EUVIMEDEUVIMED entdecken
Crossref

Vocabulary-Free Image Classification and Semantic Segmentation

Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, Elisa Ricci

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-06 · 48 (6) · 6127-6138

DOI: 10.1109/tpami.2026.3657989 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-06. Verfasst von Alessandro Conti und 5 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 79 Literaturangaben. Gefördert von: EU Projects ELIAS, ELLIOT, FAIR - Future AI Research.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large vision-language models revolutionized image classification and semantic segmentation paradigms. However, they typically assume a pre-defined set of categories, or vocabulary, at test time for composing textual prompts. This assumption is impractical in scenarios with unknown or evolving semantic context. Here, we address this issue and introduce the Vocabulary-free Image Classification (VIC) task, which aims to assign a class from an unconstrained language-induced semantic space to an input image without needing a known vocabulary. VIC is challenging due to the vastness of the semantic space, which contains millions of concepts, including fine-grained categories. To address VIC, we propose Category Search from External Databases (CaSED), a training-free method that leverages a pre-trained vision-language model and an external database. CaSED first extracts the set of candidate categories from the most semantically similar captions in the database and then assigns the image to the best-matching candidate category according to the same vision-language model. Furthermore, we demonstrate that CaSED can be applied locally to generate a coarse segmentation mask that classifies image regions, introducing the task of Vocabulary-free Semantic Segmentation. CaSED and its variants outperform other more complex vision-language models, on classification and semantic segmentation benchmarks, while using much fewer parameters.

Förderung

  • EU Projects ELIAS
  • ELLIOT
  • FAIR - Future AI Research
  • European Commission

Zitieren

APA

Conti, A., Fini, E., Mancini, M., Rota, P., Wang, Y., & Ricci, E. (2026). Vocabulary-Free Image Classification and Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(6), 6127-6138. https://doi.org/10.1109/tpami.2026.3657989

Vancouver

Conti A, Fini E, Mancini M, Rota P, Wang Y, Ricci E. Vocabulary-Free Image Classification and Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(6):6127-6138. doi:10.1109/tpami.2026.3657989

BibTeX
@article{Conti2026,
  author = {Conti, Alessandro and Fini, Enrico and Mancini, Massimiliano and Rota, Paolo and Wang, Yiming and Ricci, Elisa},
  title = {Vocabulary-Free Image Classification and Semantic Segmentation},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {6},
  pages = {6127--6138},
  doi = {10.1109/tpami.2026.3657989}
}
Kostenlos starten →