EUVIMEDEUVIMED entdecken
Crossref

Text to Image for Multi-Label Image Recognition With Joint Prompt-Adapter Learning

Chun-Mei Feng, Kai Yu, Xinxing Xu, Salman Khan, Rick Siow Mong Goh, Wangmeng Zuo, Yong Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-09 · 47 (9) · 7660-7674

DOI: 10.1109/tpami.2025.3573852 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-09. Verfasst von Chun-Mei Feng und 6 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 54 Literaturangaben. Gefördert von: Agency for Science, Technology and Research, AME Programmatic Funding Scheme, National Research Foundation (NRF) Singapore.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Benefited from image-text contrastive learning, pre-trained vision-language models, e.g., CLIP, allow to direct leverage texts as images (TaI) for parameter-efficient fine-tuning (PEFT). While CLIP is capable of making image features to be similar to the corresponding text features, the modality gap remains a nontrivial issue and limits image recognition performance of TaI. Using multi-label image recognition (MLR) as an example, we present a novel method, called T2I-PAL to tackle the modality gap issue when using only text captions for PEFT. The core design of T2I-PAL is to leverage pre-trained text-to-image generation models to generate photo-realistic and diverse images from text captions, thereby reducing the modality gap. To further enhance MLR, T2I-PAL incorporates a class-wise heatmap and learnable prototypes. This aggregates local similarities, making the representation of local visual features more robust and informative for multi-label recognition. For better PEFT, we further combine both prompt tuning and adapter learning to enhance classification performance. T2I-PAL offers significant advantages: it eliminates the need for fully semantically annotated training images, thereby reducing the manual annotation workload, and it preserves the intrinsic mode of the CLIP model, allowing for seamless integration with any existing CLIP framework. Extensive experiments on multiple benchmarks, including MS-COCO, VOC2007, and NUS-WIDE, show that our T2I-PAL can boost recognition performance by 3.47% in average above the top-ranked state-of-the-art methods.

Förderung

  • Agency for Science, Technology and Research
  • AME Programmatic Funding Scheme
  • National Research Foundation (NRF) Singapore
  • AI Singapore Programme AISG
  • RIE2020 Health and Biomedical Sciences
  • Industry Alignment Fund Pre-Positioning
  • A*STAR Central Research Fund

Zitieren

APA

Feng, C. M., Yu, K., Xu, X., Khan, S., Goh, R. S. M., Zuo, W., & Liu, Y. (2025). Text to Image for Multi-Label Image Recognition With Joint Prompt-Adapter Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(9), 7660-7674. https://doi.org/10.1109/tpami.2025.3573852

Vancouver

Feng CM, Yu K, Xu X, Khan S, Goh RSM, Zuo W, et al. Text to Image for Multi-Label Image Recognition With Joint Prompt-Adapter Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(9):7660-7674. doi:10.1109/tpami.2025.3573852

BibTeX
@article{Feng2025,
  author = {Feng, Chun-Mei and Yu, Kai and Xu, Xinxing and Khan, Salman and Goh, Rick Siow Mong and Zuo, Wangmeng and Liu, Yong},
  title = {Text to Image for Multi-Label Image Recognition With Joint Prompt-Adapter Learning},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {9},
  pages = {7660--7674},
  doi = {10.1109/tpami.2025.3573852}
}
Kostenlos starten →