Enhance Vision-Language Alignment with Noise
Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (16) · 17449-17457
DOI: 10.1609/aaai.v39i16.33918 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Sida Huang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 43-mal zitiert (Crossref).
Abstract (Originalsprache)
With the advancement of pre-trained vision-language (VL) models, enhancing the alignment between visual and linguistic modalities in downstream tasks has emerged as a critical challenge. Different from existing fine-tuning methods that add extra modules to these two modalities, we investigate whether the frozen model can be fine-tuned by customized noise. Our approach is motivated by the scientific study of beneficial noise, namely Positive-incentive Noise (Pi-noise) , which quantitatively analyzes the impact of noise. It therefore implies a new scheme to learn beneficial noise distribution that can be employed to fine-tune VL models. Focusing on few-shot classification tasks based on CLIP, we reformulate the inference process of CLIP and apply variational inference, demonstrating how to generate Pi-noise towards visual and linguistic modalities. Then, we propose Positive-incentive Noise Injector (PiNI), which can fine-tune CLIP via injecting noise into both visual and text encoders. Since the proposed method can learn the distribution of beneficial noise, we can obtain more diverse embeddings of vision and language to better align these two modalities for specific downstream tasks within limited computational resources. We evaluate different noise incorporation approaches and network architectures of PiNI. The evaluation across 11 datasets demonstrates its effectiveness.
Zitieren
APA
Huang, S., Zhang, H., & Li, X. (2025). Enhance Vision-Language Alignment with Noise. Proceedings of the AAAI Conference on Artificial Intelligence, 39(16), 17449-17457. https://doi.org/10.1609/aaai.v39i16.33918
Vancouver
Huang S, Zhang H, Li X. Enhance Vision-Language Alignment with Noise. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(16):17449-17457. doi:10.1609/aaai.v39i16.33918
BibTeX
@article{Huang2025,
author = {Huang, Sida and Zhang, Hongyuan and Li, Xuelong},
title = {Enhance Vision-Language Alignment with Noise},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2025},
volume = {39},
number = {16},
pages = {17449--17457},
doi = {10.1609/aaai.v39i16.33918}
}
EUVIMED