EUVIMEDEUVIMED entdecken
Crossref

Enhance Vision-Language Alignment with Noise

Sida Huang, Hongyuan Zhang, Xuelong Li

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (16) · 17449-17457

DOI: 10.1609/aaai.v39i16.33918 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Sida Huang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 43-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the advancement of pre-trained vision-language (VL) models, enhancing the alignment between visual and linguistic modalities in downstream tasks has emerged as a critical challenge. Different from existing fine-tuning methods that add extra modules to these two modalities, we investigate whether the frozen model can be fine-tuned by customized noise. Our approach is motivated by the scientific study of beneficial noise, namely Positive-incentive Noise (Pi-noise) , which quantitatively analyzes the impact of noise. It therefore implies a new scheme to learn beneficial noise distribution that can be employed to fine-tune VL models. Focusing on few-shot classification tasks based on CLIP, we reformulate the inference process of CLIP and apply variational inference, demonstrating how to generate Pi-noise towards visual and linguistic modalities. Then, we propose Positive-incentive Noise Injector (PiNI), which can fine-tune CLIP via injecting noise into both visual and text encoders. Since the proposed method can learn the distribution of beneficial noise, we can obtain more diverse embeddings of vision and language to better align these two modalities for specific downstream tasks within limited computational resources. We evaluate different noise incorporation approaches and network architectures of PiNI. The evaluation across 11 datasets demonstrates its effectiveness.

Zitieren

APA

Huang, S., Zhang, H., & Li, X. (2025). Enhance Vision-Language Alignment with Noise. Proceedings of the AAAI Conference on Artificial Intelligence, 39(16), 17449-17457. https://doi.org/10.1609/aaai.v39i16.33918

Vancouver

Huang S, Zhang H, Li X. Enhance Vision-Language Alignment with Noise. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(16):17449-17457. doi:10.1609/aaai.v39i16.33918

BibTeX
@article{Huang2025,
  author = {Huang, Sida and Zhang, Hongyuan and Li, Xuelong},
  title = {Enhance Vision-Language Alignment with Noise},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {16},
  pages = {17449--17457},
  doi = {10.1609/aaai.v39i16.33918}
}
Kostenlos starten →