EUVIMEDEUVIMED entdecken
Crossref

HALoRA: Low-Rank Adaptation with Hierarchical Budget Allocation for Efficient Vision-Language Alignment

Letian Zhang, GuangHao Meng, XuDong Ren, Jinpeng Wang

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (33) · 28283-28291

DOI: 10.1609/aaai.v40i33.40056 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Letian Zhang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the emergence of large multimodal models, dual-encoder alignment via contrastive learning has seen a resurgence. However, the escalating model size demands effective Parameter-Efficient Fine-Tuning (PEFT). While LoRA is a promising inference-free alternative to adapters, we find that its naive application to multimodal tasks causes a severe rank imbalance, favoring the text modality and FFN layers. To address this, we propose HALoRA (Hierarchical Allocation LoRA), which introduces a component-wise budget allocator to ensure balanced fine-tuning across both modalities and their internal components. This is complemented by a gradient-approximated initialization to accelerate convergence. With only half the parameters of adapters, HALoRA achieves superior or competitive performance in retrieval and zero-shot classification. Our work presents a more principled approach to multimodal LoRA, uncovering an intriguing asymmetry in vision-language alignment.

Zitieren

APA

Zhang, L., Meng, G., Ren, X., & Wang, J. (2026). HALoRA: Low-Rank Adaptation with Hierarchical Budget Allocation for Efficient Vision-Language Alignment. Proceedings of the AAAI Conference on Artificial Intelligence, 40(33), 28283-28291. https://doi.org/10.1609/aaai.v40i33.40056

Vancouver

Zhang L, Meng G, Ren X, Wang J. HALoRA: Low-Rank Adaptation with Hierarchical Budget Allocation for Efficient Vision-Language Alignment. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(33):28283-28291. doi:10.1609/aaai.v40i33.40056

BibTeX
@article{Zhang2026,
  author = {Zhang, Letian and Meng, GuangHao and Ren, XuDong and Wang, Jinpeng},
  title = {HALoRA: Low-Rank Adaptation with Hierarchical Budget Allocation for Efficient Vision-Language Alignment},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {33},
  pages = {28283--28291},
  doi = {10.1609/aaai.v40i33.40056}
}
Kostenlos starten →