EUVIMEDEUVIMED entdecken
Crossref

Low-Rank Bayesian Structured Adaptation of Large Language Models

Deepak Kandel, Dimah Dera

Proceedings of the AAAI Symposium Series · 06.11.2026 · 10 (1) · 319-328

DOI: 10.1609/aaaiss.v10i1.43422 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Symposium Series, online erschienen am 06.11.2026. Verfasst von Deepak Kandel und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning of large language models. Still, its deterministic formulation provides only point estimates of the adapter parameters and therefore cannot represent epistemic uncertainty. Existing Bayesian LoRA methods address this limitation by learning distributions over adapter weights; however, commonly used variational posteriors fully factorize across adapter parameters, limiting them from capturing posterior correlations and potentially underestimating uncertainty. We introduce HiVE, a Bayesian LoRA method with a structured posterior. At each LoRA layer, HiVE uses a layer-specific stochastic latent parameter to introduce structured and correlated adaptation in the adapter weights. These stochastic parameters modulate the down-projection LoRA factors, increasing the posterior expressiveness and allowing it to capture correlated uncertainty among its parameters. Marginalizing the latent variable induces a diagonal-plus-low-rank covariance structure over the effective adapter weights, capturing posterior dependencies while retaining parameter efficiency. HiVE learns the resulting variational posterior end-to-end using stochastic-gradient descent and integrates into existing LoRA pipelines with only modest trainable-parameter overhead. HiVE achieves comparable or superior predictive accuracy while providing more reliable uncertainty estimates across natural-language benchmarks under both in-distribution and distribution-shift conditions.

Zitieren

APA

Kandel, D., & Dera, D. (2026). Low-Rank Bayesian Structured Adaptation of Large Language Models. Proceedings of the AAAI Symposium Series, 10(1), 319-328. https://doi.org/10.1609/aaaiss.v10i1.43422

Vancouver

Kandel D, Dera D. Low-Rank Bayesian Structured Adaptation of Large Language Models. Proceedings of the AAAI Symposium Series. 2026;10(1):319-328. doi:10.1609/aaaiss.v10i1.43422

BibTeX
@article{Kandel2026,
  author = {Kandel, Deepak and Dera, Dimah},
  title = {Low-Rank Bayesian Structured Adaptation of Large Language Models},
  journal = {Proceedings of the AAAI Symposium Series},
  year = {2026},
  volume = {10},
  number = {1},
  pages = {319--328},
  doi = {10.1609/aaaiss.v10i1.43422}
}
Kostenlos starten →