EUVIMEDEUVIMED entdecken
Crossref

An Overview of Low-Rank Structures in the Training and Adaptation of Large Models: From Implicit Low-Dimensionality to Efficient Training and Fine-Tuning [Special Issue on the Mathematics of Deep Learning]

Laura Balzano, Tianjiao Ding, Benjamin D. Haeffele, Soo Min Kwon, Qing Qu, Peng Wang, Zhangyang Wang, Can Yaras

IEEE Signal Processing Magazine · 2026-05 · 43 (3) · 52-70

DOI: 10.1109/msp.2026.3666749 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Signal Processing Magazine, 2026-05. Verfasst von Laura Balzano und 7 weiteren Autor:innen. Die Arbeit nennt 42 Literaturangaben. Gefördert von: University of Macau.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The advent of deep learning has immeasurably changed the ways we process data in signal processing and machine learning. However, training and deploying modern deep learning models demand substantial computational resources, raising concerns about exorbitant training costs, GPU shortages, and heightened energy consumption. Several lines of research over the last decade have explored the emergence of low-dimensional structures during the training process, where basic elements such as weight matrices and representations tend to be approximately low rank even though not explicitly trained to be. These low-dimensional structures arise in part due to the implicit bias of the methods used to train deep networks, providing the potential to partially explain why deep models need fewer samples than the number of model parameters. This implicit low dimensionality has inspired the exploration of low-rank structures in training and fine-tuning large-scale deep learning models more efficiently. In this article, we review recent exciting advances in using low-rank structure in deep learning and aim to clarify the mathematical foundations underlying their design. Specifically, we highlight key insights from a rich line of research focused on theoretically understanding and leveraging low-rank structures in deep learning both at every iteration of training as well as at the global minimum.

Förderung

  • University of Macau

Zitieren

APA

Balzano, L., Ding, T., Haeffele, B. D., Kwon, S. M., Qu, Q., Wang, P., Wang, Z., & Yaras, C. (2026). An Overview of Low-Rank Structures in the Training and Adaptation of Large Models: From Implicit Low-Dimensionality to Efficient Training and Fine-Tuning [Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine, 43(3), 52-70. https://doi.org/10.1109/msp.2026.3666749

Vancouver

Balzano L, Ding T, Haeffele BD, Kwon SM, Qu Q, Wang P, et al. An Overview of Low-Rank Structures in the Training and Adaptation of Large Models: From Implicit Low-Dimensionality to Efficient Training and Fine-Tuning [Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine. 2026;43(3):52-70. doi:10.1109/msp.2026.3666749

BibTeX
@article{Balzano2026,
  author = {Balzano, Laura and Ding, Tianjiao and Haeffele, Benjamin D. and Kwon, Soo Min and Qu, Qing and Wang, Peng and Wang, Zhangyang and Yaras, Can},
  title = {An Overview of Low-Rank Structures in the Training and Adaptation of Large Models: From Implicit Low-Dimensionality to Efficient Training and Fine-Tuning [Special Issue on the Mathematics of Deep Learning]},
  journal = {IEEE Signal Processing Magazine},
  year = {2026},
  volume = {43},
  number = {3},
  pages = {52--70},
  doi = {10.1109/msp.2026.3666749}
}
Kostenlos starten →