EUVIMEDEUVIMED entdecken
Crossref

LYRA: Low-Frequency Rank Adaptation via Factored DCT Coefficients for Parameter Efficient Fine Tuning of Transformers

Sayed Muhsin, Seok-Bum Ko

IEEE Signal Processing Letters · 2026 · 33 · 3073-3077

DOI: 10.1109/lsp.2026.3714737 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Signal Processing Letters, 2026. Verfasst von Sayed Muhsin und 1 weiteren Autor:innen. Die Arbeit nennt 23 Literaturangaben. Gefördert von: Industrial Technology Innovation Program of the Ministry of Trade, Industry and Energy, Republic of Korea, NSERC Discovery.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As pretrained language models grow, full parameter updates become prohibitively expensive, motivating parameter-efficient fine-tuning (PEFT). LoRA decomposes weight updates into low-rank factors, but its per-module cost $r(m{+}n)$ imposes a structural floor at rank one. FourierFT breaks this floor via frequency-domain coefficients, yet requires a dense inverse transform at each forward step. We propose LYRA (Low-frequencY Rank Adaptation), which parameterizes each weight update with a small set of low-frequency 2D discrete cosine transform coefficients, chosen separately along each axis and contiguous in its canonical form. This separable structure admits a factored forward pass of three small matrix multiplications, avoiding dense reconstruction entirely. On GLUE and SuperGLUE with the BERT-base and RoBERTa-base encoders, LYRA matches FourierFT at identical parameter and optimizer-memory budgets while achieving the lowest peak GPU memory among all methods tested.

Förderung

  • Industrial Technology Innovation Program of the Ministry of Trade, Industry and Energy
  • Republic of Korea
  • NSERC Discovery

Zitieren

APA

Muhsin, S., & Ko, S. (2026). LYRA: Low-Frequency Rank Adaptation via Factored DCT Coefficients for Parameter Efficient Fine Tuning of Transformers. IEEE Signal Processing Letters, 33, 3073-3077. https://doi.org/10.1109/lsp.2026.3714737

Vancouver

Muhsin S, Ko S. LYRA: Low-Frequency Rank Adaptation via Factored DCT Coefficients for Parameter Efficient Fine Tuning of Transformers. IEEE Signal Processing Letters. 2026;33:3073-3077. doi:10.1109/lsp.2026.3714737

BibTeX
@article{Muhsin2026,
  author = {Muhsin, Sayed and Ko, Seok-Bum},
  title = {LYRA: Low-Frequency Rank Adaptation via Factored DCT Coefficients for Parameter Efficient Fine Tuning of Transformers},
  journal = {IEEE Signal Processing Letters},
  year = {2026},
  volume = {33},
  pages = {3073--3077},
  doi = {10.1109/lsp.2026.3714737}
}
Kostenlos starten →