EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Probing the Pitfalls: Understanding SVD’s Shortcomings in Language Model Compression

Sergey Pletenev

Journal of Language and Education · 30.12.2024 · 10 (4) · 85-97

DOI: 10.17323/jle.2024.22368 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Language and Education, online erschienen am 30.12.2024. Verfasst von Sergey Pletenev. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Modern computational linguistics heavily relies on large language models that demonstrate strong performance in various Natural Language Inference (NLI) tasks. These models, however, require substantial computational resources for both training and deployment. To address this challenge, a range of compression and acceleration techniques has been developed, including quantization, pruning, and factorization. Each of these approaches operates differently, can be applied at various levels of the model architecture, and is suited to different deployment scenarios.

Purpose

The objective of this study is to analyze and evaluate a factorization-based compression technique that reduces the computational footprint of large language models while preserving their accuracy in NLI tasks, particularly for resource-constrained or latency-sensitive applications.

Method

To evaluate the impact of factorization-based compression, we conducted probing experiments. First, we chose a widely-used pre-trained model (Bert-base and Llama 2) as our baseline. Then, we applied low-rank factorization to its transformer layers using various singular value decomposition algorithms at different compression rates. After that, we used probing tasks to analyze the changes in the internal representations and linguistic knowledge of the compressed models. We compared the changes in the model's internal representations with its ability to solve natural language inference (NLI) tasks and the compression rate achieved through factorization.

Results

Naive uniform factorization often led to significant accuracy drops, even at small compression rates, reflecting a noticeable degradation in the model's ability to understand textual entailments. Probing tasks showed that these uniformly compressed models lost important syntactic and semantic information, which aligned with the performance decline we observed. However, targeted compression approaches, such as selectively compressing the most redundant parts of the model or weighting algorithms, mitigated these negative effects.

Conclusion

These results demonstrate that factorization, when used properly, can significantly reduce computational requirements while preserving the core linguistic capabilities of large language models. Our research can inform the development of future compression techniques that adapt factorization strategies to the inherent structure of models and their tasks. These insights can help deploy LLMs in scenarios with limited computational resources.

Zitieren

APA

Pletenev, S. (2024). Probing the Pitfalls: Understanding SVD’s Shortcomings in Language Model Compression. Journal of Language and Education, 10(4), 85-97. https://doi.org/10.17323/jle.2024.22368

Vancouver

Pletenev S. Probing the Pitfalls: Understanding SVD’s Shortcomings in Language Model Compression. Journal of Language and Education. 2024;10(4):85-97. doi:10.17323/jle.2024.22368

BibTeX
@article{Pletenev2024,
  author = {Pletenev, Sergey},
  title = {Probing the Pitfalls: Understanding SVD’s Shortcomings in Language Model Compression},
  journal = {Journal of Language and Education},
  year = {2024},
  volume = {10},
  number = {4},
  pages = {85--97},
  doi = {10.17323/jle.2024.22368}
}
Kostenlos starten →