Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-06 · 48 (6) · 6107-6126
DOI: 10.1109/tpami.2026.3657354 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-06. Verfasst von Lingling Xu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 75-mal zitiert (Crossref) und nennt 128 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Medical Text Feature Representations based on Pre-trained Language Models, Faculty Research, Direct Grant.
Abstract (Originalsprache)
With the continuous growth in the number of parameters of the Transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter-Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, extensive experiments are conducted using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.
Förderung
- Medical Text Feature Representations based on Pre-trained Language Models
- Faculty Research
- Direct Grant
- Lingnan University
- Research Grants Council of the Hong Kong Special Administrative Region
Zitieren
APA
Xu, L., Xie, H., Qin, S. J., Tao, X., & Wang, F. L. (2026). Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(6), 6107-6126. https://doi.org/10.1109/tpami.2026.3657354
Vancouver
Xu L, Xie H, Qin SJ, Tao X, Wang FL. Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(6):6107-6126. doi:10.1109/tpami.2026.3657354
BibTeX
@article{Xu2026,
author = {Xu, Lingling and Xie, Haoran and Qin, S Joe and Tao, Xiaohui and Wang, Fu Lee},
title = {Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
volume = {48},
number = {6},
pages = {6107--6126},
doi = {10.1109/tpami.2026.3657354}
}
EUVIMED