EUVIMEDEUVIMED entdecken
Crossref

Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

Chuanpeng Yang, Yao Zhu, Wang Lu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, Yiqiang Chen

ACM Transactions on Intelligent Systems and Technology · 31.12.2025 · 16 (6) · 1-27

DOI: 10.1145/3699518 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 24.11.2025. Verfasst von Chuanpeng Yang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 92-mal zitiert (Crossref) und nennt 169 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This article presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field.

Zitieren

APA

Yang, C., Zhu, Y., Lu, W., Wang, Y., Chen, Q., Gao, C., Yan, B., & Chen, Y. (2025). Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application. ACM Transactions on Intelligent Systems and Technology, 16(6), 1-27. https://doi.org/10.1145/3699518

Vancouver

Yang C, Zhu Y, Lu W, Wang Y, Chen Q, Gao C, et al. Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application. ACM Transactions on Intelligent Systems and Technology. 2025;16(6):1-27. doi:10.1145/3699518

BibTeX
@article{Yang2025,
  author = {Yang, Chuanpeng and Zhu, Yao and Lu, Wang and Wang, Yidong and Chen, Qian and Gao, Chenlong and Yan, Bingjie and Chen, Yiqiang},
  title = {Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2025},
  volume = {16},
  number = {6},
  pages = {1--27},
  doi = {10.1145/3699518}
}
Kostenlos starten →