Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
ACM Transactions on Intelligent Systems and Technology · 31.12.2025 · 16 (6) · 1-27
DOI: 10.1145/3699518 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 24.11.2025. Verfasst von Chuanpeng Yang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 92-mal zitiert (Crossref) und nennt 169 Literaturangaben.
Abstract (Originalsprache)
Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This article presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field.
Zitieren
APA
Yang, C., Zhu, Y., Lu, W., Wang, Y., Chen, Q., Gao, C., Yan, B., & Chen, Y. (2025). Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application. ACM Transactions on Intelligent Systems and Technology, 16(6), 1-27. https://doi.org/10.1145/3699518
Vancouver
Yang C, Zhu Y, Lu W, Wang Y, Chen Q, Gao C, et al. Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application. ACM Transactions on Intelligent Systems and Technology. 2025;16(6):1-27. doi:10.1145/3699518
BibTeX
@article{Yang2025,
author = {Yang, Chuanpeng and Zhu, Yao and Lu, Wang and Wang, Yidong and Chen, Qian and Gao, Chenlong and Yan, Bingjie and Chen, Yiqiang},
title = {Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application},
journal = {ACM Transactions on Intelligent Systems and Technology},
year = {2025},
volume = {16},
number = {6},
pages = {1--27},
doi = {10.1145/3699518}
}
EUVIMED