EUVIMEDEUVIMED entdecken
Crossref

LLM-Oriented Token-Adaptive Knowledge Distillation

Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu, Jiangning Zhang

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (40) · 34070-34078

DOI: 10.1609/aaai.v40i40.40701 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Xurong Xie und 7 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge Distillation (KD) is a key technique for compressing Large-scale Language Models (LLMs), but prevailing logit-based methods employ static strategies misaligned with the student’s dynamic learning process. By treating all tokens indiscriminately with a fixed temperature, these methods result in suboptimal knowledge transfer. To address this, we propose LLM-oriented token-Adaptive Knowledge Distillation (AdaKD), a framework that adapts the distillation process to each token’s real-time learning state. AdaKD consists of two synergistic modules driven by a unified token difficulty metric. First, the Loss-driven Adaptive Token Focusing (LATF) module dynamically concentrates distillation on valuable tokens by monitoring the student’s learning stability. Second, Inverse Difficulty Temperature Scaling (IDTS) introduces a counterintuitive token-level temperature: low for difficult tokens to target error correction, and high for easy tokens to learn the teacher’s smooth output distribution for better generalization. As a plug-and-play framework, AdaKD consistently improves performance across diverse distillation methods, model architectures, and benchmarks.

Zitieren

APA

Xie, X., Xue, Z., Wu, J., Li, J., Wang, Y., Hu, X., Liu, Y., & Zhang, J. (2026). LLM-Oriented Token-Adaptive Knowledge Distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40), 34070-34078. https://doi.org/10.1609/aaai.v40i40.40701

Vancouver

Xie X, Xue Z, Wu J, Li J, Wang Y, Hu X, et al. LLM-Oriented Token-Adaptive Knowledge Distillation. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(40):34070-34078. doi:10.1609/aaai.v40i40.40701

BibTeX
@article{Xie2026,
  author = {Xie, Xurong and Xue, Zhucun and Wu, Jiafu and Li, Jian and Wang, Yabiao and Hu, Xiaobin and Liu, Yong and Zhang, Jiangning},
  title = {LLM-Oriented Token-Adaptive Knowledge Distillation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {40},
  pages = {34070--34078},
  doi = {10.1609/aaai.v40i40.40701}
}
Kostenlos starten →