EUVIMEDEUVIMED entdecken
Crossref

Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models

Xiao Cui, Mo Zhu, Yulei Qin, Liang Xie, Wengang Zhou, Houqiang Li

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (22) · 23724-23732

DOI: 10.1609/aaai.v39i22.34543 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Xiao Cui und 5 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge distillation (KD) has become a prevalent technique for compressing large language models (LLMs). Existing KD methods are constrained by the need for identical tokenizers (i.e., vocabularies) between teacher and student models, limiting their versatility in handling LLMs of different architecture families. In this paper, we introduce the Multi-Level Optimal Transport (MultiLevelOT), a novel approach that advances the optimal transport for universal cross-tokenizer knowledge distillation. Our method aligns the logit distributions of the teacher and the student at both token and sequence levels using diverse cost matrices, eliminating the need for dimensional or token-by-token correspondence. At the token level, MultiLevelOT integrates both global and local information by jointly optimizing all tokens within a sequence to enhance robustness. At the sequence level, we efficiently capture complex distribution structures of logits via the Sinkhorn distance, which approximates the Wasserstein distance for divergence measures. Extensive experiments on tasks such as extractive QA, generative QA, and summarization demonstrate that the MultiLevelOT outperforms state-of-the-art cross-tokenizer KD methods under various settings. Our approach is robust to different student and teacher models across model families, architectures, and parameter sizes.

Zitieren

APA

Cui, X., Zhu, M., Qin, Y., Xie, L., Zhou, W., & Li, H. (2025). Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 39(22), 23724-23732. https://doi.org/10.1609/aaai.v39i22.34543

Vancouver

Cui X, Zhu M, Qin Y, Xie L, Zhou W, Li H. Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(22):23724-23732. doi:10.1609/aaai.v39i22.34543

BibTeX
@article{Cui2025,
  author = {Cui, Xiao and Zhu, Mo and Qin, Yulei and Xie, Liang and Zhou, Wengang and Li, Houqiang},
  title = {Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {22},
  pages = {23724--23732},
  doi = {10.1609/aaai.v39i22.34543}
}
Kostenlos starten →