EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation

Yang Xu, Wanxiang Che

Electronics · 06.04.2026 · 15 (7) · 1533

DOI: 10.3390/electronics15071533 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 06.04.2026. Verfasst von Yang Xu und 1 weiteren Autor:innen. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Mainstream Large Language Models (LLMs) work under the paradigm of Next-Token Prediction (NTP). Multi-Token Prediction (MTP) is motivated by higher decoding efficiency, extending NTP to enable LLMs to draft multiple tokens during each forward pass. However, existing MTP approaches pretrain MTP along with the target LLM, making it difficult to unlock MTP for LLMs without official support. In this work, we propose a post-hoc approach to training an MTP module for a target LLM, providing an efficient way to evolve the LLM from NTP to MTP. The proposed approach features two main characteristics. (1) No changes to the target LLM, since it is frozen during MTP training. (2) Efficient MTP training via self-distillation from the target LLM’s native NTP capability. Results show that our approach can post-hoc train a performant MTP module via lightweight pretraining.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Xu, Y., & Che, W. (2026). Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation. Electronics, 15(7), 1533. https://doi.org/10.3390/electronics15071533

Vancouver

Xu Y, Che W. Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation. Electronics. 2026;15(7):1533. doi:10.3390/electronics15071533

BibTeX
@article{Xu2026,
  author = {Xu, Yang and Che, Wanxiang},
  title = {Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {7},
  pages = {1533},
  doi = {10.3390/electronics15071533}
}
Kostenlos starten →