Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation
Electronics · 06.04.2026 · 15 (7) · 1533
DOI: 10.3390/electronics15071533 ↗Kurz gefasst
Fachartikel in der Zeitschrift Electronics, online erschienen am 06.04.2026. Verfasst von Yang Xu und 1 weiteren Autor:innen. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Mainstream Large Language Models (LLMs) work under the paradigm of Next-Token Prediction (NTP). Multi-Token Prediction (MTP) is motivated by higher decoding efficiency, extending NTP to enable LLMs to draft multiple tokens during each forward pass. However, existing MTP approaches pretrain MTP along with the target LLM, making it difficult to unlock MTP for LLMs without official support. In this work, we propose a post-hoc approach to training an MTP module for a target LLM, providing an efficient way to evolve the LLM from NTP to MTP. The proposed approach features two main characteristics. (1) No changes to the target LLM, since it is frozen during MTP training. (2) Efficient MTP training via self-distillation from the target LLM’s native NTP capability. Results show that our approach can post-hoc train a performant MTP module via lightweight pretraining.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Xu, Y., & Che, W. (2026). Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation. Electronics, 15(7), 1533. https://doi.org/10.3390/electronics15071533
Vancouver
Xu Y, Che W. Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation. Electronics. 2026;15(7):1533. doi:10.3390/electronics15071533
BibTeX
@article{Xu2026,
author = {Xu, Yang and Che, Wanxiang},
title = {Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation},
journal = {Electronics},
year = {2026},
volume = {15},
number = {7},
pages = {1533},
doi = {10.3390/electronics15071533}
}
EUVIMED