EUVIMEDSkopri EUVIMED
Open access · CC BY 4.0

Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation

Yang Xu, Wanxiang Che

Electronics · 2026-04-06 · 15 (7) · 1533

DOI: 10.3390/electronics15071533 ↗

At a glance

Journal article in Electronics, published online on 2026-04-06. Written by Yang Xu and 1 co-authors. 44 references. The full text is freely available (open access, CC BY 4.0). Funded by: National Natural Science Foundation of China.

AstrattDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Mainstream Large Language Models (LLMs) work under the paradigm of Next-Token Prediction (NTP). Multi-Token Prediction (MTP) is motivated by higher decoding efficiency, extending NTP to enable LLMs to draft multiple tokens during each forward pass. However, existing MTP approaches pretrain MTP along with the target LLM, making it difficult to unlock MTP for LLMs without official support. In this work, we propose a post-hoc approach to training an MTP module for a target LLM, providing an efficient way to evolve the LLM from NTP to MTP. The proposed approach features two main characteristics. (1) No changes to the target LLM, since it is frozen during MTP training. (2) Efficient MTP training via self-distillation from the target LLM’s native NTP capability. Results show that our approach can post-hoc train a performant MTP module via lightweight pretraining.

Funding

  • National Natural Science Foundation of China

Cite

APA

Xu, Y., & Che, W. (2026). Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation. Electronics, 15(7), 1533. https://doi.org/10.3390/electronics15071533

Vancouver

Xu Y, Che W. Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation. Electronics. 2026;15(7):1533. doi:10.3390/electronics15071533

BibTeX
@article{Xu2026,
  author = {Xu, Yang and Che, Wanxiang},
  title = {Evolving LLMs from Next-Token Prediction to Multi-Token Prediction via Self-Distillation},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {7},
  pages = {1533},
  doi = {10.3390/electronics15071533}
}
Start for free →