EUVIMEDEUVIMED entdecken
Crossref

RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning

Shicheng Xu, Liang Pang, Yunchang Zhu, Jia Gu, Zihao Wei, Jingcheng Deng, Feiyang Pan, Huawei Shen, Xueqi Cheng

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (40) · 34151-34159

DOI: 10.1609/aaai.v40i40.40710 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Shicheng Xu und 8 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Distilling reasoning paths from teacher to student models via supervised fine-tuning (SFT) provides a shortcut for improving the reasoning ability of the smaller Large Language Models (LLMs). However, the reasoning paths generated by teacher models often reflect only surface-level traces of their underlying authentic reasoning. Insights from cognitive neuroscience suggest that authentic reasoning involves a complex interweaving between meta-reasoning that selects the appropriate sub-problem from multiple candidates, and solving, which addresses the sub-problem. It means that authentic reasoning has implicit multi-branch structure. Supervised fine-tuning collapses this rich structure into a flat sequence of token prediction in teacher's reasoning path, which cannot distill this structure to student. To address this limitation, we propose RLKD, a reinforcement learning (RL)-based distillation framework guided by a novel Generative Structure Reward Model (GSRM). Our GSRM converts the reasoning path into multiple meta-reasoning-solving steps and gives the reward to measure the alignment between the reasoning structures of student and teacher. Our RLKD combines this reward with RL, enables the student LLM to internalize the teacher’s implicit multi-branch structure in authentic reasoning, rather than merely mimicking fixed teacher's output paths. Experiments show that RLKD, even when trained on only 0.1% of the data under an RL-only regime, surpasses the performance of standard SFT-RL pipelines and further unleashes the potential reasoning ability of the student LLM than SFT-based distillation.

Zitieren

APA

Xu, S., Pang, L., Zhu, Y., Gu, J., Wei, Z., Deng, J., Pan, F., Shen, H., & Cheng, X. (2026). RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40), 34151-34159. https://doi.org/10.1609/aaai.v40i40.40710

Vancouver

Xu S, Pang L, Zhu Y, Gu J, Wei Z, Deng J, et al. RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(40):34151-34159. doi:10.1609/aaai.v40i40.40710

BibTeX
@article{Xu2026,
  author = {Xu, Shicheng and Pang, Liang and Zhu, Yunchang and Gu, Jia and Wei, Zihao and Deng, Jingcheng and Pan, Feiyang and Shen, Huawei and Cheng, Xueqi},
  title = {RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {40},
  pages = {34151--34159},
  doi = {10.1609/aaai.v40i40.40710}
}
Kostenlos starten →