EUVIMEDEUVIMED entdecken
Crossref

Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement

Jiashu Yao, Heyan Huang, Shuang Zeng, Chuwei Luo, Wangjie You, Jie Tang, Qingsong Liu, Yuhang Guo, Yangyang Kang

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (40) · 34405-34413

DOI: 10.1609/aaai.v40i40.40738 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Jiashu Yao und 8 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Through reinforcement learning (RL) with outcome correctness rewards, large reasoning models (LRMs) with scaled inference computation have demonstrated substantial success on complex reasoning tasks. However, the one-sided reward, focused solely on final correctness, limits its ability to provide detailed supervision over internal reasoning process. This deficiency leads to suboptimal internal reasoning quality, manifesting as issues like over-thinking, under-thinking, redundant-thinking, and disordered-thinking. Inspired by the recent progress in LRM self-rewarding, we introduce self-rewriting framework, where a model rewrites its own reasoning texts, and subsequently learns from the rewritten reasoning to improve the internal thought process quality. For algorithm design, we propose a selective rewriting approach wherein only "simple" samples, defined by the model's consistent correctness, are rewritten, thereby preserving all original reward signals of GRPO. For practical implementation, we compile rewriting and vanilla generation within one single batch, maintaining the scalability of the RL algorithm and introducing only 10% overhead. Extensive experiments on diverse tasks with different model sizes validate the effectiveness of self-rewriting. In terms of the accuracy-length tradeoff, the self-rewriting approach achieves improved accuracy (+0.6) with substantially shorter reasoning (-46%) even without explicit instructions in rewriting prompts to reduce reasoning length, outperforming existing strong baselines. In terms of internal reasoning quality, self-rewriting achieves significantly higher scores (+7.2) under the LLM-as-a-judge metric, successfully mitigating internal reasoning flaws.

Zitieren

APA

Yao, J., Huang, H., Zeng, S., Luo, C., You, W., Tang, J., Liu, Q., Guo, Y., & Kang, Y. (2026). Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40), 34405-34413. https://doi.org/10.1609/aaai.v40i40.40738

Vancouver

Yao J, Huang H, Zeng S, Luo C, You W, Tang J, et al. Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(40):34405-34413. doi:10.1609/aaai.v40i40.40738

BibTeX
@article{Yao2026,
  author = {Yao, Jiashu and Huang, Heyan and Zeng, Shuang and Luo, Chuwei and You, Wangjie and Tang, Jie and Liu, Qingsong and Guo, Yuhang and Kang, Yangyang},
  title = {Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {40},
  pages = {34405--34413},
  doi = {10.1609/aaai.v40i40.40738}
}
Kostenlos starten →