VGPO-MCTS: Distilling Step-Level Supervision from Value-Guided Tree Search for Mathematical Reasoning
AI · 17.04.2026 · 7 (4) · 146
DOI: 10.3390/ai7040146 ↗Kurz gefasst
Fachartikel in der Zeitschrift AI, online erschienen am 17.04.2026. Verfasst von Pin Wu und 2 weiteren Autor:innen. Die Arbeit nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Large language models (LLMs) are increasingly used in applied intelligent systems, but mid-sized models still lag on mathematical reasoning, partly because reliable step-level supervision is scarce. Many existing remedies rely on costly human annotation, stronger teacher models, or heavy training pipelines, which limits practical adoption. We propose VGPO-MCTS (Value-Guided Group-wise Policy Optimization over Monte Carlo Tree Search), a search-and-distillation framework that constructs reusable step-level supervision from datasets that provide only problems and final answers. VGPO-MCTS augments a frozen backbone with (i) a lightweight value model that scores candidate reasoning states formed by a reasoning prefix and its candidate next step, and (ii) a policy updated with parameter-efficient adaptation. During search, the value model guides tree expansion and selection, while verified outcomes are propagated backward to correct node utilities. The corrected search trees are then distilled into two complementary datasets: a value regression dataset for value learning and group-wise sibling candidate sets for GRPO-style policy optimization. Experiments on GSM8K and the MATH dataset with ChatGLM3-6B and SciGLM-6B show stable round-wise improvements in final-answer exact match under a lightweight adaptation setting. After three rounds of self-training, the proposed framework improves performance by about 6.3 percentage points on GSM8K and about 3.9 percentage points on MATH across the two backbones.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Wu, P., Zhu, Y., & Wang, H. (2026). VGPO-MCTS: Distilling Step-Level Supervision from Value-Guided Tree Search for Mathematical Reasoning. AI, 7(4), 146. https://doi.org/10.3390/ai7040146
Vancouver
Wu P, Zhu Y, Wang H. VGPO-MCTS: Distilling Step-Level Supervision from Value-Guided Tree Search for Mathematical Reasoning. AI. 2026;7(4):146. doi:10.3390/ai7040146
BibTeX
@article{Wu2026,
author = {Wu, Pin and Zhu, Yufei and Wang, Huiyan},
title = {VGPO-MCTS: Distilling Step-Level Supervision from Value-Guided Tree Search for Mathematical Reasoning},
journal = {AI},
year = {2026},
volume = {7},
number = {4},
pages = {146},
doi = {10.3390/ai7040146}
}
EUVIMED