EUVIMEDEUVIMED entdecken
Crossref

GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, Bowen Zhou

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (41) · 34932-34940

DOI: 10.1609/aaai.v40i41.40797 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Jian Zhao und 10 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advancements in Large Language Models (LLMs) have shown that it is promising to utilize Process Reward Models (PRMs) as verifiers to enhance the performance of LLMs. However, current PRMs face three key challenges: (1) limited process supervision and generalization capabilities, (2) dependence on scalar value prediction without leveraging the generative abilities of LLMs, and (3) inability to scale the test-time compute of PRMs. In this work, we introduce GenPRM, a generative process reward model that performs explicit Chain-of-Thought (CoT) reasoning with code verification before providing judgment for each reasoning step. To obtain high-quality process supervision labels and rationale data, we propose Relative Progress Estimation (RPE) and a rationale synthesis framework that incorporates code verification. Experimental results on ProcessBench and several mathematical reasoning tasks show that GenPRM significantly outperforms prior PRMs with only 23K training data from MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally, GenPRM demonstrates strong abilities to serve as a critic model for policy model refinement. This work establishes a new paradigm for process supervision that bridges the gap between PRMs and critic models in LLMs.

Zitieren

APA

Zhao, J., Liu, R., Zhang, K., Zhou, Z., Gao, J., Li, D., Lyu, J., Qian, Z., Qi, B., Li, X., & Zhou, B. (2026). GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(41), 34932-34940. https://doi.org/10.1609/aaai.v40i41.40797

Vancouver

Zhao J, Liu R, Zhang K, Zhou Z, Gao J, Li D, et al. GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(41):34932-34940. doi:10.1609/aaai.v40i41.40797

BibTeX
@article{Zhao2026,
  author = {Zhao, Jian and Liu, Runze and Zhang, Kaiyan and Zhou, Zhimu and Gao, Junqi and Li, Dong and Lyu, Jiafei and Qian, Zhouyi and Qi, Biqing and Li, Xiu and Zhou, Bowen},
  title = {GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {41},
  pages = {34932--34940},
  doi = {10.1609/aaai.v40i41.40797}
}
Kostenlos starten →