Process Reward Models in Mathematical Reasoning: Insights from Development

Process reward models (PRMs) are gaining traction as a promising approach for process control in the mathematical reasoning of large language models (LLMs). The goal is to identify and minimize intermediate errors in the reasoning processes. However, the development of effective PRMs faces significant challenges, particularly regarding data annotation and evaluation methods.

A common method for data synthesis for PRMs is Monte Carlo (MC) estimation. Studies show, however, that this often leads to poorer performance and generalization compared to LLM-based evaluation methods and human annotation. MC estimation relies on completion models to assess the correctness of individual steps, which can lead to inaccurate step verification.

Furthermore, conventional best-of-N (BoN) evaluation strategies for PRMs exhibit potential biases:

- Unreliable policy models generate answers with correct results but faulty processes, leading to a discrepancy between the evaluation criteria of BoN and the PRM goals of process verification. - The tolerance of PRMs to such answers leads to inflated BoN scores. - Existing PRMs show a significant concentration of minimal values on the final answer steps, revealing a shift from process-based to result-based evaluation in BoN-optimized PRMs.

To address these challenges, consensus filter mechanisms have been developed that effectively integrate MC estimation with LLM-based evaluations. In addition, a more comprehensive evaluation framework is advocated, combining both answer-level and step-level metrics. These mechanisms significantly improve both model performance and data efficiency in BoN evaluation and stepwise error identification.

New Approaches and Open-Source Models

Recent research in the field of PRMs has led to new, high-performing models available as open-source solutions. These models, trained on extensive datasets and using advanced techniques such as the "chain-of-thought" method, achieve impressive results in mathematical reasoning processes. They demonstrate the potential of PRMs to significantly improve the capabilities of LLMs in the mathematical domain and go beyond the limitations of pure imitation or distillation.

The development of PRMs for mathematical reasoning is a dynamic research field. Current findings underscore the need to critically examine and continuously improve both data generation and evaluation methods. The combination of different approaches and the development of new metrics are crucial to further optimize the robustness and reliability of PRMs and to fully exploit the potential of LLMs in mathematical reasoning.

Bibliography: Zhang, Z., Zheng, C., Wu, Y., Zhang, B., Lin, R., Yu, B., Liu, D., Zhou, J., & Lin, J. (2025). The Lessons of Developing Process Reward Models in Mathematical Reasoning. arXiv preprint arXiv:2501.07301. Ma, Y., Chen, Z., Liu, T., Tian, M., Liu, Z., Liu, Z., & Luo, W. (2024). What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning. arXiv preprint arXiv:2412.15904. Luo, L., Liu, Y., Liu, R., Phatale, S., Guo, M., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., & Rastogi, A. (2024). Improve Mathematical Reasoning in Language Models by Automated Process Supervision. arXiv preprint arXiv:2406.06592. Evaluating Robustness of Reward Models for Mathematical Reasoning. ICLR 2025 Conference Submission. https://openreview.net/forum?id=0er6aOyXUD Evaluating Robustness of Reward Models for Mathematical Reasoning. Supplementary Material. https://openreview.net/pdf/f259c2f7d81edeb81143c5174e95afd08a930d7b.pdf Park, J., Lee, H., Kim, S., Ryu, S., & Lee, J. (2024). Is Your Math-Solving LLM a Real Reasoner? Benchmarking Reasoning Ability of LLMs on Math Word Problems. Findings of EMNLP, 1023–1033. https://aclanthology.org/2024.findings-emnlp.78.pdf Evaluating Robustness of Reward Models for Mathematical Reasoning. https://www.researchgate.net/publication/384599537_Evaluating_Robustness_of_Reward_Models_for_Mathematical_Reasoning Mathematical Reasoning. Papers with Code. https://paperswithcode.com/task/mathematical-reasoning/codeless?page=2&q= Dhingra, B., Soltani, S., Lazaridou, A., & Lewis, M. (2024). Towards Generalisable Neuro-Symbolic Reasoning with Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7998–8022. https://aclanthology.org/2024.acl-long.510.pdf Noguer I Alonso, M. (2023). Large Language Models Reasoning and Reinforcement Learning. Available at SSRN 4656090. https://papers.ssrn.com/sol3/Delivery.cfm/SSRN_ID4672606_code4425638.pdf?abstractid=4656090 Ansari, S. (2025, January 4). PRIME: An Open-Source Solution for Online Reinforcement Learning with Process Rewards to Advance Reasoning Abilities of Language Models Beyond Imitation or Distillation. MarkTechPost. https://www.marktechpost.com/2025/01/04/prime-an-open-source-solution-for-online-reinforcement-learning-with-process-rewards-to-advance-reasoning-abilities-of-language-models-beyond-imitation-or-distillation/