BoostStep: A New Approach to Improving Mathematical Abilities of Large Language Models

Large language models (LLMs) demonstrate impressive performance in solving complex mathematical problems, particularly through the use of divide-and-conquer strategies and in-context learning (ICL). Despite this progress, there remains potential for improvement, especially regarding the accuracy of individual calculation steps. A new research approach called BoostStep addresses this challenge and promises to significantly enhance the mathematical capabilities of LLMs.

Challenges in In-Context Learning

Current LLMs effectively master the process of problem decomposition (divide), but often fail in the precise execution of individual solution steps (conquer). Two main problems in the ICL process limit performance: granularity differences and the resulting problem of negative noise. ICL examples retrieved at the level of the entire problem often do not offer sufficient relevance for specific, demanding calculation steps. This discrepancy between the granularity of the problem and the required level of detail in the individual steps can lead to incorrect results.

BoostStep: Step-wise Learning for More Precise Results

BoostStep addresses this issue and focuses on improving the quality of individual calculation steps. The core of the approach lies in aligning the granularity between retrieving ICL examples and executing the individual calculation steps. Through a novel "first-try" strategy, BoostStep provides highly relevant ICL examples for each individual step. The model first attempts to solve the step independently. Based on this attempt, suitable examples are then specifically retrieved from a database organized by calculation steps and not by entire problems. This approach provides more relevant examples than the conventional method, which is based on the granularity of the entire problem, thus improving the quality of the individual calculation steps.

Integration with Monte Carlo Tree Search

BoostStep can be seamlessly integrated into Monte Carlo Tree Search (MCTS) methods. MCTS methods are frequently used to solve complex problems by creating and evaluating a search tree of possible solution paths. By integrating BoostStep, both the generation of solution candidates and the decision-making within the MCTS process can be improved.

Quantitative Results

Initial results show that BoostStep improves the performance of LLMs like GPT-4o and Qwen2.5-Math-72B on various mathematical benchmarks by 3.6% and 2.0%, respectively. In combination with MCTS, a performance gain of 7.5% was even achieved. These results underscore the potential of BoostStep to significantly enhance the mathematical capabilities of LLMs.

Outlook

BoostStep represents a promising approach to improving the mathematical abilities of LLMs. The targeted provision of relevant ICL examples at the level of individual calculation steps allows for more precise and robust problem solving. Integration with MCTS methods opens further possibilities for optimizing complex mathematical calculations. Future research could focus on expanding the ICL database and applying BoostStep to other application areas.

Bibliography: https://arxiv.org/abs/2501.03226 https://www.chatpaper.com/chatpaper/fr/paper/96150 https://huggingface.co/papers https://chatpaper.com/chatpaper/ja?id=3&date=1736179200&page=1 https://arxiv-sanity-lite.com/ https://arxiv.org/list/cs.CL/recent https://aclanthology.org/2024.emnlp-main.758.pdf https://aclanthology.org/2024.eacl-srw.17.pdf https://www.sciencedirect.com/science/article/pii/S2949719123000298