TLDR: A new research paper introduces RLAD, a method that trains large language models (LLMs) to discover and utilize ‘reasoning abstractions’—concise natural language hints—to solve complex problems. This two-player reinforcement learning framework, involving an abstraction generator and a solution generator, significantly improves LLM performance on math reasoning benchmarks by guiding models to explore diverse and effective solution strategies, leading to more efficient and accurate problem-solving.
Large Language Models (LLMs) have shown impressive capabilities, but when it comes to complex reasoning problems, they often struggle. Their typical approach, known as ‘chains of thought,’ can become overly verbose, repetitive, and sometimes lead to dead ends, a phenomenon the researchers refer to as ‘degenerate exploration.’ This makes it difficult for LLMs to consistently identify and apply the core ‘algorithmic procedures’ needed to solve challenging tasks.
A new research paper, titled “RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems,” introduces an innovative approach to tackle this issue. The authors, Yuxiao Qu, Anikait Singh, Yoonho Lee, Amrith Setlur, Ruslan Salakhutdinov, Chelsea Finn, and Aviral Kumar, propose the concept of ‘reasoning abstractions.’ These are concise, natural language descriptions of procedural and factual knowledge that act like high-level hints, guiding the LLM towards more successful and efficient reasoning.
What are Reasoning Abstractions?
Imagine you’re solving a complex math problem. Instead of blindly trying every possible step, you might recall a useful formula, a common pitfall to avoid, or a strategic way to reframe the problem. These are essentially reasoning abstractions. In the context of LLMs, these abstractions are short, textual insights that summarize useful procedures, intermediate results, or shared problem-solving strategies. They help the model explore a wider range of solution strategies, rather than getting stuck on a single, potentially flawed path.
How RLAD Works: A Two-Player Game
The core of this new method, RLAD (Reinforcement Learning with Abstraction Discovery), involves a unique two-player reinforcement learning setup. It jointly trains two distinct LLMs:
1. An Abstraction Generator: This model learns to propose one or more reasoning abstractions when presented with a problem. It’s rewarded based on how much these abstractions improve the accuracy of the second model.
2. An Abstraction-Conditioned Solution Generator: This model then uses the proposed abstractions as guidance to generate a solution. It’s rewarded for solving the problem accurately while effectively utilizing the information from the abstraction.
This cooperative training process allows for structured exploration of the solution space and helps decouple the learning signals for proposing good abstractions and generating correct solutions. The abstraction generator is initially ‘warmstarted’ by summarizing multiple candidate solutions from stronger models, ensuring it learns to produce meaningful hints from the outset.
Key Findings and Benefits
The researchers evaluated RLAD on several challenging math reasoning benchmarks, including AIME 2025, DeepScaleR Hard, and AMC 2023. The results were significant:
- RLAD consistently outperformed state-of-the-art reinforcement learning approaches that don’t use abstractions, showing an average 44% improvement on AIME 2025.
- Even when no abstraction was provided during inference, models trained with RLAD showed improved general reasoning capabilities, suggesting that exposure to diverse abstractions during training enhances the model’s overall problem-solving skills.
- A crucial insight was found in how to allocate computational resources. When faced with a large inference budget, it was more beneficial to generate a diversity of abstractions rather than simply sampling more solutions from a single approach. This highlights abstractions as a new, effective way to scale test-time compute.
- Solutions generated with different abstractions were found to be semantically more diverse, indicating that abstractions successfully guide the model to explore distinct reasoning paths. The solution generator also showed high adherence to the guidance provided by the abstractions.
The study also demonstrated that abstractions generated by a comparatively weaker model could effectively transfer and improve the performance of a stronger solution generator, showcasing a ‘weak-to-strong’ generalization capability.
Also Read:
- ROVER: A Minimalist Method for High-Quality and Diverse LLM Reasoning
- DIVER: A New Approach to Enhance LLM Reasoning Through Diverse Exploration
Looking Ahead
The introduction of reasoning abstractions and the RLAD training framework offers a promising new direction for enhancing LLM reasoning. It provides a complementary axis for scaling model performance, especially as traditional methods like increasing chain-of-thought length begin to show diminishing returns. Future work will explore training a single model to both propose and utilize abstractions, and further investigate how training with abstractions improves performance even when no abstractions are explicitly used during evaluation. You can read the full research paper here: RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems.
