TLDR: HAMMER introduces a novel reinforcement learning method for large language models that uses semantic diversity to order training samples. By creating a “Hamiltonian Curiosity Order” based on minimum semantic similarity, it encourages early exploration, prevents local optimization, and leads to more stable and faster convergence, achieving 3-4% accuracy gains on various mathematical benchmarks without requiring costly difficulty assessments.
Large Language Models (LLMs) have become incredibly powerful, especially when enhanced with Reinforcement Learning with Verifiable Rewards (RLVR) for complex reasoning tasks. However, a common challenge in training these models is instability and slow convergence, particularly in the early stages. Traditional curriculum learning methods, which often sequence training data from ‘easy-to-hard,’ can inadvertently lead to problems. By focusing too much on simple samples early on, the model might get stuck in local optima, losing its crucial ability to explore diverse solutions.
A new approach, called Hamiltonian Curiosity Augmented Large Language Model Reinforcement (HAMMER), offers an innovative solution to this problem. Instead of relying on difficulty assessments, HAMMER leverages the inherent diversity within the training data to guide the learning process. It transforms diversity, usually a static measure for dataset evaluation, into an active principle for dynamic reinforcement learning.
How HAMMER Works
HAMMER operates on two main principles. First, it uses the LLM’s own internal mechanisms to generate ‘semantic embeddings’ for each training sample. Think of these embeddings as numerical representations that capture the meaning of a sentence. By using the backbone LLM itself, HAMMER ensures that these representations are perfectly aligned with how the model understands and processes information.
Second, these semantic embeddings are then used to construct what the researchers call a ‘Hamiltonian Curiosity Order.’ Imagine all your training samples as points in a vast semantic space. HAMMER finds a path through these points that minimizes the cumulative semantic similarity between consecutive samples. In simpler terms, it intentionally orders the training data so that the model encounters the most semantically diverse samples early in its training. This is like a curious student being exposed to a wide range of topics from the start, rather than just mastering one simple concept before moving on.
This ‘curiosity-driven’ ordering prevents the model from prematurely overfitting to narrow sets of easy problems. By forcing it to explore a broader spectrum of knowledge early on, HAMMER encourages more balanced exploration, stabilizes the optimization process, and ultimately accelerates convergence towards a better solution.
Also Read:
- DIVER: A New Approach to Enhance LLM Reasoning Through Diverse Exploration
- Knapsack RL: Optimizing LLM Exploration for Enhanced Learning
Theoretical and Empirical Validation
The researchers provide strong theoretical backing for HAMMER. They demonstrate that this diversity-driven ordering does not compromise the model’s ability to find the optimal policy. In fact, by training on diverse subsets, HAMMER effectively tightens the generalization bound, meaning the model is better at applying what it learns to new, unseen problems. They also show that minimizing semantic similarity in the Hamiltonian path is equivalent to maximizing the overall diversity of the dataset.
Empirical evaluations across various mathematical benchmarks, including AIME 2024, AIME 2025, AMC 2023, and Olympiad, consistently show HAMMER’s effectiveness. When integrated with popular RLVR algorithms like DAPO and GRPO, HAMMER achieves an average accuracy gain of 3% to 4% over baselines that use randomly shuffled data. This improvement is seen not just in raw pass rates but also in the consistency of the answers generated by the models. The gains remain stable even with larger models, highlighting HAMMER’s efficiency and generality.
Furthermore, ablation studies confirmed that HAMMER outperforms both maximal similarity ordering (which would group similar samples) and traditional difficulty-based (easy-to-hard) training. Crucially, HAMMER achieves these benefits without the need for costly and complex difficulty assessments, requiring only a straightforward forward pass of the backbone LLM to generate embeddings.
In conclusion, HAMMER represents a significant step forward in making reinforcement learning for LLMs more stable and efficient. By intelligently leveraging semantic diversity to guide the training curriculum, it fosters a more exploratory learning environment, leading to faster convergence and improved performance across diverse reasoning tasks. You can read the full research paper for more technical details here: HAMMER Research Paper.
