TLDR: A study on Group Relative Policy Optimization (GRPO) fine-tuning reveals that training language models on the hardest examples, rather than easy or random ones, yields significantly larger performance gains (up to 47%) on reasoning tasks. This is because hard examples provide more sustained learning opportunities for GRPO. This strategy also improves out-of-distribution generalization, offering practical guidance for budget-constrained LLM alignment.
Training large language models (LLMs) to perform specific tasks, a process known as fine-tuning, often requires a lot of high-quality data. However, collecting and annotating this data can be very expensive, leading to practical limits on how much data can be used. This raises a crucial question for developers working with limited resources: when fine-tuning an LLM, which types of examples should be prioritized – easy, medium, hard, or a random mix?
A recent research paper titled “Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets” by Benjamin Pikus, Pratyush Ranjan Tiwari, and Burton Ye, delves into this very question. The researchers focused on a specific fine-tuning method called Group Relative Policy Optimization (GRPO), which is a technique similar to PPO (Proximal Policy Optimization) but designed to be more memory-efficient and rely on variations in rewards within groups of examples for learning signals.
The study investigated GRPO fine-tuning across different model sizes and families, including Qwen3-4B, Qwen3-14B, Phi-4, and Llama3.1-8B. They compared four different strategies for selecting a subset of training examples from a larger pool, all while sticking to a fixed budget that allowed only 10% of the available data to be used. The difficulty of each example was estimated by how often the base model (before fine-tuning) succeeded on it across multiple attempts.
The findings were quite striking and consistent across various models and tasks, such as grade-school math problems (GSM8K) and a task involving tracking shuffled objects (from BIG-Bench Hard). The experiments revealed that training on the hardest examples consistently led to the largest improvements in performance. In some cases, these gains were as high as 47% compared to the baseline model. In stark contrast, training on easy examples resulted in the smallest performance gains, often being significantly less effective than even random selection.
Why do hard examples make such a difference? The researchers’ analysis provides a clear explanation rooted in how GRPO learns. GRPO requires a certain amount of “variance” or difference in outcomes within a group of examples to generate effective learning signals. If all examples in a group are either perfectly correct or perfectly incorrect, the learning signal becomes zero, and the model stops learning from that group. Hard examples, by their nature, are those where the model struggles but can occasionally succeed. This means they maintain a mix of correct and incorrect outcomes for a longer period during training, providing more continuous “learnable opportunities” for the GRPO algorithm. Easy examples, on the other hand, are quickly “solved” by the model, leading to uniform success within their groups and thus, a rapid halt in learning from them.
The benefits of training on hard examples also extended beyond the specific tasks the models were fine-tuned on. When evaluated on a significantly harder, out-of-distribution test set (AIME2025-I), models trained on the hardest examples were the only ones to show meaningful improvements over the base model. This suggests that exposure to more challenging problems during training helps models generalize better to new, more difficult scenarios.
These findings have immediate and significant practical implications for anyone fine-tuning language models using GRPO, especially when faced with budget constraints. Instead of trying to collect a broad range of data, practitioners should prioritize acquiring and annotating examples where the base model struggles but still has a chance of success. This focused approach on challenging data can transform a marginally effective fine-tuning effort into a highly successful improvement for reasoning tasks. For more details, you can read the full research paper here.
Also Read:
- Adaptive Guidance Boosts Reasoning in Small Language Models
- AI Models and Uncertainty: How Reinforcement Learning Can Lead to Overconfident Predictions
In summary, the research strongly suggests that when it comes to GRPO fine-tuning for reasoning tasks, focusing your limited data budget on the most challenging examples is the most effective strategy for maximizing performance gains and improving generalization.
