TLDR: PREPO is a new method for training Large Language Models (LLMs) using Reinforcement Learning with Verifiable Rewards (RLVR) that significantly improves data efficiency. It does this by using a “Perplexity-Schedule” to select prompts from easy to hard, and “Relative-Entropy Weighting” to prioritize uncertain, exploratory responses. This approach reduces the number of generated responses (rollouts) needed for training by up to three times while maintaining or improving performance on mathematical reasoning tasks.
Large Language Models (LLMs) have shown remarkable progress in reasoning, largely thanks to techniques like Reinforcement Learning with Verifiable Rewards (RLVR). However, training these models with RLVR is often a computationally intensive and costly process. A new research paper introduces a novel approach called PREPO, designed to make this training significantly more efficient by leveraging inherent properties of the data itself.
The Challenge of Efficient LLM Training
Traditional RLVR methods face a major bottleneck: generating numerous “rollouts” or self-generated responses, many of which contribute little to the model’s learning. This inefficiency stems from two main areas. Firstly, some prompts given to the model might be too simple or too complex to provide meaningful learning signals. Current ways to estimate prompt difficulty, such as pass rates or manual criteria, are often expensive and don’t truly reflect how the model perceives the difficulty. Secondly, the responses generated by the model (rollouts) vary in their “confidence.” Responses with low confidence (high entropy) often indicate alternative reasoning paths and opportunities for exploration, but standard methods don’t always prioritize these effectively.
Introducing PREPO: Intrinsic Exploration for Smarter Learning
To address these inefficiencies, researchers from the National University of Singapore and Ant Group propose Perplexity-Schedule with Relative-Entropy Policy Optimization (PREPO). This method introduces two complementary components that work together to improve data efficiency:
- Prompt Perplexity (PPL) Schedule: PREPO uses prompt perplexity as a dynamic indicator of how adaptable the model is to learning from a given context. Perplexity, in simple terms, measures how “surprised” the model is by a sequence of words. A lower perplexity suggests the model is more familiar or confident with the prompt. By scheduling prompts based on their perplexity, PREPO allows the model to first master well-understood contexts before moving on to more challenging ones. This is like a curriculum where learning progresses from easy to hard.
- Relative-Entropy Weighting: This component focuses on the “rollouts” or responses generated by the model. It amplifies the differences among these responses by looking at their relative entropy. Entropy here indicates the level of uncertainty or diversity in the model’s generated sequence. Responses that show a higher degree of exploration (more uncertainty) are prioritized, ensuring that the model actively seeks out and learns from diverse reasoning paths, even within seemingly confident responses.
How PREPO Delivers Efficiency
The PREPO framework integrates these two ideas seamlessly. Before the model generates its responses, the PPL-schedule filters prompts, ensuring that the training data evolves with the model’s learning progress. After responses are generated, the relative-entropy weighting scheme emphasizes those uncertain yet potentially informative rollouts. This adaptive balancing allows the model to “seek uncertainty within certainty,” preserving its exploratory capacity throughout training.
Impressive Results on LLM Benchmarks
The effectiveness of PREPO was demonstrated on popular LLMs like Qwen and Llama, across various mathematical reasoning benchmarks. The results are compelling: PREPO achieved competitive performance while significantly reducing the number of rollouts needed for training—up to three times fewer than baseline methods. For instance, on Qwen2.5-Math-1.5B, it reduced rollouts by 63.3%, and on Llama3.1-8B, by 48.9%. This translates directly into substantial savings in computational resources and training time, making advanced LLM capabilities more accessible and sustainable.
Beyond these empirical gains, the researchers also provide theoretical and in-depth analyses that explain the fundamental reasons why PREPO improves the data efficiency of RLVR. You can read the full research paper for more technical details here: Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration.
Also Read:
- Adaptive Effort Control: AI Models Learn to Optimize Reasoning for Cost and Accuracy
- Predicting Transit Delays: How AI Learns from Unstructured Alerts
Looking Ahead
PREPO represents a significant step forward in making reinforcement learning for LLMs more efficient. By intelligently selecting prompts and prioritizing exploratory responses, it allows models to learn more effectively with less data. This approach opens doors for further research into leveraging intrinsic data signals and integrating them with system-level optimizations to push the boundaries of LLM capabilities.
