TLDR: A new research paper introduces “Input Time Scaling,” a paradigm that enhances LLM reasoning by refining input queries during both training and testing. It reveals a “training-testing co-design” phenomenon, where consistent application of strategies across both phases is crucial. Surprisingly, the research contradicts “garbage in, garbage out,” showing that less-filtered, diverse data can outperform high-quality, larger datasets. This simple, low-labor method achieves State-of-the-Art performance on math reasoning benchmarks like AIME with 32B models, emphasizing diversity over traditional quality and quantity in data curation.
Large Language Models (LLMs) have made remarkable strides in complex tasks like mathematics and reasoning. Traditionally, their impressive capabilities stem from two main approaches: extensive training on vast, meticulously prepared datasets (data and training scaling) and sophisticated reasoning processes during testing (inference time scaling). However, these methods often come with significant costs, demanding immense computational resources and extensive human effort for data curation.
A new research paper introduces an innovative approach called “Input Time Scaling,” which offers a fresh perspective on boosting LLM reasoning abilities. This paradigm shifts focus to refining the inputs or queries themselves, both during the training and testing phases. It complements existing scaling methods by strategically allocating resources to the initial query, making the process more efficient and less labor-intensive.
A key discovery within this research is the “training-testing co-design” phenomenon. The paper highlights that for optimal performance, query refinement strategies must be applied consistently across both training and testing. Simply applying these strategies during one phase while neglecting the other can severely diminish the performance gains. This emphasizes a holistic approach where the model learns and applies these input-centric strategies as an integrated whole.
Perhaps the most surprising finding, which challenges a long-held belief in AI development, is the notion of “garbage in, garbage out.” The researchers observed that datasets traditionally considered to be of lower quality, or even those containing seemingly irrelevant information, could lead to higher reasoning performance. For instance, using just 1,000 randomly selected examples from a minimally filtered dataset yielded superior results compared to larger, more carefully curated ones. This suggests that the conventional inductive biases used in data curation might inadvertently limit an LLM’s potential performance ceiling. Furthermore, models trained on smaller, diverse datasets (1k examples) sometimes outperformed those trained on much larger datasets (15k examples) of similar quality, aligning with the “Less is More” phenomenon where diversity, rather than sheer quantity or strict quality, becomes a critical factor.
The Input Time Scaling method is remarkably simple and transparent, avoiding the complex data and training pipelines typically associated with high-performing LLMs. It leverages the meta-knowledge within LLMs to refine inputs using different strategies, such as generating “personas” that are similar, unsimilar, or random to the problem queries. This automated approach requires minimal human intervention.
Through comprehensive experiments using models like Qwen2.5-32B-Instruct, the researchers demonstrated that this simple Input Time Scaling approach can achieve or even surpass State-of-the-Art (SOTA) performance among 32B models. For example, it reached 76.7% pass@1 on both AIME24 and AIME25 math benchmarks using greedy decoding. When combining the outputs of three models using a majority vote, the AIME25 score further increased to 80%. Starting with a stronger base model like DeepSeek-R1-Distill-Qwen-32B, the method achieved an impressive 86.6% on AIME24 and 76.7% on AIME25, solidifying its SOTA standing.
The paper also delves into the effectiveness of different persona strategies. While intuition might suggest that adding relevant information (Persona-Similar) would yield the best results, the experiments showed that combinations like training with Persona-Similar and testing with Persona-Unsimilar (S-U), or vice-versa (U-S), often led to the highest performance. Even random personas (R-R) performed well. This highlights that diverse starting points, rather than strictly “high-quality” or “relevant” ones, can be more effective in evoking an LLM’s reasoning capabilities and preventing overfitting.
Also Read:
- Unlocking Advanced LLM Reasoning: The Combined Power of Adaptive Exploration and Scaled Training Data
- Adaptive Guidance Boosts Reasoning in Small Language Models
This research marks a significant step towards more efficient and accessible LLM development, challenging established norms in data curation and training methodologies. The findings are compatible with the “Less is More” phenomenon, emphasizing that a small, diverse set of examples can be sufficient to unlock high-level reasoning. The authors are committed to open-sourcing their datasets, data pipelines, and evaluation results to foster reproducibility and further research in this promising new scaling paradigm. You can read the full research paper here.
