TLDR: LATTS (Locally Adaptive Test-Time Scaling) is a novel method that dynamically adjusts computational effort during Large Language Model inference based on the ‘local difficulty’ of each reasoning step. By using a verifier-based acceptance criterion, LATTS efficiently allocates resources, leading to significantly superior accuracy-compute tradeoffs and requiring substantially fewer tokens compared to traditional uniform test-time scaling approaches.
Large Language Models (LLMs) have become incredibly powerful, but getting the best performance out of them, especially on complex tasks, often requires increasing computation during the inference phase. This approach, known as test-time scaling, allows models to engage in deeper reasoning. However, many existing methods apply this increased computation uniformly across all parts of a problem, regardless of how difficult individual steps might be. This can lead to inefficient use of valuable computational resources.
A new approach, called Locally Adaptive Test-Time Scaling (LATTS), addresses this limitation by dynamically adjusting the computational effort allocated to each step of the generation process. Instead of a one-size-fits-all strategy, LATTS uses a verifier model to assess the “local difficulty” of each reasoning step. This allows the system to decide whether to accept a generated step, resample, backtrack, restart, or stop the generation, effectively allocating more compute where it’s truly needed.
How LATTS Works
At its core, LATTS frames the generation process as a series of sampling problems. For each reasoning step, the system proposes a candidate step from the base language model. A verifier then evaluates this candidate, assigning a score that indicates its likelihood of correctness. Based on this score, and a modulator function, LATTS decides whether to accept the step. If the step isn’t accepted, the system retries, sampling new candidates until an acceptable one is found or a maximum number of trials is reached.
The beauty of this acceptance-rejection sampling mechanism is that it implicitly scales computation. “Difficult” steps, where the model struggles to generate high-quality outputs, will naturally require more sampling attempts before an acceptable candidate emerges. This means more computational effort is automatically directed to the parts of the problem that truly demand it, without needing explicit difficulty estimation.
LATTS offers different variants through its “modulator” function. For instance, “LATTS Tilted” uses the raw verifier score to guide acceptance, while “LATTS Truncated” applies a hard threshold, only accepting steps that meet a certain minimum score. These choices allow for fine-grained control over the stringency of step acceptance and, consequently, the accuracy-compute tradeoff.
Handling Rejections and Improving Performance
Since continuously sampling new candidates can be computationally expensive, LATTS incorporates fallback strategies when a step isn’t accepted within a predefined number of trials. These strategies include: stopping the generation, choosing the best (though still rejected) candidate, backtracking to a previous step, or restarting the entire generation process from scratch. Experiments show that the “backtrack” strategy often provides the best balance between accuracy and computational cost.
To further boost performance, LATTS can also generate multiple independent “completions” (full reasoning chains) and then use a weighted majority voting system to select the final answer. This combines the adaptive sampling of LATTS with the benefits of parallel generation, leading to even more robust results.
Impressive Results Across Benchmarks
Empirical evaluations demonstrate that LATTS achieves significantly better accuracy-compute tradeoffs compared to standard test-time scaling methods. For example, on the challenging MATH500 dataset, LATTS can match or even surpass the performance of much larger LLMs while generating substantially fewer tokens. In some cases, it achieved similar accuracy to leading baselines with about 50 times fewer tokens, and improved accuracy by over 10% at maximal compute budgets.
The method proved effective with different base models (like Llama 3.2-Instruct-1B and 3B) and verifier configurations. Notably, LATTS also performs well when a general-purpose LLM (like Llama 3.1-Instruct-70B) is used as a critic to score reasoning steps, rather than a specialized Process Reward Model (PRM). This opens up possibilities for applying LATTS to domains beyond mathematical reasoning where dedicated PRMs might not be readily available.
Also Read:
- Rethinking LLM Performance: Why System-Optimal Trumps Compute-Optimal in Test-Time Scaling
- SIM-CoT: Enhancing LLM Efficiency and Accuracy Through Supervised Implicit Thoughts
A Smarter Way to Scale
LATTS represents a significant step forward in making LLM inference more efficient and effective. By intelligently allocating computational resources based on the dynamic difficulty of reasoning steps, it enables models to achieve higher accuracy with less computational overhead. This adaptive approach is crucial for deploying powerful LLMs in real-world applications where both performance and cost efficiency are paramount. You can read the full research paper for more details. Read the full paper here.
