TLDR: This research paper introduces a framework for parallel test-time scaling (TTS) in latent reasoning models, which traditionally lacked sampling and aggregation mechanisms due to their continuous vector space operations. The authors propose two uncertainty-inspired stochastic sampling strategies: Monte Carlo Dropout (MC-dropout) and Additive Gaussian Noise (AGN), to generate diverse reasoning paths. For aggregation, they developed a Latent Reward Model (LatentRM) trained with a step-wise contrastive objective to score and guide latent reasoning trajectories. Experiments demonstrate that both sampling methods effectively scale with compute, exhibiting distinct exploration dynamics, while LatentRM enables robust trajectory selection, leading to consistent performance gains.
Large Language Models (LLMs) have shown incredible abilities in solving complex tasks, often by using a technique called Test-Time Scaling (TTS). This means that by giving the model more computational power during inference (when it’s making predictions), it can achieve better results. Traditionally, this scaling is done through explicit Chain-of-Thought (CoT) reasoning, where LLMs verbalize their intermediate steps in natural language, generating long sequences of tokens.
One powerful way to apply TTS is through parallel scaling. This involves sampling multiple reasoning paths simultaneously and then combining their outcomes, perhaps through voting or searching for the best one. This allows models to directly convert extra computing power into stronger capabilities without needing to be retrained.
However, recent advancements have introduced a more efficient approach: latent reasoning. In this paradigm, the intermediate reasoning steps unfold in continuous vector spaces, rather than as discrete tokens. This ‘continuous CoT’ (CCOT) can be more compact and efficient, potentially matching or even surpassing explicit CoT. It’s like human intuition, where thoughts aren’t always fully verbalized but exist in a more abstract form.
The big question has been whether these latent reasoning models can also benefit from parallel TTS. The challenge lies in two main areas: first, latent models don’t have clear sampling mechanisms in continuous space, unlike token-based models that use probabilities to sample tokens. Second, there’s a lack of probabilistic signals or scores for effectively aggregating these continuous reasoning trajectories.
Enabling Parallel Scaling for Latent Reasoning
A new research paper, Parallel Test-Time Scaling for Latent Reasoning Models, addresses these fundamental issues, opening up parallel TTS for latent reasoning models. The authors, Runyang You, Yongqi Li, Meng Liu, Wenjie Wang, Liqiang Nie, and Wenjie Li, propose innovative solutions for both sampling and aggregation.
Uncertainty-Inspired Sampling Strategies
To introduce controlled stochasticity (randomness) into latent reasoning for sampling diverse paths, the researchers drew inspiration from uncertainty estimation theory. They introduced two distinct strategies:
- Monte Carlo Dropout (MC-dropout): This method captures ‘epistemic uncertainty,’ which reflects the model’s own uncertainty due to its limited knowledge. During inference, dropout is kept active, randomly masking parts of the model’s weights. This generates different plausible configurations of the model, leading to varied reasoning paths.
- Additive Gaussian Noise (AGN): This strategy simulates ‘aleatoric uncertainty,’ which arises from inherent noise or ambiguity in the inputs. It involves adding small, random Gaussian noise directly to the latent thoughts at each reasoning step. This introduces controlled perturbations, encouraging broad exploration around the deterministic path.
Experiments showed that both MC-dropout and AGN effectively scale with increased computation, meaning more samples lead to better coverage (more problems solved). MC-dropout tended to achieve higher coverage overall, promoting a more structured and directed exploration towards unconventional solutions. AGN, on the other hand, drove a broader, more isotropic exploration, enriching diversity around the central reasoning path.
Latent Reward Model for Aggregation
For aggregating the sampled latent trajectories, the paper introduces the Latent Reward Model (LatentRM). Unlike token-based models that use log-likelihoods, latent trajectories are continuous vectors without explicit scores. Existing reward models designed for linguistic steps cannot interpret these abstract latent thoughts.
LatentRM is a dedicated scorer that evaluates and guides the progression of latent reasoning at each intermediate step. It’s trained using a step-wise contrastive objective, which means it learns to discriminate between good and bad candidate thoughts at each step, providing fine-grained, position-sensitive scoring. During inference, LatentRM sums the scores of a generated sequence to estimate its quality.
The LatentRM enables effective aggregation strategies like ‘best-of-N selection’ (picking the best trajectory out of N sampled ones) and ‘beam search’ (a guided search that explores promising paths). Both of these strategies consistently outperformed a simple ‘majority voting’ baseline, confirming LatentRM’s ability to distinguish promising reasoning trajectories.
Also Read:
- DeepPrune: A New Approach to Efficient Parallel Reasoning in Large Language Models
- MixReasoning: A Smart Approach to Efficient Language Model Thinking
Conclusion
This work successfully brings parallel test-time scaling to latent reasoning models, a capability previously exclusive to token-based approaches. By providing principled methods for sampling in continuous latent spaces and a novel reward model for aggregation, it opens a new direction for scalable and robust inference in the latent regime. Future work aims to integrate these sampling and aggregation methods into a reinforcement learning framework, transforming TTS into an adaptive reasoning process.
