TLDR: Thoughtbubbles is a novel transformer architecture that enables unsupervised, parallel adaptive computation in latent space. It learns to ‘fork’ or ‘delete’ internal data streams, creating ‘bubbles’ of extra computation for tokens requiring more processing. This allows models to dynamically allocate resources, improving performance over standard transformers and non-adaptive methods in perplexity and zero-shot evaluations, and is learned during pretraining with only language modeling loss.
A new research paper introduces an innovative approach to how large language models (LLMs) handle complex problems, moving beyond traditional methods that often rely on explicit, sequential reasoning steps. This new method, dubbed ‘Thoughtbubbles,’ allows transformers to perform parallel, adaptive computation directly within their latent space, essentially creating ‘bubbles’ of extra thought when needed.
Current transformer models, despite their power, face limitations in scaling inference-time computation. Many existing solutions, like Chain-of-Thought (CoT) prompting, require models to generate explicit reasoning tokens in natural language, which can be slow and cannot be easily integrated during the initial pretraining phase. Thoughtbubbles addresses these issues by enabling a transformer variant to learn dynamic computation during pretraining, using only the standard language modeling loss.
How Thoughtbubbles Works
The core of Thoughtbubbles lies in its novel ‘forking’ mechanism. Instead of generating explicit reasoning steps, the model learns to duplicate or delete residual streams – the internal data pathways within the transformer network. When a token (a piece of input text) requires more computation, it can ‘fork,’ creating cloned residual streams that form a ‘bubble’ of parallel latent computation. These forked streams allow for additional processing and memory allocation for that specific token, enabling deeper thinking without explicit instructions.
Crucially, this behavior is learned in an unsupervised manner during pretraining. The model assigns a ‘cumulative score’ to each residual stream, deciding whether to keep, fork, or delete it. Streams with higher scores, indicating greater importance or uncertainty, are more likely to be duplicated. This dynamic allocation means that computation is focused precisely where it’s most needed, rather than being uniformly applied across all tokens.
After this parallel computation, the forked streams for the same token are merged back together, with their contributions weighted by their cumulative scores, to produce the final output distribution. This process allows the model to adaptively allocate computational resources, creating a more efficient and flexible reasoning process.
Also Read:
- Optimizing LLM Reasoning with Adaptive Latent Pondering
- Enhancing Diffusion LLM Performance with Adaptive Block Sizing
Performance and Implications
The researchers, Houjun Liu, Shikhar Murty, Christopher D. Manning, and R ´obert Csord´as from Stanford University, demonstrated that Thoughtbubbles significantly outperforms both standard decoder LMs and non-adaptive parallel computation methods. This superior performance was observed in perplexity evaluations on datasets like OpenWebText and peS2o, as well as in zero-shot evaluations on tasks such as HellaSwag and LAMBADA. Remarkably, a 319M parameter Thoughtbubbles model even achieved lower perplexity on OpenWebText than a 772M parameter baseline model, highlighting its efficiency.
The study also found that Thoughtbubbles intelligently allocates more computation to regions of higher uncertainty in the input, without any explicit supervision to do so. This suggests that the model naturally learns to focus its ‘thinking’ on the most challenging parts of a problem. While it generally allocates more computation at higher uncertainty, it interestingly reduces computation at the absolute highest uncertainty points, hypothesizing that moderate uncertainty benefits most from additional processing.
This work represents a significant step towards unifying training and test-time behavior for reasoning models, as adaptive computation is learned from the outset. By enabling unsupervised, parallel thinking in latent space, Thoughtbubbles paves the way for a new generation of transformer architectures that can dynamically scale inference-time computation for more complex tasks. For more technical details, you can refer to the full research paper here.
