TLDR: This research introduces two new hybrid strategies for In-Context Learning (ICL) in large language models (LLMs) that significantly reduce computational costs while maintaining or improving performance. By combining a small number of dynamically selected, similar examples with a larger set of cached, either randomly or k-means clustered, examples, these methods offer a compute-optimal approach for many-shot ICL, making it more practical and scalable.
Large Language Models, or LLMs, have revolutionized how we interact with artificial intelligence, capable of processing vast amounts of information. A key technique that helps these models adapt to new tasks is called In-Context Learning (ICL). This involves providing the model with examples, or ‘demonstrations,’ directly within the input prompt to guide its responses.
Traditionally, ICL started with ‘few-shot’ settings, meaning only a handful of examples were given. However, with the advent of long-context LLMs, which can handle millions of tokens, it’s now possible to use hundreds or even thousands of demonstrations. This shift to ‘many-shot’ ICL has the potential to further boost performance.
The Challenge of Many-Shot ICL
While many-shot ICL offers great promise, it comes with significant challenges. One major hurdle is the high inference cost associated with processing such large inputs. To manage this, a common practice has been to randomly select a fixed set of demonstrations. This allows for ‘caching,’ where computations for these demonstrations are reused, saving time and resources. However, selection strategies based on specific criteria, like similarity to the test sample, often perform better but are computationally expensive because they require dynamically updating the prompt for each new input, preventing caching.
Another observation in many-shot settings is the ‘diminishing effectiveness’ of selection criteria. As the number of demonstrations grows very large, the impact of sophisticated selection methods on performance tends to level off, becoming almost as effective as random selection beyond a certain point.
Introducing Hybrid Selection Strategies
To address these challenges, researchers Shahriar Golchin, Yanfei Chen, Rujun Han, Manan Gandhi, Tianli Yu, Swaroop Mishra, Mihai Surdeanu, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister proposed two innovative strategies for demonstration selection in many-shot ICL. Their work, detailed in the paper “Towards Compute-Optimal Many-Shot In-Context Learning”, aims to combine the performance benefits of dynamic selection with the cost-efficiency of caching.
The first strategy, called Hybrid Similarity-Random Selection, works by combining a small number of demonstrations that are highly similar to each new test sample with a much larger set of randomly chosen demonstrations. The key insight here is that only a small portion of demonstrations needs to be dynamically selected for performance gains, while the bulk can be fixed and cached, significantly reducing computational overhead.
The second strategy, an improvement on the first, is named Hybrid Similarity-k-Means Selection. Instead of using random demonstrations for the cached portion, this method selects a diverse set of demonstrations using k-means clustering. This involves identifying ‘centroids’ based on the representations of test samples and then selecting demonstrations most similar to these centroids. This approach aims to enhance diversity in the cached demonstrations, potentially leading to better overall performance.
How They Work and Why They’re Efficient
Both hybrid strategies ensure that the majority of the input prompt remains fixed and can be cached. Only a small, dynamic part (the similar demonstrations) changes for each test sample. This means new computations are only needed for this small portion, leading to substantial cost savings. For instance, in a 100-shot setting, only 20 similar demonstrations might be dynamically selected, while the other 80 remain cached.
The paper highlights that traditional similarity-based selection, which updates the entire prompt, incurs a quadratic increase in inference cost. In contrast, the proposed hybrid methods scale linearly with the number of cached tokens, making them significantly more efficient, especially in many-shot scenarios.
Experimental Results and Impact
The researchers tested their strategies using Gemini Pro and Flash models across various datasets, including ANLI, TREC, GSM Plus, and MetaTool, as well as Big-Bench Hard tasks for low-data scenarios. They found that their hybrid methods consistently outperformed random selection and either matched or surpassed the most performant (but costly) similarity-based selection approach.
Crucially, this improved performance came with a drastically reduced inference cost—up to an order of magnitude lower in 200-shot settings compared to the similarity-based strategy. The studies also confirmed the diminishing impact of selection criteria, showing that beyond a certain point, adding more similar demonstrations yields little extra benefit, making the hybrid approach’s balance of dynamic and cached content highly effective.
Even in low-data environments, where the pool of available demonstrations is limited, the hybrid strategies proved more effective than simply using all available demonstrations. This suggests that intelligently selecting demonstrations, even from a small pool, can lead to significant performance gains.
Also Read:
- LLMs Learn to Think Smarter with Hierarchical Budget Policy Optimization
- Solving Forgetting and Scalability in Continual Prompt Tuning with GRID
Conclusion
The research demonstrates that by combining multiple criteria for demonstration selection—specifically, a small set of dynamically chosen similar examples with a larger set of preselected, cached examples (either random or k-means based)—it’s possible to achieve a powerful balance between performance and computational cost in many-shot ICL. This makes many-shot in-context learning more practical and scalable for real-world applications of large language models.
