TLDR: A new research paper introduces a unified Bayesian framework explaining how both in-context learning (ICL) and activation steering control Large Language Models (LLMs). The theory posits that ICL updates beliefs by accumulating evidence, while activation steering alters concept priors. Their model accurately predicts sigmoidal learning curves for ICL, how steering shifts these curves, and the additive effects of both interventions, leading to predictable, sudden behavioral shifts in LLMs. This work offers a foundational understanding for more effective and safer LLM control.
Large Language Models (LLMs) have become incredibly powerful, but controlling their behavior during inference remains a key challenge. Researchers have typically approached this control through two distinct methods: In-Context Learning (ICL), which involves providing examples or instructions within the prompt, and Activation Steering, which directly manipulates the model’s internal activations. While both methods aim to guide an LLM’s output, their underlying mechanisms have often been viewed as separate.
A recent research paper, titled “BELIEFDYNAMICSREVEAL THEDUALNATURE OF IN-CONTEXTLEARNING ANDACTIVATIONSTEERING,” proposes a groundbreaking unified framework to understand how these two seemingly different control mechanisms operate. Authored by Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Noah Goodman, Tomer Ullman, Hidenori Tanaka, and Ekdeep Singh Lubana, the paper suggests that both ICL and activation steering influence LLM behavior by altering the model’s “belief” in latent concepts.
The core idea is rooted in a Bayesian perspective. Imagine an LLM holds various beliefs about different concepts. According to this new theory, activation steering works by changing the model’s initial assumptions or “priors” about these concepts. It’s like telling the model, “Before you even look at the input, consider this concept more or less likely.” In contrast, in-context learning functions by accumulating evidence. When you provide examples in a prompt, the model gathers data that strengthens or weakens its belief in a particular concept, much like a Bayesian agent updating its beliefs based on new observations.
This unifying perspective led to the development of a closed-form Bayesian model that accurately predicts how LLMs behave under both types of interventions. The researchers conducted experiments across several domains, particularly focusing on manipulating an LLM’s “persona”—such as making it adopt traits like psychopathy or moral nihilism. They observed three significant behavioral phenomena predicted by their model.
First, they found that in-context learning often follows a “sigmoidal learning curve.” This means that as more examples are provided, the model’s behavior initially changes slowly, then rapidly shifts at a certain point (a “transition point”), and finally plateaus. This pattern, which explains prior observations of sudden learning curves in ICL, is effectively captured by their model, which accounts for evidence accumulation in a sub-linear fashion.
Second, the research demonstrated that activation steering predictably shifts these ICL learning curves. Positive steering magnitudes (encouraging a concept) effectively make the model learn faster or require fewer examples to adopt a persona, shifting the curve to the left. Negative magnitudes have the opposite effect, requiring more examples. This shows how steering directly modulates the model’s initial belief state.
Third, and crucially, the study revealed an “additive effect” of these interventions in a log-belief space. This means that the impact of ICL and activation steering combine in a straightforward way, leading to distinct phases in the model’s behavior. Small changes in either context length or steering magnitude can induce sudden and dramatic shifts in how the LLM responds. Their model can even predict the exact “crossover points” where the model’s belief in one concept surpasses another, offering a concrete prediction for phenomena like many-shot jailbreaking.
This work offers a powerful theoretical framework for understanding and controlling LLMs. By viewing both prompting and activation steering through the lens of Bayesian belief updating, researchers gain a deeper insight into how LLMs process information and adapt their behavior. This could have significant practical implications for designing more reliable control protocols and for enhancing AI safety by predicting when and how LLM behavior might suddenly change. The full paper can be accessed here.
Also Read:
- Understanding Knowledge Dynamics in LLM Explanations with a New Framework
- Decoding In-Context Learning: How Induction Heads Emerge in Transformers
While the current work focuses on binary concepts and a specific steering method, it opens up exciting avenues for future research. Exploring how this theory generalizes to more complex, non-binary concepts, or different steering techniques, will be important. Understanding the precise layers where belief updates occur and whether these processes are localized within the neural network are also critical next steps. Ultimately, this unified perspective on LLM control as belief updating redefines our understanding of representation in these advanced AI systems.
