TLDR: This research introduces a novel rank-2 projection subspace to analyze how Large Language Models (LLMs) combine internal Parametric Knowledge (PK) and external Context Knowledge (CK) when generating Natural Language Explanations (NLEs). It demonstrates that previous rank-1 models are insufficient, and the new framework accurately captures diverse knowledge interactions. Key findings include that hallucinated NLEs strongly align with PK, context-faithful ones balance PK and CK, and Chain-of-Thought prompting shifts NLEs towards CK by reducing PK reliance. This work provides a systematic framework for multi-step knowledge interaction analysis, offering insights into LLM interpretability and factual consistency.
Large Language Models (LLMs) are increasingly used to generate Natural Language Explanations (NLEs), which are human-readable descriptions of how these models arrive at their decisions. These explanations are crucial for understanding whether an LLM’s reasoning is based on external information (Context Knowledge, or CK) or knowledge stored within its own parameters (Parametric Knowledge, or PK). However, the intricate ways in which PK and CK interact during the generation of these explanations have largely remained a mystery.
Previous research often simplified this interaction, viewing it as a single, binary choice where the model either relies on PK or CK. This approach, often modeled in a ‘rank-1 subspace,’ failed to capture the full spectrum of interactions, such as when PK and CK complement or support each other, rather than just conflicting. This limited understanding meant that richer forms of knowledge interaction were overlooked, making it difficult to truly assess the grounding and faithfulness of NLEs.
A new study, titled Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglement, by Sekh Mainul Islam, Pepa Atanasova, and Isabelle Augenstein from the University of Copenhagen, introduces a groundbreaking approach to address this challenge. The researchers propose a novel ‘rank-2 projection subspace’ that can more accurately separate and analyze the individual contributions of PK and CK. This new framework allows for the first multi-step analysis of how these knowledge sources interact across longer NLE sequences, providing a much clearer picture of the LLM’s internal reasoning process.
The experiments, conducted on four different Question Answering (QA) datasets and three open-weight instruction-tuned LLMs (Llama-3.1-8B, Gemma-2 9B, and Mistral-v0.3 7B), yielded significant insights. A key finding was that the traditional rank-1 subspace indeed struggles to represent diverse knowledge interactions effectively. In contrast, the proposed rank-2 formulation successfully captures these complex dynamics.
The multi-step analysis revealed fascinating patterns in how LLMs utilize knowledge:
Knowledge Dynamics During NLE Generation
Initially, during NLE generation, models tend to start with a higher reliance on Context Knowledge, then gradually incorporate both PK and CK, often with a slight prioritization of Parametric Knowledge. For longer and more complex NLEs, especially those involving multi-hop reasoning or higher token uncertainty, PK and CK contributions show more fluctuation, indicating an iterative process where the model reconciles both knowledge sources.
Hallucinations and Parametric Knowledge
One of the most critical findings relates to hallucinations. The study found that hallucinated NLEs align strongly with the PK direction. This suggests that hallucinations are not merely random generation errors but rather reflect a systematic bias towards recalling parametric knowledge, even when it’s incorrect or unsupported by the context. Conversely, NLEs that are faithful and well-grounded in the provided context demonstrate a balanced interaction between PK and CK.
Chain-of-Thought Prompting’s Role
The research also explored the impact of Chain-of-Thought (CoT) prompting, a widely adopted reasoning methodology. It was observed that CoT prompting for NLEs shifts the generated explanations towards Context Knowledge by reducing the model’s reliance on Parametric Knowledge. This clarifies why CoT often improves contextual grounding and factual consistency in LLM outputs.
Also Read:
- ExplicitLM: A New Architecture for Transparent and Updatable Knowledge in Language Models
- Decoding In-Context Learning: How Induction Heads Emerge in Transformers
Implications for LLM Understanding and Control
This work fundamentally changes our understanding of PK-CK interaction, moving it from a binary choice to a multidimensional phenomenon. The rank-2 subspace provides a mechanistic, internal signal for detecting hallucinations and assessing the factual reliability of generated sequences. This framework opens doors for future research, including extending subspace-based probing to other generative tasks like summarization and dialogue, and potentially integrating controllable subspace steering into model training to fine-tune the PK-CK balance. Such advancements could significantly enhance both the interpretability and factual consistency of LLMs.
