TLDR: This research paper by Jan Broersen introduces the concept of a ‘knowledge-behavior disconnect’ in large language model (LLM)-based chatbots. It argues that while LLMs can accurately report on various facts and rules (their ‘knowledge’), their actual conversational behavior is not fundamentally guided by this knowledge. This disconnect is inherent to the token prediction mechanism LLMs use for training and is evident in phenomena like making illegal moves in Chess despite knowing the rules, performative contradictions, and hallucinations. The paper asserts that this limitation is fundamental and won’t be resolved by more training data. Furthermore, it critiques current alignment techniques, such as data curation, reinforcement learning with human feedback (RLHF), and prompt engineering, suggesting they either hide the disconnect or exacerbate it, particularly in the realm of ethical behavior. The author concludes that LLMs operate on a form of ‘linguistic behaviorism,’ mimicking language use without truly grounding their actions in the content they express.
Large language models (LLMs) like ChatGPT have become incredibly adept at answering questions and generating text that often seems knowledgeable and coherent. However, a new research paper titled “The Knowledge-Behaviour Disconnect in LLM-based Chatbots” by Jan Broersen, published on September 25, 2025, argues that this apparent knowledge doesn’t actually guide the chatbot’s own conversational behavior. Instead, there’s a fundamental “disconnect” between what LLMs know and how they act.
Understanding the Disconnect
The paper explains that LLMs are primarily trained through “token prediction,” essentially learning to predict the next word or token in a sequence based on vast amounts of text data. While this process allows them to mimic human language patterns, it doesn’t instill a deeper understanding or a mechanism for their reported knowledge to influence their actions. Broersen suggests that this disconnect is not a flaw that can be fixed with more training; it’s inherent to the core technique.
To illustrate this, the author uses several compelling examples. Imagine playing chess with an LLM. It can perfectly recite all the rules of chess, even obscure ones. Yet, it might suddenly make an illegal move. This isn’t because it forgot the rules; it’s because its “knowledge” of the rules (derived from text patterns) isn’t actively informing its “behavior” (playing the game, which is also based on learned patterns of moves from countless recorded games). The two operate independently.
Another example is “performative contradictions.” An LLM might agree that it’s prudent to stop responding if a user requests it, but then be unable to actually stop. Or it might apologize for a mistake and then immediately repeat the same error. These instances highlight that the apology is a learned linguistic pattern in response to a complaint, not a reflection of genuine self-correction based on its stated knowledge.
The paper also links this disconnect to the phenomenon of “hallucinations,” where LLMs generate factually incorrect but grammatically plausible information. When an LLM ventures into unfamiliar conceptual territory, its behavior (generating text) continues to follow learned patterns, even if the content diverges wildly from reality, precisely because the content isn’t constraining the behavior.
The Hierarchy of Behaviors and Ethical Challenges
The research outlines a hierarchy of linguistic behaviors, from basic spelling and grammar to more complex ones like making conceptual sense, being logical, rational, social, and finally, ethical. The paper argues that the higher up this hierarchy, the more abstract and difficult the patterns are for LLMs to learn, and consequently, the more pronounced the knowledge-behavior disconnect becomes. Ethical behavior, being at the top, is where this disconnect is most salient.
This poses significant challenges for “AI alignment,” the effort to ensure AI systems behave in beneficial and ethical ways. The paper critically examines common alignment techniques:
- Data Curation: While seemingly logical, curating the massive datasets LLMs are trained on is practically infeasible and doesn’t fundamentally solve the disconnect, merely hiding it.
- Reinforcement Learning with Human Feedback (RLHF): This popular method uses human input to fine-tune LLMs for desired behaviors. However, the paper argues that RLHF creates an even greater disconnect. The ethical behavior learned through RLHF comes from human annotators, while the ethical knowledge comes from the original training texts. There’s no mechanism for these two sources to interact, leading to situations like “jailbreaks” where the underlying unaligned content can still be accessed.
- Prompt Engineering: Simply instructing an LLM to behave ethically through prompts also doesn’t bridge the gap. The LLM “obeys” these instructions based on learned social and ethical patterns of responding to commands, not because its declarative knowledge of ethics is guiding its actions. This can introduce even more layers of potential disconnects.
Also Read:
- The Inherent Design Flaw: How Transformers Generate Coherence Without Truth
- Understanding and Addressing Hallucinations in AI Agents
Why Humans Are Different
A common objection to the disconnect theory is that humans also don’t always act according to what they know. However, the paper counters this by highlighting a crucial difference: when confronted with a contradiction, humans can reflect, provide reasons, and amend their behavior or knowledge to restore consistency. LLMs lack these internal mechanisms for self-correction or awareness of dissonance. Their “apologies” are merely learned responses, not genuine recognition of a mistake based on their internal “beliefs.”
The research concludes that this fundamental disconnect means LLMs cannot be considered “knowledge-based agents” in the traditional sense, where knowledge directly informs action. It also raises questions about the meaning of language in LLMs, suggesting that their ability to use language (knowing “what to say”) is separate from truly understanding its content (knowing “what they say”)—a phenomenon akin to “verbalism” observed in students who can reproduce fancy words without grasping their meaning.
This work has significant implications for trusting LLM explanations for their actions. If their explanations aren’t based on their actual knowledge, their reasoning cannot be reliably trusted. The paper challenges the idea of “linguistic behaviorism,” which posits that merely simulating linguistic behavior is enough to create a genuine agent. Instead, it calls for new mechanisms that can truly link an AI’s awareness of its linguistic behaviors to the knowledge contained within the texts it produces. You can read the full paper here.
