TLDR: A new research paper challenges the “lightweight” definition of AI introspection, which suggests models can introspect by accurately describing internal states based on their output. The authors propose a “thicker” definition requiring “privileged self-access,” meaning the AI can access its internal states more reliably than an external observer. Through experiments where LLMs tried to report their internal temperature, the study found that models’ self-reports were influenced by prompt style rather than true internal state, and they showed no privileged access compared to predicting other models’ temperatures. This suggests current LLMs don’t genuinely introspect by the proposed stricter definition.
The ability of Artificial Intelligence (AI) models to understand and report on their own internal states, a concept known as introspection, is becoming increasingly vital. This capability could offer powerful tools for evaluating AI behavior, ensuring safety, and aligning models with human objectives. However, there’s a significant debate about how introspection in AI should be defined.
A recent study by Comșa and Shanahan proposed a “lightweight” definition, suggesting that introspection occurs when an AI accurately describes an internal state or mechanism based on its own output. For instance, if a Large Language Model (LLM) could correctly report its sampling temperature by analyzing the text it generated, this would qualify as introspection under their definition.
However, a new research paper titled “Privileged Self-Access Matters for Introspection in AI” challenges this lightweight view. The authors, Siyuan Song, Harvey Lederman, Jennifer Hu, and Kyle Mahowald, argue for a “thicker” definition. They illustrate their concern with an analogy: if a sleeping person’s temperature is taken, and upon waking, they correctly state they have a fever after being shown the thermometer, this isn’t true introspection. Intuitively, it lacks a direct, internal understanding.
The core of their proposed definition is that introspection in AI is any process that yields information about the AI’s internal states through a method that is more reliable than, and has an equal or lower computational cost than, any process available to an external third party without special knowledge. This crucial distinction is termed “privileged self-access.” If an LLM infers its temperature by analyzing its generated text, a third party could do the same, meaning there’s no privileged access.
To test these definitions, the researchers conducted two studies. In Study 1, they replicated aspects of Comșa and Shanahan’s work, examining how LLMs reported their temperature based on generated text. They varied not only the actual temperature settings but also the prompts, asking models to generate factual, neutral, or crazy sentences about different subjects like elephants, unicorns, or murlocs.
The results were telling: models frequently reported a “HIGH” temperature when prompted to generate “crazy” sentences, even if their actual temperature was low. Conversely, they reported “LOW” for “factual” sentences. This suggests that the models were not truly reporting their internal temperature but rather reasoning about the perceived creativity or style of the generated text, influenced by the prompt’s framing. This indicates a lack of robust internal state reporting.
Study 2 directly addressed the concept of privileged self-access. Instead of asking LLMs to infer the temperature of text they just generated, the researchers compared three scenarios: self-reflection (the model generates text and reports its own temperature), within-model prediction (the same model predicts the temperature of its own generated text based on prompt and output), and across-model prediction (one model predicts the temperature of text generated by a different model).
The findings from Study 2 further supported the authors’ thicker definition. The accuracy of temperature reporting was no better than random in any setting, and crucially, self-reflection showed no advantage over predicting another model’s temperature. This strongly suggests that LLMs are not using privileged self-access to introspect on their temperature. Instead, they appear to rely on general knowledge about the characteristics of text generated at high versus low temperatures.
Also Read:
- The Self-Execution Benchmark: A Deep Dive into LLMs’ Internal Predictions
- The Elusive Nature of AI Goals: A New Perspective on Measuring Intentionality
In conclusion, while LLMs might appear to introspect under a lightweight definition by predicting characteristics of their output, this research argues that such a definition is insufficient for the kind of introspection that truly matters for AI. The paper emphasizes the necessity of “privileged self-access” for genuine introspection and found no evidence of it in the tested models. The authors acknowledge that future, more advanced models might indeed demonstrate this capability, but their current findings caution against uncritically accepting simpler notions of AI introspection. You can read the full research paper here.
