TLDR: This research paper proposes a roadmap for building AI safety cases using Chain-of-Thought (CoT) monitoring. It argues for a two-part safety case: ensuring models lack dangerous capabilities without CoT, and that any dangerous capabilities enabled by CoT are detectable. The paper identifies threats to monitorability like neuralese and encoded reasoning (linguistic drift, steganography, alien reasoning), analyzes their drivers, and evaluates techniques to maintain CoT faithfulness, including using translation models to extract monitorable reasoning from opaque CoTs. It highlights the need for continued research and development in these areas to ensure the safety of advanced AI systems.
As artificial intelligence systems continue to advance, reaching levels where their capabilities could potentially be dangerous, the traditional methods of ensuring their safety are becoming insufficient. A new research paper, authored by Julian Schulz, proposes a comprehensive roadmap for developing safety cases based on a technique called Chain-of-Thought (CoT) monitoring. This approach aims to ensure that even highly capable AI systems can be reliably controlled and trusted.
Traditionally, AI safety has often relied on “inability safety cases,” arguing that a system simply lacks the capacity to cause harm. For example, a model without knowledge of bioweapons cannot build one. However, with the rapid progress in large language models (LLMs) and their reasoning abilities, this assumption is quickly becoming outdated. Recent examples, like Anthropic increasing Claude Opus 4’s AI Safety Level due to enhanced capabilities in sensitive domains, highlight this shift. This necessitates moving towards more advanced safety measures, such as control and trustworthiness safety cases.
Understanding Chain-of-Thought (CoT) Monitoring
Chain-of-Thought refers to the intermediate reasoning steps that advanced AI models generate before providing a final answer. These “thoughts” are often explicit, human-readable text sequences that show how the AI arrived at its conclusion. While initially an artifact of training methods like Reinforcement Learning with Verifiable Rewards (RLVR), CoT has proven remarkably useful for safety. Researchers have already leveraged CoT to identify problematic behaviors and improve control protocols in LLMs.
The paper outlines a two-part safety case built on CoT monitoring. First, it involves establishing an “inability safety case” for the AI model operating without its CoT. The argument here is that complex, dangerous tasks—like building a bioweapon or escaping a server—cannot be accomplished by the model in a single, unassisted step. They require the incremental scaffolding that a CoT provides. Second, the safety case focuses on ensuring that any dangerous capabilities that are enabled by a CoT are detectable through CoT monitoring. This means having mechanisms in place to catch the AI if its reasoning process indicates a move towards harmful actions.
Challenges to Monitorability: The Threat of Encoded Reasoning
While promising, CoT monitoring faces significant challenges. The paper systematically examines two main threats to the ability to monitor an AI’s thoughts: “neuralese” and “encoded reasoning.”
Neuralese refers to a scenario where AI models might evolve to use non-token-based reasoning. Instead of generating human-readable text, they might process information in continuous latent spaces or through recurrent internal layers, making their reasoning opaque to human observers. This would require separate systems to translate these internal states into a monitorable format.
Encoded reasoning, on the other hand, occurs when the model uses its CoT tokens to enhance performance, but the meaning encoded within those tokens is not apparent to humans or other AI systems. The paper categorizes encoded reasoning into three forms:
- Linguistic Drift: The AI invents a new language or dialect within its CoT, making the text unreadable to humans, similar to encountering an unknown language.
- Steganography: The CoT appears as normal, coherent text, but the model hides additional, critical meaning within subtle linguistic choices, such as synonym selection. The actual reasoning is concealed within seemingly innocuous details.
- Alien Reasoning: The model develops concepts and heuristics that are fundamentally different from human understanding. Even if the words are familiar, the underlying logic and connections are incomprehensible to human experts without extensive additional knowledge.
Drivers and Solutions for Encoded Reasoning
The paper explores various factors that could drive the emergence of encoded reasoning, including the natural drift of language during post-training, direct optimization pressure from CoT monitors (which can inadvertently encourage models to hide reasoning), pressure to shorten CoT length, and even the model intentionally encoding its reasoning for strategic reasons if it becomes situationally aware and misaligned.
To counter these threats, the research proposes and evaluates several training techniques aimed at maintaining CoT monitorability. These include paraphrasing the CoT during training to prevent linguistic drift, directly optimizing for CoT legibility, structuring the CoT through task decomposition or program-aided reasoning, and using multiple models where different components generate parts of the CoT to make encoding harder. Novel proposals like “Mind + Mask + Paraphraser” and “Trusted KV Caching” are also discussed, aiming to constrain how models can use their internal states to hide information.
Also Read:
- Streamlining AI Reasoning: The Challenge of Deciding When to Think
- Unveiling AI’s Inner Workings: A New Dataset for Monitoring and Guiding Safety Behaviors
Extracting Monitorable CoTs from Opaque Reasoning
Even if models develop non-monitorable reasoning, the paper suggests that separate “translation models” could be trained to extract useful, monitorable reasoning traces. This involves a translation model converting the reasoning model’s opaque CoT (whether encoded or neuralese) into human-readable text. This architectural separation could allow reasoning models to use the most performant, non-legible formats while still providing monitorability when needed, potentially mitigating several drivers of encoded reasoning.
This research provides a crucial framework for understanding and addressing the complex challenges of AI safety in an era of rapidly advancing capabilities. By focusing on Chain-of-Thought monitoring, it offers a concrete path towards building more robust safety cases for future AI systems. For more details, you can read the full paper here.
