TLDR: A new research paper argues that progress towards Artificial General Intelligence (AGI) is limited by our theoretical understanding, not just data or scale. It critiques current AI’s reliance on observational learning within fixed hypothesis spaces, which prevents the discovery and correction of ‘unreachable errors.’ The paper proposes an ‘error-centric’ shift, focusing on how errors evolve, the limits of current hypotheses, and the role of conjecture and criticism in expanding knowledge. It introduces ‘Causal Mechanics’ and structural principles like the Locality–Autonomy Principle (LAP), Independent Causal Mechanisms (ICM), and Compositional Autonomy Principle (CAP) to guide the development of systems capable of unbounded explanatory knowledge creation.
A new research paper, “Towards Error-Centric Intelligence I: Beyond Observational Learning,” challenges the prevailing belief that artificial general intelligence (AGI) is primarily limited by data and computational scale. Instead, author Marcus A. Thomas argues that the path to true AGI is fundamentally constrained by our theoretical understanding of intelligence itself. This paper proposes a radical shift in perspective, moving from a data-centric approach to an “error-centric” one.
The Limits of Current AI
The paper critically examines the current paradigm of machine learning, particularly large language models (LLMs). It draws an analogy between LLMs and simple pocket calculators, highlighting that both operate within a fixed set of possibilities, or a “hypothesis space.” While training allows LLMs to refine their internal states and improve performance within this space, it doesn’t enable them to fundamentally alter or expand their understanding. This means that certain types of errors, termed “unreachable errors,” cannot be discovered or corrected because the system lacks the representational capacity to even conceive of them.
A key critique is leveled against the “Platonic Representation Hypothesis,” which suggests that massive observational learning will eventually lead to a universal, shared model of reality. The paper argues that simply observing data, no matter how vast, cannot by itself reveal underlying causal mechanisms. For instance, observing a correlation between two events doesn’t tell us if one causes the other, or if a third, unobserved factor causes both. This limitation means that observational learning, while achieving high “competence” in specific tasks, falls short of true “intelligence,” which is defined as the efficiency with which a system creates explanatory knowledge.
The paper also reinterprets “catastrophic forgetting” – where a model forgets old information when learning new tasks – not as a mere optimization glitch, but as a symptom of a deeper structural problem: “fractured, entangled representations.” This occurs when a single concept is spread across disconnected parts of the model, or when distinct capabilities are mixed, leading to interference during sequential learning.
Beyond Observational Learning: An Error-Centric Shift
To overcome these limitations, the paper advocates for an error-centric approach, reframing the problem of general intelligence around three core questions:
- How do both explicit (observable failures) and implicit (flawed internal models) errors evolve as an AI agent acts?
- What types of errors remain inherently unreachable given the agent’s current understanding or “hypothesis space”?
- How can an agent generate new ideas and theories (conjecture) and rigorously test them (criticism) to expand its understanding and address previously unreachable errors?
This framework emphasizes that true learning involves not just correcting known errors, but actively discovering new ones and, crucially, expanding the very space of possible explanations.
Also Read:
- Rethinking AGI Evaluation: From Simple Scores to Robust Intelligence Clusters
- Building AI That Welcomes Change: A New Approach to Corrigible Goals
Causal Mechanics and New Structural Principles
To guide the design of future AGI systems, the paper proposes a “Causal Mechanics” program, which treats changes to an AI’s hypothesis space as a fundamental operation. This program is built upon three structural principles:
- Locality–Autonomy Principle (LAP): This principle ensures that different parts of an AI’s internal mechanisms can be modified independently without unintended side effects on unrelated parts. This modularity is crucial for localizing and correcting errors efficiently.
- Independent Causal Mechanisms (ICM): This principle suggests that the process generating a cause should be independent of the mechanism by which it produces its effects. In simpler terms, if A causes B, the way A operates shouldn’t depend on how B is generated, and vice versa. This helps in identifying true causal relationships.
- Compositional Autonomy Principle (CAP): This principle addresses analogical reasoning, ensuring that when an AI transfers knowledge or solutions from one domain to another (e.g., from family trees to corporate hierarchies), the underlying structural relationships are preserved. It ensures that “map-then-compose” (translating elements and then applying operations) yields the same result as “compose-then-map” (applying operations and then translating the result).
These principles aim to create a scaffold for systems that can convert unreachable errors into reachable ones and correct them, fostering a continuous cycle of conjecture, criticism, and knowledge revision. The paper concludes by emphasizing that AGI requires a system capable of “unbounded creation and improvement of explanatory knowledge,” a capability that goes far beyond what current data-driven approaches can offer. For more details, you can read the full paper here.
