TLDR: Marco Grossi’s paper critiques the Swiss Cheese model for assessing AI existential risk, arguing that its layers of protection are interdependent, not independent, leading to a higher probability of “Doom” than initially calculated. He emphasizes the crucial distinction between quantifiable “risk” and unquantifiable “uncertainty,” introducing “State-Space Uncertainty” (unknown unknowns) and “Option Uncertainty” (unforeseen consequences of actions) as critical factors overlooked by linear models. Grossi concludes that incorporating these uncertainties is vital for a more accurate understanding and effective strategy against AI existential threats.
A recent paper by Marco Grossi, titled “What are the odds? Risk and uncertainty about AI existential risk,” offers a critical commentary on existing models used to assess the likelihood of AI-related catastrophic events. Published on January 23, 2025, this work delves into the philosophical underpinnings and limitations of ‘linear’ risk models, particularly focusing on the Swiss Cheese model as applied to AI existential risk by Cappelen, Goldstein, and Hawthorne (2025).
Grossi’s paper serves as a crucial reminder that while models provide a structured way to think about complex problems, they often simplify reality in ways that can significantly impact our understanding of risk. The core of his analysis revolves around the model proposed by Cappelen et al., which uses a Swiss Cheese analogy to estimate the probability of “Doom” (P(D)) – the failure of all protective layers against AI existential risk.
The Swiss Cheese Model and Its Layers
The Swiss Cheese model, originally conceived by Reason (1997), visualizes safety barriers as slices of Swiss cheese. Each slice represents a protective layer, and the holes within them represent failures. An accident occurs when the “arrow” of danger passes through all the holes in sequence. Cappelen et al. applied this to AI existential risk with four specific layers:
Technical Plateau: Scientific limitations prevent AI from becoming excessively powerful.
Cultural Plateau: Humanity collectively bans research into extremely powerful AI.
Alignment: Powerful AI systems do not harm humanity because their goals are aligned with ours.
Oversight: We can reliably detect and disable dangerous powerful AI systems.
The original proponents of the Swiss Cheese model emphasize the reliability and independence of these layers. However, Grossi argues that the model used by Cappelen et al. defines these layers in a way that makes them sequentially dependent, meaning each layer activates only if the previous ones fail. This is explicitly stated by Cappelen et al. regarding Oversight, which is considered only if previous layers like Alignment fail.
The Interdependence Problem and Probability of Doom
Grossi contends that this interdependence is not merely a definitional choice but significantly impacts the estimation of P(D). He illustrates this by refining the probabilities. If one assumes epistemic indifference, assigning a 50% chance of success to each layer conditional on the previous one failing, the initial P(D) is calculated as 0.5 x 0.5 x 0.5 x 0.5 = 6.25%. However, Grossi points out that certain layers are inherently linked. For instance, if AI becomes powerful, the best methods for Oversight might involve AI itself (O1), which presupposes AI alignment. If AI is not aligned (¬A), then O1 cannot succeed. By distinguishing between Oversight methods that rely on AI alignment (O1) and those that do not (O2), and assuming indifference between them, the probability of Oversight succeeding given Alignment failure (P(O/¬A)) drops. This refinement increases P(D) from 6.25% to 9.375% – a significant 33% increase.
He extends this argument to Cultural Plateau and Oversight, suggesting that a failure in one might imply a failure in the other, further increasing the calculated P(D) to 10.416%. Grossi’s core message here is that even under epistemic indifference, a more nuanced understanding of layer interdependencies reveals a higher risk of AI existential doom than initially suggested by a simplistic application of the Swiss Cheese model.
Risk Versus Uncertainty: A Crucial Distinction
A central theme of Grossi’s paper is the distinction between ‘risk’ and ‘uncertainty,’ drawing on the works of Knight (1921) and Keynes (1937). Risk refers to situations where probabilities are known (e.g., the odds in a dice roll), while uncertainty describes situations where probabilities are unknown or undefined. Grossi argues that while uncertainty is a form of risk, it often remains unquantified and unrepresented in linear models like the Swiss Cheese model, despite its relevance to P(D).
Limitations of Linear Models and Types of Uncertainty
Swiss Cheese models, with their linear progression and independent barriers, struggle to capture the complexity of systems influenced by social, cultural, technological, and political factors. AI, being a rapidly evolving technology, presents a constantly moving target. Grossi identifies two key types of uncertainty that are often overlooked:
State-Space Uncertainty (OU): This refers to the unknown unknowns – the possibilities we haven’t even considered. The “catch-all hypothesis” or “other survival stories” attempts to address this, but Grossi highlights challenges like the “No-calibration problem” (we don’t know how many unknown stories exist) and the “Unknown Relations problem” (we don’t know how these unknown elements interact). Simply adding a catch-all with an arbitrary probability can drastically alter P(D) without a sensible basis.
Option Uncertainty (SU): This concerns the unknown consequences of our chosen actions. Grossi introduces the concept of “reflexivity,” where our assessment of risk and subsequent actions can actually influence the risk itself. He provides compelling examples:
Cultural Plateau and Oversight: If humanity attempts to implement a Cultural Plateau by halting AI research, but a powerful AI emerges anyway (perhaps through recursive self-improvement), our outdated methods for control might make Oversight less effective, thus increasing the risk of Doom. Our attempt to reduce risk inadvertently increases it.
Cultural Plateau and Alignment: Similarly, if humanity tries to prevent AI’s existence and fails, a powerful AI might perceive humanity as a threat, leading to dis-alignment. This scenario, inspired by “Roko’s basilisk,” suggests that efforts to prevent AI could paradoxically make alignment harder.
Also Read:
- Enhancing Decision-Making: A Framework for Human-AI Uncertainty Collaboration
- Navigating Trust: When Should We Defer to AI’s Expertise?
Conclusion: Towards a More Nuanced Understanding
Marco Grossi’s commentary underscores that while models like the one by Cappelen et al. provide a valuable starting point for systematic thinking about AI existential risk, they must evolve to incorporate the profound dimensions of uncertainty. By acknowledging the interdependence of protective layers and the inherent unpredictability of “unknown unknowns” and the consequences of our actions, we can develop a more realistic and nuanced understanding of the likelihood of AI-related catastrophic events. This deeper insight is crucial for formulating optimal strategies for humanity’s survival in an increasingly AI-driven world. For a deeper dive into this critical discussion, you can read the full research paper here.
