TLDR: AI Harmonics is a new framework for assessing AI risks that focuses on human impact and the severity of harms, rather than just internal compliance. It uses real-world incident data and ranks harms based on their severity without needing exact numbers. The framework helps identify which types of AI harms, like political and physical ones, are most concentrated and thus require urgent attention, offering a robust and adaptable tool for policymakers.
Artificial Intelligence (AI) has become an undeniable force, shaping critical decisions across various sectors like healthcare, finance, and governance. While offering immense opportunities, AI also introduces unprecedented societal harms and risks. Traditional AI risk assessment models often fall short by focusing primarily on internal compliance and neglecting the diverse perspectives of those actually affected by AI systems.
A new research paper, titled “AI Harmonics: a human-centric and harms severity-adaptive AI risk assessment framework,” proposes a significant shift in how we evaluate AI risks. Authored by Sofia Veia, Paolo Giudici, Pavlos Sermpezis, Athena Vakalia, and Adelaide Emma Bernardelli, the paper introduces the AI Harmonics framework, designed to be human-centric and adaptive to the severity of harms, grounded in real-world incident data.
The Problem with Current AI Risk Assessments
Existing frameworks tend to adopt an internal viewpoint, concentrating on mitigating risks for AI providers and adopters through simple compliance measures. These often rely on limited registered risks, internal audits, and predefined checklists, which can fail to recognize new and unforeseen harms. Crucially, they largely overlook the experiences of external stakeholders—users, communities, and affected individuals—who bear the brunt of harmful AI outcomes. Furthermore, these frameworks are often rigid, applying generic fairness or privacy checks across all applications, without adjusting for differences in harm severity or specific domain nuances. For example, the impact of a life-threatening medical misdiagnosis is vastly different from a minor ad bias, yet current systems might treat them with similar generic recommendations.
Introducing AI Harmonics: A Human-Centric Approach
AI Harmonics addresses these limitations by proposing a paradigm shift. It introduces a novel AI harm assessment metric (AIH) that leverages ordinal severity data. This means it captures the relative impact of harms without requiring precise numerical estimates, which are often arbitrary or inconsistently assigned in real-world scenarios. Instead of asking ‘how much’ harm, it asks ‘how much worse’ one harm is compared to another.
The framework combines a robust, generalized methodology with a data-driven, stakeholder-aware approach to explore and prioritize AI harms. It integrates human perspectives by mapping text annotations collected from external stakeholders—such as affected end-users, community representatives, and domain experts—to incidents reported in open repositories like the AIAAIC Repository.
Key Features and Benefits
One of the core strengths of AI Harmonics is its reliance on **ordinal scales** for harm severity. This is crucial because precise numerical scores for harms are often unavailable or unreliable. For instance, while annotators might agree that financial loss is worse than a minor inconvenience but not as severe as bodily harm, they would likely disagree on exact numerical intervals between these harms. Ordinal scales accommodate this uncertainty by requiring only a consistent ordering of severity levels (e.g., Minor < Moderate < Severe), not exact numerical distances.
Experiments conducted on annotated incident data confirm that **political and physical harms exhibit the highest concentration** and therefore warrant urgent mitigation. Political harms can erode public trust, while physical harms pose serious, even life-threatening risks. This finding underscores the real-world relevance of the AI Harmonics approach, enabling policymakers and organizations to target their mitigation efforts effectively by identifying uneven harm distributions.
The framework’s **robustness and adaptiveness** are demonstrated through extensive sensitivity analyses. It shows that harm rankings remain stable even under different severity orderings and annotation noise scenarios. This means that the prioritization of harm categories is not easily swayed by minor variations in how severity levels are ranked or by incomplete data.
Also Read:
- Unpacking AI’s Moral Compass: How Language Models Prioritize Values
- Designing the Future: Navigating the Emergence of AI Agent Economies
Practical Implications
By embedding stakeholder-informed annotations, AI Harmonics directly captures the impact of AI harms on humans, moving beyond internal, compliance-driven processes. It operates effectively on ordinal rankings, allowing for meaningful prioritization without the need for precise numeric inputs. This makes it a flexible and reproducible tool for policymakers and practitioners to pinpoint and mitigate the most critical AI-driven harms. For example, a governing body like the European Union could use this framework to prioritize harm mitigation efforts based on empirical data and expert consensus.
The research paper, available at https://arxiv.org/pdf/2509.10104, provides a detailed explanation of the methodology, its comparison with existing metrics like the Criticality Index, and comprehensive experimental results.
