TLDR: A new study analyzed AI model cards, researcher repositories, and real-world incidents to compare how developers and researchers perceive AI risks versus actual harms. It found that developers focus on technical issues, researchers on broader societal impacts, but both largely overlook common real-world harms like fraud, manipulation, and misinformation. The paper emphasizes the need for standardized risk reporting and a more integrated approach to understanding AI’s dangers.
A new research paper titled “The AI Model Risk Catalog: What Developers and Researchers Miss About Real-World AI Harms” delves into how AI risks are perceived and reported by different stakeholders. Authored by Pooja S. B. Rao, Sanja Š´cepanovi´c, Dinesh Babu Jayagopi, Mauro Cherubini, and Daniele Quercia, this study highlights significant discrepancies between the risks identified by AI developers, those envisioned by researchers, and the harms that actually occur in the real world.
The researchers analyzed nearly 460,000 AI model cards from Hugging Face, a popular platform for sharing AI models. From these, they extracted almost 3,000 unique risk mentions, which were then compiled into a comprehensive AI Model Risk Catalog. This catalog was then compared against the MIT Risk Repository, which compiles risks identified by academic researchers, and the AI Incident Database, a collection of real-world AI-related harms.
Understanding Different Perspectives on AI Risks
The study revealed that AI developers primarily focus on technical issues such as model bias, safety concerns, and operational limitations. These are the kinds of problems they encounter directly when building and testing AI systems. For instance, developers frequently report issues like models underperforming on specific data types or generating irrelevant responses. While these technical risks are important and account for a significant portion of real-world harms, they don’t cover the full spectrum.
On the other hand, AI researchers tend to emphasize broader societal impacts, governance challenges, and threats to human agency. Their concerns often include issues like power centralization, increased inequality, and the economic devaluation of human effort. While these are crucial long-term considerations, they appear less frequently in real-world incident reports compared to the attention researchers give them.
The Overlooked Dangers: Fraud, Manipulation, and Misinformation
A critical finding of the paper is a shared blind spot among both developers and researchers: risks related to fraud, manipulation, and misinformation. These types of harms, which often arise from how people interact with AI systems, are significantly more common in real-world incidents than they are anticipated by either group of experts. Malicious uses of AI, such as deepfake scams or disinformation campaigns, account for a substantial portion of recorded harms but receive comparatively less attention in model cards and research frameworks.
The study also observed how risk reporting has evolved over time. Between 2022 and 2024, while the total number of model cards increased, the percentage with completed risk sections actually decreased. However, there was a notable shift in the types of risks mentioned: discrimination and toxicity became more prevalent than AI system safety issues, and there was a fourfold increase in mentions of malicious use risks, alongside a sixfold increase in privacy and security concerns. This suggests a growing awareness of these specific issues, possibly linked to the rise of multimodal AI models.
Also Read:
- The Slippery Slope of AI Exploits: From Simple Hacks to Systemic Misalignment
- Beyond Jailbreaks: Unpacking the True Criminal Potential of Large Language Models
Implications for a Safer AI Future
The research underscores the urgent need for clearer and more structured risk reporting. The authors advocate for a standardized format that specifies the situation and context in which harm might occur, making risk information more actionable for everyone involved. The AI Model Risk Catalog itself serves as a valuable resource, offering granular, model-specific risk data that complements the broader insights from researcher repositories and real-world incident databases.
For researchers, the paper suggests a greater focus on human-interaction and social engineering risks. Developers are encouraged to adopt a broader risk assessment approach, incorporating cybersecurity and privacy-first practices, and utilizing tools that ground risk suggestions in real-world evidence. Media professionals can use these findings to broaden their reporting beyond sensational incidents, while policymakers can leverage the consolidated risk data to develop more targeted regulations and compliance tools.
Ultimately, this research highlights that a comprehensive understanding of AI risks requires integrating perspectives from developers, researchers, and real-world incidents. By bridging these gaps, the AI community can work towards building more transparent, accountable, and safer AI systems for everyone. You can read the full paper for more details: The AI Model Risk Catalog: What Developers and Researchers Miss About Real-World AI Harms.
