TLDR: A new study evaluated ten leading large language models (LLMs) on professional privacy and AI governance certification exams from the International Association of Privacy Professionals (IAPP). The research found that several frontier models, including Google’s Gemini 2.5 Pro and OpenAI’s GPT-5, consistently achieved scores exceeding human passing thresholds, demonstrating substantial expertise in privacy law, technical controls, and AI governance. While top models performed exceptionally well across CIPP/US, CIPM, CIPT, and AIGP exams, the study also highlighted domain-specific gaps, particularly in privacy program management, and suggested that LLMs could significantly assist privacy professionals in their work.
The rapid advancement of large language models (LLMs) has sparked crucial discussions about their capabilities and limitations across various professional fields. For privacy experts, a key question is whether these AI systems can offer reliable support in areas like regulatory compliance, privacy program management, and AI governance.
A recent study, titled Can We Trust AI to Govern AI? Benchmarking LLM Performance on Privacy and AI Governance Exams, delves into this very question. Conducted by Zane Witherspoon, Thet Mon Aye, and YingYing Hao from Superset Labs PBC, the research evaluates the performance of ten leading LLMs – including models from OpenAI, Anthropic, Google DeepMind, Meta, and DeepSeek – on industry-standard certification exams from the International Association of Privacy Professionals (IAPP).
Evaluating AI on Human Certification Standards
The study benchmarked these LLMs against official sample exams for four IAPP certifications: CIPP/US (Certified Information Privacy Professional – US), CIPM (Certified Information Privacy Manager), CIPT (Certified Information Privacy Technologist), and AIGP (Artificial Intelligence Governance Professional). These exams are widely recognized as comprehensive assessments of expertise in privacy and AI governance. The models were tested in a closed-book setting, and their scores were compared against IAPP’s human passing thresholds.
Key Findings Across Certifications
The results were quite striking. Several frontier models, such as Google DeepMind’s Gemini 2.5 Pro and OpenAI’s GPT-5, consistently achieved scores that exceeded the standards for human professional certification. This demonstrates a substantial level of expertise in privacy law, technical controls, and AI governance within these advanced LLMs.
On the CIPP/US exam, which covers U.S. privacy laws, top models like OpenAI’s GPT-5 (93.4% correct) and Anthropic’s Claude 3.7 Sonnet (91.1%) showed a strong grasp of legal frameworks. Most large models surpassed the “definite pass” boundary, indicating deep encoding of legal and regulatory text from their training data.
The CIPM exam, focusing on operationalizing privacy within organizations, revealed the widest performance gap. Gemini 2.5 Pro led with 92.2% correct, followed closely by Claude 3.7 Sonnet (91.1%). While eight out of ten models exceeded the human passing threshold, smaller models like Meta-LLaMA-3-8B struggled significantly, highlighting a pronounced difference in capabilities for managerial privacy content.
For the CIPT exam, which assesses technical privacy know-how, DeepSeek-R1 and Gemini 2.5 Pro both achieved 92.2%. This exam showed a clear tiering of scores, with advanced models performing well on foundational principles and privacy by design, but all models showing weaknesses in areas like “Evolving or Emerging Technologies in Privacy.”
The AIGP exam, extending privacy competence into AI ethics and oversight, saw strong results across almost all models. Nine out of ten models cleared the guaranteed pass threshold, with Gemini 2.5 Pro scoring 93.9%. Interestingly, OpenAI’s GPT-5-Mini even slightly outperformed its more advanced counterpart, GPT-5, on this specific exam.
Also Read:
- GPT-5 Achieves Breakthrough in Multimodal Medical Reasoning, Outperforming Human Experts
- Unpacking AI’s Moral Compass: How Language Models Navigate Ethical Dilemmas
Overall Performance and Implications
Aggregating the results across all four exams, Gemini 2.5 Pro emerged as the overall leader with an average score of 92.1%, closely followed by GPT-5 at 91.3%. DeepSeek-R1 also performed strongly with 90.2%. These results suggest that while broad pretraining ensures reliable competence, there are still meaningful gaps, particularly in managerial privacy content, which could be addressed through more targeted training.
The study also found strong correlations between scores on the CIPT, CIPP/US, and AIGP exams, indicating an overlap in legal, technical, and AI governance knowledge. However, CIPM scores showed a weaker correlation with the others, suggesting that privacy program management requires a somewhat distinct skill set.
In conclusion, the research indicates that the best LLMs can match or even surpass human-level performance on these complex privacy and AI governance certification exams. This suggests that LLMs have the potential to significantly assist privacy professionals in tasks such as drafting policies, answering compliance questions, and conducting automated risk checks, thereby augmenting, automating, or accelerating their work. While further development is needed to address remaining knowledge gaps, the ability of AI to pass human benchmarks in these high-stakes domains is a significant step forward in the governance of AI itself.
