Building Trustworthy AI: A Framework for Private, Verifiable, and Auditable Systems

TLDR: Tobin South’s doctoral thesis presents a comprehensive framework for creating secure, accountable, and privacy-preserving AI systems. It addresses critical challenges in AI development and deployment by integrating advanced cryptographic techniques like zero-knowledge proofs (zkSNARKs) for verifiable model evaluations and data attestations. The research also introduces solutions for private data access in AI, such as Private Retrieval Augmented Generation (PRAG) using multi-party computation (MPC) and Trusted Execution Environments (TEEs). Furthermore, it proposes authenticated delegation and personhood credentials to manage and audit the actions of autonomous AI agents, ensuring they operate within defined permissions and are accountable to human users. The work emphasizes combining these technologies to build robust, end-to-end secure AI systems that balance innovation with societal values.

In an era where artificial intelligence is rapidly becoming the invisible backbone of our modern world, ensuring its security, accountability, and trustworthiness is paramount. A groundbreaking doctoral thesis by Tobin South, titled Private, Verifiable, and Auditable AI Systems, delves into the complex interplay of privacy, verifiability, and auditability in modern AI, particularly within powerful foundation models. This work argues that integrating these elements through technical solutions is crucial for responsible AI innovation.

The thesis highlights that our increasing reliance on AI necessitates robust frameworks to manage its security and accountability. It identifies key risks in the AI pipeline, drawing from international policy contributions and technical research, and introduces novel solutions for critical privacy and verifiability challenges.

Understanding the Core Concepts

Before diving into the solutions, it’s essential to understand three core concepts: privacy, auditability, and verifiability. Privacy, in this context, refers to the right of individuals to control their personal information and how AI systems use their data. Auditability is the ability to understand how AI systems are used and deployed, ensuring they behave as expected and allowing for traceability when issues arise. Verifiability is the capability to confirm that specific properties or claims about an AI system are true, often using technical methods, without needing to blindly trust a third party.

Risks to Privacy from AI Systems

The research categorizes privacy risks from general-purpose AI into three main groups:

Training Risks: These relate to AI models memorizing sensitive data during training, potentially allowing users to infer or reconstruct private information. Many systems are trained on publicly available data without explicit consent, leading to concerns about data leakage.
Use Risks: These arise when AI systems handle sensitive information during their application, such as personalized responses using private data in Retrieval Augmented Generation (RAG). This can lead to information leakage to third parties.
Intentional Harm Risks: Malicious actors can exploit AI’s advanced information processing capabilities for privacy abuse, such as identifying individuals across platforms, inferring sensitive characteristics, or creating non-consensual deepfakes.

A Roadmap for End-to-End Security

The thesis proposes a comprehensive roadmap for end-to-end privacy and security in generative AI, addressing these risks. It defines security goals based on two attacker models:

Internal Attacker: An entity within the computational pipeline (e.g., a malicious model provider) trying to extract sensitive data or tamper with computation.
External Attacker: An entity with only black-box access to the model (e.g., an end-user) trying to extract training data or model weights by querying the model.

To counter these, the paper outlines various technical solutions:

For Privacy (Internal Attacker): Homomorphic encryption (HE), multi-party computation (MPC), confidential computing (Trusted Execution Environments – TEEs), and on-premise hosting.
For Privacy (Training Data Specific): Federated learning and synthetic data.
For Privacy (User Input Specific): Local execution and Private Information Retrieval (PIR).
For Privacy (External Attacker): Differential privacy (DP) and other statistical noising approaches.
For Verifiability: Remote attestation of confidential computing, zero-knowledge model proofs (zkSNARKs), and attestations about data.

These solutions are not mutually exclusive and can be combined to provide robust, layered security across the AI supply chain, from data collection and pre-training to fine-tuning and inference.

Verifiable Claims About Models and Data with zkSNARKs

A significant part of the research focuses on zero-knowledge cryptography, specifically zkSNARKs (Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge). These powerful tools allow someone to prove a statement is true without revealing any additional information beyond the truth of the statement itself. This is crucial for:

Verifiable Model Evaluations: Proving a model’s performance or fairness on a benchmark dataset without revealing its proprietary weights or architecture. This allows for independent verification of claims made by model developers, even for closed-source models.
Verifiable Computation of Partial AI Systems: Addressing the computational cost of zkSNARKs by selectively proving only critical sub-computations, such as a fine-tuned classifier head or specific layers in a LoRA-tuned model. This balances verifiability with practical performance.
Portable Data: Enabling verifiable statements about data without revealing the data itself. This can be used for privacy-preserving training data attestations (e.g., proving compliance with copyright or privacy standards) or for selective disclosures of sensitive information, like in a “redact-and-prove” system.

Private Retrieval Augmented Generation (RAG)

The thesis also tackles the privacy and auditability challenges in Retrieval Augmented Generation (RAG), where LLMs retrieve external information to answer queries. While RAG enhances auditability by grounding model outputs in specific sources, it introduces privacy risks if queries or retrieved data are sensitive. The research proposes two main technical solutions:

Private Retrieval Augmented Generation (PRAG) with MPC: A novel approach using multi-party computation (MPC) to securely query distributed, private databases. This ensures that no single server observes a client’s query or the database content, allowing for private similarity searches over federated vector stores.
TEEs for Private Data Management and RAG: Utilizing Trusted Execution Environments (TEEs) to create secure enclaves for confidential data pooling, management, and RAG. Recent advancements, like NVIDIA’s H100 GPU secure enclaves, allow high-performance LLM inference to occur within these protected hardware environments, creating an end-to-end confidential pipeline.

Security with Agentic AI

As AI agents become more autonomous and capable of taking actions on behalf of users, new security challenges arise. The thesis addresses these by proposing:

Authenticated Delegation for AI Agents: A framework that extends existing authentication protocols (like OAuth 2.0 and OpenID Connect) to allow human users to securely delegate and restrict permissions for AI agents. This ensures traceability, accountability, and clear access controls for agent actions.
Personhood Credentials: Digital credentials that allow users to prove they are real people (not AIs) to online services without disclosing personal information. This helps in distinguishing human users from AI agents, protecting online spaces from scalable deception, and linking AI agent actions to verified human principals for accountability.

Also Read:

Tying It All Together

The thesis concludes by demonstrating how these diverse technologies—zkSNARKs, TEEs, MPC, and authorization protocols—can be combined to create end-to-end secure and auditable AI systems. This layered approach allows for tailored solutions that balance performance, security guarantees, and trust assumptions across the entire AI lifecycle. By integrating these sophisticated technical approaches, the research provides practical blueprints for system designers and informs policy discussions on AI safety and governance, paving the way for a more trustworthy and responsible AI future.