TLDR: A new framework uses Large Language Models (LLMs) to automatically extract and formalize privacy practices from natural-language privacy policies into a knowledge graph called Pr2Graph. This system, grounded in standard vocabularies, aims to make privacy policies understandable and auditable at scale, supporting tasks like automated compliance checking and user decision-making. Benchmarking shows LLMs can effectively perform these tasks, offering a more efficient alternative to manual analysis.
In today’s digital world, we constantly sign up for new online services, each coming with its own set of Terms of Service and Privacy Policies. The truth is, very few of us actually read these lengthy and complex documents. This creates a significant barrier to understanding how our personal data is collected, used, and shared, making it difficult for individuals to make informed decisions about their online privacy.
Existing efforts to address this issue often involve using formal languages to verify policy compliance. However, a major challenge has been the inability to create or acquire these formal policies on a large scale. This is where a new research paper from the University of Oxford steps in, proposing an innovative solution.
Introducing Pr2Graph: An AI-Powered Framework for Privacy Policies
Researchers Rui Zhao, Vladyslav Melnychuk, Jun Zhao, Jesse Wright, and Nigel Shadbolt have developed a semantic-centric framework that leverages state-of-the-art Natural Language Processing (NLP) tools, specifically Large Language Models (LLMs), to automatically understand privacy policies. The core of their work is the creation of Pr2Graph, a knowledge graph designed to capture key information about privacy practices.
The framework works by processing natural-language privacy policies through a sophisticated NLP pipeline. This pipeline uses LLMs to perform several crucial tasks:
- Identifying Data Entities: Recognizing specific types of data mentioned, like “medical data.”
- Identifying Purpose Entities: Pinpointing the reasons for data collection, such as “advertisement purpose.”
- Recognizing Parties: Distinguishing between the service provider (first-party) and other entities (third-party).
- Detecting Actions: Identifying data practices like “data collection,” “third-party data sharing,” or “data security protection.”
- Classifying Entities: Grounding the identified data and purpose entities to a standard vocabulary called the Data Privacy Vocabulary (DPV) for consistent understanding.
- Extracting Relationships: Understanding how different entities and actions are connected within the policy.
Once this information is extracted, it is used to build the Pr2Graph knowledge graph. This graph is centered around the concept of a “DataPractice,” which links together all relevant information: what data is involved, why it’s used, who uses it, and with whom it’s shared. By grounding these terms in formal semantics using DPV, Pr2Graph ensures better interoperability and makes automated analysis much more feasible. Importantly, the system also retains the original text segments, allowing for auditing to ensure the extracted information is accurate.
Also Read:
- Enhancing Legal AI: A Structured Prompting Method for Long Documents
- Mapping Disease Outbreaks: How AI is Structuring WHO’s News for Public Health
Real-World Impact and Evaluation
To demonstrate the practical utility of their framework, the researchers applied their NLP pipeline to the privacy policies of the top-100 most-visited websites. The resulting Pr2Graph for these websites has been released as a public resource, offering a valuable dataset for further research and analysis. This knowledge graph can also be used for downstream tasks, such as automatically constructing formal policy representations like Open Digital Right Language (ODRL) or perennial semantic Data Terms of Use (psDToU).
The team rigorously evaluated the capabilities of different LLMs within their pipeline. They enriched an existing dataset, Policy-IE, with custom annotations created by legal experts, providing a high-quality benchmark. Their experiments, using models like gpt-4o and gpt-4o-mini, showed promising results. Most tasks achieved an f1-score of around 0.9 or higher, indicating that the models are largely successful in correctly identifying the desired entities. While there’s still room for improvement in precisely labeling every entity, the overall performance is reasonably comparable to human annotators, suggesting a path towards faster and more cost-efficient privacy policy analysis.
This research offers a significant step forward in making privacy policies more transparent and understandable. By providing an easily accessible tool for analyzing these complex documents, the Pr2Graph framework and its associated resources lay a solid foundation for auditing the web and empowering users to make better decisions about their data. You can find more details about this research in the paper available at arXiv:2509.01716.
