TLDR: ODKE+ is a production-grade system developed by Apple Inc. that uses large language models (LLMs) and ontology-guided prompting to automatically extract and ingest millions of high-precision, open-domain facts from web sources into knowledge graphs. It features a modular pipeline including evidence retrieval, hybrid extraction, LLM-based grounding for verification, and corroboration for normalization and ranking, significantly improving knowledge graph coverage, freshness, and accuracy compared to traditional methods.
Knowledge graphs (KGs) are essential for many artificial intelligence applications, providing a structured way to organize vast amounts of information. However, keeping these KGs current and complete is a significant and expensive challenge. The web is constantly changing, presenting issues of sheer volume, diverse information formats, varying trustworthiness, and the rapid pace of updates. Traditional methods often struggle to keep up, relying on manual curation or rigid rule-based systems that don’t scale well.
Introducing ODKE+ for Automated Knowledge Extraction
To tackle these challenges, Apple Inc. researchers have developed ODKE+ (Ontology-Guided Open-Domain Knowledge Extraction), a production-grade system designed to automatically extract and integrate millions of facts from web sources with high accuracy. This innovative system leverages large language models (LLMs) and an ontology-guided approach to ensure extracted information is not only precise but also consistent with the existing knowledge graph structure.
How ODKE+ Works: A Modular Pipeline
ODKE+ operates through a series of interconnected, modular components, forming a scalable pipeline:
The Extraction Initiator is the first step, identifying facts that are either missing from the knowledge graph or have become outdated. It often does this by monitoring updates to web pages, such as Wikipedia articles, which signal potential new or changed information.
Next, the Evidence Retriever collects supporting documents from trusted sources, primarily Wikipedia in the current implementation. This ensures that the system works with reliable information.
The core of the system lies in its Knowledge Extractors, which employ a hybrid strategy. Pattern-based extractors are used for highly structured data, like Wikipedia infoboxes, while more flexible LLM-based extractors handle a wider variety of contexts and data formats. A key innovation here is the use of “ontology snippets” – dynamically generated, tailored schema fragments that guide the LLMs to extract facts that align perfectly with the knowledge graph’s predefined structure and constraints. This allows for scalable, type-consistent fact extraction across a wide range of predicates.
Following extraction, a lightweight Grounder module acts as a validator. It uses a second LLM to verify whether the extracted facts are explicitly supported by the provided source context, significantly reducing the risk of LLM hallucinations and ensuring factual accuracy.
Finally, the Corroborator takes the validated candidate facts and performs normalization, ranking, and consolidation. This step resolves potential conflicts, standardizes values (e.g., converting “6 ft” to “183 cm”), and assigns confidence scores to determine the most trustworthy information for ingestion.
The system then proceeds with Data Export & KG Ingestion, where high-confidence, normalized facts are added to the knowledge graph. ODKE+ supports both batch processing for large-scale periodic updates and streaming modes for near real-time ingestion of ongoing changes, making it highly adaptable to different operational needs.
Real-World Impact and Performance
ODKE+ has been successfully deployed in a production knowledge graph infrastructure since May 2025. It processes an average of 150,000 to 250,000 new facts daily in streaming mode for high-priority entities, with most updates completed within two hours. The system demonstrates impressive performance metrics:
It achieves a remarkable 98.8% precision in factual accuracy, consistently verified through weekly audits.
ODKE+ provides 10% more answerable facts for user queries compared to previous pipelines.
It significantly improves freshness, with newly extracted facts appearing approximately 50 days earlier than in legacy workflows.
The grounding module alone reduced hallucinated extractions by 35%, and the corroboration process further refined factual precision from 91% (raw LLM output) to 98.8%.
Overall, ODKE+ has extracted 19 million high-confidence facts, supporting 195 predicates across 4 million Wikipedia pages.
The researchers emphasize that ODKE+ showcases the viability of LLM-based extraction in real-world knowledge graph pipelines, offering a blueprint for deploying trustworthy, ontology-aligned fact extraction at scale. For more in-depth technical details, you can refer to the full research paper here.
Also Read:
- Unlocking Advanced Question Answering with KERAG’s Knowledge Graph Approach
- SePA: An AI Agent for Proactive and Personalized Health Coaching
Ethical Considerations
The development team also addressed ethical considerations, acknowledging the risk of bias propagation from LLM training data. They highlight that ODKE+ exclusively processes public sources like Wikipedia, avoids user-generated or private data, and incorporates robust verification steps to mitigate hallucination risks. Computational efficiency is also optimized through hybrid operations, using LLMs selectively.
