TLDR: AgREE is a novel agent-based framework that uses iterative retrieval and multi-step reasoning to dynamically build knowledge graph triplets, especially for new and emerging entities. It requires no training and significantly outperforms existing methods by intelligently searching the web and reflecting on information to keep knowledge graphs up-to-date, addressing the challenge of incorporating constantly emerging information.
In today’s fast-paced world, new information and entities emerge constantly, from new startups and public figures to novel products and TV shows. This continuous influx poses a significant challenge for Knowledge Graphs (KGs), which are structured representations of entities and their relationships. Traditional methods for completing these graphs often struggle to incorporate these “emerging entities” because they rely on static, pre-trained knowledge that quickly becomes outdated.
Existing approaches typically depend on the pre-existing knowledge within language models, pre-defined queries, or single-step information retrieval. While effective for established entities, these methods often fall short when it comes to capturing comprehensive and up-to-date information about entities that are new or not widely popular. This limitation creates a persistent gap between the static knowledge in KGs and the dynamic real world, impacting applications that require current information, such as question answering and recommendation systems.
Introducing AgREE: A Dynamic Agent-Based Framework
To address this critical challenge, researchers from Apple and Sapienza University of Rome have introduced a novel framework called AgREE: Agentic Reasoning for Knowledge Graph Completion on Emerging Entities. AgREE is an agent-based system designed to dynamically construct rich knowledge graph triplets by combining iterative retrieval actions with multi-step reasoning. Unlike previous methods, AgREE does not rely on static internal knowledge or predefined query templates. Instead, it employs a flexible, adaptive process that progressively builds comprehensive entity representations.
The core of AgREE’s methodology involves an intelligent agent that can strategically use a set of tools to explore the web, self-reflect on the gathered information, and reason to complete KG triplets. This iterative process allows AgREE to adapt to the unpredictable nature of new entities and gather diverse information. The framework is designed to be training-free, meaning it doesn’t require extensive training data or computational overhead for retraining models, making it highly efficient for applications that need frequent knowledge base updates.
How AgREE Works
AgREE operates through a multi-step process. It starts with a query about a specific entity and its relation. The agent first decides whether it has enough internal knowledge to answer directly or if it needs to use external tools. If external information is required, the agent can choose between a basic retriever (like Wikipedia API) and an advanced retriever (like Google search API). This choice is strategic, with the agent often trying the basic tool first before escalating to the advanced one for broader coverage.
After retrieving information, a crucial step called “self-reflection” takes place. The agent evaluates the quality and sufficiency of the retrieved information. If the information is incomplete or imbalanced, the agent initiates additional retrieval rounds with refined queries or different tool choices. This iterative refinement continues until the agent deems the information sufficient or reaches a maximum number of iterations. Finally, the accumulated information is synthesized into a final answer, which undergoes a rule-based format check before being linked to entities in the knowledge graph. For more technical details, you can refer to the full research paper here.
Also Read:
- SEAgent: An AI Framework for Autonomous Software Proficiency
- AI Agents Master Collaboration: A Hybrid Approach to Ad Hoc Teamwork
Performance and Impact
Experiments show that AgREE significantly outperforms existing methods in constructing knowledge graph triplets, especially for emerging entities that were not seen during language models’ training processes. On standard Knowledge Graph Completion (KGC) datasets, AgREE shows performance improvements of up to 13.7% compared to previous models. More impressively, on a newly constructed dataset specifically for emerging entities, AgREE demonstrates up to 45.3% improvement in performance, highlighting its effectiveness in handling truly novel information.
The researchers also identified a fundamental weakness in existing evaluation methodologies for KGC, particularly the standard Hits@N metric, which can be unfair for relations with multiple valid answers. To address this, they proposed a new “relation-aware Hits@N” metric, which adapts the evaluation threshold based on each relation’s inherent cardinality. When evaluated with this fairer metric, AgREE shows even greater performance gaps, ranging from 21.8% to 60.5% over previous baselines.
AgREE’s ability to combine agent-based reasoning with strategic information retrieval represents a promising direction for maintaining up-to-date knowledge graphs in dynamic information environments. This framework is particularly valuable for applications where knowledge bases need to be continuously updated with the latest real-world information without constant retraining.
