TLDR: KG-Agent is a new AI framework that uses a State-Action Knowledge Graph (SA-KG) to help agents learn and plan more efficiently in environments without traditional APIs, relying only on visual interfaces. It connects similar but visually different states to improve exploration and uses a hybrid reward system for better long-term strategy, showing significant performance gains in complex games like Civilization V and Slay the Spire.
In the rapidly evolving landscape of artificial intelligence, a new frontier is emerging: AI agents that can operate without predefined Application Programming Interfaces (APIs). These “API-free” agents interact with software and environments much like humans do, by observing and manipulating Graphical User Interfaces (GUIs) through pixels. While this approach promises ultimate generality and human-like autonomy, it also introduces significant challenges, particularly in efficiency and long-term strategic planning.
Most existing software lacks accessible APIs, forcing AI agents to rely solely on pixel-based GUIs. This limitation often leads to what researchers call “efficiency bottlenecks.” Current large language model (LLM)-based agents, when confined to local visual experiences, tend to make short-sighted decisions and depend heavily on inefficient trial-and-error methods. This hinders their ability to learn new skills and plan for the future effectively.
To tackle these critical issues, a new framework called KG-Agent has been proposed. KG-Agent is an experience-driven learning system designed to transform an agent’s raw pixel-level interactions into a structured, persistent State-Action Knowledge Graph (SA-KG). Think of the SA-KG as the agent’s long-term memory, organizing visual observations into “state nodes” and learned skills into “edges” connecting these states. This innovative approach converts unstructured pixel data into a coherent network that supports strategic decision-making in API-free environments.
One of KG-Agent’s key innovations is its ability to overcome inefficient exploration. It achieves this by linking functionally similar but visually distinct GUI states using “similarity edges.” This creates a rich “neighborhood of experience,” allowing the agent to generalize from a diverse set of past successful strategies rather than getting stuck in isolated, myopic data points. This means the agent can recognize that two different-looking screens might offer similar functional opportunities, drastically reducing the need for repetitive trial-and-error.
For supporting long-horizon reasoning, KG-Agent introduces a hybrid intrinsic reward mechanism. This mechanism is based on the structure of the knowledge graph itself. It combines a “state value reward,” which encourages the agent to pursue known high-value pathways, with a “novelty reward” that promotes targeted exploration of new areas. This clever design helps the agent value “setup actions” – moves that might not yield immediate benefits but are crucial for long-term success – effectively decoupling strategic planning from pure discovery.
The effectiveness of KG-Agent was rigorously tested in two complex, open-ended GUI-based decision-making environments: the strategy game Civilization V and the deck-building roguelike Slay the Spire. The results were impressive, demonstrating significant improvements in exploration efficiency and strategic depth compared to state-of-the-art methods. For instance, KG-Agent achieved higher in-game progression and scores while maintaining a high execution-responsive rate, and notably, with reduced computational overhead in terms of LLM token costs.
Ablation studies further highlighted the importance of KG-Agent’s core components. Removing the similarity edges severely hampered long-term progression, while disabling the graph-driven reward mechanisms reduced strategic coherence. This underscores the necessity of these features for effective open-ended gameplay. The framework showed its capacity to prioritize strategic value over mere visual changes, executing high-reward skills even when the visual impact was minimal, such as “Advance Tur” or “Move Worker” in Civilization V.
Also Read:
- Enhancing LLM Multi-Agent Reasoning Through Strategic Self-Play
- Enhancing AI Model Alignment by Resolving Feedback Inconsistencies
While KG-Agent marks a significant step forward, its current knowledge remains tied to specific environments. Future work aims to enhance its ability to abstract and transfer knowledge across unseen tasks and diverse contexts, moving closer to human-like adaptability and robust general intelligence. This research, published as a conference paper at ICLR 2026, represents a promising direction for the future of AI agents. You can find the full paper here.
