TLDR: mKGQAgent is a novel LLM agent framework designed to convert natural language questions into SPARQL queries for knowledge graphs, supporting multiple languages. It employs a human-inspired reasoning process, breaking down complex tasks into modular subtasks, and utilizes an ‘experience pool’ for in-context learning and a feedback mechanism for query refinement. The framework achieved first place in the Text2SPARQL challenge 2025, demonstrating superior performance across various languages. The research also highlights that translating non-English questions into English can often improve question answering quality, while acknowledging the increased computational costs associated with the agent paradigm.
Accessing vast amounts of structured knowledge, often stored in what are known as knowledge graphs, can be challenging, especially when the information is needed in various languages. While knowledge graphs are typically queried using a specific language like SPARQL, the goal is to allow users to ask questions in their natural language and have the system automatically translate those questions into the necessary queries. This process, known as Knowledge Graph Question Answering (KGQA), faces an emerging challenge: extending its capabilities beyond English to support a multitude of languages.
Traditional approaches to KGQA often combine different components, such as rule-based or neural networks, to handle subtasks like named entity recognition or relation detection, eventually leading to an answer. More recently, Large Language Models (LLMs) have been leveraged to directly generate structured queries from non-English input. The advent of LLM agents, which are augmented language models, has further improved performance compared to standard prompting techniques, but their application in multilingual settings has remained largely unexplored.
Introducing mKGQAgent: A Human-Inspired Approach
A new framework called mKGQAgent has been introduced to address these challenges. It takes a human-inspired approach, breaking down the complex task of converting natural language questions into SPARQL queries into smaller, more manageable, and interpretable subtasks. This framework leverages a coordinated LLM agent workflow for crucial steps like planning, identifying entities, and refining queries. A key feature is its ‘experience pool,’ which guides the agent through in-context learning, allowing it to efficiently handle multilingual KGQA.
The effectiveness of mKGQAgent was rigorously evaluated on the DBpedia- and Corporate-based KGQA benchmarks within the Text2SPARQL challenge 2025, where it secured first place among all participants. This achievement opens new avenues for developing reasoning systems that mimic human-like thought processes in multilingual semantic parsing.
How mKGQAgent Works: A Modular Architecture
The mKGQAgent operates in two main phases: an offline phase and an evaluation (online) phase. The offline phase is crucial for building the ‘experience pool,’ a non-parametric memory that stores both successful and unsuccessful SPARQL query generation attempts along with their intermediate steps. This pool helps the agent learn from past interactions.
The core workflow in the evaluation phase involves several steps:
-
Plan Step: This step uses the LLM to generate a structured, step-by-step plan for constructing a SPARQL query from a given question. It simplifies the task by breaking it into subtasks like entity recognition and query refinement. In the evaluation phase, this step benefits from the experience pool by retrieving successful planning examples to inform its current plan.
-
Named Entity Linking (NEL) Tool: Similar to how humans look up resource identifiers in a knowledge graph, the NEL tool interacts with the knowledge graph to retrieve resource labels. This is vital because LLMs might not have direct access to URI-label mappings, which are essential for generating accurate SPARQL queries. The tool identifies and links entity and relation candidates proposed by the LLM.
-
Action Step: Once a plan is generated, the action step executes each task sequentially. It uses the NEL tool for entity linking and also leverages the experience pool to benefit from in-context SPARQL query examples from the training data. This ensures the agent follows the structured plan and interacts with necessary tools to refine the query.
-
Feedback Step: After an initial SPARQL query is generated, the feedback step executes it on a triplestore (a database for knowledge graphs). It collects the response and integrates it into a prompt for the action step, allowing the agent to refine the query based on the execution results. This feedback loop is designed to occur only once per input question to prevent infinite loops.
A significant advantage of this framework is that it does not require supervised fine-tuning, which substantially reduces computational costs and helps preserve the generalizability of the original LLMs.
Also Read:
- DynaSearcher: Advancing AI Search with Knowledge Graphs and Smart Reinforcement Learning
- INRAExplorer: Advancing Scientific Knowledge Exploration with Agentic RAG and Knowledge Graphs
Performance and Multilingual Insights
Experiments were conducted on the QALD-9-plus benchmark, which includes questions in multiple languages. The mKGQAgent, particularly when implemented with GPT-4o, achieved state-of-the-art results on English questions, with an F1 score of 54.83%. It also demonstrated superior quality on German, Spanish, Belarusian, and Bashkir languages. While languages using Cyrillic-based scripts generally showed poorer results compared to Latin-based ones, the system’s overall multilingual performance was robust.
A notable finding was the impact of machine translation. The evaluation showed that translating non-English questions into English before processing them generally led to higher KGQA performance across most models and languages. This effect was particularly pronounced in Russian and Spanish, suggesting that translation can significantly improve results, especially for languages where native processing might be more challenging for LLMs. However, the effectiveness of translation varies by language and model, indicating that some models might already be well-optimized for certain native languages.
The study also analyzed the computational costs, noting that the full mKGQAgent setup, while achieving better quality, requires more LLM calls on average compared to simpler configurations. This highlights a trade-off between performance and computational resources.
In conclusion, mKGQAgent represents a promising advancement in multilingual Text-to-SPARQL tasks. Its human-inspired, modular architecture, combined with an experience pool and feedback mechanism, significantly enhances the accuracy of SPARQL query generation across diverse languages. For more in-depth technical details, you can refer to the full research paper: Text-to-SPARQL Goes Beyond English: Multilingual Question Answering Over Knowledge Graphs through Human-Inspired Reasoning.
