TLDR: The research paper introduces FinReflectKG – MultiHop, a new benchmark for multi-hop financial question answering. It demonstrates that using knowledge graphs to precisely retrieve evidence significantly improves the accuracy and efficiency of large language models (LLMs) in answering complex financial queries. Compared to traditional text-based retrieval, KG-guided methods boost correctness scores by approximately 24% and reduce token utilization by about 84.5%, highlighting the value of structured knowledge in financial reasoning.
In the complex world of finance, getting accurate answers to intricate questions often means sifting through vast amounts of information spread across numerous documents, companies, and years. This challenge is particularly acute for multi-hop questions, which require connecting several pieces of information to form a complete answer. Traditional methods, often relying on large language models (LLMs) with broad text searches, struggle with this, leading to either incorrect answers or inefficient use of computational resources.
A new research paper introduces a groundbreaking solution: FinReflectKG – MultiHop. This benchmark and methodology highlight the critical role of Knowledge Graphs (KGs) in making financial question answering more precise and efficient. The core idea is that by guiding LLMs with structured, KG-linked evidence, they can focus their reasoning on composing already retrieved facts, rather than getting lost in noisy, irrelevant context.
Understanding the Challenge
Imagine asking a question like, “How did a specific macroeconomic event impact a company’s risk factors, and how did the market react to that event?” To answer this, an LLM would need to find information about the company, its disclosed risks, the macroeconomic conditions, and financial market indices, potentially across different annual reports and financial statements. This is a multi-hop query, demanding connections between disparate pieces of information.
Existing approaches often fall short because financial data has unique characteristics: it’s highly temporal (changes year-to-year), structural (organized in specific ways within filings), and organizational (spread across different companies). Current benchmarks for LLMs haven’t fully addressed these finance-specific complexities, especially when it comes to multi-document and multi-year reasoning.
Introducing FinReflectKG – MultiHop
The researchers behind FinReflectKG – MultiHop have built a benchmark on top of FinReflectKG, a sophisticated, temporally indexed financial Knowledge Graph. This KG links audited financial facts to their original source chunks from S&P 100 company filings between 2022 and 2024. Essentially, it creates a map of financial information, showing how different facts are connected.
To create the benchmark, the team mined frequent 2-hop and 3-hop subgraph patterns across various financial sectors. These patterns represent common analytical questions a financial professional might ask. Using these patterns, they generated financial analyst-style questions, each with exact supporting evidence directly from the Knowledge Graph. A rigorous two-phase pipeline, involving pattern-specific prompts and multi-criteria quality control, ensured the validity and relevance of these questions and answers.
How it Works: Precise Evidence Retrieval
The benchmark evaluates LLMs under three controlled retrieval scenarios:
- S1: Precise KG-linked paths: The LLM is given only the exact, interconnected chunks of information from the Knowledge Graph that are needed to answer the question. This is like giving the LLM a direct, clean path to the answer.
- S2: Text-only page windows: The LLM receives broader text windows centered around the relevant text spans. This simulates a more traditional semantic search where the model gets a larger, but still somewhat focused, chunk of text.
- S3: Relevant page windows with distractors: This scenario adds random, irrelevant pages or semantically similar but non-answering chunks to the page windows. This mimics real-world retrieval noise, where a search might return some useful information mixed with less relevant data.
Key Findings: Knowledge Graphs Lead the Way
The results were compelling. Across various reasoning and non-reasoning LLMs, KG-guided precise retrieval (S1) consistently delivered substantial gains. Specifically, it boosted correctness scores by approximately 24% while dramatically reducing token utilization by about 84.5% compared to the page-window setting (S2), which reflects the traditional vector retrieval paradigm.
This means that not only did the LLMs answer more accurately when guided by the Knowledge Graph, but they also did so much more efficiently, consuming significantly fewer computational resources. The improvement was even more noticeable for smaller LLMs, underscoring the importance of efficient retrieval for models with limited capacity.
The study also found that intra-document questions (where all evidence is within a single filing) were generally easier, while inter-year questions (same company across different years) proved most challenging. Cross-company questions (different companies in the same year) showed moderate accuracy, likely because financial contexts are often aligned across firms for the same year.
Also Read:
- Enhancing Access to UK Clinical Guidelines with AI: A RAG System for Healthcare
- Guiding LLM Reasoning: A New Approach to Maintain Focus in Complex Tasks
Looking Ahead
The FinReflectKG – MultiHop benchmark, with its curated subset of 555 QA pairs, offers a robust testbed for financial reasoning. It spans intra-document, inter-year, and cross-company scopes, highlighting the pivotal role of knowledge graphs in efficiently connecting evidence for multi-hop financial QA. The researchers plan to expand the dataset, incorporate more expert audits, and diversify LLM judges to further enhance the benchmark’s reliability.
This work paves the way for more trustworthy, interpretable, and cost-efficient financial question answering systems. By grounding LLMs in structured knowledge, we can build more effective tools for navigating the complexities of financial disclosures. You can read the full research paper here.
