TLDR: FINDER is a novel AI framework designed to significantly improve large language models’ (LLMs) ability to perform complex numerical reasoning in the financial domain. It employs a two-step process: a generative retriever extracts relevant facts from financial documents, and then a Program of Thought prompting mechanism with dynamically selected in-context examples guides an LLM (like GPT-4) to generate executable Python code for answers. This approach achieved new state-of-the-art performance on FinQA and ConvFinQA datasets, demonstrating enhanced accuracy and parameter efficiency for financial numerical tasks.
Large language models (LLMs) have made incredible strides in understanding and generating human-like text, but they often hit a wall when it comes to complex numerical reasoning, especially in specialized fields like finance. Financial data, often presented in a mix of unstructured text and detailed tables, requires not just information extraction but also precise logical inference and mathematical computation. This is where even minor errors can lead to significant inaccuracies.
A new research paper titled “Program of Thoughts for Financial Reasoning: Leveraging Dynamic In-Context Examples and Generative Retrieval” introduces a novel framework called FINDER. Developed by Subhendu Khatuya, Shashwat Naidu, Pawan Goyal, and Niloy Ganguly from the Indian Institute of Technology Kharagpur, FINDER aims to significantly boost LLMs’ capabilities in financial numerical reasoning.
How FINDER Works: A Two-Step Approach
FINDER operates through a clever two-step process, designed to overcome the limitations of existing AI models in handling financial numerical tasks:
1. Relevant Fact Retriever: The first step involves an instruction-tuned generative model, specifically a fine-tuned FLAN-T5 Large model. This retriever’s job is to accurately identify and extract only the most relevant facts from diverse financial data, including both text and tables. Unlike older methods that might rely on fixed rules or score-based selections, FINDER’s generative retriever dynamically pulls out information, ensuring it’s precise and context-aware. This is crucial because providing too much irrelevant information can confuse even powerful LLMs.
2. Target Answer Computation: Once the relevant facts are retrieved, they are fed into a powerful decoder-only LLM, such as GPT-4. This model uses a technique called Program of Thought (PoT) prompting. Instead of directly giving an answer, the LLM generates executable Python code that outlines the reasoning steps. This code is then run by an external interpreter (SymPy) to get the final, accurate numerical answer. A key innovation here is the dynamic selection of “in-context examples.” FINDER doesn’t rely on static examples; instead, it intelligently selects the most relevant examples for a given question using clustering techniques and a policy gradient method. This ensures the LLM is guided by the most appropriate prior problem-solving patterns, enhancing its ability to generalize across different financial scenarios.
Breaking Benchmarks in Financial AI
FINDER has achieved impressive results, setting new state-of-the-art performance benchmarks on two widely recognized financial numerical reasoning datasets: FinQA and ConvFinQA. It surpassed the previous best model, APOLLO, with execution accuracy improvements of 5.98% on FinQA and 4.05% on ConvFinQA. This means FINDER is significantly more accurate in solving complex financial questions.
Beyond its accuracy, FINDER also demonstrates remarkable parameter efficiency. Its FLAN-T5 retriever requires only 0.59 million trainable parameters, a stark contrast to some state-of-the-art retrievers that need hundreds of millions of parameters. This makes FINDER a more efficient and scalable solution.
Also Read:
- VeritasFi: An Advanced RAG System for Multi-modal Financial Data
- FinVet: A Smarter Way to Detect False Financial Claims
Robustness and Future Directions
The framework’s robustness was also tested by integrating it with other LLMs like Gemini-2.0 and GPT-3.5-turbo for the target computation module. Even with these alternative LLMs, FINDER continued to outperform existing state-of-the-art models, highlighting the strength of its overall design.
While FINDER excels in many areas, particularly with table-only questions and single-step programs, the researchers acknowledge areas for improvement. Error analysis revealed that logical errors, such as misinterpreting sign conventions or percentage representations, are the most common issues. Future work will explore integrating external financial knowledge and expanding the framework to other domains requiring complex numerical analysis.
This research marks a significant step forward in making AI more reliable and capable for intricate financial tasks, paving the way for more sophisticated financial analysis tools. You can read the full research paper here.
