TLDR: The FinRpt research introduces the first formal definition and open-source benchmark for Equity Research Report (ERR) generation using LLMs. It provides a high-quality dataset constructed from seven financial data types and a comprehensive 11-metric evaluation system. The paper also proposes FinRpt-Gen, a multi-agent LLM framework trained with SFT and RL, which significantly outperforms single LLMs and even leading commercial models in generating detailed and accurate ERRs.
Equity Research Reports (ERRs) are vital documents for investors, offering in-depth assessments of a company’s financial health, market standing, and investment potential. Traditionally, these reports are crafted by professional analysts, a process that demands specialized financial knowledge, market understanding, and considerable time. However, recent advancements in Large Language Models (LLMs) are opening new avenues for automating this complex task.
A groundbreaking new research paper introduces FinRpt, an open-source evaluation benchmark and a multi-agent framework designed to automate the generation of Equity Research Reports. This work addresses significant challenges in the field, including the scarcity of suitable data and the absence of comprehensive evaluation metrics for ERR generation. The paper, titled “FinRpt: Dataset, Evaluation System and LLM-based Multi-agent Framework for Equity Research Report Generation,” is a collaborative effort by Song Jin, Shuqi Li, Shukun Zhang, and Rui Yan. You can find the full research paper here: FinRpt Research Paper.
Addressing Data Scarcity and Evaluation Gaps
The FinRpt benchmark tackles the data scarcity issue by presenting a novel Dataset Construction Pipeline. This pipeline automatically integrates seven diverse financial data types, including company information, financial indicators, announcements, news, historical stock prices, and market indices. This robust data collection process results in a high-quality ERR dataset, which is crucial for training and evaluating models effectively. The dataset comprises 6,825 samples from 800 stocks in the Chinese CSI800 Index, covering a period from September to November 2024.
Beyond data, FinRpt also introduces a comprehensive evaluation system featuring 11 distinct metrics. These metrics go beyond traditional natural language processing (NLP) evaluations like ROUGE-L and BERTScore. They include “CompletionRate” and “Accuracy” for recommendation ratings, but also specialized LLM-based evaluations such as “Financial Numeric” (FN) for data precision, “News” for relevance of news analysis, “Company & Market & Industry” (CMI) for insights into management and market trends, “Invest” for the logic of investment recommendations, “Risk” for thoroughness of risk analysis, and “Writing” for overall coherence and readability. This holistic approach ensures a more accurate assessment of generated ERRs.
Introducing FinRpt-Gen: A Multi-Agent Approach
Recognizing the complexity of ERR generation, the researchers propose FinRpt-Gen, a multi-agent framework specifically tailored for this task. This framework decomposes the intricate process into three main modules: an Information Extraction Module, an Analysis Module, and a Prediction Module. These modules involve nine specialized agents working collaboratively.
For instance, the Information Extraction Module includes agents like the News Extraction Agent, Income Extraction Agent, Balance Extraction Agent, and Cash Extraction Agent, which meticulously gather and process relevant data. The Information Analysis Module then uses agents like the Finance Analysis Agent, News Analysis Agent, Status Analysis Agent, and Risk Analysis Agent to summarize financial health, assess news impact, derive management insights, and identify potential risks. Finally, the Prediction Module’s Prediction Agent synthesizes all this information to forecast investment potential and recommend a “buy” or “sell” rating.
Training and Performance
The FinRpt-Gen framework leverages Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to train its LLM-based agents on the proposed datasets. Experimental results demonstrate the high quality of the FinRpt dataset and the strong performance of FinRpt-Gen. The multi-agent framework significantly outperforms single LLMs, and fine-tuning further enhances its capabilities, even surpassing leading closed-source models like GPT-4o and Gemini-2.5-Pro in many evaluation aspects, especially after the RL phase.
The ablation study further validates the necessity of each component within FinRpt-Gen, showing a clear performance drop when specific extraction or analysis agents are removed or consolidated. Human evaluation studies also confirm the high quality of the FinRpt dataset, with generated reports closely matching expert-written ERRs.
Also Read:
- AI Prompts for Finance: A Self-Optimizing Approach Using Synthetic Data
- A New Framework for Evaluating Financial Information Retrieval in Banking
Future Implications
This research marks a significant step towards fully automating Equity Research Report generation. By providing an open-source benchmark, a robust dataset, and an effective multi-agent framework, FinRpt has the potential to drive innovation in the financial technology (Fintech) sector. It could enable organizations and researchers to generate timely insights and recommendations, allowing for quicker responses to market changes and industry trends, and ultimately improving stock price prediction and other financial applications.
