TLDR: This research evaluates Large Language Models (LLMs) for detecting positive, negative, or neutral stances on financial targets (debt, EPS, sales) within SEC reports and earnings call transcripts. It introduces a new sentence-level dataset annotated with ChatGPT-o3-pro and human validation. The study finds that few-shot learning combined with Chain-of-Thought prompting yields the best results, with ChatGPT 4.1-mini performing strongest. It also highlights that models perform better on conversational earnings call transcripts than on complex SEC reports, and that selecting semantically similar examples improves accuracy. The findings suggest LLMs are viable for financial stance detection with minimal labeled data.
Financial documents like annual reports (Form 10-K) and earnings call transcripts (ECTs) are crucial for investors, auditors, and regulators. However, their sheer length, complex financial language, and subtle meanings make them difficult to analyze in detail. Traditionally, understanding the sentiment or “stance” (positive, negative, or neutral) towards specific financial topics like debt, earnings per share (EPS), and sales has required large, expensive datasets for training. This new research introduces a novel approach to tackle this challenge.
Researchers from Kansas State University, Mathinvestments, Inc., and the University of Illinois Chicago have developed a sentence-level dataset specifically for detecting stance on three key financial metrics: debt, EPS, and sales. Sentences were extracted from Form 10-K annual reports and ECTs. To ensure high quality, these sentences were labeled for their stance (positive, negative, or neutral) using the advanced ChatGPT-o3-pro model, with rigorous human validation confirming over 97% agreement with the model’s justifications.
The Challenge of Financial Text Analysis
Previous methods for financial sentiment analysis, such as lexicon-based approaches or traditional machine learning, often struggled with the specialized language and context-dependent nuances of financial texts. These methods could identify general optimism or pessimism but failed to pinpoint how sentiment varied for different financial targets. For instance, an increase in debt could be positive (strategic investment) or negative (increased risk) depending on the context. Even advanced deep learning models like BERT and FinBERT, while improving contextual understanding, still required extensive labeled data and struggled with target-specific stance detection.
Leveraging Large Language Models (LLMs)
The emergence of modern Large Language Models (LLMs) offers a promising solution. These models can achieve strong results with minimal or no task-specific labeled data, which is a significant advantage given the high cost of creating financial datasets. This study systematically evaluates how well LLMs perform in detecting stances towards specific financial targets (debt, EPS, sales) at the sentence level.
The researchers compared four LLMs: Llama 3.3, Gemma 3-27B, Mistral 3 Small, and the proprietary ChatGPT 4.1-mini. They tested these models using various prompting strategies: zero-shot (no examples), few-shot (a small number of examples), and Chain-of-Thought (CoT) prompting, which encourages the model to show its reasoning steps. They also explored the impact of providing different levels of contextual information, such as full or summarized sections from SEC reports and ECTs.
Also Read:
- Bridging Language and Structure: A Look at Large Language Models and Text-Attributed Graphs
- The Reasoning Paradox: How LLM Thinking Can Undermine Safety and Hallucination Detection Precision
Key Findings
The study revealed several important insights:
Few-shot with Chain-of-Thought (CoT) Prompting Excels: The combination of providing a few labeled examples and guiding the LLM to think step-by-step (CoT) consistently led to the best performance. This approach outperformed methods without CoT and zero-shot learning.
ChatGPT 4.1-mini Leads: Among the evaluated models, ChatGPT 4.1-mini achieved the highest overall accuracy, followed by Llama 3.3.
Importance of Example Selection: When using few-shot learning, selecting examples that are semantically similar to the test instance significantly improved accuracy compared to randomly chosen examples.
Contextual Information: While providing full or summarized company context improved zero-shot performance for some models (like GPT-4.1-Mini and Gemma3:27B), its benefit diminished when sufficient few-shot examples were provided. This suggests that in-context learning from examples becomes more dominant.
Dataset Differences: Models performed consistently better on Earnings Call Transcripts (ECTs) than on SEC filing reports. This is because ECTs are more conversational and often contain explicit financial changes, making stance detection clearer. SEC reports, being formal and numerically dense, require deeper quantitative reasoning, which remains a challenge for current LLMs. For example, understanding that a decrease in debt-to-capital ratio is positive requires the model to compute and interpret the change, a task where LLMs sometimes struggle.
This research demonstrates the practical potential of using LLMs for target-specific stance detection in finance, reducing the need for extensive, costly labeled data. The authors have made their prompts, data, and code publicly available to support further research in this area. You can find the full research paper here: Evaluating Large Language Models for Stance Detection on Financial Targets.
