TLDR: A JPMorgan Chase & Co. study found that small, proprietary AI models, specifically a Decoder-Only Transformer, can match or surpass large language models (LLMs) like Llama3-8b in accuracy for financial transaction understanding, while being significantly faster and more cost-effective. This specialized approach improved transaction coverage by 14% and is projected to save over $13 million annually by reducing customer inquiry calls.
In the fast-paced world of finance, accurately understanding every transaction is vital for everything from regulatory compliance and fraud detection to enhancing customer experience. However, financial transaction data is often messy, unstructured, and comes in vast volumes, making this a challenging task. Traditional rule-based systems struggle with the sheer variety of merchants and transaction formats, proving inefficient and unscalable.
Enter the era of advanced language models. Transformer-based models, including Large Language Models (LLMs) like GPT, Llama, and T5, have shown remarkable capabilities in processing complex data. Yet, their application in real-time financial production environments comes with its own set of hurdles: high computational costs, latency issues, and the need for extensive fine-tuning to adapt general-purpose models to highly specific financial nuances.
The JPMorgan Chase & Co. Study: Small Models, Big Impact
A recent research paper from JPMorgan Chase & Co., titled “Better with Less: Small Proprietary Models Surpass Large Language Models in Financial Transaction Understanding,” delves into this challenge. The study explores whether powerful, general-purpose LLMs are truly the best solution for specialized tasks like standardizing point-of-sale (POS) transactions, or if smaller, custom-built proprietary models offer a more effective path. You can read the full paper here: Better with Less: Small Proprietary Models Surpass Large Language Models in Financial Transaction Understanding.
The core objective was to take messy transaction text and identify the correct merchant ID, enriching the information available to users on the Chase mobile app. Imagine seeing “SWA * EARLYBRD XQQJWQ9V4F4” transformed into “Southwest Air” with its logo, formal name, and address – a significant improvement for users trying to understand their spending or detect fraud.
Comparing the Contenders: LLMs vs. Proprietary Models
The researchers conducted extensive experiments, evaluating three types of Transformer models: Encoder-Only, Decoder-Only, and Encoder-Decoder. For each type, they compared three options: out-of-the-box pretrained LLMs (like Llama3-8b, Flan-T5, and SBERT), fine-tuned versions of these LLMs, and small proprietary models developed from scratch, specifically tailored for financial transaction data.
The findings were compelling. While fine-tuning significantly improved the performance of LLMs compared to their out-of-the-box versions, the small proprietary models consistently performed comparably or even better, despite having drastically fewer parameters. For instance, a proprietary Decoder-Only model achieved 72.07% accuracy with just 1.7 million parameters, nearly matching Llama3-8b’s 72.89% accuracy, which uses 8 billion parameters.
Speed and Cost: Where Proprietary Models Shine
The real advantage of proprietary models became evident in terms of speed and cost efficiency. Processing over 50 million transactions daily requires models to respond within milliseconds. The proprietary Decoder-Only model processed a transaction in a mere 95.02 milliseconds, while Llama3-8b took 735 milliseconds. This difference is critical for real-time applications and operational costs.
The study highlighted that proprietary models were not only faster in inference but also significantly cheaper and quicker to train. This efficiency translates into substantial budget savings, making them a more practical choice for high-volume, low-latency financial applications.
Also Read:
- Domain-Specific LLMs Outperform General Models in Accounting and Finance, Study Finds
- Large Language Models Tested in Business Simulations: A Benchmark for Managerial AI
Deployment and Business Impact
Based on these results, JPMorgan Chase & Co. chose to deploy the Proprietary Decoder-Only model to handle complex transactions that their previous rule-based and string-distance methods couldn’t manage. This strategic deployment led to a 14% increase in overall transaction coverage, boosting it from 80% to an impressive 94%.
The improved accuracy and coverage directly translated into significant business benefits, including a projected annual cost saving of over $13 million by reducing the number of transaction inquiry calls from customers. This demonstrates that for highly specialized, domain-specific tasks, a tailored approach with smaller, efficient models can yield superior results and substantial economic advantages over general-purpose large language models.
The research underscores a crucial lesson: when it comes to specialized applications like financial transaction understanding, bigger isn’t always better. Customized, proprietary models, designed with domain-specific needs in mind, can offer a powerful combination of accuracy, speed, and cost-effectiveness, ultimately leading to better business outcomes and enhanced user experiences.
