TLDR: A new benchmark, TaxCalcBench, evaluates large language models (LLMs) on their ability to calculate U.S. personal income taxes. The study found that even state-of-the-art models correctly calculate less than a third of federal tax returns, consistently misusing tax tables, making calculation errors, and incorrectly determining eligibility. The findings highlight the need for additional infrastructure to enable LLMs to reliably perform tax calculations.
Can artificial intelligence handle the complexities of filing your taxes? A new research paper introduces TaxCalcBench, a benchmark designed to evaluate how well frontier AI models can calculate U.S. personal income tax returns. The findings suggest that while AI has made significant strides in coding and math, it’s not yet ready to reliably compute your tax liability.
The paper, titled “TaxCalcBench: Evaluating Frontier Models on the Tax Calculation Task,” was authored by Michael R. Bock, Kara Molisee, Zachary Ozer, and Sumit Shah from Column Tax. It highlights that calculating personal income taxes requires not just understanding vast amounts of English text but also meticulously computing results based on that knowledge.
Understanding the Tax Calculation Challenge
Tax filing typically involves three main steps: collecting documents (like W-2s), preparing and entering information into software, and finally, calculating the actual tax return. TaxCalcBench focuses exclusively on this third, crucial step: calculation. Traditionally, this has been handled by highly specialized “tax engines” – deterministic software built to compute returns with 100% accuracy. Only about a dozen such engines exist, with very few developed in recent decades.
The core difficulty lies in translating the U.S. income tax code, which spans over 75,000 pages and a million lines of text across federal and state regulations, from English into precise programmatic code. Unlike common belief, the IRS doesn’t provide an “answer key” or its own tax engine; each software provider must build their own. The interconnectedness of rules and the sheer volume of dependencies make this task incredibly complex, far beyond simple arithmetic.
Introducing TaxCalcBench
TaxCalcBench is a dataset comprising 51 test cases, each a pair of user inputs and the expected correct tax return output. These test cases represent a modest range of federal-only personal income tax returns for Tax Year 2024 (TY24), covering various filing statuses, income sources, and credits/deductions. The input data is synthetic, formatted in a proprietary JSON, while the expected outputs are in the IRS-specified XML format.
The benchmark was constructed using a subset of tests from Column Tax’s own tax calculation engine, verified by human experts and a deterministic system. This ensures the accuracy of the expected results against which AI models are evaluated.
Experimenting with Frontier Models
The researchers tested state-of-the-art models with a 2025 knowledge cutoff, including Gemini 2.5 Pro, Gemini 2.5 Flash, Claude Opus 4, and Claude Sonnet 4. Models were prompted to calculate a tax return based on the provided inputs and output the results in a simplified text-only format, as generating schema-compatible XML is currently beyond their capabilities.
Evaluation was strict: model outputs were considered correct only if amounts matched exactly for every important line on the main Form 1040. A lenient metric, allowing for a +/- $5 difference, was also used to provide additional insight. Models were tested at various “thinking levels” to see if increased computational budget improved performance.
Key Takeaways: AI’s Current Limitations
The results were clear: frontier models cannot reliably calculate tax returns today. Even the best-performing model, Gemini 2.5 Pro, achieved a strict correctness rate of only around 32%. This is far from the 100% accuracy required for tax calculation.
Models showed inconsistency in their calculations and struggled particularly with complex tax laws, such as those governing the Child Tax Credit or Earned Income Tax Credit, which involve intricate eligibility requirements. However, there were some positive signs: models performed better on the lenient metric, often being only a few dollars off, and showed higher accuracy on a per-line basis, suggesting that single mistakes often cascade through the entire return.
Analysis of Errors
The study identified two primary reasons for model failures:
- Incorrect Use of Tax Tables: Models frequently used percentage-based tax bracket calculations instead of the IRS-mandated lookup tables for taxable incomes under $100,000. This often led to small discrepancies, accounting for the 15-20% difference between strict and lenient scores.
- Calculation Errors: The majority of incorrect returns stemmed from outright calculation errors. Examples included hallucinating incorrect line numbers on forms (like Form 8962 for the Premium Tax Credit) or using incorrect reference values (like the Federal Poverty Level). These errors often led to a chain of incorrect calculations throughout the return.
Model-Specific Observations
Gemini 2.5 Pro emerged as the top performer, though its accuracy didn’t significantly improve beyond a certain thinking budget. Gemini’s API, by default, includes a dynamic thinking budget, which proved beneficial for the task.
Claude Opus 4 was the second-best, but its performance lagged behind Gemini 2.5 Pro, especially at lower thinking budgets. Claude models, both Opus and Sonnet, showed significant performance improvements when their thinking budgets were increased, highlighting that Claude’s API defaults to disabled thinking budgets, which hampers its performance on this benchmark.
Also Read:
- Unpacking How Question Types Affect Large Language Model Performance
- AI’s Achilles’ Heel: Why More Feedback Can Harm Large Language Model Performance
The Path Forward
The TaxCalcBench is a simplified version of the real-world tax calculation task, focusing only on federal returns and a modest range of situations, and not requiring full XML output. The researchers plan to release yearly updates, expanding the dataset to include state returns, more complex scenarios, and eventually requiring models to output the IRS-mandated XML format.
The findings underscore the need for continued development in AI infrastructure and orchestration to augment LLMs, potentially by providing them with the correct tax tables or scaffolding for additional forms, to enable them to reliably compute tax returns. For more details, you can refer to the full research paper here.
