TLDR: Large Language Models (LLMs) were tested on their ability to understand, evaluate, and generate U.S. tax-minimization strategies using a novel dataset called Shelter Check. While LLMs showed mixed performance, one model (o1-preview) notably generated an entirely new, working tax strategy, highlighting their potential to assist tax agencies in identifying and combating tax abuse.
Tax minimization schemes represent a significant challenge for governments worldwide, costing the U.S. government alone over $600 billion annually. This complex problem, involving vast amounts of data and millions of words of constantly updated tax-law authorities, is uniquely suited for artificial intelligence solutions. A recent research paper, Can LLMs Identify Tax Abuse?, delves into the capabilities of large language models (LLMs) in understanding, evaluating, and generating U.S. tax-minimization strategies.
Introducing Shelter Check: A Novel Dataset for Tax Strategy Analysis
To rigorously test LLMs, the researchers developed a unique dataset called Shelter Check. Handcrafted by U.S. tax-law domain experts, this dataset comprises 36 known U.S. tax-minimization strategies. Each strategy is meticulously detailed, including the relevant tax-law authorities (statutes, regulations, case law), background facts necessary for its use, the specific goals of the scheme (e.g., creating an artificial tax deduction), the concrete steps involved, and a legal analysis explaining how the authorities apply to achieve the goals. Crucially, the dataset also includes an ‘adversarial step’ for each strategy, designed to make the scheme definitively fail, allowing for testing the LLMs’ ability to identify non-working strategies.
Evaluating LLM Performance Across Key Tasks
The study benchmarked leading LLMs, specifically OpenAI’s o3, Anthropic’s Claude-4, and Google’s Gemini-2.5, across several critical tasks:
- Analysis Verification: LLMs were tested on their ability to interpret and verify the legal analysis steps within each strategy. The evaluation distinguished between a ‘viable’ standard (a reasonable basis for the strategy) and a more stringent ‘correct’ standard (full support by authorities). While o3 and Claude-4 found all 193 analysis steps viable after iterative refinement, Gemini-2.5 struggled with some, indicating a lack of certain background legal knowledge.
- Goal Verification: This task assessed whether LLMs could determine if a tax strategy successfully met its stated goals, both with and without the expert-written legal analysis provided. Generally, models performed better when given the analysis, and more strategies were deemed ‘viable’ than ‘correct’.
- Adversarial Step Goal-Failure Verification: Here, the adversarial step was swapped into strategies to make them fail. Surprisingly, LLMs showed a strong bias towards finding strategies viable, even when designed to unambiguously fail. Domain experts had to iteratively refine the adversarial steps to ensure the models returned 0/36 as correct, highlighting the LLMs’ resistance to identifying non-working schemes.
- Step-Cloze Task: LLMs were asked to fill in a blanked-out step in partially specified strategies. Using a 0-to-3 grading rubric, models achieved ‘3’ (semantically identical) on slightly over half the steps. All models showed improvement from 0-shot to 1-shot prompting, but a decrease from 1-shot to 2-shot, suggesting a single example might be optimal for such tasks.
A Breakthrough: LLM Generates a Novel Tax Strategy
Perhaps the most significant finding came from the ‘From-Scratch Strategy Generation’ task, where LLMs were prompted to create entire tax strategies based only on authorities, background facts, and goals. While overall performance was mixed, o1-preview generated what appears to be an entirely novel, working tax-minimization strategy. This strategy leverages common trust funds and financial derivatives called straddles, exploiting common trust funds’ ability to set their accounting periods to circumvent restrictions on deducting tax losses from straddles (tax code § 1092). The strategy allows a taxpayer to recognize a loss while avoiding the corresponding gain, a mechanism previously unknown to domain experts. This discovery underscores the revolutionary potential of LLMs in helping tax agencies identify and shut down such schemes.
Challenges in LLM-Based Grading
The study also explored using LLMs to grade strategies generated by other LLMs. However, the agreement between LLM grades and human domain expert grades was poor, with Spearman’s ρ values significantly lower than for the step-cloze task. This suggests that grading complex, free-form legal strategies remains a challenging task for current LLMs.
Also Read:
- AI and Logic Unite for Trustworthy Tax Assistance
- Unlocking LLM Copyright: A Deep Dive into Fingerprinting for Model Protection
Conclusion
The research demonstrates that while LLMs exhibit mixed performance in understanding, evaluating, and generating tax-minimization strategies, they possess significant potential. Their responsiveness to different legal standards and the remarkable generation of a novel tax strategy highlight their future role in improving tax administration and combating tax abuse. The study also points to limitations, such as the small dataset size and the challenges in LLM-based grading of complex legal outputs, suggesting areas for future research and development.
