TLDR: DeCE is a new evaluation framework for large language models (LLMs) in high-stakes fields like law. It moves beyond single scores by breaking down evaluation into precision (factual accuracy) and recall (coverage of required information), using automatically extracted criteria. DeCE shows much stronger alignment with human expert judgments than traditional methods and reveals specific trade-offs and weaknesses in different LLMs across various legal tasks and jurisdictions.
Evaluating the performance of large language models (LLMs) is a critical challenge, especially when these powerful AI systems are deployed in high-stakes fields like law, medicine, or finance. In these domains, a simple ‘good’ or ‘bad’ score isn’t enough; the nuances of factual accuracy, relevance, and comprehensive coverage are paramount. Traditional evaluation methods often fall short, either by failing to capture semantic correctness or by reducing complex answer quality into a single, undifferentiated score.
A new research paper titled “Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses” introduces an innovative framework called DeCE (Decomposed Criteria-Based Evaluation). Developed by Fangyi Yu, Nabeel Seedat, Dasha Herrmannova, Frank Schilder, and Jonathan Richard Schwarz from Thomson Reuters Foundational Research and Labs, DeCE offers a more interpretable and actionable way to assess LLM outputs.
Understanding DeCE’s Approach
DeCE tackles the evaluation problem by breaking it down into two distinct, measurable dimensions: precision and recall. Precision focuses on the factual accuracy and relevance of the claims made by the LLM. It asks: Is what the model says correct and pertinent? Recall, on the other hand, measures the completeness of the model’s answer, specifically its coverage of all required concepts and information. It asks: Did the model include everything it was supposed to?
What makes DeCE particularly powerful is its ability to automatically extract instance-specific evaluation criteria directly from ‘gold standard’ answers – the ideal responses crafted by human experts. This means DeCE doesn’t rely on predefined taxonomies or handcrafted rubrics, making it highly adaptable across different domains and specific questions. It’s also ‘model-agnostic,’ meaning it can be used to evaluate any LLM.
The DeCE framework operates through two main workflows:
-
Precision Scoring: First, the model’s answer is broken down into individual factual elements. Each of these elements is then verified against the gold standard answer to confirm its factual correctness and relevance.
-
Recall Scoring: Here, evaluation criteria are extracted from the ‘Required Information’ section of the gold standard answer. The LLM’s response is then checked to see if it satisfies each of these essential criteria, quantifying how fully it covers the necessary concepts.
Real-World Validation in Legal QA
To test DeCE, the researchers applied it to a real-world legal question-answering task. This involved a dataset of 224 legal QA pairs, curated by expert attorneys, spanning various U.S. jurisdictions. The gold standard answers in this dataset were meticulously structured, distinguishing between ‘Required Information’ (essential for completeness) and ‘Helpful Information’ (supportive material).
The study evaluated five diverse LLMs: general-purpose models like GPT-4o and Gemini-2.5-Pro, reasoning models such as DeepSeek-R1, the open-source Llama-3.1-405B, and a domain-specific model, Legal Llama-3.1-70B, which was fine-tuned for legal contexts.
Key Findings and Insights
The results were compelling. DeCE demonstrated a substantially stronger correlation with human expert judgments (a Pearson correlation of r=0.78) compared to traditional metrics like ROUGE (r=0.12), pointwise LLM scoring (r=0.35), and even modern multidimensional evaluators (r=0.48). This indicates that DeCE’s instance-specific, domain-aware criteria and explicit precision-recall decomposition effectively capture the nuanced signals that human experts prioritize.
Beyond just a higher correlation, DeCE provided valuable diagnostic insights into LLM behavior:
-
Precision-Recall Trade-offs: The evaluation revealed distinct trade-offs among the LLMs. Larger, generalist models like Gemini-2.5-Pro and DeepSeek-R1 often excelled in recall (providing comprehensive answers) but sometimes at the cost of precision (factual accuracy). Conversely, the specialized Legal Llama-3.1-70B showed higher precision but lower completeness. This highlights that different models prioritize different aspects of answer quality.
-
Reliability of LLM-Generated Criteria: A significant finding was the high reliability of the criteria automatically generated by the LLM judge. Only 11.95% of these criteria required expert revision, underscoring DeCE’s scalability and potential to drastically reduce the manual effort typically involved in creating evaluation rubrics.
-
Jurisdictional and Query-Type Performance: DeCE helped identify specific strengths and weaknesses across different U.S. jurisdictions and types of legal questions. For instance, models performed exceptionally well on Ohio-State queries, while New York-State queries often suffered from low precision due to unsupported or outdated citations. Complex legal reasoning problems and source-specific requests consistently challenged all models, indicating areas for targeted improvement.
Also Read:
- Steering AI Agents Towards Expert-Level Behavior
- CultureScope: A Deeper Look into AI’s Cultural Competence
Implications for AI Development
The convergence of failure modes across diverse models for certain jurisdictions and query types suggests systemic limitations in current LLMs for legal QA. This implies that simply scaling up models might not be enough; targeted corpus expansion and human-in-the-loop routing for challenging cases could yield greater benefits.
In conclusion, DeCE offers an interpretable and actionable framework for evaluating LLM responses in expert domains. By providing decomposed scores for precision and recall, it moves beyond opaque single scores to offer fine-grained insights that can guide targeted improvements in AI systems. For more details, you can read the full research paper here.
