TLDR: The FETCH classifier is a new hybrid AI system that combines large language models (LLMs), keyword matching, and traditional machine learning to accurately classify legal problems. Developed for civil legal intake and referral, it achieved 97.37% accuracy, outperforming the GPT-5 model, while being significantly more cost-effective. FETCH also generates automatic follow-up questions to gather more information, streamlining the process of connecting individuals to the right legal help.
Navigating the legal system can be a daunting task, especially for those seeking help from legal aid programs or lawyer referral services. The initial step of identifying the correct legal problem is critical, as misdirection can lead to severe consequences, from missed deadlines to loss of housing or child custody. A new research paper introduces an innovative solution to this challenge: the FETCH classifier. [RESEARCH_PAPER_URL]This system aims to streamline the process of matching individuals with the appropriate legal assistance, significantly improving accuracy and reducing costs.
The paper, titled “That’s So FETCH: Fashioning Ensemble Techniques for LLM Classification in Civil Legal Intake and Referral,” was authored by Quinten Steenhuis from Suffolk University Law School. It delves into the development and evaluation of the FETCH classifier, which leverages a hybrid approach combining large language models (LLMs) with traditional machine learning (ML) and keyword matching techniques.
Addressing a Critical Need
Millions of people annually seek legal help, often through hotlines or referral services. The first point of contact requires accurately identifying their legal issue. Errors in this initial assessment can prevent applicants from receiving timely and appropriate help, leading to potentially devastating outcomes. The FETCH classifier was designed to tackle this problem head-on, offering a more reliable and efficient method for legal problem classification.
The FETCH Classifier: A Hybrid Approach
FETCH stands for Fast Ensemble Tagging and Classification Helper. It is a multi-label weighted ensemble voting classifier that integrates several components: LLMs, keyword matching, and traditional ML models. The system employs a novel dataset of 419 real-world queries submitted to a nonprofit lawyer referral service, allowing for robust evaluation against practical scenarios.
The core of FETCH involves prompting three inexpensive LLM models—GPT-5-nano, gemini-2.5-flash, and mistral-small—alongside a simple keyword matching system and the traditional ML Spot API. These models are instructed to classify the applicant’s problem into one or more of 244 taxonomy nodes, determine if more information is needed, or identify if the query does not describe a legal problem at all. The results from these diverse models are then combined using a weighted voting system, with weights based on past performance, to produce a highly accurate classification.
Exceeding State-of-the-Art Performance
One of the most striking findings of the research is FETCH’s classification accuracy. The ensemble method achieved a hits@2 accuracy of 97.37%, meaning that in nearly all cases, the correct legal category was among the top two suggestions. This performance not only significantly surpasses older machine learning and keyword-matching approaches but also slightly exceeds the current state-of-the-art GPT-5 model, which achieved 96.66% accuracy.
Beyond accuracy, FETCH offers substantial cost and speed advantages. The ensemble method operates at approximately one-third the cost of using the GPT-5 model alone, with significantly lower latency. This makes it a highly practical and cost-effective solution for legal aid programs and bar referral services that handle a high volume of inquiries.
Automatic Follow-Up Questions
A unique feature of the FETCH classifier is its ability to automatically generate follow-up questions when it lacks sufficient confidence in a classification. Each LLM in the ensemble proposes three candidate questions, which are then semantically merged to present the applicant with the top three clarifying questions. This mechanism allows the system to gather more precise information without requiring immediate human intervention, further streamlining the intake process.
The Oregon State Bar has already integrated the FETCH classifier into its online referral system. Here, follow-up questions are presented in a familiar form metaphor, using radio buttons and checkboxes. This design choice aims to reduce user effort and mitigate the “uncanny valley” effect often associated with direct interactions with LLMs.
Insights from Error Analysis
The research also included a detailed error analysis of the ensemble classification method. Out of 11 errors, four were identified as probable human annotation errors in the dataset itself, suggesting that the LLM ensemble’s performance might be comparable to, or even highlight inconsistencies in, human classification. The remaining LLM errors often involved classifying a problem as “general litigation” instead of a more specific category, indicating areas for potential refinement. Importantly, the errors did not show a bias towards any particular type of legal problem.
Also Read:
- Optimizing Large Language Model Routing for Cost and Performance
- Boosting Classroom LLMs: A Comparative Look at AI Retrieval for Accurate Learning
Future Directions
While the FETCH classifier demonstrates strong results, the authors acknowledge areas for future work. This includes more extensive testing of cases where applicants may not have a legal problem (e.g., due to delusions), quantifying the real-world benefits to applicants from faster and more accurate referrals, and further exploring methods to route complex cases to best-of-class LLMs based on confidence levels. The study also highlights the ongoing need to establish human baseline performance for comparison.
The FETCH classifier represents a significant step forward in applying AI to enhance access to justice. By combining the strengths of various models, it offers a powerful, cost-effective, and accurate tool for legal problem classification, ultimately helping more people connect with the legal help they need.
