TLDR: The Agentic Classification Tree (ACT) is a novel AI framework that merges the semantic reasoning capabilities of large language models (LLMs) with the transparent, rule-based structure of decision trees. It enables AI systems to make decisions on unstructured data, like text, by formulating each decision point as a natural-language question, which is iteratively refined by an LLM. This approach ensures competitive accuracy while providing clear, auditable, and human-understandable decision paths, addressing the critical need for transparency in high-stakes AI applications.
In today’s world, as artificial intelligence (AI) systems become more involved in critical areas like healthcare, finance, and legal decisions, there’s a growing demand for these systems to be transparent, understandable, and auditable. This is not just a good practice but is increasingly becoming a regulatory requirement. While traditional decision trees offer clear, verifiable rules, they are limited to structured data, meaning they can’t directly handle complex information like text or images.
On the other hand, large language models (LLMs) are excellent at processing unstructured data. However, their decision-making process often relies on free-form reasoning, which can be difficult to trace, verify, or audit. This makes it challenging to trust LLMs in high-stakes environments where accountability is paramount.
Introducing the Agentic Classification Tree (ACT)
A new research paper introduces the Agentic Classification Tree (ACT), a novel framework that aims to bridge this gap. Developed by Vincent Grari, Tim Arni, Thibault Laugel, Sylvain Lamprier, James Zou, and Marcin Detyniecki, ACT extends the well-understood methodology of decision trees to unstructured inputs. It does this by formulating each decision point, or ‘split,’ in the tree as a natural-language question. These questions are then refined through a process that evaluates how well they separate different categories of data, using feedback from an LLM.
Imagine a doctor diagnosing a patient. Instead of following a rigid flowchart, the doctor asks a series of questions, each tailored to the patient’s symptoms, to narrow down the possibilities. ACT works similarly, but with an AI. For instance, in a tuberculosis diagnosis scenario, an ACT might ask, “Does this example involve coughing up blood or weight loss?” or “Does the example show evidence of severe pain and either shortness of breath or swelling in unusual locations?” The LLM answers these questions, guiding the decision down the tree until a final classification is reached.
How ACT Works
At its core, ACT combines the best of both worlds: the semantic understanding of LLMs and the transparent, rule-based structure of decision trees. Instead of relying on predefined numerical or categorical features, ACT uses an LLM to generate and refine natural-language questions at each node of the tree. This process is iterative: an initial generic question is posed, and then, based on how well it separates the data and where errors occur, the LLM refines the question to be more precise and effective. This refinement is guided by both quantitative measures (like Gini impurity, which assesses how mixed the data is at a given point) and qualitative semantic feedback from the LLM itself.
This approach ensures that the decision path is not only effective but also human-understandable. Each step in the tree is a clear, binary question that can be inspected and verified, making the entire decision process transparent and auditable. This is a significant improvement over ‘black-box’ LLM methods where the reasoning is often opaque.
Key Benefits and Performance
The benefits of ACT are manifold:
- Transparent and Structured Decisions: ACT provides fully traceable and interpretable decision paths, making it easier for humans to oversee and intervene.
- Optimized Decision Process: By breaking down complex tasks into a series of sub-questions, ACT can improve LLM performance without needing to retrain the model. It automatically finds the most relevant questions at each step.
Experiments conducted on various text classification tasks, including medical diagnosis, spam detection, and jailbreak prompt classification, show that ACT performs competitively, often matching or surpassing existing LLM-based methods like Chain-of-Thought, DSPy, and TextGrad. Crucially, it also outperforms traditional decision trees that rely on basic text features, demonstrating its ability to handle the nuances of natural language.
For example, on the medical diagnosis dataset, the best ACT configuration improved test accuracy by several percentage points over other advanced LLM baselines, all while providing clear, question-based decision paths. The questions generated by ACT for tuberculosis diagnosis were also found to align strongly with symptoms identified by medical experts, further validating its utility in high-stakes scenarios.
Also Read:
- Chain-in-Tree: Streamlining LLM Reasoning with Adaptive Branching
- Enhancing Medical AI with Logic-Driven Multi-Agent Systems
The Future of Interpretable AI
The Agentic Classification Tree represents a promising step towards building more trustworthy and understandable AI systems, especially for complex unstructured data. By combining the powerful reasoning of LLMs with the inherent transparency of decision trees, ACT offers a new avenue for interpretable and effective decision-making. Future work aims to extend ACT to more complex tasks like multi-class classification and regression, and to explore its application beyond text to other data types. You can read the full research paper here.
