TLDR: This research paper introduces a novel ‘rectification-based’ approach for distilling complex boosted tree models into simpler, more interpretable decision trees. The method incrementally corrects the decision tree by identifying misclassified instances, deriving abductive explanations from the boosted tree, and applying these as classification rules. This process offers logical guarantees that corrections are effective, making the decision tree progressively align with the boosted tree’s predictions. Experiments show that this approach outperforms traditional retraining methods, significantly improving the efficiency of generating explanations for AI decisions, despite a manageable increase in decision tree size.
In the rapidly evolving world of Artificial Intelligence, machine learning models have achieved impressive predictive performance across various applications. However, in critical domains like healthcare or legal systems, simply having accurate predictions isn’t enough. Users often need to understand why a system made a particular decision, receive explanations, and even correct errors. This demand has given rise to the field of eXplainable AI (XAI).
A core challenge in XAI is the trade-off between a model’s accuracy and its interpretability. Highly accurate models, such as boosted trees, are often complex and difficult to interpret. Conversely, models that are easy to understand, like decision trees, may not always be as accurate. The goal is to find a balance, creating models that are both precise and interpretable.
A new research paper, titled A Rectification-Based Approach for Distilling Boosted Trees into Decision Trees, introduces an innovative method to address this challenge. Authored by Gilles Audemard, Sylvie Coste-Marquis, Pierre Marquis, Mehdi Sabiri, and Nicolas Szczepanski, the paper proposes using a technique called ‘rectification’ to distill complex boosted trees into simpler, more interpretable decision trees.
What is Model Distillation?
Model distillation is a process where a ‘simpler’ target machine learning model is built from a more complex ‘source’ model. The aim is to transfer the knowledge from the complex model to the simpler one, often to reduce memory requirements, speed up predictions, or, crucially for XAI, to provide better explanations. In this context, the complex source model is a boosted tree, known for its high accuracy, and the simpler target model is a decision tree, valued for its inherent interpretability.
The Rectification Approach
The core idea of this new approach is an incremental distillation process. Instead of trying to convert the entire boosted tree into a decision tree at once, the system corrects the decision tree gradually. This happens whenever the decision tree misclassifies an instance compared to the more accurate boosted tree. When such a discrepancy occurs, the boosted tree acts as an ‘oracle’ to guide the correction.
Here’s how it works in simple terms:
- An instance (a piece of data) is encountered, and the decision tree classifies it differently than the boosted tree.
- An ‘abductive explanation’ is computed from the boosted tree for that specific instance. This explanation identifies the key conditions that led the boosted tree to its classification.
- This explanation is then converted into a simple classification rule.
- The current decision tree is ‘rectified’ (modified in a minimal way) by this new rule. This ensures that the decision tree now classifies that specific instance, and potentially others covered by the rule, in the same way as the boosted tree.
A significant advantage of this rectification-based approach is its ‘logical guarantees.’ It ensures that each correction step effectively brings the decision tree logically closer to the boosted tree, meaning the number of instances classified identically by both models increases with each step. This is a crucial difference from other distillation methods, such as retraining, where corrections might not always be effective or could even be undone by subsequent steps.
Also Read:
- Enhancing AI Predictions and Explanations with Association Rules
- Unlocking Complex Simulations: How AI Makes Expensive Models Faster and More Transparent
Empirical Evidence and Benefits
The researchers conducted experiments comparing their rectification method with a retraining approach. They found that rectification consistently improved the decision tree’s accuracy relative to the boosted tree, eventually leading to a complete distillation where the decision tree matched the boosted tree’s classifications on the test set. In contrast, retraining sometimes led to a decrease in accuracy, making complete correction impossible in some cases.
While the rectification process can lead to an increase in the size and depth of the decision tree, the computational benefits for XAI queries are substantial. For instance, computing ‘sufficient reasons’ (a type of explanation) for instances was significantly faster when using the distilled decision tree compared to the original boosted tree. The rectification method also drastically reduced the number of timeouts encountered during explanation computations, making it a much more practical solution for generating explanations rapidly.
In essence, this rectification-based distillation offers a practical way to create interpretable decision trees from complex boosted trees, providing a valuable tool for applications where both accuracy and clear explanations are paramount. The lazy and opportunistic nature of the approach allows for controlled growth of the decision tree and ensures effective, guaranteed corrections.
