TLDR: Narrative Learning is a novel explainable machine learning methodology where AI models are entirely defined and iteratively refined using natural language prompts, rather than traditional numerical optimization. It employs two LLMs, an ‘Overseer’ for generating rules and an ‘Underling’ for applying them. Experiments show that Narrative Learning often outperforms traditional explainable models in accuracy, while maintaining the comprehensibility of its natural language explanations over time. This approach aims to make AI systems inherently more understandable and trustworthy.
In the rapidly evolving landscape of artificial intelligence, a new methodology called Narrative Learning is emerging as a significant advancement in explainable machine learning. This innovative approach, detailed in a recent research paper, proposes a future where AI models are not only powerful but also inherently understandable, defined entirely through natural language rather than complex mathematical equations.
The Challenge of Explainable AI
Traditionally, machine learning models fall into two categories: “explainable models” like decision trees, which offer some transparency, and “blackbox models” such as many neural networks, whose internal workings are opaque. While techniques like LIME and SHAP can provide insights into blackbox models, they often offer only approximations of how a model arrived at a decision. The demand for truly explainable AI is growing, driven by legal requirements in regions like the EU (GDPR) and China (PIPL), where understanding the rationale behind an AI’s decision can be more valuable than raw accuracy.
Narrative Learning addresses this by unifying the computational model and its human-language explanation. Instead of having separate systems for computation and explanation, this method uses a single, human-readable narrative that both humans and AI systems can understand and execute. This narrative becomes the core object that the machine learning process learns and refines.
How Narrative Learning Works
The Narrative Learning algorithm operates as a supervised binary classification system, employing two distinct large language models (LLMs):
- Overseer LLM: This model is responsible for generating classification instructions in natural language. It learns by observing training data and receiving feedback on its performance.
- Underling LLM: This model evaluates individual data points based on the instructions provided by the Overseer, returning classifications. Ideally, the Underling is a smaller, more cost-effective LLM, as it performs the bulk of the evaluation work.
The process is iterative. Initially, the Underlings might start with random classifications. The Overseer then receives performance metrics (like accuracy) and examples of correctly and incorrectly classified data points. Based on this feedback, the Overseer refines its natural language instructions, issuing a new prompt to the Underlings. This cycle continues until the model’s performance on validation data stops improving.
To enhance robustness and accuracy, individual narratives are combined into ensembles of three, with a majority vote determining the final classification. The system also has tuneable parameters, such as “patience” (how many rounds without improvement before stopping) and the number of examples shown to the Overseer each round.
Experiments and Results
A key challenge in evaluating Narrative Learning was preventing the LLMs from using prior knowledge about common datasets. To overcome this, researchers transformed well-known datasets (like the Titanic survival data, Wisconsin Breast Cancer data, and South German Credit data) into anonymized, re-scaled, and relabeled versions. They also created synthetic datasets with varying levels of noise, ensuring no pre-existing knowledge could influence the LLMs.
The experiments, which utilized a variety of frontier LLMs from vendors like Anthropic, OpenAI, and Google, demonstrated promising results. Narrative Learning ensembles consistently improved over time as new LLM models were released, eventually surpassing seven baseline explainable machine learning models on five out of six datasets. The paper highlights that on the transformed Titanic data, the improvement was statistically significant.
Maintaining Comprehensibility
A crucial concern for any explainable AI technique is whether the explanations remain understandable as the models become more sophisticated. The researchers addressed this by quantifying the lexical diversity of the generated narratives using Herdan’s Law. This analysis revealed no statistically significant increase in the complexity of the language used in the explanations over time, even as the models’ accuracy improved. This suggests that Narrative Learning is likely to maintain its comprehensibility, preventing explanations from becoming overly complex or difficult for humans to process.
Also Read:
- NARRABENCH: A New Framework to Assess AI’s Grasp of Stories
- Bridging AI Learning Methods: A New Approach to Scalable Task Adaptation
Future Directions
The paper concludes by outlining numerous avenues for future research, ranging from testing with newer LLMs and different datasets to exploring prompt engineering techniques, evaluating the impact of different Underling models, and investigating the stability and robustness of Narrative Learning. The authors also suggest that this methodology could be an ideal topic for undergraduate research projects due to its accessibility and the availability of code and data.
Narrative Learning represents a compelling step forward in making AI more transparent and trustworthy. By leveraging the power of large language models to define and refine models in natural language, it offers a path to AI systems that can explain themselves in a way humans can truly understand. You can read the full research paper here.
