TLDR: This research explores whether Large Language Models (LLMs) can generate high-quality explanations of business processes from significantly reduced event log data. By progressively decreasing the input size for process model discovery, the study found that explanation quality improves non-linearly, with a “sweet spot” where substantial data reduction still yields good explanations. This suggests a practical cost-quality trade-off, enabling more efficient LLM-assisted process analysis in resource-constrained settings.
In the world of business operations, understanding how processes flow is crucial for efficiency and improvement. This is where process mining comes in, a field that extracts valuable insights from event logs – digital footprints left by every activity in a process. Traditionally, process mining tools have matured significantly, but a new frontier has emerged with the advent of Large Language Models (LLMs).
LLMs are increasingly being used to generate textual explanations of these complex process models. Imagine an AI that can look at the intricate steps of a manufacturing line or a customer service workflow and then describe it in plain language, identify bottlenecks, and even suggest improvements. This capability is incredibly powerful, but it comes with a challenge: the sheer volume of data. Process models are often derived from massive event logs, leading to large and structurally complex “behavioral abstractions” (like directly-follows graphs or Petri nets) that can be computationally expensive for LLMs to process.
A recent study by P. van Oerle, R.H. Bemthuis, and F.A. Bukhsh from the University of Twente explores a novel approach to tackle this computational hurdle: progressive behavioral-input reduction. Their research investigates whether high-quality explanations can still be generated by LLMs even when the input data used to build the process models is significantly reduced. This is particularly relevant for scenarios where computational resources are limited or real-time analysis is needed.
The Research Approach
The researchers developed a four-stage pipeline to evaluate this concept. First, they took a complete event log and generated progressively smaller “sub-logs” by taking only the first ‘k’ events. From each of these sub-logs, a process model was discovered. This means they created multiple process models, each based on a different amount of behavioral information, ranging from very few events to almost the full log.
Next, a first LLM (LLM1) was prompted to generate a textual explanation for each discovered model. These explanations were designed to cover three key aspects: a description of the process, identification of any bottlenecks, and suggestions for improvements. Finally, a second LLM (LLM2) was used to evaluate the quality of these explanations. LLM2 scored each explanation against the full process model (derived from the complete log) on a scale of 1 to 10 across completeness, bottleneck identification, and improvement suggestions, then averaged these scores.
Key Findings: A Sweet Spot for Efficiency
The study, conducted using synthetic job-shop scheduling logs, revealed some compelling insights. Unsurprisingly, very small inputs (10-20 events) resulted in low-scoring, incomplete explanations. However, as the input size increased, explanation quality saw substantial gains. A “practical knee” was observed between 100 and 1,000 events, where average scores significantly improved. Beyond 1,000 events, gains became more modest, suggesting diminishing returns.
Specifically, around 100,000 events, the quality stabilized near an average score of 7.92, with higher consistency (narrower error bars). Interestingly, using the full log provided only a marginal additional increase to about 7.95. This suggests that a significant portion of the behavioral information can be omitted without a drastic loss in explanation quality. For scenarios prioritizing latency or budget, around 1,000 events might offer a stable mid-range quality, while 100,000 events could be justified for higher-stakes settings requiring maximum consistency.
These findings indicate that meaningful LLM-based process explanations can indeed be derived from significantly reduced behavioral input. This opens up possibilities for more scalable and resource-efficient LLM-assisted process intelligence, especially in environments with limited computational power or where quick insights are paramount. The research paper can be found here: Evaluating LLM-Based Process Explanations under Progressive Behavioral-Input Reduction.
Also Read:
- AI’s Role in Software Teams: Boosting Collaboration and Productivity
- A New Framework for AI-Powered Customer Support Efficiency
Implications and Future Directions
While the study used LLMs to grade other LLMs and relied on synthetic data, the results offer a promising path forward. The non-linear improvement in explanation quality with input size highlights a crucial cost-quality trade-off. Future work will involve validating these findings on real-world logs, exploring different process discovery algorithms and model notations, and incorporating human evaluations to calibrate LLM scores. This research paves the way for making advanced process analysis more accessible and efficient.
