TLDR: StepFly is a novel AI-powered framework designed to automate the execution of troubleshooting guides (TSGs) for IT incident management. It addresses key challenges such as TSG quality issues, complex control flows, and data-intensive queries. The framework features a three-stage workflow: a TSG Mentor tool for quality improvement, offline preprocessing to extract structured execution DAGs and Query Preparation Plugins (QPPs), and an online DAG-guided scheduler-executor system with a memory for efficient and parallel execution. Evaluations show StepFly achieves a high success rate (around 94% with GPT-4.1) with reduced time and token consumption, and can significantly cut execution time (32.9% to 70.4%) for parallelizable TSGs.
In the fast-paced world of large-scale IT systems, keeping services running smoothly is paramount. When things go wrong, a process called incident management kicks in, relying heavily on detailed troubleshooting guides (TSGs). These guides are essentially step-by-step instructions that Site Reliability Engineers (SREs) follow to diagnose and fix issues. However, manually going through these guides can be slow, prone to human error, and ultimately delay getting services back online.
While artificial intelligence, particularly large language models (LLMs), has shown great promise in automating many IT tasks, existing solutions haven’t fully tackled the unique challenges of TSGs. These challenges include dealing with poorly written or unclear guides, understanding complex decision paths, handling large amounts of data for queries, and taking advantage of steps that can be done at the same time.
To address these issues, a team of researchers from Tsinghua University and Microsoft conducted an in-depth study of 92 real-world TSGs. Their findings led to the development of StepFly, an innovative, end-to-end AI agent framework designed to automate troubleshooting guide execution. You can read the full research paper here: Agentic Troubleshooting Guide Automation for Incident Management.
Understanding StepFly’s Approach
StepFly operates through a clever three-stage workflow, making the automation process more reliable and efficient:
1. Improving TSG Quality: The first step acknowledges that not all TSGs are perfectly written. The researchers found that many guides suffer from issues like unclear instructions, missing information, or poorly defined decision points. StepFly introduces a tool called TSG Mentor, which acts like a helpful assistant for SREs. It analyzes TSGs, identifies common quality problems, and provides feedback to improve their clarity and structure. This ensures the guides are easier for both humans and AI to understand and follow.
2. Offline Preparation: Before an incident even occurs, StepFly gets to work behind the scenes. It uses LLMs to process the unstructured text of TSGs and extract a structured ‘Execution DAG’ (Directed Acyclic Graph). Think of this DAG as a precise map of the troubleshooting process, showing each step and how they connect, including conditional branches. This map guides the AI agent during execution, preventing it from skipping steps or going off track. Additionally, StepFly creates ‘Query Preparation Plugins’ (QPPs). Many TSGs involve running complex database queries. Instead of having the LLM generate these queries from scratch every time (which can be error-prone and slow), QPPs encapsulate the query templates, making query generation faster and more accurate.
3. Online Execution: When an incident strikes, StepFly’s online execution stage takes over. It features a ‘DAG-guided scheduler-executor’ framework. The scheduler uses the pre-built DAG to orchestrate the execution of steps, assigning them to individual ‘executors’. Each executor is a specialized agent responsible for carrying out a single TSG step. A crucial ‘memory system’ is also in place, allowing the agents to store and manage structured data from various tools and plugins. This prevents large data outputs from cluttering the agent’s conversation history and makes data-intensive tasks much more efficient. For example, if one step retrieves time-series data, the memory system stores it in a structured format, allowing a subsequent step to easily access and analyze it without re-parsing raw text.
The Power of Parallelization
One of StepFly’s standout features is its ability to support parallel execution of independent TSG steps. The empirical study revealed that many troubleshooting steps don’t strictly need to be done one after another. Just like a team of SREs might divide and conquer different parts of a problem simultaneously, StepFly’s scheduler can assign multiple executors to work on independent tasks concurrently. This significantly reduces the overall time needed to diagnose and resolve an incident. For instance, in some cases, parallelization led to a remarkable reduction in execution time, ranging from 32.9% to 70.4% for parallelizable TSGs.
Also Read:
- Bridging JIRA and GitHub: AI-Powered System Accelerates Software Issue Resolution
- Unlocking Enterprise Knowledge: A Hybrid Approach to Scalable and Explainable Information Retrieval
Impressive Results
The evaluation of StepFly on real-world TSGs and incidents demonstrated its effectiveness. When powered by advanced LLMs like GPT-4.1, StepFly achieved an impressive success rate of approximately 94%, significantly outperforming previous baseline methods. It also proved to be more efficient, consuming less time and fewer computational tokens. Even with a more modest LLM like GPT-4.1-mini, StepFly maintained a high success rate of around 84%.
StepFly represents a significant leap forward in automating incident management. By focusing on improving TSG quality, structuring workflows, optimizing query generation, and enabling parallel execution, it offers a robust and efficient solution for minimizing service disruptions and ensuring smoother IT operations.
