TLDR: CodeSight is a novel system that combines process mining and machine learning to analyze and predict outcomes in software development workflows. By extracting data from GitHub, transforming it into event logs, and applying an LSTM model, CodeSight can accurately predict Pull Request (PR) resolution times and deadline compliance. This enables early identification of potential delays and provides actionable insights through interactive dashboards, enhancing operational efficiency and fostering data-driven development.
In the fast-paced world of software development, ensuring projects meet their deadlines and maintaining efficient workflows can be a significant challenge. As software systems and teams grow, the complexity of development pipelines, from initial code contributions to final deployment, often makes it difficult to gain clear visibility into the entire process. While high-level metrics like those from the DORA framework offer a general overview, they often fall short in revealing the detailed process dynamics that lead to inefficiencies or bottlenecks.
Addressing this critical gap, a new system called CodeSight has been introduced. This end-to-end platform is designed to anticipate whether software development workflows will meet their deadlines. It achieves this by integrating process mining with advanced machine learning techniques, providing a comprehensive view of software delivery performance.
CodeSight operates by capturing development and deployment data directly from GitHub. This raw data, which includes information about pull requests (PRs), commits, and workflow runs, is then transformed into structured process mining logs. These logs are crucial for detailed analysis, allowing the system to reconstruct actual process flows and identify the timing between various activities. From these logs, CodeSight generates a variety of metrics and interactive dashboards that offer actionable insights into PR activity patterns and overall workflow efficiency.
Beyond just descriptive analysis, CodeSight incorporates a predictive layer. It uses a Long Short-Term Memory (LSTM) model, a type of neural network particularly effective at understanding temporal dependencies in sequential data. This model predicts the remaining resolution times for PRs based on their sequential activity traces and other static features. This predictive capability is vital for early identification of potential deadline breaches, enabling teams to take proactive measures.
The architecture of CodeSight is divided into four main components: a Data Acquisition Layer that retrieves information from the GitHub REST API; a Data Transformation Layer that converts this raw data into standardized event logs; a Process Mining and Visualization Layer that performs process mining and displays results through interactive dashboards; and a Predictive Layer that implements the LSTM-based model.
The system’s workflow is a continuous loop: raw GitHub data is collected, normalized, and transformed into event logs. These logs are then analyzed using process mining to generate metrics and feed into dashboards. Crucially, they also serve as training input for the LSTM predictor. This pipeline supports both descriptive analysis and predictive forecasting, creating a feedback loop for continuous improvement in software development workflows.
For its internal tests, CodeSight utilized data from Inverbis Analytics’ own process mining platform development repositories on GitHub, specifically frontend and backend projects. This data was structured into detailed datasets covering commit histories, pull request information, and workflow execution details, allowing for a thorough reconstruction of development activities.
The process mining capabilities of CodeSight enable the reconstruction of workflow structures and the computation of operational metrics such as activity and transition durations, waiting and review times, and the identification of bottlenecks and rework patterns. The system’s dashboards, powered by Power BI, offer a multi-faceted view of development processes, including DORA metrics, general development indicators, pull request activity, process variants, user-based analysis, temporal evolution of PRs, and deployment and incident metrics.
The LSTM model for deadline compliance prediction demonstrated strong performance. In tests, the model achieved high precision and F1 scores in predicting deadline compliance, with an accuracy of 0.944 and an F1-score of 0.963 on the test set. This indicates a significant ability to anticipate whether a PR will meet its deadline even from incomplete traces. The mean absolute error (MAE) for remaining time prediction was approximately 8.8 hours on the test set, which is considered acceptable for practical deadline forecasting.
Also Read:
- Advanced Code Clone Detection Using Multiple Code Representations
- Gistify: A New Challenge for AI Code Understanding
CodeSight represents a significant step towards process intelligence for DevOps. By unifying event data, applying process mining, and augmenting insights with machine learning, it empowers organizations to continuously monitor, understand, and predict the behavior of their development pipelines through data-driven insights. Future work aims to expand data integration to other repository services like GitLab and Bitbucket, automate the mapping of events into standardized process logs, and incorporate explainability mechanisms into the predictive models to facilitate broader industrial adoption. For more details, you can refer to the original research paper.
