TLDR: W ALL is a web application that automates software quality assurance by integrating SonarQube for issue detection with large language models (LLMs) like GPT-3.5 Turbo and GPT-4o for automated code revisions. It features modules for issue extraction, code revision, and comparison, demonstrating significant reductions in human effort and improved revision rates through a hybrid LLM approach. The system aims to streamline software development by detecting bugs, code smells, and vulnerabilities, then generating and evaluating code fixes automatically, with future plans for full automation and integration of open-source LLMs.
In the rapidly evolving landscape of software development, projects are becoming increasingly complex, leading to a surge in the volume and variety of code-related issues. Manually identifying, resolving, and evaluating these issues is a time-consuming and error-prone process, even for dedicated quality assurance teams. To address this critical challenge, researchers have introduced W ALL, a novel web application designed to automate software quality assurance using the power of large language models (LLMs).
W ALL, which stands for WebApplication usingLargeLanguage Models, integrates established static code analysis tools like SonarQube with advanced LLMs such as GPT-3.5 Turbo and GPT-4o. This integration creates a seamless pipeline for detecting software issues, generating automated code revisions, and evaluating the accuracy of these revisions, significantly reducing the human effort traditionally required.
The Core Modules of W ALL
The application is structured around three primary modules, each playing a crucial role in the automated quality assurance workflow:
1. Issue Extraction Tool: This module leverages SonarQube, a widely recognized static code analysis platform, to identify various software issues. SonarQube is chosen for its robust support across numerous programming languages, making it ideal for large, multi-language projects. It effectively detects bugs (unintentional errors), code smells (potential design flaws), and vulnerabilities (security weaknesses). Users connect their projects to SonarQube, and W ALL extracts all necessary issue details—type, line number, file location, file name, and description—into a CSV file, which then serves as input for the next stage.
2. Code Issues Reviser: This is where the large language models come into play. Based on the extracted issues, W ALL employs LLMs to generate automated code revisions. The application offers two modes: a “Processing All Files” mode, which revises all identified files using pre-configured prompts and LLMs, and a “One-by-One File Processing” mode, allowing users to manually select and view individual files. In the one-by-one mode, users can even customize the prompt to tailor the LLM’s revision process. The system strategically uses GPT-3.5 Turbo for cost-effective revisions of most issues, reserving the more advanced GPT-4o for complex problems requiring sophisticated reasoning.
3. Code Comparison Tool: The final module facilitates the evaluation of the LLM-generated revisions. It provides a side-by-side comparison of the original and revised code files, visually highlighting differences. Lines removed from the original are marked in yellow, while newly added lines in the revised version appear in green. This visual aid helps users quickly identify changes and assess revision quality. Furthermore, the tool calculates evaluation metrics like precision, recall, and F1-score to quantitatively measure the accuracy of the revisions, based on line-level changes.
Also Read:
- AI Agents Fortify LLM-Generated Firmware for Embedded Systems
- Assessing the Energy Footprint of AI-Generated Code: Human Expertise Still Leads the Way
Experimental Validation and Future Outlook
W ALL’s effectiveness was rigorously tested on a substantial dataset comprising 563 files with over 7,599 issues. The experiments demonstrated impressive revision rates: GPT-3.5 Turbo alone achieved a 71.6% revision rate, while GPT-4o boosted this to 85.5%. A key finding was that a hybrid approach—first using GPT-3.5 Turbo for initial revisions and then GPT-4o for remaining complex issues—could reduce overall costs by up to 40% while maintaining high performance. This makes W ALL a cost-efficient solution for large-scale projects.
While W ALL is currently in its initial phase, future enhancements are planned. These include integrating open-source LLMs like the LLAMA models for even more cost-effective solutions, adopting Retrieval-Augmented Generation (RAG) techniques to broaden capabilities, and exploring fine-tuning to improve revision accuracy. A major objective is to achieve a fully automated workflow, eliminating human intervention entirely for a seamless pipeline that continuously improves software quality.
W ALL represents a significant step forward in automating software quality assurance, offering a powerful solution to manage the increasing complexity of modern software projects. By combining static analysis with the intelligence of large language models, it promises to enhance productivity and code reliability across diverse programming environments. You can learn more about this research in the full paper available here.
