TLDR: Meta’s Code Quality Score (CQS) system leverages fine-tuned Llama models to automate code quality issue detection and provide actionable review critiques in large software systems. This AI-powered solution has achieved high precision and a 60% user helpfulness rate in an industrial setting, aiming to enhance developer productivity and codebase health by reducing manual review efforts.
Maintaining high code quality in vast software systems, especially where numerous engineers collaborate, presents a significant challenge. Traditional methods, often relying on manual peer code reviews and basic static analysis tools, struggle to evaluate nuanced aspects like code modularity, readability, and error handling.
Introducing the Code Quality Score (CQS) System
Meta Platforms has introduced the Code Quality Score (CQS) system, an innovative AI-powered solution designed to automatically detect code quality issues and provide actionable insights. At its core, the CQS system is powered by two Llama3 models, which have been fine-tuned using advanced techniques like Supervised Fine-tuning (SFT) and offline Reinforcement Learning (RL) approaches, specifically Direct Preference Optimization (DPO).
The CQS system serves two primary functions: first, it identifies common code quality issues based on best practices; second, it provides constructive critiques for LLM-generated code reviews. To ensure a positive user experience and filter out incorrect or hallucinated responses, the system incorporates a layer of hand-crafted rules.
How CQS Works: A Multi-Agent Approach
The CQS system operates with a “multi-agent” architecture, assigning distinct, specialized roles to different components. It comprises two main agents:
- The Issue Collector Agent: This agent, based on a Llama 3.1-70b model, scans proposed code changes (in a “diff” format) and generates a list of potential quality issues. These issues can range from code readability and complexity to robustness and modularity. The model is prompted to output a summary including an issue tag, function name, rationale, file path, and line number.
- The Issue Validator Agent: The issues identified by the collector are then passed to this agent, which uses another fine-tuned Llama 3.1-70b model. This validator critically assesses the validity and relevance of each identified issue, assigning a score from 0 to 10. A post-processing step then filters out issues based on this score and additional rule-based filters before the code review is presented to developers.
Training the AI Models
Both the issue collector and validator LLMs are fine-tuned using the Llama 3.1 70b Instruct model as their base. The issue collector undergoes a two-phase training process: an initial SFT phase using a large dataset of human-reviewed code changes from Meta’s internal codebase, followed by an offline RL phase using DPO. The DPO training leverages an LLM-judge model to create preference pairs, where the judge scores different model-generated code reviews.
The LLM-judge model itself is also a fine-tuned Llama 3.1-70b Instruct model. It was trained using developer feedback, where engineers provided textual critiques and thumbs-up/down signals on model-generated code reviews. This approach allowed the system to distill human preferences into the judge model, making it effective in evaluating and grading LLM-generated responses.
Also Read:
- Automating Code Lint Error Resolution with LLMs at Scale
- ToolTrain: Enhancing AI’s Code Issue Localization Through Integrated Learning
Real-World Impact and Future Directions
The CQS system has been successfully rolled out to over 5,000 engineers at Meta, demonstrating its effectiveness in a real-world industrial setting. It has consistently achieved an impressive 60% week-over-week user helpfulness rate, indicating strong developer acceptance and utility. Offline evaluations show that the CQS system achieves a high precision rate for identifying valid issues, with the multi-stage prompt optimization and the issue validator agent significantly boosting precision to 78.20%.
While Llama models currently lag behind state-of-the-art closed-source models like Claude in raw issue collection tasks, the strategic fine-tuning with developer feedback and the multi-agent system design have enabled CQS to achieve superior overall precision. The developers behind CQS hope that such systems will alleviate the burden on senior engineers for manual reviews and serve as a valuable learning resource for junior developers.
Future work for the CQS system includes leveraging continuous online developer feedback to create a data flywheel for ongoing model improvement. Additionally, there are plans to expand the system’s capabilities beyond general code quality to identify more complex issues, such as security vulnerabilities and performance bottlenecks. For more technical details, you can refer to the full research paper: A Note on Code Quality Score: LLMs for Maintainable Large Codebases.
