TLDR: Meta developed MetaMateCR, an AI-assisted system, to provide fixes for code review comments at scale. By fine-tuning Llama models on a large internal dataset and conducting rigorous safety trials, they achieved significant improvements in patch acceptance rates (LargeLSFT outperforming GPT-4o by 9.2 percentage points in production) without negatively impacting review times, by strategically showing AI suggestions only to code authors.
At Meta, the sheer volume of code review comments—tens of thousands each week—presented a significant challenge. To address this, the company developed MetaMate for Code Review (MetaMateCR), an innovative AI-assisted system designed to provide automated fixes for these comments at an unprecedented scale.
The development of MetaMateCR involved a meticulous process. Researchers first created an extensive internal benchmark comprising 64,000 pairs of review comments and their corresponding code patches. This large dataset was then used to fine-tune Llama models, tailoring them specifically for Meta’s unique codebase and development practices. A crucial aspect of their methodology involved rigorous randomized controlled safety trials and full production experiments. These trials were essential to ensure that the AI-assisted fixes did not inadvertently slow down the code review process, a common concern when integrating new AI tools into existing workflows.
Performance and Model Comparison
In offline evaluations, MetaMateCR’s LargeLSFT model demonstrated impressive performance, achieving an exact match patch rate of 68%. This significantly outperformed GPT-4o, a leading public model, by a notable 9 percentage points. A key advantage of Meta’s internal models was their ability to utilize modern Hack functions, which are more performant and relevant to Meta’s environment, unlike GPT-4o which often suggested outdated PHP functions due to its reliance on publicly available information.
Ensuring Safety and Efficiency
The initial rollout of MetaMateCR into production revealed an unexpected challenge: reviewers were taking over 5% longer to complete reviews. Further investigation uncovered that reviewers felt compelled to manually check the AI-generated patches, adding an extra layer of scrutiny to their already demanding tasks. This critical finding underscored the importance of safety trials in preventing AI from inadvertently hindering engineering productivity.
In response, the user experience (UX) of MetaMateCR was modified. AI patches were subsequently shown only to the code authors, and were collapsed for reviewers. This strategic change eliminated the regression in review times, demonstrating that the system could be safely integrated without burdening reviewers. This highlights a key insight: while AI can generate fixes, the responsibility of applying and verifying those fixes should primarily rest with the code authors.
Also Read:
- LoopServe: Accelerating LLMs for Dynamic Conversations
- Unlocking LLM Potential: A New Approach to Positional Bias
Real-World Impact
When the LargeLSFT model was fully deployed in production, it achieved an ‘ActionableToApplied’ rate of 19.7%. This represents a substantial 9.2 percentage point improvement over GPT-4o, indicating a significant increase in the rate at which AI-suggested fixes were accepted and integrated into the codebase. The success of MetaMateCR illustrates the profound impact of fine-tuning AI models on specific, large-scale internal datasets and the necessity of real-world safety trials to ensure beneficial outcomes.
The MetaMateCR system is now widely adopted across Meta, processing thousands of code review comments weekly and generating patches for actionable ones. This system, detailed further in the research paper available at AI-Assisted Fixes to Code Review Comments at Scale, stands as a testament to the power of AI in streamlining complex software engineering processes while prioritizing developer efficiency and experience.
