TLDR: This study explores fine-tuning multilingual language models (CodeReviewer, CodeLlama-7B, DeepSeek-R1-Distill) on C# code for automated code review tasks like quality estimation, comment generation, and code refinement. It finds that monolingual fine-tuning generally improves performance, especially for quality estimation, but human reviewers remain superior for complex, nuanced feedback. The research highlights the potential of AI tools for efficiency gains in industrial settings, suggesting a hybrid approach combining AI speed with human expertise.
Code review is a fundamental practice in software development, crucial for maintaining high code quality, enforcing standards, and detecting defects early. However, this essential process is often time-consuming and cognitively demanding, especially in large industrial projects where developers might spend several hours per week on review tasks. The sheer volume of code changes in major companies, like Microsoft Bing processing thousands of reviews monthly, underscores the significant manual effort involved and its potential impact on productivity.
While Automated Static Analysis Tools (ASATs) like SonarQube have been developed to reduce manual effort by identifying issues such as bugs and style violations, they frequently suffer from high false-positive rates and lack the contextual understanding needed for nuanced code evaluations. This often leads to developers sifting through numerous irrelevant warnings, diminishing the tools’ effectiveness.
Recent advancements in artificial intelligence, particularly in language models (LMs), have opened new avenues for automating parts of the code review process, known as Automated Code Review (ACR). These LMs are trained to understand and generate natural language text, and various pretrained language models (PLMs) have been proposed to support ACR tasks like generating review comments, suggesting improvements, and refining code. However, much of this research has focused on dominant programming languages like Java and Python, leaving languages like C#, widely used in enterprise software, relatively underexplored.
This study addresses these limitations by fine-tuning existing multilingual LMs on monolingual C# data. The researchers evaluated three open-source models: CodeReviewer, a transformer-based PLM designed for review comment generation; CodeLlama-7B, a decoder-only multilingual Large Language Model (LLM) pretrained for general code tasks; and DeepSeek-R1-Distill, a multilingual instruction-tuned LLM optimized for multi-domain applications without specific code pretraining.
The evaluation focused on three core ACR tasks: Code Change Quality Estimation (determining if a change needs human review), Review Comment Generation (generating natural language feedback), and Code Refinement (automatically applying suggested improvements). The study utilized a unique C#-specific dataset, combining public benchmarks with proprietary industrial repositories from Lovion GmbH, a software company specializing in infrastructure asset and network management. This industrial dataset provided high-quality, production-grade review comments, originally in German and translated to English for consistency.
Key Findings from the Study
For Code Change Quality Estimation, the monolingual C# fine-tuned CodeReviewer consistently outperformed its multilingual baseline across all metrics (precision, recall, F1 score, and accuracy). This suggests that tailoring LMs to a specific domain and language can significantly improve their ability to identify relevant code changes, even with smaller training datasets.
In Review Comment Generation, the results were more nuanced. While the fine-tuned CodeReviewer achieved the highest BLEU-4 score (a metric for lexical similarity), human evaluators rated the fine-tuned CodeLlama-7B higher in terms of Information and Relevance. This highlights a known limitation of automated metrics like BLEU-4, which may not fully capture the quality, informativeness, or contextual relevance of human-like feedback. Interestingly, the DeepSeek-R1-Distill model, which is reasoning-focused, showed a decline in its ability to generate rich, explanatory comments after fine-tuning, despite an improved BLEU-4 score. This indicates that fine-tuning approaches need to be carefully designed to preserve or enhance reasoning capabilities in such models.
Conversely, for the Code Refinement task, monolingual fine-tuning led to a decline in performance for CodeReviewer compared to its multilingual baseline. This suggests that tasks requiring broader generalization or exposure to diverse examples, like code refinement, might suffer from the limited size and variety of a monolingual fine-tuning dataset.
When comparing LMs, SonarQube, and human reviewers, the study found that human reviewers consistently provided the highest quality and most context-aware feedback, especially for nuanced and complex code changes. SonarQube excelled at detecting rule-based issues (e.g., security vulnerabilities, maintainability concerns) but struggled with deeper logic or architectural flaws. The fine-tuned LMs, particularly CodeLlama-7B, showed better generalization across various issue categories, including refactoring suggestions. While LMs and SonarQube completed reviews significantly faster than humans (often under one minute compared to five to seven minutes for humans), their feedback lacked the depth and accuracy required for critical assessments.
Also Read:
- Assessing LLM Code Generation: Introducing ReCatcher for Regression Testing
- LOTUS: A New Framework for Evaluating Advanced Image Captioning
Implications for Software Development
The findings suggest that fine-tuned LMs can serve as valuable, fast, and reasonably accurate assistants in continuous integration (CI) pipelines. They can help filter routine pull requests (PRs), thereby reducing the cognitive load on human reviewers. A hybrid approach, combining the speed and efficiency of AI tools with the deep, contextual understanding of human experts, appears to be the most effective strategy for industrial code review workflows.
The study also emphasizes the importance of human-centered evaluations over sole reliance on automated metrics like BLEU for assessing code review quality. Furthermore, it highlights a need for future research to develop fine-tuning strategies that preserve the reasoning capabilities of LLMs, potentially through the use of chain-of-thought data or multi-objective optimization. For a deeper dive into the methodology and detailed results, you can refer to the full research paper: Fine-Tuning Multilingual Language Models for Code Review: An Empirical Study on Industrial C# Projects.
