TLDR: This research investigates how different instruction strategies, inspired by Martin Fowler’s guidelines, can improve Large Language Models’ (LLMs) ability to perform diverse code refactoring tasks. The study found that explicit, structured instructions (like step-by-step or rule-based) significantly boost LLM success rates, especially for DeepSeek-V3. While LLMs can handle many refactoring types, including complex ones, they sometimes struggle with semantic preservation and introduce compilation errors due to hallucinations. Interestingly, giving LLMs a high-level objective to improve code quality, without specifying a refactoring type, can lead to better quality metrics like reduced cyclomatic complexity, even if the exact intended refactoring isn’t applied. The paper highlights the potential of LLMs in software engineering and suggests ways to improve code assistant tools.
Code refactoring is a crucial practice in software development, aimed at improving the internal structure of code without changing its external behavior. It makes code cleaner, easier to maintain, and more efficient. However, developers often find themselves skipping refactoring due to the significant time and effort it demands, coupled with a lack of immediate functional rewards. While automated refactoring tools exist, they often fall short in supporting a wide variety of refactoring types.
A recent study explores how instruction strategies, inspired by human best practices, can significantly enhance the ability of Large Language Models (LLMs) to perform diverse refactoring tasks automatically. Leveraging the advanced instruction-following and code comprehension capabilities of state-of-the-art LLMs like GPT-mini and DeepSeek-V3, the researchers designed multiple instruction strategies based on Martin Fowler’s renowned refactoring guidelines. These strategies encode the motivations, procedural steps, and transformation objectives for 61 well-known refactoring types.
Exploring Instruction Strategies for LLMs
The study evaluated these strategies on both benchmark examples and real-world code snippets from GitHub projects. The findings reveal that instruction designs grounded in Fowler’s guidelines enable LLMs to successfully perform all benchmark refactoring types while preserving program semantics—a critical aspect of effective refactoring. Interestingly, while descriptive instructions are more human-interpretable, rule-based instructions often led to better performance in specific scenarios. A particularly insightful discovery was that allowing models to focus on the overall goal of refactoring, rather than prescribing a fixed transformation type, could yield even greater improvements in code quality.
The researchers investigated two primary questions: how instruction strategies guide LLMs in applying diverse refactoring types, and what impact these strategies have on the quality of the refactored code. For the first question, they found that explicit guidance significantly improved LLM performance. DeepSeek, for instance, achieved a 100% success rate with both Step-by-Step and Rule-based instructions on benchmark scenarios, while GPT-4o-mini performed best with Step-by-Step learning. This suggests that breaking down transformations into smaller reasoning steps or providing explicit rules helps LLMs achieve higher accuracy. Conversely, a high-level “Objective Learning” prompt, which only described the general goal of refactoring, resulted in the lowest success rates for both models in correctly applying specific refactoring types.
When looking at real-world scenarios, the study used metrics like compilation rate, new failed tests, and new test errors. GPT-4o-mini generally achieved higher compilation rates but sometimes introduced more semantic errors (test failures/errors). DeepSeek, while having slightly lower compilation rates, showed stronger semantic preservation once the code compiled. Simple, localized refactorings like “Split Variable” and “Extract Variable” were consistently easier for LLMs to perform, regardless of the instruction type. More complex refactorings, such as “Change Function Declaration” or “Inline Variable,” proved challenging for both models.
Impact on Code Quality
For the second research question, focusing on code quality, the study used metrics like CodeBLEU (similarity to ground truth), Lines of Code (LOC), Cyclomatic Complexity (CC), and Fan-Out (FOUT). All instruction strategies resulted in moderate to low CodeBLEU scores, suggesting that LLMs often generate novel solutions rather than simply memorizing and reproducing existing code. Rule-based and Step-by-Step learning generally produced refactorings most similar to human-applied transformations.
Intriguingly, the “Objective Learning” strategy, which provided only a high-level goal of improving readability, maintainability, and quality without specifying a refactoring type, often led to the lowest Cyclomatic Complexity and Lines of Code for DeepSeek. This indicates that when unconstrained by specific transformation types, LLMs can still improve code quality by focusing on the broader objective of refactoring, sometimes even outperforming human-applied refactorings in terms of complexity reduction. However, this also meant the LLMs might not apply the intended refactoring type from the ground truth.
Also Read:
- Assessing Code Quality: A Deep Dive into LLM-Generated Code Smells
- Enhancing Automated Program Repair with Intelligent Filtering Policies
Challenges and Future Directions
The study also delved into the compilation issues faced by LLMs, identifying lexical, parsing, and compilation errors. Many errors stemmed from “hallucinations,” where LLM-generated code referenced non-existent symbols or introduced invalid syntax. Duplicated elements and incompatible types were also common. These issues highlight the need for better contextual understanding and error-checking mechanisms in LLM-generated code.
The findings have significant implications for improving current code assistant tools. By understanding common error patterns, these tools could provide more targeted support, such as suggesting fixes for “cannot find symbol” errors or informing LLMs about project-specific dependencies. The research also suggests that for conventional refactorings, LLMs are highly capable, but for more unconventional or complex cases, precise rule-based prompts could be more effective. Future work could explore autonomous systems for continuous refactoring and human-in-the-loop approaches to ensure semantic accuracy and transparency in LLM-driven refactoring processes.
This comprehensive study underscores the promising potential of LLMs in automating and enhancing code refactoring practices, bridging the gap between human expertise and machine understanding in software engineering. You can read the full paper here.
