TLDR: A new research paper introduces a comprehensive benchmark to evaluate and improve Large Language Models’ (LLMs) ability to understand code semantics, beyond just syntax. By fine-tuning models like QWQ-32B, Codestral-22B, and Granite-8B on tasks such as code grading, bug fixing, and question answering, the study demonstrates significant improvements in their code comprehension, with QWQ-32B’s accuracy on a key task increasing from 70% to 83.47%.
Large Language Models (LLMs) have made significant strides in coding tasks like generating new code and completing existing snippets. These models are often trained to predict the next token in a sequence, which helps them learn the surface-level syntax of code. However, this doesn’t automatically mean they truly understand the underlying meaning or “semantics” of the code. This gap in comprehension can lead to challenges in more complex tasks such as debugging or optimizing code.
To address this, new research proposes a method to improve LLMs’ code comprehension by fine-tuning them on large datasets specifically designed for understanding code semantics. The goal is to help these models develop a more robust grasp of what the code actually does, beyond just its structure.
The researchers evaluated three different code models, varying in size from 8 billion to 32 billion parameters, across a suite of code comprehension tasks. These tasks were carefully designed to assess semantic understanding, moving beyond simple syntactic pattern matching. The study observed that model performance significantly improved after fine-tuning on these relevant tasks. For instance, the QWQ-32B model saw its accuracy on the Subjectivity Grading Task jump from 70% to 83.47%. Similar positive trends were noted across other models, clearly indicating an enhancement in their ability to comprehend code. Notably, the DPO-fine-tuned Codestral-22B achieved the highest micro-accuracy of 87.66% on the Subjectivity Grading Task.
Understanding Code Comprehension
Code comprehension is the ability to grasp both the functionality and the deeper meaning of source code. It’s not just about recognizing what the code does, but also understanding its structure, how it executes, and its overall purpose. This skill is vital for many development activities, including finding and fixing bugs, refactoring code to make it more efficient, and analyzing code for potential issues. While current LLMs excel at generating code, their training often focuses on “next token prediction,” which doesn’t fully capture the structural aspects of code, such as how it’s represented in Abstract Syntax Trees (ASTs) or how data and control flow through it.
The paper highlights that incorporating these structural aspects into training is essential for building robust and reliable code models. Since deep learning models are data-driven, training them with code’s structural data can significantly boost their comprehension abilities.
Key Tasks for Evaluation
To rigorously evaluate code comprehension, the researchers selected several tasks that specifically target the structural and semantic aspects of code. These tasks assess how well models can reason about functionality, structure, and correctness. Each task is supported by a large dataset to ensure comprehensive training and evaluation:
- Subjectivity Grading Task (SGT): Given a programming problem, a student’s code submission, and a grading criterion, the model predicts an appropriate rating for the submission. This task uses the CS101-Gold dataset, curated from an introductory programming course.
- Code Question-Answering Task (QAT): The model generates a free-form textual answer to a natural language question about a given code snippet. This requires understanding both natural language and programming logic, using the CodeQA dataset.
- Code Search Task (CST): Given a natural language query, the model extracts the most relevant code fragment. This task, crucial for developer productivity, uses the CodeSearchNet dataset.
- Test Case Task (TCT): An upgraded version of SGT, where the model predicts whether a provided code passes specific test cases.
- Bug Fix Task (BFT): Given buggy code and a prompt, the model identifies errors and generates a corrected version. This uses a dataset of buggy and fixed Java functions, augmented with synthetic pairs.
- Code Comparison Task (CCT): The model distinguishes which of two student codes better aligns with a high rating for a specific criterion, testing its ability to map code to natural language descriptions.
Models and Experiments
The study evaluated three distinct models to demonstrate the generalizability of their approach across different sizes:
- QWQ-32B: Alibaba’s 32-billion parameter model, built on Qwen 2.5-32B, focusing on analytic reasoning and computational efficiency, enhanced with reinforcement learning.
- Codestral-22B: Mistral AI’s 22-billion parameter open-weight model, trained on over 80 programming languages, excelling in code generation and completion.
- Granite-8B: IBM’s 8-billion parameter model, part of the Granite series, designed for enterprise software development with a focus on long-context understanding.
The experiments were structured along three dimensions: Post-hoc vs. Pre-hoc evaluation (analyzing existing code vs. generating new code), Intrinsic vs. Extrinsic grounding (internal understanding vs. applying understanding to broader goals), and Abstractive vs. Extractive capability (generating new insights vs. retrieving specific information).
Also Read:
- LiveRepoReflection: A New Standard for Evaluating AI’s Code Understanding in Complex Projects
- CodeJudgeBench: A New Benchmark for Evaluating AI Code Judges
Results and Future Directions
The results consistently showed that fine-tuning on these downstream tasks significantly improved the models’ code comprehension abilities. For example, the QWQ-32B model saw substantial gains across tasks like Code Search, Question Answering, and Bug Fixing after being combined with the Subjectivity Grading Task. The Granite-8B model also showed notable improvements in accuracy and F1 scores across various tasks when subjectivity grading was integrated.
While Codestral-22B started with very high baseline performance, its improvements after integrating subjectivity grading were less dramatic, and in some cases, performance even slightly reduced. The researchers suggest this might be due to “catastrophic forgetting,” where the model forgets previously learned knowledge during multiple fine-tuning tasks, or that its architecture isn’t optimized for leveraging multi-task fine-tuning in the same way.
The paper concludes that training LLMs on semantically rich tasks like grading and bug fixing is crucial for enhancing their code comprehension. This research advocates for a shift in how code LLMs are evaluated—moving from just checking syntactic correctness to assessing true semantic understanding. This lays the groundwork for developing more robust and interpretable code models in the future. For more detailed information, you can refer to the full research paper here.
