TLDR: A new dataset called ReDef, built with high-confidence labels from revert commits and GPT-assisted filtering, offers a more reliable benchmark for Just-in-Time Software Defect Prediction (JIT-SDP). A study using ReDef found that while diff-style input encodings improve Pre-trained Language Model (PLM) performance, these models largely rely on superficial cues rather than genuine semantic understanding of code modifications, even when change information is deliberately distorted.
In the fast-paced world of software development, ensuring code quality and preventing defects is paramount. Just-in-Time Software Defect Prediction (JIT-SDP) is a crucial technique that helps developers identify and prioritize risky code changes during the code review process or continuous integration. By flagging potentially problematic modifications early, JIT-SDP can significantly reduce downstream failures and maintenance costs.
However, a major challenge in this field has been the reliability of existing datasets. Many datasets used to train JIT-SDP models suffer from noisy labels, meaning they often misidentify which code changes actually caused a bug. This makes it difficult for models to learn effectively and truly understand what makes a code change defective.
Introducing ReDef: A High-Confidence Dataset
To tackle this problem, researchers from the Korea Advanced Institute of Science and Technology (KAIST) and Jeonbuk National University have introduced ReDef, a new dataset designed for high-confidence Just-in-Time Defect Prediction. ReDef stands for Revert-based Defect dataset, and it focuses on function-level code modifications. The key to its reliability lies in how it identifies defective cases: by anchoring them to “revert commits.” When a developer reverts a previous commit, it often indicates that the original change introduced a problem. ReDef uses these explicit signals to identify defective code.
For clean (non-defective) cases, the dataset employs a rigorous post-hoc history check to ensure accuracy. Furthermore, ambiguous instances are filtered out using a sophisticated GPT-assisted triage process, involving multiple votes and audits to guarantee high precision. This meticulous approach results in a dataset with 3,164 defective and 10,268 clean modifications, offering substantially more reliable labels than previous resources. A manual audit confirmed an impressive 92% precision for positive labels, far surpassing the noise levels found in other datasets.
Probing Language Models on Code Changes
Beyond creating a robust dataset, the research also provides the first systematic evaluation of how Pre-trained Language Models (PLMs) interpret and reason about code modifications. PLMs like CodeBERT, CodeT5+, and UniXcoder have shown remarkable success in various software engineering tasks, such as code search and vulnerability detection. The central question here was whether these models could genuinely understand the semantics of code changes – the relationship between the ‘before’ and ‘after’ states of a code modification.
The evaluation was conducted in two stages. First, the researchers compared five different ways of presenting code modification information to the PLMs, known as “input encodings.” These ranged from simply providing the ‘after’ version of the function to more structured formats like “Diff with tags” (showing only changed lines with add/delete markers) and “Added→Deleted blocks” (separating added and deleted lines). The findings revealed that compact, diff-style encodings consistently outperformed whole-function formats across all PLMs. This suggests that explicitly highlighting the changes, rather than providing the entire function which might get truncated or contain distracting context, is more effective for defect prediction.
The Surprising Truth: Reliance on Superficial Cues
The second stage of the evaluation involved a series of “counterfactual perturbations.” Here, the researchers deliberately distorted the change information to see if the PLMs would degrade as expected. For example, they swapped added and deleted code blocks, inverted the polarity of diff tags (making additions look like deletions and vice-versa), or injected spurious change markers. The results were surprising: the PLMs’ performance remained largely unchanged, with F1 score differences staying below 1%.
This unexpected stability indicates that the models were not truly leveraging the semantic signals from the code changes. Instead, they appeared to rely on superficial cues, such as token distributions or positional patterns, that persisted even when the actual meaning of the change was distorted. In essence, what seemed like robustness was, in fact, an insensitivity to the genuine semantics of code modifications.
Also Read:
- Bridging the Semantic Gap: Enhancing Software Traceability with Multi-Strategy Approaches
- LoCoBench: A New Standard for Evaluating AI in Complex Software Development
Implications for Future Software Quality
These findings highlight a critical limitation of current PLMs: while they excel at understanding static code snapshots, they struggle with genuinely comprehending code modifications. This suggests a fundamental gap between snapshot-level code understanding and true modification reasoning. The research calls for new approaches, such as pre-training objectives that explicitly expose models to edits and diffs, or richer change representations that make modification semantics more accessible to PLMs.
ReDef, along with this systematic evaluation protocol, provides a valuable foundation for the next generation of change-aware and semantics-sensitive defect prediction models. For more in-depth information, you can read the full research paper here.
