TLDR: The CORE benchmark evaluates Large Language Models (LLMs) on fundamental static analysis tasks like data dependency, control dependency, and information flow across C/C++, Java, and Python. It features 12,553 human-verified instances, revealing that while LLMs can identify dependencies, they struggle with generating accurate traces and enumerating all sources, especially in complex code structures or with reverse dependencies. The benchmark aims to foster deeper understanding of LLMs’ code reasoning capabilities.
Large Language Models (LLMs) are increasingly integrated into various aspects of software development, from generating code to fixing bugs and detecting vulnerabilities. While these AI models show impressive capabilities, their true understanding of how code works beyond surface-level patterns has remained largely unexamined by existing evaluation methods.
A new benchmark called CORE has been introduced to address this gap. CORE, which stands for Code Reasoning, is a high-quality, human-verified dataset specifically designed to evaluate LLMs on fundamental static analysis tasks. These tasks require a deep understanding of program semantics, such as how values flow through a program, how control structures dictate execution, and how different parts of a program interact.
Existing benchmarks often focus on end-to-end outcomes, like whether a piece of code is correctly repaired. However, they don’t provide insights into the underlying reasoning abilities of LLMs. For instance, to fix a bug or find a vulnerability, an AI needs to understand value propagation (data dependency), how conditions affect execution paths (control dependency), and how information implicitly or explicitly flows through the code (information flow).
CORE fills this critical need by offering 12,553 task instances across programs written in C/C++, Java, and Python. These tasks cover three core semantic properties:
Data Dependency
This refers to how the value of one variable depends on another. For example, if variable ‘y’ is calculated using variable ‘x’, then ‘y’ is data-dependent on ‘x’. Understanding these “def-use chains” is crucial for tasks like tracking untrusted input in security applications.
Control Dependency
This captures whether the execution of one part of the code is governed by a conditional statement. For instance, if a line of code only runs when a specific ‘if’ condition is true, that line is control-dependent on the ‘if’ statement. This is vital for reasoning about program paths and finding bugs that depend on specific execution flows.
Also Read:
- FuzzFeed: Enhancing AI-Driven Program Verification with Fuzzing
- The Coding Triangle: A New Lens on AI’s Programming Abilities
Information Flow
This is a broader concept that combines both data and control dependencies. It tracks how the value of one variable can influence another, either directly through assignments (explicit flow) or indirectly through control conditions (implicit flow). Information flow analysis is fundamental for security, privacy, and debugging tasks like isolating relevant code for a fault.
The benchmark was constructed using a diverse sampling strategy, drawing programs from competitive coding datasets like CodeNet and Google Code Jam. To ensure accuracy, the task instances were meticulously human-verified, a process that involved significant manual effort due to the inherent difficulty of automatically extracting these complex semantic properties.
An extensive evaluation of 10 mainstream LLMs, including top reasoning models, revealed interesting findings. While LLMs generally perform well at simply identifying if a dependency exists (achieving high F1 scores), they significantly struggle with tasks that require deeper semantic understanding, such as generating the exact sequence of steps (traces) for a dependency or enumerating all relevant sources for a given target. Gemini 2.5 Pro showed the best overall performance among the evaluated models.
The research also highlighted factors that increase task difficulty for LLMs. Longer functions, complex control structures (like nested ‘if’ statements or early exits), and “reverse dependencies” (where a later variable influences an earlier one, common in loops) consistently led to a drop in performance. This suggests that LLMs are biased towards processing information in a sequential, left-to-right manner and find it challenging to reason about non-linear or backward influences.
Furthermore, the study found that asking models to provide a trace in addition to classifying a dependency can lead to higher precision but lower recall, indicating a tendency for models to assert a dependency when a trace is required. Providing only the target function code, rather than the entire file, improved performance, suggesting LLMs struggle with irrelevant context. Surprisingly, few-shot learning with additional examples did not improve results, possibly due to increased input length or the limitations of current retrieval methods in capturing structural and semantic similarities in code.
CORE represents a significant step forward in evaluating the true code reasoning capabilities of LLMs, moving beyond simple input-output correctness to probe their understanding of fundamental program semantics. The dataset is publicly available for researchers to further explore and improve AI models’ abilities in this crucial area. You can find more details about the benchmark and access the dataset at the CORE website.
