TLDR: This research introduces Gistify, a novel task designed to evaluate how well large language models (LLMs) understand and interact with entire codebases. Unlike previous benchmarks that might be solved with shortcuts, Gistify requires an LLM to create a single, minimal, self-contained file that accurately reproduces a specific functionality from a larger codebase. The generated “gistified” file must match the original runtime behavior, contain only essential code, and be derived solely from the original codebase. Experiments with state-of-the-art models and frameworks reveal that current LLMs struggle with Gistify tasks, particularly those involving long execution traces. The findings highlight the importance of faithfully preserving test functions and suggest that while execution tools offer some benefit, dynamic reasoning and access to global code context are crucial for improving performance. Gistified files themselves are presented as valuable outputs, offering simplified, executable representations of complex codebase functionalities for various downstream applications.
As artificial intelligence models become increasingly integrated into software development, their ability to understand and navigate large, intricate codebases is more critical than ever. However, evaluating this capability has been a significant challenge. Many existing benchmarks for coding agents can often be solved using shortcuts or by retrieving small, localized code snippets, rather than requiring a deep, holistic understanding of how code executes across an entire project.
To address this gap, researchers have introduced a new task called Gistify. This task challenges a coding large language model (LLM) to create a single, minimal, and self-contained file that can accurately reproduce a specific functionality from a larger codebase. Imagine a developer trying to understand a new project: they often start by running a specific command or test, then trace its execution to grasp the codebase’s structure and functionality. Gistify formalizes this process, requiring an AI agent to extract the “gist” of a given command.
What Makes a Gistified File Successful?
For a file to be considered successfully “gistified,” it must meet four key requirements:
- Self-Contained: The file must include all necessary components from the original codebase to run independently, without relying on external modules from the original project.
- Execution Fidelity: When executed, the gistified file must produce the exact same outputs and runtime behavior as the original codebase under the specified command.
- Minimalism: It should only contain the essential code required for execution, with all unused functions, objects, and redundant lines removed.
- Grounded Preservation: The generated code must be derived directly from the original codebase, without any “hallucinated” or fabricated content.
To evaluate models on Gistify, researchers use a consistent setup involving a Docker image of the target codebase and a specific entrypoint, such as a pytest command. Performance is measured using three metrics: Execution Fidelity (a binary pass/fail), Line Execution Rate (how much of the generated code is actually run, indicating minimality), and Line Existence Rate (how much of the generated code is directly from the original, indicating preservation).
Key Findings from the Research
The study involved experiments with various leading LLM frameworks (mini-SWE-agent, SWE-agent, and Copilot) and models (GPT-5-mini, GPT-5, Claude-3.7-Sonnet, and Claude-Sonnet-4). The results revealed several interesting insights:
- Even state-of-the-art models and frameworks struggle with Gistify tasks, especially when the execution traces are long and cover a significant portion of the repository. The strongest combination (Copilot with Claude-4) achieved only 58.7% average Execution Fidelity.
- Different model families showed distinct strengths. Claude-4 excelled in Line Existence, faithfully extracting code, while GPT-5 produced more concise outputs with higher Line Execution rates.
- Smaller models, like GPT-5-mini, showed significant performance improvements when provided with richer scaffolding and tools, such as those offered by the Copilot framework.
- Frontier models (GPT-5 and Claude-4) demonstrated strong capabilities even in basic bash-only setups.
- Surprisingly, enabling execution tools (like running Python or pytest commands) yielded only small, consistent performance gains. This suggests that current LLMs don’t fully leverage these tools for deep runtime analysis, such as using a debugger.
Understanding Failure and Improvement
An analysis of common errors showed that models often fail due to “Import Errors” (incorrectly importing the original codebase instead of inlining modules), “File Creation Failures” (failing to generate the gistified file), “Missing Test Functions” (not including the necessary test code), or “Pytest Runtime Errors” (execution failures). Claude-4, despite being a top performer, frequently encountered Import Errors.
A crucial finding was the strong correlation between successfully preserving the original test function and overall task success. Models that accurately copied the test function were far more likely to produce a successful gistified file. An ablation study confirmed that providing the correct test function upfront significantly boosted performance, highlighting this as a critical first step in the Gistify process.
Further analysis explored the impact of different strategies and tools. Prompting models with explicit guidance (like “tracing” or “reading” strategies) improved performance. Providing global context through tools like RepoGraph (a codebase graph) or a Tracing tool (exposing gold execution traces) also led to gains, with the latter being particularly effective. Interestingly, having access to fewer, more focused execution tools (like “Edit And Execute” for the gistified file) sometimes outperformed unrestricted bash access, as the latter could lead to models exploring too many options and hitting step limits.
The research also confirmed that tasks with longer and more complex execution traces (measured by function calls and unique files touched) were significantly harder for LLMs to gistify. Agentic models, which can dynamically select files and iterate on solutions, consistently outperformed static approaches, even when static models were given all relevant files at once.
Also Read:
- Evaluating Agentic Potential in LLMs During Pre-Training with APTBench
- LongWeave: A New Standard for Assessing AI’s Long Text Capabilities
The Future of Codebase Understanding
Gistify not only serves as a challenging benchmark for evaluating AI’s codebase-level understanding but also produces valuable “gistified” files. These compact, executable representations of specific functionalities can be incredibly useful for human developers and other AI agents. They can simplify debugging, aid in code refactoring, and provide a clearer understanding of complex systems without the overhead of navigating an entire large codebase. This research paves the way for more robust and capable AI agents in software development. You can read the full research paper here.
