TLDR: CODE2BENCH is a new framework for creating dynamic, contamination-resistant benchmarks from real-world GitHub code to evaluate Large Language Models (LLMs) on coding tasks. It uses automated code ingestion, dependency analysis to classify tasks (Self-Contained and Weakly Self-Contained), and Property-Based Testing for rigorous test generation. The first benchmark instance, CODE2BENCH-2505, reveals that LLMs struggle with complex, non-standard logic and cross-language tasks, but perform better on tasks involving common library interactions. The benchmark effectively exposes subtle model failures and provides a more realistic assessment of LLM capabilities in software development.
As Large Language Models (LLMs) become increasingly vital in software development, accurately assessing their performance on complex, real-world code generation tasks is crucial. However, existing evaluation methods often fall short due to issues like data contamination and insufficient testing rigor, which can hide model weaknesses.
To tackle these challenges, researchers have introduced CODE2BENCH, a systematic, end-to-end pipeline designed to dynamically build robust and contamination-resistant benchmarks directly from real-world GitHub repositories. This innovative framework aims to provide a more reliable and realistic way to evaluate how well LLMs can handle actual coding problems.
Key Innovations of CODE2BENCH
CODE2BENCH stands out with three core innovations:
First, it features Automated Dynamism. This is achieved by continuously ingesting recent code from active GitHub repositories. This periodic update mechanism is essential for minimizing the risk of training data contamination, a common problem with static benchmarks where models might have already seen the test data during their training. By using fresh, recent code, CODE2BENCH ensures the benchmark remains relevant and challenging as the software development landscape evolves.
Second, it employs Scope Graph-based dependency analysis. This sophisticated technique allows for the structured classification of functions into benchmark instances with controlled dependency levels. It distinguishes between two main types of tasks: Self-Contained (SC) tasks, which are entirely independent and suitable for cross-language evaluation, and Weakly Self-Contained (WSC) tasks, which involve controlled usage of specific, commonly used external libraries. This classification ensures that tasks are realistic yet manageable in complexity, providing clear boundaries for evaluation.
Third, CODE2BENCH utilizes Property-Based Testing (PBT) for the automated synthesis of rigorous, high-coverage test suites. Unlike traditional testing that relies on fixed examples, PBT dynamically generates diverse inputs based on a function’s properties. This approach enables the discovery of subtle errors and edge cases that might be missed by less comprehensive test cases, ensuring thorough functional verification of the generated code.
Introducing CODE2BENCH-2505
Using this advanced pipeline, the researchers constructed CODE2BENCH-2505, the first dynamic benchmark instance derived from 880 recent Python projects across various domains. This benchmark comprises 1,163 code generation tasks, boasting an impressive 100% average branch coverage on ground-truth implementations. This high coverage signifies the exceptional rigor of the generated test suites.
What the Evaluation Revealed
Extensive evaluation of 16 state-of-the-art LLMs using CODE2BENCH-2505 yielded several surprising and crucial insights:
- Struggles with Complex Logic: Models consistently struggled with Self-Contained (SC) tasks, especially those requiring complex, non-standard logic and cross-language transfer. This suggests that LLMs find it harder to generate code for problems that demand deep algorithmic understanding rather than pattern recall.
- Better on Library Interactions: Conversely, LLMs showed relatively stronger performance on Weakly Self-Contained (WSC) tasks in Python. This indicates that models are more proficient at replicating common library interaction patterns, even if these involve complex data structures.
- Cross-Language Challenges: Performance varied significantly across different programming languages for SC tasks. Python consistently yielded the highest scores, while models showed substantial drops in other languages like Go, TypeScript, and Java. This highlights limitations in LLMs’ ability to generalize language-agnostic task descriptions into diverse linguistic contexts.
- Rigorous Testing Pays Off: The Property-Based Testing approach proved highly effective in revealing nuanced model limitations. Even for highly capable models, a significant percentage of problems were nearly correct but failed on a small number of subtle test cases, which would have been incorrectly marked as solved by less thorough testing.
- Difficulty Matters: As expected, LLM performance decreased significantly with increasing task difficulty (Easy, Medium, Hard), confirming CODE2BENCH-2505’s ability to capture a wide spectrum of real-world coding challenges.
The detailed analysis of failure modes further supported these findings. For SC tasks, most failures were due to incorrect logic (Test Failures), while for WSC tasks, a higher proportion of failures were due to code not running at all (Execution Errors), indicating issues with correct API usage or library-specific data structures.
Also Read:
- Enhancing LLM Code Generation with Klear-CodeTest’s Advanced Test Case Synthesis
- Benchmarking AI Agents: A New Standard for Evaluating Tool Use Capabilities
A New Standard for Evaluation
CODE2BENCH introduces a contamination-resistant, language-agnostic methodology for dynamic benchmark construction. It offers a principled foundation for the comprehensive and realistic evaluation of LLMs on real-world software development tasks, providing deeper insights into their strengths and weaknesses than previously possible. The codes, data, and results are publicly available for further exploration at the project website.
Future work will focus on expanding CODE2BENCH’s language support and enabling domain-specific benchmark construction, leveraging its language-agnostic core and the extensibility of its Scope Graph-based dependency analysis.
