TLDR: The COMPASS benchmark evaluates large language models for code generation across three critical dimensions: functional correctness, algorithmic efficiency, and code quality, moving beyond the traditional focus on correctness alone. It uses 50 real-world competitive programming problems with human baselines. Findings show that models achieving high correctness don’t always produce efficient or maintainable code, highlighting the need for multi-dimensional assessment to understand real-world capabilities.
Large Language Models (LLMs) have made significant strides in generating code, but a new research paper introduces a critical perspective on how we evaluate their true capabilities. Traditionally, benchmarks for code generation have focused almost exclusively on functional correctness – essentially, whether the code produces the expected output for given test cases. However, real-world software development demands much more than just correct code; it requires code that is also efficient and of high quality.
The paper, titled COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models, highlights a major gap in current evaluation methods. Existing benchmarks, such as HumanEval and MBPP, might score a solution as perfect even if it runs incredibly slowly or is poorly structured. This overlooks crucial aspects like algorithmic efficiency, which determines how well code performs with large inputs, and code quality, which impacts maintainability, readability, and long-term productivity.
To address this, James Meaden, Michał Jarosz, Piotr Jodłowski, and Grigori Melnik, all affiliated with Codility, introduce COMPASS (COdility’s Multi-dimensional Programming ASSessment). This comprehensive evaluation framework assesses code generation across three vital dimensions: correctness, efficiency, and quality. COMPASS is built upon 50 competitive programming problems sourced from real Codility competitions, providing authentic human performance baselines from over 393,000 submissions.
How COMPASS Works
The benchmark evaluates code solutions in three distinct ways:
-
Correctness: This measures the percentage of test cases passed, including basic functionality, edge cases, and corner cases. It provides a granular view of how robust a solution is.
-
Efficiency: This dimension specifically tests algorithmic scalability. Test cases involve large input sizes and are evaluated against strict runtime thresholds derived from expert-crafted, optimal solutions. This helps differentiate between code that merely works and code that works efficiently.
-
Quality: Beyond just working and being fast, COMPASS assesses how well-written the code is. It uses static analysis via CodeScene, an industry-standard platform, to analyze aspects like cyclomatic and cognitive complexity, function length, nesting depth, code duplication, cohesion, and adherence to best practices. Each solution receives a composite quality score, normalized to a 1-100 scale.
The researchers evaluated three leading reasoning-enhanced models: Anthropic Claude Opus 4, Google Gemini 2.5 Pro, and OpenAI O4-Mini-High. They used a sampling strategy of 64 independent solutions per model-problem combination and explored four different prompt types (neutral, correctness-optimized, efficiency-optimized, and dual objective) to see how prompt phrasing influenced model behavior.
Key Findings
The study revealed several important insights:
-
Distinct Dimensions: Correlation analyses confirmed that correctness and efficiency are moderately related, but code quality is largely independent of both. This validates COMPASS’s multi-dimensional approach, showing that each metric provides unique and valuable information about a model’s performance.
-
Overall Performance: While all models achieved high correctness scores, significant differences emerged in efficiency and quality. O4-Mini-High consistently delivered the highest performance in correctness (mean 95.6%) and efficiency (mean 93.0%), making it the top overall performer. Gemini 2.5 Pro also performed strongly across all dimensions, with the highest code quality (mean 93.2%). Claude Opus 4 showed strong code quality (mean 92.3%) but significantly lower efficiency (mean 35.4%) and more variable correctness (mean 72.2%), leading to the lowest overall composite score.
-
Human Comparison: When compared to human performance baselines from competitive programming challenges, O4-Mini-High and Gemini 2.5 Pro significantly outperformed human participants, achieving average percentile ranks of 97.840 and 96.280, respectively. Claude Opus 4 was lower at 76.580.
-
Consistency: O4-Mini-High demonstrated the most consistent performance across repeated runs and different tasks, indicating more stable and reliable behavior. Claude Opus 4, conversely, showed the highest variability.
-
Prompting Effects: The study found that prompt phrasing had only modest effects on overall model performance. However, Claude Opus 4 was more responsive to prompt framing, showing noticeable improvements when given explicit guidance on correctness or efficiency. O4-Mini-High and Gemini 2.5 Pro were largely robust to minor prompt variations.
The findings underscore that models achieving high correctness scores do not necessarily produce efficient algorithms or maintainable code. This disconnect highlights the practical limitations of benchmarks that treat software development as a simple pass/fail task. In real-world scenarios, inefficient or unmaintainable code, even if functionally correct, can lead to significant costs, delays, and scalability issues.
Also Read:
- Large Language Models Struggle to Accurately Verify Code, Study Finds
- EvolMathEval: A Dynamic Approach to Challenging AI’s Mathematical Reasoning
Implications for the Future
COMPASS serves as a guiding framework for future research, charting a path toward AI systems that are robust, reliable, and ready for production use. For researchers, it provides a tool to evaluate not just if models can solve problems, but how they solve them and at what cost. For practitioners, it offers insights into which models are production-ready and where trade-offs might exist. The authors urge the broader research and engineering communities to move beyond syntactic correctness as the sole proxy for software engineering capability, emphasizing that the question is no longer “Can they write code that works?” but “Can they write code that lasts?”
