TLDR: LiveOIBench is a new benchmark featuring 403 expert-curated Informatics Olympiad problems to evaluate large language models’ (LLMs) coding capabilities against human contestants. The study found that while GPT-5 performs strongly at the 81.76th percentile, it still lags behind top human programmers. Open-weight models are improving, but LLMs generally struggle with complex algorithms like dynamic programming and exhibit persistent runtime errors, highlighting the need for more strategic reasoning and refined training for efficiency.
A new study introduces LiveOIBench, a comprehensive benchmark designed to rigorously evaluate how well large language models (LLMs) perform in competitive programming, specifically against human contestants in Informatics Olympiads. This research, conducted by a team from the University of Michigan, Ann Arbor, highlights the growing importance of competitive programming problems as a challenging yet verifiable metric for assessing LLMs’ coding abilities.
The researchers, including Kaijian Zou, Aaron Xiong, and Yunxiang Zhang, identified several limitations in existing coding benchmarks. These included a lack of exceptionally difficult problems, insufficient test case coverage, and reliance on online platform APIs that hindered accessibility and reproducibility. LiveOIBench was developed to overcome these issues, offering a robust platform for evaluating advanced LLMs.
What is LiveOIBench?
LiveOIBench is a unique benchmark comprising 403 expert-curated, Olympiad-level competitive programming problems. These problems are sourced directly from 72 official Informatics Olympiads held between 2023 and 2025. Each problem comes with an average of 60 expert-designed test cases, ensuring thorough evaluation and minimizing false positives that plagued previous benchmarks.
The benchmark distinguishes itself through four key features:
- Meticulously curated high-quality tasks with detailed subtask rubrics and extensive private test cases.
- Direct integration of elite contestant performance data, allowing for meaningful comparisons against top human programmers.
- A commitment to continuous, contamination-free updates with newly released Olympiad problems.
- A self-contained evaluation system that facilitates offline and easily reproducible assessments.
Key Findings from the Evaluation
The study benchmarked 32 popular general-purpose and reasoning LLMs. The results provided significant insights into the current state of LLM coding capabilities:
- GPT-5 Leads, But Humans Still Ahead: GPT-5 achieved a notable 81.76th percentile, a strong performance that nonetheless falls short of top human contestants, who typically place above the 90th percentile. This indicates that while LLMs are highly capable, they still have room to grow to match elite human performance in competitive programming.
- Open-Weight Models Show Progress: Among open-weight reasoning models, GPT-OSS-120B achieved a 60th percentile, demonstrating significant advancements and narrowing the performance gap with frontier closed models like GPT-5. However, substantial capability disparities remain.
- Thinking Models Outperform Non-Thinking Models: Models equipped with extended thinking capabilities performed significantly better. Non-thinking LLMs generally struggled, with most failing to exceed a 10% pass rate, underscoring the critical importance of structured reasoning for complex coding tasks.
- Challenges with Advanced Algorithms: LLMs, even top performers like GPT-5, showed weaknesses in algorithms requiring creative observation, intricate state designs, or hierarchical reasoning, such as dynamic programming, segment trees, and tree problems. They performed better on tasks involving straightforward application of standard formulas or well-known patterns.
- Strategic Reasoning Allocation: Detailed analyses of reasoning traces revealed that high-performing models strategically allocate more tokens to precise problem analysis rather than excessive exploration. This suggests that carefully managed reasoning behaviors are crucial for robust performance on challenging tasks.
- Persistent Runtime Errors: While stronger models reduced failure rates related to time limits, memory limits, and compilation errors, runtime errors remained a notable challenge. Researchers hypothesize this might be due to these models pursuing more aggressive and optimized coding patterns, which can increase the potential for execution faults in edge cases.
Also Read:
- The Evolving Paradigm of Trust in Agentic AI: Opportunities, Challenges, and Human-AI Collaboration
- India to Unveil Indigenous Large Language AI Model by End of 2025, Bolstering Digital Sovereignty
Future Directions
The LiveOIBench project aims to drive significant advancements in the reasoning and coding capabilities of LLMs. The researchers suggest future work could explore curriculum-driven fine-tuning for complex algorithms and optimize how models allocate reasoning effort across different cognitive behaviors. Incorporating fine-grained reward signals during training, targeting efficiency and memory management, could also help models produce more reliable and efficient code.
All data, code, and leaderboard results from LiveOIBench will be made publicly available on their website, fostering collaborative research and continuous improvement in the field. You can read the full research paper here.
