TLDR: A study by York University researchers empirically examines the reasoning processes of six state-of-the-art ‘thinking LLMs’ for code generation. It analyzes step counts, verbosity, and human-evaluated quality (efficiency, logic, completeness) across 100 coding tasks. Findings reveal that reasoning chain length has a model-specific impact, completeness issues (especially edge case handling) are the dominant failure mode, and task complexity significantly affects reasoning quality. The research also explores model stability and self-correction capabilities, highlighting the need for fundamental improvements in how LLMs conceptualize problem scope and requirements.
Large Language Models (LLMs) are increasingly used in software engineering tasks like code generation, but their internal reasoning processes have often been a ‘black box’. This lack of transparency can make it hard to verify solutions, debug failures, and ensure robust performance on complex coding challenges. A recent study titled An Empirical Study of Reasoning Steps in Thinking Code LLMs by Haoran Xue, Gias Uddin, and Song Wang from York University, Canada, delves into this crucial area, examining how ‘thinking LLMs’—models that generate explicit intermediate reasoning traces—perform in code generation.
Unpacking the ‘Thinking’ Process of LLMs
The research focuses on understanding the quality and structure of these explicit reasoning chains. Unlike conventional LLMs, thinking LLMs articulate their problem understanding, solution strategy, and consideration of constraints step-by-step. This transparency is expected to improve trustworthiness and accuracy. However, the reliability of these generated reasoning steps has been largely unexplored.
The study evaluated six state-of-the-art thinking LLMs: DeepSeek-R1, OpenAI-o3-mini, Claude-3.7-Sonnet-Thinking, Gemini-2.0-Flash-Thinking, Gemini-2.5-Flash, and Qwen-QwQ. These models were tested on 100 diverse code generation tasks from the BigCodeBench benchmark, which includes problems of varying difficulty levels (standard and hard).
How LLMs Structure Their Thoughts: Step Counts and Verbosity
One key aspect of the study was analyzing the structure of reasoning chains. Researchers quantified reasoning-chain structure through step counts (how many steps an LLM takes) and verbosity (how detailed each step is). They found that the relationship between the number of reasoning steps and success rate is not straightforward; it’s highly dependent on the specific model and the complexity of the task.
For instance, on challenging ‘hard’ tasks, Gemini-2.0-Flash-Thinking often benefited from taking more steps, suggesting that extended exploration helped it find solutions. Conversely, models like DeepSeek-R1 and Qwen-QwQ sometimes performed better with fewer, more concise steps on hard tasks, indicating that longer chains might signify struggle rather than progress for these models. For easier ‘full’ tasks, step count had little impact on success rates across all models.
The study also experimented with directly manipulating step counts. They found that a small, targeted increase in steps could sometimes improve resolution rates for previously failed tasks, but this benefit was not consistent or monotonic. Similarly, reducing steps by 10-30% often preserved success on standard tasks but significantly degraded performance on hard problems, highlighting the critical role of comprehensive reasoning for complex challenges.
Regarding verbosity, the research showed model-specific strategies. Gemini-2.0-Flash-Thinking, for example, succeeded by using many concise steps, while Qwen-QwQ on hard tasks used fewer but more detailed steps. High verbosity without a clear strategic progression (as seen in DeepSeek-R1 and Gemini-2.5-Flash) did not guarantee success.
Human Perspective: Evaluating Reasoning Quality
To assess the actual quality of the reasoning, a human evaluation was conducted with 21 participants. They evaluated the reasoning traces across three dimensions: efficiency, logical consistency, and completeness. This human-centered approach provided crucial insights into how developers perceive LLM reasoning.
The study developed a ‘reasoning-problematic taxonomy,’ categorizing common failure patterns. The most dominant issue identified was ‘completeness’ (44.5% of problems), particularly the lack of ‘edge case handling’ (32.17%). This means LLMs often fail to consider crucial scenarios like file access permissions, input validation, or boundary conditions. ‘Efficiency issues’ (33.5%) like redundancy and ‘overthinking’ were also common, where models provided unnecessary elaboration or got stuck in circular deliberations. ‘Logic consistency issues’ (7.5%), such as logical gaps or mismatches between claims and implementation, were less frequent but represented fundamental breakdowns.
When looking at individual models, Gemini-2.0-Flash-Thinking showed exceptional completeness, while Gemini-2.5-Flash exhibited the weakest performance across all metrics, often producing incomplete or cut-off reasoning.
Complexity and Self-Correction
Task complexity significantly impacted reasoning quality. On hard tasks, incomplete reasoning was a much stronger predictor of failure compared to standard tasks. While OpenAI-o3-mini generally performed well, it experienced a notable drop in success rate when transitioning from full to hard tasks. Claude-3.7-Sonnet-Thinking, however, demonstrated remarkable resilience, showing a smaller performance decrease on harder problems.
The study also explored the stability and self-correction capabilities of OpenAI-o3-mini. It found that the model maintained consistent logical structures across different ‘reasoning effort’ levels (low, medium, high), even if the exact phrasing varied. Furthermore, o3-mini proved capable of self-correcting previous errors, both when guided with specific cues about the problem type and when simply re-prompted without explicit guidance. This suggests a promising avenue for improving LLM reliability in iterative development workflows.
Also Read:
- Unifying Software Engineering Evaluation for AI Coding Agents with SWE-Compass
- Evaluating AI’s Coding Prowess in Ukrainian: Introducing UA-Code-Bench
The Path Forward for Thinking LLMs
This comprehensive study highlights a critical gap: while thinking LLMs show strong logical depth, they systematically struggle with comprehensive problem analysis, especially in anticipating edge cases and implicit constraints. Simply making LLMs ‘think deeper’ by extending reasoning chains or allocating more computational resources doesn’t reliably solve these issues. Instead, fundamental improvements are needed in how these models conceptualize problem scope, identify critical requirements, and strategically allocate their reasoning effort. Addressing these reasoning quality issues will be vital for building trustworthy AI-assisted programming tools that can truly augment human developers.
