TLDR: CodeJudgeBench is a new benchmark designed to evaluate Large Language Models (LLMs) acting as judges for coding tasks like code generation, repair, and unit test generation. The study found that ‘thinking models’ significantly outperform non-thinking models, even smaller ones, but all models exhibit biases related to response order and the source of the code. It also highlights that pair-wise evaluation and providing full, unprocessed responses (including comments) lead to better judging performance, emphasizing the need for more robust and generalizable LLM-as-a-Judge systems.
Large Language Models (LLMs) have become incredibly powerful tools for various coding tasks, from generating code to fixing errors. Beyond just writing code, LLMs are also being used as ‘judges’ to evaluate the quality of code produced by other models. This capability is vital for comparing different LLMs and improving their outputs. However, despite this growing trend, there hasn’t been a dedicated benchmark to properly assess how well LLMs perform as judges in coding scenarios.
To fill this crucial gap, researchers have introduced CodeJudgeBench, a new benchmark specifically designed to evaluate LLM-as-a-Judge models across three key coding tasks: code generation, code repair, and unit test generation. This comprehensive benchmark includes 4,260 carefully selected pairs of responses, sourced from challenging coding problems on platforms like LeetCode, AtCoder, and CodeForces, ensuring the data is fresh and avoids contamination from previously seen problems.
The study evaluated 26 different LLM-as-a-Judge models and found some compelling results. A significant finding is that ‘thinking models’—LLMs that use advanced reasoning techniques like chain-of-thought to analyze problems—consistently outperform models that don’t employ such deep thinking. Surprisingly, even relatively smaller thinking models, such as Qwen3-8B, were able to surpass much larger, specially trained LLM-as-a-Judge models (up to 70B parameters) in overall accuracy. This suggests that the quality of reasoning is more important than sheer model size for judging code.
However, the research also highlighted some challenges. All models showed a notable degree of randomness in their judgments. For instance, simply changing the order in which two code responses were presented could significantly impact the judge’s accuracy. This ‘positional bias’ was observed across different tasks, with some models consistently preferring the first or second response, while others showed task-dependent biases. The study also revealed that LLM-as-a-Judge models perform differently when evaluating code generated by various ‘LLM programmers’ (e.g., Gemini versus Claude). This suggests that judges might be influenced by factors like coding style or formatting, not just the functional correctness of the code.
The benchmark also sheds light on the difficulty of different coding tasks for LLM judges. Evaluating the correctness of unit test generation proved to be the most challenging, followed by code generation, with code repair being the easiest. This might be because code generation and repair tasks provide more contextual information, such as code snippets and error messages, which aids in more accurate judgment, whereas unit test generation often provides only the problem statement.
Regarding optimal prompting strategies, the study found that ‘pair-wise comparison’—where the judge compares two responses directly—outperforms ‘point-wise judging,’ where each response is scored independently. Pair-wise comparison is more suitable for coding tasks, which often have clear correct or incorrect answers, and it helps avoid frequent ties in scoring. Furthermore, the research indicates that providing the LLM-as-a-Judge with the full, unprocessed response, including comments and reasoning, leads to better performance than just providing the raw code. This suggests that the additional context helps the judge make more informed decisions.
Also Read:
- New Research Uncovers Security Concerns in AI Coding Agents
- VerifyBench: A New Benchmark for Evaluating AI Reasoning Verifiers
In conclusion, CodeJudgeBench provides a robust new tool for evaluating LLM-as-a-Judge capabilities in coding. While thinking models show great promise, there’s still a significant need to improve the robustness and generalization of these systems to ensure their reliability and consistency in real-world coding scenarios. For more details, you can refer to the original research paper.
