TLDR: A new study evaluated eight open-source, locally hosted large language models (LLMs) on over 3,500 complex programming problems from Kattis. The research found that while local LLMs offer benefits like cost control and data privacy, their pass@1 accuracy was modest, with the best models performing at approximately half the acceptance rate of proprietary models like Gemini 1.5 and ChatGPT-4. The study highlights a persistent performance gap, especially with increasing problem difficulty, but also points to the rapid advancements in open models and the potential for in-house evaluation workflows.
A recent study delves into the capabilities and limitations of open-source, locally hosted large language models (LLMs) when faced with complex programming challenges. The research, conducted by Kadin Matotek, Heather Cassel, Md Amiruzzaman, and Linh B. Ngo from West Chester University, sheds light on the performance gap between these private, cost-controlled LLM deployments and their state-of-the-art proprietary counterparts like Gemini 1.5 and ChatGPT-4.
The increasing reliance on AI for code generation has brought forth concerns regarding data privacy, latency, and cost associated with cloud-based, proprietary LLMs. This has led many organizations to explore deploying open-source models locally. However, a comprehensive evaluation of these local models on complex coding tasks has been largely absent until now.
Building upon an existing framework called FACE (Framework for AI-driven Code Generation Evaluation), the researchers significantly enhanced it to operate entirely offline using the Ollama runtime. This retrofitted pipeline streamlined data organization, consolidating thousands of problem files into a handful of JSON files, and introduced robust checkpointing. This crucial addition allows multi-day evaluation runs to resume seamlessly after any interruptions, making large-scale testing feasible.
The extended framework was then used to generate, submit, and record solutions for the entire Kattis corpus, a publicly available platform featuring over 3,500 programming problems of varying difficulty. Eight code-oriented local LLMs, ranging from 6.7 billion to 9 billion parameters, were put to the test. These models included CodeLlama, CodeQwen, DeepSeek-Coder, DolphinCoder, Granite-Code, Llama 3.1, Qwen2.5-Coder, and Yi-Coder.
The experiments were conducted on a departmental GPU server, but the models were chosen to be runnable on consumer-grade GPUs, highlighting the accessibility of local LLM deployment. The total runtime for these extensive experiments exceeded three weeks, underscoring the scale of the evaluation.
The study evaluated model performance across several dimensions: solution generation speed, correctness (measured by ‘Accepted’ status), and types of failures (e.g., ‘Wrong Answer’, ‘Run Time Error’). The results were also analyzed across different problem difficulties: Easy, Medium, and Hard.
While some models, like Qwen2.5-Coder and Yi-Coder, exhibited slower solution generation times, they paradoxically performed better in terms of acceptance rates. For ‘Easy’ problems, all models showed modest performance, with Qwen2.5-Coder and Yi-Coder leading with 157 and 139 accepted submissions, respectively. However, even at this level, a significant number of attempts resulted in ‘Wrong Answers’ or ‘Run Time Errors’.
As the problem difficulty increased to ‘Medium’, performance deteriorated sharply. Most models achieved fewer than 10 accepted solutions, with only Yi-Coder and Qwen2.5-Coder showing slightly better results (52 and 47 accepted solutions). On ‘Hard’ problems, the limitations became starkly evident, with only Yi-Coder and Llama3.1 managing to produce a single accepted response each, indicating a significant performance ceiling for current local LLMs in high-difficulty scenarios.
When compared to previous benchmarks of cloud-based models, the local LLMs showed a clear performance gap. Gemini 1.5 and ChatGPT-4 previously achieved acceptance rates of 10.9% and 10.7% respectively on a subset of Kattis problems. In contrast, the best-performing local models in this study, Qwen2.5-Coder and Yi-Coder, achieved pass@1 rates of 5.7% and 5.4%. This means they performed at approximately half the acceptance rate of their proprietary counterparts.
Despite this gap, the findings highlight the rapid progress of open models and the practical benefits of an evaluation workflow that organizations can replicate on in-house hardware. The trade-off lies in the ability to run local models an unlimited number of times without incurring token limitations or monetary costs associated with proprietary solutions. The full research paper can be accessed here: Evaluating the Limitations of Local LLMs in Solving Complex Programming Challenges.
Also Read:
- AI Learns to Fix C++ Code: A New Approach to Compilation Repair
- When LLMs Invent: The Challenge of Knowledge Overriding Evidence in Process Modeling
The study concludes by emphasizing the promise and current limitations of locally hosted LLMs for complex code generation. Future directions include fine-tuning these models on task-specific datasets, combining local and cloud-based inference in hybrid workflows, and using smarter prompt designs or built-in debugging steps to improve submission rates. These benchmarks are crucial for guiding model improvements and informing deployment decisions, especially where data privacy or limited computing resources are key concerns. The implications also extend to education, where local code-centric LLMs could power explain-as-you-grade autograders and in-IDE tutoring agents, providing scalable private feedback loops for students.
