TLDR: A new report by code quality firm Sonar highlights that the unique “personalities” or coding styles of generative AI models significantly influence the quality, security, and maintainability of the code they produce. The study, which evaluated five leading Large Language Models (LLMs) on Java programming tasks, found varying levels of competency and a propensity for generating “code smells” and severe vulnerabilities, underscoring the critical need for developers to understand these model characteristics for safe and effective AI integration.
A recent report titled “The coding personalities of leading LLMs” by Swiss-based code quality company Sonar has shed light on a crucial aspect of generative AI in software development: the distinct “personalities” or coding styles of AI models profoundly affect the quality, security, and maintainability of the code they generate. Published on August 13, 2025, the findings emphasize that while these models offer significant acceleration in coding, their inherent characteristics necessitate human oversight and a deep understanding from developers to mitigate risks.
Sonar’s comprehensive evaluation involved testing five prominent Large Language Models (LLMs): Anthropic’s Claude Sonnet 4 and 3.7, OpenAI’s GPT-4o, Meta’s Llama 3.2 90B, and the open-source OpenCoder-8B. These models were tasked with completing 4,442 Java programming assignments, including benchmarks like MultiPL-E-mbpp-java and ComplexCodeEval. The study aimed to move beyond raw performance metrics to uncover the nuanced ways each model approaches code generation.
The report’s overarching conclusion aligns with the broader industry perspective: generative AI models possess both strengths and weaknesses, making them valuable tools when complemented by human review and supervision. However, the study revealed significant disparities in competency. On benchmarks such as HumanEval, scores ranged from a high of 95.57 percent for Claude Sonnet 4, indicating its strong capability in producing valid, executable code, down to 61.64 percent for Llama 3.2 90B.
A critical finding was the models’ tendency to introduce “code smells”—design patterns that signal deeper underlying problems in the code structure, making it messy, complex, and difficult to maintain. Furthermore, the report highlighted a concerning propensity for generating severe security vulnerabilities. For instance, the models were found to produce hard-coded secrets like passwords, a flaw often inherited from their training data. They also frequently allowed resource leaks due to a lack of contextual awareness regarding software engineering principles, such as failing to close file streams.
The severity of these vulnerabilities was particularly alarming. The report categorized them on a scale including “Blocker,” “Critical,” “Major,” and “Minor.” Llama 3.2 90B, for example, generated over 70 percent of its vulnerabilities at the “Blocker” severity, meaning bugs severe enough to crash an application. GPT-4o followed with 62.5 percent of its vulnerabilities reaching this critical level, and Claude Sonnet 4, despite its high competency in generating executable code, still produced almost 60 percent of its vulnerable code at the “Blocker” severity.
Tariq Shaukat, CEO of Sonar, underscored the importance of this understanding, stating, “To really get the most from them, it is crucial to look beyond raw performance to truly understand the full mosaic of a model’s capabilities.” He added, “Understanding the unique personality of each model, and where they have strengths but also are likely to make mistakes, can ensure each model is used safely and securely.”
Also Read:
- AI Transforms Software Development: From Code Suggestions to Autonomous Agents
- Generative Engineering: Revolutionizing Software Development for Business Impact
The report advocates for developers to recognize these distinct “archetypes” or “personalities” of AI models. By understanding where each model excels and where it is prone to errors, developers can more effectively integrate AI-generated code into their workflows, ensuring it meets stringent standards for quality, security, and maintainability. This proactive approach is vital as AI continues to accelerate the volume of code being produced, making robust quality assurance more critical than ever.
