Navigating Code Hallucinations: An Automotive Deep Dive into LLM Reliability

TLDR: This research paper investigates the phenomenon of hallucinations in Large Language Models (LLMs) used for code generation, specifically within the automotive domain. It presents a case study evaluating models like GPT-4o, GPT-4.1, and Codex across varying prompting complexities (baseline, signal-augmented, and template-augmented) for a task involving COVESA Vehicle Signal Specifications. The study identifies common hallucination types, including syntax violations, invalid reference errors, and API knowledge conflicts. It demonstrates that while iterative refinement helps, a correct solution was only achieved with the most context-rich, template-augmented prompts for GPT-4o and GPT-4.1, highlighting the need for effective mitigation strategies and structured prompting to ensure reliable LLM-generated code in safety-critical systems.

Large Language Models (LLMs) are transforming how we generate code, offering exciting possibilities across various software engineering fields. However, their widespread adoption is hindered by a significant challenge: hallucinations. These are outputs that seem correct but are factually wrong, unverifiable, or simply nonsensical. This issue is particularly critical in code generation, where even minor errors can lead to serious bugs, security vulnerabilities, or system failures, especially in safety-critical areas like automotive software.

Understanding Hallucinations in Code Generation

Hallucinations in LLMs stem from their statistical nature; they predict the most likely next token based on patterns in vast datasets, rather than truly understanding the meaning or factual basis of the content. When faced with ambiguity or a lack of information, LLMs might fill in gaps with plausible but incorrect details. In code generation, these errors can be categorized into several types:

• Syntactic hallucinations: These cause compilation failures, like missing brackets or incorrect indentation, or result in incomplete code.
• Runtime execution hallucinations: Code that compiles but fails during execution, often due to misusing libraries or APIs (e.g., incorrect function signatures) or referencing undefined elements.
• Functional correctness hallucinations: Code that runs without errors but doesn’t meet the specified requirements, such as flawed logic or incorrect calculations.
• Code quality hallucinations: Issues related to resource management, security vulnerabilities, or poorly structured code that makes maintenance difficult.

These issues often arise from problems with training data (quantity or quality), the models themselves (limited reasoning, non-deterministic output), or unclear user prompts.

Strategies to Combat Hallucinations

Addressing hallucinations is complex, with no single solution fitting all scenarios. Researchers are developing various strategies:

• De-Hallucinator: This approach pre-analyzes and indexes project source code. When a user provides a prompt, relevant APIs are retrieved to enhance the context for the LLM, reducing function misuse.
• ClarifyGPT: Aims to reduce hallucinations from ambiguous prompts by generating diverse test inputs and checking code consistency. If ambiguity is detected, it asks clarifying questions to the user to refine the prompt.
• Refining ChatGPT-Generated Code: This framework focuses on improving code quality by prompting the LLM to refine its generated code based on feedback, which can range from simple notifications of quality issues to detailed static analysis and runtime error reports. This process can be iterative until issues are resolved.

An Automotive Case Study: Wipers and Hoods

To better understand the extent of hallucinations, especially in unfamiliar domains, a study was conducted focusing on an automotive task: generating Python code to turn off windshield wipers when the vehicle’s hood is open. This task utilized the COVESA Vehicle Signal Specification (VSS), a standardized set of signals for vehicle components.

The study evaluated several LLMs, including GPT-4o, GPT-4.1, and Codex, using three different prompting strategies:

1. Baseline prompt: A simple, one-line instruction.
2. Signal-augmented prompt: The baseline prompt plus a list of 20 potentially relevant VSS signals.
3. Template-augmented prompt: The signal-augmented prompt combined with a code skeleton containing ‘TODO’ sections for the LLM to fill in.

The ‘Refining ChatGPT-generated code’ technique was applied iteratively to improve the generated code.

Key Findings from the Case Study

The results highlighted the impact of prompt complexity on hallucination frequency and severity:

• Baseline Prompt: Models frequently produced syntax violations, invalid reference errors, and API knowledge conflicts. Notably, they often hallucinated VSS signals that didn’t exist, indicating a significant reliability issue in unfamiliar domains.
• Signal-Augmented Prompt: Scores improved, and the hallucination of non-existent VSS signals disappeared. However, models still struggled to differentiate between similar signals, often using an incorrect but similar VSS signal.
• Template-Augmented Prompt: This strategy yielded the best results. Initial scores were significantly higher, with only API knowledge conflicts remaining. GPT-4o and GPT-4.1 successfully produced correct solutions after a few refinement iterations. This demonstrates that providing a structured starting point (code template) and a constrained set of signals, combined with iterative feedback, can lead to a complete and correct solution. However, Codex was unable to produce a correct solution even with this advanced prompting, eventually declaring the task unsolvable given the provided signals.

The study underscores that while iterative feedback can improve code quality, its effectiveness depends heavily on the initial prompt structure and the model’s capabilities. Offline code generation models, in particular, struggled significantly with domain-specific tasks they weren’t explicitly trained on, even with context-rich prompts.

Also Read:

• AI’s Hidden Flaws: Uncovering Cognitive Biases in General-Purpose AI for Software Engineering
• Unpacking Prompt Sensitivity: A Deep Dive into LLM Robustness

Conclusion and Future Outlook

This automotive case study reveals that different types of hallucinations in code generation require tailored mitigation strategies. While syntax errors can often be resolved through iterative feedback, other issues like invalid references and API knowledge conflicts persist, especially with simpler prompts. The most critical hallucinations involved generating non-existent signals or misidentifying correct ones among similar options. The research emphasizes the need for more targeted research to close the remaining gaps in understanding and mitigating these challenges, especially in safety-critical applications. For more details, you can read the full research paper here.