TLDR: A study at ASML investigated the effectiveness of Large Language Models (LLMs) in generating functional and maintainable code for proprietary industrial settings. Researchers developed a custom evaluation framework and introduced a new metric, build@k, to assess code compilation and integration. Key findings indicate that providing examples in prompts significantly improves code quality, code-specific LLMs generally outperform generic ones (though with varying buildability results), and larger models show improved performance with diminishing returns beyond a certain size. The study highlights that ensuring functional correctness remains a primary challenge for LLM-based code generation in complex industrial environments.
Large Language Models (LLMs) have made significant strides in various fields, including generating code. Tools like GitHub Copilot and ChatGPT are already assisting developers in writing and understanding code. However, the true potential and challenges of these AI models in highly specialized, proprietary industrial environments, where code is often complex and interconnected, have remained largely unexplored.
Bridging the Gap: LLMs in Industrial Software
A recent study conducted in collaboration with ASML, a global leader in photolithography systems for semiconductor manufacturing, sheds light on this critical area. The research focused on evaluating LLMs for generating functional and maintainable code within ASML’s leveling department, a highly specialized software environment.
Industrial software systems are far from simple, often featuring legacy components, tightly coupled modules, unique naming conventions, and specialized APIs. These characteristics pose significant hurdles for LLM-based code generation, which are typically trained on broad, open-source data and evaluated on simpler, standalone code snippets.
The ASML Case Study: Automating ‘Garage’ Interfaces
The study specifically investigated the applicability of LLMs in generating ‘garage’ interfaces within ASML’s Data Control and Algorithms (DCA) architecture. These interfaces are crucial for storing and retrieving data, acting as ‘glue code’ that connects various software components. While structurally repetitive, creating these interfaces demands significant manual effort and domain expertise, making it an ideal candidate for automation.
To tackle this, the researchers developed a custom evaluation framework tailored to ASML’s proprietary codebase and introduced a novel metric called build@k. This metric assesses whether LLM-generated code successfully compiles and integrates within real industrial repositories, offering a more realistic measure of usefulness than traditional code similarity metrics.
Key Research Questions Explored:
The study addressed three core questions:
- How do different prompting techniques (e.g., zero-shot, few-shot, chain-of-thought) affect the quality of generated domain-specific code?
- How do generic LLMs compare to code-specific LLMs in generating interdependent code for industrial settings?
- To what extent does the size of an LLM influence its performance in generating domain-specific code?
Insights from the Experiments:
Prompting Techniques: Examples are Key
The research revealed that prompting techniques significantly impact the quality and buildability of generated code. Prompts that included examples (few-shot and chain-of-thought prompting) consistently outperformed zero-shot prompting, which relies solely on a task description. This highlights the importance of providing contextual examples to guide the model in understanding domain-specific patterns. Interestingly, while Chain-of-Thought (CoT) prompting was expected to enhance reasoning, its gains were sometimes marginal, suggesting that a single, well-crafted example might be more effective than multiple reasoning steps in certain scenarios.
Generic vs. Code-Specific LLMs: A Mixed Bag
Code-specific LLMs generally showed better performance in generating code that closely matched reference implementations. For instance, code-specific Gemma and DeepSeek-V2 models demonstrated substantial improvements over their generic counterparts in terms of code similarity. However, this didn’t always translate to buildability. While the code-specific Gemma model significantly improved buildability, the generic DeepSeek-V2 model surprisingly achieved higher build@k scores than its code-specific variant. This finding underscores that code similarity alone isn’t a sufficient indicator of functional correctness in complex industrial environments.
Impact of Model Size: Diminishing Returns
As expected, larger models generally achieved higher similarity scores and generated better quality code, with the 14B and 32B parameter models performing best. However, the improvements tapered off beyond the 14B mark, suggesting diminishing returns on scaling model size. Smaller models, particularly the 0.5B variant, often struggled with functionality and exhibited phenomena like the “repeat curse,” where they generated long lists of imports without actual functional code. While some smaller models (like the 1.5B) showed high build@k scores, manual inspection revealed they often produced non-functional virtual methods, highlighting a limitation of relying solely on buildability as a proxy for correctness.
Also Read:
- Diffusion Models Emerge as a Powerful Alternative for Code Generation
- Boosting Code Security in AI-Generated Software with Fine-Tuning Techniques
Challenges and Future Directions
Despite these advancements, ensuring functional correctness remains a significant challenge, partly due to limited unit test coverage and the tendency of smaller models to generate non-functional boilerplate. The study emphasizes that addressing these limitations is crucial for the reliable deployment of LLM-based code generation in industrial settings.
Future work includes integrating human-in-the-loop assessments by domain experts, broadening unit and system-level test coverage, exploring parameter-efficient fine-tuning techniques, and implementing agentic pipelines for iterative, self-correcting code generation. These directions aim to bridge the gap between research prototypes and industrial deployment of LLM-based code generation.
For a deeper dive into the methodology and detailed results, you can read the full research paper here.
