TLDR: Researchers developed a new methodology for using Large Language Models (LLMs) to generate code for novel scientific algorithms. Instead of directly asking for code, they found it more effective to iteratively refine problem specifications with the LLM before code generation. This approach, integrated with their Code-Scribe tool, improved code quality and developer productivity, especially for complex tasks like particle-mesh interactions in astrophysics simulations, by reducing model hallucinations and fostering clearer communication.
The rapid advancement of large language models (LLMs) has sparked immense interest in automating coding tasks. While many studies focus on LLMs generating code for existing patterns, a less explored area is their ability to write code for entirely new algorithms, where no prior examples exist in their training data. This challenge arises because natural language prompts, inherently imprecise, can lead to model hallucinations if specifications are not perfectly clear.
Researchers Anshu Dubey and Akash Dhruv from Argonne National Laboratory have introduced a novel methodology to address this. Their approach significantly reduces model hallucination by focusing on iterative interaction with the LLM on *specifications* rather than directly on code. This means repeatedly asking the model to articulate its understanding of the task and correcting it in natural language, making it easier to catch and fix incomplete or erroneous specifications.
Enhancing Code-Scribe for New Code Generation
To facilitate this methodology, the team enhanced Code-Scribe, an LLM-assisted tool originally designed for translating large legacy Fortran code to C++. Code-Scribe now supports new code generation by enabling chat-completion through easy-to-read and modify TOML files. This allows for a structured and maintainable way to interact with LLMs.
The methodology was developed in the context of a new technique for handling interactions between Cartesian meshes and Lagrangian particles in Flash-X, a multiphysics software system used for simulations across various scientific domains. The specific problem involved optimizing how particles deposit density onto a mesh, aiming to eliminate a costly ‘reverse ghost-cell fill’ step in adaptive mesh refinement (AMR) simulations.
The New Algorithm: Virtual Particles
The proposed algorithm introduces ‘virtual copies’ of particles located near block boundaries. Instead of a reverse ghost-cell fill, these virtual particles are moved to remote processors and deposit directly into their destination cells, then destroyed. This approach aims to reduce communication overhead, especially in AMR where irregular communications can be very expensive.
A Test-Driven Development Approach with LLMs
The researchers adopted test-driven development (TDD) principles throughout the project to ensure robust verification. They started with a simplified mesh for testing and generated helper functions using LLM assistance. The overall coding effort was broken down into smaller, verifiable tasks, such as generating mirroring positions for virtual particles, handling various boundary conditions (periodic, outflow, reflective), and assigning correct attributes to particles for deposition.
Iterative Prompt Engineering and Model Performance
Early attempts with over-specified prompts proved counterproductive, leading to hallucinations. The breakthrough came from changing the interaction mode: instead of asking for code, the models were first asked to *explain their understanding* of the problem without writing any code. This natural language interaction, interspersed with mathematical notations, allowed for much easier iteration and refinement of specifications until they were precise enough.
In their experiments, ChatGPT significantly outperformed other models like Codellama and Kimi in this specification-generation mode, demonstrating a higher level of reasoning sophistication. For models that struggled with generating specifications, breaking the problem into simpler, sequential tasks (e.g., first creating all mirrors, then applying boundary conditions) proved more successful.
Also Read:
- Navigating CI/CD Configuration Changes with Large Language Models
- AI Streamlines Code Migration with Change Diffs
Integrating with Code-Scribe for Seamless Development
The final stage involved integrating this methodology into Code-Scribe. By iteratively building a TOML file with user queries and model responses, they created a chat completion template that consistently produced correct and functional code across all tested models. Code-Scribe was further enhanced to automatically extract and integrate the generated code into Flash-X source files, streamlining the development process.
This research highlights that while LLMs are powerful tools for code generation, the key to their effective use for novel algorithms lies in a specification-first, iterative approach. This not only improves developer productivity and code quality but also encourages better software engineering practices like rigorous verification, modular design, and comprehensive documentation. You can read the full paper here: Adding New Capability in Existing Scientific Application with LLM Assistance.
