TLDR: A user study with 36 participants analyzed how different human-LLM interaction features impact code generation productivity with ChatGPT. The study introduced new project-level coding benchmarks and found that prompting patterns (Few-Shot, Reflection) and allocating more time to debugging significantly boost productivity. It also identified a taxonomy of 29 runtime and logic errors, categorizing them by user- and model-originated causes, offering mitigation strategies. The research suggests that effective prompting strategies are more critical for productivity than simply using more advanced LLM models.
Large Language Models (LLMs) like ChatGPT are increasingly being used in software development. While their ability to boost productivity is recognized, understanding how developers can interact with them most effectively for optimal results remains a key challenge. Many existing studies have focused on simpler, function-level coding tasks and haven’t fully explored the complexities of real-world software projects or the nuances of human-LLM interactions.
A recent study titled Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Tasks, conducted by Sangwon Hyun, Hyunjun Kim, Jinhyuk Jang, Hyojin Choi, and M. Ali Babar, aimed to bridge these gaps. The researchers designed a comprehensive experiment to analyze how different features of Human-LLM Interactions (HLIs) impact code generation productivity, extending beyond simple function-level evaluations to include complex project-level tasks.
Understanding the Experiment
The study involved 36 participants with diverse backgrounds, who were asked to solve coding tasks by interacting with a GPT assistant using specific prompting patterns. The researchers meticulously analyzed screen recordings and chat logs to understand participants’ experiences and behaviors. They introduced two new project-level benchmark tasks that involve multi-class dependencies, reflecting real-world software engineering challenges more accurately than previous benchmarks.
Key Findings on Productivity
The statistical and empirical investigation revealed several crucial insights:
- Three out of 15 analyzed HLI features significantly impacted code generation productivity: the prompting pattern used, the time distribution between initial implementation and debugging, and the user’s algorithm-solving experience.
- Specifically, the Few-Shot prompting pattern was found to be highly productive for both function-level and project-level tasks. The Reflection pattern also showed strong productivity in project-level scenarios and was highly rated by participants.
- A significant finding was the importance of time allocation: participants who dedicated more time to debugging the generated code than to initial implementation achieved higher productivity. This suggests that even with LLMs, iterative refinement is crucial, as initial outputs often contain faults.
- The study also highlighted that using test case codes as context during debugging led to more accurate fixes compared to relying solely on design documents.
- Interestingly, while Paid GPT models generally showed higher average productivity, the statistical analysis did not find a significant difference in overall productivity compared to Free models. This emphasizes that effective prompting strategies can be more influential than just using a more advanced model.
Errors in Human-LLM Interactions
The research also delved into the types of errors that occur during HLI processes for code generation. They identified a taxonomy of 29 categories of runtime and logic errors. These errors stemmed from various sources:
- User-Originated Errors: These often resulted from users providing insufficient context, making minor mistakes, or misunderstanding design document content. For example, omitting object initialization or providing incorrect responses to GPT’s clarifying questions could lead to errors.
- Model-Originated Errors: Even with proper context, GPTs sometimes produced erroneous code. This included missing required attributes, incorrect parameter types, or issues with developing accurate conditions (e.g., in string processing or regex). A notable issue was GPTs struggling with multi-class dependencies, sometimes generating ad-hoc objects instead of correctly linking to previously defined classes.
- Errors During Debugging: During the code-fixing phase, GPTs could introduce new issues, such as unexpectedly removing attributes, changing parameter orders, or altering function names. Multi-class dependency issues also appeared here, with GPTs sometimes moving functions between classes or creating redundant ones.
The study suggests that providing sufficient and clear context, especially for multi-class dependencies, is vital. While manual context formulation can be beneficial for clarifying understanding, it should ideally be combined with copying original specifications to avoid hallucinations due to missing details. Users should also be vigilant during debugging, carefully reviewing changes to prevent new errors or code overlaps.
Also Read:
- Decoding Student Habits: How CS Students Leverage AI and Online Resources for Coding
- Unraveling Why AI Reasoning Models Struggle with Complex Multi-Hop Questions
Implications for Software Engineering
This research offers valuable guidelines for developers, especially students and early-career practitioners, on how to interact more productively with GPT assistants for coding tasks. It underscores the importance of explicit requirement integration and a focused effort on debugging. For researchers, the newly introduced project-level benchmarks and the detailed error taxonomy provide a robust foundation for developing and evaluating new prompting paradigms that can better address the complexities of code generation and debugging with LLMs.
