Understanding Developer Frustrations with AI Codebase Generators

TLDR: A new study reveals why AI codebase assistants (CBAs) haven’t achieved widespread adoption among developers. Despite their ability to generate entire codebases, users report low satisfaction due to issues like missing functionality, poor code quality, and communication breakdowns. Key challenges include CBAs ignoring existing code context and explicit requirements, producing unusable outputs, and lacking clear usage instructions. Barriers to integration stem from perceived limited capabilities, increased effort overhead, lack of control, minimal time savings, and privacy concerns. The research proposes design improvements focusing on better context handling, transparent planning, self-verification, and guided prompting to enhance usability and integrate CBAs more effectively into developer workflows.

Artificial intelligence has dramatically changed how programmers work, moving beyond simple code suggestions to generating entire codebases from natural language descriptions. These advanced tools, known as Codebase AI Assistants (CBAs), can also extend or adapt existing code, allowing developers to focus on higher-level design. While snippet-level code assistants like GitHub Copilot are widely used, CBAs haven’t seen the same level of adoption, despite their promising capabilities.

A recent study, titled Exploring the Challenges and Opportunities of AI-assisted Codebase Generation, delves into this gap, investigating how developers interact with CBAs and why these tools often fall short of their needs. The research involved a user study and interviews with 16 students and developers, who worked on coding tasks using CBAs like GitHub Copilot and GPT-Engineer.

How Developers Prompt Codebase AI Assistants

The study found that developers use a variety of information in their prompts, including problem descriptions (48% of prompts), required functionality (98%), and code structure (48%). Despite these varied strategies, overall satisfaction with the generated codebases remained low, with a mean score of 2.8 out of 5. Participants often structured their prompts as direct commands, using an imperative tone, and included medium to high levels of detail.

Interestingly, prompts written in a highly imperative style showed a positive correlation with satisfaction, suggesting that CBAs might respond better to clear, directive language. Conversely, prompts that included specific implementation guidance (e.g., “use a PUT route”) had a negative correlation with satisfaction. The prompt writing process itself was iterative, with participants spending significant time writing, editing, and pausing, reflecting different problem-solving phases like planning, exploring, and executing.

Why Developers Are Dissatisfied

The primary reason for dissatisfaction (77% of instances) was issues with functionality, such as missing features or features that didn’t meet expectations. Other significant factors included poor code quality (42%) and communication issues (25%). Executability was also a key concern, with participants expressing frustration when code didn’t run “off the bat.”

Participants also noted issues with code completeness, readability (lack of comments), and maintainability (e.g., duplicate code, divergence from existing code structure). The study highlighted that while some participants adjusted their expectations based on the detail in their prompts, their frustration grew when clear requirements were ignored.

Key Challenges When Using CBAs

The research identified six core challenges developers faced:

• Missing and Blank Code: CBAs frequently generated incomplete code, including references to non-existent functions, unimplemented sections, or even missing entire files.

• Inadequate Communication: Developers often omitted verbally stated requirements in their written prompts, leading to misaligned outputs. Additionally, the generated code often lacked clear variable names, comments, or explanations, making it difficult to understand.

• Ignored Context: In editing tasks, CBAs often failed to consider existing code, leading to poor maintainability and incorrect modifications.

• Usage Instructions: Participants were frequently confused about how to use the generated code, where to place it, or how to integrate jumbled parts into their projects.

• Partially Correct or Unusable Code: Code that only partially worked or was completely incorrect and non-executable was a major source of frustration, often requiring significant additional effort to debug or rewrite.

• Ignored Requirements: In some instances, CBAs directly ignored specifications provided by the participants, leading to missing functionality.

Barriers to Widespread Adoption

Beyond the immediate challenges, developers cited several broader reasons for their reluctance to integrate CBAs into their daily workflows:

• Limited Capability: Many questioned the ability of CBAs to handle complex reasoning, contextual understanding, or integration with existing codebases, often finding them “not as smart as a human.”

• Effort Overhead: Using CBAs often added more effort than it saved, due to the need for extensive debugging, comprehension of poorly structured code, and elaborate prompt writing.

• Lack of Control: The unpredictable nature of AI agents and the ambiguity around what they could or couldn’t do reduced developers’ sense of control over the output.

• No Time Gains: Despite promises of increased productivity, many participants found that the time spent prompting, clarifying, and revising CBA output often offset any time saved.

• Legal and Privacy Concerns: Issues around code licensing and data confidentiality for AI-generated code were also raised as significant barriers.

Also Read:

• Developer Survey Reveals Shifting Landscape of AI Adoption and Trust
• AI-Powered Tools for Navigating Complex Codebases

Opportunities for Future CBA Design

The study also surveyed 21 commercial CBAs to compare their capabilities with the identified challenges and barriers, offering several design opportunities:

• Improved Context Retrieval (R-CTX): Future CBAs should not only retrieve context but also intelligently identify and dynamically use the optimal context for a given prompt.

• Transparent Planning (PL-SHARE): CBAs should explicitly share their implementation plans with users and allow for collaborative planning to give developers more control.

• Enhanced Self-Verification (S-VER): Tools need to not only self-correct errors but also provide clear instructions and affordances for users to verify outputs.

• Controlled Proactive Features (PROACT): While proactive suggestions can be helpful, they should be carefully managed to avoid reducing user control and introducing unintended behavior.

• Clear Change Proposals (PR): CBAs should communicate proposed changes effectively, using mechanisms like “diff” files or pull requests, along with stepwise explanations.

• Teaching and Onboarding (TEACH): Tools should offer interactive walkthroughs and explanations to help users understand the generated code and underlying changes, fostering trust and maintainability.

• Ambiguity Clarification (ASK): CBAs should actively ask clarifying questions when prompts are ambiguous, helping developers refine their requirements and retrieve forgotten details.

The study concludes that a more guided prompting process, where CBAs scaffold prompt development by asking clarifying questions and suggesting refinements, could significantly reduce cognitive load and improve alignment between user expectations and code generation. By addressing these challenges and leveraging these design opportunities, AI codebase assistants can become more reliable, efficient, and truly user-centered tools for software development.