TLDR: This research paper introduces AgentBuilder, a no-code prototyping system designed to help a wider range of individuals, beyond AI experts, create and refine user experiences for interface agents. Through studies with developers, the paper identifies key activities and desired capabilities for such systems, highlighting the need for intuitive interfaces, better debugging tools, and clear management of agent interactions and information display.
The world of artificial intelligence is rapidly evolving, with generative AI models powering sophisticated interface agents that can automate tasks based on user commands. These agents are becoming increasingly integrated into our daily digital lives, from navigating websites to managing personal schedules. However, designing the user experience, or ‘agent experience,’ for these AI entities has traditionally been a complex task, often limited to specialized AI engineers.
A recent research paper, AgentBuilder: Exploring Scaffolds for Prototyping User Experiences of Interface Agents, delves into this challenge, proposing a solution that empowers a broader range of individuals—including UX designers, product managers, and even end-users—to contribute to the design and prototyping of agent experiences. Authored by Jenny T. Liang, Titus Barik, Jeffrey Nichols, Eldon Schoop, and Ruijia Cheng, the paper highlights the growing need for accessible tools that can bridge the gap between AI engineering and user-centered design.
The Challenge of Agent Prototyping
Developing interface agents typically demands expertise in AI engineering and knowledge of specialized APIs. This technical barrier means that individuals like Mei, an interaction designer mentioned in the paper, struggle to test different agent behaviors without a formal background in AI. Mei might wonder if an agent should frequently ask for user input, perform a task and summarize, or something in between. Without proper tools, exploring these design ideas is difficult.
Uncovering Design Needs: The Requirements Elicitation Study
To understand what an effective agent prototyping system should offer, the researchers conducted a requirements elicitation study with 12 participants, ranging from AI experts to those with no prior agent familiarity. This study identified five key activities involved in agent experience prototyping:
- Designing the scope and boundaries of the agent.
- Designing the agent’s information display.
- Designing the interactions between the agent and user.
- Running the agent prototype.
- Understanding the agent’s runtime behavior.
From these activities, six desired capabilities for agent prototyping systems were derived, emphasizing the need for no-code interfaces, tools to constrain agent tasks and knowledge, ways to define the agent’s UI, components for user interactions, an environment to run and control the agent, and debugging support for runtime behavior.
AgentBuilder: A No-Code Solution
Based on these requirements, the team developed AgentBuilder, a design probe that serves as a graphical, no-code tool for prototyping agent experiences. AgentBuilder features two main interfaces:
- Prototyping Interface: This is where designers can build agent workflows using a visual editor with nodes and edges, representing actions and conditions. It also includes a Prompt interface for writing natural language instructions, allowing for a blend of visual and textual programming.
- Execution Interface: Implemented as a browser extension, this interface allows users to run their agent prototypes in a real web environment. It includes a chat panel for interaction, visual indicators of agent actions on the webpage, and controls like pause and cancel. Crucially, it also features a Debug Mode to help developers understand the agent’s internal reasoning and inputs during execution.
AgentBuilder allows users to define how an agent appears, what information it displays (e.g., action names, descriptions, reasoning), and how it interacts with users (e.g., asking for confirmation, presenting options in a dropdown menu). It supports bidirectional editing, meaning changes in the visual workflow can update the natural language prompt, and vice-versa.
Insights from the In Situ Prototyping Study
To validate AgentBuilder’s design and gather further insights, an in situ study was conducted with 14 participants. The study confirmed that AgentBuilder enabled individuals of varying technical backgrounds to successfully prototype agent experiences. Participants typically followed an iterative loop: designing an initial prototype, running and monitoring it, debugging any issues, and then refining the agent.
Key findings from this study highlighted several developer needs:
- No-Code Interfaces: While highly valued for visualization and customization, participants desired more granular control over agent behaviors.
- Constraining Agent Knowledge: Users actively considered the agent’s access to personal information, emphasizing privacy concerns.
- Information Display: A tension emerged between the minimal information end-users prefer and the detailed insights developers need for debugging. Systems need to offer mechanisms to help developers make sense of past, present, and anticipated agent behavior.
- Interaction Components: Participants found it challenging to precisely specify granular interaction requirements and verify them without extensive trial and error.
- Runtime Environment: Users often lost track of the agent’s status (running, paused, error) and found it difficult to process multiple sources of runtime information simultaneously.
- Debugging: While the debug mode was useful, raw inputs like the webpage accessibility tree were hard to interpret, and agent reasoning could sometimes be insufficient or even inaccurate.
Also Read:
- Navigating the New Frontier: How Early Adopters Understand Multi-Agent Generative AI
- Simplifying Multi-Agent System Development with LLMs and the AJAN-Editor
Future Directions for Agent Prototyping
The research underscores that prototyping agents presents unique challenges compared to other forms of generative AI, primarily due to the diverse forms of agent inputs and outputs (visual, textual, actions) and the need for developers to build a robust mental model of the agent’s decision-making process. The paper concludes with design implications for future agent prototyping systems:
- Make interaction components easy to understand, use, and customize.
- Allow toggling between different amounts of agent runtime information, catering to both end-user and developer needs.
- Enable comprehensive reviewing and customization of the agent’s runtime, including status indicators and adjustable execution speeds.
This work provides a foundational step towards democratizing AI design, enabling a wider audience to shape the user experiences of interface agents and ensure they are effective, trustworthy, and aligned with human needs.
