TLDR: OnGoal is a new chat interface that helps users manage and visualize their conversational goals when interacting with large language models (LLMs). It features a three-stage pipeline to infer, merge, and evaluate goals, providing real-time feedback, explanations, and visual summaries of goal progress. A user study showed OnGoal reduces user effort, increases confidence, and encourages more effective communication strategies, offering key insights for future LLM interface design.
As conversations with large language models (LLMs) become longer and more intricate, users often find it challenging to keep track of their original goals and evaluate the progress of their dialogue. This can lead to frustration, repeated prompts, or even abandoning a conversation entirely. To address this growing problem, researchers have developed OnGoal, an innovative LLM chat interface designed to help users manage and visualize their conversational objectives more effectively.
OnGoal introduces a novel approach by integrating a goal-tracking system directly into the chat interface. This system provides real-time feedback on how well the LLM’s responses align with the user’s stated goals. It offers clear explanations for its evaluations, complete with examples from the LLM’s output, and provides overviews of goal progression throughout the conversation. This empowers users to navigate complex dialogues with greater clarity and control.
The core of OnGoal is its three-stage goal pipeline, which operates independently of the main chat LLM. First, it infers conversational goals from the user’s messages, categorizing them as questions, requests, offers, or suggestions. Next, it merges these new goals with existing ones, either combining similar goals, replacing contradictory ones, or keeping unique objectives. Finally, it evaluates each merged goal against the LLM’s response, determining if the response confirms, contradicts, or ignores the goal. Crucially, it also generates explanations and extracts supporting evidence from the LLM’s text to justify its evaluation.
The interface visualizes this goal data in several intuitive ways. Inline goal glyphs appear under each message, offering a quick summary of inferred and evaluated goals, color-coded for easy understanding (green for confirmed, red for contradicted, yellow for ignored). Clicking on these glyphs reveals detailed explanations, showing how the goal was inferred or why a particular evaluation was given. A dedicated progress panel on the side offers a comprehensive view of goal management. Here, users can see a list of all goals, control them by locking or completing them, and even create their own. A timeline visualization provides a historical overview of how goals have been inferred, merged, and evaluated over time, while an events tab lists all pipeline operations in detail.
For deeper analysis, users can select an individual goal to filter the chat, showing only LLM responses relevant to that specific objective. This view also incorporates text highlighting features. It highlights examples from the LLM’s response that support the goal evaluation and allows users to toggle between highlighting key phrases, similar sentences, and unique sentences. These features help users quickly identify patterns in LLM behavior, such as distractions, topic drift, or consistent progress.
A user study involving 20 participants demonstrated OnGoal’s effectiveness. Participants using OnGoal spent less time and effort on reading and reviewing chat logs compared to those using a baseline interface without goal tracking. They also reported lower mental demand and higher confidence in evaluating their goals. OnGoal encouraged users to adopt new communication strategies, such as iteratively refining prompts based on the system’s feedback, rather than repeatedly sending the same message. The study highlighted that while OnGoal significantly improved goal management, there’s still room for refinement in how goal evaluations are presented to avoid user confusion.
Also Read:
- Rethinking How We Interact with AI: Insights from a Design Thinking Workshop on LLM Interfaces
- Beyond Chatbots: How LLMs Are Learning to Build Interactive User Interfaces
The findings from OnGoal inspire several design implications for future LLM chat interfaces. These include enabling multiple methods for communicating goals, visualizing where goal evaluations align with user focus, designing proactive goal alerts and progress snapshots to further reduce cognitive load, and supporting user feedback on goal evaluations to personalize the system over time. By explicitly tracking and visualizing conversational goals, OnGoal paves the way for more efficient, resilient, and user-friendly interactions with large language models. For more in-depth information, you can read the full research paper here.
