TLDR: Researchers at Harvard Medical School have developed a generative AI system that uses natural language to create interactive, linked visualizations for biomedical data discovery. This prototype helps scientists find relevant datasets more efficiently by progressively building dashboards and allowing flexible filtering through both chat and traditional UI elements. The system leverages a multi-agent architecture and a fine-tuned language model to interpret user queries and generate dynamic, interconnected visualizations across multiple data tables.
Biomedical data discovery is a crucial step for researchers aiming to identify relevant datasets for their scientific inquiries. However, building interfaces that cater to the diverse and evolving needs of all scientists has proven to be a significant challenge. Traditional data portals, despite considerable effort in their development, often fall short in supporting the wide array of specific visualization combinations or interactions required, especially for complex ‘edge cases’ that might hold the key to groundbreaking discoveries.
This challenge highlights a fundamental tension in user interface design: increasing features to cover more use cases often leads to increased complexity and difficulty of use. To address this, natural language interfaces (NLIs) have emerged as a promising solution. These interfaces allow users to express their data exploration goals in plain language, even if they don’t know the exact sequence of actions needed within a complex software environment. Recent advancements in large language models (LLMs) have further fueled a renaissance in integrating LLMs with visualization systems.
Researchers at Harvard Medical School have developed a novel prototype application that combines generative AI with grammar-based visualizations to streamline biomedical data discovery. This system stands out due to several unique features:
Key Innovations of the System
- It interacts with an LLM specifically fine-tuned for generating interactive biomedical metadata visualizations.
- It progressively constructs multi-view visualizations, building an interactive dashboard step-by-step.
- Visualizations are automatically linked through a brushing and linking pattern, allowing for cohesive data exploration.
- The system can generate visualizations across multiple related data tables, providing a comprehensive view.
- Filtering, a critical aspect of data discovery, can be performed through both traditional interactive visualization patterns and the chatbot interface. The chatbot also generates interactive UI widgets, enabling users to adjust or correct filters.
The core of the prototype is a linked multi-view visualization progressively built from natural language queries. The interface features a conversational chat interface with filter widgets and a multi-view visualization panel.
How the System Works: A Multi-Agent Approach
When a user inputs a message into the chat, a multi-agent system processes it. An ‘orchestrator’ agent first determines if the request requires filtering data, creating a visualization, or both. Subsequently, a ‘filter agent’ applies new filters to reduce the dataset, and a ‘visualization agent’ produces a new visualization to be added to the dashboard. The filter agent uses OpenAI’s GPT-4.1 for structured outputs, handling both interval and point filters across different data entities and fields. The visualization agent, on the other hand, uses a fine-tuned model to generate visualization specifications for a custom grammar designed specifically for biomedical metadata visualizations. This grammar supports tabular information and linking visualizations across multiple specifications, offering greater flexibility than existing grammars like Vega-Lite.
The fine-tuning process involved supervised fine-tuning (SFT) on reasoning-and-action traces, teaching the model to decide when to call a visualization tool and integrate its output. The training dataset, DQVis, consists of natural language queries and visualization specifications related to biomedical repository metadata.
The system intelligently interprets the structured outputs from these AI agents. Filter calls lead to data filtering and the display of interactive filter components in both the chat and dashboard, allowing users to adjust the agent’s initial selections. Visualization specifications are rendered as interactive components, with additional logic injected by the system to create linked visualizations that support selections and display data based on the global filter state. Crucially, the system supports linking across multiple entities, meaning a filter applied to donors (e.g., age range) will also filter related biological samples and derived datasets.
Also Read:
- EHR-MCP: Bridging Large Language Models with Electronic Health Records
- AI System Automates Long-Form Video Editing with Natural Language Prompts
A Practical Example
Consider a user wanting to explore donor data. They might start by asking to see all donor data, which generates a tabular representation. Follow-up queries like “How many donors are there for each sex?” or “Show a scatterplot of donor height and weight” progressively add new visualizations to the dashboard. If the user then requests to “filter to adults,” the system applies an age filter (e.g., 18-90 years). If this isn’t precise enough, the user can easily adjust the age range using the generated filter widget (e.g., 21-90 years). Further filtering, such as for “violent death events,” can also be applied and refined, ultimately allowing the user to download specific subsets of data.
This prototype, presented at VIS x GenAI, a workshop co-located with IEEE VIS 2025, represents a significant step towards more intuitive and powerful tools for biomedical data discovery. The full research paper can be accessed here.
