TLDR: A new research paper explores how Large Language Models (LLMs) can assist in ‘visualization retargeting’ – adapting existing charts to new datasets. It categorizes challenges into syntax, semantics, and pragmatics, evaluating two LLM approaches. The study found that both LLM methods struggle with data transformations and that a structured pipeline can introduce more errors. Recommendations include mixed-initiative assistance, surfacing data dependencies, and improved transformation support to enhance future systems.
Adapting existing data visualizations to new datasets can be a surprisingly complex and time-consuming task. Even with a wealth of examples available online, making an existing chart work with your own data often requires deep familiarity with the chart’s original code and an understanding of how your new data needs to be prepared. This process, known as visualization retargeting, is the focus of a new research paper titled “Challenges & Opportunities with LLM-Assisted Visualization Retargeting.”
The paper explores how Large Language Models (LLMs), like the AI models behind advanced chatbots, can help automate this tedious process. Traditionally, retargeting has focused on restyling images of charts, but LLMs offer the potential to work directly with the chart’s underlying code. This means they can not only change the visual style but also handle complex data transformations, map new data fields to visual elements, and update titles and labels.
The researchers identified three main categories of challenges in visualization retargeting: syntax, semantics, and pragmatics.
Syntax Challenges
Syntax refers to the rules of the code. When you change a dataset, you often need to update variable names, remove old data transformations, or adjust how data fields are encoded. For example, if a chart originally showed monthly temperature and you want to adapt it to show unemployment rates over time, you’d need to change the data source, update field names like ‘Month’ to ‘date’, and remove any temperature-specific calculations. The paper highlights that LLMs can help generate syntactically correct code for different charting tools, reducing the ‘gulf of execution’ for designers.
Semantic Challenges
Semantics deals with the meaning of the data. This involves deciding how new data fields should be mapped to the chart’s visual elements. For instance, mapping ‘unemployment’ to a color scale and ‘date’ to the x-axis makes semantic sense for showing changes over time. It also involves refining scales (like the range of values on an axis) and transformations based on the new data’s characteristics. Updating axis labels and chart titles to reflect the new data’s context is also a key semantic task.
Also Read:
- AI Models Streamline Metadata Creation for Data Catalogs
- The Coding Triangle: A New Lens on AI’s Programming Abilities
Pragmatic Challenges
Pragmatics considers whether the visualization effectively conveys information to its audience and looks aesthetically pleasing. This includes decisions like whether a legend is needed, ensuring enough spacing to prevent elements from overlapping, and how labels are formatted. While the paper primarily focuses on LLM assistance for syntax and semantics, it acknowledges that pragmatic decisions are more challenging to automate due to their dependence on external factors like audience and communication goals.
To evaluate LLM assistance, the researchers tested two approaches: a baseline where the LLM received minimal context (just the chart code and new dataset summary), and a more structured ‘program synthesis pipeline’. The pipeline aimed to guide the LLM by providing detailed information about the chart’s structure, like visual encodings and transformations, in three stages: syntax parsing, data mapping, and specification refinement.
The results showed that while the baseline approach rendered more charts successfully, both methods struggled significantly when the new data hadn’t been appropriately transformed beforehand. The pipeline, despite its structured approach, actually generated more syntactic errors, possibly because errors in early stages propagated through the process. More complex charts and datasets, whether very simple or very complex, also led to more errors.
Based on their findings, the researchers offer several design recommendations for future LLM-assisted retargeting systems. They suggest enabling ‘mixed-initiative assistance’ where users can interact with the LLM at key points to correct errors and steer the process. Surfacing data dependencies (how different parts of the code relate to each other) could help designers inspect and refine LLM inferences. Finally, integrating better transformation support, perhaps through user-guided heuristics or flags, is crucial to prevent common issues like overplotting.
This research underscores that while LLMs hold great promise for simplifying visualization retargeting, there’s still a need for intelligent interfaces that balance automation with human oversight. For more details, you can read the full research paper here.
