TLDR: A new research paper evaluates the capabilities of LLMs like GPT-4o, Gemini, and Claude in generating and understanding data visualizations. The study found that while LLMs are proficient at creating simpler charts using Python, they struggle with complex visualizations and Vega-lite scripts. For understanding charts, LLMs perform well on basic questions but face difficulties with subtle visual distinctions and specific chart types like line graphs, though system prompts can improve performance.
Information visualization is crucial for making sense of the vast and complex data we encounter daily. Traditionally, creating these visualizations requires specialized tools, programming libraries, or scripts, which can be a barrier for those without a strong background in data science or programming. This is where Large Language Models (LLMs) come into play, offering a promising avenue to automate and simplify the process of generating and understanding visualizations through natural language.
A recent research paper, titled “Evaluating LLMs for Visualization Generation and Understanding,” delves into the capabilities of popular LLMs in these two critical areas. Authored by Saadiq Rauf Khan, Vinit Chandak, and Sougata Mukherjea from the Indian Institute of Technology, Delhi, this study provides valuable insights into how well current LLMs perform and where their limitations lie. You can find the full paper here: Research Paper.
LLMs and Visualization Generation
The researchers evaluated four prominent LLMs—OpenAI’s GPT-3.5 and GPT-4o, Google’s Gemini-1.5-pro, and Anthropic’s Claude 3 Opus—on their ability to generate visualization code. The evaluation covered 24 different chart types, ranging from simple bar and pie charts to more complex ones like violin plots and choropleths. They tested two primary methods: generating Python code (using libraries like Matplotlib) and generating Vega-lite scripts.
For generating Python code with default configurations, GPT-4o emerged as the top performer, successfully creating about 95% of the charts. GPT-3.5 followed with 79%, while Gemini and Claude performed similarly at 75% and 70% respectively. The study noted that LLMs generally excelled at simpler visualizations but struggled with less common or more complex types. For instance, only GPT-4o could correctly generate a bullet chart, and Gemini and Claude often produced incorrect charts when asked for a range plot, instead generating violin or box plots.
When it came to generating Python code for visually modified charts (e.g., changing bar width, line colors, or point shapes), GPT-4o maintained its lead with 92% accuracy. GPT-3.5 and Gemini also performed well, achieving 88% and 84% accuracy. However, some models still showed quirks; Gemini, for example, tried to compress the entire chart width instead of reducing individual bar widths when prompted.
The most significant challenge for LLMs in visualization generation was creating Vega-lite scripts. Performance dropped considerably across the board. GPT-4o’s accuracy fell to 70%, and Gemini’s plummeted to a mere 24%. This suggests that while LLMs are proficient with established programming libraries like Matplotlib, they find it much harder to work with declarative visualization grammars like Vega-lite, often producing incorrect or blank charts.
LLMs and Visualization Understanding
Beyond generation, the paper also explored the LLMs’ capacity to understand existing visualizations and answer questions about them. Using the FigureQA dataset, which features common charts like bar, line, and pie charts with binary (yes/no) questions, the researchers initially found GPT-4o to be the best performer (66.02% accuracy), followed by Gemini (64.31%) and Claude (54.61%).
However, the researchers recognized the limitation of binary questions, which can be susceptible to random guessing. To gain a deeper understanding, they conducted a manual analysis using custom, non-binary questions on a subset of charts. This manual evaluation revealed more nuanced insights:
- All models struggled with identifying relationships between elements with close boundaries or determining the exact lengths of shapes.
- Claude 3 Opus sometimes hallucinated information not present in the charts and often used its own color names instead of the ones provided.
- Gemini-1.5-Pro occasionally failed to recognize dotted lines, especially when mixed with non-dotted lines.
- Line charts proved to be the most challenging for all models, with significantly lower accuracy compared to bar or pie charts. This might be due to dotted lines being misinterpreted as noise.
- The use of a simple system prompt, such as “Analyse the following chart carefully and answer the following questions correctly,” generally improved the performance of Gemini-1.5-Pro and GPT-4o, highlighting the importance of clear instructions.
Also Read:
- Evaluating AI’s Data Science Prowess: A New Benchmark for Real-World Tasks
- ChartM3: Enhancing Chart Editing Through Multimodal Instructions
Key Takeaways
The study concludes that while LLMs show great promise in automating visualization tasks, they still have significant limitations. They are quite capable of generating simpler charts and performing basic customizations, especially with Python libraries. However, their ability to handle complex visualizations, provide non-trivial customizations, or work with less common visualization grammars like Vega-lite is still developing.
In terms of understanding, LLMs can answer basic questions but struggle with nuanced interpretations, particularly when visual elements are very similar or when dealing with specific chart features like dotted lines. The research suggests that future improvements in LLMs, possibly through advanced prompting techniques or expanded training datasets, could address these limitations, paving the way for more sophisticated and reliable AI-driven visualization systems.
