TLDR: INTERACTSCIENCE is a new benchmark designed to evaluate Large Language Models (LLMs) on their ability to generate interactive scientific demonstration code. It uses a hybrid framework combining programmatic functional testing for interaction logic and visually-grounded qualitative testing for visual accuracy against reference snapshots. Evaluations of 30 LLMs reveal that while models can create interactive UI components, they struggle to integrate accurate scientific knowledge into functional and visually correct demonstrations, highlighting a significant challenge in advanced AI code generation for science and education.
Large Language Models (LLMs) are transforming how we create software, moving towards a future where we simply describe what we want, and the AI builds it. This is particularly exciting for science and education, where interactive demonstrations can bring complex concepts to life. Imagine asking an AI to create a tool that visualizes protein folding or simulates planetary orbits, and it delivers a fully functional application.
However, building these interactive scientific demonstrations is a complex task. It requires LLMs to not only possess accurate scientific knowledge but also to implement interactive front-end code that works correctly and responds to user actions. Current evaluation methods often fall short, either focusing on knowledge without code or static web code without interactivity. This highlights a significant gap: LLMs can perform individual tasks well, but struggle to combine them effectively for interactive scientific applications.
Introducing INTERACTSCIENCE
To address this challenge, a new benchmark called INTERACTSCIENCE has been developed. This benchmark aims to rigorously evaluate LLMs’ ability to generate interactive scientific demonstration code. It uses a unique hybrid evaluation framework that combines two key components:
-
Programmatic Functional Testing (PFT): This component uses automated unit tests to verify the interaction logic. It ensures that when a user performs an action, like clicking a button or moving a slider, the system behaves exactly as intended.
-
Visually-Grounded Qualitative Testing (VQT): This part assesses the visual correctness and quality of the generated demonstration. It compares the AI-generated visuals against reference snapshots, using advanced Vision-Language Models (VLMs) as judges, guided by specific checklists to ensure scientific accuracy.
The INTERACTSCIENCE benchmark includes a substantial set of carefully designed questions across five scientific domains: mathematics, physics, chemistry, earth science, and computer science. Each problem comes with unit tests, reference snapshots, and checklists, providing a comprehensive evaluation suite.
Key Findings from the Evaluation
The researchers evaluated 30 leading open- and closed-source LLMs on INTERACTSCIENCE. The results revealed ongoing weaknesses in how these models integrate domain knowledge with interactive front-end coding. While models often achieve high scores in basic interactivity (meaning they can generate UI components that respond to actions), their ability to connect these interactions with correct scientific logic and visualizations remains a significant hurdle. For instance, a model might create a visually plausible interface, but the underlying physics or mathematical calculations could be incorrect.
Closed-source models generally performed better, especially in functional correctness, with models like Claude-Sonnet-4 and GPT-5 showing stronger adherence to complex implementation plans. Among open-source models, larger models tended to perform better, emphasizing the importance of scale for this type of complex code generation.
The study also found that the difficulty of the problem, particularly in terms of visual complexity, significantly impacted performance. Disciplines requiring complex animations or 3D effects, like Physics and Earth Science, proved more challenging for LLMs to render accurately compared to those with simpler, static visuals like Mathematics.
Interestingly, providing multimodal LLMs with reference snapshots as input offered modest improvements in performance, suggesting that visual context can help, but the effect is not always uniform and can sometimes even introduce confusion depending on the model.
Also Read:
- BIGCODEARENA: Elevating Code Generation Evaluation Through Execution
- Clarifying User Intent in Data Visualization Code Generation Through Dialogue
The Path Forward
INTERACTSCIENCE is the first benchmark to automatically measure this combined capability with realistic interactive operations. It provides a crucial foundation for advancing the development of reliable and educationally useful scientific demonstration code generation. While the current dataset has some limitations in size and expert verification, it opens doors for future work, including expanding the benchmark, incorporating broader expert validation, and exploring more flexible agent-driven testing approaches.
This research highlights that while LLMs are incredibly powerful, there’s still a journey ahead to enable them to seamlessly blend scientific reasoning with interactive web development. For more details, you can read the full paper here.
