TLDR: Researchers have developed A VR-Eval, an automated metric for evaluating interactive multimedia content like games and animations using audio-visual recordings, and A VR-Agent, a multi-agent system that generates JavaScript code for such content. A VR-Agent significantly improves content quality over single-shot generation, especially by selecting the best initial version. However, current AI models struggle to effectively use pre-made assets and audio-visual feedback, unlike human developers, highlighting a key difference in their creative processes.
Artificial intelligence has made incredible strides in generating various forms of content, from text and audio to images and videos. However, creating interactive audio-visual experiences, such as video games and animations, has remained a significant challenge. Traditional large language models (LLMs) can generate simple JavaScript games, but they often lack automated ways to evaluate the quality of the content and struggle with the complexity typically handled by human teams over many months, especially when dealing with multiple agents or rich artistic assets.
To address these limitations, a new research paper introduces two innovative solutions: A VR-Eval and A VR-Agent. This work aims to push the boundaries of autonomous generation and evaluation of interactive multimedia content, particularly focusing on experiences that include audio.
A New Metric for Multimedia Quality: A VR-Eval
The first key contribution is A VR-Eval, a novel evaluation metric designed to assess the quality of multimedia content using Audio-Visual Recordings (A VRs). Instead of evaluating content in isolation, A VR-Eval takes a relative approach, comparing two pieces of content (Content A and Content B) to determine which is superior. The process involves an omni-modal model (capable of processing text, video, and audio) that first describes each content based on its A VR. Following this, a more powerful text model reviews these descriptions and, based on predefined criteria, decides which content is better.
The evaluation criteria for games include description fidelity, visual design, audio quality, behavior correctness, gameplay quality, and AI player quality. For animations, smoothness and creativity are also considered. This multi-step, relative comparison, and review process has been shown to be highly effective at distinguishing functional content from broken or mislabeled content. It also demonstrates a preference for human-made, high-quality content over AI-generated content, indicating its alignment with human preferences.
Automated Content Generation: A VR-Agent
Building upon the A VR-Eval metric, the researchers developed A VR-Agent, a multi-agent system specifically designed for generating JavaScript multimedia content. This framework leverages a vast bank of multimedia assets, including images, audio, and 3D models, along with audio-visual recording feedback and console logs, to iteratively improve generated content.
The A VR-Agent operates in several stages. Initially, a coding agent selects relevant assets from the bank based on the content description. Then, it generates multiple candidate initial codes. A VR-Eval is then used to identify the best of these initial versions. In subsequent stages, the selected content is rendered, an A VR is made, and console logs are extracted. This A VR and the logs are fed to the omni-modal model, which provides feedback. The coding agent then uses this feedback, along with the original description and assets, to iteratively refine and improve the code. This iterative process allows for the generation of more complex and polished content than what is possible with a single-shot generation.
Also Read:
- Unlocking the AI Black Box: A New Framework for Transparent and Personalized Learning
- The Rise of ‘Vibe Coding’: AI Reshapes Software Development Landscape
Experimental Insights and Future Directions
Experiments were conducted on a benchmark of 5 games and 5 animations, using various coding models. The results are compelling: A VR-Agent significantly improves the quality of generated content compared to one-shot generation. A key finding was that selecting the best initial content from multiple candidates (best-of-k) was more beneficial than simply extending the generation with more iterations.
However, the research also revealed a critical gap: current coding models do not seem to effectively leverage high-quality human-made assets or audio-visual feedback. Unlike humans, who greatly benefit from these resources for debugging and improvement, AI models did not show a significant increase in win rate when provided with them. This suggests fundamental differences in how humans and machines approach content creation and debugging.
The study identified Qwen3-Coder-480B and Kimi-K2-1T as the strongest coding models among those tested. While A VR-Eval and A VR-Agent represent significant steps towards automated game design, the authors note that truly impressive feats will require stronger omni-modal models that are also capable of robust coding. The code for this research is publicly available on GitHub: https://github.com/SamsungSAILMontreal/AVR-Eval-Agent.
