TLDR: ComboBench is a new benchmark that evaluates how Large Language Models (LLMs) translate high-level commands into precise VR device manipulations across four popular VR games. The study found that while top LLMs like Gemini-1.5-Pro are strong at task decomposition, they significantly struggle with procedural reasoning, spatial understanding, and motor action mapping compared to humans. Few-shot examples improve performance, particularly for procedural tasks, but current LLMs still lack the embodied understanding crucial for seamless VR interaction, highlighting the need for multimodal AI training approaches.
Virtual Reality (VR) games offer immersive experiences, but they demand a unique skill: translating complex, high-level intentions into precise physical actions using controllers and head-mounted displays. While humans perform this intuitively, leveraging common sense and embodied understanding, the ability of Large Language Models (LLMs) to replicate this intricate process has been largely unexplored until now.
A new benchmark, called ComboBench, has been introduced to rigorously evaluate how well LLMs can bridge this gap. This research delves into whether these advanced AI models can effectively manipulate virtual reality devices to play games, much like a human player would. The study, conducted by researchers including Shuqing Li, Jiayi Yan, Chenyu Niu, Jen-tse Huang, Yun Peng, Wenxuan Wang, Yepang Liu, and Michael R. Lyu, provides a comprehensive look at the current state of LLM capabilities in this challenging domain.
Introducing ComboBench
ComboBench is the first benchmark specifically designed to assess LLMs’ fine-grained cognitive abilities in translating high-level semantic actions into VR device manipulations. It comprises 262 diverse scenarios drawn from four popular VR games: Half-Life: Alyx, Into the Radius, Moss: Book II, and Vivecraft (Minecraft in VR). For each scenario, human experts meticulously annotated the exact sequence of VR device manipulations required to achieve the semantic goal, providing a ‘ground truth’ for comparison.
The benchmark evaluates LLMs across six key cognitive abilities crucial for VR interaction:
- Task decomposition: Breaking down a complex goal into smaller, manageable steps.
- Procedural reasoning: Understanding the correct order and dependencies of actions.
- Spatial reasoning & contextual awareness: Interpreting instructions within a 3D environment.
- Object interaction & tool use understanding: Knowing how to use virtual objects and tools.
- Motor action mapping & VR procedural transfer: Translating abstract actions into specific controller commands.
- Judgment of termination/continuation conditions: Knowing when a task is complete or needs repetition.
Seven state-of-the-art LLMs were put to the test: GPT-3.5, GPT-4, GPT-4o, Gemini-1.5-Pro, LLaMA-3-8B, Mixtral-8x7B, and GLM-4-Flash. Their performance was measured using a multi-dimensional scoring approach that assessed semantic action understanding, procedural step correctness, and device-specific manipulation accuracy.
Key Findings: Strengths and Struggles
The evaluation revealed a nuanced picture of LLM capabilities:
- Task Decomposition: LLMs generally excel at breaking down high-level tasks into component steps, with top models like Gemini-1.5-Pro achieving strong scores in this area. This suggests that their training on hierarchical text structures translates well to understanding sequential goals.
- Procedural Reasoning and Motor Action Mapping: These areas proved to be the most significant challenges for all models. LLMs struggled to maintain the correct temporal order of actions (low Sequential Order Preservation scores) and to precisely translate abstract actions into specific VR controller movements (motor action mapping). Even the best models fell short of robust capability here.
- Spatial Understanding: Compared to humans, LLMs also showed pronounced weaknesses in spatial reasoning, particularly in games requiring complex 3D navigation and object manipulation.
- Game-Specific Performance: Performance varied significantly across games. Models performed best in games with consistent, discrete interaction patterns, like Vivecraft. Conversely, games featuring realistic physics simulations, complex inventory management, and nuanced controller manipulations, such as Into the Radius and Half-Life: Alyx, posed much greater difficulties.
The study also highlighted the impact of ‘few-shot examples’ – providing the LLM with a small number of demonstrations. Few-shot examples dramatically improved performance, especially for procedural understanding, with Sequential Order Preservation scores increasing by 10-20 times from near-zero baselines. This indicates that while LLMs can learn from examples, there are diminishing returns, suggesting fundamental architectural limitations rather than just a lack of exposure.
Comparing AI to Human Performance
When compared to human players, LLMs showed competitive performance in identifying relevant steps (Normalized Step Alignment Score). However, humans maintained a decisive advantage in Sequential Order Preservation, spatial reasoning, and the judgment of termination conditions. This suggests that while LLMs can understand the ‘what’ of VR interactions, they still struggle with the ‘how’ and ‘when’ that comes naturally with embodied experience.
The research paper, available at https://arxiv.org/pdf/2510.24706, concludes that current text-trained LLMs lack the embodied understanding necessary for reliable VR interaction. This points to a critical need for future AI systems to incorporate multimodal training approaches that integrate spatial, visual, and haptic information, moving beyond purely text-based learning.
Also Read:
- Butter-Bench: A New Benchmark Reveals LLMs Struggle with Practical Robot Intelligence
- Unpacking LLM Long-Context Abilities: Insights from the LooGLE v2 Benchmark
Future Implications
The findings from ComboBench have significant implications. On one hand, LLMs capable of reasoning about VR interactions could revolutionize accessibility for users with motor impairments, enable more intuitive natural language interfaces for VR applications, and accelerate the development of intelligent training systems. On the other hand, as LLMs gain greater control over virtual (and potentially physical) systems, critical questions arise regarding safety, security, user autonomy, and privacy. The path forward requires responsible development practices that prioritize these concerns while pushing the boundaries of VR-capable AI.
