TLDR: BrowserArena is a novel live evaluation platform for Large Language Model (LLM) agents on open-web navigation tasks. It uses user-submitted tasks, pairwise comparisons, and step-level human feedback to assess agent performance and identify common failure modes. The study found that DeepSeek-R1 performed well despite lacking multimodal capabilities, and identified three key failure modes: captcha resolution, pop-up banner removal, and direct URL navigation. The research highlights the diversity and brittleness of current web agents and provides a new methodology for understanding their limitations at scale.
Large Language Models (LLMs) are increasingly capable of navigating the open web, acting as agents to complete complex tasks. However, evaluating these agents effectively has been a significant challenge. Traditional evaluation methods often rely on sandboxed environments or artificial tasks, which don’t accurately reflect the complexities of real-world web browsing. These ‘closed’ benchmarks suffer from limited task diversity and require extensive engineering effort to incorporate new tasks, often needing ground-truth success criteria that non-technical users cannot easily contribute to.
A new research paper introduces BrowserArena, an innovative live evaluation platform designed to address these limitations. BrowserArena allows for the assessment of LLM agents on real-world, open-web navigation tasks. It builds upon the successful Chatbot Arena framework, using a similar approach of pairwise comparisons to gather human preferences.
How BrowserArena Works
When a user interacts with BrowserArena, they submit a natural language description of a task. This task is then given to two randomly selected LLM agents, which utilize the BrowserUse library to interact with and navigate various websites. These agents operate independent Chromium browser instances, performing actions like clicking elements, inputting text, or navigating to URLs. For models with multimodal capabilities, a screenshot of the current browser with labeled HTML elements is also provided.
After both agents attempt the task, the user is presented with their outputs, including a GIF rendering of each step the agent took. Users then vote on which agent performed better and provide step-level feedback on the agent traces. This granular feedback is crucial for identifying specific failure modes.
Key Findings and Agent Performance
The researchers collected user preference data from 109 user-submitted tasks. Based on these evaluations, a leaderboard was constructed using Bradley-Terry coefficients. Interestingly, DeepSeek-R1, a language model without multimodal capabilities, achieved the highest ELO rating among the tested models, which also included AnthropicClaude 3.7Sonnet, MetaLlama-4-Maverick, OpenAIo4-mini, and GoogleGemini 2.5-Pro-Preview-03-25.
The study also explored the reliability of Vision-Language Models (VLMs) as judges compared to human evaluators. It was found that while GPT-4o showed relatively high agreement with human annotations (68%), o4-mini had lower agreement (58%). Surprisingly, providing GIFs alongside agent traces sometimes *decreased* GPT-4o’s agreement with human baselines, suggesting that multimodality can, in certain contexts, hinder judge reliability.
Identifying Common Failure Modes
A significant contribution of BrowserArena is its methodology for identifying recurring agent failure modes through step-level human feedback. By analyzing user annotations, three consistent failure modes were identified:
- Captcha Solving: Agents often struggle when encountering CAPTCHA puzzles, as the components may not be clickable DOM elements.
- Pop-Up Banner Closure: Pop-up banners (like privacy policies) can block agents from progressing on tasks.
- Direct Navigation to URLs: Agents sometimes directly navigate to a URL they believe is relevant, rather than performing a Google Search first, which can lead to delays if the initial website is complex.
To further investigate these, targeted datasets were created. For captcha solving, tasks involving Expedia.com were used. It was observed that o4-mini deployed a wider variety of strategies to circumvent captchas, including using Google’s cache, mobile versions, or even public proxies, compared to other models. For pop-up banner closure, tasks on bbc.com were used. DeepSeek-R1 consistently failed to detect pop-up banners due to its lack of multimodal capabilities, yet often marked tasks as completed. In contrast, o4-mini and Llama-4 were more successful at closing banners. For direct navigation, TriviaQA questions were used, revealing that agents generally prefer invoking the Google Search API to retrieve information rather than directly navigating to sites like Wikipedia.
Also Read:
- Unpacking AI’s Thought Process: A New Framework for Evaluating Tool-Augmented Agents
- BayesianRouter: A Smart Approach to Aligning Language Models with Human Preferences
Looking Ahead
BrowserArena provides a robust platform and methodology for evaluating and understanding the diverse and sometimes brittle nature of current web agents. While the evaluation method is dependent on the BrowserUse system and the identified failure modes might be system-specific, this approach offers valuable insights into improving LLM agent performance on real-world web tasks. You can read the full research paper for more details here: BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks.
