TLDR: A new method called Behavior Best-of-N (bBoN) significantly improves the reliability and success rates of computer-use agents (CUAs) by generating multiple task attempts (rollouts) and intelligently selecting the best one. It achieves this by converting complex agent actions into concise “behavior narratives” and using a judge to compare these narratives, setting a new state-of-the-art in digital task automation and approaching human-level performance on benchmarks like OSWorld.
Computer-use agents, or CUAs, are designed to automate everyday digital tasks across various operating systems and applications. While they hold immense promise for making our digital lives easier, current CUAs often struggle with complex, long-duration tasks. Their unreliability and high variability in performance mean that a task an agent succeeds at one time might fail catastrophically the next. This is due to small errors accumulating, delayed feedback, unpredictable solution paths, and environmental factors like UI changes or pop-ups.
A new research paper introduces a novel method called Behavior Best-of-N (bBoN) that aims to tackle these challenges head-on. The core idea behind bBoN is to leverage “wide scaling” – instead of relying on a single attempt from one agent, it generates multiple parallel attempts (called rollouts) and then intelligently selects the best one. This approach significantly boosts the robustness and success rates of CUAs.
How Behavior Best-of-N Works
The bBoN framework is built on two key components: the Behavior Narrative Generator and the Behavior Best-of-N Judge.
The **Behavior Narrative Generator** takes the raw, information-dense trajectories of an agent’s actions (which include screenshots before and after each action) and converts them into concise “behavior narratives.” These narratives are summaries that capture what the agent actually did and how its actions affected the environment, filtering out irrelevant details. For instance, if an agent clicks a button, the narrative would describe the change that occurred as a result, rather than just stating the click itself. To ensure accuracy, the generator uses visual augmentations like markers on screenshots to highlight where actions occurred and zoomed-in views to verify fine-grained changes.
The **Behavior Best-of-N Judge** then steps in. Given a set of these behavior narratives from multiple agent rollouts, a Vision-Language Model (VLM) judge performs a comparative evaluation. Instead of evaluating each narrative in isolation, the judge compares them directly, often in a multiple-choice question (MCQ) format. This allows the judge to focus on the differences between agent behaviors and select the trajectory that best fulfills the task requirements. This comparative approach is crucial for reliable selection, especially when multiple valid solution paths exist.
An Improved Foundation
The bBoN method also builds upon an improved agentic framework called Agent S3. This baseline agent itself achieves state-of-the-art performance by incorporating programmatic edits (allowing the agent to write and execute code for complex data manipulation) and using a “flat policy” that can replan at any time, removing the need for a rigid hierarchical planning structure. Agent S3 provides high-quality initial trajectories for bBoN to then scale and refine.
Impressive Results and Generalization
The impact of bBoN is substantial. On the OSWorld benchmark, which involves 361 real-world Ubuntu tasks, bBoN achieved a new state-of-the-art success rate of 69.9% (at 100 steps) with GPT-5, significantly outperforming previous methods and closely approaching human-level performance of 72%. The research demonstrates that performance generally increases with the number of rollouts, validating the effectiveness of wide scaling.
Furthermore, bBoN shows strong generalization capabilities. It improved performance on WindowsAgentArena and AndroidWorld benchmarks, demonstrating its applicability across different operating systems. The use of behavior narratives proved more effective than simply using screenshots or naive captioning for trajectory representation, and comparative selection by the judge was superior to independent ranking of trajectories.
Also Read:
- GRAPH2EVAL: Crafting Complex Evaluation Tasks for Advanced AI Agents
- Smarter Supervision for Automated GUI Operations
Looking Ahead
While bBoN offers a powerful leap forward, the researchers acknowledge limitations. The method currently assumes access to environments that allow for multiple independent rollouts, such as virtual machines with snapshot capabilities. Running agents on a user’s actual desktop or dealing with shared online resources (like shopping carts or email) introduces complexities that require further research to manage cross-run interference. However, the findings highlight the “unreasonable effectiveness” of scaling CUAs when done correctly, providing a practical framework for structured trajectory understanding and selection.
For more details, you can read the full research paper: The Unreasonable Effectiveness of Scaling Agents for Computer Use.
