TLDR: WALT (Web Agents that Learn Tools) is a novel framework that enables web agents to reverse-engineer and utilize a website’s inherent functionalities (like search, filter, or post) as robust, reusable tools. This approach moves beyond fragile, step-by-step UI interactions, allowing agents to call high-level operations directly. WALT involves a two-stage process of tool discovery and iterative construction/validation, leading to significantly higher success rates and efficiency on complex web automation tasks compared to traditional methods.
Web agents are designed to automate complex tasks within a browser, but current methods often struggle. They typically rely on a series of step-by-step interactions with a website’s user interface (UI), which can be fragile and break easily when layouts change or tasks become very long. Imagine trying to find the cheapest blue kayak on a classifieds site: a traditional agent might meticulously click search boxes, hover over dropdowns, select categories, and then sort results. This process is prone to errors if even a small part of the website’s design shifts.
Humans, on the other hand, approach such tasks differently. We understand the underlying functionality of a website. We think in terms of high-level operations like ‘search for kayaks,’ ‘filter by price,’ or ‘identify the blue one.’ We abstract away the nitty-gritty details of clicking and typing, focusing instead on the goal. This human ability to recognize and leverage reusable patterns across websites—whether for searching, filtering, creating content, or interacting socially—is what inspired a new framework called WALT (Web Agents that Learn Tools).
Introducing WALT: Learning Website-Provided Functionality
Developed by researchers Viraj Prabhu, Yutong Dai, Matthew Fernandez, Jing Gu, Krithika Ramakrishnan, Yanqi Luo, Silvio Savarese, Caiming Xiong, Junnan Li, Zeyuan Chen, and Ran Xu from Salesforce AI Research, WALT introduces a novel approach. Instead of agents trying to figure out how to click and type for every single step, WALT reverse-engineers the latent functionality already built into websites and exposes it as reusable, invocable tools. These tools are robust implementations of automations that website designers have already engineered, covering areas like discovery (search, filter, sort), communication (post, comment, upvote), and content management (create, edit, delete).
With WALT, an agent doesn’t need to reason about ‘how’ to click a search button and type a query. It simply calls a high-level tool like search(query='blue kayak', category='Boats', sort_by='price'). This dramatically reduces the number of fragile UI steps to a single, robust operation. The computational burden shifts from brittle, step-by-step reasoning to reliable tool invocation, making browser automation more efficient and less prone to errors.
How WALT Works: Discovery, Construction, and Validation
WALT operates through a two-stage pipeline: strategic discovery of tool candidates, followed by their construction and rigorous validation.
The first stage, Tool Discovery, involves a web agent systematically exploring different sections of a website. It identifies interactive elements and proposes a list of reusable tool candidates based on common user goals, such as searching, creating content, or posting comments. The agent prioritizes functionality that maximizes coverage and minimizes redundancy.
The second stage, Tool Construction & Validation, transforms these candidates into validated, executable tools. A specialized ‘tool construction agent’ works with a ‘browser agent.’ The browser agent demonstrates the functionality, creating execution traces of its interactions. The tool construction agent then analyzes these traces to synthesize structured action scripts. These scripts are designed to be as deterministic as possible, often replacing multi-step UI sequences with more robust URL manipulations (like modifying query parameters in a URL for a search). It also infers a validated input schema for each tool, defining what parameters it accepts (e.g., text for a search query, options for a dropdown menu).
Crucially, each candidate tool undergoes a rigorous validation process against pre-vetted test inputs. If a tool fails, structured feedback is used to refine its selectors, amend its input schema, or edit the action script. This iterative loop ensures that only robust and reliable tools are exposed to the agent at runtime. WALT also includes an ‘agentic fallback’ mechanism as a last resort, allowing a fresh agent to handle unexpected failures on the fly.
Also Read:
- GRAPH2EVAL: Crafting Complex Evaluation Tasks for Advanced AI Agents
- LLM Agents Learn New Skills: Distilling Retrieval Guidance for Enhanced Performance and Efficiency
Impressive Results and Future Potential
WALT has been benchmarked on challenging web agent environments like VisualWebArena and WebArena, achieving state-of-the-art success rates. For instance, it attained an average success rate of 52.9% on VisualWebArena and 50.1% on WebArena, significantly outperforming previous methods. These gains come with increased efficiency, requiring 1.3 to 1.4 times fewer steps on average compared to baseline methods. The research paper, available at https://arxiv.org/pdf/2510.01524, provides a detailed account of these findings.
The discovered tools span a wide range of functionalities, from simple search operations to complex content management tasks like creating or editing listings, and social interactions such as commenting. While WALT represents a significant leap forward, the authors acknowledge limitations, including the cost of offline tool discovery per website and challenges with highly dynamic interfaces or anti-automation measures. However, these also present exciting opportunities for future work, such as online tool patching and hybrid integration with official APIs.
Overall, WALT transforms browser automation by shifting the paradigm from brittle, step-by-step UI reasoning to efficient, tool-based abstraction. By enabling web agents to learn and leverage the inherent functionality of websites, WALT paves the way for more robust, reliable, and auditable automation in the digital world.
