TLDR: MacroBench is a new benchmark that evaluates how well Large Language Models (LLMs) can create reusable web automation programs (macros) using Python and Selenium from natural language instructions. It uses seven self-hosted, synthetic websites mimicking popular platforms and 681 tasks of varying complexity. The study found that while LLMs like GPT-4o-Mini perform well on simple tasks, they struggle with complex workflows and don’t generate production-ready code. MacroBench also includes a safety evaluation, highlighting the need for models to not just refuse harmful requests but also propose safer alternatives. The benchmark aims to provide actionable diagnostics for building more capable and responsibly deployable web automation agents.
Large Language Models (LLMs) are rapidly changing how we interact with technology, and one area where their potential is particularly exciting is web automation. Imagine telling an AI what you want to do on a website, and it automatically writes the code to perform that task. This is the core idea behind “macro synthesis” – where LLMs generate reusable programs (macros) for browser actions like clicking, typing, and submitting forms.
To truly understand how well LLMs perform in this domain, researchers Hyunjun Kim and Sejong Kim from KAIST introduced MacroBench, a novel benchmark designed specifically for evaluating LLMs in synthesizing web automation scripts. This benchmark takes a “code-first” approach, meaning the LLM interacts directly with HTML/DOM (the underlying structure of a webpage) and outputs Python code using the industry-standard Selenium library.
What is MacroBench?
MacroBench isn’t just another test. It’s a comprehensive environment built to rigorously assess LLMs. Here’s what makes it unique:
- Synthetic Website Ecosystem: To ensure reproducible and safe evaluations, MacroBench features seven self-hosted, synthetic websites that mimic popular real-world platforms. These include Airbnb-like (hospitality marketplace), TikTok-like (short-video feed), Reddit-like (forums), Instagram-like (photo feed), Facebook-like (social network), Discord-like (chat), and Threads-like (microblogging) sites.
- Extensive Task Set: The benchmark includes 681 distinct automation tasks, categorized into simple, medium, and complex levels. These tasks cover a wide range of interactions, from basic clicks to multi-step workflows.
- End-to-End Validation: Generated code undergoes a thorough validation process, including static checks (linting, safety), sandboxed execution in a headless browser, and outcome verification using DOM assertions and database snapshots.
- Safety Evaluation: Recognizing the dual-use risks of web automation (e.g., spam, scraping, credential harvesting), MacroBench also includes a safety suite to probe whether LLMs refuse or reshape harmful requests.
Key Findings from the Research
The study evaluated four contemporary LLMs: GPT-4o-Mini, GPT-4.1, Gemini-2.5-Pro, and DeepSeek-V3.1. The results revealed a clear stratification in their performance:
- Overall Performance: GPT-4o-Mini achieved the highest success rate at 96.8%, followed by GPT-4.1 (95.3%), Gemini-2.5-Pro (89.0%), and DeepSeek-V3.1 (83.4%).
- Task Complexity: While models performed reliably on simple tasks (91.7% success), their effectiveness significantly degraded on medium-complexity workflows (84.1%). Crucially, complex tasks involving conditional logic and error recovery remained largely unsolved (0.0% success).
- Website-Specific Patterns: Performance varied by platform archetype. Discord-like chat platforms had the highest success rate (99.5%), likely due to simpler workflows, while TikTok-like short-video platforms proved most challenging (81.5%) due to dynamic content and infinite-scroll feeds.
- Code Quality: A significant finding was that despite functional completion, none of the models generated production-quality code under strict rubrics. The generated macros often lacked essential robustness features like explicit waits, structured error handling, and proper parameterization.
- Safety Insights: All models consistently refused explicitly harmful requests. However, GPT-4.1 and GPT-4o-Mini demonstrated superior “refuse-and-repair” behavior, declining unsafe requests while proposing policy-compliant alternatives. Other models often provided bare refusals without constructive redirection.
Also Read:
- BrowserArena: A New Platform for Evaluating AI Web Agents on Real-World Tasks
- Beyond Final Answers: TRAJECT-Bench Evaluates AI Agents’ Tool-Use Journeys
Implications for the Future
The MacroBench study highlights that while current LLMs are capable of automating simple, well-structured web tasks, they are not yet ready for complex, multi-step workflows that demand robust error handling and dynamic adaptation. The absence of production-quality coding practices means that human review and refinement are still crucial before deploying LLM-generated automation scripts.
For developers and practitioners, these findings suggest starting with simple automation tasks, implementing comprehensive testing, and developing model-specific prompting strategies that emphasize robustness. The safety evaluation also underscores the need for more nuanced safety training for LLMs, distinguishing between legitimate productivity automation and potential policy-violating abuse.
MacroBench provides a valuable tool for diagnosing the strengths and weaknesses of LLMs in web automation, paving the way for more capable and responsibly deployable AI agents. You can find more details about this research paper here: MacroBench: A Novel Testbed for Web Automation Scripts via Large Language Models.
