TLDR: Ang Li, a former Google DeepMind engineer and co-founder of Simular, asserts that the current approach to developing AI agents, particularly their over-reliance on Large Language Models (LLMs), is fundamentally flawed. He highlights the low success rates of existing AI agents in real-world tasks and advocates for a more comprehensive reinforcement learning framework.
Ang Li, co-founder of the agent software company Simular, has publicly stated that the prevailing methods for developing AI agents are misguided. Drawing from his experience at Google DeepMind, where he worked from 2017 to 2019, Li recalls a pervasive skepticism among software engineers regarding the practical utility of machine learning in production environments. He noted that despite the hype surrounding AI, the production team often remarked, ‘machine learning never works in production.’
Li recounted a specific instance where the Google Ads team attempted to apply DeepMind’s AlphaGo system, renowned for conquering the game of Go, to boost Google’s ad revenue. Surprisingly, this application led to a drop in revenue, illustrating the complexity of real-world systems compared to controlled environments.
Simular’s core philosophy challenges the industry’s current trajectory. Li firmly believes, ‘We believe everyone else is doing the wrong thing.’ He clarifies that it’s not entirely ‘wrong,’ but rather ‘they are not going in the right direction.’ His primary critique is that most AI agents are exclusively based on Large Language Models (LLMs), which he argues represent only a fraction of the necessary reinforcement learning framework.
This perspective is underscored by data on current AI agent performance. Reports indicate that AI agents successfully complete office tasks only about 30 percent of the time. The OSWorld benchmark, established in April 2024 to assess agent software’s ability to handle real-world computer tasks, showed that the top-performing AI agent, GPT-4 (with vision), achieved an overall success rate of merely 12.24 percent. More recently, the GUI Test-time Scaling Agent (GTA1), when paired with OpenAI’s o3 model, improved this to 45.2 percent on the OSWorld benchmark.
Also Read:
- Andrew Ng Urges Focus on Practical AI Tool Mastery, Dispelling AGI Hype
- Agentic AI Transforms Real Estate: Unlocking Efficiency Amidst Governance Challenges
Simular is initially targeting industries such as insurance and healthcare, which are characterized by a high volume of form-filling and administrative tasks. Li suggests that automating these processes effectively could lead to significant productivity gains. He emphasizes that achieving organizational interest in such automation necessitates a much higher success rate, ideally matching or exceeding human employees. Simular aims to achieve this by adopting a more holistic approach to AI agent development, moving beyond the LLM-centric paradigm.
