TLDR: Runloop.ai and Fermatix.ai have joined forces to launch a Custom Benchmarks product, empowering organizations to develop specialized, private benchmarks for evaluating AI agents against their unique, proprietary codebases and business logic. This collaboration aims to address the growing demand for precise and relevant evaluation of AI agents as they transition from development to real-world deployment.
SAN FRANCISCO, October 1, 2025 – Runloop.ai, a prominent enterprise infrastructure platform for AI agents, has officially announced the launch of its innovative Custom Benchmarks product. This new offering is designed to enable organizations to create highly specialized, private benchmarks, crucial for accurately measuring and refining AI agents based on their unique, proprietary codebases and specific business logic. To underscore the product’s extensive applications and strategic value, Runloop.ai is collaborating with Fermatix.ai, a company renowned for its expertise in full-cycle data generation, on a significant pilot program.
The proliferation of AI agents has highlighted a critical need for rigorous and relevant evaluation and functional training. While existing public benchmarks serve a vital role in general model assessment, they frequently fall short in capturing the specific requirements of enterprise AI agents or the nuanced validation needs of businesses. Runloop.ai’s Custom Benchmarks directly addresses this gap by providing a secure and scalable platform. This platform allows companies to construct benchmarks that test against their internal business logic, diverse tech stacks, and precise performance metrics.
Key features of Runloop.ai’s Custom Benchmarks product include:
Private benchmarking: Facilitates secure testing of AI agents on proprietary code, safeguarding intellectual property.
Accurate performance evaluation: Measures agent effectiveness under real-world, business-specific conditions.
Scalable infrastructure: Offers a reliable and isolated environment capable of running thousands of tests concurrently.
Strategic model refinement: Generates data essential for targeted improvements and retraining of AI agents for specific tasks.
Jonathan Wall, CEO of Runloop.ai, emphasized the significance of this evolution, stating, “As AI agents move from prototypes to production, the benchmarks we use to evaluate them must evolve from generic tests to strategic assets. Our new Custom Benchmarks product empowers enterprises to define what ‘good’ looks like for their specific AI agent deployments.”
Also Read:
- Netlify Unveils AI Agent Runners to Drastically Reduce Development Backlogs and Accelerate Deployments
- Avalara Revolutionizes Compliance with Launch of Agentic AI Tax Solutions
Fermatix.ai, known for creating expert-level training data with annotators who are practicing industry experts, has strategically expanded its capabilities through this partnership. By leveraging Runloop.ai’s Custom Benchmarks infrastructure, Fermatix.ai can now offer custom, in-house verification for its clients. This allows them to move beyond providing one-time training data to establishing ongoing testing standards and verification, ensuring AI agent performance against specific proprietary logic. This collaboration positions Fermatix.ai to offer a new level of assurance and become a crucial verification layer for their clients’ AI agent deployments, creating high-fidelity, multilingual benchmarks on a secure, scalable platform.
