TLDR: Cloudflare and Microsoft have announced a new collaboration, integrating Microsoft’s NLWeb standard with Cloudflare’s AutoRAG infrastructure. This initiative aims to transform traditional websites into ‘answer engines’ that can respond to natural language queries from both human users and AI agents. The system automates content crawling, indexing into a vector database, and deployment via Cloudflare Workers, offering a one-click solution for websites to become ‘AI-ready’ and potentially disrupting traditional keyword-based search models.
Cloudflare and Microsoft have unveiled a groundbreaking partnership designed to revolutionize how users interact with websites, moving beyond traditional keyword-based search to a conversational, AI-driven model. This collaboration introduces the integration of Microsoft’s open standard, NLWeb, with Cloudflare’s managed Retrieval-Augmented Generation (RAG) engine, AutoRAG, enabling any website to function as an ‘AI app’ or ‘answer engine’.
Historically, website navigation has relied on users typing keywords and sifting through lists of links to find information. However, the rise of AI systems like Copilot, Claude, and ChatGPT has shifted user expectations towards direct answers to natural language questions. This new paradigm also addresses the needs of a growing class of visitors: AI agents, which currently struggle with inefficient keyword searches and scraping to gather information.
NLWeb, an open project from Microsoft, defines a standard protocol for natural-language queries on websites. It also functions as a Model Context Protocol (MCP) server, providing a structured interface for trusted AI agents to access site data reliably. Cloudflare is actively contributing to this specification, aiming to extend the standard to allow every site to behave like an AI application.
Cloudflare’s AutoRAG acts as the underlying infrastructure, automating the entire process of making a website conversational. It crawls websites, respecting `sitemap.xml` and `robots.txt` directives, and captures dynamic JavaScript content using Browser Rendering. The crawled data is stored in an R2 bucket, parsed, and then embedded into a managed vector database (Vectorize), enabling semantic search capabilities beyond simple keyword matching. AutoRAG ensures the content remains current through continuous re-crawling and re-indexing.
Deployment is streamlined, with AutoRAG deploying a Cloudflare Worker that serves as the access layer, implementing the NLWeb standard. This Worker includes a `/ask` endpoint for conversational UI, supporting chat history and automatic query decontextualization, and an `/mcp` endpoint for structured agent access. This ‘one-click’ setup in the Cloudflare Dashboard allows website owners to quickly enable conversational search and structured agent interfaces.
R.V. Guha, creator of NLWeb, CVP, and Technical Fellow at Microsoft, stated, “Together, NLWeb and AutoRAG let publishers go beyond search boxes, making conversational interfaces for websites simple to create and deploy. This integration will enable every website to easily become AI-ready for both people and trusted agents.” Joe Marchese, General & Build Partner at Human Ventures, expressed optimism that this could open new monetization models for publishers by fostering more direct engagement in brand-owned environments.
Also Read:
- Cloudflare Integrates Advanced Image Generation and Text-to-Speech AI Models into Workers AI Platform
- Shopify Revolutionizes AI-Powered Commerce with Interactive Widgets via MCP-UI
This initiative positions Microsoft and Cloudflare in direct competition with traditional search engines like Google, suggesting a future where AI models and LLM systems rely on structured access rather than web scraping, potentially altering search traffic patterns significantly. Cloudflare has also enhanced AutoRAG’s backend with new JobManager and FileManager Durable Objects to improve data consistency, scalability, and visibility into indexing syncs, allowing parallel processing of up to 20 files simultaneously.
