TLDR: Cloudflare, a major internet infrastructure provider, has announced a significant shift in how AI models access web content. Moving forward, AI crawlers will be blocked by default unless they obtain explicit permission or compensate content creators, aiming to establish a more equitable economic model for the internet in the age of artificial intelligence.
Cloudflare, a company that silently underpins a substantial portion of the internet’s infrastructure, is now taking a proactive stance in shaping the future of artificial intelligence. On July 1, 2025, Cloudflare announced it would become the first internet infrastructure provider to block AI crawlers from accessing content without explicit permission or compensation, a move designed to safeguard the internet’s stability and ensure fair value for content creators. This initiative comes as generative AI models increasingly scrape web content without directing traffic or revenue back to original sources, disrupting the long-standing economic model of the web.
Matthew Prince, co-founder and CEO of Cloudflare, emphasized the necessity of this change, stating, ‘If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself.’ He added that the goal is to ‘put the power back in the hands of creators, while still helping AI companies innovate.’ This sentiment was echoed by Roger Lynch, CEO of Condé Nast, who called Cloudflare’s approach ‘a game-changer for publishers’ that ‘sets a new standard for how content is respected online.’ Neil Vogel, CEO of Dotdash Meredith, also expressed support, noting, ‘We can now limit access to our content to those AI partners willing to engage in fair arrangements.’
Cloudflare’s new policy means that upon signing up, every new domain will be prompted to choose whether to allow AI crawlers, defaulting to denial. Existing customers have had the option to block AI crawlers since September 2024, with over a million already adopting this measure. The company is also working on new protocols to help AI bots authenticate themselves, allowing publishers to make nuanced decisions, potentially permitting AI for search but not for training.
Beyond simply blocking, Cloudflare aims to build a ‘pay per crawl’ marketplace where content creators and AI companies can collaborate. Matthew Prince envisions a future where ‘traffic was always a poor proxy for value. We think we can do better,’ suggesting a shift towards valuing content based on its contribution to knowledge rather than mere page views. Dane Knecht, CTO of Cloudflare, highlighted that even AI companies desire a fair competitive environment and prefer not to bear the sole cost of content creation, supporting a system that promotes equal opportunities.
Also Read:
- Microsoft Forges Ahead with AI Agents, Redefining Workflows and Facing Infrastructure Demands
- Google Cloud’s Q2 Revenue Surges 32% Driven by Strategic OpenAI Infrastructure Partnership
This move has garnered support from major media entities like The Atlantic, Fortune, TIME, and The Associated Press, as well as tech platforms such as Pinterest and Reddit, which have faced issues with content scraping. Will Allen, Head of AI Control, Privacy and Media Products at Cloudflare, noted that these companies have signed on in support of Cloudflare’s permission-based approach. Cloudflare’s infrastructure, which manages and protects traffic for 20% of the web and handles trillions of requests daily, positions it uniquely to enforce this new model, ensuring that original content, which Prince calls ‘one of the greatest inventions in the last century,’ continues to be created and fairly compensated.
