Cloudflare to block mixed-use AI crawlers on ad-supported sites

On Wednesday, Cloudflare announced a policy shift targeting how artificial intelligence companies access web content. Starting September 15, 2026, the cloud platform and security provider will block “mixed-use crawlers”—which it defines as crawlers that blend search, agent use, and training—from pages that host advertisements by default. The policy change could impact how AI model providers access web content for training and agentic services. The new default settings will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers.

The company’s decision is partly driven by the inefficiencies of current automated web traffic. According to Cloudflare, over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages. This activity consumes bandwidth and compute resources for publishers. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Matthew Prince, Cloudflare co-founder and CEO. To address this, Cloudflare is also evolving its “Pay Per Crawl” tool—a marketplace that lets websites charge AI bots for scraping—into a “Pay Per Use” model. This evolved model allows publishers to charge AI companies when their content creates value, rather than only when it is fetched. Cloudflare is initially partnering with Ceramic.ai and You.com to implement this monetization system.

The policy also directly addresses the market dominance of search engines. Cloudflare alleges that Google, the world’s largest search engine, makes it difficult for customers to remain discoverable without being used for AI. Specifically, Cloudflare claims that this search provider has access to about 2x more information than other AI companies because of these practices. While Google has previously pointed to its Google Extended bot as a way for site owners to opt out of AI training without affecting their Google Search presence, its Googlebot continues to crawl for search and integrated AI features. Prince stated that the company hopes its proposed default changes will encourage mixed-use crawlers to separate search functions from agent use and training.

Why it matters

Cloudflare is shifting the power dynamic by forcing AI companies to distinguish between search and training bots, while simultaneously monetizing publisher content through an evolved “Pay Per Use” model.