AI companies face new rules on web crawling. Cloudflare will block mixed-use bots from ad pages by default in 2026. Publishers may gain more control and revenue from AI training access.
Cloudflare is introducing a major policy shift that will affect how AI companies access publisher content for training and agentic services. Starting September 15, 2026, Cloudflare will by default block web crawlers that serve both traditional search and AI training purposes-so-called "mixed-use" crawlers-from accessing any pages that display ads. This change will apply to all new Cloudflare customers, new sites created by existing customers, and all current free-tier users, unless site owners manually adjust their settings.
The company said this move is designed to give publishers more control over how their content is used by AI models, while still allowing discoverability through search. Cloudflare noted that most publishers want their content to be found in search engines and, increasingly, through AI-powered services, but they also want to prevent their intellectual property from being used without compensation.
Cloudflare specifically referenced the "world’s largest search engine"-a clear nod to Google-stating that it currently has access to about twice as much information as other AI companies, due to the way Google structures its crawling and opt-out mechanisms. Google has previously responded to such claims by highlighting its Google Extended bot, which allows publishers to opt out of AI training without affecting their search visibility. However, Googlebot, the main crawler, still collects data for both search and certain AI features.
Cloudflare’s CEO, Matthew Prince, said the policy is a response to the recent milestone where automated bots now account for the majority of internet traffic, surpassing human users earlier than expected. Prince explained that the new tools and partnerships are intended to give website owners more transparency and commercial opportunities, while encouraging AI companies to clarify the intent of their bots.
To support publishers, Cloudflare has expanded its suite of tools, including a marketplace called Pay Per Crawl, which lets sites charge AI bots for scraping. This is evolving into Pay Per Use, enabling publishers to charge AI companies not just for fetching content, but for actual value created when their material is used. Cloudflare’s data shows that over half of AI crawler traffic is spent re-fetching unchanged pages, so the new approach could also help conserve bandwidth and computing resources.
Cloudflare is initially partnering with Ceramic.ai and You.com to pilot these monetization models. When a publisher opts in, they receive payment when their content appears in Ceramic’s AI search results or when You.com accesses premium material. Cloudflare said other AI companies can adapt this model to fit their own workflows.
This policy shift comes as publishers and technology providers experiment with new ways to balance discoverability and content protection. For example, some major media brands are now building AI-optimized site formats to maintain visibility as agentic search grows, as reported in coverage of publishers redesigning for AI agents.
Cloudflare, founded in 2009, is one of the world’s largest web infrastructure and security companies. As of 2026, it serves millions of websites globally and reported annual revenue exceeding $1.5 billion in its most recent filings. The company’s platform is widely used by publishers, e-commerce businesses, and enterprises seeking to manage traffic, security, and performance at scale.
