Publishers are seeing new revenue as AI crawlers face stricter blocks. Cloudflare’s default bot restrictions are changing how content is licensed and monetized. Pay-per-use models and new tracking tools are reshaping negotiations.
Publishers using Cloudflare are gaining leverage in AI licensing negotiations as the company intensifies its efforts to block unauthorized AI crawlers. By default, Cloudflare now restricts AI bots from scraping content unless publishers explicitly allow access, typically through a licensing agreement or a clear value exchange. This approach is creating what Cloudflare calls “reliable scarcity,” enabling publishers to demand compensation for their content’s use in AI products.
Cloudflare, which powers about 20% of global websites, has introduced a “pay-per-crawl” model, allowing site owners to charge AI crawlers for access. This system is currently in closed beta, with publishers from various sectors participating and providing feedback. The company is also piloting a “pay-per-use” model, where publishers receive payment when their content appears in AI-generated answers or directly contributes value. Cloudflare’s new AI visibility dashboard lets publishers monitor how often their content is used by platforms like ChatGPT and Claude.
Executives at People Inc report significant progress in AI licensing, attributing their success to Cloudflare’s blocking tools. The company says it blocks nearly all AI crawlers except those from OpenAI, with whom it has a deal, and Google, which uses a unified crawler for both search and AI. Cloudflare’s chief strategy officer, Stephanie Cohen, noted that many publishers-large and small-are using these tools to negotiate better terms. She highlighted that even niche and local publishers, such as Park Record in Utah, are finding new revenue opportunities by restricting AI access to their unique content.
Cloudflare’s pay-per-use model is being tested with Ceramic.ai and You.com, where publishers can earn revenue each time their content is surfaced in AI search results or delivered to enterprise users. Cohen explained that pay-per-use offers a more accurate measure of content value than pay-per-crawl, since not all scraped content is actually used. She also pointed out that over half of AI crawler visits are redundant, accessing unchanged content and wasting resources for both crawlers and publishers. To address this, Cloudflare is collaborating with OpenAI on a research project to help AI search engines crawl only when content has been updated, aiming to reduce unnecessary traffic and resource consumption.
Despite these advances, some in the industry remain concerned about sophisticated bots that disguise themselves as human users or legitimate crawlers. Reports from Tollbit and People Inc describe how aggressive bots rotate IP addresses, impersonate Google, and use residential proxies to bypass protections. People Inc’s innovation officer, Dr. Jonathan Roberts, detailed the multi-stage tactics used by bad actors and acknowledged that while many are blocked, some still get through due to the complexity of identifying compromised residential connections.
Cohen stated that most Cloudflare publisher clients now block AI bots by default, only allowing those with formal agreements. She expressed confidence in the effectiveness of Cloudflare’s blocking tools, noting that rare exceptions are publicly disclosed. The company has criticized platforms like Perplexity for allegedly attempting to circumvent blocks. Cloudflare continues to enhance its Precursor bot management product, which now operates within browsers to better detect and stop evasive bots. Cohen also warned publishers to monitor for “leaky buckets,” such as syndicated content appearing on third-party sites, which can undermine blocking efforts.
Starting September 15, Cloudflare will block mixed-purpose AI crawlers-those used for both search and training-by default on ad-supported pages. This policy will affect crawlers like Googlebot unless Google separates its search and AI crawlers as promised. Cohen explained that this move is intended to give publishers more transparency and control over how their content is accessed and used, supporting a more sustainable content ecosystem. The company is providing a transition period to encourage platforms to adapt before enforcement begins.
For additional perspective on how companies are responding to regulatory and technological shifts in data and content access, see this report on businesses navigating universal privacy opt-outs.
Cloudflare, founded in 2009, has grown into a major internet infrastructure provider, serving millions of websites worldwide. The company reported annual revenue exceeding $1 billion in 2025 and continues to expand its suite of security, performance, and content management tools for publishers and enterprises.
