Major publishers are weighing whether to block Google's crawler to prevent their content from being used for AI training without compensation. This move could impact the quality of Google Search results and reshape traffic strategies.
Digital publishers are now openly considering blocking Google's crawler, a move that could reshape how content is discovered and monetized online. For years, Google Search has been the primary driver of audience and revenue for many media companies. But as Google uses its crawler for both search indexing and AI model training, publishers are questioning whether the trade-off still makes sense.
Cloudflare, which hosts about 20% of global websites, recently announced that starting September 15, new sites and free-tier customers will have bot management settings default to blocking multi-purpose crawlers on ad-supported pages. This targets crawlers that collect data for both search and AI training-most notably Google's-unless site owners opt in. Cloudflare's leadership said they want a technical solution that allows sites to be discoverable in search without giving away content for free.
While Google has introduced Google Extended, an option that lets publishers opt out of AI training while remaining in search, some media executives remain skeptical. They worry that using these controls could still hurt their search visibility. New controls in the UK offer more granular management, but publishers are hesitant to rely on Google's discretion rather than a hard block.
Historically, leaving Google Search was seen as commercial suicide, according to SEO consultant Lily Ray, because of the platform's unmatched audience reach. However, as search traffic has declined for many publishers, some are exploring alternative audience channels such as newsletters, social media, and events. USA Today Inc., which includes a network of news sites, has maintained stable traffic by diversifying its sources and is now considering delisting from Google within the next year if no licensing deal is reached. The company has already signed licensing agreements with Meta, Microsoft, and Amazon, but not with Google.
Beehiiv, a creator network, has also enabled its users to block the Google crawler through a partnership with Cloudflare. USA Today Inc.'s CEO Mike Reed said the company blocks crawlers from platforms without licensing agreements, while allowing access to those who pay for content rights.
Executives at other major media companies are modeling what a Google Bot block would look like, weighing the value of search traffic against the leverage gained by withholding content. The decision often comes down to numbers: if search traffic drops below a certain threshold, blocking Google becomes a viable bargaining tactic.
Publishers would prefer Google to negotiate licensing agreements, but if that does not happen, more premium publishers could block Google's crawler. This could degrade search result quality, making it harder for users to find reliable news. As one executive noted, their content powers generative experiences on Google hundreds of millions of times daily, and removing it would worsen those experiences.
Still, some media leaders are unsure about the benefits of opting out. Blocking Google may not yield immediate gains, and there is uncertainty about future compensation if Google eventually offers payouts. The dilemma leaves publishers with few appealing options.
The willingness of publishers to consider leaving Google Search marks a significant shift in the open web's evolution. While the standoff is part of ongoing negotiations, the threat of blocking Google is now real. Publishers are preparing for a future where they may need to operate without Google, while Google itself has not outlined an alternative plan.
Other publishers have faced similar crossroads. For example, Hello! Magazine rebuilt its audience after a sharp drop in Google referrals by focusing on cross-channel performance and editorial independence.
