Reuters and Time are taking a hard line against AI-powered bots, blocking them by default and only allowing select crawlers through carefully managed whitelists.
Reuters and Time have moved to restrict access to their online content by artificial intelligence bots, according to a report from Digiday. Both publishers are now blocking all AI-driven crawlers by default, only permitting those specifically approved on their internal whitelists. This marks a significant shift in how major media outlets are responding to the rapid proliferation of AI tools scraping the web for data.
Time has implemented ScalePost to manage which bots can interact with its site, providing granular control over AI crawler activity. Meanwhile, Reuters is relying on its robots.txt file to deny access to most bots, a method that has become increasingly common among publishers seeking to protect their intellectual property from unauthorized AI training.
Despite these efforts, some industry observers question the effectiveness of using robots.txt as a barrier, likening it to placing a gate in an open field. The move reflects growing concerns about how AI companies collect and use publisher content, as well as the need for more robust technical and legal safeguards.
This trend toward tighter control echoes broader shifts in the digital content landscape, where publishers are reevaluating their relationships with AI platforms. For example, the rise of agentic AI tools is already reshaping how content is discovered and consumed, as seen in the evolution of travel planning platforms highlighted in recent coverage of next-generation itinerary engines.
