Legislation in New York and Congress aims to stop bots from hiding their identity. Publishers face growing risks as unidentified crawlers scrape content. Cloudflare data shows bots now dominate web traffic.
Media and publishing professionals are facing a new challenge as lawmakers move to address the rise of stealth crawlers-bots that scrape publisher content while concealing their identity. This issue has become urgent for publishers who rely on understanding who accesses their content and for what purpose, as unidentified crawlers increasingly pull material from their sites without disclosure.
Recent legislative action reflects the growing concern. Last month, New York passed a bill designed to prohibit bots from masking their identity. In July, similar legislation was introduced in the U.S. House of Representatives, signaling a coordinated effort to bring more transparency to automated web traffic. These measures aim to give publishers more control and insight into who is accessing their content and how it is being used.
Cloudflare data highlights the scale of the problem: more than half of all web traffic now comes from bots, with a significant portion attributed to crawlers that do not identify themselves or their intentions. This leaves publishers unable to track or manage the use of their content, raising concerns about unauthorized distribution and potential revenue loss.
The issue of content scraping is not limited to the United States. In the UK, publishers have begun enforcing new contracts to charge AI companies for unauthorized use of their material, with legal action and asset seizures possible if payments are not made. Industry groups have voiced support for these moves, as reported in a recent article on UK publishers' efforts to bill AI firms for scraping.
