A new UK bill aims to force AI bots to reveal their identity when scraping content. Publishers say hidden bots threaten revenue and transparency. The move follows similar action in New York.
UK publishers are backing a proposed law that would require AI bots scraping website content to clearly identify themselves and disclose their purpose. Industry leaders argue that undisclosed scraping by automated bots undermines their ability to protect content, negotiate licensing, and maintain sustainable revenue streams.
The Automated Online Software (Access and Transparency) Bill, introduced by Conservative MP Damian Hinds, seeks to address the growing problem of bots that disguise their identity or intent when accessing online content. The bill has support from the News Media Association and is designed to give website owners more control over who accesses their data and how it is used. However, the bill is unlikely to become law without government backing, though it could influence future legislation.
This legislative push comes after New York state passed the Stealth Crawler Prohibition Act, which targets AI companies deploying bots that scrape news content without proper disclosure. The New York law, awaiting the governor's signature, would make it an offense to harm a news site's operations or cause economic damage through unauthorized scraping. It also allows news organizations to seek subpoenas to identify violators and pursue injunctions or damages.
Industry data cited by media consultant Matthew Scott Goldstein suggests that third-party scrapers and brokers now drive a $1 billion market. Some bots impersonate humans or legitimate services like Google, making it difficult for publishers to track or block unwanted access. According to a recent Fastly report, bots now account for 49% of all website traffic, with 99% of that bot traffic considered unwanted or unverifiable-often due to scraping copyrighted content or impersonating trusted services.
Publishers say these stealth bots not only siphon off valuable journalism but also force them to invest heavily in bandwidth and cybersecurity to defend their sites. Some bots have even been found bypassing paywalls to access full articles. The News Media Association's chief executive, Theo Bamber, said the proposed UK bill would give publishers the right to see who is trying to access their sites and negotiate terms for any content use.
To counter unwanted scraping, some publishers have started adding search-only contracts to their website terms, replacing traditional robots.txt files. This allows them to invoice AI firms for each article scraped, potentially avoiding lengthy copyright litigation.
Efforts to increase transparency and control over web scraping reflect a broader push for fairer treatment of publishers in the digital ecosystem. Similar regulatory actions have been seen in other areas, such as when the UK's competition authority ordered Google to make its search rankings more transparent and fair for publishers-a move detailed in this recent report.
