A Google representative says the new Content-Signal directive in robots.txt files, introduced by Cloudflare, is not recognized by major web crawlers or AI models. Find out why this matters for publishers and SEO professionals.
John Mueller, a well-known representative from Google, has publicly stated that the Content-Signal directive-recently added to robots.txt files by Cloudflare-serves no practical purpose for search engines or large language models. In a comment posted on Reddit, Mueller clarified that, to his knowledge, no major web crawler or LLM currently supports or interprets the Content-Signal directive.
Mueller explained that the directive was developed by a CDN provider and does not influence how web crawlers or AI models process site content. He emphasized that adding unsupported directives like Content-Signal only increases the size and maintenance burden of the robots.txt file, without delivering any functional benefit. According to Mueller, web crawlers will continue to recognize only the directives they are programmed to support, ignoring any others present in the file.
This statement comes as some publishers and site owners consider implementing Content-Signal in hopes of influencing how their content is accessed or indexed by AI systems. Mueller’s comments suggest that, at least for now, such efforts are unlikely to have any impact on how Google or other major platforms handle robots.txt instructions.
