A new shared signal in robots.txt allows editors to block AI training once, instead of per company. Major AI firms like OpenAI, Google, and Microsoft are supporting the voluntary system.
AI content restrictions get a new tool
Editors can now use a single rule in their robots.txt file to signal that they do not want their content used for training AI models, instead of creating separate rules for each AI company.Shared signal replaces individual rules
Major AI companies like OpenAI, Google, and Microsoft have agreed to support a common signal in robots.txt, which allows editors to specify their preferences for AI training once, rather than separately for each provider.The signal is based on an IETF draft
The signal is based on a draft proposal submitted to the IETF, an organization that develops Internet standards. The draft was inspired by work from the AIPREF working group but is not an official standard.How to use the new signal
Editors can add two lines to their robots.txt file: "User-Agent: __usage_ai-training" and "Disallow: /" to indicate they do not want their content used for AI training. Specific paths on a site can also be allowed or restricted.Implementation timelines vary
OpenAI plans to implement the signal in the coming months, while Google aims to support it by 2027. Microsoft will begin using the signal for Bing on February 1, 2027, with a transitional tool available in December 2026.The signal is voluntary and not enforceable
The new signal does not block access to a site or its indexing by search engines. It only expresses a preference and is not retroactive. AI companies can choose to follow it, but there is no enforcement mechanism.Existing controls remain valid
Current crawling rules, such as those for GPTBot or Google-Extended, remain in effect. Editors are encouraged to keep these rules alongside the new signal for more effective control over AI access to their content.
