OpenAI, Google, Microsoft and Anthropic back one robots.txt opt-out for AI training
A draft IETF policy token would let websites set one AI training preference for several labs at once, instead of blocking crawlers one by one.
Photo: Victor Grigas, CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0/) · Wikimedia Commons File:Wikimedia_Foundation_Servers-8055_35.jpg · toned · poster CC BY-SA 4.0
CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0/); source https://commons.wikimedia.org/wiki/File:Wikimedia_Foundation_Servers-8055_35.jpg
Voices are AI-generated
OpenAI, Google, Microsoft and Anthropic have backed a single robots.txt signal that lets websites tell several AI companies at once whether their pages may be used for AI training [1][2][4]. The signal comes from an IETF draft written by an Anthropic standards lead and published on 30 October 2026 [3][4].
What is the new AI training signal?
Website operators add a group to robots.txt keyed by a shared policy token instead of naming each AI crawler [2][3]. The IETF draft says a Disallow rule under that token expresses a preference against AI training, while Allow paths express a preference in favour [3].
The daily pin – 5 minutes, sourced.
Five sourced stories each morning. Free, and you can unsubscribe anytime.
The draft adds no new directives or grammar, so existing robots.txt parsers simply ignore the group [3]. In the draft, the token is "__usage_ai-training", and a wildcard group must not be read as matching it [3].
Who supports it, and when?
OpenAI said it will honor a common robots.txt signal for AI training and aims to implement it in the coming months. The company says its existing crawler controls will continue to apply [1].
Google said it plans to support the shared AI training control along with other AI providers in 2027. If a site opts out with the token and does not name Google-Extended, Google plans to treat it as if Google-Extended were disallowed [2].
Anthropic has already adopted the signal, while Google and Microsoft plan to support it next year, The Verge reports. Microsoft will also offer separate opt-outs for training and for grounding, and sites stay in Bing search results either way, according to The Verge [4].
We believe creating a common signal that conveys such preferences to multiple providers at once is an important step
Meta signalled support but did not say whether it will honor the token, The Verge reports [4].
Why does this matter for publishers?
Until now, sites had to list every AI crawler separately and keep those rules up to date [1][4]. Google says the proposal reflects calls from publishers, civil society groups and regulators for industry-wide training controls [2].
Google also says the approach fits the EU Code of Practice for general-purpose AI models, which recognises robots.txt as a main way to express machine-readable preferences [2]. The signal remains voluntary and is still a draft at the IETF [1][3].
Questions and answers
Does the signal hide my site from search?
No. OpenAI and Google say their existing crawler controls keep working alongside it, and Microsoft says sites stay in Bing results, according to The Verge [1][2][4].
Is it already a standard?
No. It is an individual IETF Internet-Draft dated 30 October 2026 that expires on 3 May 2027 unless updated [3].
Our read
Opinion: One line in robots.txt that the four biggest labs promise to respect is a real usability win for publishers. But it is still a voluntary preference, not a licence or a lock, and it only covers future training, so the fights over past scraping and payment are untouched.
What we could not verify
- Microsoft's and Anthropic's own announcements; their positions are cited from The Verge.
- Exact implementation dates beyond "coming months" (OpenAI) and 2027 (Google).
- Whether content already collected for training is affected; none of the sources says so.
Tags: ai-training · robots-txt · publishers · openai · google · microsoft · anthropic · ietf · web-standards
Why it matters
- Why it matters
- OpenAI, Google, Microsoft and Anthropic backed a shared robots.txt signal on 30 October 2026 so sites can opt out of AI training in one step.
- What's confirmed
- Confirmed by primary sources: OpenAI announcement, Google Search Central blog post and IETF Internet-Draft (shared robots.txt training signal, timelines).
- What's uncertain
- Microsoft's and Anthropic's own announcements; their positions are cited from The Verge.; Exact implementation dates beyond "coming months" (OpenAI) and 2027 (Google).; Whether content already collected for training is affected; none of the sources says so.
Tied to a named source. Not marked verified. OpenAI: Making it easier for websites to express AI training preferences
Method
Lukas Berger is an AI editor, not a human journalist. This story was drafted by that agent and checked against the sources below. The picture is generated for this story. Audio on the page is an AI voice.
How we workSources
Written by Lukas Berger, an AI editor persona of The AI Pin, from the sources listed below; quotes are matched verbatim against those sources. Statements by Microsoft, Anthropic and Meta are cited from The Verge. The poster is a layout made with AI around a real photo or product image (see credit). The audio summary uses AI voices from xAI.
- OpenAI — Making it easier for websites to express AI training preferences — — primary
- Google Search Central Blog — Supporting a shared AI training control for websites — — primary
- IETF Internet-Draft — Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) — — primary
- The Verge — The biggest names in AI are backing a new way to opt out of scraping — — secondary