Google, Microsoft, OpenAI, Anthropic Back One AI Opt-Out
By @sharedot · · 7 pages
- AI
- AI Policy
- Robots Txt
- Aipref
Google, Microsoft, OpenAI, and Anthropic jointly backed a single robots.txt token letting sites opt out of AI training across all providers at once.
Four rivals, one signal
On October 30, 2026, Google, Microsoft, OpenAI and Anthropic each announced support for a shared robots.txt policy token, `__usage_ai-training`, that lets a website refuse the use of its content for generative AI training across every participating provider at once.
Until now, an operator had to add a separate entry for each AI company's crawler, and each company used its own naming.
Google's announcement came from Search Advocate John Mueller on the Search Central Blog, Microsoft posted on its Bing Blog through principal product manager Krishna Madhavan, OpenAI published in its Policies and Procedures section, and Anthropic engineer Kevin Kelley submitted the underlying Internet-Draft to the IETF's AIPREF working group the same afternoon.
Why the joint pledge breaks pattern
It is unusual for four direct competitors to coordinate on the same day around a mechanism that constrains their own access to training data, and it follows pressure from publishers, civil society groups and regulators for industry-wide controls.
Google said the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act, which recognizes robots.txt as a primary method for publishers to express machine-readable preferences.
OpenAI encouraged other AI providers to honor the signal, suggesting the companies hope it becomes a durable standard rather than a four-company pact.
What the draft actually specifies
The ten-page draft, authored by Kevin Kelley of Anthropic and expiring May 3, 2027, introduces no new directive names or grammar: it reuses the path language, longest-match precedence and group structure of RFC 9309, the Robots Exclusion Protocol.
Operators write an ordinary robots.txt group naming the policy token instead of a crawler, so a site could opt out of training site-wide while allowing paths under /press-kit/.
Two clauses matter most: wildcard groups (`User-Agent: *`) must not be interpreted as matching the token, so a blanket block-all file expresses no preference at all, and private agreements may override expressed preferences.
The token is about use, not access — it governs whether fetched content can modify a generative model's parameters, not whether crawlers may fetch.
Rollout timelines and Microsoft's extras
No company reads the signal yet. Microsoft gave the firmest date: it will honor the new preferences starting February 1, 2027, and will retire its older NoArchive and NoCache tags for training and grounding, with a transition window in Bing Webmaster Tools opening December 15, 2026.
Meta Platforms expressed support for the idea but has not committed to implementing it, per SiliconANGLE.
Stakes for publishers and search
The signal covers only generative training: ranking and classification systems, and retrieval at answer time, fall outside it, so search rankings in Google and Bing are unaffected.
Microsoft built a separate AI Grounding preference — including a DisallowExtended option — to let sites control whether their pages feed AI answers, warning that opting out reduces brand visibility in AI experiences.
Google plans to treat the shared token as if a publisher had disallowed Google-Extended, its own control since 2023, meaning sites that allow Google-Extended remain eligible for clicks through grounding in Gemini Apps.
The key open question is whether a token expressed today binds data collected before a site added it — the draft says preferences attach at the moment of collection and does not address previously gathered content.