Four AI giants pledge one opt-out signal. Only Anthropic honors it today
What happened
Google, OpenAI, Anthropic and Microsoft have committed to honor a single robots.txt directive that lets publishers refuse use of their content for generative AI training. Publishers add a user agent named "__usage_ai-training" with Disallow and Allow rules, instead of listing each company's crawler separately. Support is staggered: Anthropic already honors the signal, Microsoft says February 1, 2027, Google "early 2027," and OpenAI "in the coming months," per Engadget.
What we know
- The directive follows a proposal under consideration at the Internet Engineering Task Force (IETF), the open internet standards body, written by Anthropic's technical lead for standards, Kevin Kelley, per Engadget.
- Rollout dates differ: Anthropic already supports it, Microsoft starts February 1, 2027, Google says early 2027, and OpenAI gave only "the coming months," per Engadget.
- Publishers can block AI-training crawlers site-wide and carve out exceptions, such as allowing a /press-kit/ path, per Engadget.
- The directive does not block crawling altogether and does not remove pages from Google Search. Google will read it as a stop signal for its Google-Extended user agent, which gathers material for training Gemini models, per Engadget.
- It cannot undo past scraping. Content already crawled stays with the companies, per Engadget.
- The Verge, via Techmeme, reports that the four companies say they will adopt a standard letting publishers apply AI training preferences across multiple crawlers.
What we don't know yet
- The IETF proposal is still under consideration, and the sources do not say whether it will be adopted as a formal standard.
- It is unconfirmed how each company will enforce compliance, and whether any independent audit will check it.
- OpenAI has given no firm date.
- Whether the directive will cover other AI developers beyond these four is unstated.
- The sources do not say whether training data already collected will be retained, or if any such data could be removed on request.
Why it matters
Publishers currently need separate robots.txt rules for each AI company's crawler. A shared signal cuts that to one rule, but it only protects future crawls. Until Google, Microsoft and OpenAI go live, most of the four companies' crawlers will still need their own opt-outs.
Claims
- ConfirmedGoogle, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive for opting out of generative AI training.
- ConfirmedThe directive is based on a proposal under consideration at the IETF, written by Anthropic's Kevin Kelley.
- ConfirmedAnthropic already supports the directive.
- ConfirmedMicrosoft said it will begin supporting the directive on February 1, 2027.
- ConfirmedGoogle said it will support the directive in early 2027.
- ConfirmedOpenAI pledged to implement the signal in the coming months, without a specific date.
- ConfirmedThe directive does not remove content from Google Search and does not undo past scraping.
Related coverage
Why you can trust this story
75%Source map · 2 outlets / 2 articles
- Engadget
- Article 1engadget.comreport
- The Verge (via Techmeme)
- Article 1theverge.comreport
How this credibility score is calculated
- Source reliability
- Corroboration
- Primary source
- Claims checked against the source
- No contradiction
- Settled
- Claim attribution
- AI disclosure
Checked claims were verified against the cited source text. Weights are fixed and explainable.
Human accountability
- 2 independent origins / 2 sources
- Drafted by an automated system; a human editor approves every story before it is published.
Verification ledger · 7
- Claim: Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive for opting out of generative AI training.Status: ConfirmedVerification trailVerifiedChecked by verification engine
At least one checkable detail was matched against the full text of the cited source.
- Claim: The directive is based on a proposal under consideration at the IETF, written by Anthropic's Kevin Kelley.Status: ConfirmedVerification trailVerifiedChecked by verification engine
At least one checkable detail was matched against the full text of the cited source.
- Claim: Anthropic already supports the directive.Status: ConfirmedVerification trailVerifiedChecked by verification engine
At least one checkable detail was matched against the full text of the cited source.
- Claim: Microsoft said it will begin supporting the directive on February 1, 2027.Status: ConfirmedVerification trailVerifiedChecked by verification engine
At least one checkable detail was matched against the full text of the cited source.
- Claim: Google said it will support the directive in early 2027.Status: ConfirmedVerification trailVerifiedChecked by verification engine
At least one checkable detail was matched against the full text of the cited source.
- Claim: OpenAI pledged to implement the signal in the coming months, without a specific date.Status: ConfirmedVerification trailVerifiedChecked by verification engine
At least one checkable detail was matched against the full text of the cited source.
- Claim: The directive does not remove content from Google Search and does not undo past scraping.Status: ConfirmedVerification trailVerifiedChecked by verification engine
At least one checkable detail was matched against the full text of the cited source.
Update log · 0
AI accelerates. High-risk topics wait for a human editor.