🔍 Read the full analysis: The Shared AI Training Opt-Out: Benefits, Boundaries, And Setup on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will support a shared robots.txt signal asking that a website not be used to train generative AI models. The opt-out is voluntary, does not block AI answers based on live web content, and will take effect on different schedules; Meta has voiced support but has not committed to honoring it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared robots.txt signal asking that a website’s content not be used to train generative AI models. The move offers publishers one rule to communicate that preference across participating services, but it is voluntary, does not cover every use of web content, and will roll out on different timelines.
The proposed rule uses the user-agent name __usage_ai-training with Disallow: / in a website’s robots.txt file. It is set out in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Anthropic standards lead Kevin Kelley and published on October 30. The draft is not an adopted internet standard.
The companies’ plans differ. Anthropic says it has started implementing support, while OpenAI says it aims to implement the signal “in the coming months.” Google says it aims to support it in 2027, and its crawler documentation specifies “early 2027.” Microsoft says the shared training preference will take effect as it crawls from February 1, 2027.
Microsoft is also adding a separate page-level preference for “AI Grounding,” which concerns content used to inform AI answers in Bing, Copilot and Microsoft 365. It says both preferences begin February 1, 2027. Microsoft’s existing NoArchive and NoCache tags will stop counting as training or grounding preferences after March 1, 2027. Meta spokesperson Andy Stone expressed support for publishers’ ability to state preferences, but did not commit the company to honoring this signal.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
One Rule, Limited Control
A common signal could reduce the work for publishers who currently need to manage different crawler names and controls for different companies. A site can state its training preference in one place for the companies that adopt the signal, rather than relying only on separate entries for tools such as GPTBot, Google-Extended and ClaudeBot. That may make the preference easier to maintain, but it does not make compliance automatic.
The rule communicates a request; it is not a technical barrier that prevents a crawler from accessing a site. It also does not determine whether a publisher receives payment, traffic or inclusion in a service. Microsoft says neither choice of preference guarantees any of those outcomes. For organizations managing public content, the key distinction is between expressing a preference and controlling how each company uses or accesses material.
Publishers also need to distinguish model training from AI answers. The shared signal addresses training, not every use of web content to generate answers. Microsoft is offering a separate grounding preference, while the other announcements described in the source material do not establish a common rule for live-answer use.
Popular right now
Breville Bambino Espresso Machine, Brushed Stainless Steel (BES450BSS)
Samsung 85-Inch QN70H Neo QLED 4K Smart TV (2026)
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
As an affiliate, we earn on qualifying purchases.
Existing Crawler Rules Remain
Before this proposal, publishers could use company-specific robots.txt rules to express preferences about crawlers. The shared signal is intended to offer a common option, but the companies have not said that existing controls should be removed immediately. Their current instructions remain relevant while support is introduced at different times.
Google says that if a site uses the new token and has no Google-Extended rule, it plans to treat the shared preference as if the publisher had disallowed Google-Extended. Google also says Google-Extended does not affect eligibility for Google Search. Microsoft says its preferences do not change eligibility for Bing search results. These statements concern the companies’ described controls; they do not establish a universal effect on search or other services.
The proposal is connected to the Internet Engineering Task Force’s AI Preferences working group. However, the draft’s publication does not mean the group has approved it as a standard. At a working-group meeting on October 21, one of the chairs said, “We haven’t declared consensus on anything yet.”
““has already started implementing support for this draft””
— Anthropic
Adoption and Enforcement Remain Open
Who will honor the signal beyond the four announcing companies remains unsettled. The source material reports no commitment from Meta and no statement from Apple, Amazon, xAI or Perplexity. Meta’s public support for publishers expressing preferences is not the same as a promise to follow this particular rule.
There is also no adopted IETF standard yet: the document is an individual draft, and the working group had not declared consensus as of its October 21 meeting. The companies’ statements describe intended or ongoing implementation, but do not establish that every product or service will interpret the signal identically. The signal itself is a preference, not a technical block, and the announcements do not say how compliance will be independently monitored.
The announced scope leaves other questions distinct from training. The common rule does not remove material from models that have already been trained, prevent crawling, or specify how live web content may be used in AI-generated answers. Microsoft has announced a separate grounding control, but the material does not establish an equivalent shared control across all participating companies.
Check Rules Before Each Deadline
Publishers that want to opt out of training can add the shared user-agent and disallow lines to robots.txt, while retaining company-specific rules until each provider’s support is in place. They should check the providers’ documentation for changes and avoid treating the shared line as a substitute for controls covering AI answers, search or other uses.
Microsoft users of NoArchive or NoCache should review their setup before the company’s stated March 1, 2027 cutoff for those tags as training or grounding preferences. Microsoft says its new training and grounding preferences take effect as it crawls from February 1, 2027; Google’s stated target is early 2027, and OpenAI’s is the coming months. Anthropic says support has begun, but describes a gradual rollout across services.
The next milestones are the companies’ actual deployments and any changes to the IETF draft or working-group status. Until those develop, publishers should treat the rule as a shared preference supported by announced company plans—not as a binding standard or a guarantee about how content will be used.
Key Questions
What does the shared AI training opt-out do?
It adds a robots.txt preference asking participating companies not to use a website’s content to train generative AI models. The proposed user-agent is __usage_ai-training, paired with Disallow: /.
Does the rule block crawlers or AI-generated answers?
No. The signal expresses a training preference; it does not technically block crawling. It also does not cover AI answers based on live web content. Microsoft has described a separate grounding preference for certain services.
When will the companies support it?
Anthropic says implementation has begun. OpenAI aims for the coming months, Google aims for 2027 and lists early 2027 in its crawler documentation, and Microsoft says its controls take effect from February 1, 2027 as it crawls. These are company statements and timelines may differ by service.
Has Meta agreed to honor the signal?
Not according to the statement cited in the source material. Meta expressed support for publishers’ ability to state preferences, but its spokesperson did not say the company would follow this specific signal.
Is this an official internet standard?
No. It is an individual Internet-Draft associated with the IETF’s AI Preferences working group. The draft has no formal standing in the IETF standards process, and a working-group chair said consensus had not been declared.
Source: ThorstenMeyerAI.com
