Read the full analysis: The Limits Of The Shared AI Training Opt-Out, Explained on ThorstenMeyerAI.com
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced plans to recognize a shared robots.txt signal asking that website content not be used to train generative AI models. The signal is voluntary, does not cover every use of web content, and will take effect on different schedules; Meta has expressed support but made no commitment to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they plan to recognize one shared robots.txt signal asking companies not to use a website’s content to train generative AI models. The move could simplify a publisher’s opt-out instructions, but it is a voluntary request rather than a technical block, and each company has its own implementation schedule.
The proposed signal uses the robots.txt user-agent label __usage_ai-training with Disallow: /. It is set out in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” published by Anthropic standards lead Kevin Kelley on October 30. The draft builds on work by the Internet Engineering Task Force’s AI Preferences working group, but it is not an adopted internet standard.
Anthropic says it has started implementing support, though it expects the work to take time across its services. OpenAI says it aims to honor the signal for foundational generative AI training “in the coming months.” Google aims to support it in 2027; its crawler documentation gives the timing as early 2027. Microsoft says it will honor the training preference from February 1, 2027, as it crawls websites.
Microsoft also announced a separate page-level preference for AI Grounding—the use of web content to inform answers in Bing, Copilot and Microsoft 365. It says both new preferences will take effect on February 1, 2027. Microsoft’s existing NoArchive and NoCache controls will stop counting as training or grounding preferences after March 1, 2027. The companies’ existing crawler-specific controls, including GPTBot, Google-Extended and ClaudeBot, remain relevant during the rollout.
Status 31 Oct 2026
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
OpenAI calls it a “voluntary effort”.The vocabulary draft does not “ensure that preferences are followed”.The draft “has no formal standing in the IETF standards process”.Private agreements may override it, the draft says.
Who else?
Meta supports a common signal, but did not say whether it will honor it.Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
What the Shared Signal Changes
For publishers, a shared instruction could reduce the need to maintain separate training opt-outs for every participating company. A site can add one rule expressing its preference, while continuing to use company-specific controls where needed. That may make the preference easier to communicate, especially during the period when providers are adopting it at different speeds.
But the signal does not make a site’s content inaccessible. Robots.txt communicates a preference; it does not technically prevent a crawler from accessing a page, and the draft itself does not create a mechanism to enforce compliance. The practical effect depends on whether a company implements and honors the request. For publishers deciding what to publish or license, that distinction matters: an opt-out signal is not equivalent to a guaranteed exclusion from training.
The scope is also narrower than a general ban on AI use. The shared rule concerns model training, not every way a provider may use web pages. Microsoft’s separate grounding preference illustrates that content used to produce or inform live AI answers can be treated differently. The signal does not remove material from models already trained on it, set payment terms, or promise that a site will appear in search results or receive traffic. Microsoft said its preferences do not guarantee inclusion, traffic or payment.
From Separate Rules to One Request
Before this proposal, website owners seeking to express preferences had to contend with different mechanisms. OpenAI, Google and Anthropic offered crawler-specific controls, while Microsoft used page tags including NoArchive and NoCache for certain preferences. A publisher could therefore need several instructions, and a rule aimed at one company did not automatically communicate the same preference to another.
The shared signal comes from an individual Internet-Draft, not a completed standards process. The draft’s publication records a proposal; it does not mean the IETF has approved it or that all AI companies have agreed to follow it. At an October 21 working-group meeting, one of the group’s chairs said, “We haven’t declared consensus on anything yet.” The companies’ announcements are commitments or plans made by those companies, not evidence of universal industry adoption.
Meta spokesperson Andy Stone said the company “strongly support[s] publishers’ ability to express preferences” and called a common signal an important step. That statement expresses support for publishers’ ability to communicate preferences, but does not commit Meta to honor this particular signal. The source material also reports no statement from Apple, Amazon, xAI or Perplexity about whether they will follow it.
““has already started implementing support for this draft””
— Anthropic
Adoption and Enforcement Remain Open
Industry-wide adoption is not confirmed. Anthropic, OpenAI, Google and Microsoft have stated plans or implementation work, but the available announcements do not establish that every service operated by each company will honor the preference on the same date. Anthropic itself says rollout across its services will take time. Meta has voiced support without specifying whether it will comply, and the position of several other major AI companies is unknown.
The signal’s final status is also unsettled. The draft has no formal standing in the IETF standards process, and the working group had not declared consensus at its October 21 meeting. It remains unclear whether the proposal will be revised, adopted as a standard, or changed as implementation proceeds. Nor does the material establish a way for publishers to verify that all training systems have excluded opted-out content.
Publishers should also distinguish the signal’s stated purpose from broader questions about content use. It addresses training, while AI answer generation, search indexing and other uses may have separate controls or policies. The companies’ announcements do not establish a shared rule for all those activities.
Rollout Dates and Publisher Choices
The next developments are the companies’ implementation work and any changes to the draft or its status in the IETF process. Anthropic says its support is underway; OpenAI’s target is the coming months; Google’s stated aim is early 2027. Microsoft’s announced date is February 1, 2027, when it plans to begin honoring the training and grounding preferences as it crawls, followed by the March 1, 2027 cutoff for treating NoArchive and NoCache as those preferences.
For now, website owners who want to express a training preference can add the shared user-agent group and disallow rule, but should not assume that doing so blocks crawling or substitutes immediately for existing company-specific controls. They can keep those rules in place while providers adopt the new signal. Sites that use Microsoft’s current page tags should review the announced March deadline and decide whether to add Microsoft’s separate grounding preference as well.
The announcement concerns website content, not a person’s conversations with an AI assistant. Users seeking to manage whether their own chats are used for model improvement need to consult the relevant service’s account or privacy settings. Whether the shared web signal becomes a widely honored convention will depend on adoption beyond the four companies that announced plans.
Key Questions
What does the shared AI training opt-out signal say?
It is a robots.txt rule using User-Agent: __usage_ai-training and Disallow: /. The rule communicates a website owner’s preference that content not be used to train generative AI models.
Does the signal stop AI companies from crawling a website?
No. It is a voluntary preference signal, not a technical barrier that prevents access. Its effect depends on whether a company adopts and honors it.
Does it opt a site out of AI-generated answers?
Not by itself. The shared signal addresses training. Some uses for AI answers or grounding have separate controls; Microsoft, for example, announced a distinct page-level AI Grounding preference.
When will the companies honor the signal?
Anthropic says implementation has started, OpenAI aims for the coming months, and Google aims for early 2027. Microsoft says its preferences take effect February 1, 2027. These are different company schedules, not one shared launch date.
Has Meta agreed to honor the opt-out?
Meta has publicly supported publishers’ ability to express preferences and called a common signal an important step. The statement described in the source material did not confirm that Meta will honor this specific signal.
Source: ThorstenMeyerAI.com