🔍 Read the full analysis: The Shared AI Opt-Out: How To Set It And Understand Its Limits on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal asking them not to use a site’s content to train generative AI models. It is voluntary, does not block crawling or AI answers, and will take effect on different schedules; Meta has voiced support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared instruction in a website’s robots.txt file asking companies not to use its content to train generative AI models. The move offers publishers one common signal instead of separate training rules for each company, but it is a voluntary preference, not a technical block, and the companies have set different implementation timelines.
The proposed rule uses the user-agent name __usage_ai-training with Disallow: /. A publisher can add that group to robots.txt to express a preference against training use. The signal appears in an Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Anthropic standards lead Kevin Kelley and published October 30. The draft is not an adopted internet standard.
Anthropic says it has begun implementing support, although its announcement says it will take time to add the signal across its services. OpenAI says it aims to support it “in the coming months.” Microsoft says it will honor the shared training preference beginning February 1, 2027, as it crawls websites. Google aims to support it in 2027; its crawler documentation specifies “early 2027.”
Meta has not made the same commitment. Its communications chief, Andy Stone, said the company supports publishers expressing preferences and called a common signal an important step, but did not say Meta would honor this one. The source report also found no statement on the signal from Apple, Amazon, xAI or Perplexity.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
What Publishers Can Control
The shared rule could reduce the work of maintaining separate AI-training instructions for participating companies. For publishers, that makes it easier to express a consistent preference in one place. But the signal should not be treated as an access-control mechanism: robots.txt communicates a request, and the announcement does not describe a technical system that prevents a company from retrieving or using material.
The distinction between training and other uses matters. The new signal is for foundational model training; it does not opt a site out of AI-generated answers that draw on live web content. Microsoft has announced a separate, page-level preference for “AI Grounding,” its term for web content used to inform answers in Bing, Copilot and Microsoft 365. That is a separate choice, not an effect of the shared training rule.
Publishers also should not infer that opting out will remove material from models already trained on it, generate payment, or determine whether a page appears in search. Microsoft says its preferences do not change eligibility for Bing search results, and Google says Google-Extended does not affect Google Search. Microsoft also states: “Neither choice guarantees inclusion, traffic, or payment.”
Products Worth Considering
LG UltraGear 32GX870A-B 32-inch 4K OLED Gaming Monitor
LG UltraGear 45GX900A-B 45-Inch Curved OLED Gaming Monitor
LG UltraGear 27G640A-B 27-Inch QHD IPS Gaming Monitor, 300Hz, 1ms, G-SYNC Compatible, FreeSync Premium, DisplayHDR 400
As an affiliate, we earn on qualifying purchases.
Before this proposal, website operators generally needed company-specific controls to express training preferences. Examples include GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. Microsoft has used page-level tags called NoArchive and NoCache for relevant preferences. The new signal is intended to give participating companies a common instruction while existing controls remain in place during the rollout.
Google says that if a site uses the new token and has no Google-Extended rule, it plans to interpret the shared preference as though the publisher had disallowed Google-Extended. OpenAI says its existing crawler controls will continue to apply. Microsoft says its new training and grounding preferences begin February 1, 2027, and that NoArchive and NoCache stop counting as training or grounding preferences after March 1, 2027.
The proposal is linked to the Internet Engineering Task Force’s AI Preferences working group, but it remains an individual draft. At a working-group meeting on October 21, a chair said, “We haven’t declared consensus on anything yet.” The draft therefore documents a proposal and company plans; it does not establish a formal IETF standard.
““has already started implementing support for this draft””
— Anthropic, in its announcement about the draft
Where Adoption Still Stands
Compliance remains voluntary, and the announcements do not establish a shared enforcement process or penalties for ignoring the preference. The companies have also not all provided the same level of detail about rollout timing or which products and services will recognize it. Anthropic’s crawler help page still described its existing per-crawler controls on October 31, according to the source report.
Meta’s support statement does not confirm implementation, and the available source material reports no commitments from several other major technology companies. It is also not yet clear how quickly publishers’ instructions will be picked up as participating companies crawl sites, or how the proposed draft may change during the IETF process. The signal does not address AI answers based on live web content except where a separate company control applies.
Rollout Dates and Site Updates
Publishers who want to express a preference can add the shared user-agent group and keep their existing company-specific rules while providers implement the change. They should check each company’s published crawler documentation rather than assume that an announcement means the control is already active. Sites using Microsoft’s NoArchive or NoCache tags should review Microsoft’s planned replacement controls before March 1, 2027, when those tags stop counting as training or grounding preferences under its announcement.
The next milestones are the companies’ implementation updates and any further development of the Internet-Draft within the IETF’s AI Preferences work. Until providers confirm their support is active, and explain its scope, the practical effect of the shared rule will vary by company. Publishers will still need to make separate decisions about training, AI answers and search; the new instruction addresses only the first of those by itself.
Key Questions
What should a publisher add to robots.txt?
The proposed group uses User-Agent: __usage_ai-training followed by Disallow: /. It expresses a preference against using the site’s content for generative AI training by companies that honor the signal.
Does the rule block AI companies from crawling a website?
No. The signal communicates a training preference; the announcements do not describe it as a technical barrier to crawling. Publishers may still need company-specific crawler rules for other purposes.
Does opting out stop AI answers from using a site?
Not by itself. The shared signal covers model training, not answers generated using live web content. Microsoft has separately announced an AI Grounding preference for certain services.
When will the companies honor the signal?
Anthropic says implementation has started, OpenAI aims for the coming months, Google aims for early 2027, and Microsoft says its controls begin February 1, 2027. These are company statements, and the rollout may vary across products.
Has the signal become an official internet standard?
No. It is an individual Internet-Draft associated with the IETF’s AI Preferences work. The source report says the working group had not declared consensus on a proposal.
Source: ThorstenMeyerAI.com
