Google, Microsoft and OpenAI have announced plans to honor a common robots.txt signal that lets website owners state whether their content can be used to train AI models. If adopted, a site could express one training preference to every participating provider instead of writing separate rules for each company’s crawler.
Where the Proposal Comes From
The signal comes from an Internet Draft submitted to the AI Preferences working group at the IETF by Kevin Kelley of Anthropic. It is listed as an individual draft, so it does not represent official IETF standardization. Kelley has indicated on the working group’s mailing list that Anthropic has begun work on supporting it, though no announcement from Anthropic had appeared as of October 30.
None of the three companies supports the signal yet. Each has described its own timeline and approach.
How the Signal Works
The draft does not add a new directive to robots.txt. Site owners write an ordinary rule group under a policy token called __usage_ai-training, using the familiar Allow and Disallow rules. For example, a site could block AI training across the whole site while still permitting the /press-kit/ folder:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
The draft draws on the working group’s vocabulary, which defines AI training as using content to modify a generative AI model’s learned parameters. Non-generative uses, such as classification or ranking, fall outside that definition.
A few details limit its scope:
- A wildcard group (User-Agent: *) never matches the token.
- A robots.txt file with no __usage_ai-training group is treated as expressing no preference, which describes every robots.txt file published today.
- The token does not control crawling.
Google’s Approach
Google plans to read the shared opt-out as if the site had disallowed Google-Extended, provided the site has not specified Google-Extended itself. Google is aiming for support in 2027, and its crawler documentation says early 2027.
Google-Extended governs whether content can train future Gemini models and whether it can be used for grounding in Gemini Apps and on Vertex AI. Grounding means supplying content from the search index to the model while it answers a question. Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode are controlled separately through the Search generative AI control in Search Console.
Microsoft’s Approach
Microsoft is adopting the shared line for training and adding a separate AI Grounding preference, which is set at the page level and governs whether content can inform AI-generated answers. A meta tag or HTTP header can also set the training preference for a single page, overriding robots.txt for that page.
Grounding defaults to Allow. Disallow keeps a page out of grounding entirely, while a third option, DisallowExtended, permits grounding in Microsoft’s own tools such as Copilot and Bing’s AI Mode but not through the grounding services Microsoft provides to other businesses. Microsoft says either restriction can reduce a brand’s visibility in AI answers but will not affect its listing in Bing search results.
These new preferences replace the existing NoArchive and NoCache tags for training and grounding. The timeline is:
- December 15: A temporary preference opens in Bing Webmaster Tools for sites still using the old tags on that date and wanting to stay out of grounding.
- February 1, 2027: Microsoft begins honoring the new preferences as it crawls.
- March 1, 2027: The old tags stop being recognized for training or grounding. Pages without a preference by then default to allowing grounding until they are recrawled.
Once Microsoft detects either new preference on a site, it ignores any NoArchive or NoCache tags remaining there.
OpenAI and Anthropic
OpenAI says it intends to honor the signal within the next few months, and its current robots.txt policy remains in effect for now. Neither its announcement nor its crawler documentation explains what happens if a site’s shared rule conflicts with a GPTBot rule.
Anthropic’s crawler help page lists three controls at present: ClaudeBot, Claude-User and Claude-SearchBot. Kelley has said that supporting the signal across all of Anthropic’s services will take time.
What This Means for Website Owners
A single robots.txt group could eventually take the place of several per-company training rules, but the opt-out will not behave identically everywhere. At Microsoft, training and grounding are separate settings. At Google, both fall under Google-Extended, and the documentation describes no way to keep content out of Gemini training while still allowing Gemini Apps to use it for grounding.
Microsoft is the only company whose announcement replaces controls sites already use. Google, OpenAI and Kelley have all said existing controls will keep working, so sites relying on NoArchive or NoCache to limit how Copilot uses their content have changes to make before the March 2027 cutoff.
What Comes Next
OpenAI has encouraged other AI providers to honor the signal, and Kelley has said he hopes more will join. Google and OpenAI have both said they will share further details as their support rolls out. Kelley’s draft expires on May 3, 2027, unless a new version replaces it, and he plans to propose his design as a change to the working group’s Content-Usage draft.
Note: Nothing is live yet. Opting out of AI training without losing AI answer visibility is possible under Microsoft's planned approach, but not under Google's described Google-Extended approach.
Audit your PPC & SEO strategy with TEQTOP.
Let our specialists review your Google Ads compliance, integrate paid and organic strategy, and protect your campaigns.