Short answer
Google, Microsoft and OpenAI have said they plan to honour a single shared robots.txt line for AI training preferences, so sites will not need separate rules for each company's crawler. None of them supports the line today, the timelines differ, and Microsoft manages appearing in AI answers through a separate setting.
Highlights
- The shared line adds no new directive to robots.txt, and existing rules do not count as a training preference.
- Google plans to treat it as a Google-Extended block when a site does not specify Google-Extended.
- Microsoft is replacing NoArchive and NoCache with two new preferences for training and for appearing in answers.
2 min readAuthor: UNIT Journal EditorEditor-in-chief: Uğur Deniz İlhan
As Search Engine Journal reports, Google, Microsoft and OpenAI plan to read AI training preferences from one shared robots.txt line. The line comes from an individual draft that Anthropic's Kevin Kelley submitted to the IETF's AI Preferences working group, so it is not an official standard.
How does the shared line work?
The draft adds no new directive to robots.txt. Sites write ordinary User-Agent, Allow and Disallow rules for a policy token called "__usage_ai-training". For example, a site can opt the whole site out of training while allowing only a press kit folder.
One detail matters: today's robots.txt files count as expressing no preference, a "User-Agent: *" group never matches the token, and the token does not control crawling.
When will the companies support it?
According to the timelines in the report, none of the three supports the line yet. Google aims to support it in 2027 and plans to treat a site that uses the line without specifying Google-Extended as if it had disallowed Google-Extended. OpenAI aims to honour the signal in the next few months.
Microsoft takes a more detailed route: it adopts the shared line for training and adds a separate page-level "AI Grounding" preference for answers. It will honour the new options from 1 February 2027 and stop recognising NoArchive and NoCache for this purpose after 1 March 2027. A temporary preference in Bing Webmaster Tools for sites still using the old tags opens on 15 December.
What should sites do?
Settle your goal before writing the preference: opting out of training and appearing in AI answers are not the same decision, and Microsoft notes that either restriction can reduce brand visibility in AI answers. Treat it together with your GEO (generative engine optimisation) work.
- List your current robots.txt rules and any NoArchive or NoCache tags on your pages.
- Decide in writing, with the business, which content types (blog, pricing, product pages) to keep out of training.
- If you use the old tags and want to stay out of Microsoft's answers, check the Bing Webmaster Tools preference after 15 December.
- Once the line is live, test it and monitor the effect in Search Console and Bing Webmaster Tools.
Frequently asked
- Does the shared line affect Google Search rankings?
- According to Google's documentation cited in the report, Google-Extended does not affect inclusion in Google Search. AI Overviews have a separate control in Search Console.
Sources
- Search Engine Journal ·
Follow UNIT Journal
What's new in search, AI and technology, in your feed every day.
