Short answer
Google's Gary Illyes said robots.txt is an access preference protocol: it tells a crawler which URLs it may request but not what the content may be used for once crawled. To keep a specific AI bot out entirely, a site must name that bot's user-agent token explicitly in the file.
Highlights
- robots.txt governs which URLs can be crawled, not how crawled content is used afterwards.
- Blocking an AI bot means naming its user-agent token explicitly in the file.
- A Content-Usage rule for stating training preferences exists only as a draft so far.
2 min readAuthor: UNIT Journal EditorEditor-in-chief: Uğur Deniz İlhan
As Search Engine Journal reports, Google's Gary Illyes used a new Search Central video to explain what robots.txt does and does not do for AI training. The question was simple: can a site allow AI crawling and still keep the same information out of model training?
Does robots.txt stop AI training?
Not by itself. According to Illyes, robots.txt as defined in RFC 9309 is an "access preference protocol": it tells a crawler which URLs it may request and says nothing about what happens to the data afterwards. If you want to keep a particular AI crawler out completely, you need to target its user-agent token explicitly in the file.
A made-up directive shown in the video ("No-AI-Chicken-Training") would simply be ignored by a parser. Google's specification page lists four supported elements: User-Agent, Allow, Disallow and Sitemap; invalid lines are skipped.
What does Google-Extended control?
Google-Extended is a robots.txt token with no user agent of its own. Google crawls with its existing crawlers; the token governs whether that content can be used to train future Gemini models and for grounding in Gemini apps. Google's page says it does not affect inclusion in Google Search and is not a ranking signal.
There is still no standard way to state a usage preference independently. An IETF draft co-written by Illyes proposes a Content-Usage rule for robots.txt and an HTTP header; it remains a working group document, and a separate vocabulary draft defining the usage labels reflects no consensus within the working group.
What should sites in Turkey do?
Settling your stance on AI bots in robots.txt means weighing content use in training against the visibility goals of your GEO (generative engine optimisation) work: blocking a bot entirely can also make it harder to appear in the answers it powers. The decision is a business one, not a technical one.
- Open your current robots.txt and write down which AI bots are named and what each one is blocked from.
- Name each bot you want to block by its own user-agent token; a generic "no AI" line will not work.
- Write paths containing Turkish characters or emoji in percent-encoded form: Google handles raw characters the same way, but older or different crawlers may misread them.
- Review the blocking decision alongside your AI-answer visibility goals at least once a year.
Frequently asked
- Does blocking Google-Extended affect Google Search rankings?
- According to Google's documentation, no; the token does not affect a site's inclusion in Google Search and is not a ranking signal. It only governs Gemini training and grounding use.
Sources
- Search Engine Journal ·
Follow UNIT Journal
What's new in search, AI and technology, in your feed every day.
