Short answer
Google added an "Only index content meant for Search" section to its spam guide. Sites where users create shareable pages, generated outputs or tools are advised to apply noindex, or block crawling in robots.txt, by default for pages made for direct sharing. Using robots.txt alone can backfire, because Googlebot then cannot read a noindex rule.
Highlights
- The new section covers link-shared pages, generated outputs and tool pages.
- Google suggests the noindex rule, or a default robots.txt block, for these pages.
- On a page blocked in robots.txt, Googlebot never sees a noindex rule.
2 min readAuthor: UNIT Journal EditorEditor-in-chief: Uğur Deniz İlhan
As Search Engine Journal reports, Google added a new section to its spam guide, updated on 26 October. It targets platforms where users can publish pages, outputs or web apps.
What does Google recommend in the new section?
The section asks services where users can create shareable pages, generated outputs or interactive tools whether that content is meant to be found through search engines or only reached by people with a link. For pages created for direct sharing, collaboration or utility, it advises applying the noindex rule (through a robots meta tag or an X-Robots-Tag header) or blocking crawling in robots.txt by default.
Should you use noindex or robots.txt?
Drawing on Google's own documentation, the source reminds readers that the two methods can undermine each other. A page blocked in robots.txt can still be indexed if other sites link to it, in which case the URL and link text can appear in results without a description. A noindex rule only works if the page is not blocked in robots.txt, because the crawler never sees the rule otherwise.
The "Indexed, though blocked by robots.txt" status in Search Console signals this clash. Google's help page advises removing the block and using noindex.
What should websites do?
If your site offers dashboards, quotes, report pages, order-tracking screens or AI outputs that users share by link, this guidance applies to you directly.
- List every page type that is shared by link and decide whether each one should be findable in search.
- Add noindex to those that should not appear, and do not also block them in robots.txt, so Googlebot can read the rule.
- Watch the Page Indexing and Performance reports in Search Console for unexpected jumps in indexed pages or impressions.
- Track newly published public pages on your platform and set an alert if a few users generate most of the indexable URLs.
These checks are part of designing index control from the start in web and e-commerce projects that host user-generated content and tool pages.
Frequently asked
- Does blocking in robots.txt remove a page from Google?
- No. According to Google's documentation, a page blocked in robots.txt can still be indexed if other sites link to it. To keep a page out of search, use a noindex rule and leave the page open to crawling.
Sources
- Search Engine Journal ·
Follow UNIT Journal
What's new in search, AI and technology, in your feed every day.
