Why does a valid sitemap show 'couldn't fetch'?

Speaking on Google's Search Off the Record podcast, John Mueller and Martin Splitt explained why Search Console can say 'couldn't fetch' even for a valid XML sitemap. According to Search Engine Journal, Google staff are aware of users' frustration over this.

Why can a valid sitemap not be fetched?

The first cause Mueller described is server load: if Google's systems are busy crawling other pages on the site, they may not find time to fetch the sitemap file, and Search Console reports that as 'couldn't fetch' too. SEJ adds that a server returning errors under heavy traffic can have the same result.

The second cause is crawl demand. According to Mueller, if Google's systems see no need to crawl much more from a site, they may skip the sitemap. That demand often rests on the perceived quality of the site, so the matter is not purely technical. He added that if quality improves significantly, Google will start using the file.

How do AI crawlers find a sitemap?

Mueller said most AI training crawlers have no console where a sitemap can be submitted. In a separate SEJ report, he suggests two routes for sites that want their content to appear in AI systems: publish the file under its generic name, sitemap.xml, or focus on RSS feeds. He said he has seen some AI crawlers access his sitemap and RSS files in his own server logs, but did not name them.

He was cautious about llms.txt: Mueller compared the Markdown file to an HTML sitemap and said Google's systems cannot use it as a sitemap. He sees no harm in trying it, but does not advise relying on it.

What should websites do?

The comments show that a red warning in Search Console should not automatically be treated as a technical fault. If the file validates and is accessible, the places to look are the server logs and the quality of the site's content.

• Check your server logs for error responses to Googlebot requests; if the server fails at busy times, fix the infrastructure first.
• Publish the sitemap as sitemap.xml and declare it in robots.txt; avoid a custom name unless you have a reason to keep it private.
• For sites that publish news or blog content, link the RSS feed from the page head; it gives AI crawlers an extra route to discover content.
• Do not use llms.txt in place of a sitemap; if you want to try it, keep it alongside the XML sitemap.