Short answer
Cloudflare and OpenAI say network signals helped ChatGPT's search crawler discover new pages. The signals show when a page has changed and aim to cut unnecessary crawl requests. Being found still does not mean being cited in ChatGPT answers, and crawler permissions continue to decide access.
Highlights
- The pilot ran on roughly 1% of Cloudflare's network and helped OpenAI find and index hundreds of millions of new pages.
- The post gives no data on how often the new pages were cited in ChatGPT or how many visits they sent.
- Allowing OAI-SearchBot while blocking GPTBot keeps a site open to ChatGPT search while declining training use.
2 min readAuthor: UNIT Journal EditorEditor-in-chief: Uğur Deniz İlhan
On 15 October, Cloudflare and OpenAI shared the first results of a research pilot on Cloudflare's blog. According to Search Engine Journal the pilot aims to help OpenAI's search crawler discover web pages more easily, and it ran on roughly 1% of Cloudflare's network.
What did the pilot change in ChatGPT search?
The shared results say the pilot helped OpenAI find and index hundreds of millions of new web pages. Cloudflare's signals include how recently a page was updated, the quality of a site's traffic and what actually changed on the page. Cloudflare announced the pilot on 8 July.
The examples given were pages that are useful even with few links from other sites: a washing machine's user guide, passport requirements and fees, and a restaurant's menu and opening hours.
What data is still missing?
The post does not report how many times the newly found pages were cited in ChatGPT or how many visits they sent. The companies have not said which sites took part. And because OpenAI ranks pages by its own criteria, being found does not guarantee a citation.
Do crawler settings still decide who gets in?
Yes. In the blog post's words, a signal about a page does not grant permission to access it. OpenAI's crawler documentation splits search and training into two separate robots.txt rules: a site that blocks OAI-SearchBot is left out of ChatGPT search results, while a site that allows GPTBot as well lets its content be used for training too.
What should companies in Turkey do?
A business that wants to appear in ChatGPT should first make its site ready for AI answers. In generative engine optimisation work, this news puts the following checks first:
- Open your robots.txt and check that OAI-SearchBot and GPTBot each have their own rule; open to search and closed to training is possible.
- Watch your server logs, and the AI Crawl Control screen if you use Cloudflare, to see which AI crawlers visit; the screen is available on every plan.
- Keep up-to-date pages such as guides, prices, opening hours and terms at their own addresses and refresh them when they change; one of the signals is how recently a page was updated.
- Track ChatGPT traffic separately: measuring visits depends on a UTM tag or a referrer.
Frequently asked
- What happens if I block OAI-SearchBot?
- Your site will not appear in ChatGPT search results and may show up only as a navigational link. You can block GPTBot separately to decline training use of your content as well.
Sources
- Search Engine Journal ·
Follow UNIT Journal
What's new in search, AI and technology, in your feed every day.
