Search & SEO

Brave sells its search index to AI: is your site in it?

Brave sells access to its search index to AI companies and does not name its crawler. Why robots.txt is not enough for your site, and what to do.

Short answer

Brave Search sells access to its search index to AI companies through the Brave Search API. Because Brave does not announce a separate user agent for its crawler, you cannot block it alone with robots.txt. You can check for yourself whether your site is in the index; your visibility in AI answers may draw on it too.

Highlights

  • Brave sells access to its search index to AI companies such as Cohere, Mistral, Perplexity and You.com.
  • Brave's crawler uses no separate user agent name, so Brave cannot be blocked without also blocking Google.
  • You can see in seconds whether your site is in the index by using the site: operator on Brave Search.
A laptop showing a Google search, a phone and a notebook on a wooden desk
Photo: Caio / Pexels

3 min readAuthor: UNIT Journal EditorEditor-in-chief: Uğur Deniz İlhan

AI assistants do not always find fresh information by crawling the web themselves; some pay to query another search engine's index instead. According to an analysis published by Search Engine Journal, one of those indexes belongs to Brave Search, and your site is probably in it.

Who does Brave sell its index to?

Brave sells access to its search index to AI companies through its Brave Search API. Brave's own pages say the API is used by names such as Cohere, Mistral, Perplexity and You.com. According to the same source, the index holds 30 billion pages, with millions of new pages added every day.

On 2 November, the New York Times filed a lawsuit against Brave. The complaint alleges that Brave copied pages without identifying its crawler and sold the content to AI companies. Brave has not yet responded; for now these are allegations. The outcome of the case is not the subject of this article, but it is a good reason to review how your site relates to this index.

Why can't you block only Brave with robots.txt?

A robots.txt file works by crawler name. OpenAI and Anthropic publish the names of their crawlers, so you can admit one and turn the other away. Brave's help page, however, says its crawler does not advertise a differentiated user agent. The reason given: some sites admit only Google. Brave follows the rules given to Googlebot.

The upshot: there is no robots.txt line that lets Google in and keeps Brave out. The noindex tag, cited as the way out of the index, would drop the page from Google as well.

Is your site in Brave's index?

The check is simple: on search.brave.com, type site: followed by your domain. Every page that appears is in the index Brave sells. The article's author tried this on their own site and saw a stored copy of the site in the results.

What should businesses in Turkey do?

The source gives no Turkey-specific data, but the lesson holds for sites of any size: content that is visible on Google can also reach AI answers through a different index. Take this index into account in your generative engine optimisation work:

  • Run a site: check for your domain on Brave Search; note which pages are in the index and whether the stored descriptions are current.
  • Measure regularly how your brand appears in AI answers; referral figures in traffic reports show only part of this visibility.
  • If you are a publisher or have paid content, decide in a written content policy which pages stay open; do not rely on robots.txt alone.
  • Keep titles and descriptions of your key pages up to date; an index can serve a stored copy rather than the live page.

Frequently asked

Can I block Brave with robots.txt?
Because Brave does not announce a separate user agent name, you cannot write a line that targets only Brave. Rules you set for Googlebot apply to Brave as well.
What do I do to leave the index?
Brave's help page suggests the noindex tag, which also removes the page from Google. Before deciding, weigh which pages genuinely need it.

Sources

  1. Search Engine Journal ·

Follow UNIT Journal

What's new in search, AI and technology, in your feed every day.

Related articles

← Back to UNIT Journal

Let us measure
your visibility today.

We map your current search visibility and your standing inside generative engines. Free, one page, real data.

Request an Analysis
Get a Quote