Short answer
llms.txt is a plain-text file placed in a website's root directory that summarises, in Markdown, what the site is and which of its pages matter most, for AI assistants. It is not an official standard but a proposed convention. It does not control access or secure citations; it works as a readable site summary for AI.
8 min read
What exactly is llms.txt?
llms.txt is a proposal that takes its name from familiar root-directory files such as robots.txt and sitemap.xml. The file is published at yourdomain.com/llms.txt and offers a short summary designed to help tools built on large language models understand a site quickly. It contains the site's name, a one-paragraph description and lists of links, with short descriptions, to its most important pages.
The reason it was proposed is simple: a web page is designed for people. It is full of menus, cookie notices, scroll effects, footer areas repeated on every page and scripts. For a language model's limited context window, that noise gets in the way of the actual information. llms.txt aims to gather the same information in one place, unadorned, as plain text and Markdown.
In short, llms.txt is not an access rule or a ranking signal but an introduction: the answer to "what should someone looking at this site for the first time know, and in what order?"
How is llms.txt different from robots.txt and sitemap.xml?
The three files sit in the same directory but do different jobs. Mixing them up creates the wrong expectations; the most common misconception is that llms.txt steers or blocks AI crawlers.
| File | What does it do? | Who reads it? | Is it binding? |
|---|---|---|---|
| robots.txt | States which crawler may access which parts of the site | Search engine and AI crawlers | A directive for compliant crawlers; it is the established way to control access |
| sitemap.xml | Lists the URLs you want crawled and when they were last updated | Search engines | A suggestion; it makes discovery easier |
| llms.txt | Summarises in plain language what the site is and its key pages | Tools and assistants that use language models | No; whether it is read depends on the tool |
The practical upshot: if you want to restrict or allow an AI crawler's access, the right place is robots.txt and, where needed, your server and content delivery network settings. llms.txt only answers the question "how should someone arriving here understand the site?"
Is llms.txt a standard, and where does it stand today?
No, it is not a standard. The format emerged as a proposal published at llmstxt.org and has gradually become a widespread convention. It is not a specification approved by any standards body; it has been adopted mostly by projects that publish software documentation and by some content management tools.
The real question is whether AI systems actually read the file. There is no general, official commitment on this from the major providers. Google, however, has been explicit about its own AI features: according to Google Search Central documentation, you do not need new machine-readable files, AI text files or additional markup to appear in AI Overviews and AI Mode.
So llms.txt needs to be viewed realistically. It is easy to produce and does no harm; it may be useful for some tools and for links that users paste into chat windows. But it is not a lever that brings visibility on its own, and it should not be sold as one.
How do you write an llms.txt file?
The proposed format is Markdown, and the order of elements is set. Whether the file reads easily for both people and machines depends less on strict syntax and more on following this simple order:
- Title: The name of the site or project, as a top-level heading. This is the only mandatory element in the proposal.
- Summary: A short blockquote describing what the site does. The aim is for an assistant to give an accurate description after reading this line alone.
- Free-form description: Optional paragraphs or bullet points: context, important constraints, notes on how to read the file.
- Sections: Lists separated by second-level headings. Each line holds a link in square brackets, followed by a colon and a short description.
- "Optional" section: Secondary links that can be skipped when context is limited. This heading has a special meaning in the proposal.
Which pages should llms.txt include?
Not all of them. The file's value comes from being selective. For an agency or a service business, a sensible order is usually: service pages, sector or solution pages, guide-style articles, then contact and company information. Campaign pages, filtered list URLs, tag archives and temporary announcements should be left out. On e-commerce sites, category pages and frequently asked information such as returns and delivery terms can take priority over individual products.
How should the link descriptions be written?
Each link's description should state, in one sentence, the question that page answers. Marketing language backfires here: phrases like "the best in the industry" give a model no information at all. A concrete description such as "channel choice and measurement for clinics acquiring patients from abroad" tells both people and models far more. Brand names, addresses and service names in the descriptions should be written exactly as they are on the rest of the site.
What does an example llms.txt look like?
Here is an example from our own site. The file at unitistanbul.com/llms.txt is not written by hand; it is generated automatically at build time from our service, sector and article data. Its structure follows the proposal:
- The agency's name as the title, with a one-sentence description of the site directly below it as a blockquote.
- A short description paragraph giving the founding year, city and contact details.
- "Services" and "Sectors" sections with a link and a one-line description for each page.
- A "Guides" section listing current guide articles.
- An "Optional" section with secondary URLs such as the Journal, our work and the sitemap.
The biggest advantage of automatic generation is consistency: when a new service page is added, the file updates itself and dead links do not pile up. The most common problem with hand-maintained llms.txt files is that after a while they no longer reflect the site as it really is. If automatic generation is not possible, tying the file to the same publishing process as sitemap and menu changes at least keeps the update from being forgotten.
What are the limits of llms.txt?
- It does not create visibility. Adding the file does not mean a brand will be mentioned in ChatGPT, Gemini or Perplexity answers. Mentions depend on content quality, third-party sources and consistent entity information.
- It does not control access. Blocking or allowing a crawler is robots.txt's job.
- There is no certainty it will be read. Which tool uses the file, when and how is mostly not public.
- It is no substitute for the pages themselves. If the pages are weak, the summary in the file achieves nothing; the model ends up going to the page.
- If not updated, it spreads wrong information. An unmaintained file keeps pointing to old services, closed branches or changed contact details.
Where does llms.txt fit in GEO and AEO work?
llms.txt is one of the last steps of a generative engine optimisation plan, not the first. First, pages need a structure that answers questions directly, brand information needs to be written the same way everywhere, structured data needs to match visible content and crawlers need to be able to reach the site. Once that groundwork is in place, llms.txt becomes a complementary step that summarises that order in a single file.
A short checklist for prioritising:
- Are your important pages indexed by search engines and allowed to show snippets?
- Is robots.txt accidentally blocking AI search crawlers you want to allow?
- Does the first paragraph of each service page answer that page's question directly?
- Are your company name, address and contact details the same on your site, in directories and on social profiles?
- If those four are in place, add llms.txt and make sure it updates automatically.
We explain the full scope of work on being cited by generative engines on our GEO service page, and answer surfaces such as featured snippets and voice assistants on our AEO service page. For the difference between the concepts, see our article What is GEO?.
Frequently asked
- Does adding llms.txt affect SEO rankings?
- No direct effect should be expected. Google states that such a file is not needed to appear in search results or in AI Overviews. The file is added not for rankings but so that AI tools can understand the site more easily.
- What is the difference between llms.txt and llms-full.txt?
- Some sites also publish a longer version that combines all their documentation into a single file alongside the link list. This is especially common for software documentation. For service and corporate sites, the short version is usually enough.
- Does llms.txt block AI crawlers?
- No. The file provides a summary; it sets no access rules. To block or allow specific crawlers you use robots.txt; in some cases your server and content delivery network settings also need checking.
- How often should I update the file?
- Whenever important pages on the site change. The safest approach is to generate the file automatically from your content management system or build process, so the site and the file never drift apart.