GuidesIndexed

llms.txt explained

Someone has probably told you your firm’s website needs an llms.txt file. It does no harm, but very little reads it. Here is what the file is, who actually fetches it, and the three jobs to finish first.

John Geater, founder of IndexFirst5 min read
The short answer

llms.txt is a plain Markdown file at yoursite.co.uk/llms.txt that points AI tools to your most useful pages. Jeremy Howard proposed it in September 2024. Google says its Search doesn’t use the file, and no major AI answer engine has said it does. It is quick to add, but on its own it won’t get your firm named when buyers ask AI who to use.

What goes in an llms.txt file?

Not much, on purpose. The proposal at llmstxt.org requires only a top heading (an H1) with the name of your site or business. The rest is optional: a short summary in a quote line, a few lines of background, then sections under their own headings, each a list of links with a brief note after a colon. A section headed “Optional” holds links a tool can skip when it is short of space. The proposal also suggests offering a clean Markdown copy of each page at the same address with .md added.

The parts of an llms.txt fileFour lines from a made-up IT support firm’s llms.txt, each labelled. “# Example IT Support” is the H1 with the business name, the only part that is required. “> Managed IT support…” is an optional one-line summary. “## Services” opens an optional section of links. “- [Cyber Essentials](…): help…” is one link with a note after the colon.Required: your business name, as an H1# Example IT SupportOptional: a one-line summary> Managed IT support…Optional: a section of links## ServicesEach link, then a note after a colon- [Cyber Essentials](…): …
Lines from an llms.txt for a made-up firm, Example IT Support at example.co.uk, shortened to fit. Format from the llms.txt proposal.

To make one, type it in any text editor, save it as llms.txt and upload it to the root of your site so it opens at yoursite.co.uk/llms.txt as plain text. Only link to pages that actually load: your service pages, location pages, team pages and accreditations. If your site runs on WordPress, Yoast SEO can create the file for you.

Does llms.txt help you appear in AI answers?

Not in any way anyone has been able to measure. Google’s guidance on generative AI search says you don’t need to create these files, because “Google Search itself doesn’t use them.” Google’s John Mueller put it more sharply, likening it to the old keywords meta tag: what a site owner claims the site is about.

The large studies point the same way. Ahrefs looked at 137,210 domains in May 2026; of the roughly 38,000 that had a valid llms.txt, 97% saw no requests for the file that month. SE Ranking analysed nearly 300,000 domains and found no effect on how often a domain is cited by AI.

For a service business the practical point is simple. When a buyer asks who does employment law for small firms in Leeds, the answer is drawn from pages the engine finds in search: firms’ own service pages, registers, directories and review sites. That is where your effort pays back.

So who does fetch llms.txt?

Very few, and mostly not the tools you care about. In Ahrefs’ data, named AI bots made 19.5% of the requests to the small number of files that were fetched at all. GPTBot, OpenAI’s training crawler, made the most; Claude Code, Anthropic’s coding tool, came second. Anthropic publishes an llms.txt for its own developer documentation.

So if clients connect to your booking, ticketing or tracking system through an API, an llms.txt for those developer docs has a real reader: their developers’ coding tools. But a fetch is not a citation, and a coding tool reading your API docs is not ChatGPT recommending you to a buyer.

Chrome’s Lighthouse also checks the file. It only flags a page when requesting llms.txt returns a server error; if you have no file, the check is marked not applicable.

How does llms.txt compare with robots.txt and a sitemap?

The three are often mentioned together, but only one of them can keep you out of AI answers.

The job of robots.txt, a sitemap and llms.txtrobots.txt tells crawlers which pages they may access; blocking an AI search crawler can keep your pages out of its answers. sitemap.xml lists your pages so crawlers can find them more efficiently. llms.txt is a Markdown guide to your key pages; it controls nothing, and Google Search doesn’t use it.robots.txtTells crawlers which pages they may visit.Can keep you out of AI answers.sitemap.xmlLists the pages on your site.Helps crawlers find them.llms.txtA Markdown guide to your key pages.Controls nothing.
Sources: Google’s guides to robots.txt and sitemaps, and OpenAI’s crawler documentation.

Block OAI-SearchBot in robots.txt and OpenAI says your pages won’t be shown in ChatGPT search answers. PerplexityBot is the crawler Perplexity uses to surface and link websites in its results, so blocking it works against you too. llms.txt can’t override either, because it contains no rules, only links. Our robots.txt checker shows which search and AI crawlers can read a page on your site.

Should your firm add an llms.txt file?

If you like. It costs very little. Just leave it until last:

  1. Let the AI crawlers inMake sure robots.txt doesn’t block OAI-SearchBot, PerplexityBot or Googlebot, and that your firewall doesn’t either.
  2. Get your pages indexedGoogle says a page must be indexed and eligible to show with a snippet to appear in AI Overviews and AI Mode.
  3. Get onto the pages AI readsWrite service and location pages that answer buyers’ questions, and make sure registers and directories list you. Our ChatGPT guide shows what the answers in our test leaned on.
  4. Then add llms.txtIt is a short job. If you publish API or integration docs, move this step up.

Steps 1 to 3 are where AI visibility comes from. That is the work IndexFirst does for UK service businesses that sell to other businesses: real changes to your site every working day, each checked before it goes live, for £1,750 a month.

About the examples: Example IT Support and example.co.uk are made up for this guide. The figures quoted come from the studies and documentation linked above, checked on 13 November 2026. IndexFirst has not run its own study of llms.txt.