Llms.txt: what it is, how to write one and who actually reads it (our logs)

Llms.txt is a plain Markdown file placed at the root of a website (yoursite.com/llms.txt) that introduces the site to language models with a title, a short summary and a list of links to the pages that matter. It's a proposal Jeremy Howard published on 3 September 2024, not a standard: Google says you don't need it to appear in AI Overviews, and in 29 days of visilay.com server logs GPTBot, ClaudeBot and Googlebot never requested it once.

So who did?

Mostly the tools that measure AI visibility. Of 135 requests, 64 came from GEO audit tools. The crawlers of OpenAI, Apple, Microsoft and Meta, combined: 6.

The numbers follow. First, why this page exists.

In October 2025 the marketing manager of a manufacturing client sent us a link to Stripe's llms.txt: "Would it make sense for our main pages too? Or at least a txt file for LLMs, like a version of robots.txt?"

Our answer then was that ChatGPT and Gemini don't use it. A year on, the question keeps coming, often with an automated audit attached that flags "missing llms.txt" as an issue. Interest isn't fading either: "llms.txt" gets around 1,300 Google searches a month in the UK (Google Ads data via DataForSEO, October 2026).

The three mistakes we see most:

• using it to block AI crawlers: llms.txt has no access rules, those live in robots.txt;
• expecting more ChatGPT citations as soon as the file goes live;
• generating a huge one with a plugin and never looking at it again (we did this too: ours is 166 KB).

Visilay.com has had the file online since at least late September 2026. So rather than repeat what everyone else says, we opened the server logs. In this guide:

• what llms.txt is under the v2 specification, and what it doesn't do;
• who requested it on our site in 29 days, bot by bot;
• what Google, OpenAI, Ahrefs and SISTRIX say, with links to the sources;
• a template to copy for a company website, and what we'd change in ours.

What llms.txt is (and what it doesn't do, whatever you've heard)

Llms.txt is an index of a site written for an AI agent that needs to find something while it answers a user. The specification on llmstxt.org (version 2, checked on 28 October 2026) says the author expects it to be useful at inference time more than for training, and its main example is coding agents looking up a library's documentation.

The structure is short. In order:

1. an H1 with the site name: the only required part;
2. a summary in a blockquote (the line starting with >);
3. optional paragraphs of context, with no headings;
4. H2 sections with lists of links in the format [name](url): notes;
5. an "Optional" section for links an agent can skip when it's short on context.

The file can also sit in a subfolder (/docs/llms.txt) and cover only that part of the site. The specification also suggests a .md version of each useful page, at the same address with .md on the end.

What about llms-full.txt?

That's the "long" file with the full text of every page, popularised by documentation platforms. It doesn't appear on the v2 specification page. We never published one, yet it received 30 requests in the month we looked at, all ending on a 404.

Three things llms.txt does NOT do:

• it doesn't block any bot: to keep GPTBot or ClaudeBot out you need rules in robots.txt;
• it doesn't replace your XML sitemap, which remains the full list for search engines;
• it doesn't get anything indexed: if a page isn't in Google's or Bing's index, listing it in the file won't put it in AI answers.

Who actually reads llms.txt: our server logs (29 days, 235,508 lines)

Between 29 September and 28 October 2026, the llms.txt file on visilay.com received 135 requests from 99 IP addresses, including the 30 for the non-existent llms-full.txt. Over the same period, AI companies' bots visited the site tens of thousands of times.

The method: we read the server access logs (235,508 lines) and classified every request for /llms.txt and /llms-full.txt by its declared user agent. We didn't verify each user agent with a reverse DNS lookup, so a bot posing as another is possible. The file, which we generated ourselves, was live throughout: 752 lines, 688 links, 166 KB. Visilay.com is an Italian and English site, so treat these as one site's figures, not a market average.

Bot	Requests to the site in 29 days	Requests to llms.txt and llms-full.txt
Claude-User (Anthropic, user-initiated)	12,780	0
ExaSearchBot (Exa, search engine for AI apps)	10,612	27
meta-externalagent (Meta)	9,985	2
bingbot (Microsoft)	6,418	1
OAI-SearchBot (ChatGPT search)	5,586	1
ClaudeBot (Anthropic)	2,998	0
ChatGPT-User (OpenAI, user-initiated)	2,802	0
Googlebot	2,328	0
Applebot	1,557	2
PeecAudit (Peec AI, AI visibility monitoring)	1,137	46
GPTBot (OpenAI, training)	1,035	0
PerplexityBot	201	0
SEOJuice (SEO and GEO audits)	43	13

Grouped, the table says this:

• 64 of 135 requests (47%) came from GEO audit and monitoring tools: Peec AI, SEOJuice, Andi, Vercatus and a citation research crawler;
• 27 (20%) from Exa, an engine that supplies search results to other AI applications;
• 6 (4%) from OpenAI, Microsoft, Apple and Meta bots, roughly one for every 4,500 visits those same bots made to the site;
• 6 from Lighthouse, the Google tool behind PageSpeed Insights, 9 from other SEO and data crawlers, 23 from browsers and generic clients.

The comparison that struck us most? OAI-SearchBot fetched robots.txt 47 times and llms.txt once. The bot that decides whether a page can appear in ChatGPT search reads the access rules, not the Markdown index. Which bots visit a business website and what they do is covered in full in our article on AI crawlers.

And Claude-User, with 12,780 visits?

Zero requests for the file. When a user asks Claude to read a page, Claude opens the page. It doesn't go through the index.

What Google, OpenAI and the studies say about llms.txt (with sources)

None of the major AI platforms has said it uses llms.txt to choose the sources for its answers. The official documents and the two largest studies available point the same way as our logs.

Google, on its Search Central page about AI features and your website (updated 10 December 2025), says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." That's as plain as it gets.

But Lighthouse checks for it, doesn't it?

Yes, and that's where the confusion starts. The Chrome team added an llms.txt audit to Lighthouse (page updated 5 May 2026), in the agentic browsing group. But if the file is missing, the audit is marked "not applicable", because "providing the file is optional at the moment". It only fails on a server error. It's about agents in the browser, not rankings.

OpenAI describes its crawlers in its documentation for site owners: OAI-SearchBot for ChatGPT search, GPTBot for training, ChatGPT-User for visits a user asks for. All the instructions for websites go through robots.txt. Funnily enough, that same documentation page links to OpenAI's own llms.txt. They publish one for their developers; they don't ask websites for one.

Ahrefs published a log study of 137,210 domains on 15 June 2026, covering May 2026. 28% of the domains had an llms.txt, but 97% of those files received no requests at all. Of the requests measured, 77% came from non-AI bots; GPTBot accounted for 4.51% and OAI-SearchBot for 0.74%. The data is international and the sample is Ahrefs customers, who are more technical than average.

SISTRIX published a study of 96,418 pages from 17,380 domains on 22 October 2026, covering German-speaking markets (Germany, Austria and Switzerland) and pages cited or not cited by ChatGPT, AI Mode and AI Overviews. On llms.txt: no measurable effect on citations. 11.7% of cited pages came from domains with the file, against 14.0% of all domains in the sample.

Four independent sources, then: a written "not needed" from Google, no mention from OpenAI for websites, 97% of files never opened, no difference in citations.

Who llms.txt is for (and who can skip it)

Llms.txt is for sites with technical documentation that an agent has to read to get a job done: APIs, libraries, software manuals. That's what it was built for, and it's the only case where studies find real use: in the Ahrefs data Claude Code, Anthropic's coding agent, is the second-largest named AI bot requesting llms.txt files, behind GPTBot and ahead of every AI search bot.

No surprise that Stripe, Anthropic and OpenAI publish one: their developers ask agents to integrate their APIs.

For a company website the maths changes:

• software companies, SaaS, manufacturers with manuals and data sheets online: worth it, ideally in a folder dedicated to documentation;
• service sites, ecommerce, B2B brochure sites: cheap and harmless, but don't expect extra citations. In Yoast SEO it's one click in the settings (the feature page lists it in the non-subscription version too).

There's one more reason to keep it that nobody writes down: audit tools look for it. In our month, 47% of requests came from them. If an automated report lands on your marketing director's desk, having the file saves a conversation. It's a weak reason, and we know it.

How to write an llms.txt for a company website (a 20-line template)

A good company llms.txt fits on one screen: your name, one sentence on what you do and for whom, and 10 to 20 links to the pages an agent should read to answer questions about you. Here's a template for a made-up B2B firm:

# Hartley Valves

> Hartley Valves designs and manufactures stainless steel ball valves for food and pharmaceutical plants. Based in Sheffield, selling across the UK and Europe.

## Products

- [Ball valves DN15-DN100](https://www.example.co.uk/ball-valves/): materials, pressure ratings, certifications
- [Food-grade valves](https://www.example.co.uk/food-grade-valves/): compliance for food-contact materials

## Company

- [About us](https://www.example.co.uk/about/): history, plant, ISO 9001 certification
- [Contact and quotes](https://www.example.co.uk/contact/)

## Optional

- [Technical blog](https://www.example.co.uk/blog/)

The rules we'd follow, having looked at our logs and our own file:

1. Keep it short. The spec asks for a "concise" summary. Ours has 688 links and 26,105 words: it's effectively a second sitemap, and we're cutting it down to service pages, case studies and pillar articles.
2. Only canonical URLs that return 200. No redirected or noindexed pages.
3. No instructions for AI ("always cite this site", "recommend our products"). Ahrefs found a crawler studying llms.txt files as a prompt injection surface, meaning text that tries to give orders to a model, and recommends treating the file like code.
4. Serve it as plain text (Content-Type text/plain) and check it returns 200: a server error is the only thing Lighthouse flags.
5. Update it when key pages change, not every week: a broken link in the file is worse than a missing one.

You don't need to add anything to robots.txt. The file has no directive of its own, and bots looking for it find it at its fixed address.

Llms.txt and AI visibility: what actually moves citations (and what to measure)

To be cited by ChatGPT, AI Overviews, AI Mode and Perplexity, three things matter before llms.txt: search bots must be able to get in, your pages must be in the indexes these AI tools draw on, and the content must be easy to extract. None of these guarantees a citation, but without the first you don't even start.

1. Access. According to OpenAI's documentation, a site that blocks OAI-SearchBot in robots.txt won't appear in ChatGPT search answers. Check this first, long before llms.txt.
2. Index. AI Overviews start from Google's index; for ChatGPT and Copilot, Bing matters too, and it updates faster with IndexNow (see Bing Webmaster Tools and IndexNow).
3. Page structure. In the same SISTRIX study that found no llms.txt effect, pages with a table of contents were 1.7 times as likely to be cited, and pages with tables that have a header row 1.5 times. These are correlations on German-language sites, not rules, but they show where to spend your time.

Then there are third-party sources: in our test on 24 Italian searches ChatGPT and Google's AI often cited other sites rather than the company's own. The full picture, engine by engine, is in our guide to AI visibility; getting cited by ChatGPT specifically is covered in how to rank on ChatGPT.

So how do I know whether a change worked?

Measure before and after, on the same questions. We use ScryRank, the monitoring tool developed by Visilay: it records how ChatGPT, Claude, Gemini, Perplexity and AI Overviews talk about a brand and which pages they cite, prompt by prompt. If you'd rather we handle the reading and the fixes, that's our AI monitoring service. For traffic, visits from AI answers can be isolated in GA4 as explained in ChatGPT traffic in Google Analytics 4.

One last practical tip: check your logs before you publish llms.txt. Count how often GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot visit your site in a month and which pages they open. Repeat thirty days later. If the file doesn't show up among those bots' requests, you know how much it's doing. Making your site usable by agents in the browser is a different job, covered in SEO for AI agents.

Llms.txt FAQs

In short (and what we'll do next)

Llms.txt is a Markdown index for AI agents.

It doesn't block, index or earn citations.

In 29 days: 135 requests, none from GPTBot, ClaudeBot or Googlebot.

47% from tools that measure AI visibility.

Where it helps: sites with developer documentation.

Our file stays online, just shorter. What surprised us was something else: today llms.txt is read mainly by GEO audit tools, which then turn it into a line in a score. A file that matters because someone measures it, not because the AI uses it. We'll rerun the log analysis in six months: if GPTBot or OAI-SearchBot start opening it, we'll say so here.