How big should llms.txt be?

The 30-second answer: an llms.txt size limit is not in the spec, and a few kilobytes is the right order of magnitude

Aim for a file between about 2 KB and 20 KB, holding roughly 10 to 80 links with one sentence each. Half of the real files we measured are under 15 KB, and the ones that read best are in the single-digit kilobytes. Treat 50 KB as the point where you split the index into section files or move content into llms-full.txt, and 100 KB as the point where tools start to object.

The reason is cost. An assistant reads llms.txt before it reads anything else on your site, and every kilobyte of index is budget it cannot spend on your pages. A short file is read in full, so the order and annotations you wrote actually reach the model. A long one is skimmed or truncated, and the end of the file, where Optional lives, goes first.

What the spec and the tools say

• The spec: no maximum size, no maximum link count, no guidance on either. It describes a curated index with a short sentence per link, which is a shape, not a number.
• This site’s validator: passes a file up to 100 KiB, warns above that, and stops reading at 512 KiB, reporting a failure because the file could not be checked in full.
• Mintlify’s generator: caps the generated index at 100,000 characters. Above that, the root llms.txt becomes a directory pointing at group files under /_llms/. A platform that generates thousands of these files decided that 100 KB is the ceiling for one index.

What real files weigh

From the 28 public files we saved on September 19, 2026, for the examples article, 24 parsed as Markdown (two fetches failed and two returned HTML). A selection, sorted by size:

Two things stand out. Bytes per link cluster around 150 to 280 when every link carries a sentence, and fall to 65 to 110 when most do not: Hono and Anthropic are bare link lists. And the files above 30 KB all read as generated from a complete page inventory, while the smallest ones read as written by hand. Size is a symptom of whether anyone chose what to include.

The token arithmetic

For English prose, a working rule is about four characters per token. That makes a 5 KB file roughly 1,200 tokens, a 20 KB file about 5,000, and a 100 KB file about 25,000. Japanese and other non-Latin scripts cost more tokens per character, so a Japanese index of the same byte size is more expensive to read.

The comparison that matters is not against a model’s context window, which in 2026 is large, but against what the assistant is willing to spend on one fetched file before it has seen any of your pages. A 25,000-token index is the same cost as reading ten typical documentation pages. If the assistant has a budget for your site, you want most of it spent on the pages that answer the question, not on the list of pages.

Cutting rules

• One sentence per link, not a paragraph. The annotation says what the page answers. Fifteen words is plenty; the files in the table average under 300 bytes per link including the URL.
• Sections, not pages. Link the guide, not every subsection of it. If a section of your site has more than a dozen pages, link its index page and let the assistant go from there.
• Drop what a sitemap already carries. Tag pages, pagination, legal pages, old versions, and the long tail of the blog belong in sitemap.xml, not here.
• Split at 50 KB. Keep the root as a short directory with one link per section index, each under 20 KB. This is what Mintlify does automatically above its cap, and it works by hand too.
• Move bodies to llms-full.txt. If the index is growing because you are pasting content into it, stop. The bundle is the place for full text; the index is the map.

One structural rule sits above the others: the first two kilobytes must stand alone. H1, blockquote, and the first section should let a reader that stops early still describe the site correctly. Everything after that is a bonus for readers with budget.

Checking the result

curl -s https://example.com/llms.txt | wc -c
curl -s https://example.com/llms.txt | grep -c '^- \['
curl -s https://example.com/llms.txt | head -c 2048

The first number is bytes, the second is links, and the third command shows what a reader with a two-kilobyte budget would see. Then run the file through a validator, which reports the size against its own threshold and lists every link so you can see where the weight is.

FAQ