The 30-second answer: 200 at /llms.txt, text/plain or text/markdown with charset=utf-8, no BOM, an hour of cache with revalidation
Serve the file at the root of the canonical origin with a 200 status, a Content-Type of either text/plain; charset=utf-8 or text/markdown; charset=utf-8, saved as UTF-8 without a byte-order mark. Cache it publicly for about an hour and send an ETag or Last-Modified so caches revalidate instead of re-downloading. Allow at most one redirect, from http to https or between www and the bare domain, and never to another host. If you want to go further, add the Link header the spec describes so pages point at their Markdown versions and at the file.
Then run one curl command against the live URL and read the headers rather than the body. Everything below explains why each line of that answer is there.
What the specification says, and does not
The specification, version 2 as modified on 10 August 2026, says the file is Markdown, lives at /llms.txt at the root or at a subpath such as /docs/llms.txt, and is used on demand when an agent needs information rather than crawled on a schedule. It does not name a Content-Type. It does describe link relations that a site may send: rel=“alternate” with type=“text/markdown” for the Markdown version of a page, and rel=“describedby” for the llms.txt that covers it, either as HTML link elements or as an HTTP Link response header. So the media type is convention, settled by what parsers accept; the Link header is the one place the spec touches HTTP at all.
Content-Type: the two good values and the three bad ones
The media type for Markdown is text/markdown, registered in RFC 7763; text/plain is what a .txt extension maps to on nearly every server. Readers accept both, and this site’s validator passes text/plain, text/markdown, and the older text/x-markdown. Everything else falls into one of three failure shapes:
| Response | What it means | Validator | Fix |
|---|---|---|---|
| text/html with a 200 | The host answered with a fallback page: an SPA shell or a styled 404 | fail | Put a real file at the path; check the host does not catch unknown routes |
| application/octet-stream | Object storage or a CDN did not map the extension | warn | Set the type on the object or in a path rule |
| No Content-Type at all | A hand-written handler or proxy dropped the header | warn | Add the header in the handler |
| text/plain, no charset, non-ASCII body | Readers must guess the encoding | warn | Append ; charset=utf-8 |
| charset=iso-8859-1 or shift_jis | The file will be decoded wrongly even if the bytes are UTF-8 | warn | Declare utf-8 and re-save the file as UTF-8 |
The HTML case is the one that hides. The fetch succeeds, the status is 200, and a browser even shows something. The validator checks both the header and the first 512 bytes of the body for a doctype or an html, head, or body tag, because some hosts send the fallback page with a text type. If your check tool reports text/html, there is no llms.txt at that URL, whatever the file manager shows.
Charset and the byte-order mark
Two separate things go wrong with encoding. The bytes may not be UTF-8 at all, which the validator fails: a file saved as Shift_JIS or Latin-1 is read as garbage by every LLM fetcher, which assume UTF-8. Or the bytes are fine but the declaration is wrong or missing: a charset other than utf-8, no charset on a file with non-ASCII characters, or a UTF-8 byte-order mark at the start, which makes the first line something other than a heading in a strict parser. Those are warnings. The fix is the same in every editor: save as UTF-8 without BOM, and put charset=utf-8 in the header. An ASCII-only file needs no charset, but declaring it costs nothing.
Cache-Control: short, public, revalidated
Readers fetch the file when a question needs it and keep it under their own rules; the spec says nothing about caching. What you control is the shared caches and CDNs between you and them. The pattern on the public files checked for this article:
| Site (18 Oct 2026) | Content-Type | Cache-Control | Validators |
|---|---|---|---|
| platform.claude.com | text/plain; charset=utf-8 | public, max-age=3600 | Last-Modified |
| developers.openai.com | text/plain; charset=utf-8 | public, max-age=0, must-revalidate | ETag, Last-Modified |
| vercel.com | text/plain; charset=utf-8 | public, max-age=0, s-maxage=3600, stale-while-revalidate=86400 | ETag |
| svelte.dev | text/plain; charset=utf-8 | public, max-age=0, must-revalidate | ETag, Last-Modified |
| docs.stripe.com | text/markdown; charset=utf-8 | (none sent) | (none) |
| knownbyllm.com | text/markdown; charset=utf-8 | public, max-age=3600, s-maxage=86400 | (none) |
Two shapes recur. Either a short positive max-age of an hour, or max-age=0 with must-revalidate and an ETag, which makes every read a cheap conditional request. Both are fine. The choice to avoid is a long max-age with immutable, which is right for hashed assets and wrong for a file whose whole value is being current: a cached llms.txt that still lists last year’s pricing is the stale-copy problem in another form. If your CDN caches at the edge for longer, purge the path on deploy.
Redirects: one hop, same host, permanent
Every reader follows at least one redirect, and the validator follows them and reports the final status. The cases that lose files:
- A different host. /llms.txt on your domain redirecting to a raw file on a code host or a pastebin. The final response has that host’s headers, often a generic type, and a reader that insists on a 200 at the origin stops at the 3xx.
- A chain. http to https to www to a trailing slash. Each hop is a chance for a fetcher with a low redirect limit to give up. Collapse to one.
- A 302 to a page. A login wall, a region picker, or a cookie notice that intercepts unknown paths. The final body is HTML, and the file is gone.
- Mixed canonical hosts. The site is on www, the file is on the bare domain, and the links inside the file use a third form. Pick one origin, serve the file there, and write absolute links to it.
Use 301 or 308 for the one permanent hop; 308 preserves the request method, which matters for fetchers that probe with HEAD first. The validator samples your links with HEAD and falls back to GET when a server refuses HEAD, which is a reasonable model of what other tools do.
The optional Link header
If your pages also exist as Markdown, the spec’s example shows how to say so without touching the HTML:
Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"
The spec notes the header form also works for non-HTML resources, such as the Markdown files themselves, and that it can be set in server or CDN configuration. It is optional, and none of the six sites above sent it when checked; treat it as the next step after the four headers are right.
Setting the headers on common hosts
nginx, for a static file:
location = /llms.txt {
types { }
default_type "text/markdown; charset=utf-8";
add_header Cache-Control "public, max-age=3600";
}
Netlify, in a _headers file at the publish root:
/llms.txt Content-Type: text/plain; charset=utf-8 Cache-Control: public, max-age=3600
Next.js, in next.config:
async headers() {
return [
{
source: "/llms.txt",
headers: [
{ key: "Content-Type", value: "text/plain; charset=utf-8" },
{ key: "Cache-Control", value: "public, max-age=3600" },
],
},
];
}
On hosted builders you usually cannot set these; the no-root-access article covers what each platform sends and the proxy option when it is wrong. Build-time generation in Next.js, Astro, and Hugo is in the framework article.
Checking it
- Headers only, no redirects followed. Read the status, the Content-Type, and any Location:
curl -sI https://example.com/llms.txt
- Follow redirects and report how many hops and where you landed:
curl -sL -o /dev/null -w '%{num_redirects} %{http_code} %{content_type} %{url_effective}\n' https://example.com/llms.txt - Check for a byte-order mark; the first three bytes must not be ef bb bf:
curl -s https://example.com/llms.txt | head -c 3 | xxd
- Confirm the body is the file, not a page:
curl -s https://example.com/llms.txt | head -n 3
The validator runs the same four checks in order, served at root, content type, encoding, then the Markdown rules, and reports the header value it saw. Run it after every change to hosting, not only after editing the file.
Validate your llms.txt
Paste your URL. The validator fetches /llms.txt the way a bot does, reports the status, Content-Type, and encoding it got back, then checks the Markdown against the spec and samples your links.
Validate →FAQ
Which Content-Type should llms.txt have?
Either text/plain; charset=utf-8 or text/markdown; charset=utf-8. The specification does not name one, and both are what parsers expect. Of the public files checked for this article on 18 October 2026, Anthropic, OpenAI, Vercel, and Svelte serve text/plain and Stripe and this site serve text/markdown, all with charset=utf-8. What matters is that it is not text/html, which almost always means your host returned a fallback page instead of the file.
My host serves the file as application/octet-stream. Is that a problem?
Usually a sign of object storage or a CDN that did not recognise the extension. Most readers will still parse the bytes, and this site's validator only warns on it, but some fetchers treat an unknown binary type as not text and skip it. Set the type explicitly in the bucket metadata or the CDN rule; every provider has a way to do it for one path.
Does the charset parameter matter if the file is ASCII?
Not for an ASCII-only file. It matters as soon as the file contains a non-ASCII character, which includes a curly quote, an accented name, or any Japanese. Without a declared charset, a reader may guess, and a wrong guess turns the file into mojibake. Declare charset=utf-8 and save the file as UTF-8 without a byte-order mark.
Should llms.txt be cached?
Yes, briefly. The file is read on demand rather than crawled on a schedule, so a short public max-age such as an hour, with an ETag or Last-Modified so caches can revalidate cheaply, is the common pattern. Avoid long immutable caching unless your deploy also purges the path; a stale llms.txt is the first thing an assistant reads. The files checked for this article use max-age values between zero with must-revalidate and one hour.
Is a redirect from /llms.txt acceptable?
One hop from http to https, or from www to the bare domain, is normal and every reader follows it. A redirect to a different host, a chain of several hops, or a 302 that lands on an HTML page is where files get lost: a reader that only accepts a 200 at the root gives up, and the final response carries the other host's headers. Keep the file at the canonical origin and make the redirect a single 301 or 308.
What is the Link header the spec mentions?
Version 2 of the specification, modified in August 2026, describes two link relations: rel="alternate" with type="text/markdown" pointing to the Markdown version of a page, and rel="describedby" pointing to the llms.txt that covers it. They can be sent as HTML link elements or as an HTTP Link response header, which the spec notes can be added in web server or CDN configuration without touching pages. It is optional, and most sites do not send it yet.
Next steps
- → 12 common llms.txt mistakes (and how to fix each) (the content-side checks that follow these)
- → Generating llms.txt at build time in Next.js, Astro, and Hugo (where the file comes from on a framework site)
- → llms.txt when you can’t upload to the root
- → All Learn articles