KnownByLLM

Reference · 9 min read

llms.txt HTTP headers: Content-Type, caching, and redirects

The file is Markdown; the headers are what make it readable by a fetcher. The values to use, what the spec says and does not say, what large sites actually send, and the commands to check your own.

Most llms.txt problems are not in the file. They are in the response around it: a host that answers with an HTML page and a 200, a bucket that labels the file as binary, a charset nobody declared, a redirect that lands somewhere else. A validator sees these as the first three checks it runs, before it reads a single heading.

This reference covers the four headers that matter, Content-Type with its charset, Cache-Control with its validators, Location when a redirect is involved, and the optional Link header the specification added in 2026. For each: what the spec says, what readers do, what six public files sent on 18 October 2026, and how to set it on common hosts. The checking commands are at the end.

The 30-second answer: 200 at /llms.txt, text/plain or text/markdown with charset=utf-8, no BOM, an hour of cache with revalidation

Serve the file at the root of the canonical origin with a 200 status, a Content-Type of either text/plain; charset=utf-8 or text/markdown; charset=utf-8, saved as UTF-8 without a byte-order mark. Cache it publicly for about an hour and send an ETag or Last-Modified so caches revalidate instead of re-downloading. Allow at most one redirect, from http to https or between www and the bare domain, and never to another host. If you want to go further, add the Link header the spec describes so pages point at their Markdown versions and at the file.

Then run one curl command against the live URL and read the headers rather than the body. Everything below explains why each line of that answer is there.

What the specification says, and does not

The specification, version 2 as modified on 10 August 2026, says the file is Markdown, lives at /llms.txt at the root or at a subpath such as /docs/llms.txt, and is used on demand when an agent needs information rather than crawled on a schedule. It does not name a Content-Type. It does describe link relations that a site may send: rel=“alternate” with type=“text/markdown” for the Markdown version of a page, and rel=“describedby” for the llms.txt that covers it, either as HTML link elements or as an HTTP Link response header. So the media type is convention, settled by what parsers accept; the Link header is the one place the spec touches HTTP at all.

Content-Type: the two good values and the three bad ones

The media type for Markdown is text/markdown, registered in RFC 7763; text/plain is what a .txt extension maps to on nearly every server. Readers accept both, and this site’s validator passes text/plain, text/markdown, and the older text/x-markdown. Everything else falls into one of three failure shapes:

ResponseWhat it meansValidatorFix
text/html with a 200The host answered with a fallback page: an SPA shell or a styled 404failPut a real file at the path; check the host does not catch unknown routes
application/octet-streamObject storage or a CDN did not map the extensionwarnSet the type on the object or in a path rule
No Content-Type at allA hand-written handler or proxy dropped the headerwarnAdd the header in the handler
text/plain, no charset, non-ASCII bodyReaders must guess the encodingwarnAppend ; charset=utf-8
charset=iso-8859-1 or shift_jisThe file will be decoded wrongly even if the bytes are UTF-8warnDeclare utf-8 and re-save the file as UTF-8

The HTML case is the one that hides. The fetch succeeds, the status is 200, and a browser even shows something. The validator checks both the header and the first 512 bytes of the body for a doctype or an html, head, or body tag, because some hosts send the fallback page with a text type. If your check tool reports text/html, there is no llms.txt at that URL, whatever the file manager shows.

Charset and the byte-order mark

Two separate things go wrong with encoding. The bytes may not be UTF-8 at all, which the validator fails: a file saved as Shift_JIS or Latin-1 is read as garbage by every LLM fetcher, which assume UTF-8. Or the bytes are fine but the declaration is wrong or missing: a charset other than utf-8, no charset on a file with non-ASCII characters, or a UTF-8 byte-order mark at the start, which makes the first line something other than a heading in a strict parser. Those are warnings. The fix is the same in every editor: save as UTF-8 without BOM, and put charset=utf-8 in the header. An ASCII-only file needs no charset, but declaring it costs nothing.

Cache-Control: short, public, revalidated

Readers fetch the file when a question needs it and keep it under their own rules; the spec says nothing about caching. What you control is the shared caches and CDNs between you and them. The pattern on the public files checked for this article:

Site (18 Oct 2026)Content-TypeCache-ControlValidators
platform.claude.comtext/plain; charset=utf-8public, max-age=3600Last-Modified
developers.openai.comtext/plain; charset=utf-8public, max-age=0, must-revalidateETag, Last-Modified
vercel.comtext/plain; charset=utf-8public, max-age=0, s-maxage=3600, stale-while-revalidate=86400ETag
svelte.devtext/plain; charset=utf-8public, max-age=0, must-revalidateETag, Last-Modified
docs.stripe.comtext/markdown; charset=utf-8(none sent)(none)
knownbyllm.comtext/markdown; charset=utf-8public, max-age=3600, s-maxage=86400(none)

Two shapes recur. Either a short positive max-age of an hour, or max-age=0 with must-revalidate and an ETag, which makes every read a cheap conditional request. Both are fine. The choice to avoid is a long max-age with immutable, which is right for hashed assets and wrong for a file whose whole value is being current: a cached llms.txt that still lists last year’s pricing is the stale-copy problem in another form. If your CDN caches at the edge for longer, purge the path on deploy.

Redirects: one hop, same host, permanent

Every reader follows at least one redirect, and the validator follows them and reports the final status. The cases that lose files:

  • A different host. /llms.txt on your domain redirecting to a raw file on a code host or a pastebin. The final response has that host’s headers, often a generic type, and a reader that insists on a 200 at the origin stops at the 3xx.
  • A chain. http to https to www to a trailing slash. Each hop is a chance for a fetcher with a low redirect limit to give up. Collapse to one.
  • A 302 to a page. A login wall, a region picker, or a cookie notice that intercepts unknown paths. The final body is HTML, and the file is gone.
  • Mixed canonical hosts. The site is on www, the file is on the bare domain, and the links inside the file use a third form. Pick one origin, serve the file there, and write absolute links to it.

Use 301 or 308 for the one permanent hop; 308 preserves the request method, which matters for fetchers that probe with HEAD first. The validator samples your links with HEAD and falls back to GET when a server refuses HEAD, which is a reasonable model of what other tools do.

The optional Link header

If your pages also exist as Markdown, the spec’s example shows how to say so without touching the HTML:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

The spec notes the header form also works for non-HTML resources, such as the Markdown files themselves, and that it can be set in server or CDN configuration. It is optional, and none of the six sites above sent it when checked; treat it as the next step after the four headers are right.

Setting the headers on common hosts

nginx, for a static file:

location = /llms.txt {
    types { }
    default_type "text/markdown; charset=utf-8";
    add_header Cache-Control "public, max-age=3600";
}

Netlify, in a _headers file at the publish root:

/llms.txt
  Content-Type: text/plain; charset=utf-8
  Cache-Control: public, max-age=3600

Next.js, in next.config:

async headers() {
  return [
    {
      source: "/llms.txt",
      headers: [
        { key: "Content-Type", value: "text/plain; charset=utf-8" },
        { key: "Cache-Control", value: "public, max-age=3600" },
      ],
    },
  ];
}

On hosted builders you usually cannot set these; the no-root-access article covers what each platform sends and the proxy option when it is wrong. Build-time generation in Next.js, Astro, and Hugo is in the framework article.

Checking it

  1. Headers only, no redirects followed. Read the status, the Content-Type, and any Location:
    curl -sI https://example.com/llms.txt
    
  2. Follow redirects and report how many hops and where you landed:
    curl -sL -o /dev/null -w '%{num_redirects} %{http_code} %{content_type} %{url_effective}\n' https://example.com/llms.txt
    
  3. Check for a byte-order mark; the first three bytes must not be ef bb bf:
    curl -s https://example.com/llms.txt | head -c 3 | xxd
    
  4. Confirm the body is the file, not a page:
    curl -s https://example.com/llms.txt | head -n 3
    

The validator runs the same four checks in order, served at root, content type, encoding, then the Markdown rules, and reports the header value it saw. Run it after every change to hosting, not only after editing the file.

Validate your llms.txt

Paste your URL. The validator fetches /llms.txt the way a bot does, reports the status, Content-Type, and encoding it got back, then checks the Markdown against the spec and samples your links.

Validate →

FAQ

Which Content-Type should llms.txt have?

Either text/plain; charset=utf-8 or text/markdown; charset=utf-8. The specification does not name one, and both are what parsers expect. Of the public files checked for this article on 18 October 2026, Anthropic, OpenAI, Vercel, and Svelte serve text/plain and Stripe and this site serve text/markdown, all with charset=utf-8. What matters is that it is not text/html, which almost always means your host returned a fallback page instead of the file.

My host serves the file as application/octet-stream. Is that a problem?

Usually a sign of object storage or a CDN that did not recognise the extension. Most readers will still parse the bytes, and this site's validator only warns on it, but some fetchers treat an unknown binary type as not text and skip it. Set the type explicitly in the bucket metadata or the CDN rule; every provider has a way to do it for one path.

Does the charset parameter matter if the file is ASCII?

Not for an ASCII-only file. It matters as soon as the file contains a non-ASCII character, which includes a curly quote, an accented name, or any Japanese. Without a declared charset, a reader may guess, and a wrong guess turns the file into mojibake. Declare charset=utf-8 and save the file as UTF-8 without a byte-order mark.

Should llms.txt be cached?

Yes, briefly. The file is read on demand rather than crawled on a schedule, so a short public max-age such as an hour, with an ETag or Last-Modified so caches can revalidate cheaply, is the common pattern. Avoid long immutable caching unless your deploy also purges the path; a stale llms.txt is the first thing an assistant reads. The files checked for this article use max-age values between zero with must-revalidate and one hour.

Is a redirect from /llms.txt acceptable?

One hop from http to https, or from www to the bare domain, is normal and every reader follows it. A redirect to a different host, a chain of several hops, or a 302 that lands on an HTML page is where files get lost: a reader that only accepts a 200 at the root gives up, and the final response carries the other host's headers. Keep the file at the canonical origin and make the redirect a single 301 or 308.

What is the Link header the spec mentions?

Version 2 of the specification, modified in August 2026, describes two link relations: rel="alternate" with type="text/markdown" pointing to the Markdown version of a page, and rel="describedby" pointing to the llms.txt that covers it. They can be sent as HTML link elements or as an HTTP Link response header, which the spec notes can be added in web server or CDN configuration without touching pages. It is optional, and most sites do not send it yet.

Next steps