ClaudeBot, Claude-User, Claude-SearchBot: how Anthropic crawls and what llms.txt changes

The 30-second answer: block ClaudeBot to opt out of training, allow the other two to stay visible, and treat llms.txt as a bonus

Anthropic runs three bots. ClaudeBot collects web content that may contribute to model training; disallowing it signals that your future material should be excluded from training datasets, and nothing else. Claude-SearchBot indexes content to improve search result quality, and Claude-User fetches a page when a person asks Claude a question; disallowing either of those, Anthropic says, may reduce your visibility in user search results. All three are documented as honouring robots.txt, including the non-standard Crawl-delay.

On llms.txt, Anthropic publishes one for its own developer documentation and says nothing about reading yours. The public log evidence shows a handful of fetches. Write your pages for Claude-SearchBot and Claude-User; publish llms.txt for the tools that read it.

What Anthropic documents

The source is one help-center article, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, last updated 7 April 2026. The parts a site owner needs:

• ClaudeBot helps enhance the utility and safety of Anthropic’s generative AI models by collecting web content that could potentially contribute to their training. Disallowing it signals that the site’s future materials should be excluded from training datasets.
• Claude-User supports Claude users: when a person asks Claude a question, it may access a website using this agent. Disabling it prevents Anthropic’s system from retrieving your content in response to a user query, which may reduce your visibility for user-directed web search.
• Claude-SearchBot navigates the web to improve search result quality for users. Disabling it prevents indexing of your content for search, which may reduce your visibility and accuracy in user search results.
• robots.txt. Anthropic’s bots respect do-not-crawl signals by honouring industry-standard directives in robots.txt. A block goes in the robots.txt at the top level of each subdomain you want to opt out.
• Crawl-delay. Anthropic supports the non-standard Crawl-delay extension, with an example of a one-second delay for ClaudeBot, and says it aims for minimal disruption by respecting it where appropriate.
• IP addresses. The crawlers’ IPs are published at claude.com/crawling/bots.json. A source IP on that list indicates the crawler is coming from Anthropic. Blocking by IP is not recommended as an opt-out, because it may not work correctly or persistently.
• Anti-circumvention. The bots respect anti-circumvention technologies and will not attempt to bypass CAPTCHAs.

The IP file is the same shape as OpenAI’s and Perplexity’s: a creation time plus a list of IPv4 prefixes. One difference is that Anthropic publishes a single list for all three bots, where OpenAI publishes one JSON per agent. When this article was checked the file was dated 7 October 2026, so expect it to change and re-fetch it rather than hard-coding the ranges.

What each block actually does

The decision most sites want is “do not train on my content, but do cite me”, and Anthropic’s split makes it a two-line choice. The mistake to avoid is a single rule for anything containing “Claude”: that removes you from the answers to get out of the training set.

How this differs from OpenAI and Perplexity

1. The user fetcher. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiated the action; Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic lists Claude-User among the bots that honour robots.txt. It is the only one of the three vendors to say so.
2. Crawl-delay. Anthropic supports it; OpenAI and Perplexity do not document it; Google ignores it entirely. If your server is sensitive to crawl rate, this is the one vendor where a robots.txt line is enough.
3. Opting out of training. All three vendors separate training from search, and all three say the training block does not affect visibility. Anthropic’s wording is the most explicit about what the block means: future material excluded from training datasets.
4. What is not documented. Anthropic does not say how often Claude-SearchBot recrawls, how long a robots.txt change takes to be reflected, or whether JavaScript is rendered. OpenAI gives a figure of about 24 hours for robots.txt changes; Perplexity says up to 24 hours. Assume a day and check the logs.

The full user-agent table across vendors is in the crawler list, and the Googlebot comparison explains why the three-job split exists at all.

Two robots.txt recipes

Cite me, do not train on me. Name the groups explicitly, so that none of the three falls back to the wildcard:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

Allow everything, but slow the training crawler down. Keep the delay in a named group; Google will ignore it and a wildcard Crawl-delay confuses tools:

User-agent: ClaudeBot
Crawl-delay: 5
Allow: /

Remember that robots.txt is per host. Anthropic’s page says to repeat the block in the robots.txt of every subdomain you want to opt out, so docs.example.com and example.com each need their own.

What llms.txt changes for Claude

Anthropic publishes an llms.txt for its developer documentation at platform.claude.com. When this article was checked it was an index of 760 Markdown links grouped by section, with a pointer to an llms-full.txt that concatenates the whole documentation set and to an MCP server for searching it. That is a strong signal that Anthropic finds the format useful for the tools that read documentation: coding agents, editors, and anything that can fetch a URL when told to.

It is not a statement that Claude’s bots read the file from your site, and the crawler page does not mention llms.txt at all. The evidence from logs points the same way. The 12-week panel of 83 sites described in this site’s ChatGPT article counted 9 llms.txt fetches from Anthropic against 3,120 robots.txt requests: not zero, but not a crawl target either.

So the file changes two things for Claude, and neither is the search index. First, when a user pastes your URL into Claude or a Claude-based agent and asks it to read your docs, a well-built llms.txt gives the fetcher a map instead of a homepage. Second, Claude Code and similar tools can be pointed at the file directly. For Claude-SearchBot the surface is your pages: text in the HTML, the answer in the first paragraph, dates on everything. The how-to guide covers writing the file for the readers that exist.

Checking your logs

1. Search access logs for the three user-agent tokens: ClaudeBot, Claude-SearchBot, Claude-User. Count each per week.
2. Verify a sample against claude.com/crawling/bots.json. A hit with a Claude user agent from an address outside the list is not Anthropic.
3. If ClaudeBot is heavier than you like, add Crawl-delay and watch the next week’s count. If Claude-SearchBot never appears, check that nothing in front of robots.txt (a CDN or firewall rule for non-browser agents) is challenging it.

The first check is the one that fails silently. A checker fetches your robots.txt the way a bot does and reports what each of Anthropic’s three agents, and the other common ones, is allowed to read.

FAQ