KnownByLLM

How-to · 9 min read

How to measure AI search impact

The four signals that actually move.

There is no Google Search Console for ChatGPT yet. The marketers who say “we got 30% lift from llms.txt” are usually pointing at a single chart and ignoring the rest. That doesn’t mean impact is unmeasurable — it means you have to triangulate from four different signals, each imperfect on its own.

This is the playbook we use ourselves. It assumes you already have llms.txt published and you want to know whether it’s doing anything.

The four signals worth tracking

  1. 01

    Server-log crawler activity

    How often AI bots — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended — actually fetch your pages and your llms.txt. The first thing that moves after you publish.

  2. 02

    Referral traffic from AI domains

    Real users clicking through from chat.openai.com, claude.ai, perplexity.ai, gemini.google.com, copilot.microsoft.com. Visible in GA4 or your analytics tool of choice.

  3. 03

    Manual citation audits

    Once a month, ask the AI assistants the questions your customers ask. Note whether your site appears as a cited source. Tedious but irreplaceable — it's the only direct signal of citation share.

  4. 04

    On-site behaviour from AI traffic

    Bounce rate, session depth, conversions on the segment of users who arrived from AI domains. Tells you whether the traffic is qualified, not just how much there is.

Each on its own can mislead. Together they paint a fairly clear picture within 4–8 weeks of publishing.

1. Server-log crawler activity

What you’re measuring: Are AI assistants actually fetching your llms.txt and the pages it points to?

Why it matters: If AI bots aren’t hitting the file, nothing else can work. This is the leading indicator and the troubleshooting baseline.

How to capture it

On a traditional server, parse your access logs for these User-Agent strings:

GPTBot/         (OpenAI / ChatGPT browsing)
ClaudeBot/      (Anthropic / Claude)
PerplexityBot/  (Perplexity)
Google-Extended (Google AI products)
Applebot-Extended (Apple Intelligence)
Bingbot          (used by Copilot via Bing index)

On a modern stack you usually have something better. Common options:

  • Cloudflare Logpush: ship HTTP logs to R2 / S3 and run SQL on top. cf-bot-management categorises AI crawlers natively.
  • Cloudflare Workers Analytics Engine: fire-and-forget event stream, query with SQL for 90 days. This site uses one for /api/crawl hits, for example.
  • Vercel Web Analytics: aggregates AI bot traffic into a dedicated “AI” segment.
  • Plausible / Fathom / Umami: most modern analytics tools now break out AI bot traffic in a separate report.

What to look at

  • Daily fetches of /llms.txt by bot. Should be non-zero within a week of publishing.
  • Daily fetches of the URLs listed inside your llms.txt. The pages you’ve curated should see disproportionately more bot traffic than other pages.
  • Status-code distribution. 4xx or 5xx responses to AI bots are a regression — fix them before measuring anything else.

Common gotcha: a robots.txt rule that disallows GPTBot or ClaudeBot will silently kill the entire funnel. Audit it once before you start measuring.

2. Referral traffic from AI domains

What you’re measuring: Real humans clicking through from an AI assistant’s answer.

Why it matters: This is the only signal that proves the citation translated into a visit. AI bots fetching the page is necessary but not sufficient.

The referrer list to track

chat.openai.com
chatgpt.com
claude.ai
perplexity.ai
www.perplexity.ai
copilot.microsoft.com
gemini.google.com
search.brave.com  (Brave AI)
you.com           (You.com Smart)

In GA4, create a custom channel group called “AI search” and add those domains. The session attribution will look right out of the box, and you can build reports against it.

What to look at

  • Sessions per week, segmented by referrer. Watch the trend, not the absolute number — early-stage AI traffic is noisy.
  • Landing pages by AI referrer. AI traffic almost never lands on the home page. The pages that win are usually specific service pages, pricing pages, comparison pages, and technical documentation. If your llms.txt entries are surfacing the right pages, those are the ones that should get the traffic.
  • Search query (when available). Bing, Google AI Overviews, and some Perplexity referrals carry a query string. Treat them like organic search queries.

3. Manual citation audits

What you’re measuring: When a user asks an AI assistant the questions your site answers, does your site show up as a cited source?

Why it matters: It’s the only direct view of citation share. Bots fetching and visitors clicking are downstream of this.

How to run it

  1. Pick 5–10 representative queries. Use your real customer-search queries, the things people email you about, the keywords you optimise pages for, and a couple of comparison-style prompts (“X vs Y”, “best X for Y”).
  2. Run them in a logged-out browser session against ChatGPT, Claude, Perplexity, Google AI Overviews, and optionally Copilot. Use private/incognito to avoid personalised answers.
  3. Record three things per query × per assistant: cited or not, position in the citation list, and snippet text (whose wording got picked up).
  4. Re-run monthly. AI answers shift more slowly than Google rankings — monthly cadence is plenty for trend tracking.

What to track

  • Citation rate = (queries where you appear) / (total queries). Compare across assistants and over time.
  • Citation position. Top citation matters more than fifth.
  • Snippet attribution. If the AI is paraphrasing you, you’re cited but not quoted. If it’s quoting you, your wording is propagating into other people’s answers — that’s the strongest form of AI citation.

For a small site, 10 queries × 4 assistants × monthly is about an hour of work. Worth it. Tools that automate this (Otterly, Profound, Athena, etc.) exist but the manual baseline is what calibrates them.

4. On-site behaviour from AI traffic

What you’re measuring: Once AI traffic arrives, does it convert?

Why it matters: If 200 visitors a week from ChatGPT bounce immediately, your citation may be misleading users — or your page may be loading badly, or the AI may be quoting an outdated section. Volume without conversion is a problem to investigate, not celebrate.

The metrics to compare

  • Bounce rate for the AI-search segment vs. your site average.
  • Pages per session. AI users typically read one page deeply rather than browse — pages-per-session of 1.2–1.5 is normal and healthy.
  • Conversion rate on the AI-search segment. Conversions can be anything you care about: form submissions, free-trial signups, newsletter signups, demo bookings.
  • Time on page. Surprisingly low time-on-page with high bounce can mean the AI is sending visitors who already got their answer in the AI’s reply — they’re just double-checking the source. That’s actually a good outcome, not a bad one. Read it in context.

A simple monthly dashboard

You don’t need a fancy BI tool. A spreadsheet with one row per month and these columns is enough to manage by:

  • llms.txt fetches by GPTBot / ClaudeBot / PerplexityBot
  • Page fetches by the same bots (sum across all URLs in your llms.txt)
  • AI-search referral sessions (from analytics)
  • AI-search referral conversions
  • Citation rate from your monthly query audit
  • One-line note: anything that changed (content, llms.txt, robots.txt)

Three months of rows and you have a baseline. Six months and you can tell trend from noise.

What not to measure

  • Vanity counts of total bot hits. A bot fetching/llms.txt 10,000 times because of a misconfigured cache header is not impact.
  • Anonymous “AI traffic” clusters in logs that don’t identify the assistant. Treat unidentified traffic as background noise; track only sessions you can attribute.
  • One-off citation appearances. An AI assistant cited you once doesn’t mean a pattern. Look at trends over 4+ weeks before claiming impact.

FAQ

How long until llms.txt shows measurable impact?

AI bot crawls (GPTBot, ClaudeBot, PerplexityBot) usually appear in your server logs within days. Citation impact and on-site referrals build over weeks, not days, because they depend on users asking AI assistants the questions your pages answer. A reasonable expectation: weeks 1–2 you should see crawler activity in logs, weeks 4–8 you should see early referral traffic, and the first quarter is when steady patterns emerge.

What does AI referral traffic look like in Google Analytics?

AI assistants link out using their own domains. Common referrers include chat.openai.com, chatgpt.com, claude.ai, perplexity.ai, copilot.microsoft.com, and gemini.google.com. Set those as a custom channel group called 'AI search' in GA4. Sessions usually look short and high-intent — visitors arrive on a specific page, often deep in your site, and bounce or convert quickly.

Does Google Search Console track AI Overviews citations?

Partially. As of 2026, Google AI Overviews citations are bundled into ordinary Search Console impressions and clicks for the underlying query. There is no dedicated 'AI Overview' filter, but pages that get cited typically show a higher impressions-to-clicks ratio because the snippet on the SERP answers the question directly. Compare your impressions trend against your competitor mentions in AI answers to back out the impact.

Can I A/B test llms.txt?

Not cleanly. AI assistants are not split-testable on demand the way Google rankings can be A/B tested with redirects. The closest you can do is a before/after comparison: measure the baseline for 4 weeks, publish llms.txt, and watch the same metrics for the next 4–8 weeks. Larger sites can split-test by section (publish llms.txt entries pointing to half the docs sections, watch which sections get cited more).

What's the single best leading indicator?

AI bot fetch counts in your server logs. They move within days of publishing, they're easy to capture, and they correlate with eventual citation activity. If GPTBot and ClaudeBot are not fetching your llms.txt at all after two weeks, something is wrong (blocked in robots.txt, returning the wrong Content-Type, or the file is not at the root). Fix that first; everything else depends on it.

Should I track every AI bot or just the major three?

Track GPTBot, ClaudeBot, and PerplexityBot at minimum — those are the volume drivers. Add Google-Extended (Google's AI training crawler) and Applebot-Extended for completeness if you care about Bard/Gemini and Apple Intelligence respectively. A long tail of smaller bots exists but rarely drives meaningful citation traffic; tracking them is usually noise.

Next steps