Do AI Answers Cite You? The Measurement Protocol, Pre-Registered

A citation list under an AI answer is the output of two stochastic processes stacked on each other: a retrieval pass over an index that is being rewritten while you read it, and a decode that samples. Nothing in that stack promises the same ten URLs twice. Run-to-run variation is not an anomaly to be explained away; it is the expected behaviour of the mechanism, and it sets the floor under every effect anyone claims to have caused.

That floor is the part the generative engine optimization literature keeps skipping. A tactic is applied, a prompt is re-run, the citation set changed, and the change is attributed to the tactic. The same procedure applied to no change at all would produce the same story, and nobody runs that arm.

So this post is the protocol, published before the results: the harness, the sampling design, the effect size that would count as an effect, the confounds the design cannot remove, and the commitment that the raw data goes up when the window closes whether it shows something or nothing. Everything below is executable today on your own site. None of it requires my numbers to be useful, which is the point of writing it in this order.

What is actually observable

Start by drawing the boundary, because the boundary is narrower than the discourse assumes.

You can observe answers to prompts you issue yourself, at a timestamp you record, from a locale and account state you control, with the citation set as rendered. Your own request logs, with agent tokens and source IPs you can check against published prefix lists. Impressions inside Google’s AI features, since the June 2026 launch of the Search Generative AI performance reports, for the subset of properties that have them. Citation counts and a sample of grounding queries from Bing, since AI Performance went to public preview in February 2026. Arrivals that carry a referrer or a parameter from a surface that links out.

Three things are claimed as observations and are not.

“We rank third in ChatGPT.” There is no position to hold. A citation set is assembled per answer; there is no persistent index position you can query, no stable ordering semantics across runs, and no guarantee the same set exists a minute later. Ordinal language imported from the SERP describes a structure that is not there. Google’s own help documentation sidesteps the question differently but tellingly: an AI Overview occupies one position, and every link in it is assigned that same position.

“We were cited for query X.” Unless you issued X yourself, you did not observe that. The prompt is on the other side of the boundary. Bing’s grounding queries are the only surface that returns anything at all here, and Bing describes them as a sample of overall citation activity, which is a different object from a query report. Google’s generative AI report has pages, countries, devices and dates. It has no queries and no clicks.

“The model used our page without citing it.” Not observable from outside. You can observe that a claim only your page publishes appears in an answer, which is suggestive and worth recording, but a shared upstream source explains it equally well. Treat it the way you would treat any unfalsifiable mechanism claim. The same standard I use for grading patent-derived claims applies here without modification.

The variance you have to measure first

Before any intervention, the harness runs against an unchanged site for long enough to estimate four things separately.

Within-day, run-to-run: the same prompt, same locale, same hour, repeated. This is the decode and retrieval sampling, and it is the largest component in my expectation, though that expectation is exactly what the pre-period is for.

Then the same prompt at the same hour on different days, which mixes index refresh with whatever else drifts.

Locale and account state come next: the same prompt from different countries, and from a signed-out versus signed-in session. Anything with personalization makes the account a variable, not a constant.

Model version is not noise. A version change is a regime change, and an intervention window that straddles one has a discontinuity in it. Record the model label the surface reports on every run, and pre-commit to treating a version change as a segment boundary rather than pooling across it.

Now the arithmetic that decides how many runs you need. Citation is a binary outcome per run, so the sampling error on a citation rate is the binomial one. At a true rate near 0.3, thirty runs gives a standard error of 0.084 and a 95 percent interval roughly sixteen points wide on each side. Concretely, nine citations in thirty runs yields a Wilson interval of about [0.167, 0.479].

from math import sqrt

def wilson(k, n, z=1.96):
    p, d = k / n, 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return centre - half, centre + half

print(wilson(9, 30))    # ~ (0.167, 0.479)
print(wilson(0, 30))    # ~ (0.000, 0.114)

Read the second line carefully, because it is the most useful number in the post. Never being cited in thirty runs is consistent with a true citation rate as high as about 11 percent. A zero is not an absence.

Thirty runs per prompt is a screening instrument. It tells you which prompts your site is in contention for. It cannot resolve the size of an intervention on its own, and here is why: to distinguish a rate of 0.25 from 0.35 at 80 percent power and the conventional 5 percent two-sided level takes roughly 326 observations per arm. To distinguish 0.30 from 0.35 takes roughly 1,374. Those are the honest numbers for the effect sizes people casually report.

Runs are also clustered inside prompts, and the clustering is severe: whether a prompt cites you at all is mostly a property of the prompt. With thirty runs per prompt and an intra-cluster correlation of 0.5, the design effect is 15.5, meaning 300 runs carry about as much information as 19 independent ones. So the unit of analysis is the prompt, not the run. Runs estimate a prompt’s citation rate; prompts are what you count.

Pre-register the run count and the window before you look at any of it. Deciding how long to sample after seeing the first week is how a null becomes a finding.

The prompt panel, the cadence and the storage schema

The panel is the experiment’s external validity, and it is the step that gets rushed. Build it by stratifying rather than by picking prompts you hope you win.

Sample from your actual query footprint, not from a keyword tool: pull the queries your pages already receive impressions on, then write prompts a person would plausibly type into an assistant to satisfy that same need. Stratify on intent shape, on whether the topic is commercially contested, on whether your page is currently a top-ten organic result for the underlying query, and on freshness sensitivity. Include a control stratum of prompts you have no plausible claim to, because a panel that only contains prompts you win cannot show you a loss. The SERP fingerprint fields you already record for a query transfer directly: the AI surface’s source set is another field on the same record.

Size follows from the arithmetic above. If the prompt is the unit and you want to compare arms, you need prompts in the dozens per arm, not five hand-picked ones. Cadence: fix an hour, fix a locale set, run the whole panel every day rather than opportunistically, and never re-run a prompt because the answer looked wrong.

Store runs and citations separately, and store the raw payload so that a change in your extraction logic can be replayed against old data instead of invalidating it.

CREATE TABLE run (
  run_id      TEXT PRIMARY KEY,
  prompt_id   TEXT NOT NULL,
  surface     TEXT NOT NULL,
  model_label TEXT NOT NULL,
  locale      TEXT NOT NULL,
  started_at  TIMESTAMP NOT NULL,
  answer_text TEXT NOT NULL,
  raw_payload JSON NOT NULL
);

CREATE TABLE citation (
  run_id   TEXT NOT NULL REFERENCES run(run_id),
  position INTEGER NOT NULL,
  url      TEXT NOT NULL,
  host     TEXT NOT NULL,
  norm_url TEXT NOT NULL
);

Citation extraction has three failure modes worth naming. Cited links are frequently wrapped in a redirector or decorated with tracking parameters, so resolve and strip before you store norm_url, and keep the raw url beside it. The same host can appear twice in one answer, so decide once whether the unit is host-per-run or URL-per-run and never mix them mid-window. And answers cite a URL that redirects to your canonical often enough that host-level matching and URL-level matching disagree; record both columns and report which one you used.

The two variance queries you run every morning:

WITH per_run AS (
  SELECT r.prompt_id, r.run_id, count(DISTINCT c.host) AS hosts
  FROM run r LEFT JOIN citation c USING (run_id)
  WHERE r.started_at >= now() - INTERVAL 14 DAY
  GROUP BY 1, 2
),
ever AS (
  SELECT r.prompt_id, count(DISTINCT c.host) AS hosts_ever
  FROM run r JOIN citation c USING (run_id)
  GROUP BY 1
)
SELECT p.prompt_id, avg(p.hosts) AS mean_per_run, any_value(e.hosts_ever) AS hosts_ever
FROM per_run p JOIN ever e USING (prompt_id)
GROUP BY 1
ORDER BY hosts_ever DESC;

The ratio of hosts_ever to mean_per_run is the churn measure. A prompt where eight hosts are cited per run and forty distinct hosts appear across two weeks is a prompt where a before/after screenshot proves nothing at all.

The server side: training crawl versus answer-time retrieval

The log side is where the discourse is loosest, because “AI crawler traffic” is used as one category when the operators themselves publish three. The distinction that matters is whether a fetch happens once, into a corpus, or at the moment somebody asks a question.

Those are the exact tokens from OpenAI’s bot documentation, Anthropic’s crawler page, Perplexity’s bot guide and Google’s common crawlers reference. Four of them carry qualifications you should read before building a classifier on them.

Google-Extended and Applebot-Extended never appear in a log. Google’s documentation is explicit: Google-Extended has no separate HTTP user agent string, crawling is done with existing Google user agent strings, and the robots.txt token is used in a control capacity. It governs training and grounding for Gemini Apps and Vertex AI, and Google states it does not impact inclusion in Google Search nor act as a ranking signal there. Apple’s Applebot page is structured the same way: it documents user agent strings for Applebot only, and presents Applebot-Extended purely as a robots.txt opt-out from foundation-model training. A robots.txt line addressed to either token produces no observable event either way, so neither belongs in a log-side classifier.

Perplexity-User is documented as generally ignoring robots.txt because the request originates from a user action. That is the same rationale Google gives for its user-triggered fetchers, which now include Google-Agent and Google-GeminiNotebook. Disallowing a user-triggered agent is a request, not a control.

The classifier over combined-format logs is four lines:

awk 'BEGIN{OFS="\t"}
  /GPTBot|ClaudeBot/                                      {c="train"}
  /OAI-SearchBot|Claude-SearchBot|PerplexityBot|bingbot/  {c="index"}
  /ChatGPT-User|Claude-User|Perplexity-User|Google-Agent/ {c="answer"}
  c != "" { print substr($4,2), $1, $7, $9, c; c="" }
' access.log > ai-fetches.tsv

That classification is worthless without verification, exactly as it is for Googlebot in your access logs. The user agent is a string an attacker or a scraper types. Every operator above publishes a prefix file, so check against it:

curl -s https://openai.com/chatgpt-user.json | python3 -c '
import json, sys, ipaddress as ip
nets = [ip.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"])
        for p in json.load(sys.stdin)["prefixes"]]
for line in open("candidate-ips.txt"):
    a = ip.ip_address(line.strip())
    print(line.strip(), any(a in n for n in nets))'

Two structural limits fall out of doing this properly, and both belong in any honest writeup.

Anthropic publishes one prefix file for all three bots. Verifying the source IP tells you the request came from Anthropic. It does not tell you whether it was ClaudeBot, Claude-SearchBot or Claude-User, because only the user agent string separates them, and the user agent string is the field you just refused to trust. On that operator, the training-versus-retrieval split rests on an unverifiable header. Say so in your results rather than presenting the split as if it were measured.

Google’s AI features leave no answer-time footprint at all. AI Overviews and AI Mode ground on the Search index. There is no answer-time fetcher token for them, so in your logs an AI Overview appearance and an ordinary blue-link ranking are the same Googlebot request. This is the single largest blind spot in the design and it is not fixable from the server side. On Google, the harness and the Search Console impressions series are the only instruments you have.

The cryptographic fix is in motion but not arrived. Google’s own fetcher documentation notes it is experimenting with the Web Bot Auth protocol under the identity https://agent.bot.goog. Those are individual IETF drafts, not adopted standards. Design your log schema so a signature-verified column can be added later; do not build a conclusion on it now.

What the timing tells you: an answer-class fetch is a request that happened because someone asked something. Its arrival distribution over the day should look like human demand, not like a crawler’s steady rate. If your answer-class fetches arrive at a flat rate around the clock, something is polling you and it is not a person, and you should say that in the writeup before someone else notices.

Attribution: what arrives, what is structurally invisible

Three categories, and only one of them is a referral.

Some arrivals carry identity. Surfaces that link out and preserve a referrer: chatgpt.com, perplexity.ai, copilot.microsoft.com, gemini.google.com, claude.ai. ChatGPT also appends a utm_source=chatgpt.com parameter to cited links, which is convenient and undocumented; OpenAI’s bot documentation says nothing about it. Build your channel definition on the referrer first and the parameter second, and re-check that the parameter still appears every month, because an undocumented behaviour can be withdrawn without a changelog.

Some are structurally invisible. A click from an AI Overview or AI Mode arrives as an ordinary Google organic referral. Google’s own guidance confirms the reporting side of this: sites appearing in AI features are included in overall search traffic in Search Console, reported in the Performance report within the Web search type. The dedicated generative AI report gives impressions, pages, countries, devices and dates, with no clicks, no queries and no backfill before its launch window. Which means the number “clicks from AI Overviews” does not exist in any first-party source. If you see it in a deck, it was modelled. Beyond that: answers consumed without a click leave nothing, desktop and mobile app surfaces frequently strip the referrer into direct, and an answer copied into a document is not instrumented anywhere.

What to instrument instead: define the AI channel server-side over raw access logs rather than only in your analytics product, so consent gating and parameter stripping cannot silently delete the channel. Then run the join that is actually informative: an answer-class fetch of a URL, followed within minutes by a human session on the same URL from a different address. That pair is the closest thing to a causal trace the current web offers, and the lag distribution is worth a chart on its own. Alongside it, keep the Search Console generative AI impressions series and the Bing citation series as two independent, differently-flawed views. And keep tracking clicks against position for the same queries, because CTR loss at unchanged position is the mechanism people misread as an AI citation effect.

The intervention: treatment, control, and a pre-declared null

The design is a difference-in-differences on prompt-level citation rates, with pages randomized into arms.

Randomize at the page level within matched strata. Match on template, topic cluster, baseline citation rate from the pre-period, and baseline organic impressions. Randomize within stratum. Hand-picking the treatment pages guarantees regression to the mean does the work.

The treatment is one change, a single manipulation stated in advance. If the treatment involves editing page content, the edit itself triggers a recrawl, and recrawl is a confound you have now baked in. The fix is a sham edit: control pages receive an edit of comparable size that does not implement the treatment. Without it you are measuring “was this page touched”, not “did this change help”.

Pre-declare the endpoint, the window and the minimum effect. Primary endpoint: change in prompt-level citation rate for treated pages minus the same change for controls. Fixed window, no peeking, no early stop. Pre-commit to the smallest difference worth acting on. If you would not change your behaviour for five points, do not run a design that can only resolve twenty.

Report a bootstrap interval over prompts, not runs. The clustering makes run-level intervals dishonestly narrow.

import random, statistics

def bootstrap_did(treated, control, iters=10000, seed=0):
    """treated / control: one (pre_rate, post_rate) tuple per prompt."""
    rng = random.Random(seed)
    point = lambda t, c: (statistics.mean(b - a for a, b in t)
                          - statistics.mean(b - a for a, b in c))
    draws = sorted(point([rng.choice(treated) for _ in treated],
                         [rng.choice(control) for _ in control])
                   for _ in range(iters))
    return point(treated, control), draws[iters // 40], draws[iters - iters // 40]

What this design cannot rule out, stated now rather than negotiated later. Interference between units: treatment and control pages share a hostname, so any domain-level response contaminates both arms and biases the estimate toward zero. A model version change inside the window. Index refresh lag, which means a short post-period measures propagation rather than effect. Locale and account personalization, controlled only for the locales you sampled. And the obvious one: a null on this panel is a null on this panel, not a statement about every site.

The null is a publishable outcome and is pre-declared as such. If the interval covers zero, the result is that the intervention did not move citation rate by more than the design could detect, and the interval is the finding.

llms.txt, judged by what the logs would have to show

The llms.txt proposal was published by Jeremy Howard on 3 September 2024 and revised to v2 on 10 August 2026. It specifies a Markdown file at /llms.txt or any subpath, with an H1 project name as the only required section, an optional blockquote summary, and H2-delimited lists of links. v2 adds companion .md versions of pages at the same URL and rel="alternate" type="text/markdown" plus rel="describedby" link relations, available as an HTTP Link: header so a CDN can add them without touching pages.

It is a proposal. It is not an RFC, unlike the robots exclusion protocol, which is RFC 9309. No search or assistant operator has published a commitment to consume it. Google’s AI features guidance is as close to a statement as exists and it goes the other way: you do not need to create new machine readable files, AI text files, or markup to appear in these features. Anthropic’s crawler page, updated April 2026, discusses robots.txt and the non-standard Crawl-delay extension and mentions nothing else. OpenAI’s and Anthropic’s developer sites both publish an llms.txt for their own documentation, which is a statement about their docs, not about their crawlers.

So stop arguing about it and instrument it. Three log conditions, in increasing strength.

The weak condition is that /llms.txt gets requested at all, by an IP that verifies against a published prefix list. A single probe by one operator proves curiosity, not consumption.

Moderate: the .md companion URLs listed in the file are requested by the same verified agent, shortly after the file itself, in that order. Ordering is the signal; two unrelated fetches are not a parse.

Decisive is a canary. Publish a page at a URL that appears nowhere except inside llms.txt: not linked internally, not in any sitemap, not submitted anywhere, allowed in robots.txt, containing a unique nonsense token. Any fetch of that URL means something parsed the file. The token later appearing in a harness answer means something retrieved and used it.

awk '$7 ~ /^\/llms\.txt/ || $7 ~ /^\/ref\/canary-/ {print substr($4,2), $1, $7, $9}' \
  access.log | sort

This is the cleanest test in the whole protocol, because it has no statistics in it. Either the canary is fetched or it is not, and one line of the log settles a debate that has consumed two years of conference slides. Note the ceiling: a fetch refutes “nobody reads the file” and nothing more. Whether the file changes what gets cited is a question only the intervention arm can answer.

What I would act on, and what I would ignore

I would act on the things whose mechanism I can verify in a log the same day.

First, whether answer-time fetchers can reach you at all. Check that robots.txt does not disallow ChatGPT-User, Claude-User, Perplexity-User or OAI-SearchBot, and then check the thing people forget: whether your CDN or WAF is returning 403 to them regardless of what robots.txt says. Bot-mitigation defaults and a robots.txt policy are different systems and they disagree more often than you would like. The tokens are separate precisely so you can allow retrieval while refusing training. That is a business decision, but make it deliberately rather than discovering it in a rule you inherited.

Then error rate and latency for those agents specifically. A 5xx or a timeout served to a Claude-User request is a citation that did not happen, and it is invisible in every aggregate you normally look at. Segment your error budget by the agent class from the awk above.

Whether the fetchers see what browsers see is answerable from your own logs rather than by assertion. Check whether a given agent requests your JavaScript bundles and XHR endpoints after the document. If it does not, the rendered-only content on that template is not part of what it retrieved.

And the correctness and the dating of the facts on the page. The failure mode the harness surfaces most legibly is being cited accurately for a number that is now wrong.

I would ignore, until the design says otherwise, the following. llms.txt, until a canary fires: publishing one costs nothing, and that is exactly why it is not evidence of anything. Schema types with no documented consumer. Any vendor visibility score that does not disclose run count, locale and model version, because everything in the variance section applies to their sampling too and you cannot check whether they accounted for it. Before-and-after screenshots of a single prompt. And optimizing against the observed behaviour of a model version that will be replaced inside your measurement window.

That is the boundary between what the instrument can see and what it cannot. The instrument is described above in enough detail to build; what it returns on my panel is a separate post, and it publishes with the raw JSONL, the panel definition and the analysis notebook attached, on the date the window closes, whichever way it comes out.