Reading Googlebot in Your Access Logs: Demand, Capacity, Waste

Googlebot is not a crawler. It is the name a shared fetching platform writes into the User-Agent header when it fetches on behalf of Google Search. Google put it plainly in a March 2026 post on its crawling infrastructure: “When you see Googlebot in your server logs, you are just looking at Google Search,” and dozens of other clients route through that same infrastructure under different names.

That distinction is why most log analysis is wrong before the first query runs. Google’s framing of crawl budget is a capacity limit multiplied by demand, where the capacity limit governs “the total amount of time your server spends holding connections open for Google.” Both terms sit in your access log at request granularity, in storage you own. Search Console publishes a summary; the log is the observation. What follows is the pipeline that turns one into the other.

The log you need, not the log you have

Twelve fields. Two of them are missing from almost every default log format.

Duration and the query string are the two that get dropped: nginx and Apache both omit duration by default, and many pipelines strip query strings before the log reaches storage, deleting the evidence for the largest waste class most sites have.

Log tab-separated, not Combined Log Format. Nginx escapes control characters by default, so a tab inside a user agent becomes \x09 and cannot break the delimiter:

log_format crawl '$time_iso8601\t$remote_addr\t$host\t$request_method\t$request_uri\t'
                 '$status\t$body_bytes_sent\t$request_time\t$upstream_response_time\t'
                 '$upstream_cache_status\t$server_protocol\t$http_user_agent';
access_log /var/log/nginx/crawl.tsv crawl;

The Apache equivalent, using mod_log_config format strings:

LogFormat "%{%Y-%m-%dT%H:%M:%S%z}t\t%a\t%v\t%m\t%U%q\t%>s\t%B\t%{ms}T\t%H\t%{User-Agent}i" crawl
CustomLog logs/crawl.tsv crawl

%U adheres to MergeSlashes, so //products and /products collapse into one row; add %r if doubled slashes matter to you. Retain 180 days minimum. Crawl-to-index latency needs a window longer than the latency being measured, and Search Console’s history goes back only 90 days.

CDN and origin see different sites

If a CDN fronts your site, your origin log is missing every fetch the edge served from cache. The undercount equals the bot cache-hit rate and is not uniform across templates: static-heavy templates look under-crawled, uncacheable ones look over-crawled, and both are artifacts. This is the most common reason a crawl analysis reaches a confident wrong conclusion.

Edge logs fix that and introduce their own gaps. In Cloudflare’s http_requests dataset, ClientIP is the real crawler address, but EdgeResponseStatus and OriginResponseStatus can differ, and origin latency is invisible unless you ship OriginResponseDurationMs. Check whether your edge product samples: if it does, every absolute rate below is wrong by the sampling factor while every ratio holds. Behind a proxy, $remote_addr is the CDN, and you must take only the hop your own proxy appended; DNS-verifying a client-supplied X-Forwarded-For verifies nothing.

Three CDN behaviours change the numbers themselves, all documented in Google’s post on CDNs and crawling. The capacity threshold is raised when the infrastructure detects a CDN. A cold cache means the origin still serves every URL at least once, so a large launch hits it at full crawl rate anyway. And WAF blocks are easy to misread: a 503 or 429 is legible, a network timeout is terminal, and a bot challenge served with HTTP 200 is the worst case, because it looks like a successful fetch in your log and like a duplicate page to Google. Grep verified-Google rows for 403s and suspiciously uniform response sizes before concluding anything about demand.

Verifying Googlebot without trusting the user agent

The user agent is a request header. The client picks it. Any analysis that filters on the string Googlebot is an analysis of everyone who typed Googlebot.

Google documents two checks. The first is a reverse DNS lookup on the client IP, whose hostname must sit under googlebot.com, google.com or googleusercontent.com, followed by a forward lookup on that hostname returning the original IP. The second is membership in the published prefix files, five of them, all under https://developers.google.com/static/crawling/ipranges/:

They moved in February 2026 from /search/apis/ipranges/, each carries a creationTime, and they change; fetch them on a schedule and never vendor a static copy. Use range membership as the primary test, since it is offline and deterministic, and the DNS pair to adjudicate the residue: a newly rotated address can appear in your log before it appears in the file.

The two checks, in code

Fetch the prefixes into a flat CSV the SQL side can join against:

import ipaddress, csv, json, urllib.request

BASE = "https://developers.google.com/static/crawling/ipranges/"
FILES = ["common-crawlers", "special-crawlers", "user-triggered-fetchers",
         "user-triggered-fetchers-google", "user-triggered-agents"]

with open("google_prefixes.csv", "w", newline="") as out:
    w = csv.writer(out)
    w.writerow(["crawler_class", "prefix"])
    for name in FILES:
        with urllib.request.urlopen(BASE + name + ".json", timeout=20) as r:
            doc = json.load(r)
        for p in doc["prefixes"]:
            cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
            ipaddress.ip_network(cidr)        # fail loudly on a malformed file
            w.writerow([name, cidr])
        print(name, doc["creationTime"], len(doc["prefixes"]))

The DNS adjudicator, reading one address per line on stdin:

import socket, sys

SUFFIXES = (".googlebot.com", ".google.com", ".googleusercontent.com")

def verdict(ip):
    try:
        host = socket.gethostbyaddr(ip)[0]
    except OSError:
        return "no_ptr", ""
    if not host.endswith(SUFFIXES):     # leading dot matters: notgoogle.com must fail
        return "foreign_host", host
    try:
        forward = {a[4][0] for a in socket.getaddrinfo(host, None)}
    except OSError:
        return "no_forward", host
    return ("verified" if ip in forward else "forward_mismatch"), host

for line in sys.stdin:
    ip = line.strip()
    print("\t".join((ip,) + verdict(ip)))

Two details break homegrown verifiers. The suffix tuple starts with a dot, because endswith("google.com") also accepts notgoogle.com. And match the documented domain, not the example hostname masks: the docs illustrate common crawlers with crawl-***-***-***-***.googlebot.com, but run host 35.247.243.240 and you get a googleusercontent.com hostname for an address sitting in common-crawlers.json. Matching the crawl- prefix silently discards real Googlebot. Same for address families: the common-crawler file is mostly IPv6, so compare as addresses, never as text.

Running it at log scale

DuckDB’s inet extension gives CIDR containment with <<=, which turns verification into a join. Load the parsed log first:

CREATE OR REPLACE TABLE raw AS
SELECT * FROM read_csv('logs/crawl-*.tsv.gz',
  delim = '\t', header = false, nullstr = '-',
  columns = {
    'ts':            'TIMESTAMPTZ', 'client_ip':    'VARCHAR',
    'host':          'VARCHAR',     'method':       'VARCHAR',
    'uri':           'VARCHAR',     'status':       'SMALLINT',
    'bytes':         'BIGINT',      'request_time': 'DOUBLE',
    'upstream_time': 'DOUBLE',      'cache_status': 'VARCHAR',
    'protocol':      'VARCHAR',     'ua':           'VARCHAR'});

Then classify distinct addresses once, rather than joining ranges against every row:

INSTALL inet; LOAD inet;

CREATE OR REPLACE TABLE prefix AS
SELECT crawler_class, CAST(prefix AS INET) AS net
FROM read_csv('google_prefixes.csv', header = true);

CREATE OR REPLACE TABLE ip_class AS
SELECT c.client_ip, min(p.crawler_class) AS crawler_class
FROM (SELECT DISTINCT client_ip FROM raw) c
LEFT JOIN prefix p ON CAST(c.client_ip AS INET) <<= p.net
GROUP BY c.client_ip;

The files are not guaranteed disjoint: run that join once with count(DISTINCT p.crawler_class) instead of min to see whether any address lands in two classes. Then the number this section exists for:

SELECT r.ua ILIKE '%google%'          AS claims_google,
       i.crawler_class IS NOT NULL    AS in_google_range,
       count(*)                       AS fetches,
       round(100.0 * count(*) / sum(count(*)) OVER (), 2) AS pct
FROM raw r JOIN ip_class i USING (client_ip)
GROUP BY 1, 2 ORDER BY fetches DESC;

Four cells. Claims and verifies is your real Google traffic; claims and does not verify is the spoof cell; does not claim and verifies is the interesting one, fetchers with no obvious Google token that you would otherwise have discarded; neither is everyone else.

I am not handing you a spoof percentage, because the only one that matters is yours and it varies by an order of magnitude between sites. What is generic enough to state: on any property with meaningful traffic that cell is not a rounding error, and it is not evenly distributed: it concentrates on the templates a scraper wants, at the hours a scraper runs, from a handful of networks. So skipping verification corrupts the ratios and not merely the totals, which makes an unverified crawl analysis a scraper analysis with a misleading title. Run the query.

The 2026 taxonomy

Verification tells you a request came from Google. It does not tell you which Google, and grouping all of it as “crawl” is the second error:

CREATE OR REPLACE MACRO crawler_group(ua) AS (
  CASE
    WHEN ua ILIKE '%Google-InspectionTool%'                        THEN 'tooling'
    WHEN ua ILIKE '%Googlebot/%'      OR ua ILIKE '%Googlebot-%'
      OR ua ILIKE '%Storebot-Google%'                              THEN 'search'
    WHEN ua ILIKE '%GoogleOther%'                                  THEN 'generic'
    WHEN ua ILIKE '%Google-CloudVertexBot%'                        THEN 'grounding'
    WHEN ua ILIKE '%Google-Agent%'                                 THEN 'agent'
    WHEN regexp_matches(ua, 'Google-Read-Aloud|Google-GeminiNotebook|Google-Pinpoint'
                         || '|GoogleProducer|FeedFetcher-Google|Google-CWS'
                         || '|GoogleMessages|Google-Site-Verification')  THEN 'user_triggered'
    ELSE 'other' END);

Only search is crawl demand. Google-InspectionTool is you, pressing buttons in Search Console, and counting it inflates exactly the URLs you were already worried about. GoogleOther is the generic crawler product teams use for one-off fetches. The user-triggered fetchers ignore robots.txt on purpose, because a human asked, and they are demand from people rather than from a scheduler. Google-GeminiNotebook was renamed from the NotebookLM token in July 2026, so a hardcoded old string now matches nothing.

Google-Extended deserves its own warning: it is a robots.txt control token with no user agent string. It cannot appear in your logs. If you find it there, you found a spoofer.

Google-Agent rolled out in March 2026 with its own range file, and some of its requests are signed with Web Bot Auth, an experimental scheme built on HTTP Message Signatures: signed requests carry Signature-Agent: g="https://agent.bot.goog" alongside Signature and Signature-Input. Not every request is signed yet, so it does not replace range matching, but logging those headers is nearly free. When the pipeline is later pointed at measuring which AI surfaces fetch and cite you, that column separates a signed agent from an imitation of one.

A schema for crawl analysis

Two tables: a request fact enriched with verification and grouping, and a page dimension carrying everything the log cannot know.

CREATE OR REPLACE MACRO url_template(path) AS (
  '/' || array_to_string(
    list_transform(
      string_split(trim(path, '/'), '/'),
      lambda s : CASE
        WHEN regexp_matches(s, '^\d+$')                     THEN '{int}'
        WHEN regexp_matches(s, '^[0-9a-f]{8}-[0-9a-f]{4}-') THEN '{uuid}'
        WHEN regexp_matches(s, '\d{4,}')                    THEN '{id}'
        ELSE s END),
    '/'));

CREATE OR REPLACE TABLE req AS
SELECT r.*, i.crawler_class,
       i.crawler_class IS NOT NULL              AS verified,
       crawler_group(r.ua)                      AS ua_group,
       r.host || split_part(r.uri, '?', 1)      AS url,
       url_template(split_part(r.uri, '?', 1))  AS template
FROM raw r LEFT JOIN ip_class i USING (client_ip);

CREATE OR REPLACE VIEW search_crawl AS
SELECT * FROM req
WHERE verified AND crawler_class = 'common-crawlers' AND ua_group = 'search';

The template macro collapses identifier-shaped path segments and leaves everything else alone. It will be wrong on your site the first time you run it: look at the twenty highest-volume templates, add the patterns it missed, run again. An uninspected template list produces a waste table you cannot defend.

The page dimension holds url, template, cohort, in_sitemap, depth, canonical and indexable. Sitemap membership comes from parsing the live sitemap over HTTP, not the file in your repository. Depth and canonical come from a crawl, and the canonical is the one column no log can supply, which constrains what the waste table can claim. If you have never joined those two sides, the cheapest version of the join takes twenty minutes.

Demand versus capacity

Every intervention addresses one term or the other. Confusing them is how a team spends a quarter rewriting content when it had a slow database, and how another buys servers when its pages simply are not worth scheduling.

The observable that separates them is the relationship between crawl rate and your own latency, measured with a lag, because the throttle response is not instantaneous:

WITH hourly AS (
  SELECT date_trunc('hour', ts) AS hour,
         count(*) FILTER (verified AND crawler_class = 'common-crawlers'
                          AND ua_group = 'search')                  AS crawl_fetches,
         quantile_cont(request_time, 0.95)                          AS p95_s,
         100.0 * count(*) FILTER (status >= 500 OR status = 429)
                        / count(*)                                  AS pct_err
  FROM req GROUP BY 1
)
SELECT t.k AS lag_hours,
       round(corr(a.crawl_fetches, b.p95_s),   3) AS r_latency,
       round(corr(a.crawl_fetches, b.pct_err), 3) AS r_error
FROM range(0, 25) t(k), hourly a, hourly b
WHERE b.hour = a.hour - to_hours(t.k)
GROUP BY t.k ORDER BY t.k;

p95 covers all traffic, not just bot traffic, because the capacity limit responds to how your server behaves generally.

Two shapes come out. If r_latency stays near zero at every lag while crawl rate varies, you are not capacity limited: crawl rate is a statement about demand, server tuning will not change it, and the levers are internal links, sitemap freshness and whether the pages are worth refetching. If r_latency and r_error go clearly negative at some lag, the scheduler is backing off and the content is not the problem.

That is what the correlation shows. What it means is my reading, and it carries a caveat: hour of day confounds both series, since user and crawl traffic often peak together. Rerun it restricted to hours where non-bot volume sits in a narrow band. If it survives, act on it; if not, you found a daily rhythm.

The documented mechanism makes that interpretation legitimate rather than speculative. The capacity limit rises when response times are stable and falls when the site slows or errors; 5xx and 429 prompt crawlers to slow down proportionately to the number of URLs affected, and URLs that persistently return server errors are dropped from the index. That same mechanism is the documented emergency brake, since returning 500, 503 or 429 reduces crawl rate. So check your WAF before your application: a rate-limiting rule returning 429 to verified Googlebot is indistinguishable from a server asking to be crawled less.

Google also caches robots.txt for up to 24 hours; if it starts returning 5xx, crawling pauses for 12 hours, then falls back to the last good copy for 30 days. An intermittently failing robots.txt is a crawl-rate problem with a multi-day signature and no obvious cause.

The waste taxonomy, quantified

Waste is any fetch that consumed capacity without contributing an indexable, canonical page. The classes must be mutually exclusive and evaluated in a fixed order, or a 404 on a parameterised URL is counted twice and the percentages stop summing to 100.

CREATE OR REPLACE MACRO waste_class(status, uri, template, in_sitemap) AS (
  CASE
    WHEN status = 304                          THEN 'conditional_hit'
    WHEN status BETWEEN 300 AND 399            THEN 'redirect'
    WHEN status IN (404, 410) AND in_sitemap   THEN 'dead_but_submitted'
    WHEN status IN (404, 410)                  THEN 'dead'
    WHEN status >= 500 OR status = 429         THEN 'server_error'
    WHEN strpos(uri, '?') > 0                  THEN 'parameter'
    WHEN template ILIKE '%/page/{int}%'        THEN 'pagination'
    WHEN NOT in_sitemap                        THEN 'unsubmitted'
    ELSE 'intended' END);

Then price each class in the currency the capacity limit is denominated in. The limit governs the time your server spends holding connections open, so rank by share of server time, not fetches:

SELECT waste_class(s.status, s.uri, s.template, p.in_sitemap) AS class,
       count(*)                                               AS fetches,
       round(100.0 * count(*) / sum(count(*)) OVER (), 2)     AS pct_fetches,
       round(100.0 * sum(s.request_time)
             / sum(sum(s.request_time)) OVER (), 2)           AS pct_server_time
FROM search_crawl s JOIN page p USING (url)
GROUP BY 1 ORDER BY pct_server_time DESC;

The two percentage columns disagree, and the disagreement is the finding. Redirects are cheap per fetch and overstated by fetch count; a slow parameterised search endpoint can be a small share of fetches and a large share of the time budget. Prioritise by the second column.

Four readings need saying. conditional_hit is not waste but its opposite: crawlers support ETag with If-None-Match and Last-Modified with If-Modified-Since, so a 304 costs almost nothing, and a near-zero row means caching headers are the cheapest capacity you can buy. redirect is worth expanding into chain length, since crawlers follow up to 10 hops and each is a separate fetch. dead is mostly harmless, because a 404 is a strong signal not to recrawl, while dead_but_submitted is your bug. unsubmitted is the leak: capacity spent on inventory you never meant to publish.

The class this query cannot produce is non-canonical duplicates, because the canonical is not in the log. Join page.canonical from a crawl and count fetches where it points elsewhere. A large class there is usually manufactured at the render stage rather than in routing, which is a separate investigation with its own harness for diffing response HTML against the rendered DOM.

Two accounting notes. JavaScript, CSS and image fetches are not waste, but they compete for the same capacity and carry their own byte limits, so bucket them separately. And Googlebot fetches only the first 2MB of a URL, PDFs excepted at 64MB: if a template’s logged response size exceeds 2MB, part of that page was never transferred at all.

Crawl-to-index latency per cohort

Crawl rate is an input. The output metric the crawl-budget discourse skips is how long a URL waits between its first successful fetch and the moment Google reports it indexed, per cohort. The first half is free from the log. The second comes from polling the URL Inspection API and storing a time series rather than a snapshot: one row per URL per poll, carrying coverageState, verdict, googleCanonical and lastCrawlTime. The per-property quota is 2000 queries per day and 600 per minute, so poll a stratified panel rather than the whole site: a fixed sample per cohort, refreshed daily, enough for a distribution and cheap enough to run forever.

WITH first_fetch AS (
  SELECT url, min(ts) AS first_fetch_ts
  FROM search_crawl WHERE status < 400 GROUP BY url
)
SELECT p.cohort,
       count(*)                                                 AS urls,
       round(100.0 * count(x.indexed_ts) / count(*), 1)         AS pct_indexed,
       median(date_diff('day', f.first_fetch_ts, x.indexed_ts)) AS median_days,
       quantile_cont(date_diff('day', f.first_fetch_ts,
                               x.indexed_ts), 0.9)              AS p90_days
FROM page p
JOIN first_fetch f USING (url)
LEFT JOIN first_indexed x USING (url)
GROUP BY p.cohort ORDER BY median_days DESC;

first_indexed is the earliest poll per URL whose coverageState reported it indexed.

Read pct_indexed and median_days together or you will fool yourself. The median covers only URLs that were eventually indexed, so a cohort where most never make it can post an excellent median. This is censored data and the honest summary is the pair: high pct_indexed with a long median is a scheduling problem, a short median with low pct_indexed is a selection problem, and those two failure modes wear the same Search Console label.

The panel also buys a free check on verification. Compare the API’s lastCrawlTime against the last verified fetch in your log. If Google says it crawled a URL on a day your log has no verified row for it, either your verification is discarding real Googlebot, your CDN absorbed the fetch, or retention dropped it. All three are worth knowing, and none is visible from either source alone.

Where Crawl Stats disagrees with your logs

They will disagree, and knowing why decides which source answers which question. Google’s own disclaimer settles the largest one: the Crawl Stats report carries a stated known issue, “The Crawl Stats report currently reports most crawl requests, but some requests might not be counted for various reasons.” That is Google calling its counts approximate, which makes your verified log the authority on volume.

The rest is structural rather than a defect. Each hop in a redirect chain counts as a separate request, so a redirect-heavy site’s total exceeds its count of distinct destinations by design. URLs are counted as requested and never reassigned to canonicals, unlike the Page Indexing report, so the two count different objects. Resources hosted outside the property are not counted, a cached resource shared across many pages is requested only the first time, and the report exists only for root-level properties, so a subdirectory property has none at all.

The last two rows carry the most weight. Discovery versus refresh is Google’s own statement about whether a URL had ever been crawled before; your log can only approximate it as “first fetch inside my retention window,” which is biased by how long you keep logs. And when Crawl Stats’ average response time sits far above your p95, the gap lives between your edge and Google: a network, TLS or edge finding that application profiling will never surface.

Before and after, with a control cohort

Crawl rate has weekly seasonality, responds to site-wide releases, and drifts with demand you do not control. A single before-and-after chart on a treated cohort cannot separate your fix from any of that, which is why almost every published crawl-budget win is unfalsifiable.

The design that survives: define the treatment cohort as the templates you will touch, and a control cohort you deliberately will not, matched on pre-period fetches per URL per week, depth and sitemap membership. Freeze both lists first. Four weeks pre, four weeks post, no other deployment to either cohort inside the window.

SELECT p.arm,
       CASE WHEN s.ts < TIMESTAMPTZ '2026-06-01' THEN 'pre' ELSE 'post' END AS period,
       count(DISTINCT s.url)                                      AS urls,
       round(count(*) / count(DISTINCT s.url)::DOUBLE
             / (date_diff('day', min(s.ts), max(s.ts)) / 7.0), 3) AS fetches_per_url_week
FROM search_crawl s JOIN page p USING (url)
WHERE p.arm IN ('treatment', 'control')
GROUP BY 1, 2 ORDER BY 1, 2;

Read it as a difference of differences: the treatment change minus the control change. If the control moved as much as the treatment, you measured the season. Track secondary metrics too, since the point of removing waste is not more fetches but better ones: the treated cohort’s share of intended fetches, and its crawl-to-index median. And state what the design cannot rule out. If the intervention also changed the pages, you moved demand and capacity at once; if it touched a shared resource, the control was treated too.

Expect the null. On a site whose latency did not correlate with crawl rate at any lag, capacity was never the binding constraint, so freeing it changes nothing measurable: the rate holds flat in both arms, and the conclusion is that the diagnosis two sections ago was skipped. Publish that anyway, because the alternative is another quarter optimising a constraint nobody had.

Your access log is the only place Google’s opinion of your site is written down in storage you control. Search Console is a report about it, filtered, aggregated and admittedly incomplete. Both are useful and only one of them is yours.