Discovered – Currently Not Indexed: Scheduling or Selection

Search Console defines the state in one sentence. From the Page indexing report documentation: “The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl.”

Read it literally. It reports a queue position and a capacity estimate. There is no claim about the page in it, because nothing has looked at the page.

Now read the neighbouring state in the same table: “The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.” That one reports the outcome of a decision made after a fetch.

Two labels, two upstream systems, one report. The field that separates them is whether a fetch ever happened, and that field is available per URL for every property you own. Nearly all the published advice for the first state is advice for the second one, applied without checking which one you have.

What the label asserts, and what it does not

Four conclusions get drawn from this row that the row does not support.

It is not a quality verdict. Quality evaluation happens downstream of a fetch. A URL in “Discovered” has, by the definition above, not been fetched. Rewriting the page changes nothing that the scheduler can see, because the scheduler is not reading pages; it is reading a queue.

It is not a crawl block. Blocked URLs get their own states: URL blocked by robots.txt, URL marked 'noindex', Blocked due to access forbidden (403). If your URLs land in “Discovered”, robots.txt allowed them. Checking robots.txt again is free, and it is also the step where people spend a week.

It is not terminal. “Currently” is doing real work in that string. Queues drain, so the question worth asking is not whether one URL is stuck but whether the drain rate for its cohort is above zero.

And it is not a per-URL fact you can act on. The report’s example list “is limited to 1,000 rows” and is explicitly not exhaustive. On a site with 200,000 URLs in this state, you are looking at a sample you did not design, cannot stratify, and cannot re-draw. Any conclusion you form by clicking three of those rows is unfalsifiable, which is why the genre of advice built on that workflow never gets tested.

The unit that survives all four objections is the cohort: a stable, pre-declared group of URLs that share a template, an age band, a depth band and a discovery source. A single URL’s fate is noise. A cohort’s indexation rate over four consecutive weeks is a measurement.

Two systems, one label

Scheduling decides whether and when a known URL gets fetched. Google’s crawl budget documentation models this as two factors. The crawl capacity limit is “the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration”. Crawl demand is driven by perceived inventory (the URLs Google knows about), popularity (“URLs that are more popular on the Internet tend to be crawled more often”) and staleness. That guide also names a large share of URLs in “Discovered – currently not indexed” as one of the conditions that makes the document relevant to you at all.

Selection decides whether a fetched, parsed and rendered document enters the index. Google’s how Search works overview is blunt about the boundary: “Googlebot doesn’t crawl all the pages it discovered”, and separately, “Indexing isn’t guaranteed; not every page that Google processes will be indexed.”

The two fail for different reasons and respond to different interventions. Confusing them is expensive in both directions: rewriting content that was never fetched, or buying server capacity for pages that were fetched and judged not worth keeping.

The discriminator is documented. In the URL Inspection API, lastCrawlTime is described as “Last time this URL was crawled by Google using the primary crawler. Absent if never crawled successfully.” Absent means scheduling. Present means selection. That is the whole diagnostic, and it costs one API call per URL.

One caveat before you trust it: “successfully”. A URL that Googlebot requested and got a 5xx for may show no lastCrawlTime while your access log shows the request. Before you spend quota, run the sitemap-against-logs join. It separates never-requested from requested-and-failed in twenty minutes with no API involved. The field also reports the primary crawler only.

Cohort sitemaps as a measurement instrument

Sitemaps are usually treated as a submission mechanism. Treat them instead as the partition of your URL space that Search Console will report against, because the Page indexing report lets you filter by “Specific sitemap URL – Show only URLs listed in a specific sitemap or sitemap index”. One file, one cohort, one readable indexation rate.

Partition on the axes that plausibly drive either system:

• Template, meaning the page type as your renderer sees it, not as your URL structure suggests.
• Age band: first-published quarter or half-year. New URLs and three-year-old URLs sit in different parts of the queue and should never share a denominator.
• Depth band: minimum click distance from the home page on your own crawl graph.
• Source: hand-authored, user-generated, or programmatically generated from a dataset.

Four axes multiply fast, so collapse until every cohort is large enough to measure and homogeneous enough to interpret. A single file caps at 50,000 URLs or 50MB uncompressed and an index at 50,000 sitemaps, so the format is not your constraint.

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sm/product-2025h2-d2.xml</loc><lastmod>2026-03-19</lastmod></sitemap>
  <sitemap><loc>https://example.com/sm/product-2026h1-d2.xml</loc><lastmod>2026-03-19</lastmod></sitemap>
  <sitemap><loc>https://example.com/sm/category-d1.xml</loc><lastmod>2026-03-19</lastmod></sitemap>
  <sitemap><loc>https://example.com/sm/editorial-legacy-d3.xml</loc><lastmod>2026-03-19</lastmod></sitemap>
  <sitemap><loc>https://example.com/sm/programmatic-geo-d4.xml</loc><lastmod>2026-03-19</lastmod></sitemap>
</sitemapindex>

Three rules keep the instrument honest.

A URL belongs to exactly one cohort. Overlapping membership makes the per-sitemap rates uninterpretable, and Search Console will happily attribute one URL to several files.

Membership is frozen for the duration of a measurement. Re-partition in week three and the week-four number is not comparable to the week-one number, so you will read your own bookkeeping as an effect. Version the partition and keep the old files served until the measurement closes.

Do not touch lastmod to provoke a crawl. Google uses the value “if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate”, per the sitemap build documentation, and ignores <priority> and <changefreq> entirely. A sitemap that stamps today’s date on every URL every night is not an instrument, it is noise you are generating yourself. The crawling myths page makes the same point about artificial freshness in general.

One limitation to plan around: the Search Console API’s Sitemaps resource returns contents[].submitted but its indexed counterpart is marked deprecated; do not use. Per-cohort indexed counts are a UI read. If you want them in a database on a schedule, you need the URL Inspection API.

Ground truth at scale: the URL Inspection API

The endpoint is a single POST:

https://searchconsole.googleapis.com/v1/urlInspection/index:inspect

with a body of {"inspectionUrl": ..., "siteUrl": ...} and the webmasters.readonly scope. The documented quotas at the time of writing are 2,000 QPD and 600 QPM per property, against 10,000,000 QPD and 15,000 QPM per Cloud project. The per-property daily figure is the binding one for everybody. Check that page before you size a run; it has changed before.

From inspectionResult.indexStatusResult, the fields that carry diagnostic weight:

One engineering warning that will bite you in month two: verdict, pageFetchState, robotsTxtState and indexingState are documented enums. coverageState is not. It is the report’s display string, and the request accepts a languageCode parameter that translates issue messages. Branch on verdict and lastCrawlTime; store coverageState for humans to read, never match on it.

A panel, not a snapshot

2,000 inspections per property per day is a hard ceiling, so a full-site sweep is off the table above a few thousand URLs. The answer is a stratified panel: a fixed random sample per cohort, drawn once with a recorded seed, stored in its own table, and re-inspected on a schedule. The same URLs every wave. That turns each wave into a within-URL transition rather than a comparison between two different random draws.

Sample size is arithmetic, not taste. For a proportion near 50%, the 95% confidence interval is roughly ±8 points at n=150, ±5 points at n=400, and ±3 points at n=1,000. If your intervention is expected to move cohort indexation by 5 points, a 150-URL panel cannot see it, and you will publish a null result that is an artifact of your sample size. Eight cohorts at 250 URLs each is 2,000 calls: one full day of quota, one wave per day, comfortably inside 600 QPM at any sensible pace.

CREATE TABLE IF NOT EXISTS inspection (
  checked_on       TEXT NOT NULL,
  url              TEXT NOT NULL,
  cohort           TEXT NOT NULL,
  verdict          TEXT,
  coverage_state   TEXT,
  indexing_state   TEXT,
  robots_state     TEXT,
  fetch_state      TEXT,
  last_crawl       TEXT,
  google_canonical TEXT,
  PRIMARY KEY (checked_on, url)
);

The primary key is the point. checked_on is part of the identity of a row, so a re-inspection appends rather than overwrites. Most implementations of this idea store one row per URL and update it in place, which throws away the only thing you actually need: the date a URL crossed from never-fetched to fetched, and from fetched to indexed.

import sqlite3, time, random, datetime
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError

SITE  = "sc-domain:example.com"
SCOPE = ["https://www.googleapis.com/auth/webmasters.readonly"]
api = build("searchconsole", "v1", cache_discovery=False,
            credentials=Credentials.from_authorized_user_file("token.json", SCOPE))

def inspect(url, tries=5):
    for n in range(tries):
        try:
            r = api.urlInspection().index().inspect(
                body={"inspectionUrl": url, "siteUrl": SITE}).execute()
            return r["inspectionResult"]["indexStatusResult"]
        except HttpError as e:
            if e.resp.status not in (429, 500, 503) or n == tries - 1:
                raise
            time.sleep(2 ** n + random.random())

Backoff with jitter on 429 matters more than it looks. A loop that retries immediately burns daily quota on errors and leaves a half-populated wave, which is worse than no wave at all because the missing rows are not missing at random.

PACE, DAILY_CAP = 0.15, 2000
FIELDS = ("verdict", "coverageState", "indexingState", "robotsTxtState",
          "pageFetchState", "lastCrawlTime", "googleCanonical")

def wave(db, panel, day=None):
    day = day or datetime.date.today().isoformat()
    for cohort, url in panel[:DAILY_CAP]:
        r = inspect(url)
        db.execute("INSERT OR REPLACE INTO inspection VALUES (?,?,?,?,?,?,?,?,?,?)",
                   (day, url, cohort) + tuple(r.get(f) for f in FIELDS))
        db.commit()
        time.sleep(PACE)

PACE = 0.15 puts you at roughly 400 QPM against the 600 QPM ceiling. Commit per row rather than per batch: an interrupted wave then leaves a partial but valid day rather than nothing.

The first query to run against the resulting table is the one that ends the argument:

SELECT cohort,
       SUM(last_crawl IS NULL)     AS never_fetched,
       SUM(last_crawl IS NOT NULL) AS fetched_and_declined,
       COUNT(*)                    AS not_indexed
FROM inspection
WHERE checked_on = (SELECT MAX(checked_on) FROM inspection)
  AND verdict <> 'PASS'
GROUP BY cohort
ORDER BY never_fetched DESC;

Every cohort falls somewhere on the line between those two columns. Dominated by never_fetched, it is a scheduling cohort and no amount of editing will touch it. Dominated by fetched_and_declined, it is a selection cohort and no amount of internal linking will touch it. A cohort split evenly usually means the partition is wrong: two page types your renderer treats differently have been merged.

Five hypotheses and the test that falsifies each

Each hypothesis below is stated so that a specific observation can kill it. If you cannot say in advance what would make you abandon it, you are not testing, you are decorating a decision you already made.

H1. Near-duplicate templates (selection)

Mechanism. The cohort’s members are close enough textually that selection consolidates them, keeping one representative.

Prediction. lastCrawlTime present on most members; googleCanonical differs from the inspected URL on a substantial share; coverageState skews toward duplicate-family strings.

Falsifier. If googleCanonical equals the inspected URL wherever it is present, and pairwise near-duplicate similarity across the cohort’s main content is low, H1 is dead. Compute the similarity yourself over the extracted main content, not the full HTML. Shared chrome will report every page on a site as a near-duplicate of every other. Where the cluster is real, the consolidation decision is the same one you make when two of your URLs answer one query.

H2. Thin generated URLs (selection)

Mechanism. Pages are technically fine and materially empty: a template wrapped around a row that carries no information a user could not get from the parent page.

Prediction. Fetched, verdict of NEUTRAL, self-canonical, and a template whose indexed members earn effectively no impressions.

Falsifier. The within-template contrast. Take the same template’s URLs that are indexed and compare them against the non-indexed members on the dimensions you can measure: unique token count in main content, number of distinct data fields populated, internal inlinks, external references. If the two groups are indistinguishable on every axis, thinness is not what is separating them and you are looking at a scheduling artifact that happens to have been fetched. This is the test people skip, because running it often shows that the indexed and non-indexed halves of a template differ by nothing except the order they were queued in.

H3. Depth and orphaning (scheduling, demand side)

Mechanism. Perceived inventory is not the same as perceived importance. URLs reachable only from a sitemap, or at high click depth behind pagination, generate little crawl demand.

Prediction. never_fetched dominant; referringUrls sparse or absent; the cohort concentrated at high depth on your own crawl graph.

Falsifier. Join the panel to depth and internal inlink count and compare distributions between the fetched and never-fetched members of the same cohort. If both groups sit at the same depth with the same inlink count, depth is not the discriminator. The instrument for this is internal PageRank on your own crawl graph, because raw depth counts every link as equal and a link from a paginated archive is not equal.

H4. Host capacity throttling (scheduling, capacity side)

Mechanism. Exactly the case the label’s own definition describes: Google wanted to crawl and expected to overload the site. The myths page confirms the direction: “a significant number of 5xx HTTP response status codes or connection timeouts signal the opposite, and crawling slows down”.

Prediction. A site-wide, time-correlated signature. The Crawl Stats report shows degraded host status or a rising average response time; crawl rate moves inversely to p95 latency in the same window.

Falsifier. Capacity is a property of the host, not of a template. If one cohort is starving while other cohorts on the same host are being crawled at a normal rate in the same window, it is not capacity. This single check invalidates most “we upgraded the server to fix indexing” stories. The evidence lives in your access log, where crawl rate can be plotted against latency and error rate directly. That is the demand-versus-capacity read, and Crawl Stats is a coarser view of the same thing.

H5. Insufficient crawl demand (scheduling, residual)

Mechanism. Nothing is broken. The URLs are known, allowed, shallow enough, on a healthy host, and the scheduler has simply ranked them below everything else it could spend a fetch on.

Prediction. never_fetched dominant, host status clean, depth distribution normal, robots ALLOWED, and no external references anywhere in the cohort.

Falsifier. H5 is the residual, so it cannot be confirmed by observation, only by intervention. Split the cohort at random, give the treatment half a genuine entry point (a real link from a page that is itself crawled frequently, or an external reference), hold the control half untouched, and watch the fetch rate curves for four to six weeks. If the treated half’s fetch rate does not separate from the control’s, your entry point was not worth a fetch either, and you have learned something more useful than another rewrite.

The action that follows each diagnosis

The third row is the one no listicle contains, and on a large site it is frequently correct. A cohort published in a single release is a scheduling backlog by construction: the queue is behaving as documented, and every intervention launched during that window will later be credited with a drain that was going to happen anyway. Waiting is a decision, and it is the only one that produces a clean control period.

Two anti-actions worth naming. Manually requesting indexing from the URL Inspection tool moves individual URLs and teaches you nothing about the cohort; it is a demo, not a fix. And the Indexing API “can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject”. If your pages are neither, pointing it at them is not an aggressive tactic, it is a no-op with extra logging.

The SOFT_404 row deserves its own note: a soft 404 is emitted after a fetch, by the system that evaluates the rendered result, so the URL was never a scheduling case. Those belong with the rendering-stage failures, not here.

Measuring the fix without fooling yourself

Cohort indexation rate over time, with a control, is the whole design.

SELECT cohort, checked_on,
       COUNT(*)                                                 AS panel,
       ROUND(100.0 * SUM(verdict = 'PASS')       / COUNT(*), 1) AS index_rate,
       ROUND(100.0 * SUM(last_crawl IS NOT NULL) / COUNT(*), 1) AS fetch_rate
FROM inspection
GROUP BY cohort, checked_on
ORDER BY cohort, checked_on;

Two curves per cohort, and their relationship is the diagnosis. Fetch rate flat and low: the scheduler has not arrived, and index rate cannot move. Fetch rate climbing while index rate stays flat: the scheduler arrived and selection declined, and now, only now, content work is the right response. Both climbing together: the queue drained and you are done.

The control is not optional, because everything else in the window moves: core updates, seasonality, a CDN change, a template release. All of them hit treatment and control equally, which is exactly why the control earns its keep. Draw it by randomizing within a cohort, never by picking a different section. A different section is a different template, age band and depth distribution, so it will not track your treatment group even with no intervention at all. Read the result as a difference in differences.

Three practical constraints on the reading.

Baseline first: two to three waves before you change anything. Without a pre-period you cannot distinguish a fix from a trend that was already running.

Respect the lag. Between recrawl scheduling and the report’s own processing delay, a four-week floor before reading a result is my working rule rather than a documented figure. The common failure is checking at day five, seeing nothing, adding a second intervention, and losing the ability to attribute either.

One change per cohort per window. If you improve internal linking and rewrite the template in the same fortnight, both hypotheses survive and neither is testable. Run them serially on different cohorts if you are impatient, which is what having many cohorts is for.

When the answer is null, write it down. Treatment and control curves that stayed inside each other’s confidence intervals for six weeks have told you the hypothesis was wrong, which is a result. The alternative is a backlog of unevaluated interventions and a site-wide indexation number that drifts for reasons nobody can name.

When the scheduler is right and the pages are wrong

Sometimes the measurement comes back and says the queue is correct.

The signature is specific: fetched, self-canonical, INDEXING_ALLOWED, robotsTxtState of ALLOWED, SUCCESSFUL fetch, and a stable NEUTRAL verdict across four or more waves. Nothing is broken. Selection looked at the page and declined, repeatedly, and the within-template contrast from H2 found nothing that distinguishes these URLs from their indexed siblings except that the siblings got there first.

That is a real answer, and the action is removal.

The mechanics follow from documented behaviour. 410 or 404 for pages with no successor. The crawling myths page notes that URLs serving 4xx codes, 429 excepted, do not waste crawl budget, so removal is not a crawl-budget optimization and should not be sold internally as one. 301 where a genuinely better target exists, which is consolidation rather than deletion. noindex only where the page must remain reachable for users, and remember that Google “has to crawl the page to find the noindex rule”, so a noindex on a never-fetched URL does nothing at all: it is an instruction to a system that has not arrived.

Whatever you choose, remove the URLs from the cohort sitemap in the same deploy. Leaving deleted inventory in a submitted sitemap keeps your denominator wrong and your indexation rate permanently understated, and then you will spend next quarter investigating a number you broke yourself.

The framing that makes this decision easy: index count is an inventory measure, not a health measure. Removing URLs that were fetched, evaluated and declined costs you nothing you had, because they were never in the index. What it buys is a partition in which every remaining cohort is one you would defend.

The uncomfortable version of the same sentence is the one worth sitting with. If a template’s non-indexed members are statistically indistinguishable from its indexed members, the honest reading is not that selection is wrong about the non-indexed half. It is that selection is right about the whole template, and the indexed half is on borrowed time.