Rendering-Stage Failures: Soft 404s and Canonical Conflicts

A fetched URL exists in three forms at once: the response body, the DOM a browser builds after executing it, and the verdict an indexing pipeline wrote down about a fetch from days ago. Most arguments about indexing on client-rendered sites are arguments about which of the three somebody was looking at. They are allowed to disagree, they routinely do, and nothing in your stack tells you when they start.

Google’s pipeline runs crawling, rendering and indexing in that order, with a queue in between that a page “may stay on for a few seconds, but it can take longer than that”. What your server sent is on the near side; what your bundle changes is on the far side. Below: a harness that measures the gap, and a CI job that fails a build when it opens where it should not.

Three views of one page

The response HTML is what the origin returned for a plain GET. It carries the status code and the response headers, the only place X-Robots-Tag can live.

The rendered DOM is what Chromium produced after executing the bundle. It has no status of its own: the status was decided in view one, and no client code revises it.

What Search Console reports is not a live measurement. The Page Indexing report describes the most recently indexed version, and its labels are verdicts. “Duplicate, Google chose different canonical than user” means, verbatim, “this page is marked as canonical for a set of pages, but Google thinks another URL makes a better canonical”.

A fourth view gets mistaken for the second, which is why these bugs survive code review: the Elements panel in your own browser, where the DOM was built with your cookies, your warm cache and a scroll. The renderer has none of those. The Web Rendering Service “does not retain state across page loads”, with Local Storage, Session Storage and cookies all cleared, and the lazy-loading guidance steers you off anything driven by scrolling or clicking, “which is important as Google Search does not interact with your page”.

The harness

Two passes over one URL inside one throwaway browser context, one extractor run against both.

The raw pass is the part worth stealing. Instead of parsing the response body with a regex or a Node HTML library, navigate to it in Chrome with script execution disabled, then re-enable script execution and evaluate: skipped scripts are not retroactively run, so what comes back is the response HTML through Chrome’s own parser. That matters because a malformed element inside <head> closes the head early and every <link> after it lands in the body, where canonical and hreflang annotations are not read. A tolerant Node parser reports those links as present. Chrome is the parser in the pipeline, and it does not.

// render-diff.mjs
import puppeteer, {PredefinedNetworkConditions} from 'puppeteer';
import {extract} from './extract.mjs';

const BLOCKED = new Set(['image', 'media', 'font']);
const FIELDS = ['status', 'xRobotsTag', 'canonical', 'robots', 'title',
                'h1', 'hreflang', 'links', 'mainChars', 'errorMarkers'];

async function harden(page, origin) {
  await page.setCacheEnabled(false);
  const cdp = await page.createCDPSession();
  await cdp.send('Network.clearBrowserCache');
  await cdp.send('Network.clearBrowserCookies');
  await cdp.send('Storage.clearDataForOrigin', {origin, storageTypes: 'all'});
}

storageTypes: 'all' covers Local Storage, IndexedDB, Cache Storage and service workers in one call. That reproduces the documented no-state condition, and it is what most home-grown render checks skip, which is why they pass on an app that only renders for a returning visitor.

async function pass(context, url, {javaScript}) {
  const origin = new URL(url).origin, failed = [];
  const page = await context.newPage();
  await page.setViewport({width: 412, height: 915, isMobile: true});
  await harden(page, origin);
  await page.setRequestInterception(true);
  page.on('request', req => {
    if (req.isInterceptResolutionHandled()) return;
    return BLOCKED.has(req.resourceType()) ? req.abort() : req.continue();
  });
  page.on('requestfailed', req => {
    if (new URL(req.url()).origin === origin) failed.push(`${req.resourceType()} ${req.url()}`);
  });
  if (javaScript) {
    await page.emulateCPUThrottling(4);
    await page.emulateNetworkConditions(PredefinedNetworkConditions['Slow 4G']);
  } else {
    await page.setJavaScriptEnabled(false);
  }
  const res = await page.goto(url, {waitUntil: javaScript ? 'networkidle0' : 'load'});
  if (javaScript) await new Promise(r => setTimeout(r, 2000));
  else await page.setJavaScriptEnabled(true);
  const dom = await page.evaluate(extract);
  await page.close();
  return {status: res.status(), xRobotsTag: res.headers()['x-robots-tag'] ?? null,
          failedSameOrigin: failed, ...dom};
}

The isInterceptResolutionHandled() guard is not decoration: another listener may already have resolved the request, and Puppeteer requires the check and the resolution to run synchronously together.

Blocking image, media and font approximates documented renderer behaviour: Googlebot and its rendering service “continuously analyze and identify resources that don’t contribute to essential page content and may not fetch such resources”. The requestfailed collector is the other half, because a page referencing many resources that cannot be loaded can be interpreted as a soft 404 even when it is otherwise fine.

emulateCPUThrottling and emulateNetworkConditions are page methods, so neither needs a raw CDP session; PredefinedNetworkConditions exposes Slow 3G, Fast 3G, Slow 4G and Fast 4G, where Slow 4G aliases Fast 3G. A 4x throttle is not a claim about Googlebot’s hardware; it makes the race between your data fetch and your first render fail in CI rather than once a quarter in production. One flag to check before copying an old config across: since Chrome 132 the old headless implementation ships only as chrome-headless-shell, so headless: true is the modern mode and headless: 'shell' tests the wrong browser.

// extract.mjs
export function extract() {
  const ERROR_MARKERS = [
    /\bnot found\b/i, /\bno results?\b/i, /\b0 results\b/i, /\bno longer available\b/i,
    /\bout of stock\b/i, /\bsomething went wrong\b/i, /\bsign in to (?:view|read)\b/i,
  ];
  const abs = u => { try { return new URL(u, document.baseURI).href; } catch { return u; } };
  const q = s => [...document.querySelectorAll(s)];
  const main = document.querySelector('main') || document.body;
  const text = (main.innerText || '').replace(/\s+/g, ' ').trim();
  return {
    title: (document.title || '').trim(),
    canonical: q('link[rel~="canonical" i]').map(l => abs(l.getAttribute('href'))),
    robots: q('meta[name="robots" i], meta[name="googlebot" i]')
      .map(m => `${m.name.toLowerCase()}=${(m.content || '').toLowerCase().trim()}`),
    h1: q('h1').map(h => (h.innerText || '').trim()).filter(Boolean),
    hreflang: q('link[rel~="alternate" i][hreflang]')
      .map(l => `${l.getAttribute('hreflang').toLowerCase()} ${abs(l.getAttribute('href'))}`).sort(),
    links: q('a[href]').length,
    mainChars: text.length,
    errorMarkers: ERROR_MARKERS.filter(re => re.test(text)).map(re => re.source),
  };
}

Nine fields, each changing an indexing outcome and each assertable:

• canonical as an array, because the failure mode is two of them.
• robots under both the robots and googlebot names, since a route stacking a directive on a template default is a different page from the one you shipped.
• title and h1, the cheapest evidence that the shell rendered at all.
• errorMarkers and mainChars: the status says 200, the words say otherwise.
• links, because a rendered page with zero <a href> is a dead end in the crawl graph.
• hreflang, injected on the far side of the render queue and valid only when every target renders and points back.

The wrapper: a fresh context per URL (in Chrome every non-default context is incognito), both passes inside it, then a field-by-field diff.

export async function diffRoute(browser, url) {
  const context = await browser.createBrowserContext();
  try {
    const raw = await pass(context, url, {javaScript: false});
    const rendered = await pass(context, url, {javaScript: true});
    return {url, raw, rendered,
            drift: FIELDS.filter(f => JSON.stringify(raw[f]) !== JSON.stringify(rendered[f]))};
  } finally {
    await context.close();
  }
}

Soft-404 patterns in client-side routers

A soft 404 is a URL that returns a page telling the user the page does not exist and also a 200 status code, or “a page with no main content or empty page”. A client-side router produces exactly that, because the router runs after the status code has been sent.

A 200 for a missing record. The route matched, the shell shipped with 200, the fetch came back 404, the component rendered “we could not find that”. Every deleted SKU and expired job posting becomes an indexable URL with an error message on it.

Empty states: a profile with no posts, a category with no children, a workspace for a new account. Nothing is broken, but the rendered DOM holds a dozen words and an illustration, and a classifier cannot tell “broken” from “empty”.

Zero-result listings. Filters and internal search, worse because they are combinatorial: a facet system generates a large URL space, most of it returns nothing, all of it answers 200. Google’s list of soft-404 causes names “an empty internal search result page” explicitly.

And gated content. Shipping the full article and hiding it with JavaScript after a subscription check reads as one page to a crawler and another to a user; the documentation calls that an unreliable way to limit access. Invert it and the rendered DOM is a login prompt, which is the empty-state case again.

The harness catches all four the same way: rendered.status === 200, errorMarkers non-empty or mainChars under a template floor, and no noindex anywhere. That detects your own declared error states, not Google’s classifier, which is undocumented and which I will not pretend to model.

A fifth shape is not a router bug at all: the page is correct and the render failed. Blocked resources, too many resources, a script that throws. failedSameOrigin and the mainChars floor cover it. In the Page Indexing report these sit beside the scheduling states, and separating the two is the whole job of telling a scheduling problem from a selection problem.

Canonical injected by JavaScript

The documented position is not ambiguous. You may inject rel=canonical with JavaScript, and “Google Search will pick up the injected canonical URL when rendering the page”. But you must not use JavaScript to change it to something other than the URL already in the original HTML, and if you cannot set it in HTML, leave it out of the HTML entirely rather than shipping a placeholder for JavaScript to overwrite.

App shells violate that last clause by construction. The shell has one <head>, the head has a canonical, and that canonical is a hardcoded origin, a self-reference from the CDN, or a leftover from whatever the shell was cloned from. Then the route mounts and rewrites it: the exact configuration the documentation tells you not to build, in a file nobody edits.

Why the raw-HTML value tends to win is my reading of a documented mechanism, not a documented outcome. Canonicalization is a cluster-level decision, and the signals Google lists are redirects (strong), rel=canonical (strong) and sitemap inclusion (weak), which “can stack and thus become more effective when combined”. The response-HTML canonical is present on every fetch of every URL in the cluster, including fetches whose render was queued and never got resources, and it is identical across the template. The injected one exists only where the render completed. A clustering system fed a constant and a conditional settles on the constant.

To reproduce the conflict deliberately, on staging, in two commits:

1. Put a fixed canonical in the shell (the site root will do) and ship a route that injects its own on mount. The harness now lists canonical in drift with two absolute URLs.
2. Change the injection from updating the existing tag to document.head.appendChild(link). Now rendered.canonical has two entries, and the documentation says conflicting or multiple rel="canonical" tags “may lead to unexpected results”. There is no documented precedence rule to reason from, so the only safe number is one.

The fix is the boring one: render the canonical server-side, or emit none in the shell at all. The argument applies with more force to hreflang, where annotations must be reciprocal: an injected set is valid only if every target also rendered and injected a return link, a conjunction over N renders. Put hreflang in the response HTML, the Link header, or the sitemap.

Lazy content, tab panels, infinite lists

One rule decides all three: content in the DOM at the end of the render is indexable, and content requiring an interaction to enter the DOM does not exist.

Lazy sections and images. IntersectionObserver with a generous root margin, or native loading="lazy", puts content in the DOM as the viewport approaches it. The harness never scrolls, so anything below its 915px viewport waiting on a real intersection is missing from mainChars. That is a true negative, and the reason the guidance is to load on viewport rather than on a scroll event. The documented check is to look for the URL in the src attribute of the rendered HTML, one more q('img[src]') in the extractor.

Tab panels and accordions. The distinction is not hidden versus visible, it is mounted versus not. hidden or display:none on a rendered panel keeps the text in the DOM; a conditional render ({active === 'specs' && <Specs/>}, or v-if) removes it, and the specs tab holding your product attributes is not part of the page. Render every panel, hide the inactive ones with CSS. mainChars measures innerText, which excludes display:none subtrees; swap it for textContent and diff both counts to see the gap.

Infinite lists. Two failures. Content: a virtualized list renders a window of rows and unmounts the rest. Link discovery, which is worse: if the next chunk is fetched on scroll and those URLs exist only inside the chunk, nothing links to them from any rendered page. links is in the extractor for this. The documented fix is unglamorous: give each chunk a persistent unique URL, keep its content stable, link sequentially between them, and update the displayed URL with the History API as the user scrolls. That is pagination with a scroll listener attached, and it is the only version that survives rendering. If orphan analysis on your internal crawl graph keeps surfacing detail pages with no inbound edges, this is usually where they went.

Status codes you cannot set from the client

Nothing in the browser changes the status of a response already sent. Every workaround is server-side, edge-side, or a documented substitute.

First, a correction that saves an argument: current documentation treats all 4xx codes except 429 identically: the URL is removed from the index if it was there, and crawl frequency gradually decreases. 410 is not a faster 404 in any documented sense. Use it when content is permanently gone because that is what it means to every other consumer of your API.

Render the route on the server and let the framework set the status. In Next.js, notFound() serves a real 404 from a Route Handler and renders the nearest boundary in a Server Component, with a caveat worth reading twice: if it is called after streaming has started, the status stays 200 and the framework injects a noindex meta tag instead. A documented degradation from a status code to a robots directive, which means the same call in the same codebase produces two different indexing signals depending on where in the render tree it fires. The harness tells you which one you got; reading the framework source does not.

Or set the status at the edge. A worker in front of a static shell can check the path against a source of truth the client does not have and rewrite the response:

export default {
  async fetch(request, env) {
    const {pathname} = new URL(request.url);
    const response = await env.ASSETS.fetch(request);
    if (!pathname.startsWith('/products/')) return response;
    if (await env.CATALOG.get(pathname.split('/')[2])) return response;
    return new Response(await response.text(), {
      status: 410,
      headers: {...Object.fromEntries(response.headers), 'X-Robots-Tag': 'noindex'},
    });
  },
};

The shape matters more than the vendor: one lookup, and a status the client cannot set, for one key-value read per request on one route prefix.

Prefer X-Robots-Tag to a meta tag where you have the choice. Both are documented as having the same effect, and any rule valid in a robots meta tag is valid as a header. The header applies to non-HTML responses, is set once at the edge rather than in every rendering path, and does not depend on a render completing.

The documented client-side fallbacks, when nothing above is available: redirect to a URL that really answers 404, or inject <meta name="robots" content="noindex"> on the error branch. Both are in the JavaScript SEO guide with sample code. Use them holding two facts. Put noindex in the original HTML meaning to remove it with JavaScript later, and the documentation warns that Google “may skip rendering and JavaScript execution” when it encounters the tag, so the removal may never run. And a noindex on a page blocked by robots.txt does nothing at all: the page must not be blocked and must be accessible to the crawler, or “the crawler will never see the noindex rule, and the page can still appear in search results”. Blocking a directory and adding noindex to it in the same sprint cancels itself out.

Verifying a fix without waiting a month

The harness against production, immediately. The deploy either closed the drift or it did not, and you know inside a minute. It catches the common case where the fix worked in preview and the CDN or the edge middleware undid it.

The live test, within minutes. URL Inspection’s live test returns the rendered HTML, a screenshot, the console output and the resource list. It does not test for all possible indexing issues, with duplicate detection explicitly out of scope, and the Google-selected canonical exists only in the indexed data, never in a live test, because a live test has nothing to cluster against. So it proves your noindex is gone and your content renders. It cannot prove a canonical conflict is resolved. The tools also fetch as Google-InspectionTool, a separate user agent from Googlebot, so an edge rule branching on user agent puts you on a different code path.

Log evidence of the re-fetch, over days. Nothing changes in the index until the URL is fetched again, so the question after a deploy is not “has it been reindexed” but “has it been re-crawled”. Filter verified Googlebot to the changed paths and find the first hit after the deploy timestamp; the pipeline is in the access-log analysis post, and the cheap version, the sitemap-to-log join, answers fetched-or-not in twenty minutes.

Then poll rather than refresh. The URL Inspection API returns googleCanonical, userCanonical, coverageState, pageFetchState and lastCrawlTime, at a documented 2,000 queries per day and 600 per minute per site. Store one row per URL per day; a time series converges in days where staring at one URL in the interface never does. Two constraints: the API exposes only the indexed version, since “you cannot test the indexability of a live URL”, and Google may hold pages in a duplicate cluster for up to two weeks even after the underlying issue is fixed. Two weeks is the floor for a verdict on a canonical change, not the point at which you conclude the fix failed.

The CI assertion

The harness answers a question once. The assertion keeps it answered: the build fails when fields that decide indexability move between raw and rendered on routes where they must not. The contract is a manifest rather than hardcoded expectations, because absolute values change every time somebody edits a title and drift does not.

{
  "baseUrl": "https://preview.example.com",
  "parityFields": ["status", "xRobotsTag", "canonical", "robots", "title", "hreflang"],
  "routes": [
    {"path": "/", "expect": {"status": 200, "indexable": true}},
    {"path": "/products/live-sku", "expect": {"status": 200, "indexable": true}},
    {"path": "/products/deleted-sku", "expect": {"status": 410, "indexable": false}},
    {"path": "/search?q=zzzzqq", "expect": {"status": 200, "indexable": false}},
    {"path": "/articles/gated", "expect": {"status": 200, "indexable": true},
     "allowDrift": ["title"]}
  ]
}

Note the fixtures. Two of the five are negative cases, a record deliberately deleted and a search deliberately returning nothing, seeded and owned like any other test fixture. A parity suite made only of happy paths tests nothing, because happy paths are not where soft 404s live. allowDrift is the escape hatch: a route that legitimately sets its title client-side declares that once, in review, instead of the team learning to ignore a red build.

const noindex = r => (r.robots ?? []).some(v => v.includes('noindex')) ||
                     (r.xRobotsTag ?? '').toLowerCase().includes('noindex');

function violations(route, {raw, rendered}, cfg) {
  const v = [], skip = new Set(route.allowDrift ?? []);
  for (const f of cfg.parityFields) {
    if (skip.has(f)) continue;
    const a = JSON.stringify(raw[f]), b = JSON.stringify(rendered[f]);
    if (a !== b) v.push(`${f} drifts: raw ${a} -> rendered ${b}`);
  }
  if (route.expect?.status && raw.status !== route.expect.status)
    v.push(`status ${raw.status}, expected ${route.expect.status}`);
  if (route.expect?.indexable === false && rendered.status === 200 && !noindex(rendered))
    v.push('must not be indexable but answers 200 with no noindex');
  if (route.expect?.indexable === true && noindex(rendered))
    v.push('must be indexable but carries a noindex directive');
  if (rendered.status === 200 && rendered.errorMarkers.length && !noindex(rendered))
    v.push(`soft 404: 200 + error text ${rendered.errorMarkers.join(', ')}`);
  if (rendered.failedSameOrigin.length)
    v.push(`same-origin resources failed: ${rendered.failedSameOrigin.join(', ')}`);
  return v;
}

Four of the six rules are parity or expectation checks a reviewer can argue with. The fifth is the soft-404 detector: a 200 that renders words from your own error vocabulary and carries no noindex is a bug by definition, whatever a classifier eventually decides. The sixth catches the render that half-failed. The runner is a loop over cfg.routes calling diffRoute, collecting violations, and exiting non-zero when the count is above zero.

name: rendering parity
on: [pull_request]
jobs:
  parity:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: {node-version: 24, cache: npm}
      - run: npm ci
      - run: npx puppeteer browsers install chrome
      - run: node assert-parity.mjs routes.json
        env:
          BASE_URL: ${{ vars.PREVIEW_URL }}
      - if: failure()
        uses: actions/upload-artifact@v7
        with: {name: parity-report, path: parity-*.json}

The BASE_URL indirection is the point. Run this against localhost:3000 and you test your bundle. Run it against the preview URL and you test the bundle plus the CDN, the edge middleware, the security headers and whatever else the platform injects: the system that actually answers Googlebot. Canonical conflicts are usually created by exactly one of those layers, and none of them exist on a dev server. Cache ~/.cache/puppeteer keyed on the Puppeteer version, and keep the route list to five or fifteen entries, one per template plus the negative cases: this is a smoke test for a class of bug, not a crawl. Whole-URL-space coverage is a scheduled job against production, next to your other template-level diagnostics.

What this buys is narrow. The suite cannot tell you whether Google will index a page, because indexing is a selection decision made against a corpus you do not have. It tells you that the four artifacts deciding indexability (status, canonical, robots directives, and the presence of content) are the same in the response as they are after render, on the routes where you said they must be. That is the part that is your fault when it breaks, and until somebody writes this job it is the part with no owner and no test.