Internal PageRank on Your Own Crawl Graph: Finding Underpowered Pages

Pick any page on your site and count the internal links pointing at it. On a template-driven site the number is large and almost none of it was a decision: a nav, a footer, a related-posts block and a sidebar render on every URL in the build, so each page collects an inbound link from nearly every other page for free. The contextual links an editor added last week sit in that pile, contributing what a handful of items out of several thousand contribute. That is why “link related content” is correct and useless at the same time: it is applied to a graph whose shape was set by a navigation component nobody in the SEO conversation is allowed to touch.

The graph does two jobs at once, and they fail differently. Google’s crawl-budget documentation defines a site’s budget as capacity together with demand and states that URLs that are more popular on the Internet tend to be crawled more often, so a page nothing points at competes for scheduling before it competes for position. The link documentation is blunter: every page you care about should have a link from at least one other page on your site. One set of edges, two outcomes: whether a URL is fetched at all, a scheduling question, and how much link-derived weight it carries once it is, a ranking question.

What follows is a job rather than an audit: a crawl export goes in, a ranked worklist and an experiment come out, and it runs again next week.

Building the edge list

Everything downstream is decided here, and this is the step people rush. Any crawler that exports inlinks works. Screaming Frog’s All Inlinks export is the common case; the columns used below are Type, Source, Destination, Anchor, Follow, Status Code and Link Position. Check your own header row, because the set has grown across versions.

import pandas as pd

TEMPLATE = {"Navigation", "Header", "Footer", "Aside"}

links = pd.read_csv("all_inlinks.csv")
links.columns = [c.strip() for c in links.columns]

edges = links[
    links["Type"].eq("Hyperlink")
    & links["Status Code"].eq(200)
    & links["Follow"].astype(str).str.lower().eq("true")
].copy()

edges["template"] = edges["Link Position"].isin(TEMPLATE)
edges = (edges.rename(columns={"Source": "src", "Destination": "dst"})
              .loc[lambda d: d["src"].ne(d["dst"]), ["src", "dst", "Anchor", "template"]]
              .drop_duplicates(subset=["src", "dst", "template"]))

Four decisions are hiding in those filters, and each changes the numbers.

Type == "Hyperlink" matters because the same export carries canonical references, hreflang pairs, image sources and redirect relationships, which are rows in a link table rather than edges a crawler traverses as a link. Google can only crawl a link if it is an <a> element with an href, which doubles as the membership test for the graph.

Status 200 on the destination. A link to a URL that 301s is not an edge to the redirect target. Drop those rows, or resolve them through the Redirect URI column of the crawler’s internal export and re-point the edge. Resolving is the better default, since dropping makes the final target look orphaned by an artifact of your own pipeline. Keep the choice constant across runs either way, or your week-over-week diff is noise.

Then the Follow column. Dropping nofollow edges is a modelling choice, not a reproduction of Google’s behaviour. Since 2019 all three link attributes, meaning sponsored, ugc and nofollow, are “treated as hints about which links to consider or exclude within Search”, and for crawling and indexing “nofollow will become a hint as of March 1, 2020”. The current documentation says such links will generally not be followed and sends you elsewhere for internal control: “For links within your own site, use the robots.txt disallow rule.” That is the end of sculpting as a technique.

And deduplication. Two links from one page to the same target are two links in the original formulation, because the normalising term counts links going out of the page. Collapsing them is usually right, since duplicate pairs are mostly markup artifacts: a mobile nav and a desktop nav emitting the same menu twice. Declare which you did: the two give different out-degrees, and therefore different everything.

Finally, crawl with rendering on if any links are injected client-side, or the graph is missing edges and every conclusion below inherits the gap.

Labelling template links instead of deleting them

The tempting move is to filter nav and footer rows out and compute PageRank “on the editorial graph”. Do not. That graph is a fiction: no user agent sees it, and template edges are most of the crawl paths on the site. Keep them, label them, use the label as a dimension downstream.

Screaming Frog classifies each link into Navigation, Header, Aside, Footer or Content from semantic HTML5 elements, and lets you configure rules for sites built out of undifferentiated divs. Verify it on two or three templates by hand first: if your markup has no nav or footer element, everything is classified as Content and the entire analysis below inverts.

Computing internal PageRank

The run is four lines. Note that create_using is not optional: from_pandas_edgelist produces an undirected graph by default, and an undirected PageRank on a website is meaningless.

import networkx as nx

G = nx.from_pandas_edgelist(edges, source="src", target="dst",
                            create_using=nx.DiGraph)
pr = nx.pagerank(G, alpha=0.85)

The current signature is pagerank(G, alpha=0.85, personalization=None, max_iter=100, tol=1e-06, nstart=None, weight='weight', dangling=None). In NetworkX 3.6 it delegates unconditionally to the SciPy implementation, so SciPy is a hard dependency even though it is not a NetworkX one.

One trap worth knowing before you hit it: pass edge_attr=["template"] while building a DiGraph and parallel rows collapse, so a page linking to a target from both the nav and the body becomes one edge with the last row’s attribute. Build the graph without edge attributes and compute template statistics in the dataframe, where the parallel rows still exist.

Dangling nodes are worth understanding rather than accepting. The original technical report removes them entirely: “Because dangling links do not affect the ranking of any other page directly, we simply remove them from the system until all the PageRanks are calculated” (Page, Brin, Motwani and Winograd, 1999; the Stanford InfoLab server has been erroring, so that is the archived PDF). NetworkX instead gives “dangling nodes … outedges according to the personalization vector (uniform if not specified)”, redistributing their weight across the whole site. With a long tail of leaf pages that is not a rounding difference, and it is a choice you make by not making it.

The two parameters that are not free

The damping factor comes from the published description: PR(A) = (1-d) + d(PR(T1)/C(T1) + ... + PR(Tn)/C(Tn)), where “the parameter d is a damping factor which can be set between 0 and 1. We usually set d to 0.85” (Brin and Page, 1998). You are not reconstructing Google’s ranking, only ordering your own pages against each other, so the value matters less than holding it fixed. But check that your conclusion survives it:

from scipy.stats import spearmanr

order = list(G)
a = nx.pagerank(G, alpha=0.85)
b = nx.pagerank(G, alpha=0.50)
rho = spearmanr([a[u] for u in order], [b[u] for u in order]).statistic

If the ordering moves enough to reshuffle your worklist, the worklist is a statement about alpha, not about your site.

The second is personalization, the original paper’s E vector: the “source of rank” introduced to stop rank sinks trapping weight. Its default treats every page as an equally likely entry point; seed it instead on the pages that actually receive external links, so distribution starts where authority enters the site. Where the two runs disagree, your external and internal graphs are pulling in different directions.

The baseline that makes the number mean something

PageRank values form a probability distribution summing to one over N nodes, and that single fact disqualifies the raw number from most of what people do with it. Publish ten thousand tag pages and every existing page’s PageRank falls without one thing about those pages having changed. Compare two crawls of different sizes and the deltas are dominated by N. So use a baseline:

n = G.number_of_nodes()
report = pd.DataFrame({"pr": pd.Series(pr)})
report["lift"] = report["pr"] * n
report["pct"] = report["pr"].rank(pct=True)

T = nx.from_pandas_edgelist(edges[edges["template"]], source="src", target="dst",
                            create_using=nx.DiGraph)
T.add_nodes_from(G)
pr_t = nx.pagerank(T, alpha=0.85)
report["editorial"] = report["pr"] / pd.Series(pr_t)

lift measures against a uniform baseline: 1.0 is what a structureless graph would hand this page, so the number reads as “how many times its fair share”. pct is the percentile within this crawl, the only form comparable across runs of different sizes. editorial is a counterfactual: PageRank on the template-only subgraph, node set held constant with add_nodes_from, which is the floor every page gets free from the build. A page whose editorial ratio sits at 1.0 receives nothing from contextual linking, whatever the CMS reports.

Pages with no inbound edges never appear in pr_t’s index and surface as NaN. That is your orphan list, free.

Template links versus contextual links

The dominance ratio

The count-based version, template inbound links over total inbound links, is easy and slightly dishonest, because it treats an edge from the home page and an edge from a paginated archive as equal. Report the flow-weighted version: each inbound edge carries pr(u)/outdeg(u), so sum that separately for template and contextual edges.

out_deg = edges.groupby("src").size().rename("out")
flow = edges.join(out_deg, on="src")
flow["w"] = flow["src"].map(pr) / flow["out"]

by_target = (flow.groupby(["dst", "template"])["w"].sum()
                 .unstack(fill_value=0.0)
                 .reindex(columns=[False, True], fill_value=0.0))
report["dominance"] = by_target[True] / by_target.sum(axis=1)

A dominance of 1.0 means every unit of internal weight this page receives arrived through boilerplate; NaN means no inbound internal links at all. The distribution of this column across the site is the most useful chart in the whole job.

Why three contextual links change nothing

Take the equation seriously for a moment. The direct contribution of one new edge u -> v is alpha * PR(u) / (outdeg(u) + 1):

def marginal(u, pr, G, alpha=0.85):
    return alpha * pr[u] / (G.out_degree(u) + 1)

Whatever the edge triggers downstream comes on top of that, but the first-order term is where the intuition lives. Suppose the source is an ordinary page whose PageRank sits near the uniform share 1/N, already emitting sixty template links plus a few in the body. The direct term is then 0.85 / (66 * N), roughly 1.3% of a uniform share, and three such links move the target by about 4% of one page’s fair share, into a distribution where the gap between an underpowered page and a healthy one is usually much larger than that.

That is the arithmetic behind the flat distribution. If every page links to nearly every other page the graph approaches a regular graph, and PageRank on a regular graph is uniform. Your navigation is a machine for pushing the site toward that limit, and the symptom is measurable without squinting at a histogram: take the ratio of the 90th to the 10th percentile of lift. A structure that carries information has a wide spread. A structure that is a nav component does not.

Which points at the real lever, and it is a denominator. Cutting a sixty-item mega-nav to fifteen multiplies what every surviving nav link carries by roughly four, across every page, in one release. No campaign of contextual link insertion competes with that, and the same logic governs the pages you link from: an edge from a low-out-degree page with real weight beats many edges from a hub that links to everything.

The mismatch matrix

Internal PageRank ranks pages by how the build treats them, and says nothing about whether that treatment is deserved. The join that turns it into a worklist is against demand: impressions from Search Console, or revenue per URL if you have it mapped. Pull the page dimension from the Search Analytics API, where rowLimit has a “valid range is 1–25,000; Default is 1,000”. That default is why so many of these analyses truncate to a thousand pages without saying so.

import numpy as np

gsc = pd.read_csv("gsc-pages.csv").set_index("page")   # page, clicks, impressions
m = report.join(gsc, how="left")
m["impressions"] = m["impressions"].fillna(0)
m["imp_pct"] = m["impressions"].rank(pct=True)
m["quadrant"] = np.select(
    [(m.pct >= .5) & (m.imp_pct >= .5),
     (m.pct >= .5) & (m.imp_pct < .5),
     (m.pct < .5) & (m.imp_pct >= .5)],
    ["aligned", "over-linked", "starved"], default="quiet")

Both axes are percentiles, never raw values, for the reason above. Impressions rather than clicks, because impressions measure the demand Google has already decided to assign you and are less noisy than the click layer on top. Cut on medians first, tighten to deciles once you can see your own population.

Over-linked is where internal-linking projects go to die. The page already receives internal weight and converts none of it into impressions, so more links change nothing. Either it is the wrong shape for the result set it aims at, which you answer by fingerprinting the SERP before you write, or a sibling URL is absorbing the query, which is what the query–URL matrix is for. Both are cheaper than the link campaign and both are prerequisites for it.

Starved is the list you work. Sort by impressions * (1 - pct): demand high, support low, and the site has already proved these pages can earn attention.

Quiet holds two populations that look identical here and need opposite treatment: pages never discovered, and pages discovered and correctly ignored. Fetch evidence separates them, not intuition, and the scheduling-versus-selection split is the decision procedure. Prune the second population. Link into the first.

Generating link candidates

Candidate generation takes two inputs: topical similarity, so the link makes sense to a reader, and PageRank deficit, so it makes a difference.

Embed the main content of every page, not the DOM: embed the whole document and boilerplate dominates the vector, so every page turns out similar to every page: the same template problem, one layer up. For each starved target, retrieve the top passages across the rest of the site and filter sources hard: 200, indexable, self-canonical, not already linking to the target, not in the same near-duplicate cluster. Score each survivor as similarity multiplied by alpha * pr(u) / (outdeg(u) + 1); ranking on similarity alone hands you links from pages with nothing to give, which is how this pipeline produces a large diff and no measurable change.

The veto step

The output is a diff, never an edit. One row per candidate: source URL, the exact sentence the link would go in, the proposed anchor, the similarity, the marginal contribution. A human accepts or rejects each one and rejection reasons get recorded; after a few runs those reasons become filter rules and the accept rate climbs, which is the only legitimate way this step gets faster. It is not a rate limiter you remove once the model looks trustworthy.

What unattended automation does to a site

The failure modes are consistent enough to enumerate:

• Similarity is textual, not intentional. The retriever matches a term used in a different sense. Obvious collisions are easy to filter; the ones that survive review are subtle.
• The generator writes for the target, not the source page’s reader. Every insertion is individually defensible and the source slowly turns into a link farm.
• Statelessness: successive runs stack links into the same paragraph, because the scoring does not know what last month’s run did.
• Reciprocal pairs. Two similar pages link to each other, both PageRanks rise, the matrix looks better, and no new discovery path exists.
• Near-duplicate targeting. The retriever picks whichever cluster member is nearest in embedding space, which is not necessarily the one you want to rank. That cements a cannibalization problem with structural links.
• Boilerplate contamination. A link inserted into a shared partial becomes a template link across thousands of URLs on the next build, making the dominance ratio worse.
• Stale graph. Candidates computed from last month’s crawl point at URLs that now redirect or 404. Re-validate every destination’s status at write time, not plan time.

Anchor text as a distribution

Anchor text is usually discussed one link at a time, which is the wrong unit. What a search engine sees is the distribution of phrases pointing at a URL, and a distribution has properties no individual decision can express. Compute it over contextual edges only, or a single nav anchor repeated site-wide swamps every statistic.

ctx = edges.loc[~edges["template"]].copy()
ctx["anchor"] = ctx["Anchor"].fillna("").str.strip().str.lower()

def norm_entropy(s):
    p = s.value_counts(normalize=True)
    return 0.0 if len(p) < 2 else float(-(p * np.log(p)).sum() / np.log(len(p)))

g = ctx.groupby("dst")["anchor"]
anchors = pd.DataFrame({"n": g.size(), "distinct": g.nunique(),
                        "top_share": g.agg(lambda s: s.value_counts(normalize=True).max()),
                        "empty_share": g.agg(lambda s: s.eq("").mean()),
                        "entropy": g.agg(norm_entropy)})

Four columns, four different problems. A high empty_share is image links with no alt text and anchors reading “here” or “read more”: edges carrying weight and no meaning, against documentation asking for anchor text that is “descriptive, reasonably concise, and relevant” and naming “click here” as the thing to avoid. A top_share near 1 at large n is one phrase repeated, which usually means a generator wrote it. distinct equal to n at large n is the opposite pathology: sentence fragments rather than descriptions of the destination. entropy is the summary you threshold and alert on.

I have no documented penalty to cite for exact-match saturation and will not invent one. The reason to care is diagnostic: if every inbound anchor is the exact target query, you have removed the variation that would let you observe which phrasing pulls. Pick thresholds on those four columns, tune them on your own site, treat a breach as a review trigger rather than an automated rewrite. And note the cross-check: a starved page whose few inbound anchors all describe a different topic is not starved, it is mislabelled, and new links do not fix that.

Running it as an experiment

Internal linking is one of the very few SEO interventions that can honestly be randomised. You control the treatment completely, you apply it to some pages and not others in the same release, and the holdout lives on the same site under the same updates.

Assignment

The unit is the target page. The pool is the candidate generator’s output after the veto: every page that would have been treated. Both arms are drawn from that pool. If your holdout is “the pages we did not pick”, you are comparing pages good enough to become candidates against pages that were not, and the design is dead before the first link ships.

Stratify on template and pre-period impression decile so the arms start matched, then split within each stratum:

pool["imp_decile"] = pd.qcut(pool["impressions"].rank(method="first"), 10, labels=False)
pool = pool.sample(frac=1, random_state=20260831)
pool["arm"] = np.where(pool.groupby(["template", "imp_decile"]).cumcount() % 2 == 0,
                       "treatment", "holdout")

Two hygiene rules that are easy to forget. Treated source pages must not link to holdout targets, so drop those candidate rows or the treatment leaks into the control. And exclude the source pages from the outcome set, because they were edited and their own metrics moved for reasons unrelated to the hypothesis.

The metric and the observation window

Declare the metric before you look at anything. Impressions per target page over a fixed window is the default: least noisy signal in the export, and it measures the demand Google assigns. Clicks if the pages are commercial and the volume supports it. Not average position, which averages over a result set whose composition changed underneath you.

The window has two parts and the first is not optional. The change does not exist for Google until the source pages have been re-fetched, so confirm the re-crawl in your logs rather than assuming a lag. That is what the access-log pipeline is for. Start the post-period after the last source fetch, take a pre-period of equal length, then run a difference-in-differences on log impressions and bootstrap a confidence interval over pages rather than days, because pages are the unit you randomised.

The noise floor, measured rather than assumed

Do not run a power calculation on invented variance. Measure the floor on your own data with an A/A test on the pre-period: shuffle the same pool, assign sham arms with the same stratification, compute the same statistic, repeat a few hundred times, take the 95th percentile of the absolute gap.

def sham_gap(seed, df):
    s = df.sample(frac=1, random_state=seed)
    arm = np.where(s.groupby(["template", "imp_decile"]).cumcount() % 2 == 0, "a", "b")
    med = s.assign(arm=arm).groupby("arm")["impressions_pre"].median()
    return med["a"] / med["b"] - 1

floor = np.percentile([abs(sham_gap(s, pool)) for s in range(200)], 95)

Anything the post-period produces below floor is not a result. That one number is worth more than the rest of the analysis, because it is what stops you from shipping a random walk as a finding.

Confounds, including the one the holdout cannot absorb

The holdout absorbs seasonality, core updates and site-wide releases, which is the entire reason it exists. It does not absorb anything correlated with candidacy: if the treated pages were also the ones whose content got refreshed, you measured a bundle. And one confound is specific to this intervention: PageRank is a probability distribution over the whole graph, so weight added to the treated pages is weight removed from everyone else, holdout included. The effect is second order, but it biases toward finding an effect, and the honest write-up says so.

Reporting a null

Report the point estimate, the interval, the minimum effect the design could have detected given your measured floor, and the metric you pre-declared. Then stop. Re-cutting the data by segment until something clears significance is how a null becomes a case study.

A null on the first run usually means the treatment was too small rather than that internal linking does not work. Three links from average pages, per the arithmetic above, move a target by a few percent of one uniform share. The response is a larger treatment on the next cohort, or one aimed at out-degree instead. It is not a new analysis of the same data.

Keeping it from rotting

An audit is a snapshot, and the graph changes every time someone edits a template. Schedule the crawl weekly, write one row per URL per run into a snapshot table (run_date, url, pr, lift, pr_pct, indeg_template, indeg_contextual, outdeg, dominance and the anchor columns), and make the diff the deliverable rather than the absolute values.

SELECT curr.url,
       round(curr.pr_pct - prev.pr_pct, 3) AS pct_delta,
       curr.indeg_contextual - prev.indeg_contextual AS ctx_delta
FROM snapshot curr
JOIN snapshot prev
  ON prev.url = curr.url AND prev.run_date = :previous
WHERE curr.run_date = :current
  AND (abs(curr.pr_pct - prev.pr_pct) > 0.05
       OR curr.indeg_contextual < prev.indeg_contextual)
ORDER BY pct_delta ASC;

Alerts worth wiring up: URLs that entered the orphan set this week; pages that lost contextual inbound edges; percentile drops beyond a threshold you set; edges pointing at destinations that stopped returning 200. Then one graph-level series no page-level alert will surface: the spread of lift over time, so you can watch the distribution flatten as the navigation grows.

The highest-value alert catches template releases. A nav change moves thousands of edges in one deploy, so filter the diff for changes affecting more than some count of URLs at once and you hear about it the day it ships rather than during next quarter’s audit.

All of this is instrumentation for a lever you already own outright. The internal graph is the only part of the link structure you control completely, the only part you can change on a Tuesday afternoon, and the only part where a proper holdout is available, which also means it is the part where you have no excuse for reporting a result you did not measure. Run the job, work the starved quadrant, let your own cohort tell you whether it did anything. The answer is allowed to be no.