Technical SEO Tasks: The Complete Crawl-to-Rank Reference in 2026

Here’s something I tell every client on the first call. If Googlebot can’t reach a page, nothing else on this list matters. Not the content. Not the backlinks. Not the schema markup someone spent a week building. Most technical SEO checklists bury that fact under 25 random bullet points. I want to fix that. The order is the whole point.

Everything in this guide fits into one of four stages: Crawl, Render, Index, Rank. Fix Crawl before you touch Render. Fix Render before Index. Get Index right before you spend another afternoon chasing Core Web Vitals scores. I call this the CRIR model. Once you start diagnosing problems this way, you stop wasting time on fixes that were never going to help.

I’ve watched teams spend three weeks shaving 400 milliseconds off LCP on a page stuck in “Discovered, currently not indexed.” The speed work wasn’t wrong. It just fixed a stage the page hadn’t reached yet.

Where To Start: A Real Diagnostic

Run this against your own site first. It will tell you which section actually applies to you.

Is the page showing up in Google at all?
├─ NO
│   ├─ Is it in the sitemap and linked from a live page?
│   │   ├─ NO  → Stage 1: add internal links, add to sitemap
│   │   └─ YES → Check robots.txt and meta robots
│   │       ├─ Blocked     → Stage 1: remove the block, resubmit
│   │       └─ Not blocked → Does the content need JavaScript to load?
│   │           ├─ Yes → Stage 2: check rendered HTML in URL Inspection
│   │           └─ No  → Check the GSC Coverage reason
│   │               ├─ "Discovered, not indexed" → Stage 1: crawl priority issue
│   │               └─ "Crawled, not indexed"    → Stage 3: content or duplication issue
└─ YES, but it ranks poorly
    ├─ Core Web Vitals failing → Stage 4: fix LCP, INP, CLS
    ├─ No schema, weak SERP presence → Stage 4: add structured data
    └─ Otherwise → probably a relevance problem, not a technical one

The CRIR Model

Most technical SEO advice treats crawling, indexing, and speed as separate categories. You just work through them in some rough order. I don’t think that’s right. These stages depend on each other. There’s a real sequence:

Crawl comes first. Can a bot reach the URL? This covers server responses, robots.txt, internal links, and redirect chains.

Render comes next. Once the bot has the page, can it actually see the content? This is where JavaScript either helps you or quietly costs you.

Index is the gate after that. Does the content earn a spot in Google’s index, or does it get filtered out for being thin, duplicate, or blocked?

Rank is the only stage that’s actually about competing. Among the pages that made it through, does yours perform well on speed, schema, structure, and mobile experience?

Diagnose top to bottom. Fix bottom to top. A page stuck at Crawl doesn’t need a Rank fix, no matter how tempting the Core Web Vitals dashboard looks.

Here’s a pushback I give in almost every strategy call. Crawl budget is not something you protect by blocking pages. That’s the first instinct most people have, and it’s backward. Every crawl happens somewhere. The real question is whether that effort lands on pages that can actually rank. Blocking a low-value URL in robots.txt doesn’t send that saved effort anywhere useful on its own. Fixing your internal links so crawl activity flows toward the pages you care about does that. I call this idea crawl spend allocation, because it’s different from how most people use the term “crawl budget.” I’ll come back to it later.

Here’s why a business owner should care about a four-stage model instead of just wanting things fixed. Each stage has its own cost attached, and it’s not just a ranking cost. A page stuck at Crawl gets zero traffic. Full stop. It doesn’t matter how good the content is. A page stuck at Render might get indexed with half its content missing, so it technically ranks but converts badly because the offer never loaded. A page that clears Index but loses at Rank is the cheapest problem to have. It’s visible. It’s just not winning yet, and that’s a tuning problem, not a plumbing problem. When a founder asks me to justify a technical SEO budget line, this is the argument. You’re not paying to rank higher. You’re paying to stop losing traffic you already earned through content and links, quietly, at a stage below where anyone’s looking.

Stage 1: Crawl

Task 1: Confirm Your Priority Pages Are Crawlable

Why it matters: Googlebot doesn’t crawl your whole site fresh every visit. It works within a budget based on how fast your server responds and how much value it thinks your site has. Then it spends that budget across your URLs. One stray Disallow line. A leftover noindex tag from staging. A page with zero internal links pointing to it. Any of those, and the bot never gets there.

How to check: Open Search Console’s Crawl Stats report. Fetch your live robots.txt and actually read every Disallow line. Don’t trust your memory of what it says. I’ve lost count of how many audits turn up a block nobody remembers adding. Run a full Screaming Frog crawl and filter for anything that isn’t returning a 200.

Common mistake: People assume blocking a URL in robots.txt removes it from Google. It doesn’t. If the page was already indexed, blocking it can leave it sitting in the index forever with no title and no snippet, just a bare URL and “No information is available for this page.” A block only stops future crawling. It does nothing to what’s already indexed. If you actually want a page removed, the order matters: allow crawling, add noindex, let Google recrawl and drop it, then block it afterward if you still want to.

Fix: Clean up robots.txt line by line. Make sure every page you want ranked has at least one internal link from a page Google already visits often.

Task 2: Decide On Purpose About AI Crawlers

Why it matters: GPTBot, PerplexityBot, ClaudeBot, and the rest crawl your site through their own user agents. They follow the same robots.txt rules as Googlebot. But they often get blocked by default through a CDN or security plugin’s bot settings, set up long before this was even a decision worth making. It’s fully possible to be crawlable for Google while invisible to every AI answer engine, just because some bot-blocking rule got flipped on a year ago for a reason that had nothing to do with this.

I’d also mention llms.txt, a plain-text file some sites now put at their root to summarize what they consider their best content for AI systems. It’s not a standard yet, and I wouldn’t over-invest in it. But checking your actual bot rules is worth doing now, not once the format settles.

Fix: Go through your robots.txt and your CDN or firewall bot rules. Check specifically for GPTBot, PerplexityBot, ClaudeBot, CCBot, Google-Extended, and OAI-SearchBot. Decide on each one. Don’t let a blanket “block all bots” setting make the call for you.

Task 3: Read Your Log Files, Not Just Search Console

Why it matters: Search Console’s crawl data is sampled and delayed. Your server logs show exactly what Googlebot did, and when, with no averaging and no lag. If you want to know whether your crawl effort is really going where you think it is, this is the only place that tells you the truth.

A real example: I worked with a publisher running about 50,000 pages. Log analysis showed Googlebot spending roughly 40% of its daily crawl on infinite-scroll pagination and tag archives that had never produced a single ranking page. The obvious move would be to block those in robots.txt. That would have been a mistake, because internal link equity was still flowing through those pages to content that mattered. The real fix was adding noindex to the thin archives while keeping the links, then adjusting internal links and sitemap priority to push crawl activity toward the pages meant to rank. Crawl activity shifted within three weeks. You could see it in the logs before it showed up in rankings.

Fix: Pull at least 30 days of raw server logs every quarter, and every month if you run a large or fast-publishing site. Compare where crawl activity actually goes against where you want it to go.

Task 4: Sort 404s By Evidence

Why it matters: Not every dead link deserves the same urgency. Most checklists just say “fix your 404s” without telling you which ones to fix first.

Ask three questions, in this order, before you touch a redirect. Does the dead URL have backlinks pointing to it? Did it used to get organic traffic? Is there a live page it genuinely makes sense to send it to?

A 404 with backlinks and no good redirect target is a harder call than a 404 with backlinks and an obvious target. Treat those two cases differently instead of running through one flat checklist.

Fix: Redirect to the closest genuinely relevant page. If nothing relevant exists, redirect to the nearest topical section instead of just defaulting to the homepage. A homepage redirect for an unrelated dead page sends a weaker relevance signal than most people expect.

Task 5: Collapse Redirect Chains

Why it matters: A to B to C should be A to C. Every extra hop slows the crawl down and weakens how much link equity actually gets through.

Here’s the part most guides skip. Chains build up over time. A URL redirected once during a 2023 site update, then again during a 2025 rebrand, can end up four or five hops deep with nobody noticing. That quietly holds back a page that looks perfectly healthy in every other report you’d check.

Fix: Screaming Frog’s Redirects tab finds chains automatically. Point every chain straight at its final destination. Never redirect to another redirect.

Task 6: Build a Migration Checklist Before You Need One

Migrations carry the highest risk on this entire list, and they deserve their own checklist instead of one line in a task table.

Crawl the old site completely before anything changes, so you have a full URL list. Map every indexed URL to its new home, not just your top ten pages. Have your redirects live at launch, not “soon after.” Double-check that staging’s noindex tag and password wall are actually gone before you go live. This is the most common migration disaster I see: launching with staging’s noindex tag still turned on, and nobody catching it for a week. Crawl the new site right after launch and compare it against the old URL list. Resubmit your sitemap the same day. Then check Search Console daily for the first two weeks. Weekly isn’t enough right after a migration.

Stage 2: Render

Task 7: Check What Googlebot Actually Sees After JavaScript Runs

Why it matters: This is the part most guides get half right. Googlebot works in two separate passes. First it fetches the raw HTML. Then it puts the page in a queue to render, running the JavaScript in a headless browser to see the final page, and that queue is not instant. On large sites, or lower-authority ones, the gap between the crawl and the render can run from seconds to several days.

If your main content or your internal links only show up after JavaScript runs, you’re not just “harder to crawl” in some abstract way. You depend on a second pass that a lot of pages never get promptly, and sometimes never get at all if the page gets pushed down the queue before rendering finishes.

I’d push back on the idea that “Google supports JavaScript now” settles this. Support isn’t the same as speed. A page can render correctly on the tenth attempt while its ranking already got decided around the ninth.

How to check: Search Console’s URL Inspection tool has a “View Crawled Page” option that shows the rendered HTML Google actually saw. Don’t trust your browser’s view-source, which shows the page before it renders.

Fix: For anything critical to getting indexed, use server-side or hybrid rendering instead of pure client-side rendering. At minimum, confirm through URL Inspection that your main content and internal links survive the render, not just empty placeholders.

Stage 3: Index

Task 8: Check Indexation By Reason, Not Just Count

Why it matters: A crawled page isn’t automatically an indexed one. The fix depends entirely on why Google left it out.

GSC status	What it means	Fix
Crawled, currently not indexed	Google visited and decided the content wasn’t worth indexing	Expand thin content, strengthen what makes it unique
Discovered, currently not indexed	Google found the URL but hasn’t crawled it yet	Improve internal links and sitemap priority
Duplicate without user-selected canonical	Google sees near-identical content across URLs	Add clear canonical tags
Alternate page with proper canonical tag	Working as intended	Nothing to fix

How to check: Search Console, then Index, then Pages, checked category by category. Don’t just glance at the total number.

Task 9: Fix Duplicate Content With the Right Tool for the Cause

Why it matters: Duplicate content splits your ranking signal across multiple URLs that end up competing against each other for the same search. The fix depends on why the duplication is happening in the first place.

Common causes: www versus non-www, HTTP versus HTTPS, trailing slash variants, and URL parameters that create near-identical pages at scale, like ?color=blue versus ?color=red.

I’ve seen this at very different sizes. On a simple site, it’s usually just an unconfigured www redirect. On a 2,000-product store, color and size parameters had created thousands of near-duplicate product pages, all splitting the signal that should have gone to one canonical page per item.

How to check: Screaming Frog’s Canonicals tab, and Search Console’s Coverage report under “Duplicate without user-selected canonical.”

Fix: Put a consistent canonical tag on every page, pointing to the version you prefer. Make sure www/non-www and HTTP/HTTPS all lead to a single version.

Redirect vs. Canonical

People mix these two up constantly, so here’s the actual comparison instead of a vague rule of thumb.

	301 Redirect	302 Redirect	Canonical Tag
Passes link equity	Fully	Weakly, and only once Google treats it as permanent	Yes, to the page you name
Signals it’s permanent	Yes	No, it’s temporary by design	Not really. It’s a preference, not a move
User gets redirected	Yes	Yes	No, they stay on the original URL
Best use	Content that’s gone for good	A short test or a maintenance window	Several live URLs, where you want one to be the indexing target

Using a canonical tag when you actually needed a redirect leaves the old URL live, drawing weak, split signal instead of pulling it all together. Going the other way, redirecting URLs that users still need to reach in both forms, causes its own headaches. Match the tool to what’s actually happening on the page.

Task 10: Confirm HTTPS Is Actually Complete

Why it matters: A padlock on your homepage doesn’t prove your whole site runs on HTTPS. Mixed content, where an HTTPS page quietly loads some resources over plain HTTP, still hurts trust and speed even when the address bar looks fine.

Fix: PageSpeed Insights flags mixed content directly. Redirect every HTTP URL to HTTPS across the whole site, and update your internal links to point at the HTTPS versions instead of relying on the redirect to catch it each time.

Task 11: Add Hreflang If You Serve More Than One Market

Why it matters: Without hreflang, Google can read your regional or language versions as duplicate content competing with each other, instead of correctly targeted pages for different audiences.

Fix: Add matching hreflang tags across every version, including a tag on each page pointing to itself. Check them through Search Console’s International Targeting report. Hreflang mistakes are common, and they stay silent until you actually go check.

Task 12: Keep Your Sitemap Trustworthy

Why it matters: A sitemap full of noindexed, redirected, or non-canonical URLs teaches Google to trust it a little less every time it’s crawled.

Fix: Only put indexable, canonical, live URLs in there. Set it to update automatically when you publish. Submit it in Search Console and confirm it shows zero errors, not just that it got accepted.

Task 13: Push Time-Sensitive Content Instead of Waiting

For fast-publishing or large sites, don’t sit around for a normal recrawl. IndexNow, supported by Bing and a few others, and Google’s own “Request Indexing” tool inside URL Inspection both skip ahead on discovery for content where speed actually matters.

Stage 4: Rank

A quick note before the individual metrics. Every score below is given as a labeled range (Good, Needs Improvement, Poor) instead of a raw number you’re expected to memorize. That’s on purpose. It borrows from a well-known idea in psychology called recognition over recall. People judge labeled categories faster and more accurately than they recall a raw number and compare it to some mental cutoff in their head. You don’t need to remember that 2.5 seconds is the line. You need to recognize “Good” when you see your own score land in that column. Build your own reports the same way if you’re showing these numbers to a client or a founder who isn’t going to memorize the cutoffs either.

Task 14: LCP (Largest Contentful Paint)

Under 2.5 seconds is good. Over 4.0 seconds is poor. It’s almost always caused by images, not code.

That’s worth saying plainly, because the first instinct, especially for developers, is to assume a slow LCP means a JavaScript problem and reach for code-splitting first. In most audits I’ve run, the actual largest element on the page turns out to be an oversized hero image or a heavy, unoptimized banner. Check that before you touch the JavaScript.

Fix: Compress and convert images to WebP or AVIF. Lazy-load everything below the fold, but not the LCP element itself. Lazy-loading that one will actually make LCP worse. Put a CDN in front of your assets. Defer any JavaScript that isn’t critical.

Task 15: INP (Interaction to Next Paint)

Under 200 milliseconds is good, over 500 is poor. INP replaced FID as the official Core Web Vital back in 2024. If a checklist you’re reading still treats FID as current, that’s a good sign it hasn’t been updated in a while.

A real example: on a mid-size SaaS site, the first suspect for a bad INP score was the CMS. It wasn’t. One third-party chat widget was blocking the main thread on every click. Removing it, nothing else, dropped INP from about 640 milliseconds to 190.

Fix: Check every third-party script for whether it’s actually needed before you touch your own code. Defer anything that isn’t critical. Split your code so pages only load what they need.

Task 16: CLS (Cumulative Layout Shift)

Under 0.1 is good, over 0.25 is poor. It’s almost always caused by images or ads loading without any reserved space.

Fix: Give every image explicit width and height attributes. Reserve space in your CSS for ads and embeds before they load. Use font-display: swap for web fonts. And never drop new content above what’s already on the page after it first loads.

Worth flagging: Search Console’s Core Web Vitals report uses real data from real Chrome users, while PageSpeed Insights leads with lab data from one test run. They can disagree, and that’s normal, not a bug. Field data reflects your actual visitors on their actual devices. Lab data reflects one controlled run. When the two disagree, use field data to decide what to fix first, and lab data to figure out why.

Task 17: Fix Your URL Structure

Keep URLs short, hyphenated, free of extra parameters, and shallow, three folders deep at most for most content.

Use /seo-guide, not /page?id=123&category=5&ref=nav.

How to check: Export every URL through Screaming Frog and filter for length, parameter count, and messy patterns.

Task 18: Check Internal Link Depth

Every important page should sit within three clicks of the homepage, with zero true orphans, meaning pages with no internal links pointing to them at all.

How to check: Screaming Frog’s Inlinks report ranks every page by how many internal links point to it. Anything near zero is both a crawl priority problem and a missed chance to pass authority from your strongest pages.

There’s a reason this matters to people, not just to Googlebot, and it’s worth borrowing a term from information science: information scent, the cues a person uses to guess whether clicking a link will get them closer to what they want. A page buried four clicks deep with vague link text isn’t just hard for a crawler to prioritize. It’s hard for a real visitor to trust enough to click. The same shallow, well-labeled structure that helps Googlebot also keeps a real visitor from bouncing back to the search results. That’s not a coincidence. Google built its crawl-depth signals to guess how findable a page actually is, and that was never a purely technical question to begin with.

Task 19: Treat Crawl Budget as Something You Spend On Purpose

For sites over roughly 500 pages, this deserves active attention, not a once-a-year pass at blocking obviously useless pages.

Fix: Add noindex to thin or low-value pages instead of only blocking them in robots.txt (see the common mistake in Task 1). Merge near-duplicate thin pages together. Fix redirect chains. All three of these actually push crawl effort toward pages that can rank, instead of just cutting down how much crawling happens overall.

Tasks 20 to 24: Structured Data

Schema tells search engines what your content means, and it makes a page eligible for rich results. It doesn’t guarantee them, and I’ll explain why that gap matters in a minute.

Here’s the business case for doing it anyway. Rich results take up more space in the search results and give someone a reason to click before they’ve even reached your page. An FAQ dropdown or a how-to preview competes for attention right next to plain blue links. That’s a click-through lever that costs a developer an afternoon, which makes it one of the best time-to-return ratios on this whole list, even though it doesn’t touch a single ranking factor directly.

Add Organization or LocalBusiness schema to your homepage and contact page, with your name, logo, address, phone, and social profiles. Add Article or BlogPosting schema to editorial content, linking it to a named author, which supports E-E-A-T. Add FAQPage schema to any page with real FAQ content. It’s one of the highest-impact types for search result space. Add HowTo schema to step-by-step tutorials, which can trigger step-level rich results. Add BreadcrumbList schema site-wide, so hierarchy shows directly in the search result.

Validate everything with Google’s Rich Results Test and Search Console’s Enhancements report, and check it again on a mobile user agent, since structured data can render differently across devices.

Common mistake: assuming valid schema guarantees display. It doesn’t. A page can have perfectly valid schema and still get a near-zero rich-result display rate on a competitive search, because Google decides on its own whether to actually show the enhanced result. Schema gets you eligible. It doesn’t get you shown.

Task 25: Fix Mobile Usability

Google uses mobile-first indexing across the board, so your mobile experience is your main ranking experience, not a check you run once a quarter.

Fix: Check Search Console’s Mobile Usability report. Keep tap targets at least 44 by 44 pixels. No horizontal scrolling. A 16 pixel minimum font size that doesn’t force anyone to zoom. And test your structured data specifically on a mobile user agent, since it can render differently than it does on desktop.

Task 26: Remove Intrusive Pop-ups

Full-screen pop-ups that block content on mobile get penalized directly in Google’s page experience checks.

Fix: Swap full-screen overlays for smaller notification bars or slide-in banners that don’t block the content someone actually came for.

What Most People Get Wrong

A few myths come up in nearly every client audit I run, usually stated with total confidence.

“Blocking a URL in robots.txt removes it from Google.” It doesn’t. It can stay indexed forever with no snippet. Use noindex if removal is actually the goal.

“Valid schema guarantees rich results.” It makes you eligible. Google decides display on its own, and eligibility with zero display is a completely normal outcome.

“FID is still the metric to track for interactivity.” It was replaced by INP in 2024. Anything still citing FID as current probably hasn’t been updated in a while.

“Field data and lab data should always match.” They measure different things: real users on real devices versus one simulated test run. Disagreement between them is the expected outcome, not a red flag.

“Managing crawl budget just means blocking pages.” It means pointing effort toward pages that can actually rank. Blocking is one tool in that process, not the whole strategy.

“A canonical tag and a redirect do the same job.” They solve different problems. Check the comparison table above if you’re not sure which one you need.

Symptom, Cause, Fix

Symptom	Likely cause	Where to check	Fix
Page is indexed but gets zero organic traffic	Thin content, or it matches the wrong search intent	GSC Performance, plus a manual look at the search results	Rewrite for the intent that’s actually there
LCP flagged as poor, but the page feels fast	A large, unoptimized hero image, not a JavaScript problem	PageSpeed Insights, “Opportunities” section	Compress and convert images, add a CDN
Sitemap lists 500 URLs, GSC shows 300 indexed	Quality or duplication filtering	GSC Coverage, broken down by reason	Fix the specific excluded reason, not the total number
Schema is eligible but rich results never show	Google is choosing not to display them	Rich Results Test, plus the Enhancements report	Nothing to fix. Eligibility isn’t guaranteed display
Mobile usability errors cluster on one template	A shared template bug, not a content issue	GSC Mobile Usability, filtered by URL pattern	Fix it at the template level
Heavy crawl activity, but indexing is still slow	Crawl is landing on low-value URLs instead of priority pages	Server log analysis	Add noindex to thin sections, rebalance internal links

The Cadence

Task	Frequency	Tool	Time
GSC Coverage check	Weekly	Search Console	10 min
Core Web Vitals review	Monthly	GSC and PageSpeed Insights	20 min
Full site crawl	Monthly	Screaming Frog	2 to 3 hrs
Log file review (large or fast-publishing sites)	Monthly	Log File Analyser	30 to 45 min
Broken link repair	Monthly	Screaming Frog and Ahrefs	30 to 60 min
Canonical and redirect audit	Monthly	Screaming Frog	30 min
Mobile usability review	Monthly	Search Console	15 min
Schema validation	Quarterly	Rich Results Test	30 min
Sitemap health check	Quarterly	GSC and Screaming Frog	20 min
Full architecture review	Quarterly	Screaming Frog	1 to 2 hrs
Complete technical audit	Twice a year	All tools combined	4 to 6 hrs
Migration checklist	Before any migration	Full crawl plus manual mapping	Varies

How This Fits Into a Full Campaign

Technical SEO is Phase 3 in most well-run campaigns, sitting after the first audit and before content creation. I hold a hard line on this with clients. Don’t publish new content on a site that’s technically broken. Every page you publish after that underperforms if Google can’t crawl and index it well, and you won’t know that’s the problem until you’ve already spent the budget on content.

Fix the foundation first. Then create content. Then build links. That order protects the return on everything you do after it.

FAQ

Key Takeaways

Every technical issue fits into one of four stages: Crawl, Render, Index, Rank. Diagnose the right one before you fix anything, or you’ll waste real time solving a problem the page hasn’t even reached yet.

Crawl budget gets spent, not saved. The goal is pointing it toward pages that can rank, not just blocking the ones that can’t.

Robots.txt blocking and noindex solve different problems. Blocking stops future crawls. It doesn’t touch what’s already indexed.

Core Web Vitals field data and lab data measure different things, and they will honestly disagree sometimes. Use field data to decide what to fix first, and lab data to figure out why.

Schema markup gets you eligible for rich results. It doesn’t guarantee them, and mixing up the two leads to a lot of confused clients.

AI crawlers are a separate access decision from Googlebot. Check them on purpose instead of inheriting whatever your security tools defaulted to.

At the end of the day, technical SEO removes friction. It doesn’t create an advantage on its own. Nothing in this guide will make a mediocre page outrank a genuinely better one. But leave any one item here unfixed long enough, and it can make a great page invisible.