The Technical SEO Checklist I Use on Every Website

robots.txt

This is the first file any well-behaved crawler looks for. It tells crawlers which parts of your site they're allowed to crawl and which they should stay out of. It lives at yoursite.com/robots.txt, and every website should have one.

Two things people often get wrong here:

AI crawlers in your robots.txt

This one is new, and I think it's going to matter more every year.

AI platforms like ChatGPT, Claude, and Perplexity have their own crawlers, and each one identifies itself with a user agent. Some of the common ones:

You get to decide which of these you allow. That's a business decision, and there's no single right answer. But make it a conscious decision.

Here's why I say that. A lot of sites are blocking AI crawlers without knowing it. In July 2025, Cloudflare started blocking AI crawlers by default for new domains. So you can have a perfectly clean robots.txt and still be invisible to ChatGPT because of a setting in your CDN or firewall you never touched.

If AI visibility matters to you, check your robots.txt and your CDN or firewall bot settings.

A simple example that allows the AI search crawlers in:

User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

sitemap.xml

Your XML sitemap is the list of pages you want search engines to know about. Every site should have one, submitted in Google Search Console and referenced in your robots.txt.

But having a sitemap isn't enough. It needs to be clean. I call this sitemap hygiene, and it comes down to a few rules:

1. Only include URLs you actually want indexed. That means canonical URLs that return a 200 status code and don't have a noindex tag.
2. No redirects, no 404s, no noindexed pages, no parameter URLs. If your sitemap says "index this" and the page says "don't index me" or "I've moved," you're sending mixed signals.
3. Keep lastmod honest. Only update the date when the page content actually changes. Google uses lastmod if it's consistently accurate, and ignores it if it isn't. (Google ignores priority and changefreq entirely, so don't stress over those.)
4. Split it up when it gets big. One sitemap file can hold up to 50,000 URLs (or 50 MB uncompressed). Larger sites should use a sitemap index pointing to multiple sitemaps, ideally split by page type so you can see in Search Console which sections are getting indexed.

HTML sitemap (for bigger websites)

If your website is big, an HTML sitemap is worth having. It's a regular page on your site that links out to your important pages in an organized way, which gives both users and crawlers a clear path to everything.

The best example I know is the New York Times. Have a look at their HTML sitemap: it's organized so that you can reach pretty much any article within three to four clicks, whether the article is from 1920 or 2025. That's what an ideal HTML sitemap does. It keeps your crawl depth low, so no important page is buried ten clicks deep where crawlers rarely reach.

llms.txt optional

This one's a bit controversial. llms.txt is a proposed file that gives AI models a simple, markdown-based summary of your site. Half the SEO world says add it, the other half says it's useless.

Here's where I land: it's cheap to add, and there's no harm in having it. Some AI crawlers might use it to understand the context of your website. But don't expect results from it.

"No AI system currently uses llms.txt."

John Mueller, Google, June 2025

So add it if you want, and then spend your real effort on everything else in this list.

The <head> of your page is where you tell search engines what the page is. Every page on your website should have these:

Meta title

The title that shows up in search results.

Meta description

The snippet below it.

Canonical tag

The URL you want treated as the "real" version of this page.

Meta robots tag

Your indexing instructions for crawlers.

Structured data

Your schema.org JSON-LD code (more on this in section 10).

Open Graph and Twitter cards

How your page looks when it's shared on social media and messaging apps.

Here's what a clean head looks like:

<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Technical SEO Checklist | Your Brand</title>
  <meta name="description" content="The technical SEO baseline every website needs.">
  <link rel="canonical" href="https://yoursite.com/technical-seo-checklist/">
  <meta name="robots" content="max-snippet:-1, max-image-preview:large, max-video-preview:-1">

  <meta property="og:title" content="Technical SEO Checklist">
  <meta property="og:image" content="https://yoursite.com/og-image.jpg">
  <meta name="twitter:card" content="summary_large_image">

  <script type="application/ld+json">{ ... }</script>
</head>

A note on the meta robots tag

You'll often see <meta name="robots" content="index, follow"> on pages. That's fine, but it's also the default, so it isn't doing anything. The meta robots tag really matters when you want to change the default: noindex a page, or control how your snippets appear.

And here's one I recommend for almost every site: max-image-preview:large. Google Discover requires it (along with images at least 1,200 pixels wide) to show your content with a large image preview. Without it, you're much less likely to show up prominently in Discover. I usually pair it with max-snippet:-1 and max-video-preview:-1, which tell Google there's no limit on how much of your text or video it can preview.

Keep tracking codes in one place

Your head is also where tracking codes live: Google Analytics, Google Tag Manager, the Search Console verification tag, product analytics tools, and whatever else your marketing team adds.

My recommendation: install Google Tag Manager once, and fire every other tag through it. Your head stays clean and sorted, and you aren't pasting a new script into your theme every time someone signs up for a new tool.

No invalid elements in your head

This is one most people have never heard of, and it can silently break your SEO.

Google's documentation says the <head> should only contain these elements:

If Google finds anything else in there, like an iframe or an img, it assumes the head has ended right there and ignores everything after it.

So if your canonical tag, meta robots, or hreflang comes after an invalid element, Google never sees it. Your page looks fine to you, but Google is reading half of your head.

Use semantic HTML

Semantic HTML means using tags that describe what the content is, instead of wrapping everything in <div>s:

This makes it easier for search engines (and any crawler) to understand which part of your page is the main content, which part is navigation, and which part is the footer you repeat on every page. It's a small thing that helps crawlers spend less effort figuring out your page.

Heading tags, like a book index

Your heading tags tell crawlers what your page is about and how it's organized. The H1 and H2s especially carry the main context of the page. I like to think of headings like the index of a book:

Don't pick heading levels based on how big you want the text to look. That's what CSS is for.

Images

Images are usually the heaviest thing on a page, so a few rules:

• Keep each image under 100 KB, ideally. It's a rule of thumb I follow, not a Google requirement, but it keeps pages light.
• Use modern formats like WebP or AVIF. JPG works as a fallback.
• Always add alt text. It's what screen readers read out, what shows if the image fails to load, and how search engines understand what the image is about.

One thing to keep in mind if you care about Google Discover: Discover wants images at least 1,200 pixels wide. That's in tension with the 100 KB rule, so the answer is to compress well, not to shrink the image.

Keep your HTML under 2 MB

Googlebot only reads the first 2 MB of an HTML file. Anything after that point is ignored, as if it didn't exist.

A couple of details matter here. The limit applies to the uncompressed file size. And it's per file: your CSS and JavaScript files are each fetched separately with their own 2 MB limit. Images don't count toward your HTML's limit.

Most pages are nowhere near 2 MB of HTML. But I've seen sites with massive inline scripts, huge embedded JSON blobs, or entire product catalogs dumped into a single page get dangerously close. If your important content sits at the bottom of a bloated page, make sure Google actually gets to it.

Separately, keep your total page weight (images, scripts, everything) as low as you can. That isn't a hard cutoff like the 2 MB rule, but it directly affects speed.

Server-side rendering, ideally

Ideally, your pages should be server-side rendered. That means when a crawler requests your page, the server sends back HTML with all your content already in it. The crawler sees exactly what a user sees, without having to run any JavaScript.

This matters even more now because of AI crawlers. Google can render JavaScript (eventually, and at a cost). But Vercel studied AI crawler traffic in late 2024 and found that the major AI crawlers, including GPTBot and ClaudeBot, fetch JavaScript files but don't execute them.

If your content only appears after JavaScript runs, ChatGPT and Claude simply don't see it.

Or a hybrid approach

If parts of your site have to be client-side rendered, go hybrid. Server-side render the important text: the content you want search engines and AI crawlers to read. Load the interactive components and everything else on the client side.

Hidden content needs to be in the DOM

Many pages have content that only appears when a user clicks something. FAQ accordions are the classic example, along with tabs, "read more" buttons, and the expandable sections you see on most landing pages.

That's fine, as long as the hidden text is already in the HTML when the page loads. If it only gets fetched after the click, crawlers never see it.

Every link should be a real <a href> link

This is the section I feel most strongly about, because I've seen what it does.

Every link on your website, internal or external, should be a standard HTML anchor link:

<a href="https://yoursite.com/biryani/">Biryani</a>

<span onclick="goTo('/biryani/')">Biryani</span>
<button onclick="window.location='/biryani/'">Biryani</button>

If your links are JavaScript-based, they're invisible to Google and every other search engine. Crawlers won't follow them. And here's the dangerous part: any page that's only reachable through those links won't get discovered, crawled, or indexed.

Let me tell you how much this can matter.

I audited this at my agency. When I pointed it out, the fix was simple. No new content. No link building. Just making existing pages reachable. That's the kind of impact one technical fix can have when you're a big site with a lot of client-side links.

Use absolute URLs in your links

While you're at it, try to use absolute URLs in your href values instead of relative ones:

<a href="https://yoursite.com/biryani/">Biryani</a>

<a href="/biryani/">Biryani</a>

Search engines can resolve relative URLs, so this isn't about Google failing to follow them. It's about removing room for mistakes. With absolute URLs, every link points to exactly one version of the page (remember the golden rule in the URLs section below). A relative link inherits whatever version it was loaded on, so if your site is ever reachable on http://, a www variant, or a staging domain, relative links quietly spread that wrong version across every page. Scrapers that copy your content also end up linking back to you instead of to nowhere.

Internal linking and orphan pages

Your pages should be well linked to each other. Ideally, no page should be left orphaned.

An orphan page is a page that no other page on your website links to. It might be in your sitemap, it might not, but if nothing on your site links to it, crawlers have a hard time finding it and search engines assume it isn't very important.

So make sure your important pages are linked from other relevant pages on your site: your navigation, your category pages, related articles, wherever it makes sense.

Links that open in a new tab

If a link opens in a new tab (it has target="_blank"), add rel="noopener noreferrer":

<a href="https://example.com" target="_blank" rel="noopener noreferrer">Example</a>

This is a security best practice, not a ranking factor. Modern browsers already apply noopener by default, so it's mostly hygiene now. Just know that noreferrer also hides your site as the referrer, so the site you link to won't see your traffic in their analytics. If that matters to you (say, for partner sites), use only noopener.

URLs are how people access your content, so they're a very important part of technical SEO. Here's the one golden rule that everything else builds on:

Once you understand that, most URL problems start making sense.

Your website should load on only one URL version

Every page on your website should be accessible through exactly one URL.

The common versions that cause duplicates:

If your site loads on more than one of these, search engines see duplicate content: the same page on multiple URLs, competing with itself.

The rule of thumb: pick one version, and 301 redirect every other known variation to it. Then stay consistent across your whole site. Don't have some pages that use a trailing slash and others that don't. Same pattern, everywhere.

HTTPS, and the HSTS header

Your website should be on HTTPS, full stop. It's the standard, and most browsers will warn users away from sites still on HTTP.

On top of that, enable the HSTS header (HTTP Strict Transport Security). It tells browsers to always load your site over HTTPS, even if someone types or clicks an http:// link:

Strict-Transport-Security: max-age=31536000; includeSubDomains

Use clean URLs, and parameters only when you need them

Handling URL parameters can be a nightmare if it's done badly. Remember the golden rule: every added character creates a new URL. So these are three different URLs showing nearly the same content:

My advice: use clean, static URLs wherever possible, and keep parameters for tracking purposes only.

Where you genuinely need parameters (like filters on an e-commerce category page), use canonical tags. Every filtered or parameterized version should have a canonical tag pointing to the main version of the page, not to itself:

<!-- On /shoes/?color=red&size=9 -->
<link rel="canonical" href="https://yoursite.com/shoes/">

Internal search pages should be noindexed

If your website has a search bar, searching usually creates URLs like /search?q=whatever. Make sure those search result pages have a noindex meta robots tag (unless you've deliberately planned for specific search pages to be indexed).

This isn't just about duplicate content. It's a security issue.

A noindex on your search pages shuts that down. Use noindex on its own here, not combined with a canonical pointing somewhere else. Google has said that mixing the two sends conflicting signals.

A proper 404 page, with a real 404 status

Your website should not load a page for a URL that doesn't exist.

What happens? On roughly 20 to 30% of the sites I've audited, the site loads something (often the homepage or an empty template) with a 200 OK status code. To a search engine, that means every random URL is a real page.

What should happen instead:

Soft 404s

A soft 404 is when a page returns 200 OK, but there's basically nothing on it: an empty category, a product with no content, a search with no results.

These hurt more than people think, because they drag down the overall quality of your site. Say you have 10 high-quality pages and 5 soft 404s:

On average, your site looks a lot worse than it is. So find your soft 404s and handle them: add real content, return a proper 404 or 410, or noindex them. Either way, keep them out of the index.

Redirects

If you remove a page that's been around for a while, it has probably built up some authority (some people call it "link juice"): backlinks, rankings, history. You don't want to throw that away.

Redirect it to the most relevant, most similar page you have, using a 3xx redirect:

301 and 308 do the same job, as do 302 and 307. Use whichever your CMS or server supports. And if a page is gone for good, has no relevant replacement, and you want search engines to drop it from their index, use 410 Gone. It tells search engines this page isn't coming back.

Avoid redirect chains

A redirect chain is when URL A redirects to B, which redirects to C, which redirects to D. Every hop wastes crawl budget, slows down users, and can leak some of the authority you were trying to preserve. Crawlers may also give up before they reach the end. Worse are redirect loops, where A ends up redirecting back to A and the page never loads.

Chains usually build up over years of site migrations and URL changes. The fix:

• Redirect directly to the final URL in one hop
• Update your internal links to point at the final URL, so you aren't relying on redirects at all

People handle pagination in a lot of different ways, and the right approach depends on the size of your site.

What Google recommends: give every paginated page its own self-referencing canonical tag. So /category?page=2 canonicals to /category?page=2, not to page 1. Google's docs are explicit about this: don't use the first page as the canonical for the whole series. (And if you still have rel="next" and rel="prev" tags, Google stopped using those years ago.)

That works well for most sites. But for big websites with multiple category pages, each with dozens of paginated pages, the number of URLs can run into the millions. So big sites come up with their own strategies:

There's no single right answer for every site. If your site has thousands or lakhs of pages, you need a deliberate pagination strategy that doesn't waste crawl budget, while making sure your important pages still get crawled, indexed, and performing.

Core Web Vitals

Your pages should load fast and pass Core Web Vitals. Page experience is a confirmed ranking signal. It isn't a dominant one, but beyond rankings, faster pages simply keep more people around: the bounce rate drops drastically when your page loads fast.

The three Core Web Vitals and their thresholds:

How Google judges your site's speed

Google doesn't just test one page. It looks at real user data across your site, evaluated at the 75th percentile of page loads, and groups similar pages together.

If a group of your pages performs well, Google considers that part of your site fast. If a group performs poorly, it can drag down the whole section, not just one URL.

Mobile-friendly, always

Google's main crawler is Googlebot Smartphone. That means Google primarily sees and judges the mobile version of your website. That's what mobile-first indexing means.

So your website must be mobile-friendly, and it must load fast on mobile devices, not just on desktop. If content is missing from your mobile version, as far as Google is concerned, it's missing.

Finally, every important page on your website should have relevant structured data, in schema.org JSON-LD format.

Structured data gives search engines and AI crawlers the context of your page in a format they can read directly: what type of page it is, who wrote it, what product it's about, what the price is, what the FAQs are. They understand your page better and don't have to spend as many resources figuring it out. And when crawlers spend less effort per page, your crawl budget stretches further, so more of your pages get crawled and eventually indexed.

Let me be honest about what it doesn't do, though. Adding schema isn't a guaranteed ranking boost. Plenty of sites without schema outrank sites with it. But it's always better to make your content easier for crawlers to understand than harder.

And make sure the schema you add is valid. Broken or incomplete schema helps nobody. It should also follow Google's structured data guidelines (and those of other search engines): mark up only what's actually visible on the page, use the right types, and don't stuff in things that aren't true. Check it with Google's Rich Results Test and the Schema Markup Validator before you ship.

Watch out for double-escaped characters

One more thing on structured data, because it caught a lot of sites off guard. Google now unescapes JSON-LD only once. It used to keep decoding until nothing was left, which quietly cleaned up messy code. Now, if your CMS saves a value already escaped and your theme escapes it again on the way out, that extra layer stays.