Technical SEO Checklist: Robots.txt, Sitemaps, Canonicals, Schema and Redirects Explained

Great content cannot rank if search engines cannot reach it, understand it or tell which version to show. Technical SEO is the plumbing that makes sure they can. None of it is glamorous, but a single stray line in robots.txt or a missing canonical tag can quietly cost you months of traffic.

How search engines see your site

Search engines work in three broad stages. First they crawl: automated programs follow links and sitemaps to discover pages and download them. Then they index: they analyse the page and store what they learn in a giant database. Finally they rank: when someone searches, they pick the most relevant indexed pages and order them.

Technical SEO mostly lives in the first two stages. If a page is blocked from crawling, it may never be understood. If it is crawlable but confusing, with duplicates, wrong signals or broken links, the engine may pick the wrong version or skip it. The rest of this checklist walks through the signals you control.

Robots.txt

Robots.txt is a plain text file at the root of your site, such as /robots.txt, that tells crawlers which paths they may fetch. It is a request that well-behaved bots follow, not a security measure. Never use it to hide private pages, because the file itself is public and a blocked URL can still appear in results if other sites link to it.

A simple, healthy file looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

• User-agent names the crawler the rules apply to. * means all.
• Disallow blocks a path prefix. An empty value allows everything.
• Allow creates an exception inside a blocked folder.
• Sitemap points to your sitemap and can appear anywhere in the file.

Robots.txt controls crawling, not indexing. To keep a page out of search results, use a noindex meta tag or header, and leave the page crawlable so the engine can see that tag. Blocking and noindexing the same page can have the opposite of the intended effect. Paste your file into the Robots.txt Syntax Validator to catch typos, unknown directives and conflicting rules.

XML sitemaps

An XML sitemap lists the URLs you want indexed, along with optional details such as the last modification date. It helps crawlers discover pages, especially new pages and those with few internal links. It does not guarantee indexing, but it is a clear statement of which URLs matter.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guide/</loc>
    <lastmod>2026-11-05</lastmod>
  </url>
</urlset>

Good sitemap habits:

• Include only canonical, indexable pages that return a 200 status. Leave out redirects, noindexed pages and error pages.
• Stay within the limits: a single sitemap file may hold up to 50,000 URLs and 50 MB uncompressed. Larger sites use a sitemap index that points to several files.
• Keep lastmod honest. Update it only when the content truly changes, or crawlers learn to ignore it.
• Reference the sitemap in robots.txt and submit it in the search engines’ webmaster tools.

Use the Sitemap XML Validator to confirm the structure is valid, URLs are absolute and the file is within the protocol limits.

Canonical tags and duplicate content

The same page can often be reached through several URLs: with and without www, with a trailing slash or without, with tracking parameters, or through print and sort variations. To a crawler these are separate pages with identical content. A canonical tag tells search engines which version is the preferred one:

<link rel="canonical" href="https://example.com/guide/">

Rules of thumb:

• Use absolute URLs, including the protocol.
• Give each page a self-referencing canonical unless it genuinely duplicates another page.
• Do not canonicalise to a URL that redirects, returns an error or is noindexed.
• Keep a single canonical per page. Conflicting tags are ignored.
• Treat it as a strong hint, not a command. Engines may choose another URL if other signals disagree.

The Canonical Tag Checker inspects the canonical in your HTML and flags relative URLs, duplicates and mismatches.

Title, description and meta tags

The title element is one of the strongest on-page signals and is usually the blue link people click. Make it descriptive, unique, and put the main topic early. Keep it short enough to avoid truncation, commonly under about 60 characters, though the limit is by pixel width and not a fixed number.

The meta description does not directly affect ranking, but a clear summary of 120 to 160 characters can improve clicks. Search engines often rewrite it, so make sure the page content also answers the query.

Other tags worth knowing:

• meta robots with noindex or nofollow to control indexing and link following per page.
• meta viewport for mobile-friendly layout.
• A single h1 that matches the page’s main topic, followed by logical h2 and h3 headings.

The Meta Tag Extractor pulls these elements from HTML so you can review titles, descriptions, robots directives and heading structure in one view. For counting characters and words in your own copy, use the Character Counter.

Open Graph and social previews

When someone shares your URL on a social platform or in a chat app, the preview card is built from Open Graph tags in your page head. They are not a ranking factor, but a strong preview can lift click-through from shares.

<meta property="og:title" content="Technical SEO Checklist">
<meta property="og:description" content="A practical guide to crawling, indexing and schema.">
<meta property="og:image" content="https://example.com/images/seo-checklist.jpg">
<meta property="og:url" content="https://example.com/seo-checklist/">
<meta property="og:type" content="article">

Use an image of around 1200 × 630 pixels, and keep important content away from the edges so it survives cropping. Preview your tags with the Open Graph Preview Generator, and prepare the right file with our guide to image formats and the Image Resizer.

Structured data (schema)

Structured data labels the content on a page in a machine-readable vocabulary, usually Schema.org, written as JSON-LD. It does not boost rankings directly, but it can make pages eligible for rich results such as FAQ-style snippets, ratings, breadcrumbs or product information, depending on what the search engine supports at the time.

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Technical SEO Checklist",
  "datePublished": "2026-11-05",
  "author": { "@type": "Organization", "name": "SZ Toolify" }
}

Follow these rules:

• Mark up only what is visible on the page. Hidden or misleading markup can lead to penalties.
• Pick the most specific type available and fill the required properties.
• Keep JSON-LD valid. A stray comma breaks the whole block.
• Check the search engine’s own documentation for which types still produce rich results, since those change over time.

The Schema JSON-LD Validator parses your block, finds syntax errors and shows missing common properties. If the JSON itself is the problem, the JSON Validator pinpoints the line.

Hreflang for multiple languages

If you publish the same content in several languages or for different regions, hreflang annotations tell search engines which version belongs to which audience:

<link rel="alternate" hreflang="en-us" href="https://example.com/en-us/">
<link rel="alternate" hreflang="en-gb" href="https://example.com/en-gb/">
<link rel="alternate" hreflang="x-default" href="https://example.com/">

The most common mistakes are missing return links (page A points to B but B does not point back), wrong language or region codes, and annotations pointing to redirected URLs. Every version must list all the others, including itself. The Hreflang Tag Validator checks codes and reciprocity.

Redirects and URL parameters

Redirects

When a URL changes, send visitors and crawlers to the new address with a 301 redirect (permanent). A 302 (temporary) says the move is not permanent. Guidelines:

• Redirect each old URL straight to its final destination. Chains of several hops waste crawl time and can lose signals.
• Redirect to the closest equivalent page, not blindly to the home page.
• Update internal links so they point at the final URL.
• Keep redirects in place for at least a year.

When you migrate a site, you will work from a spreadsheet of old and new addresses. The Redirect Mapping Formatter turns that list into rules for common server formats.

URL parameters

Tracking and filtering parameters, like ?utm_source=newsletter or ?sort=price, create endless URL variations of the same page. Handle them with canonical tags, consistent internal linking, and by stripping tracking tags from the links you publish. The URL Parameter Cleaner removes common tracking parameters from a list of links, and the URL Encoder and Decoder helps when parameters contain special characters. For clean, readable addresses on new pages, generate slugs with the Text to Slug Converter.

Internal links and site structure

Internal links spread authority around your site and tell crawlers what is related. Every important page should be reachable within a few clicks from the home page and linked from relevant content.

• Use descriptive anchor text like “Loan EMI Calculator”, not “click here”.
• Link new articles from older, related ones, and the other way round.
• Group content into clear topics, with hub pages that link to detailed guides. A hub-and-spoke structure makes relationships obvious.
• Find and fix broken links, since they waste crawl budget and frustrate readers.

To audit the links on any page, the HTML Link Extractor lists every anchor with its text and attributes. A good example of topic grouping in practice is how our own guides connect to tools: see the concrete calculation guide, the regex guide and the GPA guide, each linking to the tools it explains.

A 20-minute audit routine

Run this after each launch or major change:

1. Robots.txt: open /robots.txt, confirm it is not blocking the whole site, and validate the syntax.
2. Sitemap: make sure it loads, lists only canonical 200 pages and is referenced in robots.txt.
3. Status codes: spot-check key URLs. They should return 200, with old URLs returning a single 301 hop.
4. Canonicals: sample your top pages and check each is self-referencing and absolute.
5. Titles and descriptions: look for duplicates, missing tags and truncation.
6. Schema: test one page of each template for valid JSON-LD.
7. Mobile and speed: check the page loads fast on a phone, with compressed images and set dimensions.
8. Search console: review coverage and indexing reports for new errors, then fix the most common cause first.

Frequently asked questions

Related guides and tools

References

1. Google Search Central, “Introduction to robots.txt”. developers.google.com
2. Google Search Central, “Consolidate duplicate URLs” (canonicalization). developers.google.com
3. Sitemaps.org, the Sitemaps protocol (format and limits). sitemaps.org/protocol.html
4. Schema.org, vocabulary and documentation. schema.org
5. The Open Graph protocol. ogp.me

Search engines change how they treat signals and rich results over time. Treat this checklist as a foundation and confirm details in the current official documentation.