On-Page Technical SEO for Google: Crawl, Index, CWV and Schema

Google has to crawl, select, measure speed and read structured data before any ranking happens. This operator's checklist covers the on-page technical controls that give your content a fair chance to compete.

1. Crawling and indexing basics: robots.txt, crawl budget, and JavaScript

Nothing else in this article matters until Googlebot can fetch your page. robots.txt lives at the root of the domain (for example https://example.com/robots.txt) and tells crawlers which URLs they are allowed, or not allowed, to request. Treat it as an access-control hint, not a ranking signal and not a security layer. Two of the most common mistakes we see: blocking the site's own CSS and JavaScript, which quietly breaks rendering and therefore indexing; and believing a Disallow line by itself keeps a URL out of Google's results. It does not. A disallowed URL can still be indexed if it is linked from elsewhere — to remove it from the index you must pair the block with a noindex signal you can actually serve.

Crawl budget is the volume of URLs Googlebot will fetch from your site in a given window. By Google's own guidance this concept really only matters for large sites — those with millions of URLs or heavy parameter-driven duplication. If you run a few hundred or a few thousand pages, crawl budget is almost never your bottleneck; thin content and broken internal linking are. Where budget does bite, it is usually leaks: infinite sort/filter combinations, session IDs, faceted navigation with no canonical, and staging or test hosts left reachable.

How Googlebot treats JavaScript. Google does render JavaScript, through a two-wave process: it fetches the raw HTML, queues the page, and later renders it with a headless browser to read client-side content. In practice that render is asynchronous and not always complete, so content that only appears after JavaScript executes can be delayed or missed. The robust operational choice is to make the primary text and key links present in the first HTML response — server-side rendering, static generation, or inlining critical content — rather than depending on Google to execute and re-read your bundle.

2. Index coverage: canonical, noindex vs 404, and sitemaps

Once Google can crawl a page, it still has to decide whether to index it. The Pages report in Google Search Console (the successor to the older Index Coverage report) shows which URLs are indexed, excluded, or returning errors. Work from a specific URL inspection rather than the aggregate counts — the report tells you why a page is excluded, which is the only thing that helps you fix it.

The canonical tag tells Google which URL is the master copy of duplicate or near-duplicate content. Use it three ways: a self-referencing canonical on every preferred page; cross-domain canonicals when the same content is mirrored across domains; and canonicals that point parameterized URLs (utm tracking, session IDs, sorting and filtering) back to the clean, preferred version. Canonical is a strong hint rather than a directive, so make it correct — do not assume Google will guess the version you meant.

noindex versus 404. Send noindex when a page should remain reachable by users but out of the index — thank-you pages, internal search results, thin but live pages. Return a 404 (or 410 for permanently removed content) when the page is genuinely gone. The worst option is the soft-404: returning a 200 OK on deleted content, which wastes crawl and confuses the index. One rule people break constantly: never block a noindexed page in robots.txt. If Googlebot cannot crawl it, it cannot see the noindex tag and will keep indexing the page.

XML sitemaps list your important URLs so Google can discover them faster. Per Google's documentation, a sitemap does not guarantee crawling or indexing — it improves discovery and helps prioritize a large site's URLs. Keep it current, keep it free of noindexed or low-value URLs, and keep it readable at scale. Think of it as a map you hand Google, not a request it must grant.

3. Core Web Vitals: the thresholds, the causes, and what to fix first

Core Web Vitals are the real-user experience signals Google uses as a search ranking factor. The current set, with the official Google Core Web Vitals thresholds for the “good” end of the scale, is below. Interaction to Next Paint (INP) replaced First Input Delay in March 2024.

Prioritize from field data, not lab vibes. The Chrome User Experience Report (CrUX) is real-user field data — the population Google's ranking signals actually reflect. Lab tools such as Lighthouse and the lab section of PageSpeed Insights are synthetic, controlled measurements that are excellent for debugging but can disagree with what real visitors experience. Start where your CrUX data is genuinely poor, then fix the usual suspects: compress and serve responsive images with explicit width and height, defer or drop unnecessary third-party scripts (chats, ads, tag managers), and reserve space for ads and embedded media so nothing jumps after paint.

4. Structured data: pick the right Schema and keep it honest

Structured data is a shared vocabulary (Schema.org) that annotates your pages so machines can parse them. Pick types that match the page's job rather than marking everything on the page:

• Article for blog posts and news; Product (with offer, price and availability) on commerce pages; FAQPage for a genuine question-and-answer block;
• BreadcrumbList to show the navigation trail; Organization on the home or about page to anchor your entity.

Google's structured data guidelines are explicit that marked-up content must be visible on the page and match what users see. Do not inject hidden text or stuff attributes users cannot read — that is the kind of mismatch that triggers rich-result removal or manual action. Before you ship a template change, validate every affected page with Google's Rich Results Test, and re-run it after updates. Treat Schema as eligibility and clarity, not a direct ranking boost: it helps Google understand your entity and can qualify you for rich results, but it does not, by itself, move rankings.

5. hreflang: serving the right language and region

When the same content exists in more than one language or region, hreflang tells Google which version to serve. The link tag lives in the <head> (or can be delivered via HTTP headers or an XML sitemap) and pairs a language-region code with a URL:

Three rules separate working hreflang from broken hreflang. First, annotations must be bidirectional: if page A points to page B, B must point back to A, and every page must also reference itself. Second, x-default is your fallback for browsers or regions that do not match a specific variant — point it at a generic, international landing page. Third, hreflang URLs must be fully qualified and return 200, and the return links must match exactly. The most common failure is a one-way set of annotations that Google therefore ignores entirely. For large multilingual sites, delivering hreflang inside the XML sitemap is easier to maintain than head tags, and it composes naturally with the discovery work in section two.

The on-page technical layer is unglamorous but it is the gate: you can write excellent content and build strong links, but if Google cannot crawl the right page, read its speed, or tell your language variants apart, none of that work compounds. If you want a second pair of eyes on your setup, you can browse more Google SEO guidance in our Insights library, review how we run technical and visibility programs, or talk to our team about a crawl-and-index audit.