Web Crawlers: How Googlebot and Spiders Index Content
A production web crawler is a scheduler with strict safety boundaries. It repeatedly chooses an allowed URL, fetches a bounded response, extracts durable evidence, discovers new candidates, and records enough state to resume without starting over. HTML parsing is the easy part; URL identity, politeness, traps, and failure recovery determine whether the crawler can run unattended.
Specify the crawl contract before fetching
Define allowed schemes, hosts, ports, path scope, content types, body limits, redirect depth, request timeout, concurrency, and user agent. Resolve and validate every redirect target to prevent server-side request forgery. Treat authentication as an explicit crawl profile; never forward credentials to a different origin.
Fetch and apply robots.txt according to RFC 9309. Cache it for a bounded period and fail according to a documented policy when it is unavailable. Robots rules do not authorize access and do not prevent a URL from being indexed from other evidence.
Make the frontier fair and finite
Store canonicalized URL candidates in a durable frontier partitioned by origin. Schedule per-origin delays so one host cannot monopolize workers. Normalize host casing and default ports, remove fragments, and resolve relative links with the WHATWG URL Standard. Do not blindly sort or delete query parameters: parameter order and presence can change application semantics.
Control traps with maximum depth, per-origin URL budgets, repeated-path detection, and rules for calendars, faceted navigation, session IDs, and infinite pagination. A content hash catches duplicates reached through different URLs; a URL key alone does not.
Extract evidence without executing everything
For each response, retain status, redirect chain, final URL, content type, cache validators, canonical, robots meta, language, title, headings, links, and a content fingerprint. Parse static HTML first. Use a browser renderer only for pages where meaningful content or links are absent, because rendering expands the security boundary and resource cost.
Sitemaps are seed sources, not permission or truth. Validate sitemap URLs against scope and feed them into the same deduplication and scheduling path. The protocol defines URL and sitemap-index limits.
Design restart and failure behavior
Persist leases so a crashed worker releases unfinished URLs, retry only transient classes with capped exponential backoff, and send permanent failures to an inspectable dead-letter state. Honor HTTP cache validators and revisit policy instead of downloading unchanged bodies; the semantics are defined in RFC 9111.
Test redirect loops, DNS rebinding, oversized bodies, decompression bombs, malformed markup, robots changes, slow responses, duplicate pages, and restart recovery. Operational metrics should expose frontier size and age, fetch class, bytes, extraction yield, duplicates, per-origin rate, retries, and freshness lag.
Published · Updated