Search Engines Explained: Crawling, Indexing, Ranking & the AI Search Revolution
A search engine is a data pipeline, not a search box. It discovers documents, fetches them safely, converts them into an index, retrieves candidates for a query, and ranks those candidates under a latency budget. Treating those stages as separate contracts makes relevance problems diagnosable and lets each stage scale without hiding failures in the next one.
Discovery and fetching build the corpus
A crawler starts from seeds, feeds normalized URLs into a frontier, and schedules hosts according to politeness and capacity. It must apply the Robots Exclusion Protocol, but robots.txt is crawl guidance, not access control. Authentication and authorization still belong at the origin. Sitemaps provide additional canonical candidates through the Sitemaps protocol; they neither override robots rules nor guarantee indexing.
The fetcher records status, final URL, content type, caching headers, and a bounded body. Redirect loops, calendar traps, tracking parameters, duplicate paths, and unbounded faceted navigation can consume a frontier indefinitely. Normalize only transformations proven equivalent, retain redirect evidence, and fingerprint content so duplicate detection does not depend on URL spelling alone.
Indexing turns documents into retrieval structures
Indexing extracts fields, language, headings, links, timestamps, and access-control labels before analysis. Text analysis then tokenizes and normalizes terms, optionally adding stemming, synonyms, or character folding. An inverted index maps each term to matching documents and positions; doc values or columnar structures serve sorting and aggregation. Apache Lucene's core documentation is a useful reference implementation of these structures.
Every index needs an update model. Decide whether writes are append-only, replaced by stable document ID, or versioned with tombstones. Define when deletes become invisible and how a full rebuild is swapped atomically. Without those rules, a fast index can return stale, duplicated, or unauthorized documents.
Retrieval and ranking answer different questions
Retrieval asks which documents are plausible candidates; ranking asks which of them should appear first. Lexical retrieval such as BM25 is strong for identifiers and exact terminology. Vector retrieval broadens recall for semantic similarity. Filters enforce hard constraints. A second-stage reranker can use richer signals on a small candidate set, but it cannot recover a relevant document that retrieval never returned.
Evaluate stages independently with a versioned judgment set. Track recall at candidate depth, precision or nDCG near the top, zero-result rate, freshness lag, and p50/p95/p99 latency. Online clicks are useful but biased by position and presentation; pair them with human relevance judgments and task-completion evidence.
Operational failures define production quality
Test malformed documents, slow origins, partial shards, analyzer changes, queue saturation, and index rollbacks. Bound every fetch and query, expose stage-level metrics, and keep the last known-good index until a replacement passes validation. Google's original Brin and Page's search-engine architecture paper is historic, but its separation of crawling, indexing, and serving remains the durable design lesson.
Published · Updated