Fast Search Engine: Sub-200ms Latency Techniques
Fast search is search that meets a task-specific latency budget under realistic load while preserving relevance and freshness. “Sub-200 ms” is not a universal requirement: an autocomplete request, a local documentation query, and a multi-stage enterprise search have different budgets. Define the budget at the user-visible boundary, then allocate it across the stages you can measure.
Turn a speed goal into a latency budget
Measure from the client initiating the request until usable results are rendered. Split that time into network, queue, query parsing, candidate retrieval, reranking, snippet generation, serialization, and rendering. Record p50, p95, and p99 separately; an average conceals queueing and cold-start failures. Segment cold and warm caches, query classes, result counts, tenants, and index versions.
Instrument boundaries with traces and consistent attributes. The OpenTelemetry semantic conventions provide a common vocabulary, while Elasticsearch's Profile API can explain expensive query execution during diagnosis. Profiling itself has overhead and should not be enabled indiscriminately in production.
Remove work before adding capacity
Normalize the query once, apply selective filters early, retrieve a bounded candidate set, and run expensive scoring only on those candidates. Avoid leading wildcards, unbounded aggregations, deep offset pagination, and scripts in the hot path. Precompute stable fields and use search-after or cursor pagination when users can traverse deep result sets.
Cache immutable index segments and repeated public queries, but include tenant, locale, permissions, filters, and index version in cache keys. Never cache a privileged result under a key that an unprivileged user can reuse. Keep a result-count cap so a broad query cannot turn snippet generation or faceting into unlimited work.
Control queueing and overload
Search latency often fails before CPU reaches 100% because requests wait behind expensive neighbors. Bound concurrency and queue depth, assign deadlines, and cancel downstream work when the client has gone away. Reserve capacity for interactive queries separately from exports, analytics, and index maintenance. Return a clear overload response or a reduced feature set rather than letting every request time out.
Load tests need representative query distributions, not one cache-friendly term. Grafana k6 documents thresholds for percentile and error-rate assertions. Include rare filters, zero-result queries, long terms, authorization checks, and concurrent indexing.
Protect relevance and freshness during optimization
Every optimization runs against a versioned judged-query set. Compare top-result relevance, candidate recall, zero-result rate, and freshness lag alongside latency. Test node loss, cold restart, cache stampede, slow shards, and a full index swap. A faster result is an improvement only if it is still the right result and does not expose stale or unauthorized data.
Published · Updated