Semantic Search: Vector Embeddings, NLP & Retrieval-Augmented Generation
Semantic search retrieves by meaning rather than exact token overlap, but it is most reliable as one stage in a hybrid system. Use it when judged queries show a vocabulary gap—users and documents describe the same concept differently—not as a blanket replacement for lexical search, filters, or access control.
Identify the query class that needs semantics
Exact identifiers, names, error codes, and quoted phrases usually favor lexical retrieval. Natural-language questions and conceptual descriptions may benefit from embeddings. Label a representative query set by class and measure where BM25 misses relevant documents before introducing vectors. The BEIR benchmark paper demonstrates why performance varies across retrieval domains; a model that wins one benchmark is not automatically right for a product corpus.
Build an explicit embedding pipeline
Choose the unit being embedded: whole document, section, paragraph, product, or support answer. Chunks need stable IDs linked to their source document, permissions, language, model name, model version, and content version. Re-embedding must be restartable, and index activation should be atomic so queries never mix incompatible vector dimensions.
Store hard filters beside vectors. Tenant, publication state, region, inventory, and authorization must be applied inside retrieval, not after snippets have been returned. Elasticsearch documents its k-nearest-neighbor search and filtering behavior, including approximate-search trade-offs.
Fuse lexical and vector candidates deliberately
Run lexical and vector retrieval independently, normalize or rank-fuse their candidate lists, and retain the component scores for explanation. Reciprocal rank fusion is a robust baseline because it does not assume comparable raw scores. Qdrant's hybrid-query documentation shows prefetch and fusion as explicit stages.
A cross-encoder or other reranker can improve the top results, but it adds latency and cannot recover a document absent from both candidate sets. Bound candidate depth and degrade to lexical retrieval if the embedding or reranking service is unavailable.
Evaluate retrieval before adding generated answers
Track recall at candidate depth, nDCG or judged precision near the top, latency percentiles, no-answer rate, and performance by query class and language. Review hard negatives—plausible but wrong matches—because vector similarity often overvalues topical resemblance.
Retrieval-augmented generation is a consumer of search results, not proof that retrieval works. Require source references, refuse answers when evidence is weak, and test prompt injection inside indexed documents. Keep the original passages visible. The most serious failures are unauthorized chunks, stale embeddings, model-version mixtures, and fluent answers built from the wrong source.
Published · Updated