Retrieval System Design: Beyond the Basics

The “chunk the PDF, embed it, stuff the top 3 chunks into the prompt” pipeline has a well-defined ceiling.

It works in demos. It works on clean, well-structured documents with clear semantic content. It fails in production when users search for exact product codes, regulatory identifiers, or error strings that do not appear in semantic neighbors. It fails when the relevant passage uses different vocabulary from the query. It fails when the most important document is the 12th most similar by cosine distance. It fails when the same query needs different ranking signals depending on context — the marketing copy and the technical specification may both be semantically relevant, but only one answers the user’s actual question.

Pure vector search fails on the queries that matter most in production: exact identifiers, codes, error strings, product names, numbers. In 2026, hybrid search is the production standard and pure vector search is increasingly the anti-pattern.

This article covers the retrieval architectures that address these production failure modes: the hybrid retrieval pipeline that combines dense and sparse signals, Reciprocal Rank Fusion as the robust merge strategy, BGE-M3 as the single model that produces all three representation types, late interaction retrieval with ColBERT for fine-grained token-level matching, learned sparse retrieval with SPLADE, cross-encoder reranking for precision at the top of the list, multi-vector indexing, query transformation techniques, and the decision framework for choosing between them in production.

Why the Basics Fail

Standard dense retrieval (embed query, find k-nearest neighbors by cosine similarity) has three structural weaknesses:

Vocabulary mismatch under semantic similarity. A query for “myocardial infarction treatment protocol” may retrieve documents about general cardiac care rather than the specific clinical guideline, because the semantic embedding of the query is closer to general cardiology content than to the specific document whose title uses the same medical terminology.

Exact-match failure. Dense embeddings average token-level semantics into a single vector. This process loses exact lexical information. A query for “PROC-7741-B” retrieves semantically similar process identifiers rather than the exact document. A query for “TypeError: cannot read property ‘map’ of undefined” retrieves semantically similar error pages rather than the specific thread. Exact identifiers, codes, and strings require lexical retrieval — not semantic retrieval.

Single-vector compression loss. Representing an entire document passage as a single embedding vector compresses all the semantic content into one point in embedding space. When the query is short and the relevant passage is long, the single-vector similarity score blends relevant and irrelevant content. The most relevant sentence in a paragraph may be buried inside a dense embedding that does not rank first.

Each of these failure modes has a corresponding retrieval architecture that addresses it.

The Production Standard: Hybrid Retrieval + RRF

Adding BM25 (or a learned sparse representation from BGE-M3 or SPLADE) recovers lexical precision without losing semantic recall. Every major vector database — Qdrant, Weaviate, Pinecone, Elasticsearch, pgvector — supports hybrid natively as of 2026. The extra complexity is modest; the quality uplift is 10–20% on mixed query sets.

BM25: The Lexical Baseline

BM25 (Best Match 25) is a probabilistic ranking function that scores documents based on term frequency and inverse document frequency. It is purely lexical — it rewards exact term matches and penalizes common terms. It does not understand semantics: a query for “vehicle” will not retrieve documents about “cars” unless the documents also contain the word “vehicle.”

BM25 excels precisely where dense retrieval fails: exact identifiers, rare terms, codes, product names, and technical strings. It is fast (inverted index lookup), cheap (no GPU required), and deterministic (same query always returns the same results).

Reciprocal Rank Fusion: The Robust Merge

When combining BM25 and dense retrieval ranked lists, the merge function matters. Simple score fusion fails because BM25 scores and cosine similarity scores are on different scales with different distributions — a score of 0.85 means something different from BM25 and from dense retrieval, and they cannot be directly compared or weighted.

Reciprocal Rank Fusion (RRF) is a simple, score-independent way to merge ranked lists. RRF combines ranked lists from multiple retrieval systems without requiring score calibration between heterogeneous retrievers.

def reciprocal_rank_fusion(
    ranked_lists: list[list[str]],  # Each list is doc IDs in rank order
    k: int = 60                     # RRF constant — 60 is the standard default
) -> list[tuple[str, float]]:
    """
    Merge multiple ranked lists using Reciprocal Rank Fusion.
    RRF score = sum over all lists of 1 / (k + rank_in_list)
    Higher score = better. No score calibration required.
    """
    scores: dict[str, float] = {}

    for ranked_list in ranked_lists:
        for rank, doc_id in enumerate(ranked_list, start=1):
            if doc_id not in scores:
                scores[doc_id] = 0.0
            scores[doc_id] += 1.0 / (k + rank)

    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

RRF is preferred over weighted score fusion because:

• No calibration required — works correctly regardless of score scale differences between retrievers
• Robust to individual retriever quality fluctuations — a single retriever returning an anomalously low score does not pollute the merge
• Proven in TREC evaluations to outperform more sophisticated merge strategies in practice

BGE-M3: One Model, Three Representations

BGE-M3 by BAAI produces three representations of a text in one pass: a dense vector with 1,024 dimensions, learned term weights as a sparse vector (similar to SPLADE), and ColBERT-style multi-vectors. One model serves both retrieval paths and the pipeline stays lean.

from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)

def encode_document(text: str) -> dict:
    output = model.encode(
        [text],
        return_dense=True,    # 1024-dim dense vector
        return_sparse=True,   # Learned sparse weights (BM25-compatible)
        return_colbert_vecs=True  # Per-token ColBERT vectors
    )
    return {
        "dense": output["dense_vecs"][0],         # float32[1024]
        "sparse": output["lexical_weights"][0],    # dict[token_id, weight]
        "colbert": output["colbert_vecs"][0]       # float32[n_tokens, 128]
    }

BGE-M3 is the practical default for most production hybrid retrieval systems because it eliminates the need to maintain separate embedding models for dense and sparse retrieval, reducing index complexity and eliminating embedding inconsistency.

The three-representation hybrid pipeline:

class BGE_M3_HybridRetriever:

    async def retrieve(
        self, query: str, top_k_candidates: int = 50
    ) -> list[RankedDocument]:

        # Encode query — all three representations in one pass
        query_repr = self.model.encode(
            [query],
            return_dense=True,
            return_sparse=True,
            return_colbert_vecs=True
        )

        # Stage 1: Dense retrieval (ANN search)
        dense_results = await self.vector_store.search_dense(
            vector=query_repr["dense_vecs"][0],
            top_k=top_k_candidates
        )

        # Stage 2: Sparse retrieval (inverted index)
        sparse_results = await self.vector_store.search_sparse(
            weights=query_repr["lexical_weights"][0],
            top_k=top_k_candidates
        )

        # Stage 3: Merge with RRF
        fused = reciprocal_rank_fusion(
            [dense_results.doc_ids, sparse_results.doc_ids],
            k=60
        )

        # Return top candidates for optional downstream reranking
        return fused[:top_k_candidates]

Late Interaction Retrieval: ColBERT

ColBERT (Contextualized Late Interaction over BERT) is the production late interaction architecture. Rather than compressing a document into a single embedding vector, ColBERT preserves per-token embeddings for both query and document, computing similarity at query time by matching each query token to its most similar document token.

How MaxSim Works

Query:    "treatment protocol myocardial infarction"
          [q1]  [q2]      [q3]        [q4]

Document: "The standard STEMI protocol includes..."
          [d1]  [d2]      [d3]     [d4]    [d5]

ColBERT score = sum over each query token of:
                max(cosine_similarity(qi, dj)) for all document tokens dj

For each query token, find the document token most similar to it.
Sum these maximum similarities across all query tokens.

This MaxSim operation allows each query token to find its best match in the document independently — “myocardial” matches “STEMI” even if “treatment” and “protocol” match weaker document tokens. The result is substantially better retrieval on medical, legal, and technical text where single-vector compression loses the specific token-level relevance signal.

ColBERT variants achieve robust retrieval performance across low-resource languages and specialized domains while maintaining parameter efficiency. ColBERTv2 adds residual compression to reduce the storage cost of multi-vector indexing. The PLAID engine accelerates late-interaction retrieval through centroid-based pruning.

The Storage Trade-off

ColBERT stores one 128-dimensional vector per token per document. A document with 512 tokens requires 512 × 128 × 4 bytes = 262KB of index storage — roughly 25–50× more than a single-vector dense index for the same document.

For high-accuracy retrieval on specialized domains where the performance gain justifies the storage cost, this trade-off is worthwhile. For general retrieval at massive scale, token pooling reduces the ColBERT footprint significantly with minimal performance impact:

Token pooling reduces the footprint of multi-vector retrieval with minimal performance impact by pooling adjacent token embeddings, reducing the number of stored vectors per document from ~512 to ~64 with minor accuracy loss.

def token_pool(
    token_embeddings: np.ndarray,  # shape: [n_tokens, dim]
    pool_size: int = 8             # Pool every 8 tokens into 1
) -> np.ndarray:
    """
    Reduce ColBERT storage by averaging adjacent token embeddings.
    n_tokens → n_tokens // pool_size vectors.
    Typically retains >95% of retrieval quality at pool_size=8.
    """
    n_tokens, dim = token_embeddings.shape
    padded_len = ((n_tokens + pool_size - 1) // pool_size) * pool_size
    padded = np.pad(token_embeddings, ((0, padded_len - n_tokens), (0, 0)))
    pooled = padded.reshape(-1, pool_size, dim).mean(axis=1)
    return pooled

When to Use ColBERT

ColBERT is the right choice when:

• The domain has specialized vocabulary requiring fine-grained token-level matching (medical, legal, scientific)
• Retrieval quality is the primary constraint and storage/cost is secondary
• The query distribution has significant vocabulary mismatch with document vocabulary
• Multilingual retrieval is required (ColBERT’s per-token representations generalize better across languages)

ColBERT is not the right choice when:

• Storage cost is a primary constraint (25–50× more than dense)
• Latency is tight and GPU resources are limited
• The corpus is very large (billions of documents) where the storage cost becomes prohibitive

Learned Sparse Retrieval: SPLADE

SPLADE (Sparse Lexical and Expansion Model) combines the interpretability and exact-match capability of BM25 with the semantic understanding of neural models. Rather than computing term frequencies over raw vocabulary, SPLADE learns a sparse representation that expands query and document vocabulary with semantically related terms.

Get Navaneethan L’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

A SPLADE representation of “car” might include high weights for {“car”: 2.1, “vehicle”: 1.4, “automobile”: 1.2, “sedan”: 0.8} — capturing semantic synonyms in a format compatible with an inverted index.

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

class SPLADEEncoder:
    def __init__(self, model_name: str = "naver/splade-v3"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForMaskedLM.from_pretrained(model_name)

    def encode(self, text: str) -> dict[str, float]:
        """
        Encode text to a sparse vector over the full vocabulary.
        Returns: dict mapping token_id → weight for non-zero terms.
        """
        inputs = self.tokenizer(
            text, return_tensors="pt", max_length=512, truncation=True
        )
        with torch.no_grad():
            outputs = self.model(**inputs)

        # SPLADE pooling: max over token positions, RELU for sparsity
        logits = outputs.logits  # [1, seq_len, vocab_size]
        weights = torch.log(1 + torch.relu(logits)).max(dim=1).values.squeeze(0)

        # Return only non-zero terms as sparse representation
        non_zero_mask = weights > 0
        tokens = self.tokenizer.convert_ids_to_tokens(
            non_zero_mask.nonzero(as_tuple=True)[0].tolist()
        )
        scores = weights[non_zero_mask].tolist()
        return dict(zip(tokens, scores))

SPLADE vs BM25:

SPLADE occupies the middle ground between BM25 and dense retrieval — better semantic coverage than BM25, lower storage than dense retrieval. It is particularly valuable as the sparse component of a hybrid pipeline when vocabulary expansion matters.

Cross-Encoder Reranking: Precision at the Top

The two-stage retrieval architecture is the production standard for accuracy-critical systems:

1. Stage 1 — Fast candidate retrieval: hybrid retrieval (dense + sparse → RRF) returns the top 50–100 candidates. Fast, parallelizable, but approximate.
2. Stage 2 — Slow precise reranking: a cross-encoder scores each of the top 50 candidates against the full query. Slow, serial, but highly accurate.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder(
    "BAAI/bge-reranker-v2-m3",  # 570M params, multilingual, Apache 2.0
    max_length=512
)

async def two_stage_retrieve(
    query: str,
    top_k_candidates: int = 50,
    top_k_final: int = 5
) -> list[Document]:

    # Stage 1: Hybrid retrieval — fast, broad
    candidates = await hybrid_retriever.retrieve(
        query, top_k=top_k_candidates
    )

    # Stage 2: Cross-encoder reranking — slow, precise
    pairs = [[query, doc.text] for doc in candidates]
    scores = reranker.predict(pairs, show_progress_bar=False)

    # Sort by cross-encoder score, return top-k
    reranked = sorted(
        zip(candidates, scores),
        key=lambda x: x[1],
        reverse=True
    )

    # Optional: apply confidence threshold
    # Documents below threshold are too irrelevant to include
    CONFIDENCE_THRESHOLD = 0.3
    filtered = [
        (doc, score) for doc, score in reranked
        if score > CONFIDENCE_THRESHOLD
    ]

    return [doc for doc, _ in filtered[:top_k_final]]

Why Cross-Encoders Are More Accurate

A bi-encoder (standard dense retrieval) encodes query and document independently, then compares their vectors. The query and document never interact during encoding — the model cannot attend to how specific query tokens relate to specific document tokens.

A cross-encoder receives the full concatenation of query and document as a single input sequence. The transformer attention mechanism operates over both simultaneously — “treatment” in the query can directly attend to “STEMI protocol” in the document. This joint encoding is what produces substantially higher precision, at the cost of no precomputation (every candidate must be scored at query time).

A cross-encoder processes the pair jointly: query tokens and document tokens together through the transformer. This recognizes whether a passage actually answers the question or merely uses the same terms.

Reranker model selection:

For most production RAG systems, BGE-reranker-v2-m3 provides the best accuracy-latency trade-off, especially if content is multilingual. The reranker model does not need to match the embedding model — using BGE-M3 for embedding and BGE-reranker-v2-gemma for reranking is a valid and high-performing combination.

Query Transformation Techniques

Before retrieval, transforming the query can significantly improve recall for difficult queries.

Query Rewriting

A poorly-formed user query often fails to retrieve the right documents not because the documents are absent but because the query vocabulary does not match the document vocabulary. Query rewriting uses an LLM to produce a better-formed retrieval query.

QUERY_REWRITE_PROMPT = """
Rewrite the following user question as a precise retrieval query.
The retrieval query will be used to search a corporate knowledge base.

Rules:
- Expand abbreviations
- Add relevant technical synonyms
- Remove conversational filler
- Keep the query concise (under 50 words)
- Preserve specific identifiers, product names, and codes exactly

User question: {query}

Return only the rewritten query, nothing else.
"""

async def rewrite_query(query: str) -> str:
    rewritten = await llm.ainvoke(
        QUERY_REWRITE_PROMPT.format(query=query)
    )
    return rewritten.strip()

HyDE — Hypothetical Document Embeddings

HyDE is a retrieval technique where instead of embedding the query directly, an LLM generates a hypothetical document that would answer the query, and that hypothetical document is embedded for retrieval.

HYDE_PROMPT = """
Generate a short document passage that directly answers this question.
Write as if you are the document, not answering the question.
The passage should be 2-4 sentences, factual in tone.

Question: {query}

Document passage:
"""

async def hyde_retrieve(
    query: str, retriever: HybridRetriever
) -> list[Document]:
    # Generate hypothetical document
    hypothetical_doc = await llm.ainvoke(
        HYDE_PROMPT.format(query=query)
    )

    # Embed the hypothetical document (not the query)
    results = await retriever.retrieve(hypothetical_doc)
    return results

HyDE is particularly effective for queries where the user’s phrasing is very different from document phrasing — the hypothetical document is likely to use the vocabulary that real documents use, bridging the gap.

Multi-Query Retrieval and Fusion

For complex, multi-faceted queries, decompose the query into sub-queries, retrieve independently for each, and fuse the results.

async def multi_query_retrieve(
    query: str, retriever: HybridRetriever, n_queries: int = 3
) -> list[Document]:
    # Decompose into sub-queries
    sub_queries = await decompose_query(query, n=n_queries)

    # Retrieve for each sub-query independently
    all_results = []
    for sub_q in sub_queries:
        results = await retriever.retrieve(sub_q, top_k=30)
        all_results.append([r.doc_id for r in results])

    # Fuse with RRF
    fused = reciprocal_rank_fusion(all_results)
    doc_ids = [doc_id for doc_id, _ in fused[:50]]
    return await retriever.fetch_by_ids(doc_ids)

The Full Production Pipeline

The complete production retrieval pipeline for a high-accuracy enterprise RAG system:

                        USER QUERY
                            │
                    ┌───────▼────────┐
                    │ Query Transform │  ← Rewrite + HyDE (optional)
                    └───────┬────────┘
                            │
              ┌─────────────┼────────────┐
              │             │            │
        ┌─────▼────┐  ┌────▼─────┐ ┌───▼───────┐
        │  Dense   │  │  Sparse  │ │  ColBERT  │
        │ (BGE-M3) │  │(SPLADE or│ │(if domain │
        │  ANN     │  │  BM25)   │ │ requires) │
        └─────┬────┘  └────┬─────┘ └───┬───────┘
              │             │            │
              └─────────────▼────────────┘
                     Reciprocal Rank Fusion
                    (top 50–100 candidates)
                            │
                    ┌───────▼────────┐
                    │ Cross-Encoder  │  ← BGE-reranker-v2-m3
                    │  Reranking     │    Score all 50 candidates
                    └───────┬────────┘
                            │
                    ┌───────▼────────┐
                    │ Confidence     │  ← Filter below threshold
                    │  Threshold     │    (source_quality signal)
                    └───────┬────────┘
                            │
                    ┌───────▼────────┐
                    │   Top 3–5      │  → LLM generation
                    │  Documents     │
                    └────────────────┘

Retrieval Architecture Selection

Indexing and Infrastructure

Vector Database Selection

Index Freshness

The retrieval index becomes stale as the document corpus changes. Index freshness strategy:

For low-update corpora (weekly or less): full re-index nightly. Simple, reliable, no incremental complexity.

For high-update corpora (daily or more): incremental indexing with a real-time update queue. New and updated documents are processed immediately; a background process handles bulk ingestion of daily changes.

class IncrementalIndexer:

    async def handle_document_event(self, event: DocumentEvent) -> None:
        if event.type == DocumentEventType.CREATED:
            await self.index_document(event.document)
        elif event.type == DocumentEventType.UPDATED:
            await self.delete_from_index(event.document.id)
            await self.index_document(event.document)
        elif event.type == DocumentEventType.DELETED:
            await self.delete_from_index(event.document.id)

Retrieval Evaluation

The retrieval layer requires its own evaluation metrics, independent of generation quality:

• Recall@K: what fraction of relevant documents are in the top-K retrieved? The primary recall metric.
• MRR (Mean Reciprocal Rank): how high does the first relevant document rank? The primary ranking quality metric.
• NDCG@K: normalized discounted cumulative gain — accounts for graded relevance, not just binary.
• Context precision: of the K retrieved documents, what fraction are actually relevant?

Retrieval eval requires a labeled evaluation dataset — query-document pairs labeled for relevance by domain experts. This is expensive to build but essential: it is the only way to distinguish a retrieval configuration that works from one that appears to work.

Trade-offs Worth Naming

Two-stage pipelines add latency. Hybrid retrieval is slower than dense-only retrieval (two index lookups instead of one). Cross-encoder reranking adds another 50–200ms per query for scoring 50 candidates. Total pipeline latency for a production hybrid + reranker system is typically 200–500ms for the retrieval stage alone. This is acceptable for document-level retrieval but may require caching or approximation for sub-100ms latency requirements.

ColBERT storage costs are real. At 25–50× the storage of a single-vector dense index, ColBERT is prohibitive for very large corpora (billions of documents). Token pooling reduces this to 3–5× with minimal quality loss. For most enterprise corpora (millions of documents), the storage cost is manageable.

Reranker confidence scores are not calibrated probability. A reranker score of 0.7 does not mean “70% confident this document is relevant.” Cross-encoder scores need to be calibrated (via sigmoid normalization or isotonic regression on labeled data) before they can be used as probabilistic confidence signals.

SPLADE requires inference at query time. Unlike BM25 (pure lookup), SPLADE runs a transformer inference pass for each query, adding 10–50ms per query. This is acceptable for most enterprise RAG systems but rules out SPLADE as the only retrieval method for sub-10ms latency requirements.

Architect’s Takeaways

• Pure vector search is the anti-pattern in 2026. Hybrid retrieval (dense + sparse + RRF) is the production default.
• Hybrid retrieval provides 10–20% quality uplift on mixed query sets over vector-only, and every major vector database supports it natively.
• Use BGE-M3 as the default embedding model — one inference pass produces dense, sparse (SPLADE-compatible), and ColBERT multi-vector representations, eliminating the need for separate models per retrieval path.
• RRF is the robust merge strategy for combining ranked lists from multiple retrievers — it requires no score calibration and is robust to individual retriever quality fluctuations.
• Cross-encoder reranking is not a replacement for hybrid retrieval — it is an addition. Hybrid fixes recall; reranking fixes precision. The full pipeline does both: hybrid top-50, cross-encoder top-5.
• BGE-reranker-v2-m3 provides the best accuracy-latency trade-off for most production systems, especially multilingual content. The reranker model does not need to match the embedding model.
• ColBERT is the right choice for specialized domains requiring token-level matching precision. Token pooling at pool_size=8 reduces storage to 3–5× dense while retaining >95% quality.
• Query transformation (rewriting, HyDE, multi-query) improves recall for difficult queries without changing the underlying retrieval architecture.
• Retrieval evaluation requires labeled query-document relevance datasets, independent of generation quality evaluation. Recall@K and MRR are the primary retrieval metrics.