Add hybrid search to an existing vector index using RRF

TL;DR — If you already have vector search, the fastest path to hybrid search is: keep your existing vector index, add a lexical/BM25 index over the same document IDs, query both, then fuse the ranked lists with reciprocal rank fusion (RRF). The most common failure is mismatched document IDs or filters between the two retrievers; fix that first before tuning weights or k. Reading time: ~5 min

Goal

When you finish, your application will issue one query that runs both vector and lexical retrieval against the same corpus, merges the two ranked result sets with reciprocal rank fusion, and returns a single ranked list where exact-term matches and semantic matches both surface correctly.

Prerequisites

• An existing vector index already serving queries in production or staging
• Access to the code path that performs retrieval and ranking
• A lexical search engine or DB feature you can query by document text, such as PostgreSQL full-text search or Elasticsearch/OpenSearch BM25
• A stable document primary key shared across both indexes, for example doc_id
• Python 3.11+ for the example scripts — check with:

python3 --version

• curl and jq installed for verification:

curl --version
jq --version

• If using PostgreSQL full-text search: PostgreSQL 14+ and permission to create indexes — check with:

psql --version

• Your existing vector query command or API endpoint, plus any auth token needed to call it
• A representative test query set: 5-10 real queries where vector-only currently misses exact keywords, codes, names, or acronyms

Steps

Step 1: Confirm both retrievers can address the same documents

Run these checks against your current vector store metadata source and your source-of-truth table. Replace names as needed.

psql "$DATABASE_URL" -c "SELECT COUNT(*) AS docs, COUNT(DISTINCT doc_id) AS distinct_docs FROM documents;"
psql "$DATABASE_URL" -c "SELECT doc_id FROM documents WHERE body IS NULL OR body = '' LIMIT 5;"

If your vector store keeps a metadata export with doc_id, sample it and compare IDs:

python3 - <<'PY'
import json, sys
sample = json.load(open('vector-metadata-sample.json'))
ids = [x['doc_id'] for x in sample[:5]]
print(ids)
PY

You should see matching, stable doc_id values in both systems and no large class of empty text bodies.

Step 2: Add a lexical index over the same corpus

If your corpus is already in PostgreSQL, add a generated tsvector column and a GIN index. Replace documents, title, and body with your actual column names.

⚠️ Creating the index can consume CPU and I/O on large tables. Run it during a low-traffic window, and use CONCURRENTLY for the index to reduce write blocking.

psql "$DATABASE_URL" <<'SQL'
ALTER TABLE documents
ADD COLUMN IF NOT EXISTS search_tsv tsvector
GENERATED ALWAYS AS (
  setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
  setweight(to_tsvector('english', coalesce(body, '')), 'B')
) STORED;
SQL

psql "$DATABASE_URL" -c "CREATE INDEX CONCURRENTLY IF NOT EXISTS documents_search_tsv_gin ON documents USING GIN (search_tsv);"

Quick smoke test:

psql "$DATABASE_URL" -c "SELECT doc_id, ts_rank_cd(search_tsv, websearch_to_tsquery('english', 'invoice retry code')) AS rank FROM documents WHERE search_tsv @@ websearch_to_tsquery('english', 'invoice retry code') ORDER BY rank DESC LIMIT 5;"

You should see 0 or more rows with doc_id and a numeric rank; no SQL error about missing column or operator class.

Step 3: Standardize retrieval outputs to doc_id + rank position

Your fusion layer should not depend on raw vector distance or BM25 score scales. Fetch top-N from each retriever and convert each result list to rank positions starting at 1.

Example vector API call shape:

curl -sS -X POST "$VECTOR_API_URL/query" \
  -H "Authorization: Bearer $VECTOR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"invoice retry code","top_k":20,"filter":{"tenant_id":"acme"}}' | jq .

Example lexical query with the same filter semantics:

psql "$DATABASE_URL" -P pager=off -c "SELECT doc_id FROM documents WHERE tenant_id = 'acme' AND search_tsv @@ websearch_to_tsquery('english', 'invoice retry code') ORDER BY ts_rank_cd(search_tsv, websearch_to_tsquery('english', 'invoice retry code')) DESC LIMIT 20;"

You should get two ranked lists of doc_id values for the same tenant/filter scope.

Step 4: Implement reciprocal rank fusion

Create a small fusion function. This example uses the standard RRF formula 1 / (k + rank) with k=60, which is a safe default for most corpora.

from collections import defaultdict

def rrf_fuse(result_lists, k=60, weights=None):
    """
    result_lists: list of ranked doc_id lists, e.g. [["d1","d2"], ["d2","d3"]]
    weights: optional list of per-retriever weights, same length as result_lists
    """
    if weights is None:
        weights = [1.0] * len(result_lists)
    scores = defaultdict(float)
    for weight, docs in zip(weights, result_lists):
        for rank, doc_id in enumerate(docs, start=1):
            scores[doc_id] += weight * (1.0 / (k + rank))
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

# example
vector_docs = ["d10", "d7", "d2", "d1"]
lexical_docs = ["d2", "d9", "d1", "d8"]
print(rrf_fuse([vector_docs, lexical_docs], k=60, weights=[1.0, 1.0])[:5])

You should see a sorted list of (doc_id, score) tuples, with documents appearing in both lists ranked above one-list-only ties.

Step 5: Wire hybrid retrieval into your app

Call both retrievers with the same query text, top-N, and filters, then fuse. Keep top-N per retriever larger than your final return size; 20 -> 10 is a good starting point.

def hybrid_search(query, tenant_id, final_k=10, per_retriever_k=20):
    vector_docs = vector_search(query=query, top_k=per_retriever_k, filter={"tenant_id": tenant_id})
    lexical_docs = lexical_search(query=query, top_k=per_retriever_k, tenant_id=tenant_id)

    # vector_search / lexical_search should each return ranked doc_id lists
    fused = rrf_fuse([vector_docs, lexical_docs], k=60, weights=[1.0, 1.0])
    top_doc_ids = [doc_id for doc_id, _ in fused[:final_k]]
    return hydrate_documents(top_doc_ids)

If lexical search is too dominant for code-like queries, try weights [1.0, 0.7] or [0.7, 1.0] instead of changing the RRF formula.

You should see your app return one merged result list without needing score normalization between retrievers.

Step 6: Add a reproducible local test

Create a script that prints vector-only, lexical-only, and fused rankings for the same query.

queries = [
    "invoice retry code",
    "SAML audience mismatch",
    "customer ACME-4421",
]

for q in queries:
    v = vector_search(q, top_k=10, filter={"tenant_id": "acme"})
    l = lexical_search(q, top_k=10, tenant_id="acme")
    f = [doc_id for doc_id, _ in rrf_fuse([v, l], k=60)[:10]]
    print({"query": q, "vector": v[:5], "lexical": l[:5], "fused": f[:5]})

You should see exact identifiers, acronyms, and quoted terms improve in fused compared with vector-only.

Verify it works

Run an end-to-end check with a query that previously failed in vector-only retrieval because of exact token dependence.

python3 hybrid_smoke_test.py

Expected output shape:

{'query': 'SAML audience mismatch', 'vector': ['d81', 'd44', 'd12', 'd77', 'd30'], 'lexical': ['d12', 'd91', 'd44', 'd18', 'd55'], 'fused': ['d12', 'd44', 'd81', 'd91', 'd18']}
{'query': 'customer ACME-4421', 'vector': ['d7', 'd88', 'd52', 'd19', 'd63'], 'lexical': ['d52', 'd101', 'd7', 'd3', 'd48'], 'fused': ['d52', 'd7', 'd101', 'd88', 'd3']}

Also verify that both retrievers are using identical filters. For PostgreSQL lexical search, inspect the exact filtered count:

psql "$DATABASE_URL" -c "SELECT COUNT(*) FROM documents WHERE tenant_id = 'acme' AND search_tsv @@ websearch_to_tsquery('english', 'SAML audience mismatch');"

For your vector API, log the outbound request body and confirm the same tenant_id filter and top_k are present. The hybrid implementation is correct when:

• both retrievers return doc_id values from the same corpus
• fused results contain documents from either retriever
• documents present in both lists move upward in the final ranking
• exact-term queries improve without obvious regressions on semantic queries

Common pitfalls

Different doc_id values between vector and lexical indexes

Mistake: the vector index stores chunk IDs while the lexical index returns parent document IDs.

Symptom: RRF output contains duplicates, or relevant lexical hits never merge with vector hits.

Fix: return the same canonical ID from both retrievers, or map chunk IDs to parent doc_id before fusion.

Filters applied to one retriever but not the other

Mistake: tenant, language, ACL, or date filters are only passed to vector search.

Symptom: fused results leak unauthorized docs or rank irrelevant old content unusually high.

Fix: pass the exact same filter set to both retrievers before fusion.

Using raw score averaging instead of rank fusion

Mistake: averaging cosine similarity with BM25/FTS scores directly.

Symptom: one retriever dominates because score scales are incompatible; tiny tuning changes swing results wildly.

Fix: fuse by rank position with RRF, not by raw score, unless you have calibrated score normalization.

Too-small candidate pools

Mistake: querying top_k=5 from each retriever and then expecting a strong top 10 fused list.

Symptom: fused results look brittle and miss obvious lexical or semantic candidates.

Fix: fetch at least 2x your final return size from each retriever, commonly 20 per retriever for a final 10.

PostgreSQL text search dictionary mismatch

Mistake: indexing with 'english' when your corpus is mostly identifiers, product names, or non-English text.

Symptom: lexical search drops important tokens or stems terms in unhelpful ways.

Fix: switch to 'simple' for identifier-heavy corpora, or create per-language indexes/queries instead of forcing 'english'.

Building the lexical index without planning for write load

Mistake: creating a large GIN index during peak traffic.

Symptom: elevated DB latency, replication lag, or deployment rollback pressure.

Fix: use CREATE INDEX CONCURRENTLY, run off-peak, and monitor DB CPU/IO until the build completes.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.