RAG retrieval returns wrong documents: query, embeddings, chunking

TL;DR — When retrieval returns the wrong documents, start by proving the query sent to the retriever is the one you think it is. If the query is correct, verify the same embedding model and preprocessing were used for both indexing and search; only then spend time on chunking, because bad chunk boundaries usually degrade relevance after query/embedding issues are excluded. Reading time: ~6 min

The scenario

It is Tuesday afternoon, you merged what looked like a harmless prompt and ingestion cleanup, and now your RAG endpoint is confidently citing documents that have nothing to do with the user’s question. The app is up, latency is normal, and there are no 500s, but support has pasted three examples where "refund policy" retrieves onboarding docs and a query for a specific error code returns marketing copy. You need to isolate whether the retriever is being asked the wrong thing, whether the vectors are incompatible, or whether your chunking made the right content impossible to retrieve.

Symptoms

• Users report semantically irrelevant top results even though the corpus definitely contains the answer.
• Retrieval debug logs show a query that differs from the user input, for example:

user_query="How do I rotate API keys?"
retrieval_query="account access troubleshooting"

• Sudden relevance drop after a deploy that changed prompt templates, query rewriting, ingestion, or embedding model config.
• Top-k scores are tightly clustered and low, for example:

[0.412, 0.409, 0.407, 0.401, 0.398]

• Vector DB returns results, but they are from the wrong topic, wrong tenant, or duplicated adjacent chunks.
• Embedding service or app logs show dimension/model mismatches, e.g.:

ValueError: expected embedding dimension 1536, got 3072

invalid input for vector column: expected 1024 dimensions

• After reindexing, retrieval quality changes dramatically without any application code changes.
• Long documents are never retrieved for narrow questions, or retrieved chunks start/end mid-sentence with missing context.

Likely causes

| Embedding mismatch: different model/version/preprocessing at index vs query time | Common | ```bash
psql "$DATABASE_URL" -c "select model_name, embedding_dim, count(*) from document_embeddings group by 1,2 order by 3 desc;"
``` |
| Chunking is too large/too small or overlap is wrong, so relevant facts are split or diluted | Common | ```bash
python scripts/inspect_chunks.py --doc-id DOC123 --show 10
``` |
| Metadata filter bug (tenant, language, doc_type) excludes the right documents | Common | ```bash
curl -s http://localhost:8080/debug/retrieval?q='refund policy' | jq '.filters'
``` |
| ANN index/search params are too aggressive, hiding relevant neighbors | Less common | ```bash
psql "$DATABASE_URL" -c "show hnsw.ef_search;"
``` |

## Step-by-step diagnosis
1. **Capture the exact retrieval request, not the user-facing prompt**
   Use your app’s debug endpoint or temporary request logging around the retriever call.
```bash
curl -s 'http://localhost:8080/debug/retrieval?q=How%20do%20I%20rotate%20API%20keys%3F' | jq '{user_query, retrieval_query, filters, top_k}'

This is your problem if retrieval_query is materially different from user_query without a good reason, or if filters unexpectedly include tenant/doc_type/language constraints. Jump to Fixes → Query rewriting/filtering sends the wrong retrieval query or filters out the right docs.

1. Run retrieval with rewriting disabled If you have a feature flag or env var, disable rewrite/hyde/expansion for one request. If not, call the vector search directly with the raw user query embedding.

RETRIEVAL_DISABLE_REWRITE=1 curl -s 'http://localhost:8080/search?q=How%20do%20I%20rotate%20API%20keys%3F' | jq '.results[:5][] | {score, doc_id, title}'

This is your problem if relevance improves immediately when rewrite is off. Jump to Fixes → Query rewriting/filtering sends the wrong retrieval query or filters out the right docs.

1. Confirm index-time and query-time embedding model names and dimensions match Check what is stored in the index metadata and what the app is configured to use for live queries.

printenv | grep -E 'EMBED|VECTOR|MODEL'
psql "$DATABASE_URL" -c "select model_name, embedding_dim, count(*) from document_embeddings group by 1,2 order by 3 desc;"

Example bad output:

QUERY_EMBEDDING_MODEL=text-embedding-3-large
INDEX_EMBEDDING_MODEL=text-embedding-3-small

       model_name        | embedding_dim | count
-------------------------+---------------+-------
 text-embedding-3-small  |          1536 | 84210
 text-embedding-3-large  |          3072 |   412

This is your problem if more than one active model/dimension appears for the same corpus, or env/config disagrees with the indexed majority. Jump to Fixes → Embedding mismatch: different model/version/preprocessing at index vs query time.

1. Check preprocessing parity for embeddings Compare the exact text sent to the embedder at ingest and at query time: lowercasing, HTML stripping, boilerplate removal, code fence handling, Unicode normalization.

curl -s http://localhost:8080/debug/embed-text?q='Error 0x80070005' | jq .

This is your problem if ingest strips tokens the query keeps (or vice versa), especially for code, IDs, SKUs, and error strings. Treat this as an embedding mismatch and jump to Fixes → Embedding mismatch: different model/version/preprocessing at index vs query time.

1. Inspect chunk boundaries for a known-bad document Pick a document that should obviously match the query and inspect its stored chunks.

python scripts/inspect_chunks.py --doc-id DOC123 --show 12

This is your problem if the answer sentence is split across chunks with no overlap, headers are separated from body text, or chunks are huge and contain multiple unrelated sections. Jump to Fixes → Chunking is too large/too small or overlap is wrong.

1. Measure chunk size distribution across the corpus Look for pathological chunk sizes after a parser or splitter change.

psql "$DATABASE_URL" -c "select percentile_cont(0.5) within group (order by token_count) as p50, percentile_cont(0.9) within group (order by token_count) as p90, min(token_count), max(token_count) from chunks;"

This is your problem if p90 is extremely high for your retriever context budget, or min/p50 indicate many tiny fragments. Jump to Fixes → Chunking is too large/too small or overlap is wrong.

1. Only after the above, check ANN/search params If query, embeddings, and chunking all look correct, compare approximate search to exact search on a sample.

psql "$DATABASE_URL" -c "show hnsw.ef_search;"

If increasing search breadth improves results, your ANN settings are too aggressive. Jump to Fixes → ANN index/search params are too aggressive.

Fixes

Query rewriting/filtering sends the wrong retrieval query or filters out the right docs

Disable rewriting temporarily, then narrow the rewrite prompt or remove it for exact-match-heavy domains like docs with error codes, API names, and SKUs.

export RETRIEVAL_DISABLE_REWRITE=1
export RETRIEVAL_DISABLE_HYDE=1
systemctl restart rag-api

If you apply metadata filters, log and validate them before the vector call.

{
  "retrieval": {
    "log_filters": true,
    "allowed_filter_keys": ["tenant_id", "language", "doc_type"],
    "reject_unknown_filters": true
  }
}

For exact identifiers, add a lexical fallback or hybrid search instead of relying on semantic rewrite.

{
  "retrieval": {
    "mode": "hybrid",
    "bm25_weight": 0.35,
    "vector_weight": 0.65
  }
}

Verify it worked:

curl -s 'http://localhost:8080/debug/retrieval?q=Error%200x80070005' | jq '{retrieval_query, filters, results: .results[:3]}'

Embedding mismatch: different model/version/preprocessing at index vs query time

Pin one embedding model for both ingestion and query paths and reindex any mixed corpus.

⚠️ Reindexing can consume significant CPU/IO and may temporarily degrade retrieval if you swap indexes in place. Build a new index/table and cut over after validation.

export EMBEDDING_MODEL=text-embedding-3-small
export EMBEDDING_DIM=1536
systemctl restart ingest-worker rag-api

If your schema tracks model metadata, isolate mixed embeddings and rebuild.

create table document_embeddings_new as
select * from document_embeddings where 1=0;

Re-embed from source content with the same preprocessing function used at query time.

python scripts/reembed_corpus.py --model text-embedding-3-small --dim 1536 --normalize unicode_nfc --strip-html --input s3://docs-bucket/export.jsonl --output-table document_embeddings_new

Swap tables or aliases after validating sample queries.

begin;
alter table document_embeddings rename to document_embeddings_old;
alter table document_embeddings_new rename to document_embeddings;
commit;

Verify it worked:

psql "$DATABASE_URL" -c "select model_name, embedding_dim, count(*) from document_embeddings group by 1,2;"

Chunking is too large/too small or overlap is wrong

Re-chunk with boundaries that preserve section semantics. For prose docs, start with 300-600 tokens and 50-100 token overlap; for API/reference docs, chunk by heading plus short body, not fixed size alone.

{
  "chunking": {
    "strategy": "heading_then_tokens",
    "target_tokens": 450,
    "max_tokens": 600,
    "overlap_tokens": 80,
    "split_on": ["h1", "h2", "h3", "paragraph", "list"],
    "preserve_code_blocks": true
  }
}

Rebuild chunks and embeddings from raw source, not from already chunked text.

python scripts/rechunk.py --config chunking.json --input ./export/docs.jsonl --output ./build/chunks.jsonl
python scripts/reembed_corpus.py --input ./build/chunks.jsonl --model "$EMBEDDING_MODEL" --dim "$EMBEDDING_DIM" --output-table document_embeddings_new

If duplication from overlap pollutes top-k, dedupe adjacent chunks at retrieval time.

{
  "retrieval": {
    "dedupe_on": ["doc_id", "chunk_window"],
    "max_chunks_per_doc": 3
  }
}

Verify it worked:

python scripts/inspect_chunks.py --doc-id DOC123 --show 12

Metadata filter bug (tenant, language, doc_type) excludes the right documents

Log the final filter object and reject null/empty values that accidentally become hard filters.

{
  "retrieval": {
    "drop_empty_filters": true,
    "reject_null_filter_values": true
  }
}

Test the same query with and without filters.

curl -s 'http://localhost:8080/search?q=refund%20policy&tenant_id=t1' | jq '.results[:3][] | .doc_id'
curl -s 'http://localhost:8080/search?q=refund%20policy' | jq '.results[:3][] | .doc_id'

Verify it worked:

curl -s 'http://localhost:8080/debug/retrieval?q=refund%20policy&tenant_id=t1' | jq '.filters'

ANN index/search params are too aggressive

Increase search breadth before rebuilding the index.

set hnsw.ef_search = 200;
show hnsw.ef_search;

If recall improves, persist a higher value in your DB/app config and benchmark latency. For IVF-style indexes, increase probes similarly in your engine’s supported setting. Verify it worked:

curl -s 'http://localhost:8080/search?q=How%20do%20I%20rotate%20API%20keys%3F' | jq '.results[:5][] | {score, doc_id}'

Prevention

• Add a golden-query regression suite in CI that runs against a fixture corpus and fails on top-k drift.

python tests/retrieval_eval.py --fixtures tests/fixtures/retrieval.json --min-recall-at-5 0.85

• Pin embedding model name, dimension, and preprocessing in one shared config imported by both ingestion and query services.

{
  "embedding": {
    "model": "text-embedding-3-small",
    "dimension": 1536,
    "preprocess_profile": "unicode_nfc_strip_html_preserve_code"
  }
}

• Store model_name, embedding_dim, chunker_version, and preprocess_profile with every chunk row; alert if more than one active combination exists.

select model_name, embedding_dim, chunker_version, preprocess_profile, count(*) from chunks group by 1,2,3,4 having count(*) > 0;

• Emit retrieval debug logs for sampled requests: raw query, rewritten query, filters, top-k doc IDs, and scores. Redact user secrets before logging.

{
  "logging": {
    "retrieval_sample_rate": 0.05,
    "fields": ["user_query", "retrieval_query", "filters", "topk_ids", "topk_scores"]
  }
}

• Add a chunk-size distribution check to ingestion CI/CD and fail if p50/p90 move outside your expected band.

python scripts/check_chunk_stats.py --input ./build/chunks.jsonl --p50-min 250 --p50-max 550 --p90-max 700

• Keep a direct-vector-search admin path that bypasses rewriting and app filters. It cuts incident time in half because you can isolate query construction from index quality immediately.

curl -s 'http://localhost:8080/admin/vector-search?q=Error%200x80070005&raw=1' | jq .

This article was written by an AI system and published pending human review. Verify anything you intend to act on.