Why does retrieval return confidently wrong but similar documents?

What does this look like in production?

• Retrieval confidently returns a document about a similar-sounding product, plan tier, or account instead of the one the user actually asked about
• A query containing a specific ID, date, or code returns topically related but factually wrong matches
• A negated statement ("does not support X") and its positive counterpart ("supports X") both score highly against the same query
• Keyword search on the same query would have found the right document immediately, but the vector searchVector search finds the nearest matches to a query by comparing embeddings, not exact keywords. buried it

Why does it happen?

Embedding models are trained to place semantically similar text close together in vector space, and they do that well for topic and meaning. They do not reliably encode the discrete facts that decide a match. Examples are an exact identifier or a specific date. A negation that flips a claim is another. Two passages can share a topic but disagree on the one detail that matters. They still end up close together. The embedding space rewards resemblance and ignores correctness on that detail.

This is invisible in a demo because demo queries are topical, such as "how do I reset my password". Real queries are often identifier-shaped, such as "what happened to invoice #4471". Pure vector similarity handles topical queries well. The failure shows up once real users ask about specific things, such as order numbers or exact dates. The embedding space was never built to tell those apart precisely.

How do you confirm it's this?

• Build a small query set of ID, code, and date lookups. Keep it separate from your topical query set. Measure recall@k for each set independently
• Compare vector-only retrieval against keyword (BM25) retrieval on the same identifier queries. A large gap in favor of keyword search confirms the failure mode
• Inspect the top results for a query that contains a negation, such as cannot or excluding. Check whether the positive counterpart outranks the correct negated match
• Track a "wrong entity, right topic" tag on retrieval failures reported by users, and watch whether it clusters

How do you fix it?

1. Hybrid search with rank fusion

   Run BM25 (or another exact/keyword method) alongside vector search and fuse the rankings. Keyword search catches exactly the identifiers and exact phrases that pure embeddings miss.

2. Cross-encoder rerank over the candidate set

   Retrieve a wider candidate set with cheap vector search. Then rerank the top 50 or so with a cross-encoder that scores query and document jointly. This recovers precision that bi-encoder similarity alone cannot.

3. Apply metadata filters before search, not after

   If the query or session context reveals the account ID, date range, or product line, filter the candidate set to it before ranking. Similarity then only has to separate documents within the correct scope.

4. Treat identifier queries as a distinct retrieval mode

   Detect query patterns that look like IDs, codes, or dates. Where the data is available, route them through exact-match or structured lookup and skip semantic search.

What this doesn't cover

This explains wrong-but-similar retrieval specifically. If retrieval returns unrelated documents instead of close-but-wrong ones, suspect embedding quality or index corruption. This identifier and negation gap is less likely.