Why does retrieval return confidently wrong but similar documents?

What does this look like in production?

• Retrieval confidently returns a document about a similar-sounding product, plan tier, or account instead of the one the user actually asked about
• A query containing a specific ID, date, or code returns topically related but factually wrong matches
• A negated statement ("does not support X") and its positive counterpart ("supports X") both score highly against the same query
• Keyword search on the same query would have found the right document immediately, but the vector search buried it

Why does it happen?

Embedding models are trained to place semantically similar text close together in vector space, and they do that well for topic and meaning. What they do not reliably encode is the specific, discrete facts that often decide whether a match is actually correct: an exact identifier, a specific date, a negation flipping a claim's truth value. Two passages that are "about the same topic" but disagree on the one detail that matters end up close together anyway, because the embedding space rewards resemblance, not correctness on that detail.

This is invisible in a demo because demo queries tend to be topical ("how do I reset my password") rather than identifier-shaped ("what happened to invoice #4471"), and topical queries are exactly what pure vector similarity is good at. The failure shows up once real users start asking about specific things — accounts, order numbers, exact dates — that the embedding space was never built to distinguish precisely.

How do you confirm it's this?

• Build a small query set specifically of ID, code, and date lookups, separate from your topical/semantic query set, and measure recall@k for each set independently
• Compare vector-only retrieval against keyword (BM25) retrieval on the same identifier queries — a large gap in favor of keyword search confirms the failure mode
• Manually inspect the top results for a query containing a negation ("cannot," "does not," "excluding") and check whether the positive counterpart outranks the correct negated match
• Track a "wrong entity, right topic" tag on retrieval failures reported by users, and watch whether it clusters

How do you fix it?

1. Hybrid search with rank fusion

   Run BM25 (or another exact/keyword method) alongside vector search and fuse the rankings — keyword search catches exactly the identifiers and exact phrases that pure embeddings miss.

2. Cross-encoder rerank over the candidate set

   Retrieve a wider candidate set with the cheap vector search, then rerank the top 50 or so with a cross-encoder that scores query and document jointly — this recovers precision that bi-encoder similarity alone cannot.

3. Apply metadata filters before search, not after

   If the account ID, date range, or product line is known from the query or session context, filter the candidate set to it before ranking, so similarity only has to distinguish within the correct scope instead of across the whole corpus.

4. Treat identifier queries as a distinct retrieval mode

   Detect query patterns that look like IDs, codes, or dates and route them through exact-match or structured lookup instead of semantic search entirely, where that data is available.

What this doesn't cover

This explains wrong-but-similar retrieval specifically — if retrieval is returning genuinely unrelated documents rather than close-but-wrong ones, the problem is more likely embedding quality or index corruption than this identifier/negation gap.