Why does our AI cite a policy we deleted six months ago?

What does this look like in production?

• A user reports an answer that cites a policy, price, or procedure the team is certain was changed or removed
• The cited source document, when you go look, no longer exists at the URL or path the answer referenced
• Two users ask the same question a week apart and get materially different answers with no index rebuild in between
• Support tickets mention "the bot is behind" more than once from unrelated teams

Why does it happen?

Most retrieval pipelines are built around a one-time or periodically scheduled batch ingest: crawl the sources, chunk, embed, write to the vector store, done. That pipeline has no concept of a source mutating after ingest — there is no change-data-capture step watching the source of truth, no re-embed trigger on edit, and critically no tombstone process for deletes. A deleted document doesn't get removed from the index; it just keeps sitting there, embedded and searchable, because nothing ever told the index it was gone.

This is silent because retrieval quality metrics measured at build time look fine — the index matches the corpus as it existed on ingest day. The defect only shows up as the source and the index diverge over time, and it diverges fastest for exactly the documents that change most: pricing pages, policies, anything with a version history. By the time someone notices, the gap between "what's true" and "what's indexed" can be months wide.

How do you confirm it's this?

• Compare the newest source_updated_at timestamp across your content sources against the newest write timestamp in the index — a gap of more than a day or two on a source that changes often is your staleness window
• Pull 20 documents you know changed or were deleted in the last quarter and run the exact query that should surface them — check whether the old version, the new version, or nothing at all comes back
• Ask the index directly for a deleted document's known unique phrase and confirm it still returns a match
• Check whether your ingest pipeline has any code path that runs on delete, distinct from the code path that runs on create — if there is only one path, deletes are not handled

How do you fix it?

1. Move to change-data-capture ingest

   Subscribe to the actual change events of your source system (a webhook, a CMS event, a database trigger) instead of re-crawling on a timer, so an edit or delete triggers a re-embed within minutes, not at the next scheduled batch run.

2. Add soft-delete tombstones

   When a source document is deleted, write a tombstone record that excludes its chunks from retrieval immediately, even before a full re-index runs — a filter is faster to ship than a guaranteed-fresh pipeline and closes the worst failure mode first.

3. Carry source freshness into chunk metadata

   Store source_updated_at on every chunk and surface it in the answer ("as of [date]") so a reader can judge freshness themselves instead of trusting a system that cannot tell them when it last checked.

4. Alert on index staleness as a metric

   Track the age gap between source and index as a first-class number, the same way you would track error rate, and alert when it crosses a threshold instead of waiting for a user to notice first.

What this doesn't cover

This diagnosis assumes the index is genuinely out of sync with the source — if the source itself is wrong, or the model is answering from its own training data instead of the retrieved context, staleness isn't the bug you're chasing. Check that the answer is actually grounded in a retrieved chunk before assuming it's an ingest problem.