How do you keep a retrieval index from drifting away from its source?
The ingest pipeline subscribes to the actual change events of the source system — a webhook, a CMS publish hook, a database trigger — instead of re-crawling on a fixed schedule. An edit triggers a re-embed within minutes; a delete writes a tombstone that excludes the old chunks from retrieval immediately, before a full re-index even runs.
Also known as CDC ingest, event-driven re-indexing, tombstone-on-delete
What problem does this solve?
A batch ingest pipeline — crawl, chunk, embed, write, on a schedule — has no concept of a source mutating after ingest. There is no code path that runs specifically on delete, distinct from the path that runs on create, so a deleted document does not get removed from the index; it sits there, embedded and searchable, because nothing ever told the index it was gone.
This is silent because retrieval metrics measured at build time look fine — the index matches the corpus as it existed on ingest day. The gap only opens as source and index diverge over time, fastest for exactly the documents that change most, such as pricing and policy pages, and by the time a user notices a stale answer, the gap can be months wide.
How does it work?
Ingest subscribes to real change events at the source — a CMS webhook on publish/unpublish, a database change stream, a file-system watch — rather than polling on a timer. An edit event triggers a targeted re-embed of just the changed document, typically within minutes, instead of waiting for the next scheduled batch run to sweep the whole corpus.
A delete event writes a tombstone record immediately, excluding that document's chunks from retrieval results even before a full re-index has run — a filter is faster to ship and deploy than a guaranteed-fresh pipeline, and it closes the worst failure mode, serving deleted content, first and cheaply.
Every chunk carries its source's last-updated timestamp, surfaced in the final answer as "as of [date]" so a reader can judge freshness themselves, and the age gap between source and index is tracked as a first-class operational metric, alerted on the same way an error rate would be, not discovered from a user complaint.
What are the moving parts?
- Subscribe to the source's real change events
Use a webhook, database trigger, or file watch instead of polling on a schedule.
- Trigger a targeted re-embed on edit
An edit event re-embeds just the changed document, typically within minutes.
- Write an immediate tombstone on delete
Exclude a deleted document's chunks from retrieval right away, before a full re-index has run.
- Carry source_updated_at into chunk metadata
Surface freshness in the answer itself, so a reader can judge it without trusting the pipeline blindly.
- Track index staleness as an alertable metric
Watch the age gap between source and index continuously, the same way an error rate is watched.
When does it fit, and when doesn't it?
Use it when
- Source documents feeding the index are edited or deleted with any regularity (policies, pricing, product docs)
- A batch or scheduled ingest currently has a staleness window measured in days, and that window has caused a real incident
- The source system genuinely emits change events (webhooks, database triggers) that ingest can subscribe to
- Deleted or superseded content being served is a compliance or trust risk, not just a minor inconvenience
Don't use it when
- The source corpus is effectively static (a fixed reference document set that does not change post-launch) — there is no drift to capture
- The source system has no change-event mechanism at all and building one is out of scope — a tightened re-crawl schedule plus a tombstone-on-detected-404 check is the more realistic interim fix
- You are pre-launch with no production traffic yet — build this once staleness is a measured problem, not speculatively
What does it actually cost?
Edits and deletes reach the index in minutes instead of at the next scheduled batch run
Requires a real event subscription integration per source system, which is more upfront engineering than a cron job
Tombstones close the worst failure mode (serving deleted content) fast and cheaply
A tombstone-only fix still leaves edited-but-not-deleted content stale until the full re-embed catches up
Staleness becomes a tracked, alertable metric instead of a user-reported surprise
Ongoing operational surface — a broken webhook or a missed change event is now a production incident class of its own
Freshness timestamps let a reader judge whether an answer is current
Requires every source connector to reliably expose a genuine last-updated timestamp, which not every source does out of the box
How do you know it's working?
- Age gap between newest source_updated_at and newest index write timestamp, tracked continuously
- Time from a source edit or delete event to the index reflecting it
- Count of tombstoned documents still present in raw retrieval results, which should be zero
- Change-event delivery success rate from each source connector
What failure modes does this prevent?
A Ship Audit checks which of these patterns your system actually needs, prioritized against what's most likely to break first.