How do you keep a retrieval index from drifting away from its source?
The ingest pipeline subscribes to change events from the source system. This replaces re-crawling on a fixed schedule. The events come from a webhook or a CMS publish hook. An edit triggers a re-embed within minutes. A delete writes a tombstone that excludes the old chunks from retrieval at once, before a full re-index runs.
Also known as CDC ingest, event-driven re-indexing, tombstone-on-delete
What problem does this solve?#
A batch ingest pipeline crawls and chunks the source on a schedule. It then embeds the chunks and writes them to the index. It has no concept of a source that changes after ingest. No code path runs on delete, separate from the path that runs on create. So a deleted document stays in the index, embedded and searchable. Nothing told the index that the document was gone.
The failure is silent. Retrieval metrics measured at build time look fine, because the index matches the corpus as it was on ingest day. The gap opens as source and index diverge. It opens fastest for the documents that change most, such as pricing pages and policy pages. By the time a user notices a stale answer, the gap can be months wide.
How does it work?#
Ingest subscribes to real change events at the source. Examples are a CMS webhook on publish or a database change stream. This replaces polling on a timer. An edit event triggers a targeted re-embed of the changed document only. This typically happens within minutes, well before the next scheduled batch run.
A delete event writes a tombstone record at once. The tombstone excludes the chunks of that document from retrieval results, even before a full re-index runs. A filter ships faster than a guaranteed-fresh pipeline. It closes the worst failure mode, serving deleted content, first and cheaply.
Every chunk carries the last-updated timestamp of its source. The final answer shows it as "as of [date]," so a reader can judge freshness. The team tracks the age gap between source and index as an operational metric. An alert fires on that gap the same way it fires on an error rate.
What are the moving parts?#
- Subscribe to the source change events
Use a webhook, a database trigger, or a file watch in place of polling on a schedule.
- Trigger a targeted re-embed on edit
An edit event re-embeds only the changed document, typically within minutes.
- Write an immediate tombstone on delete
Exclude the chunks of a deleted document from retrieval right away, before a full re-index runs.
- Carry source_updated_at into chunk metadata
Show freshness in the answer itself, so a reader can judge it without trusting the pipeline.
- Track index staleness as an alertable metric
Watch the age gap between source and index continuously, as a team watches an error rate.
When does it fit, and when doesn't it?#
Use it when
- The team edits or deletes the source documents behind the index with some regularity, as with policies and pricing
- A batch or scheduled ingest has a staleness window of days, and that window has caused an incident
- The source system emits change events, such as webhooks or database triggers, that ingest can subscribe to
- Serving deleted or superseded content is a compliance risk or a trust risk, and a minor inconvenience at best
Don't use it when
- The source corpus is static, such as a fixed reference set that does not change after launch. No drift exists to capture
- The source system has no change-event mechanism, and building one is out of scope. A tighter re-crawl schedule plus a tombstone-on-detected-404 check is the more realistic interim fix
- The system is pre-launch and has no production traffic. Build this once staleness is a measured problem
What does it cost?#
Edits and deletes reach the index in minutes and skip the wait for the next scheduled batch run
Each source system needs its own event subscription integration, which takes more upfront engineering than a cron job
Tombstones close the worst failure mode, serving deleted content, fast and cheaply
A tombstone-only fix leaves edited content stale until the full re-embed catches up
Staleness becomes a tracked, alertable metric, so users no longer find it first
The pipeline adds operational surface. A broken webhook or a missed change event becomes its own class of production incident
Freshness timestamps let a reader judge whether an answer is current
Every source connector must expose a reliable last-updated timestamp, and some sources do not offer one by default
How do you know it's working?#
- Age gap between the newest source_updated_at and the newest index write timestamp, tracked continuously
- Time from a source edit or delete event to the moment the index reflects it
- Count of tombstoned documents still present in raw retrieval results, which should be zero
- Change-event delivery success rate from each source connector
What failure modes does this prevent?#
Sources
- Debezium - Debezium Documentation: Architecture
The change-capture architecture our ingest pattern is built on. Retrieved 2026-09-17.
- PostgreSQL - PostgreSQL: Logical Replication
The change-stream our ingest pattern reads. Retrieved 2026-09-17.
A Ship Audit checks which of these patterns your system needs. It ranks them by what is most likely to break first.