Hedronite · Ops Lesson · β-Trust/Synthesis-Lessons · 2026-06-11

Reorg-Safe Ingestion for Data Pipelines — When Upstream Retracts: Finality, Watermarks, and the Idempotent Backfill Discipline

Lesson Class: ops
Filed: 2026-06-11
Shelf: β-Trust/Synthesis-Lessons
Domains: Sec, Data, DataOps, BlockOps

<!-- hal:authoritative:yaml -->

§I — Frame

This morning's Chain lesson built an indexer that survives its upstream changing the past. A blockchain reorganization retracts blocks the indexer already processed; the cure was three primitives: a cursor that remembers where you are, a confirmation gate that holds rows until finality depth, and a backfill that replays a window idempotently when the past moves.

Now remove the blockchain. The vendor API that restates yesterday's transactions after a settlement correction. The upstream service that re-emits events after recovering from a partial outage. The labeling team that revises a batch of labels mid-week. The exchange feed that publishes a correction record for a mis-printed trade. None of these are reorgs in name. All of them are reorgs in shape: an upstream you already consumed retracting or rewriting what it told you.

Most pipelines are built as if upstream history is append-only, and most upstreams promise nothing of the kind. The Trust pair owns this lesson because the failure is a trust failure: the pipeline extended finality-grade trust to data that had only arrived, not settled. Arrival is not finality. The whole discipline falls out of refusing to confuse the two.

§II — Foundations

Four primitives, generalized from the chain frame.

Finality is the upstream's retraction contract: how far back can the source rewrite what it already served? A proof-of-stake chain states this in blocks. A payments vendor states it in settlement windows. Many sources state it nowhere, and an unstated contract is a contract you must measure from observed corrections or assume conservatively. Finality is a property of the source, not of your pipeline; the pipeline's job is to honor it.

The watermark is the pipeline's read of event-time progress: the point up to which the pipeline believes it has seen what it will be given. The streaming literature builds watermarks to handle late and out-of-order arrivals; the bridge insight is that a correction is a late arrival that contradicts an earlier one, so the machinery for lateness and the machinery for retraction are one machinery read at two depths.

Idempotent application makes replays harmless: every write keyed on a stable upstream identity, applied as an upsert, so processing an event twice converges to the same state as processing it once. Exactly-once delivery is a transport fiction; exactly-once effect is an application property you can actually build. The stream-processing canon is blunt about this distinction, and every reorg-safe design leans on it.

The bounded backfill is the repair primitive: when the past moves inside the finality window, re-run that window from the cursor, letting idempotent writes converge the state. A backfill that is safe to run is a backfill that gets run; a backfill that might double-count is a backfill the operator fears, and feared repairs rot.

§III — Mechanism

Two-tier serving: arrived versus settled

The chain indexer held rows below confirmation depth out of the canonical table. Generalized: ingest everything immediately into a provisional tier stamped with arrival time and source position, and promote to the settled tier only when the source's finality window has passed. Consumers choose their tier the way a trader chooses between an indicative quote and a settled fill. Analytics that want freshness read provisional and accept revision risk; anything feeding decisions, training sets, or financial reporting reads settled.

This is the same shape the warehouse literature reaches with late-arriving data and slowly-changing dimensions; the chain frame adds the discipline of making the promotion depth an explicit, per-source, recorded number rather than a folk assumption.

The cursor and the identity key

The pipeline's cursor records the source position last processed (offset, sequence number, block height, file manifest, API page token) in the same transaction as the writes it covers. Cursor and effect move together or not at all; a cursor that advances without its writes, or writes that land without their cursor, is the asymmetry that turns a restart into data loss or duplication.

Idempotence rests on the identity key: the stable upstream identifier each record carries (transaction id, event id, composite of source + sequence). Choose it from the upstream's own identity, never from arrival order. When a correction arrives bearing the same key, the upsert replaces; when a retraction arrives, a tombstone marks deletion. Both replay cleanly.

Detecting that the past moved

A chain announces reorgs by parent-hash mismatch. Generic upstreams are quieter, so the pipeline checks: re-read a trailing slice of the source on each cycle and compare content hashes against what was stored; subscribe to the source's correction feed where one exists; and alert on settled-tier mutations, because a correction that reaches inside the finality window means the contract itself was wrong and a human should re-measure it. That last alarm is the zero-trust posture applied to data: trust is granted to the contract, the contract is verified continuously, and a violated contract revokes trust until re-established.

The backfill discipline

When drift is detected at position P, the repair is mechanical because the primitives made it so: rewind the cursor to P, replay through present, let upserts and tombstones converge state, advance the cursor. Bound the window by the finality depth so the blast radius is known in advance. Record the backfill as an event (window, row deltas, trigger) so revisions are auditable. The pipeline that can replay any window without fear has converted corrections from incidents into routine.

§IV — Worked Example

A market-data pipeline ingests a vendor's daily trade file at 18:00 and an intraday stream through the session. The vendor's published correction policy: restatements possible for two business days. Finality depth: T+2.

The pipeline writes the stream into the provisional tier keyed on the vendor's trade id, cursor on the stream sequence number, upserts throughout. The 18:00 file is treated as a correction pass over the day: re-read, hash-compare against stored rows, upsert the diffs. Settled promotion runs nightly for data older than T+2. Wednesday, the vendor restates 4,100 Monday trades after a mis-print at one venue. The trailing hash check catches the divergence at the Monday slice, the backfill rewinds to Monday's open position and replays two days, 4,100 upserts converge, 12 tombstones land, and the settled tier never saw the bad prints because Monday had not yet promoted. Total operator involvement: reading the backfill event in the morning log.

The same pipeline without the discipline would have served the bad prints to the backtest store for two days and required a hand-written repair script under pressure. The strategy that backtested against those prints would have learned an edge that never existed.

§V — Connection to Prior Lessons

Today's Chain lesson is the source frame: cursor, confirmation gate, backfill, all built against a source that announces its retractions. This lesson removes the announcement and keeps the architecture, which is the test of whether the architecture was about blockchains or about upstreams. It was about upstreams. The 2026-06-10 zero-trust lesson supplies the posture: identity verified per request, trust granted narrowly and re-verified continuously. Here the requester is a data source and the credential is its finality contract; continuous verification is the trailing hash check. Trust the contract, verify the contract, alarm on breach. One posture, two planes.

§VI — Harmonic Ripples

This lesson ripples into BlockOps-Magus (the chain indexer becomes one instance of a general class, which is the direction that makes both teachable), QuantOps-Magus (backtest-store integrity: a restated print that reaches the research tier silently is a fabricated edge), MLOps-Magus (training-data assembly inherits settled-tier reads, so retraining runs do not learn from data that later retracts), and SecOps-Virtual-Machinist (the provisional/settled split is an integrity boundary; tampering upstream of finality is caught by the same hash checks that catch corrections).

§VII — Closing

List your pipeline's upstreams. For each one, write down its retraction contract: how far back can it rewrite, and where is that promise stated? Where you cannot answer, you have been extending finality-grade trust on arrival-grade evidence. Add the provisional tier, key every write on upstream identity, bound the backfill by the contract. Then test it: restate something on purpose and watch the pipeline converge. Do this before the upstream does it for you.

Related