Your replica copied the bad DELETE. Now recover the orders that came next.

7 minute read

You deploy a cleanup job for an orders table. The job succeeds. A few minutes later, someone notices that recent orders have disappeared.

There is a replica in another AWS Region, so you check it. The same rows are missing there too. Replication has done exactly what it was configured to do.

You can restore an earlier snapshot. But customers have placed more orders since then. Existing orders have shipped, and some have been legitimately deleted. How do you recover the missing rows without throwing away those changes?

This is the recovery problem I wanted to work through with Amazon S3 Tables: rebuild a correct analytical table at a defined point in the source event stream, then continue processing from there.

What an old snapshot cannot tell you

Consider five orders:

The earlier snapshot can restore A, but it has no knowledge of D or E. Starting from the damaged current table preserves those new orders but cannot recover A if nothing has changed it since the deletion.

The recovery needs two inputs: a snapshot we trust and the legitimate source events that follow it.

The AWS design

The normal pipeline writes full order events to a protected, general-purpose S3 bucket. EMR Serverless applies committed batches to an Iceberg table in S3 Tables. Managed replication maintains a copy in a second Region.

The recovery job reads a known-good snapshot from that replica, copies its rows into a separate writable table, and replays the retained source events. A small routing record identifies the table that the example reader should query.

S3 Tables replicas are read-only. They can retain snapshot history independently of the source, but replication is asynchronous and does not support tables with user-defined branches or tags. AWS documents these replication constraints.

If the source still has the required snapshot, recovery can start there instead. The replica is useful when it retains history the source has expired. A complete source journal also allows a full rebuild; the snapshot reduces how much history must be replayed.

Record a recovery boundary before you need one

For each successful batch, retain its table identity, snapshot ID and highest contiguous source position. Commit the table changes first, then record the checkpoint.

That ordering matters. If a job fails after the table commit but before the checkpoint, restarting may replay the batch. The write logic must tolerate that. Advancing the checkpoint first could silently skip data.

In the example package, a batch becomes accepted only after its event object and manifest are stored. The manifest records the exact object version, checksum, count and position range. Recovery verifies the manifest chain instead of assuming that the largest timestamp means every earlier event arrived.

This also defines the limit of the recovery claim. An order that never reached the retained journal cannot be reconstructed from it. A transactional application needs a dependable CDC or outbox capture path upstream.

Restore into a new writable table

Stop the table writer and confirm that its job has finished before rebuilding. Source events can continue arriving in S3, but analytical updates pause during recovery.

Select a snapshot whose contents are known to be correct. In a controlled drill, that is the validated checkpoint before fault injection. During a real incident, identifying it requires investigation; the snapshot immediately before the alert may already contain bad data.

Create the destination with an explicit schema, then copy the snapshot’s rows:

INSERT INTO recovered.lab.orders_recovery
SELECT *
FROM replica.lab.orders_state
VERSION AS OF 123456789;

The number is an illustrative snapshot ID. The executable job uses the checkpoint captured by the drill.

The destination owns the copied files. It does not depend on a metadata-only clone pointing at files that another table’s maintenance process might remove. The direct S3 Tables REST endpoint does not support CREATE TABLE AS SELECT, so creation and insertion are separate operations. S3 Tables REST endpoint documentation.

If the seed copy stops before completion can be confirmed, start a fresh recovery attempt. Appending the seed again to a partially completed table could duplicate rows.

Replay versions, not timestamps

Each event carries an order ID, a source-assigned version and the complete resulting row. Apply it only when that version is newer than the one already stored. An identical retry becomes a no-op. Conflicting payloads for the same ID or version stop the recovery.

Before MERGE, validate duplicates and reduce each batch to one row per order. Iceberg requires a single matching source record for each target row.

MERGE INTO recovered.lab.orders_recovery AS t
USING latest_valid_events AS s
ON t.order_id = s.order_id
WHEN MATCHED AND s.order_version > t.order_version
  THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

A legitimate delete becomes a versioned tombstone: the state row remains, with is_deleted = true. Readers exclude it. Removing the row and its version would allow an older replayed event to insert it again.

Order E is why event time cannot define the replay boundary. Its business timestamp may be old, but its journal position is new. Recovery includes it because it arrived within the committed range being rebuilt.

Validate, switch, then catch up

Choose a finite journal boundary, called H, and replay through it. Compare the rebuilt table with the expected state at that same boundary. Check every order’s values, version and deletion state, as well as key uniqueness. Equal row counts alone are insufficient.

For this synthetic drill, the expected state comes from an independent reference calculation. In production, the equivalent validation must reflect the source system and the business rules you trust.

Once validation succeeds and both writers have stopped, update a single S3 routing object with the new table, generation and applied position. An If-Match condition prevents a stale controller from overwriting a newer routing decision. S3 conditional-write behavior.

The next writer starts after H. Events accepted while validation was running remain in the journal and are processed afterward. The example reader resolves the route for each new query job; existing readers with cached table handles need their own coordination.

What the AWS drill showed

The source still retained the good snapshot during this run, and the complete journal remained available. This validates the recovery procedure; it does not establish that the replica was the only recovery source. Intentional failures and interactive pauses also make this a correctness exercise, not a production recovery-time benchmark. The recovered table does not automatically inherit replication; configure ongoing protection separately.

The scope is one analytical table after a bad table operation. The source journal remains available, and incoming events are valid. Corrupted source events, coordinated recovery across several tables, and a complete regional outage require additional design.

The question to take back to your pipeline is specific: can you identify a trusted snapshot, prove which source events follow it, and resume from a boundary that neither skips nor invents an order?

Source code: BuildWithAbbas S3 Tables recovery lab on GitHub. Start with the AWS test runbook to reproduce the drill in a dedicated sandbox.