Runbook: Storage-Layout Backfill

Moves every stored object in an existing environment onto the canonical object-storage layout. Read docs/reference/architecture/object-storage-layout.md first — this runbook assumes you know what the target layout is.

This has not yet been run against production.

What it does

Objects written before the canonical layout sit under inconsistent prefixes (QAPAY/… hardcodes one client code for every tenant; ops-uploads/… has no tenant segment at all). The backfill copies each one to the key the running application would now produce, and re-points the database row at it.

Three properties are worth understanding before you run it:

• Resolution is from foreign keys, never from the old key. The old keys disagree with each other, so parsing them would propagate those mistakes. Where the FK chain genuinely cannot answer, the row is SKIPPED and tagged — never guessed.
• Copy, switch, sweep — the source is never deleted in the same pass. That deferral is the rollback story: with both objects present, --rollback is a pure database revert.
• The ledger is a table (storage_key_migrations), not a file. Rollback has to be runnable from the one-shot migrate container, which has a database but no artifacts directory.

The backfill calls the same resolveFileTarget() the application uses, rather than reimplementing it — if the two disagreed, the very first write after the migration would diverge from everything the backfill had just moved.

Prerequisites

1. Both storage migrations applied: 20260723120000_storage_path_slugs and 20260723120100_storage_file_lifecycle. Confirm with pnpm db:migrate:status.
2. MIGRATE_DATABASE_URL (preferred) or DATABASE_URL set — the script needs to read across tenants, so use the privileged role.
3. S3 credentials for the target bucket in the environment.
4. A fresh database backup. --rollback reverts pointers, not schema.

Ledger statuses

Every row of storage_key_migrations carries one of:

Status	Meaning
PLANNED	Target key resolved; nothing copied yet.
COPIED	Object copied to the new key; row still points at the old one.
SWITCHED	Database row re-pointed. Both objects now exist.
VERIFIED	--verify confirmed the object is present at the new key.
SWEPT	Superseded source object deleted.
ROLLED_BACK	Pointer reverted to the old key.
SKIPPED	Not moved — see reason (e.g. SOURCE_OBJECT_MISSING, unresolvable scope).

tenant_id is nullable so an object whose tenant cannot be resolved is still recorded. NULL is invisible to any tenant-scoped read (NULL = x is NULL, not TRUE), which is the fail-closed behaviour we want — only the migrate role or an explicit bypass sees the ledger.

Procedure

All commands run from the repo root.

1. Dry run (default — writes nothing)

pnpm --filter @breezycorp/worker backfill:storage

Prints the planned old → new mapping (first 200), everything it would skip and why, and any orphans — objects present in the bucket with no database row, which are reported but never touched.

Review before continuing:

• Skipped rows: is the reason acceptable, or does an FK need fixing first?
• Orphans: expected leftovers, or evidence of a lost row?
• Spot-check a handful of target keys against the layout reference.

Narrow the scope while iterating:

pnpm --filter @breezycorp/worker backfill:storage -- --tenant <tenantId>
pnpm --filter @breezycorp/worker backfill:storage -- --table files --limit 50

2. Apply

pnpm --filter @breezycorp/worker backfill:storage -- --apply

Copies each object and re-points its row (PLANNED → COPIED → SWITCHED). Sources are retained.

Downtime is not required — the filer and the application produce the same keys the backfill does. Running it during a quiet window is still preferable, since every in-flight presigned URL for a moved object keeps working only until its TTL expires.

3. Verify

pnpm --filter @breezycorp/worker backfill:storage -- --verify

HEADs every SWITCHED entry at its new key and promotes it to VERIFIED. Anything reported as MISSING at target must be resolved before the sweep — that row's bytes are not where the database now says they are.

Then smoke-test the application: download a file from each product surface and confirm the Content-Disposition name is the new displayName.

4. Sweep — after 7 days

pnpm --filter @breezycorp/worker backfill:storage -- --sweep

Deletes the superseded source objects. The script refuses to sweep entries younger than the 7-day grace window and tells you how many it held back. Do not shorten this: the grace period is what keeps --rollback cheap.

5. Rollback (only before the sweep)

pnpm --filter @breezycorp/worker backfill:storage -- --rollback

Reverts every SWITCHED pointer to its old key and marks the entry ROLLED_BACK. Objects were never deleted, so both copies remain and no data is lost. After --sweep this is no longer possible — restore objects from the nightly bucket backup instead (see object-storage.md).

Gotchas

• Run the dry run twice and diff the output. Target keys are deterministic by construction — slugs are immutable, stamps come from createdAt, and collision suffixes are assigned in insertion order. Two dry runs that disagree mean something non-deterministic leaked into key resolution, and you should stop.
• The sweep grace window and the presign ceiling are different clocks. The filer's per-object delete waits S3_MAX_PRESIGN_TTL_SECONDS + 60; the backfill's sweep waits 7 days. The long window here is for operator confidence, not for URL expiry.
• Orphan objects are never deleted by any mode of this script. Clean them up deliberately, after confirming no row references them.
• _inbox objects are legitimate. A file that has landed but is not yet routable is expected to sit in _inbox — the backfill reports it as unfilable rather than moving it. Every ingest path enqueues the filer on finalize, so a file that stays in _inbox is one the resolver genuinely could not place (typically an Infotech output not yet linked to a cycle, or an inbound attachment awaiting a routing decision). A steadily growing _inbox is worth investigating; a stable population is normal.