Runbook: Secret Rotation

Every secret in BreezyCorp supports zero-downtime rotation via a dual-key pattern: a _PREVIOUS variant is accepted while the new key propagates, then retired.

Catalog

Secret	Rotation impact	Strategy	Owner
JWT_SECRET	Invalidates magic-link tokens	48h dual-verify window (max magic-link TTL)	Platform
MAGIC_LINK_SECRET	Same as JWT_SECRET	Same	Platform
OPENROUTER_API_KEY	All AI stops until replaced (default gateway)	Generate new key at openrouter.ai → Keys, replace in .env, restart worker	Platform
ANTHROPIC_API_KEY	All AI stops, only when AI_GATEWAY=anthropic	Generate new key in console.anthropic.com, replace in .env, restart worker	Platform
GOOGLE_VISION_CREDENTIALS_JSON	OCR pipeline fails at vision stage	Rotate service account key in GCP IAM, replace JSON blob in .env, restart worker	Platform
MFA_SECRET_KEY	Cannot decrypt stored TOTP seeds	Decrypt-with-previous at login; one-time re-encrypt migration	Platform
PAYROLL_EMPLOYEE_ENCRYPTION_KEY	Cannot read stored NRIC/FIN, passport, work pass, bank numbers, birth dates	Decrypt-with-previous on read; one-time re-encrypt migration (see below)	Platform
SEED_ADMIN_PASSWORD	Dev-only	n/a	—
DATABASE_URL password	Connection loss	Managed rotation via RDS/Neon; blue/green with PgBouncer	Platform + DBA
S3_ACCESS_KEY / S3_SECRET_KEY	Upload/download failures	New credentials in Hetzner Cloud Console, swap in .env.prod, restart api+worker, delete old pair (see object-storage.md)	Platform
SENTRY_DSN	Lost error capture during rotation	Set new DSN, roll restart	Platform

General procedure

1. Generate new secret (use openssl rand -hex 32 or provider tooling)
2. Set the new value as <SECRET>_PREVIOUS first, deploy
3. Then set the new value as <SECRET>, deploy again
4. Wait out the overlap window (see per-secret table above)
5. Remove <SECRET>_PREVIOUS, deploy final

At step 3, both keys are live; validators accept either. At step 5, only the new key remains.

JWT_SECRET rotation (worked example)

# 1. Generate
NEW=$(openssl rand -hex 32)

# 2. Stage as previous — existing tokens still work, no impact
kubectl set env deployment/spade-api JWT_SECRET_PREVIOUS=$OLD
kubectl rollout status deployment/spade-api

# 3. Promote the new key
kubectl set env deployment/spade-api JWT_SECRET=$NEW
kubectl rollout status deployment/spade-api

# 4. Wait 48h (longest magic-link TTL)

# 5. Drop the previous
kubectl set env deployment/spade-api JWT_SECRET_PREVIOUS-
kubectl rollout status deployment/spade-api

OCR provider credential rotation

OCR runs synchronously in the worker against Google Cloud Vision + Anthropic Claude (see packages/documents/src/factory.ts). There is no webhook callback path and no dual-key rotation — swapping a credential means a brief OCR outage while the worker restarts.

Which key you rotate depends on AI_GATEWAY — check it first (openrouter is the default).

OpenRouter (OPENROUTER_API_KEY) — the default gateway:

1. Create new key at openrouter.ai → Keys
2. Replace OPENROUTER_API_KEY in the worker's environment
3. Restart the worker process
4. Verify with a portal file upload that OCR jobs complete
5. Delete the old key in the OpenRouter dashboard

Note this key covers all platform AI, not just OCR — FS notes, document-intent classification, ROF/TKT drafts and PDF splitting route through the same gateway.

Anthropic (ANTHROPIC_API_KEY) — only when AI_GATEWAY=anthropic:

1. Create new key at console.anthropic.com → API Keys
2. Replace ANTHROPIC_API_KEY in the worker's environment
3. Restart the worker process
4. Verify with a portal file upload that OCR jobs complete
5. Delete the old key in the Anthropic console

Google Cloud Vision (GOOGLE_VISION_CREDENTIALS_JSON):

1. GCP Console → IAM & Admin → Service Accounts → vision-ocr@… → Keys → Add Key → JSON
2. Replace GOOGLE_VISION_CREDENTIALS_JSON in the worker's environment with the full JSON blob
3. Restart the worker process
4. Verify with a portal file upload that OCR jobs complete
5. Delete the old key in the same Keys tab

If OCR fails with DECODER routines::unsupported, the base64 private_key is malformed — copy the downloaded JSON directly without manual line-wrapping.

MFA_SECRET_KEY rotation (destructive-adjacent)

TOTP seeds are encrypted at rest with AES-256-GCM keyed on MFA_SECRET_KEY. Rotation requires decrypting with the old key and re-encrypting with the new one.

1. MFA_SECRET_KEY_PREVIOUS=<current> — deploy
2. Set MFA_SECRET_KEY=<new> — deploy
3. Run the re-encryption migration: (TODO: build scripts/reencrypt-mfa-secrets.ts) — iterates staff_users, decrypts with previous key, re-encrypts with current
4. Verify: SELECT COUNT(*) FROM staff_users WHERE mfa_secret IS NOT NULL × trial login for a sample user
5. Remove MFA_SECRET_KEY_PREVIOUS

If the migration is skipped: users can still log in (decrypt-with-previous fallback in packages/auth/src/crypto.ts), but you can never drop the previous key.

PAYROLL_EMPLOYEE_ENCRYPTION_KEY (destructive if lost)

The payroll employee record (EMP-1) encrypts its sensitive values (NRIC/FIN, passport number, work pass number, bank account number, date of birth) with AES-256-GCM keyed on PAYROLL_EMPLOYEE_ENCRYPTION_KEY, through the same helper as MFA_SECRET_KEY (encryptAtRest / decryptAtRest in packages/auth/src/crypto.ts). It is read by the API only.

Before first use in an environment: set it to 64 hex characters (openssl rand -hex 32) and store it with the database's backups. Without it, those fields are refused on save with a message naming the variable; every other field still saves. If it is lost, the stored values cannot be recovered and must be collected from clients again. Losing it does not break anything else: the screen shows such a value as "on record, but cannot be read on this server".

Rotation:

1. PAYROLL_EMPLOYEE_ENCRYPTION_KEY_PREVIOUS=<current> — deploy
2. Set PAYROLL_EMPLOYEE_ENCRYPTION_KEY=<new> — deploy. New values are written with the new key; old ones still decrypt with the previous one.
3. Re-encrypt the stored values: (TODO: build a script) — read every payroll_employee_values row with value_encrypted, decrypt (either key), encrypt with the current key, update in place. Superseded rows count too: they are the history.
4. Verify: reveal a sample value on the Employees tab of a payroll company.
5. Remove PAYROLL_EMPLOYEE_ENCRYPTION_KEY_PREVIOUS.

If step 3 is skipped, values keep working while the previous key is set, but it can never be removed.

Password change → session revocation

Changing a staff user's password already revokes all their other sessions (Phase 3A.5). Rotating the password store secret is separate — see DATABASE_URL rotation above.

Emergency rotation (compromise suspected)

1. Skip the overlap window
2. Set new secret directly (don't stage as previous)
3. Accept the downtime: all affected sessions/tokens break immediately
4. Force staff re-login by also running the UPDATE staff_sessions SET revoked_at = NOW() query from runbooks/tenant-leak-suspected.md
5. Open a post-mortem channel

Related

• packages/auth/src/crypto.ts — dual-key AES-256-GCM implementation
• packages/documents/src/factory.ts — OCR adapter construction from env
• docs/pii-inventory.md — which secrets protect which PII columns