Runbook: Object Storage (Hetzner)

Prod file storage lives in Hetzner Object Storage, not on the VPS. Local dev is unchanged and still uses the MinIO container from docker-compose.yml.

Item	Value
Bucket	breezycorp-prod
Endpoint	https://hel1.your-objectstorage.com (path-style, public)
Location	hel1 — same Hetzner location as the servers, so api/worker traffic to the bucket stays inside Hetzner's network
Auth	S3_ACCESS_KEY / S3_SECRET_KEY in /opt/breezycorp/.env.prod on hermes01

Cut over from the on-box MinIO container on 2026-07-22 (all objects mirrored with mc mirror --preserve; presigned upload/download verified end-to-end).

How the app uses it

• Both S3 clients (apps/api/src/plugins/storage.ts, apps/worker/src/lib/s3.ts) wrap @breezycorp/storage/client, which is the only place S3_BUCKET and the credentials are read (S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, S3_BUCKET, S3_REGION, forcePathStyle: true, plus S3_PRESIGN_TTL_SECONDS / S3_MAX_PRESIGN_TTL_SECONDS). No code knows which backend is behind the endpoint.

• Browsers receive presigned URLs pointing straight at the bucket endpoint — uploads via presigned PUT, downloads via presigned GET. No proxy is involved (the old files.breezycorp.app Caddy route is retired).

• Object keys are server-owned and never cross the wire. Presign returns { fileId, uploadUrl }; finalize takes the fileId and resolves the key server-side. Naming, the canonical folder layout, and the land → file lifecycle are documented in docs/reference/architecture/object-storage-layout.md.

• Because browsers PUT directly, the bucket carries a CORS policy allowing origin https://console.breezycorp.app (methods GET/PUT/HEAD). If the console ever moves domain, update the policy:

  aws s3api put-bucket-cors --bucket breezycorp-prod \
    --endpoint-url https://hel1.your-objectstorage.com --region hel1 \
    --cors-configuration '{"CORSRules":[{"AllowedOrigins":["https://console.breezycorp.app"],"AllowedMethods":["GET","PUT","HEAD"],"AllowedHeaders":["*"],"ExposeHeaders":["ETag"],"MaxAgeSeconds":3600}]}'

Credentials

• Generated ONLY in the Hetzner Cloud Console (project → Object Storage → Manage credentials). There is no API/CLI for credential management, and the secret is displayed exactly once at creation.
• Credentials are project-scoped (one keypair sees every bucket in the project), not per-bucket.
• Rotation: create a second keypair in the Console, swap the values in .env.prod, docker compose ... up -d --no-deps api worker, verify /health/ready reports s3: true, then delete the old keypair.

Backups & restore

deploy/backup.sh (nightly root cron on hermes01) mirrors the bucket via a throwaway minio/mc container and tars it to /opt/breezycorp/backups/s3-<timestamp>.tar.gz, alongside the Postgres dump. Retention is 14 days.

To restore objects (extract, then push back with mc):

mkdir /tmp/s3-restore && tar xzf backups/s3-<ts>.tar.gz -C /tmp/s3-restore
docker run --rm --entrypoint sh -v /tmp/s3-restore:/restore \
  -e EP="$S3_ENDPOINT" -e KEY="$S3_ACCESS_KEY" -e SECRET="$S3_SECRET_KEY" -e B="$S3_BUCKET" \
  minio/mc -c 'mc alias set s3 "$EP" "$KEY" "$SECRET" >/dev/null && mc mirror --preserve /restore "s3/$B"'

Gotchas

• Bucket names are shared per-location across all Hetzner customers — a "taken" name on create means someone else owns it, pick another.
• Presigned URLs embed the endpoint + bucket, so changing either invalidates URLs already handed to browsers (they're short-lived; a cutover only needs the object data mirrored first).
• The final MinIO-era safety nets on hermes01 (from the 2026-07-22 cutover): .env.prod.bak-minio, backups/minio-20260722-132255.tar.gz, and the orphaned breezycorp_miniodata docker volume. Safe to delete once the bucket has soaked.