BreezyCorp Handbook

Appearance

Ops Troubleshooting Runbook ​

Operational playbook for the on-call accountant/engineer. Each entry: symptom → where to look → what it means → action. Most "stuck" states are by-design back-pressure, not bugs — confirm against this list before escalating.

Audit trail is the source of truth. Every state change writes an audit_events row (entity_type, entity_id, event_type). When in doubt, query the audit log for the affected entity before touching data.

Payroll ​

Duplicate-receipt warning on a cycle (NOT blocking) ​

  • Symptom: validation surfaces a duplicate-receipt issue but export is still allowed.
  • Meaning: by design. Of the 5 core validation rules, 4 are BLOCKING (attendance, change docs, IR21, NS docs) and duplicate receipts is a WARNING only (packages/domain/src/payroll/rules/validation-rules.ts).
  • Action: the PE reviews and proceeds; duplicates never halt an export. Document the decision in the cycle notes. Do not "fix" by editing validation.
  • Symptom: invite/reminder/approval emails link to localhost.
  • Meaning: PORTAL_BASE_URL / PORTAL_URL_BASE not set in the environment.
  • Action: set the production domain in the server .env and restart the worker.

Bookkeeping ​

A bank statement is stuck "parsing" then goes FAILED ​

  • Symptom: BankStatement.parseStatus sits at PENDING, then FAILED after retries.
  • Where: apps/worker/src/handlers/bookkeeping/parse-bank-statement.ts.
  • Meaning: by design — the parser throws to trigger a pg-boss retry until the document's OCR extraction completes. If OCR never completes, retries exhaust and the statement is marked FAILED.
  • Action: check the source document's extraction (documents / OCR logs). If OCR failed, see "OCR / extraction failed" below, then re-trigger the statement parse.

Reconciliation run has "neither statementId nor periodId" ​

  • Symptom: error logged from reconciliation.service.ts (defensive throw).
  • Meaning: a data-integrity violation that should never occur if the schema is intact.
  • Action: alert engineering. Capture the ReconciliationRun.id; do not retry blindly.

Low-confidence journal entries auto-drafted ​

  • Symptom: entries with LOW / UNCLASSIFIED confidence appear as DRAFT, not escalated.
  • Meaning: classification confidence is advisory; the reviewer decides.
  • Action: reviewers must inspect confidence before approving; ESCALATE manually when unsure.

Client configured for SharePoint ingests nothing ​

  • Symptom: a folder-sync client receives zero documents.
  • Meaning: SharePoint's adapter is a placeholder (empty pulls). Channel creation is now gated (POST .../ingestion-channels returns 422 for SHAREPOINT), but any pre-existing SharePoint channel still no-ops.
  • Action: move the client to Google Drive or Dropbox. Delete the SharePoint channel.

Shared — OCR / extraction failed ​

  • Symptom: a document shows EXTRACTION_FAILED; downstream parse/classify stalls.
  • Meaning: the OCR provider (Vision + Claude) errored or OCR_PROVIDER=mock in prod.
  • Action: (1) confirm OCR_PROVIDER is the real provider in the server .env and the Vision/Anthropic credentials resolve; (2) re-enqueue the ocr-process job for the document; (3) if the provider is down, OCR has no automatic fallback — wait or retry.

Shared — Chatwoot messaging ​

  • Symptom: Settings → Integrations → Chatwoot is red/Unreachable; creating a Telegram inbox 503s; the Inbox composer 503s on Send.
  • Meaning: outbound calls to the Chatwoot API are failing. Most often a bot-protection / WAF challenge in front of the Chatwoot host answering our server-to-server requests with a 403 HTML page. Inbound webhooks keep working, so messages still arrive — only sending breaks.
  • Action: follow docs/runbooks/chatwoot-unreachable.md. Check chatwootFailureKind in the API logs first: BOT_PROTECTION → that runbook; AUTH → the token, see secret-rotation.md.
  • Workaround meanwhile: staff can reply from the Chatwoot UI. Those replies still reach the console via the chatwoot.mirror job, which the block does not touch.
  • Do not close until a reply sent from the console composer has been confirmed delivered — inbox creation succeeding does not prove the send path is healthy. Use Internal note mode to test against a live client thread without messaging anyone.
  • A 502 (not 503) from the composer is a different fault: Chatwoot answered and refused the message. Wrong runbook.

Financial Statements ​

Engagement stuck at NOTES_ERROR ​

  • Symptom: engagement status is NOTES_ERROR after notes generation.
  • Where: NotesService.generate (packages/domain/src/financial-statements/services/notes.service.ts).
  • Meaning: by design — a Claude/parse failure transitions to the recoverableNOTES_ERROR state and writes a fs.notes.failed audit event. The job completes (it does not retry a non-deterministic LLM call).
  • Action: read the fs.notes.failed audit reason. Fix the input if needed, then re-trigger notes generation (POST /ops/financial-statements/engagements/:id/notes/auto-generate). The FSM allows NOTES_ERROR → STATEMENTS_GENERATED → NOTES_COMPLETE.

Client requested a revision ​

  • Symptom: an engagement at NOTES_COMPLETE reverts to STATEMENTS_GENERATED.
  • Meaning: by design — the client used the portal "request revision" action (fs.client.revision_requested audit). The preparer regenerates notes after addressing the comment in the audit eventDataJson.

TB upload "succeeds" but mapping shows nothing ​

  • Meaning: the trial-balance parse produced zero rows (bad file) — see the fs.trial_balance.parse_failed audit reason.
  • Action: re-upload a clean CSV/XLSX.

XBRL Filing ​

Document ingest rejected as "not a recognizable financial statement" ​

  • Symptom: XbrlSourceDocument.parseStatus = FAILED; filing stays in DRAFT.
  • Where: DocumentIngestionService.runIngest pre-ingestion gate.
  • Meaning: by design — the document parsed but had no line items or only notes (no primary statement). A xbrl.document.ingest_rejected audit records the reason. The job is swallowed (no DLQ cycling) because a retry won't change the file.
  • Action: confirm the uploaded DOCX/PDF actually contains a balance sheet / P&L, then re-ingest the corrected document.

Export blocked (HTTP 422) ​

  • Meaning: the export gate (Rules 2 & 3) refuses output until validation has passed and every tag mapping is human-confirmed (assertExportAllowed).
  • Action: run validation; confirm any unconfirmed mappings; retry the export. This gate is non-negotiable — do not bypass it.

Worker / queue health ​

  • Dead-lettered jobs — a job that exhausts its retries is routed to the dead-letter queue (configured on every ad-hoc queue in apps/worker/src/handlers/index.ts). A single consumer logs it at error level (worker.job.dead_letter), increments the in-process worker.job.dead_letter counter, captures it to Sentry, and writes an append-only audit_events row (eventType = 'worker.job.dead_letter', entityType = 'PgBossJob').
    • To review dead-lettered jobs: query audit_events WHERE event_type = 'worker.job.dead_letter' (the event_data_json holds the original payload + pg-boss output/error), or grep worker logs for worker.job.dead_letter.
    • Note: handlers that catch their own errors (most send/notify handlers) complete normally and never dead-letter — their failures are already logged inside the handler. Dead-lettering captures the "gave up after retries" case (e.g. parse-bank-statement, xbrl ingest) — the genuinely silent-failure path before this existed.
  • Graceful restart: the worker drains in-flight jobs for up to 2 minutes on SIGTERM; the API readiness probe returns 503 during shutdown so the LB pulls it from rotation. The worker also flushes buffered Sentry events before exit.

Observability — Sentry ​

  • Default state: OFF. Sentry initialisation is a guarded no-op until SENTRY_DSN (API + worker) and NEXT_PUBLIC_SENTRY_DSN (web) are set. Dev, CI, and preview builds run with no Sentry traffic. See .env.example for the full var list.
  • To enable in production: stand up the self-hosted Sentry instance, create a project, and set SENTRY_DSN + NEXT_PUBLIC_SENTRY_DSN (usually the same project DSN) in the server environment + the web build env. NEXT_PUBLIC_* is baked at build time, so the web service must be rebuilt after setting it. Optionally set SENTRY_TRACES_SAMPLE_RATE (0..1; default 0 = errors only) and SENTRY_RELEASE.
  • What is captured: API 5xx errors (via the error-handler plugin), worker pg-boss errors + dead-lettered jobs, and web server/client errors. 4xx client errors are not captured. PII is scrubbed twice — Pino log redaction plus a Sentry beforeSend denylist (packages/observability/src/sentry.ts) — before any event leaves the process.
  • Not yet wired (follow-ups): withSentryConfig for web source-map upload (needs SENTRY_ORG / SENTRY_PROJECT / auth token), and a Prometheus/OTel exporter behind the @breezycorp/observability metrics facade. An external uptime monitor should poll /health/live + /health/ready independently of Sentry.

Internal use only — BreezyCorp