BreezyCorp Handbook

Appearance

Ops Troubleshooting Runbook ​

Operational playbook for the on-call accountant/engineer. Each entry: symptom → where to look → what it means → action. Most "stuck" states are by-design back-pressure, not bugs — confirm against this list before escalating.

Audit trail is the source of truth. Every state change writes an audit_events row (entity_type, entity_id, event_type). When in doubt, query the audit log for the affected entity before touching data.

Payroll ​

Duplicate-receipt warning on a cycle (NOT blocking) ​

  • Symptom: validation surfaces a duplicate-receipt issue but export is still allowed.
  • Meaning: by design. Of the 5 core validation rules, 4 are BLOCKING (attendance, change docs, IR21, NS docs) and duplicate receipts is a WARNING only (packages/domain/src/payroll/rules/validation-rules.ts).
  • Action: the PE reviews and proceeds; duplicates never halt an export. Document the decision in the cycle notes. Do not "fix" by editing validation.
  • Symptom: invite/reminder/approval emails link to localhost.
  • Meaning: PORTAL_BASE_URL / PORTAL_URL_BASE not set in the environment.
  • Action: set the production domain in the server .env and restart the worker.

Bookkeeping ​

A bank statement is stuck "parsing" then goes FAILED ​

  • Symptom: BankStatement.parseStatus sits at PENDING, then FAILED after retries.
  • Where: apps/worker/src/handlers/bookkeeping/parse-bank-statement.ts.
  • Meaning: by design — the parser throws to trigger a pg-boss retry until the document's OCR extraction completes. If OCR never completes, retries exhaust and the statement is marked FAILED.
  • Action: check the source document's extraction (documents / OCR logs). If OCR failed, see "OCR / extraction failed" below, then re-trigger the statement parse.

Reconciliation run has "neither statementId nor periodId" ​

  • Symptom: error logged from reconciliation.service.ts (defensive throw).
  • Meaning: a data-integrity violation that should never occur if the schema is intact.
  • Action: alert engineering. Capture the ReconciliationRun.id; do not retry blindly.

Low-confidence journal entries auto-drafted ​

  • Symptom: entries with LOW / UNCLASSIFIED confidence appear as DRAFT, not escalated.
  • Meaning: classification confidence is advisory; the reviewer decides.
  • Action: reviewers must inspect confidence before approving; ESCALATE manually when unsure.

Client configured for SharePoint ingests nothing ​

  • Symptom: a folder-sync client receives zero documents.
  • Meaning: SharePoint's adapter is a placeholder (empty pulls). Channel creation is now gated (POST .../ingestion-channels returns 422 for SHAREPOINT), but any pre-existing SharePoint channel still no-ops.
  • Action: move the client to Google Drive or Dropbox. Delete the SharePoint channel.

Shared — OCR / extraction failed ​

  • Symptom: a document shows EXTRACTION_FAILED; downstream parse/classify stalls.
  • Meaning: the OCR provider (Vision + Claude) errored. (OCR_PROVIDER=mock can no longer reach production: the worker refuses to start with it unless ALLOW_MOCK_AGENT_AI=true.)
  • Action: (1) confirm OCR_PROVIDER is the real provider in the server .env and the Vision/Anthropic credentials resolve; (2) re-enqueue the ocr-process job for the document; (3) if the provider is down, OCR has no automatic fallback — wait or retry.

Shared — Chatwoot messaging ​

  • Symptom: Settings → Integrations → Chatwoot is red/Unreachable; creating a Telegram inbox 503s; the Inbox composer 503s on Send.
  • Meaning: outbound calls to the Chatwoot API are failing. Most often a bot-protection / WAF challenge in front of the Chatwoot host answering our server-to-server requests with a 403 HTML page. Inbound webhooks keep working, so messages still arrive — only sending breaks.
  • Action: follow docs/runbooks/chatwoot-unreachable.md. Check chatwootFailureKind in the API logs first: BOT_PROTECTION → that runbook; AUTH → the token, see secret-rotation.md.
  • Workaround meanwhile: staff can reply from the Chatwoot UI. Those replies still reach the console via the chatwoot.mirror job, which the block does not touch.
  • Do not close until a reply sent from the console composer has been confirmed delivered — inbox creation succeeding does not prove the send path is healthy. Use Internal note mode to test against a live client thread without messaging anyone.
  • A 502 (not 503) from the composer is a different fault: Chatwoot answered and refused the message. Wrong runbook.

Shared — a client's file or message did not arrive (since 28 Oct 2026, stage review S2-05) ​

  • Bell: "a file the client sent could not be received" (CLIENT_FILE_NOT_RECEIVED): an attachment on a client's message did not download from Chatwoot after three tries, or could not be stored. Nothing was filed, but the client may already have had "we've received your documents". The conversation's history has a chatwoot.attachment.not_received row naming the file and the error. It goes to the company's assignee (assigned executive, else the group's relationship manager), or the Platform Admins when the thread has no company or no assignee. Action: ask the client to send that file again.
  • Bell: "Messages from Chatwoot inbox N (account A) are being dropped" (CHATWOOT_UNKNOWN_INBOX): a Chatwoot inbox no BreezyCorp inbox matches (or more than one does). Every message to it is answered ignored and audited as chatwoot.unmatched_inbox. Each Platform Admin is told once per inbox. Action: connect the inbox under Settings → Integrations, or take the agent bot off that inbox in Chatwoot.
  • A typed message read as payroll data that could not become a document now fails the msg.inbound-text job, which is retried (3 times, then the failed-jobs list), instead of being swallowed.

Financial Statements ​

Engagement stuck at NOTES_ERROR ​

  • Symptom: engagement status is NOTES_ERROR after notes generation.
  • Where: NotesService.generate (packages/domain/src/financial-statements/services/notes.service.ts).
  • Meaning: by design — a Claude/parse failure transitions to the recoverableNOTES_ERROR state and writes a fs.notes.failed audit event. The job completes (it does not retry a non-deterministic LLM call).
  • Action: read the fs.notes.failed audit reason. Fix the input if needed, then re-trigger notes generation (POST /ops/financial-statements/engagements/:id/notes/auto-generate). The FSM allows NOTES_ERROR → STATEMENTS_GENERATED → NOTES_COMPLETE.

Client requested a revision ​

  • Symptom: an engagement at NOTES_COMPLETE reverts to STATEMENTS_GENERATED.
  • Meaning: by design — the client used the portal "request revision" action (fs.client.revision_requested audit). The preparer regenerates notes after addressing the comment in the audit eventDataJson.

TB upload "succeeds" but mapping shows nothing ​

  • Meaning: the trial-balance parse produced zero rows (bad file) — see the fs.trial_balance.parse_failed audit reason.
  • Action: re-upload a clean CSV/XLSX.

XBRL Filing ​

Document ingest rejected as "not a recognizable financial statement" ​

  • Symptom: XbrlSourceDocument.parseStatus = FAILED; filing stays in DRAFT.
  • Where: DocumentIngestionService.runIngest pre-ingestion gate.
  • Meaning: by design — the document parsed but had no line items or only notes (no primary statement). A xbrl.document.ingest_rejected audit records the reason. The job is swallowed (no DLQ cycling) because a retry won't change the file.
  • Action: confirm the uploaded DOCX/PDF actually contains a balance sheet / P&L, then re-ingest the corrected document.

Export blocked (HTTP 422) ​

  • Meaning: the export gate (Rules 2 & 3) refuses output until validation has passed and every tag mapping is human-confirmed (assertExportAllowed).
  • Action: run validation; confirm any unconfirmed mappings; retry the export. This gate is non-negotiable — do not bypass it.

Worker / queue health ​

  • Dead-lettered jobs — a job that exhausts its retries is routed to the dead-letter queue (configured on every ad-hoc queue in apps/worker/src/handlers/index.ts). A single consumer logs it at error level (worker.job.dead_letter), increments the in-process worker.job.dead_letter counter, captures it to Sentry, and writes an append-only audit_events row (eventType = 'worker.job.dead_letter', entityType = 'PgBossJob').
    • To review dead-lettered jobs: query audit_events WHERE event_type = 'worker.job.dead_letter' (since SEC-5, 27 Oct 2026, the event_data_json holds the queue, the source job id, the payload's ids as payload and its key names as payloadKeys, and the failure's message as output.message — never the payload's values, which can hold a sign-in code, an address or pay), or grep worker logs for worker.job.dead_letter. To see the full payload, read the failed job itself in pgboss.job (by sourceJobId) as the migrate role.
    • Note: handlers that catch their own errors (most send/notify handlers) complete normally and never dead-letter — their failures are already logged inside the handler. Dead-lettering captures the "gave up after retries" case (e.g. parse-bank-statement, xbrl ingest) — the genuinely silent-failure path before this existed.
  • Graceful restart: the worker drains in-flight jobs for up to 2 minutes on SIGTERM; the API readiness probe returns 503 during shutdown so the LB pulls it from rotation. The worker also flushes buffered Sentry events before exit.

Observability — Sentry ​

  • Default state: OFF. Sentry initialisation is a guarded no-op until SENTRY_DSN (API + worker) and NEXT_PUBLIC_SENTRY_DSN (web) are set. Dev, CI, and preview builds run with no Sentry traffic. See .env.example for the full var list.
  • To enable in production: stand up the self-hosted Sentry instance, create a project, and set SENTRY_DSN + NEXT_PUBLIC_SENTRY_DSN (usually the same project DSN) in the server environment + the web build env. NEXT_PUBLIC_* is baked at build time, so the web service must be rebuilt after setting it. Optionally set SENTRY_TRACES_SAMPLE_RATE (0..1; default 0 = errors only) and SENTRY_RELEASE.
  • What is captured: API 5xx errors (via the error-handler plugin), worker pg-boss errors + dead-lettered jobs, and web server/client errors. 4xx client errors are not captured. PII is scrubbed twice — Pino log redaction plus a Sentry beforeSend denylist (packages/observability/src/sentry.ts) — before any event leaves the process.
  • Not yet wired (follow-ups): withSentryConfig for web source-map upload (needs SENTRY_ORG / SENTRY_PROJECT / auth token), and a Prometheus/OTel exporter behind the @breezycorp/observability metrics facade. An external uptime monitor should poll /health/live + /health/ready independently of Sentry.

Internal use only — BreezyCorp