BreezyCorp Handbook

Appearance

Runbook: Chatwoot Unreachable (bot-protection / WAF block) ​

Alert: Settings → Integrations → Chatwoot card shows Unreachable, or API logs carry chatwootFailureKind: "BOT_PROTECTION"Severity: Critical — this breaks messaging in both directions

The block can be transient. Observed 2026-07-31: every request from hermes was answered with a managed challenge for a period, then cleared on its own — an inbox created minutes later succeeded. A challenge that comes and goes is harder to diagnose than a permanent one, because the obvious retry appears to "fix" it and the real cause goes unrecorded. So: do not close this incident just because the probe is green now. Confirm from the logs whether a BOT_PROTECTION failure was ever recorded, and if so apply the allowlist below preventively.

Why this is Critical ​

Two things break, and only one of them is obvious.

The obvious one is that creating a Telegram inbox during onboarding fails. The other is that no operator can reply to a client: the console composer (POST /ops/messaging/conversations/:conversationId/messages) rides the same outbound path via ChatwootClient.sendMessage. Meanwhile the inbound webhook keeps working — Chatwoot pushes to us — so client messages keep arriving and nothing goes out. That asymmetry is what makes the incident easy to under-rate.

It does at least fail loudly now. The composer is a foreground caller, so an operator meets a 503 toast on the first attempt rather than discovering days later that replies never left. (Before the composer existed, sendMessage had no callers at all, and this section's original claim that "every agent reply has been silently failing" described a path nothing actually walked.)

Workaround while the block is in force: staff can reply from the Chatwoot UI directly. Those replies still reach the console — they arrive over the inbound webhook and are recorded by the chatwoot.mirror job, which the block does not touch. Slower, but not blind.

Always check the outbound half before closing this incident.

Symptoms ​

  • POST /ops/messaging/inboxes/telegram returns 503; the console toast reads "Chatwoot is unreachable — the API host is being blocked by a bot-protection challenge (HTTP 403). Contact an admin."
  • The inbox composer shows the same 503 toast on Send, and the typed message is deliberately left in the box rather than cleared
  • API log line chatwoot inbox creation failed or chatwoot message send failed, with chatwootStatus: 403, chatwootFailureKind: "BOT_PROTECTION", and a chatwootBodyExcerpt beginning <!DOCTYPE html> … Just a moment …
  • API log line chatwoot reachability probe failed (the GET /integrations/chatwoot probe, at most once per 60s)
  • Settings → Integrations → Chatwoot card is red / Unreachable

If the failure kind is AUTH rather than BOT_PROTECTION, this is the wrong runbook — that is a token problem, see secret-rotation.md.

Diagnose ​

Run these on hermes01. The api container NATs through the host, so both present the same public IP to the WAF. Running them from your laptop proves nothing: your IP is not the one being challenged.

The api image has no curl. Use node -e for anything that needs the container's environment; use the host's curl for the unauthenticated probes.

bash
# 1. What egress IP does the WAF actually see?  (on the host)
curl -s https://ifconfig.me; echo
#   recorded value for hermes01: 204.168.250.171

# 2. Is the host proxied through Cloudflare?
dig +short chat.breezycorp.app
#   172.67.x.x / 104.21.x.x  → Cloudflare-proxied ("orange cloud")

# 3. Is it a challenge, or is it Chatwoot itself saying no?
curl -sI https://chat.breezycorp.app/ | grep -iE '^(server|cf-mitigated|cf-ray):'
#   cf-mitigated: challenge  → confirmed WAF block, not our bug
#   no cf-mitigated, 200     → not being challenged right now (may still be transient)

# 4. The authoritative check — the exact call inbox creation makes, with the real
#    token, from the process that actually makes it.
docker exec breezycorp-api-1 node -e '
  const u = `${process.env.CHATWOOT_BASE_URL}/api/v1/accounts/${process.env.CHATWOOT_ACCOUNT_ID}/inboxes`;
  fetch(u, { headers: { api_access_token: process.env.CHATWOOT_API_ACCESS_TOKEN } })
    .then((r) => console.log(r.status, r.headers.get("cf-mitigated") ?? ""))
    .catch((e) => console.log("ERR", e.message));'
#   403 → this runbook
#   401 → token problem, NOT this runbook
#   200 → reachable right now

# 5. Has a block EVER been recorded?  A green step 4 does not rule it out.
docker logs breezycorp-api-1 --since 168h 2>&1 | grep -c 'BOT_PROTECTION'

A useful tell: under a managed challenge even GET / returns 403. If the site root is fine and only /api/ fails, suspect a path-scoped WAF rule rather than a zone-wide one.

Remediate ​

This is not a code change. It needs someone with Cloudflare dashboard access on zone breezycorp.app. Name that owner before starting.

Option A — WAF custom rule (preferred: narrow and path-scoped) ​

Security → WAF → Custom rules → Create rule, name it Allow BreezyCorp hermes API:

(ip.src eq 204.168.250.171
 and http.host eq "chat.breezycorp.app"
 and starts_with(http.request.uri.path, "/api/"))

Action Skip, with these phases checked:

  • All remaining custom rules
  • Managed rules (WAF)
  • Rate limiting rules
  • Super Bot Fight Mode / Bot Fight Mode
  • Browser Integrity Check
  • Security Level

Move the rule to position 1 in the rule order.

Option B — IP Access Rule (fallback: blunter) ​

Security → WAF → Tools → IP Access Rules → add 204.168.250.171, action Allow, scoped to this zone.

Use this when the account is on the Free plan: there, Bot Fight Mode is a zone toggle that a custom-rule Skip cannot reliably bypass, whereas an IP Access Rule Allow does. The cost is that it whitelists the whole host for that IP, not just /api/.

⚠️ Which of A or B actually works depends on the Cloudflare plan tier — the available Skip phases differ. Verify with diagnose step 4 before declaring victory. A rule that saves cleanly but does not take effect looks identical in the dashboard.

Option C — remove Cloudflare from the path (durable fix) ​

A and B are unblocks; this is the real answer. Route hermes → Chatwoot over the existing WireGuard mesh instead of the public internet:

  • add chat.breezycorp.app:<origin private IP> to extra_hosts on the api and worker services in docker-compose.hermes.yml, or
  • run a Cloudflare Tunnel to the origin.

TLS implication: SNI is preserved by extra_hosts, so the origin must present a certificate valid for chat.breezycorp.app — a real cert or a Cloudflare Origin CA cert. Server-to-server traffic then never touches the edge, so no WAF rule can break it again.

The egress IP is not stable ​

204.168.250.171 is hermes01's egress IP as of 2026-07-31. It is not guaranteed across a host rebuild, a floating-IP move, or provider maintenance. A silent change reproduces this incident exactly, with no code or config having changed on our side.

  • Recorded as HERMES_EGRESS_IP in deploy/deploy.conf.example.
  • Re-verify with diagnose step 1 after any host-level change.
  • Option C removes this failure mode entirely.

Verify ​

  1. Diagnose step 4 returns 200 — and step 5 records no new BOT_PROTECTION lines over the following day. A single green probe does not prove a transient challenge is gone.

  2. Settings → Integrations → Chatwoot goes green. The probe is memoized for 60s — hit GET /ops/messaging/integrations/chatwoot?probe=force or wait it out.

  3. Create a Telegram inbox end-to-end from a company's Settings tab.

  4. Prove the outbound half. Open a conversation at /dashboard/inbox, type into the composer and send. Confirm it appears in the transcript and lands on the client's device. Step 3 passing does not imply step 4 — inbox creation and message send are different Chatwoot endpoints, and only this one is what operators use all day.

    Read the failure code, not just "it errored":

    • 503 — the block is still in force. Same root cause; go back to Remediate.
    • 502 — Chatwoot answered and refused the message. Different problem entirely (a resolved conversation, a stale inbox); this runbook is done.

    Use Internal note mode to run this against a live client conversation. It exercises the identical outbound API call — same endpoint, same credentials, same WAF path — but Chatwoot never delivers a private: true message to the client. There is no need to message a real client to verify an incident is over.

Rollback ​

Delete the custom rule (Option A) or the IP Access Rule (Option B). No application state changes, so there is nothing to undo on our side. For Option C, remove the extra_hosts entries and redeploy.

  • docs/reference/architecture/chatwoot-single-ingestion.md — why Chatwoot is the single messaging front door, and the risk row this incident realised
  • docs/runbooks/secret-rotation.md — for AUTH-kind failures instead

Internal use only — BreezyCorp