Incident runbook

How to declare, work, and close a platform incident. The firm commits to 99.9% monthly availability (43.2 minutes of error budget per 30 days).

Severity targets

| Severity | Acknowledge within | Resolve within | |----------|-------------------|----------------| | SEV1 | 15 minutes | 4 hours | | SEV2 | 1 hour | 24 hours | | SEV3 | 8 hours | 7 days |

The drill clock starts at openedAt. A drill passes only when both the acknowledgment and the resolution land inside their targets.

Declare (SUPER_ADMIN only)

1. POST /api/status with { "action": "open", "id", "title", "severity" }.

IDs must be unique — replays return 409.

2. Confirm the public page moves: GET /api/status (no auth) and /status.

Any open SEV1 reads as an outage; any open SEV2 reads as degraded.

Work

1. POST /api/status with { "action": "acknowledge", "id" } the moment a

human owns the incident. Re-acks are safe no-ops.

2. Keep firm data paths untouched: incident records live in the shared

platform namespace, never in firm rows.

3. Page the firm through the normal dispatch channels; do not improvise new

recipient lists mid-incident.

Resolve

1. POST /api/status with { "action": "resolve", "id" }. Skipped

acknowledgments are stamped at resolve time so the clock stays honest.

2. Verify /status returns to operational. Resolved rows stay in history —

the page derives state at read time, so nothing can stick on outage.

3. Read the drill verdict off the incident timestamps before the retro:

ack minutes and resolve minutes against the table above.

Client notice

Every successful write returns { incident, notice } alongside the incident. The notice is the client-facing message for that exact event — subject carries severity and title ([SEV1] Incident Opened: ...) for inbox triage; the body carries id, severity, reached-event timestamps, and the /status pointer. Forward it verbatim through the normal dispatch channels; do not reword severity or timestamps.

Rules

• Never resolve what is still broken to make the page green.
• Never store client or firm data in incident titles.
• A corrupt or missing history resolves to operational only when no open

incident exists; absence of evidence is not evidence of health.