# Incident response

This document defines **first-response procedures** for platform-class incidents. It is not a substitute for a full company IR policy but ensures engineering **defaults** are safe and consistent.

Severity scales (example): **SEV1** user-wide / financial integrity, **SEV2** multi-tenant degraded, **SEV3** isolated tenant, **SEV4** minor.

---

## General response flow

1. **Acknowledge** — assign incident commander (IC) and comms owner.
2. **Stabilize** — stop bleeding (disable feature flag, throttle abuse, pause queue consumers if needed).
3. **Diagnose** — preserve logs, request ids, job ids; avoid destructive commands until scope is known.
4. **Remediate** — apply fix or rollback-forward plan; verify tenant boundaries after fix.
5. **Post-incident** — blameless review, action items, update runbooks.

---

## Tenant isolation incidents

**Symptoms:** Cross-tenant data visible, wrong tenant in exports, API returns other tenant’s rows.

**Immediate actions**

- [ ] Identify entry point (route, job, query scope missing).
- [ ] If exploit suspected: **rotate** affected API tokens, block abusive IPs, preserve audit trails.
- [ ] Scope: list tenants possibly affected; prepare customer communication template.

**Verification**

- [ ] Re-run affected flows under **two test tenants** with overlapping IDs.
- [ ] Add regression tests around the missing guard.

---

## Payroll integrity incidents

**Symptoms:** wrong amounts, duplicate snapshots, period unlocked after finalize.

**Immediate actions**

- [ ] Freeze writes: disable payroll mutations via flag or admin action if available.
- [ ] Snapshot DB or preserve rows for investigation.
- [ ] Do **not** silently patch financial rows without finance/legal alignment.

**Verification**

- [ ] Reconcile totals vs source attendance/HR inputs for impacted periods.
- [ ] Document corrective migrations or manual adjustments with **dual control** where required.

---

## Queue failures

**Symptoms:** backlog growth, repeated failures, poison messages.

**Actions**

- [ ] Identify failing job class and exception signature.
- [ ] Scale workers cautiously; do not infinite-retry poison payloads.
- [ ] Move to **dead-letter** / failed job store; fix code; replay with idempotency checks.

---

## Webhook delivery failures

**Symptoms:** dead-letter spikes, partner outages, signature errors.

**Actions**

- [ ] Use admin **integrations health** and ops commands (`ops:webhooks:dead-letter-inspect`, `ops:webhooks:retry-failed`) per runbook.
- [ ] Verify signing secrets not rotated without partner coordination.
- [ ] Confirm replays are **consumer-safe** (event id dedupe).

---

## Backup failures

**Symptoms:** missed backups, restore test failures, storage full.

**Actions**

- [ ] Follow ops backup commands / monitoring already in the platform’s operational posture.
- [ ] Treat backup failure as **SEV2+** until proven otherwise.
- [ ] Execute restore drill on non-prod before claiming resolution.

---

## Communication

- Customer-facing incidents require **coordinated** messaging (support + leadership).
- Engineering status updates on a fixed cadence (e.g. every 30 minutes during SEV1).

---

## References

- `docs/governance/release-management.md`
- `docs/governance/data-governance.md`
- `docs/governance/security-review-checklist.md`