# Production monitoring requirements

Defines **what** to observe and **initial alert thresholds**. Tune per environment; formalize in your APM/alerting vendor.

**Related:** [service-level objectives](../scalability/service-level-objectives.md), [v1 production launch checklist](../scalability/v1-production-launch-checklist.md), [incident-escalation-framework](./incident-escalation-framework.md).

---

## 1. Queue latency / backlog

| Metric | What | Alert threshold (initial) |
|--------|------|---------------------------|
| `jobs` table row count | Pending work | **>** `config('operations.queue_backlog_warn', 500)` sustained **15 min** |
| Failed jobs | Hard failures | Any **> 20/hour** or growth without drain |
| Oldest job age | Stuck queue | **> 30 min** P95 (instrument when possible) |

**Actions:** Scale workers; isolate hot queue (exports vs webhooks); check Redis/DB connectivity.

---

## 2. Webhook failures

| Metric | What | Alert threshold (initial) |
|--------|------|---------------------------|
| Delivery `failed` rate | Retrying | **> 10%** of attempts in 1h for **any** single webhook URL (tune) |
| New `dead_letter` rows | Exhausted | **> N/day** (set N from baseline) or sudden **5×** vs 7-day avg |
| Partner 5xx/429 | External | Same as [SLO escalation](../scalability/service-level-objectives.md) |

**Actions:** `ops:webhooks:dead-letter-inspect`; disable misconfigured webhook temporarily; partner comms.

---

## 3. Export backlog

| Metric | What | Alert threshold (initial) |
|--------|------|---------------------------|
| `exports.status = pending` | Waiting | **> 50** globally or **> 10** single tenant (tune) |
| Stuck `processing` | Worker loss | **>** 2h without `completed_at`/`failed_at` (align with `exports.processing_stale_minutes`) |
| Export failure rate | Quality | **> 2%/day** vs SLO |

**Actions:** Reclaim via next job run; scale export workers; investigate handler errors in logs.

---

## 4. Payroll failures / anomalies

| Metric | What | Alert threshold (initial) |
|--------|------|---------------------------|
| Period stuck `DRAFT` with lock contention | Generator no-op | Alerts on repeated “lock not acquired” logs if instrumented |
| Finalize 422 spike | Double-finalize attempts | Unusual rate vs baseline |
| Protection blocks | Integrity | Sudden spike in `PayrollProtection` denials (possible misconfig) |

**Actions:** Do not override finalized periods; use `docs/resilience/recovery-drills.md` payroll drill pattern.

---

## 5. API abuse spikes

| Metric | What | Alert threshold (initial) |
|--------|------|---------------------------|
| `api_audit_logs` 429/401 | Auth/rate | Spike **3×** hourly vs 7d median per tenant or global |
| Same token path volume | Abuse | Anomaly detection (manual first: top N tenants by row count) |

**Actions:** Rotate/revoke tokens; rate limit; `ApiSecurityMonitorService` rules if enabled.

---

## 6. Tenant lifecycle anomalies

| Metric | What | Alert threshold (initial) |
|--------|------|---------------------------|
| Bulk `suspended` / `archived` | Ops or bug | **> 5** tenants in 1h without change ticket |
| Onboarding stuck | Product | Tenants `lifecycle_state = onboarding` **> 14d** without completion (tune) |

**Actions:** Audit `platform_audit_events`; verify billing/subscription sync.

---

## 7. Synthetic checks (recommended)

- Scheduled `GET /up` or deep health from outside the cluster.
- Weekly: `php artisan ops:readiness` result logged to metrics.

---

## References

- `app/Modules/Admin/Operations/Services/DeploymentReadinessService.php`
- `app/Modules/Admin/Operations/Services/BusinessMetricsService.php` (coarse internal counts)
- `config/operations.php`, `config/exports.php`