# Production telemetry expansion

**Current baseline:** Laravel logs, DB counts (`BusinessMetricsService`, diagnostics snapshots), scheduled `ops:readiness`, admin health pages.

**Goal:** Expand **measurable** telemetry without speculative dashboards—prioritize rollout-critical signals.

---

## 1. Metrics to add (priority order)

| Priority | Metric | Why |
|---------|--------|-----|
| P0 | Queue: `jobs` count, oldest job age, failed_jobs count/hour | Containment of backlog incidents |
| P0 | Exports: count by status; age of oldest `pending`/`processing` | User-visible SLA risk |
| P0 | Webhooks: deliveries/hour by `status`; dead-letter rate | Partner + integration trust |
| P1 | HTTP: P95 latency, 5xx rate for web + `/api/v1/*` | User experience |
| P1 | DB: connections in use; slow query count (top queries) | Scale signals |
| P2 | Payroll: snapshot generation duration per period (histogram) | Capacity planning |

---

## 2. Latency tracking

- **App layer:** Middleware or APM spans for representative routes (`dashboard`, `exports.store`, payroll period actions).
- **Jobs:** Timestamp `started_at` / `completed_at` already on exports extend to webhook `duration_ms` on deliveries for P95 aggregates.
- **Queue wait:** Difference between job `payload` enqueue time (if serialized) or `created_at` in `jobs` vs `started_at` when worker instrumentation exists—**gap today**; add when adopting Horizon or equivalent.

---

## 3. Queue lag monitoring

- Export lag: `(now - export.created_at)` for oldest `pending` row.
- Global lag: `jobs` oldest `created_at` (MySQL JSON column—vendor-specific parsing; or Redis queue depth per queue).

Align alerts with `operations.queue_backlog_warn`.

---

## 4. Webhook delivery timing metrics

- From `integration_webhook_deliveries`: `duration_ms`, `delivered_at - first_attempted_at` for success path.
- Aggregate: P50/P95 per **webhook_id** and per **tenant_id** to spot noisy partners.

---

## 5. Tenant health scoring (lightweight v1)

Internal composite (not customer-facing score) for CS/ops:

| Factor | Source | Weight (example) |
|--------|--------|------------------|
| Subscription health | subscription status + grace | High |
| Export backlog | pending count | Medium |
| Webhook DLQ | dead_letter count 7d | Medium |
| API errors | 4xx/5xx rate | Low–medium |
| Lifecycle | suspended/archived | Override flag |

**Use:** Sort support queue; do **not** auto-suspend from score without human review.

---

## Implementation notes

- Prefer **OpenTelemetry** or vendor APM with **low-cardinality** labels (tenant as attribute only if cardinality controlled).
- Never log **secrets** or full PII in metrics.

---

## References

- `docs/scalability/service-level-objectives.md`
- `app/Modules/Admin/Operations/Services/TenantDiagnosticsService.php`
- `docs/rollout/production-monitoring.md`