# Async processing governance

**Scope:** Laravel queues, tenant-safe jobs, retries, dead letters, long-running work.  
**North star:** Predictable backlog behavior and **no silent cross-tenant execution** (ADR 0004).

---

## 1. Queue separation strategy

**Current state (v1)**

- Default Laravel queue connection (`QUEUE_CONNECTION` in env); jobs share the default queue unless `onQueue()` is applied per job class.

**Recommended evolution (when load demands)**

| Queue (name) | Workloads | Rationale |
|--------------|-----------|-----------|
| `default` | Quick domain events, light jobs | Lowest latency |
| `exports` | `ExportProcessorJob` | Isolate long CPU/IO from interactive latency |
| `webhooks-outbound` | `DeliverWebhookJob` | Isolate HTTP wait/retry storms |
| `payroll` | `GeneratePayrollSnapshotsJob` / payroll batch work | Protect OLTP from heavy payroll |
| `integrations` | Biometric / replay / bulk integration | Operational tuning separate from HR |

**Convention:** set queue name in the job class constructor (`$this->onQueue('exports')`) *or* centralize mapping in a service provider once conventions are agreed.

---

## 2. Retry policies

| Job | Pattern | Notes |
|-----|---------|--------|
| `DeliverWebhookJob` | High `tries` + `release()` backoff in executor | Bounded by business dead-letter policy, not infinite success retries |
| `ExportProcessorJob` | Default queue retries | Should align with idempotent claim/processing (see export ADR) |
| `RunAttendanceExceptionDetectionJob` | Default | Safe to retry if handlers are idempotent per (tenant, date) |

**Rules**

1. **Idempotency first:** retries are safe only if `handle()` can run twice without corrupting state.
2. **Exponential backoff** for outbound HTTP; avoid tight spin on 429/5xx from partners.
3. **Max attempts** for non-idempotent paths must be **low** with explicit human follow-up.

---

## 3. Dead-letter strategy

- Webhook deliveries: status `dead_letter` with operational commands documented (`ops:webhooks:retry-failed`, `ops:webhooks:dead-letter-inspect`).
- **Ownership:** integrations platform module owns replay semantics and audit.

**Principle:** dead-letter is a **first-class outcome**, not an error log line—monitor rate and age.

---

## 4. Long-running job handling

- **Exports / payroll:** may exceed “interactive” thresholds; workers should use `--timeout` greater than worst-case, or split work (chunk jobs).
- **Webhook delivery:** wall-clock bound by HTTP timeout in executor, not PHP max execution alone.

**Standards**

- Log **correlation IDs** (request / export / delivery id) in structured logs where available.
- Prefer **after-commit dispatch** for work triggered from DB transactions (stabilization ADR 0006 pattern).

---

## 5. Batch processing standards

- Use `chunkById` with conservative sizes (codebase uses **100–500** depending on query cost).
- Never load unbounded tenant collections into memory for “all tenants” cron—use cursor/chunk at tenant level, then bounded work per tenant.

---

## 6. Queue naming conventions

```
{domain}.{verb}
```

Examples aligned with recommended queues:

- `exports.process`
- `integrations.webhook.deliver`
- `payroll.snapshots.generate`
- `hr.attendance.exceptions.detect`
- `hr.attendance.biometric.process`

**Laravel queue name** (worker `--queue=`) should be **short and stable**: `exports`, `webhooks`, `payroll`, `integrations`, `default`.

**Payload:** always include **`tenantId`** for tenant-scoped jobs, even if a second id (exportId, deliveryId) is present.

---

## References

- [ADR 0004: Queue tenant context strategy](../architecture/adr/0004-queue-tenant-context-strategy.md)
- [ADR 0006: After-commit external effects](../architecture/adr/0006-after-commit-external-effects.md)
- `app/Modules/Integrations/Platform/Jobs/DeliverWebhookJob.php`
- `app/Modules/Exports/Jobs/ExportProcessorJob.php`
- `docs/audits/queue-consistency-notes.md`