|
| 1 | +# ADR-0042: Approval SLA escalation — a jobs-backed scanner with audit-row idempotency |
| 2 | + |
| 3 | +**Status**: Proposed (2026-06-12) |
| 4 | +**Deciders**: ObjectStack Protocol Architects |
| 5 | +**Builds on**: [ADR-0019](./0019-approval-as-flow-node.md) (approval as flow node), [ADR-0041](./0041-flow-trigger-family.md) (triggers vs jobs vs hooks — this is the canonical "jobs, not trigger" case), thread interactions (#1740) |
| 6 | +**Closes**: [#1742](https://github.com/objectstack-ai/framework/issues/1742) |
| 7 | +**Consumers**: `@objectstack/plugin-approvals` (scanner + execution), `@objectstack/service-job` (the clock), Console inbox (timeline rendering) |
| 8 | + |
| 9 | +--- |
| 10 | + |
| 11 | +## TL;DR |
| 12 | + |
| 13 | +`ApprovalEscalationSchema` (`timeoutHours`, `action: reassign | auto_approve | |
| 14 | +auto_reject | notify`, `escalateTo`, `notifySubmitter`) has been contract- |
| 15 | +complete for a while, and #1740 made the deadline *visible* (`sla_due_at` |
| 16 | +chips). Nothing *executes* it: a breached SLA changes a chip color and |
| 17 | +nothing else. |
| 18 | + |
| 19 | +**Decision**: plugin-approvals owns a periodic scanner registered directly |
| 20 | +on the `job` service (`IJobService`) — per ADR-0041's boundary, a plugin- |
| 21 | +internal clock is a job, not a trigger. The scanner escalates each overdue |
| 22 | +pending request **at most once** (single-shot), using the `escalate` audit |
| 23 | +row itself as the idempotency marker — no schema migration. Machine |
| 24 | +decisions are recorded under the reserved actor id **`system:sla`**. The |
| 25 | +SLA clock runs from `created_at` and does **not** pause during |
| 26 | +`request_info` round-trips in v1 (recorded as a future option). |
| 27 | + |
| 28 | +## Context (verified 2026-06-12) |
| 29 | + |
| 30 | +- Execution primitives all exist after #1740: `decide()` (with |
| 31 | + `context.isSystem` bypassing the approver check), approver-slot |
| 32 | + rewriting (reassign), `notify()` via the optional messaging surface, and |
| 33 | + the `escalate` member of the audit-action enum (both contract and |
| 34 | + `sys_approval_action` select). |
| 35 | +- `sla_due_at = created_at + escalation.timeoutHours` is already computed |
| 36 | + by the row mapper and rendered by the Console. |
| 37 | +- The runtime auto-loads `service-job` (`JobServicePlugin`, registered as |
| 38 | + service **`job`**) with interval/cron/once adapters. |
| 39 | +- The request table has no escalation bookkeeping column, and the |
| 40 | + append-only `sys_approval_action` table is already the audit source of |
| 41 | + truth. |
| 42 | + |
| 43 | +## The three policy decisions |
| 44 | + |
| 45 | +### 1. Idempotency & multi-instance safety → single-shot, audit-row marker |
| 46 | + |
| 47 | +An escalation fires **at most once per request, ever**. The marker is the |
| 48 | +`escalate` audit row, written **before** any mutation (the audit-first |
| 49 | +ordering #1740 established for `reassign`): |
| 50 | + |
| 51 | +``` |
| 52 | +scan: overdue && no prior 'escalate' action → escalate |
| 53 | +``` |
| 54 | + |
| 55 | +Rationale: |
| 56 | + |
| 57 | +- **No schema migration.** An `escalated_at` column would need a column add |
| 58 | + across drivers; the audit row is already durable, queryable, and |
| 59 | + tenant-scoped. |
| 60 | +- **Crash-safe.** If the process dies between the audit insert and the |
| 61 | + mutation, the next scan skips the request (marker present) — the failure |
| 62 | + mode is "escalation logged but not executed", which an operator can see |
| 63 | + on the timeline, rather than a double `auto_approve`. |
| 64 | +- **Multi-instance**: the in-process job adapters run one scheduler per |
| 65 | + node. Two nodes could race between the marker check and the insert; the |
| 66 | + window is milliseconds and the worst case is a duplicate *notify* (the |
| 67 | + mutating actions are guarded again by `decide()`'s pending-status check, |
| 68 | + which makes the second decision throw `INVALID_STATE`). A distributed |
| 69 | + lock is deliberately out of scope until clustered deployments demand it |
| 70 | + (`service-cluster` exists when that day comes). |
| 71 | + |
| 72 | +Repeat/laddered escalation (re-escalate every N hours, multi-level chains) |
| 73 | +is a recorded non-goal for v1 — single-shot covers the dominant "don't let |
| 74 | +it rot silently" need. |
| 75 | + |
| 76 | +### 2. Machine decisions → reserved actor `system:sla` |
| 77 | + |
| 78 | +`auto_approve` / `auto_reject` call `decide()` with |
| 79 | +`actorId: 'system:sla'` under a system context. Consequences: |
| 80 | + |
| 81 | +- The audit trail shows two rows: `escalate` (what policy fired, with the |
| 82 | + configured action in the comment) followed by `approve`/`reject` **by |
| 83 | + `system:sla`** — a reader can always distinguish a human decision from a |
| 84 | + policy decision. |
| 85 | +- Clients render `system:sla` as a localized display name ("System (SLA)") |
| 86 | + — it is a reserved identity, never a `sys_user` row. |
| 87 | +- Deployments that must forbid machine decisions simply don't author |
| 88 | + `auto_approve`/`auto_reject` in their flows; a platform-level kill switch |
| 89 | + is deferred until a compliance requirement actually arrives. |
| 90 | + |
| 91 | +### 3. The clock → `created_at`, no pause in v1 |
| 92 | + |
| 93 | +The deadline stays `created_at + timeoutHours` — exactly what the Console |
| 94 | +already shows. A `request_info` round-trip does **not** pause or re-arm the |
| 95 | +clock: the approver who needs more material can see the deadline coming and |
| 96 | +the submitter is incentivised to answer fast. ServiceNow-style SLA pause |
| 97 | +("clock stops while waiting on the requester") is a future option that |
| 98 | +would require per-request clock bookkeeping (schema change) and is exactly |
| 99 | +the kind of speculative state this ADR avoids; it gets built when a real |
| 100 | +tenant asks. |
| 101 | + |
| 102 | +## Mechanics |
| 103 | + |
| 104 | +- **Wiring**: `ApprovalsServicePlugin.start()` resolves the `job` service |
| 105 | + (optional, like `messaging`); when present it schedules an interval job |
| 106 | + `approvals-sla-escalation` (default every 5 min, |
| 107 | + `escalationScanIntervalMs` plugin option) and fires one **catch-up scan |
| 108 | + at boot** so restarts don't extend a breach by a scan period. `stop()` |
| 109 | + cancels the job. No job service → SLA stays display-only, exactly as |
| 110 | + today. |
| 111 | +- **Scan** (`runEscalations()`): list pending requests (the same capped |
| 112 | + read the inbox uses), keep rows whose node config declares |
| 113 | + `escalation.timeoutHours` and whose deadline has passed, skip rows with a |
| 114 | + prior `escalate` action, then per action: |
| 115 | + - `notify` (default): message pending approvers + `escalateTo`; |
| 116 | + - `reassign`: replace `pending_approvers` with `escalateTo` (falls back |
| 117 | + to `notify` when `escalateTo` is missing) and notify the new approver; |
| 118 | + - `auto_approve` / `auto_reject`: `decide()` as `system:sla` — the owning |
| 119 | + flow resumes down the matching branch like any human decision; |
| 120 | + - in all cases, notify the submitter when `notifySubmitter !== false`. |
| 121 | +- **Failure isolation**: per-request try/catch; one bad row never stops the |
| 122 | + sweep. The scan logs `{scanned, escalated}`. |
| 123 | + |
| 124 | +## Consequences |
| 125 | + |
| 126 | +- A breached SLA now *does* something, closing the "red chip, no action" |
| 127 | + gap — and the timeline explains exactly what and why. |
| 128 | +- No migrations, no new services, no new packages: one service method, one |
| 129 | + optional wiring block, UI timeline rendering for `escalate` + |
| 130 | + `system:sla`. |
| 131 | +- The single-shot + audit-marker discipline gives clustered deployments a |
| 132 | + bounded, documented race (duplicate notify at worst) instead of a silent |
| 133 | + double-decision hazard. |
| 134 | +- Future work has clean seams: laddered escalation = relax the marker rule; |
| 135 | + SLA pause = add clock bookkeeping; compliance kill-switch = plugin |
| 136 | + option. |
| 137 | + |
| 138 | +## References |
| 139 | + |
| 140 | +- Schema: `ApprovalEscalationSchema` |
| 141 | + (`packages/spec/src/automation/approval.zod.ts`) |
| 142 | +- Primitives: `decide` / `notify` / audit-first ordering in |
| 143 | + `packages/plugins/plugin-approvals/src/approval-service.ts` (#1740) |
| 144 | +- Clock: `IJobService` (`packages/spec/src/contracts/job-service.ts`), |
| 145 | + registered as `job` by `@objectstack/service-job` |
| 146 | +- Boundary rationale: ADR-0041 §1 (plugin clocks are jobs, not triggers) |
0 commit comments