Skip to content

Commit f15d6f6

Browse files
os-zhuangclaude
andcommitted
feat(approvals): ADR-0042 SLA auto-escalation + ADR-0041 mechanical landing
SLA escalation (ADR-0042, closes #1742): plugin-approvals owns a jobs-backed scanner — interval job 'approvals-sla-escalation' (5 min default, escalationScanIntervalMs option) plus a catch-up sweep at kernel:ready (the job service starts after this plugin, so wiring defers to the ready hook). Each overdue pending request escalates AT MOST ONCE: the 'escalate' audit row is the idempotency marker, written audit-first, so a crash between audit and mutation degrades to "logged but not executed" rather than a double machine decision. Actions: notify (default; approvers + escalateTo), reassign (replace approver set with escalateTo), auto_approve / auto_reject via decide() as the reserved actor 'system:sla' — the timeline always shows two rows (escalate, then the decision) so policy decisions are distinguishable from human ones. The clock is created_at + escalation.timeoutHours with no request_info pause in v1 (recorded as a future option in the ADR). ADR-0041 mechanical landing: plugin-trigger-record-change and plugin-trigger-schedule drop the plugin- prefix and become the trigger-* family (@objectstack/trigger-record-change, @objectstack/trigger-schedule); serve's nameMatch already accepted both spellings. ActionDescriptor gains maturity ('ga'|'beta'| 'reserved', default ga) so designers can grey out contract-ahead-of-runtime surfaces instead of letting authors build on them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 998c4e4 commit f15d6f6

26 files changed

Lines changed: 2222 additions & 19 deletions
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
"@objectstack/spec": minor
3+
"@objectstack/plugin-approvals": minor
4+
"@objectstack/trigger-record-change": minor
5+
"@objectstack/trigger-schedule": minor
6+
"@objectstack/cli": patch
7+
---
8+
9+
ADR-0042 SLA auto-escalation + ADR-0041 mechanical landing. plugin-approvals now owns a jobs-backed escalation scanner (`runEscalations`, interval job `approvals-sla-escalation` + boot catch-up): overdue pending requests escalate **at most once** (the `escalate` audit row is the idempotency marker, written audit-first) executing the node's `escalation.action` — notify / reassign-to-`escalateTo` / auto_approve / auto_reject as the reserved actor `system:sla`. The trigger packages drop their `plugin-` prefix (`@objectstack/trigger-record-change`, `@objectstack/trigger-schedule`) per ADR-0041, and `ActionDescriptor` gains an optional `maturity: 'ga' | 'beta' | 'reserved'` field so designers can grey out contract-ahead-of-runtime surfaces.
Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# ADR-0042: Approval SLA escalation — a jobs-backed scanner with audit-row idempotency
2+
3+
**Status**: Proposed (2026-06-12)
4+
**Deciders**: ObjectStack Protocol Architects
5+
**Builds on**: [ADR-0019](./0019-approval-as-flow-node.md) (approval as flow node), [ADR-0041](./0041-flow-trigger-family.md) (triggers vs jobs vs hooks — this is the canonical "jobs, not trigger" case), thread interactions (#1740)
6+
**Closes**: [#1742](https://github.com/objectstack-ai/framework/issues/1742)
7+
**Consumers**: `@objectstack/plugin-approvals` (scanner + execution), `@objectstack/service-job` (the clock), Console inbox (timeline rendering)
8+
9+
---
10+
11+
## TL;DR
12+
13+
`ApprovalEscalationSchema` (`timeoutHours`, `action: reassign | auto_approve |
14+
auto_reject | notify`, `escalateTo`, `notifySubmitter`) has been contract-
15+
complete for a while, and #1740 made the deadline *visible* (`sla_due_at`
16+
chips). Nothing *executes* it: a breached SLA changes a chip color and
17+
nothing else.
18+
19+
**Decision**: plugin-approvals owns a periodic scanner registered directly
20+
on the `job` service (`IJobService`) — per ADR-0041's boundary, a plugin-
21+
internal clock is a job, not a trigger. The scanner escalates each overdue
22+
pending request **at most once** (single-shot), using the `escalate` audit
23+
row itself as the idempotency marker — no schema migration. Machine
24+
decisions are recorded under the reserved actor id **`system:sla`**. The
25+
SLA clock runs from `created_at` and does **not** pause during
26+
`request_info` round-trips in v1 (recorded as a future option).
27+
28+
## Context (verified 2026-06-12)
29+
30+
- Execution primitives all exist after #1740: `decide()` (with
31+
`context.isSystem` bypassing the approver check), approver-slot
32+
rewriting (reassign), `notify()` via the optional messaging surface, and
33+
the `escalate` member of the audit-action enum (both contract and
34+
`sys_approval_action` select).
35+
- `sla_due_at = created_at + escalation.timeoutHours` is already computed
36+
by the row mapper and rendered by the Console.
37+
- The runtime auto-loads `service-job` (`JobServicePlugin`, registered as
38+
service **`job`**) with interval/cron/once adapters.
39+
- The request table has no escalation bookkeeping column, and the
40+
append-only `sys_approval_action` table is already the audit source of
41+
truth.
42+
43+
## The three policy decisions
44+
45+
### 1. Idempotency & multi-instance safety → single-shot, audit-row marker
46+
47+
An escalation fires **at most once per request, ever**. The marker is the
48+
`escalate` audit row, written **before** any mutation (the audit-first
49+
ordering #1740 established for `reassign`):
50+
51+
```
52+
scan: overdue && no prior 'escalate' action → escalate
53+
```
54+
55+
Rationale:
56+
57+
- **No schema migration.** An `escalated_at` column would need a column add
58+
across drivers; the audit row is already durable, queryable, and
59+
tenant-scoped.
60+
- **Crash-safe.** If the process dies between the audit insert and the
61+
mutation, the next scan skips the request (marker present) — the failure
62+
mode is "escalation logged but not executed", which an operator can see
63+
on the timeline, rather than a double `auto_approve`.
64+
- **Multi-instance**: the in-process job adapters run one scheduler per
65+
node. Two nodes could race between the marker check and the insert; the
66+
window is milliseconds and the worst case is a duplicate *notify* (the
67+
mutating actions are guarded again by `decide()`'s pending-status check,
68+
which makes the second decision throw `INVALID_STATE`). A distributed
69+
lock is deliberately out of scope until clustered deployments demand it
70+
(`service-cluster` exists when that day comes).
71+
72+
Repeat/laddered escalation (re-escalate every N hours, multi-level chains)
73+
is a recorded non-goal for v1 — single-shot covers the dominant "don't let
74+
it rot silently" need.
75+
76+
### 2. Machine decisions → reserved actor `system:sla`
77+
78+
`auto_approve` / `auto_reject` call `decide()` with
79+
`actorId: 'system:sla'` under a system context. Consequences:
80+
81+
- The audit trail shows two rows: `escalate` (what policy fired, with the
82+
configured action in the comment) followed by `approve`/`reject` **by
83+
`system:sla`** — a reader can always distinguish a human decision from a
84+
policy decision.
85+
- Clients render `system:sla` as a localized display name ("System (SLA)")
86+
— it is a reserved identity, never a `sys_user` row.
87+
- Deployments that must forbid machine decisions simply don't author
88+
`auto_approve`/`auto_reject` in their flows; a platform-level kill switch
89+
is deferred until a compliance requirement actually arrives.
90+
91+
### 3. The clock → `created_at`, no pause in v1
92+
93+
The deadline stays `created_at + timeoutHours` — exactly what the Console
94+
already shows. A `request_info` round-trip does **not** pause or re-arm the
95+
clock: the approver who needs more material can see the deadline coming and
96+
the submitter is incentivised to answer fast. ServiceNow-style SLA pause
97+
("clock stops while waiting on the requester") is a future option that
98+
would require per-request clock bookkeeping (schema change) and is exactly
99+
the kind of speculative state this ADR avoids; it gets built when a real
100+
tenant asks.
101+
102+
## Mechanics
103+
104+
- **Wiring**: `ApprovalsServicePlugin.start()` resolves the `job` service
105+
(optional, like `messaging`); when present it schedules an interval job
106+
`approvals-sla-escalation` (default every 5 min,
107+
`escalationScanIntervalMs` plugin option) and fires one **catch-up scan
108+
at boot** so restarts don't extend a breach by a scan period. `stop()`
109+
cancels the job. No job service → SLA stays display-only, exactly as
110+
today.
111+
- **Scan** (`runEscalations()`): list pending requests (the same capped
112+
read the inbox uses), keep rows whose node config declares
113+
`escalation.timeoutHours` and whose deadline has passed, skip rows with a
114+
prior `escalate` action, then per action:
115+
- `notify` (default): message pending approvers + `escalateTo`;
116+
- `reassign`: replace `pending_approvers` with `escalateTo` (falls back
117+
to `notify` when `escalateTo` is missing) and notify the new approver;
118+
- `auto_approve` / `auto_reject`: `decide()` as `system:sla` — the owning
119+
flow resumes down the matching branch like any human decision;
120+
- in all cases, notify the submitter when `notifySubmitter !== false`.
121+
- **Failure isolation**: per-request try/catch; one bad row never stops the
122+
sweep. The scan logs `{scanned, escalated}`.
123+
124+
## Consequences
125+
126+
- A breached SLA now *does* something, closing the "red chip, no action"
127+
gap — and the timeline explains exactly what and why.
128+
- No migrations, no new services, no new packages: one service method, one
129+
optional wiring block, UI timeline rendering for `escalate` +
130+
`system:sla`.
131+
- The single-shot + audit-marker discipline gives clustered deployments a
132+
bounded, documented race (duplicate notify at worst) instead of a silent
133+
double-decision hazard.
134+
- Future work has clean seams: laddered escalation = relax the marker rule;
135+
SLA pause = add clock bookkeeping; compliance kill-switch = plugin
136+
option.
137+
138+
## References
139+
140+
- Schema: `ApprovalEscalationSchema`
141+
(`packages/spec/src/automation/approval.zod.ts`)
142+
- Primitives: `decide` / `notify` / audit-first ordering in
143+
`packages/plugins/plugin-approvals/src/approval-service.ts` (#1740)
144+
- Clock: `IJobService` (`packages/spec/src/contracts/job-service.ts`),
145+
registered as `job` by `@objectstack/service-job`
146+
- Boundary rationale: ADR-0041 §1 (plugin clocks are jobs, not triggers)

examples/app-showcase/src/flows/index.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -271,7 +271,7 @@ export const TaskCompletedSlackFlow = defineFlow({
271271
*
272272
* A `type: 'schedule'` flow whose start node carries an interval descriptor.
273273
* The automation engine parses that into a schedule binding; the schedule
274-
* trigger plugin (`@objectstack/plugin-trigger-schedule`, paired with the job
274+
* trigger plugin (`@objectstack/trigger-schedule`, paired with the job
275275
* service) registers a job that fires this flow every interval. Each tick runs
276276
* the `notify` node, dropping a fresh `sys_inbox_message` row — so the
277277
* scheduled fire is observable end-to-end with no manual `engine.execute()`.

packages/cli/package.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,8 @@
6565
"@objectstack/plugin-reports": "workspace:*",
6666
"@objectstack/plugin-security": "workspace:*",
6767
"@objectstack/plugin-sharing": "workspace:*",
68-
"@objectstack/plugin-trigger-record-change": "workspace:*",
69-
"@objectstack/plugin-trigger-schedule": "workspace:*",
68+
"@objectstack/trigger-record-change": "workspace:*",
69+
"@objectstack/trigger-schedule": "workspace:*",
7070
"@objectstack/plugin-webhooks": "workspace:*",
7171
"@objectstack/rest": "workspace:*",
7272
"@objectstack/runtime": "workspace:^",

packages/cli/src/commands/serve.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1422,12 +1422,12 @@ export default class Serve extends Command {
14221422
// the `FlowTrigger` wiring; these plugins are the concrete triggers:
14231423
// record-change (ObjectQL lifecycle hooks) + schedule (cron/interval
14241424
// via the job service — so pair `triggers` with `job`).
1425-
pkg: '@objectstack/plugin-trigger-record-change',
1425+
pkg: '@objectstack/trigger-record-change',
14261426
export: 'RecordChangeTriggerPlugin',
14271427
nameMatch: ['trigger-record-change', 'RecordChangeTriggerPlugin'],
14281428
extras: [
14291429
{
1430-
pkg: '@objectstack/plugin-trigger-schedule',
1430+
pkg: '@objectstack/trigger-schedule',
14311431
export: 'ScheduleTriggerPlugin',
14321432
nameMatch: ['trigger-schedule', 'ScheduleTriggerPlugin'],
14331433
},

packages/plugins/plugin-approvals/src/approval-service.test.ts

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -473,6 +473,82 @@ describe('ApprovalService (node era)', () => {
473473
expect(actions.filter(a => a.action === 'comment')).toHaveLength(2);
474474
});
475475

476+
// ── SLA escalation (ADR-0042) ───────────────────────────────────
477+
478+
function makeOverdue(reqId: string) {
479+
// Push created_at into the past so a small timeoutHours is breached.
480+
const row = engine._tables['sys_approval_request'].find(r => r.id === reqId)!;
481+
row.created_at = new Date(baseTime - 10 * 3600_000).toISOString();
482+
}
483+
484+
it('runEscalations: notify action messages approvers + escalateTo + submitter, once', async () => {
485+
const emitted: any[] = [];
486+
svc.attachMessaging({ async emit(input) { emitted.push(input); } });
487+
const req = await svc.openNodeRequest(
488+
openInput(['u9'], {}, { escalation: { timeoutHours: 2, action: 'notify', escalateTo: 'boss', notifySubmitter: true } }), CTX,
489+
);
490+
makeOverdue(req.id);
491+
const first = await svc.runEscalations();
492+
expect(first.escalated).toBe(1);
493+
expect(emitted.map(e => e.topic)).toEqual(['approval.sla_breached', 'approval.sla_breached']);
494+
expect(emitted[0].audience).toEqual(['u9', 'boss']);
495+
expect(emitted[1].audience).toEqual(['u1']); // submitter
496+
const actions = await svc.listActions(req.id, SYS);
497+
expect(actions.at(-1)).toMatchObject({ action: 'escalate', actor_id: 'system:sla', comment: 'notify → boss' });
498+
// Single-shot: second sweep is a no-op.
499+
const second = await svc.runEscalations();
500+
expect(second.escalated).toBe(0);
501+
expect(emitted).toHaveLength(2);
502+
});
503+
504+
it('runEscalations: auto_approve decides as system:sla and resumes the flow', async () => {
505+
const resumed: any[] = [];
506+
svc.attachAutomation({ async resume(runId, signal) { resumed.push({ runId, signal }); } });
507+
const req = await svc.openNodeRequest(
508+
openInput(['u9'], {}, { escalation: { timeoutHours: 1, action: 'auto_approve', notifySubmitter: false } }), CTX,
509+
);
510+
makeOverdue(req.id);
511+
const out = await svc.runEscalations();
512+
expect(out.escalated).toBe(1);
513+
const fresh = await svc.getRequest(req.id, SYS);
514+
expect(fresh?.status).toBe('approved');
515+
expect(resumed[0]).toMatchObject({ runId: 'run_1', signal: { branchLabel: 'approve' } });
516+
const actions = await svc.listActions(req.id, SYS);
517+
expect(actions.map(a => a.action)).toEqual(['submit', 'escalate', 'approve']);
518+
expect(actions.at(-1)?.actor_id).toBe('system:sla');
519+
});
520+
521+
it('runEscalations: auto_reject decides as system:sla', async () => {
522+
const req = await svc.openNodeRequest(
523+
openInput(['u9'], {}, { escalation: { timeoutHours: 1, action: 'auto_reject', notifySubmitter: false } }), CTX,
524+
);
525+
makeOverdue(req.id);
526+
await svc.runEscalations();
527+
const fresh = await svc.getRequest(req.id, SYS);
528+
expect(fresh?.status).toBe('rejected');
529+
});
530+
531+
it('runEscalations: reassign replaces the approver set with escalateTo', async () => {
532+
const req = await svc.openNodeRequest(
533+
openInput(['u9', 'u2'], {}, { escalation: { timeoutHours: 1, action: 'reassign', escalateTo: 'boss', notifySubmitter: false } }), CTX,
534+
);
535+
makeOverdue(req.id);
536+
await svc.runEscalations();
537+
const fresh = await svc.getRequest(req.id, SYS);
538+
expect(fresh?.status).toBe('pending');
539+
expect(fresh?.pending_approvers).toEqual(['boss']);
540+
});
541+
542+
it('runEscalations: skips requests that are not yet due or have no SLA', async () => {
543+
await svc.openNodeRequest(
544+
openInput(['u9'], {}, { escalation: { timeoutHours: 1000, action: 'auto_approve' } }), CTX,
545+
);
546+
await svc.openNodeRequest(openInput(['u9'], { recordId: 'opp2', record: { id: 'opp2' } }), CTX);
547+
const out = await svc.runEscalations();
548+
expect(out.scanned).toBe(2);
549+
expect(out.escalated).toBe(0);
550+
});
551+
476552
// ── SLA + flow steps ────────────────────────────────────────────
477553

478554
it('rows expose sla_due_at when the node declares escalation.timeoutHours', async () => {

0 commit comments

Comments
 (0)