Skip to content

Commit a8e4f3b

Browse files
os-zhuangclaude
andauthored
feat(spec,verify): prove-it-runs proof field + ratchet, binding 4 high-risk classes (ADR-0054 #1 + Phase 2) (#2045)
* feat(spec): add prove-it-runs proof field + ratchet to liveness gate (ADR-0054 #1) ADR-0054 follow-up (1): the liveness ledger's `live` meant only a static consumer pointer — necessary but not sufficient, since a property can be live at every layer yet broken end-to-end. This adds the third leg: high-risk authorable properties must carry a `proof` (a dogfood test reference) that asserts the runtime outcome. - proof-registry.mts: the authoritative high-risk-class list (field types, analytics, RLS, flow nodes, form widgets) + which classes the ratchet enforces this phase. Field types and RLS are bound (matrix exists AND surface governed); analytics/flow/form are listed-but-blocked with honest reasons (their surface isn't governed yet / no proof yet — Phase 2). - check-liveness.mts: a bound `live` entry must carry a valid `proof` of its own class. Validation is STATIC (file exists + declares the `@proof: <id>` tag) so the gate stays seconds-cheap; running the proof remains the dogfood gate's job. Reverse check flags unregistered `@proof:` tags. - Ledger: field.type → field-zoo proof; permission.rowLevelSecurity.using → #1994 RLS proof. Dogfood proofs self-declare their `@proof:` tag. - Gate now also triggers on packages/dogfood/** so deleting/renaming a proof re-runs the check and the dangling reference is caught. - 15 unit + wiring tests; README documents the contract and ratchet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XVdnfUAx85amkerym26vdx * chore: add changeset for the liveness prove-it-runs gate change Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XVdnfUAx85amkerym26vdx * feat(verify,spec): bind flow-node high-risk class with a runtime proof (ADR-0054 Phase 2) Phase 2 extends the prove-it-runs ratchet to the flow-node class. - verify: bootStack gains an opt-in `automation` flag that registers @objectstack/service-automation, so authored flows are pulled from the registry and POST /automation/:name/trigger runs their nodes. Without it, flow execution was unreachable through the harness (the dispatcher's automation routes resolved no service). Mirrors the existing `multiTenant`/`security` opt-ins; default off. - dogfood: a self-contained flow fixture (one object + one autolaunched flow whose update_record node stamps a record) + the flow-node proof. It authors the flow, triggers it over HTTP, and asserts both directions — the targeted record is stamped (node executed) AND a bystander is untouched (the input variable wired into the node filter, not a blanket update). Runs green end-to-end. - spec: flow-node class is now `bound` in proof-registry.mts; flow.nodes.type carries the proof; the liveness gate enforces it. Three classes now bound: field types, RLS, flow nodes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XVdnfUAx85amkerym26vdx * feat(spec): govern dataset + bind the analytics high-risk class (ADR-0054) Unlocks the analytics binding that was previously blocked because its authorable surface wasn't governed. - Govern the `dataset` metadata type: new liveness/dataset.json classifies all 19 authorable props with file:line evidence from the analytics service consumer audit (26 live, 1 dead — measures.certified has no runtime consumer). `dataset` added to GOVERNED in check-liveness.mts. - Bind the analytics class: dataset.dimensions.dateGranularity carries the tz-bucketing proof (#1982/#2018) — the property whose org-timezone day-bucket behavior the proof asserts. proof-registry.mts flips analytics to bound. - Four high-risk classes now CI-enforced: field types, RLS, flow nodes, analytics. - README governed-types + high-risk tables updated; changeset description updated. Verified: liveness gate green (dataset 27 classified); 17 registry tests; the analytics proof runs green; full dogfood suite 59/59. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XVdnfUAx85amkerym26vdx --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent af0df56 commit a8e4f3b

20 files changed

Lines changed: 816 additions & 12 deletions
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@objectstack/spec": patch
3+
---
4+
5+
Add the ADR-0054 "prove-it-runs" proof field + ratchet to the spec liveness gate. A `live` ledger entry may now carry a `proof` — a reference (`<file>#<proof-id>`) to a dogfood test that asserts the property's runtime behavior. A bound high-risk `live` property must carry a valid proof, validated statically by the liveness gate (the file exists and declares the matching `@proof:` tag). Four high-risk classes are bound this phase: field types (`field.type`), RLS (`permission.rowLevelSecurity.using`), flow nodes (`flow.nodes.type`), and analytics (`dataset.dimensions.dateGranularity`). The `dataset` metadata type is now governed (new `liveness/dataset.json`). The authoritative high-risk-class list lives in `scripts/liveness/proof-registry.mts`; see `liveness/README.md`.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@objectstack/verify": minor
3+
---
4+
5+
`bootStack` gains an opt-in `automation` boot option. When set, it registers `@objectstack/service-automation` so the app's authored flows are pulled from the registry and `POST /api/v1/automation/:name/trigger` actually executes their nodes against the real in-process stack. This makes flow-node execution + variable wiring verifiable end-to-end (ADR-0054 Phase 2), mirroring the existing `multiTenant` opt-in. Default is `false`, so the standard boot stays lean for apps that don't exercise flows.

.github/workflows/spec-liveness-check.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,18 @@ name: Spec Liveness Check
66
# property of a GOVERNED type to declare a liveness status with evidence in
77
# packages/spec/liveness/<type>.json. A new unclassified property fails the check (the
88
# ratchet: no new undeclared surface). See packages/spec/liveness/README.md.
9+
#
10+
# ADR-0054 (prove-it-runs): bound high-risk 'live' properties must carry a `proof`
11+
# pointing to a dogfood test that declares the matching `@proof:` tag. The gate also
12+
# triggers on packages/dogfood/** so deleting/renaming a proof re-runs this check and
13+
# the dangling reference is caught (the proof files live outside packages/spec/).
914

1015
on:
1116
pull_request:
1217
types: [opened, synchronize, reopened]
1318
paths:
1419
- 'packages/spec/**'
20+
- 'packages/dogfood/**'
1521

1622
permissions:
1723
contents: read

packages/dogfood/test/analytics-timezone.dogfood.test.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,12 @@
33
// GOLDEN REGRESSION for #1982 / #2018 — "organization timezone drives analytics
44
// date bucketing", exercised end-to-end through the real HTTP + service stack.
55
//
6+
// @proof: analytics-tz-bucketing
7+
// ADR-0054 runtime proof for the analytics high-risk class. Registered in
8+
// proof-registry.mts but NOT yet ledger-bound: the authorable surface
9+
// (dataset/report dimensions+measures) is not a GOVERNED liveness type yet, so
10+
// there is no entry to carry the proof. Binds once dataset/report are governed.
11+
//
612
// This is the test that would have caught #2018 before merge. That bug passed
713
// every static gate: it lived in the *integration* of NativeSQLStrategy routing
814
// + in-memory count + REST execution-context resolution. Only booting the app

packages/dogfood/test/field-zoo-roundtrip.dogfood.test.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,11 @@
33
// CAPABILITY-MATRIX golden test — every authorable field type must survive a
44
// real HTTP write → read round-trip.
55
//
6+
// @proof: field-type-roundtrip
7+
// ADR-0054 runtime proof for the field-type high-risk class. Referenced by the
8+
// liveness ledger entry `field.type` (packages/spec/liveness/field.json); the
9+
// spec liveness gate fails if this tag is removed. See proof-registry.mts.
10+
//
611
// `showcase_field_zoo` carries one field of (almost) every protocol FieldType.
712
// Until now it was only *static*-checked (the metadata bundle registers it);
813
// nothing wrote a record and read it back. But the platform's value is that an
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// Flow-node execution fixture — the deterministic ADR-0054 Phase-2 flow proof.
4+
//
5+
// A flow is `live` in the ledger because the automation engine *reads* its
6+
// nodes — but "reads" is not "runs correctly end-to-end". A node's value
7+
// crosses flow-trigger → variable context → CEL/template interpolation → the
8+
// data engine, and the break can live in any seam (e.g. an input variable that
9+
// never reaches a node's config, or an `update_record` that ignores its
10+
// filter). This fixture proves the integrated path with ZERO dependence on an
11+
// example app: one object `flow_note` and one `autolaunched` flow whose single
12+
// `update_record` node stamps `status: 'processed'` on the record whose id was
13+
// passed in as the `noteId` input variable.
14+
//
15+
// The proof asserts BOTH directions, mirroring the RLS fixture's rigor:
16+
// • the targeted record IS mutated → the node executed,
17+
// • a bystander record is NOT → the input variable actually flowed into
18+
// the node's filter (not a blanket update). A flow that didn't wire the
19+
// variable would either touch nothing or touch everything; only correct
20+
// execution + wiring flips exactly the target.
21+
22+
import { defineStack } from '@objectstack/spec';
23+
import { ObjectSchema, Field } from '@objectstack/spec/data';
24+
25+
/** The one object under test: a note the flow stamps as processed. */
26+
export const FlowNote = ObjectSchema.create({
27+
name: 'flow_note',
28+
label: 'Flow Note',
29+
pluralLabel: 'Flow Notes',
30+
fields: {
31+
name: Field.text({ label: 'Name', required: true }),
32+
status: Field.text({ label: 'Status' }),
33+
},
34+
});
35+
36+
/**
37+
* `flow_touch` — start → update_record → end. The `noteId` input variable is
38+
* interpolated into the update filter (`{noteId}` template), and the node sets
39+
* `status` to `processed`. Triggered via `POST /automation/flow_touch/trigger`
40+
* with `{ params: { noteId } }`.
41+
*/
42+
export const flowTouch = {
43+
name: 'flow_touch',
44+
label: 'Flow Touch',
45+
type: 'autolaunched',
46+
variables: [{ name: 'noteId', type: 'text', isInput: true }],
47+
nodes: [
48+
{ id: 'start', type: 'start', label: 'Start' },
49+
{
50+
id: 'mark_processed',
51+
type: 'update_record',
52+
label: 'Mark processed',
53+
config: {
54+
objectName: 'flow_note',
55+
filter: { id: '{noteId}' },
56+
fields: { status: 'processed' },
57+
},
58+
},
59+
{ id: 'end', type: 'end', label: 'End' },
60+
],
61+
edges: [
62+
{ id: 'e1', source: 'start', target: 'mark_processed' },
63+
{ id: 'e2', source: 'mark_processed', target: 'end' },
64+
],
65+
};
66+
67+
/** A minimal, self-contained app config the dogfood harness can boot. */
68+
export const flowFixtureStack = defineStack({
69+
manifest: {
70+
id: 'com.dogfood.flow_fixture',
71+
namespace: 'flow',
72+
version: '0.0.0',
73+
type: 'app',
74+
name: 'Flow Node Fixture',
75+
description: 'Single-object app whose flow exercises node execution + variable wiring (ADR-0054 Phase 2).',
76+
},
77+
objects: [FlowNote],
78+
flows: [flowTouch],
79+
});
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// FLOW NODE execution proof (ADR-0054 Phase 2), exercised end-to-end through the
4+
// real HTTP + automation stack.
5+
//
6+
// @proof: flow-node-execution
7+
// ADR-0054 runtime proof for the flow-node high-risk class (node execution +
8+
// variable wiring). Referenced by the liveness ledger entry `flow.nodes.type`
9+
// (packages/spec/liveness/flow.json); the spec liveness gate fails if this tag
10+
// is removed. See proof-registry.mts.
11+
//
12+
// A flow being `live` means the engine reads its nodes — necessary but not
13+
// sufficient. This authors a flow, triggers it over HTTP, and asserts the
14+
// observable runtime outcome: the `update_record` node ran AND the `noteId`
15+
// input variable wired into its filter, so EXACTLY the targeted record changed.
16+
17+
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
18+
import { bootStack, type VerifyStack } from '@objectstack/verify';
19+
import { flowFixtureStack } from './fixtures/flow-touch-fixture.js';
20+
21+
describe('objectstack verify FLOW: node execution + variable wiring (#flow-node)', () => {
22+
let stack: VerifyStack;
23+
let token: string;
24+
25+
beforeAll(async () => {
26+
// `automation: true` registers @objectstack/service-automation so the app's
27+
// flows are pulled from the registry and the trigger route runs them.
28+
stack = await bootStack(flowFixtureStack, { automation: true });
29+
token = await stack.signIn();
30+
}, 60_000);
31+
32+
afterAll(async () => {
33+
await stack?.stop();
34+
});
35+
36+
async function createNote(name: string): Promise<string> {
37+
const res = await stack.apiAs(token, 'POST', '/data/flow_note', { name, status: 'new' });
38+
expect(res.status, `create ${name} failed: ${res.status} ${await res.clone().text()}`).toBeLessThan(300);
39+
const j = (await res.json()) as { id?: string; record?: { id?: string } };
40+
const id = j.id ?? j.record?.id;
41+
expect(id, 'no id returned from create').toBeTruthy();
42+
return id as string;
43+
}
44+
45+
async function statusOf(id: string): Promise<unknown> {
46+
const res = await stack.apiAs(token, 'GET', `/data/flow_note/${id}`);
47+
expect(res.status).toBe(200);
48+
const j = (await res.json()) as { record?: Record<string, unknown> } & Record<string, unknown>;
49+
return (j.record ?? j).status;
50+
}
51+
52+
it('precondition: the automation service is wired and the flow is registered', async () => {
53+
// The flow-list route is served by the same dispatcher; if automation were
54+
// unregistered this would not return the flow (the whole proof would be moot).
55+
const res = await stack.apiAs(token, 'GET', '/automation/flow_touch');
56+
expect(res.status, `automation service not wired: ${res.status}`).toBe(200);
57+
});
58+
59+
it('runs the update_record node and wires the input variable into the filter', async () => {
60+
const target = await createNote('target');
61+
const bystander = await createNote('bystander');
62+
expect(await statusOf(target)).toBe('new');
63+
expect(await statusOf(bystander)).toBe('new');
64+
65+
const res = await stack.apiAs(token, 'POST', '/automation/flow_touch/trigger', {
66+
params: { noteId: target },
67+
});
68+
expect(res.status, `trigger failed: ${res.status} ${await res.clone().text()}`).toBeLessThan(300);
69+
const body = (await res.json()) as { success?: boolean; data?: { success?: boolean; error?: string } };
70+
expect(body.success).toBe(true);
71+
expect(body.data?.success, `flow run not successful: ${JSON.stringify(body.data)}`).toBe(true);
72+
73+
// The node executed → the targeted record was stamped.
74+
expect(await statusOf(target)).toBe('processed');
75+
// Variable wiring is REAL → the filter used noteId, so the bystander is
76+
// untouched. (A flow that dropped the variable would touch nothing or, with a
77+
// filterless update, touch every row — both caught here.)
78+
expect(await statusOf(bystander)).toBe('new');
79+
});
80+
});

packages/dogfood/test/rls-fixture.dogfood.test.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@
22
//
33
// The HARD, revert-provable #1994 gate.
44
//
5+
// @proof: rls-by-id-write
6+
// ADR-0054 runtime proof for the RLS / sharing high-risk class. Referenced by the
7+
// liveness ledger entry `permission.rowLevelSecurity.using`
8+
// (packages/spec/liveness/permission.json); the spec liveness gate fails if this
9+
// tag is removed. See proof-registry.mts.
10+
//
511
// `auto-verify-rls.dogfood.test.ts` runs the cross-owner runner over the real
612
// apps, but single-tenant boot strips the tenant policy so every object is
713
// `member-visible` — the by-id-write path is never exercised. This test boots a

packages/spec/liveness/README.md

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,57 @@ Resolution per property: **ledger entry → spec `.describe()` marker → UNCLAS
3535
Framework provenance/lock fields (`_lock*`, `_provenance`, `_packageId/Version`,
3636
`protection` — ADR-0010) are auto-classified `live`.
3737

38+
## Runtime proofs — prove-it-runs (ADR-0054)
39+
40+
`live` today means only *a static pointer to a consumer* — proof that something
41+
*reads* the property. That is necessary but not sufficient: a property can be live
42+
at every layer yet **broken end-to-end** (the break lives in the integration —
43+
engine ↔ driver ↔ service ↔ HTTP). [ADR-0054](../../../docs/adr/0054-runtime-proof-for-authorable-surface.md)
44+
adds the third leg: for a defined class of **high-risk** authorable properties, a
45+
`live` entry must carry a **`proof`** — a reference to a `@objectstack/dogfood` test
46+
that authors the property against the real in-process stack and asserts the runtime
47+
outcome.
48+
49+
```jsonc
50+
"type": {
51+
"status": "live",
52+
"evidence": "packages/objectql/src/engine.ts",
53+
"proof": "packages/dogfood/test/field-zoo-roundtrip.dogfood.test.ts#field-type-roundtrip"
54+
}
55+
```
56+
57+
**The contract.** A `proof` is `"<repo-relative-file>#<proof-id>"`. The dogfood test
58+
self-declares the id with a greppable tag near its top:
59+
60+
```ts
61+
// @proof: field-type-roundtrip
62+
```
63+
64+
The gate validates **statically** (it never runs the test — that's the dogfood
65+
gate's job, keeping this gate seconds-cheap): the file must exist **and** declare the
66+
`@proof: <id>` tag. A bound entry must point at *its own class's* proof. The reverse
67+
is also checked: a `@proof:` tag under `packages/dogfood/test/**` that isn't
68+
registered in `../scripts/liveness/proof-registry.mts` is flagged (warning) so a new
69+
proof gets wired in.
70+
71+
**The ratchet (the authoritative high-risk-class list).** Defined in
72+
`../scripts/liveness/proof-registry.mts`. A class is **CI-enforced** (`bound`) only
73+
once it has *both* a runtime proof *and* a governed ledger entry to carry it — the
74+
binding lands one class at a time (ADR-0054 §3), never as a big-bang backfill.
75+
76+
| High-risk class | Bound? | Ledger binding | Proof |
77+
|---|---|---|---|
78+
| Field types | ✅ enforced | `field.type` | `field-zoo-roundtrip.dogfood.test.ts#field-type-roundtrip` |
79+
| RLS / sharing | ✅ enforced | `permission.rowLevelSecurity.using` | `rls-fixture.dogfood.test.ts#rls-by-id-write` |
80+
| Flow nodes | ✅ enforced | `flow.nodes.type` | `flow-node.dogfood.test.ts#flow-node-execution` |
81+
| Analytics dims/measures | ✅ enforced | `dataset.dimensions.dateGranularity` | `analytics-timezone.dogfood.test.ts#analytics-tz-bucketing` |
82+
| Form layout/section/widget | ⛔ pending || none yet (form surface not yet governed) |
83+
84+
To bind a pending class: add its dogfood proof + `@proof:` tag, set `bound: true` and
85+
its `ledgerBindings` in `proof-registry.mts`, add the `proof` to the ledger entry, and
86+
confirm the gate is green. Because the gate also triggers on `packages/dogfood/**`,
87+
deleting or renaming a proof re-runs this check and the dangling reference is caught.
88+
3889
## Author warnings — closing the loop (`authorWarn`)
3990

4091
Classification is also fed back to the *author* at build time. The CLI `compile`
@@ -121,7 +172,8 @@ The governed set is `GOVERNED` at the top of `check-liveness.mts`. To add a type
121172
| agent | 18 | 4 | 5 | access/permissions/visibility dead (chat route hardcodes perms); autonomy experimental |
122173
| tool | 13 | 1 | 5 | write-only metadata; runtime uses a parallel AIToolDefinition |
123174
| skill | 15 || 2 | triggerPhrases dead (no matcher); permissions dead |
175+
| dataset | 26 || 1 | analytics semantic layer (compiled to a Cube); `measures.certified` dead; `dimensions.dateGranularity` carries the org-tz bucketing proof |
124176

125177
The `dead` set across types is the enforce-or-remove worklist (ADR-0049). Not yet governed
126-
(rollout): view, page, dashboard, app, report, dataset, job, datasource, translation,
178+
(rollout): view, page, dashboard, app, report, job, datasource, translation,
127179
email_template, doc, book, validation, seed.

0 commit comments

Comments
 (0)