|
| 1 | +# ADR-0054: A live authorable property must be proven at runtime, not merely have a consumer (prove-it-runs gate) |
| 2 | + |
| 3 | +**Status**: Accepted (2026-06-18) |
| 4 | +**Deciders**: ObjectStack Protocol Architects |
| 5 | +**Builds on**: [ADR-0049](./0049-no-unenforced-security-properties.md) (enforce-or-remove gate), [ADR-0005](./0005-metadata-customization-overlay.md) (artifact vs runtime), [ADR-0053](./0053-date-and-datetime-semantics.md) (the domain of the motivating regression) |
| 6 | +**Consumers**: `@objectstack/spec` (liveness ledger `packages/spec/liveness/<type>.json`), the Spec Liveness Check CI gate (#1919), `@objectstack/dogfood` (the runtime gate, [#2020](https://github.com/objectstack-ai/framework/pull/2020)), spec authors, platform contributors. |
| 7 | +**Surfaced by**: PR [#2018](https://github.com/objectstack-ai/framework/pull/2018) — "organization timezone drives analytics date bucketing" was **green on every static gate** (build, ~900 unit tests, spec-liveness, CodeQL) yet broken end-to-end across three integration seams; and the field-type capability-matrix dogfood ([#2022](https://github.com/objectstack-ai/framework/pull/2022)), which on its first run found `rating`/`slider`/`toggle` reading back wrong-typed. |
| 8 | + |
| 9 | +--- |
| 10 | + |
| 11 | +## TL;DR |
| 12 | + |
| 13 | +ObjectStack is a development platform: **third parties have an AI author |
| 14 | +arbitrary metadata**, and the promise is that it works at runtime. ADR-0049 |
| 15 | +closed *false compliance* — a property declared but unenforced. The liveness |
| 16 | +ledger (#1919) then made every authorable property declare a status |
| 17 | +(**live / experimental / dead**) with evidence, killing silent dead surface. |
| 18 | + |
| 19 | +But "live" today means only **a static `file:line` pointer to a consumer** — |
| 20 | +proof that *something reads the property*. That is necessary but **not |
| 21 | +sufficient**. A property can be live at every individual layer and still be |
| 22 | +**broken end-to-end**, because the break lives in the *integration* — engine ↔ |
| 23 | +driver ↔ service ↔ HTTP ↔ execution-context. #2018 is the proof: `timezone`, |
| 24 | +date bucketing, the analytics strategy, and the REST context were each |
| 25 | +individually correct (and individually unit-tested against mocks); the bucket |
| 26 | +was wrong only when they ran together. Call this gap **unproven liveness**: the |
| 27 | +ledger says "live", the AI is told "you may author this", and it silently |
| 28 | +misbehaves at runtime. |
| 29 | + |
| 30 | +A metadata-driven platform whose authors are AI cannot ship unproven liveness |
| 31 | +for the primitives that matter. The static pointer must be upgradable to a |
| 32 | +**runtime proof** — a [`@objectstack/dogfood`](../../packages/dogfood) test that |
| 33 | +authors the property against the real, in-process stack and asserts the runtime |
| 34 | +result. |
| 35 | + |
| 36 | +**Decision.** Extend the enforce-or-remove gate (ADR-0049) with a third leg — |
| 37 | +**prove-it-runs**. For a defined high-risk class of authorable properties, a |
| 38 | +`live` classification must carry a `proof` (a dogfood test reference), not just a |
| 39 | +consumer pointer. Applied as a **ratchet, not a retrofit**: required for newly |
| 40 | +added/changed high-risk properties and for any property implicated in a shipped |
| 41 | +runtime regression — never as a one-shot demand to prove all 200 live properties. |
| 42 | + |
| 43 | +--- |
| 44 | + |
| 45 | +## Context |
| 46 | + |
| 47 | +Three gates already guard the authorable surface, each at a different layer: |
| 48 | + |
| 49 | +| Gate | Question it answers | Where it can be fooled | |
| 50 | +|---|---|---| |
| 51 | +| **AI-authoring guardrails** (build-time lint, broken→error / fragile→warning) | *Is the authored metadata valid?* | Valid ≠ correct at runtime. | |
| 52 | +| **Spec liveness ledger** (ADR-0049 + #1919) | *Does any code read this property?* | A consumer existing ≠ the integrated path being correct. | |
| 53 | +| **Dogfood gate** ([#2020](https://github.com/objectstack-ai/framework/pull/2020)) | *Does authoring it produce correct runtime behavior?* | Coverage is currently incidental — whatever the example apps happen to exercise. | |
| 54 | + |
| 55 | +The liveness ledger's evidence is a static pointer |
| 56 | +(`packages/spec/liveness/<type>.json`, e.g. `field:line`). It is excellent at |
| 57 | +killing *dead* surface (parsed, no consumer). It is **blind to integration |
| 58 | +correctness**: #2018's properties all had valid consumer pointers and were |
| 59 | +classified live, yet the end-to-end result was wrong. The same blindness let |
| 60 | +`rating`/`slider`/`toggle` be live (they persist) while reading back as the |
| 61 | +wrong JS type — found only when [#2022](https://github.com/objectstack-ai/framework/pull/2022) |
| 62 | +wrote one and read it back over the real API. |
| 63 | + |
| 64 | +The cost is asymmetric for *this* platform. When a human authors metadata and it |
| 65 | +misbehaves, they notice and adjust. When an **AI** is told a property is live and |
| 66 | +emits it across the combinatorial space the examples never cover, the |
| 67 | +misbehavior ships silently into a third-party app. "Live" must therefore carry a |
| 68 | +stronger guarantee for the primitives an AI is most likely to combine in ways the |
| 69 | +curated examples don't. |
| 70 | + |
| 71 | +## Decision |
| 72 | + |
| 73 | +### 1. The contract — `live` may be backed by a runtime proof |
| 74 | + |
| 75 | +The liveness ledger gains an optional, stronger evidence form for `live` |
| 76 | +properties: alongside the static consumer pointer, an entry may carry a |
| 77 | +**`proof`** — a reference to a `@objectstack/dogfood` test that authors the |
| 78 | +property against the real in-process stack and asserts the runtime outcome |
| 79 | +(a value, a bucket, a count, a denied write — observable behavior, not "no |
| 80 | +error"). |
| 81 | + |
| 82 | +A proof supersedes a static pointer: it subsumes "a consumer exists" and adds |
| 83 | +"the integrated path is correct." |
| 84 | + |
| 85 | +### 2. The ratchet — required for the high-risk class, on change |
| 86 | + |
| 87 | +A `proof` is **required** (CI-enforced via the liveness gate) only for: |
| 88 | + |
| 89 | +- **(a) High-risk authorable classes, on add/change.** The classes whose values |
| 90 | + cross the engine↔driver↔service↔HTTP boundary and have repeatedly broken in |
| 91 | + *integration* despite green unit tests: |
| 92 | + - field types — persistence + read-coercion fidelity (the field-zoo matrix), |
| 93 | + - analytics dimensions / measures — bucketing, aggregation, timezone, |
| 94 | + - RLS / sharing — read **and** by-id-write enforcement, |
| 95 | + - flow nodes — execution + variable wiring, |
| 96 | + - form layout/section/widget — server-side resolution. |
| 97 | +- **(b) Regression carriers.** Any property implicated in a *shipped* runtime |
| 98 | + regression: the fix PR must add (or un-quarantine) its dogfood proof — exactly |
| 99 | + as #2018 added the tz proof, and as the `rating`/`slider`/`toggle` fix must |
| 100 | + lift the `it.fails` quarantine in the field-zoo matrix. |
| 101 | + |
| 102 | +Properties outside these classes (labels, descriptions, pure presentation hints) |
| 103 | +**do not** require a proof — a static pointer remains sufficient. The ratchet |
| 104 | +grows coverage where silent runtime breakage is plausible, not everywhere. |
| 105 | + |
| 106 | +### 3. Phasing |
| 107 | + |
| 108 | +- **Phase 1 (in progress).** Capability-matrix proofs for the two classes with a |
| 109 | + demonstrated break: field types ([#2022](https://github.com/objectstack-ai/framework/pull/2022), field-zoo) and analytics ([#2018](https://github.com/objectstack-ai/framework/pull/2018), tz bucketing). |
| 110 | +- **Phase 2.** Extend the matrix to flow nodes, form widgets, and RLS patterns |
| 111 | + (the member-edit-others by-id-write hole, #1994, is the seed RLS proof — it |
| 112 | + also drives a multi-user harness capability reused by every later RLS proof). |
| 113 | +- **Phase 3 (deferred, evidence-gated).** A generative pass that emits random |
| 114 | + valid metadata from the spec's Zod surface and asserts invariants — pursued |
| 115 | + **only** once the matrix proves the harness scales, and scoped to narrow |
| 116 | + high-value slices. Generative testing is high-ceiling and high-maintenance; it |
| 117 | + does not lead. |
| 118 | +- **CI binding lands incrementally.** The liveness gate begins requiring `proof` |
| 119 | + for class (a) one class at a time as its matrix is populated, and for class (b) |
| 120 | + immediately. No big-bang demand to backfill all live properties. |
| 121 | + |
| 122 | +### 4. Dogfood is the proof mechanism |
| 123 | + |
| 124 | +A proof is a dogfood test because dogfood is the only gate that boots the real |
| 125 | +stack in-process and exercises a property end-to-end (the thing that caught |
| 126 | +#2018). In-process Hono request-injection keeps a proof at ~2s with no ports, so |
| 127 | +the proof corpus stays CI-cheap as it grows. |
| 128 | + |
| 129 | +## Consequences |
| 130 | + |
| 131 | +- **Positive.** Closes *unproven liveness*: every high-risk authorable primitive |
| 132 | + an AI can emit carries a runtime guarantee, not just a "someone reads it" |
| 133 | + pointer. Every shipped regression leaves behind a permanent guard (the fix |
| 134 | + carries its proof). The three gates compose into one honest chain — *valid* |
| 135 | + (build) → *has a consumer* (liveness) → *runs correctly* (dogfood). |
| 136 | +- **Negative / cost.** A proof is more work than a static pointer, and the |
| 137 | + dogfood harness must scale (per-class fixtures, boot cost). Mitigated by |
| 138 | + in-process inject (~2s/proof) and by scoping the requirement to high-risk |
| 139 | + classes on change — not a retrofit. Risk that the proof corpus slows CI; |
| 140 | + bounded by the same scoping and by keeping generative testing deferred. |
| 141 | +- **Follow-up.** (1) Define the authoritative high-risk-class list and add the |
| 142 | + `proof` field + ratchet to the liveness gate. (2) The field-fidelity fix |
| 143 | + (`rating`/`slider`/`toggle`) is the first "regression carrier" instance — it |
| 144 | + must lift the field-zoo quarantine. (3) Seed the RLS proof (#1994) and the |
| 145 | + multi-user harness capability. |
| 146 | + |
| 147 | +## Non-goals |
| 148 | + |
| 149 | +- **Proving all 200 live properties now.** Trivial static properties don't need |
| 150 | + runtime proofs; the ratchet targets the high-risk class. |
| 151 | +- **Building the generative tester now.** Deferred to Phase 3, evidence-gated. |
| 152 | +- **Replacing unit tests.** Dogfood proofs add the *integration* dimension; they |
| 153 | + complement, not replace, layer-level unit tests. A proof must assert something |
| 154 | + a mocked unit test structurally cannot. |
| 155 | +- **Client-side render proofs.** The backend dogfood harness covers |
| 156 | + server-reachable behavior. Pure objectui/React render correctness belongs in |
| 157 | + objectui's own suite; a property whose only failure mode is client render is |
| 158 | + out of scope for this gate. |
0 commit comments