Skip to content

Commit c9c98c1

Browse files
os-zhuangclaude
andauthored
fix(spec): the liveness gate stops trusting the direction it never checked (#4094)
The gate walked the schema and looked each property up in the ledger. That direction was well defended — a property with no row is UNCLASSIFIED and fails CI. The reverse had no check at all, and the two removal routes make that gap load-bearing: - a `retiredKey()` tombstone leaves `z.never()` in the walked shape, so the key is still a property and its row must STAY (deleting it fails as UNCLASSIFIED — 14 at once in the #3896 close-out sweep); - a strict removal takes the key OUT of the shape, so its row must GO. Nothing asked. The forward pass stops enquiring about a key it can no longer see, and a stale `dead`/`live` claim about a nonexistent property rots in place — still read by anyone treating the ledger as the capability catalogue it doubles as. Not hypothetical: the report `aria`/`performance` rows outlived their keys by a full release and were deleted by hand in the #3896 sweep, noticed only because someone happened to be reading the file. One direction failed loudly; the other never failed at all — the same silent-no-op shape this whole ledger exists to catch, one layer up. `scripts/liveness/orphans.mts` closes it, as a FAILURE rather than a warning. The tree is orphan-free across all sixteen governed types, so there is no debt to amortise and nothing to soften: a warning here would have re-created the original defect (the README's own verdict is that a permanently-noisy check is a check nobody reads, the same way a stale row is a claim nobody re-tests). The error text names BOTH causes, because which one applies is the author's judgement: delete the row, or — if the property really is still authorable — fix the WALK, since a property the walk cannot see is one the ratchet cannot govern. Deliberately silent in one case: a row declaring `children` on a non-container. The forward pass already reports that as UNCLASSIFIED with a more precise message, and two headings for one fix reads as two problems. Confirmed against `tool.parameters` (a `z.record`, undrillable) — the forward pass claimed it and the orphan scan stayed quiet. Because the tree is clean, a green gate proves nothing about whether the scan can fire, so the proof lives in `orphans.test.ts` (9 cases: both orphan levels, the tombstone that must NOT flag, the deferred non-container, entry fields that are not child props, absent/empty props). Both legs were also driven end-to-end by injecting a real orphan into a real ledger: `tool/category` (top level) and `view/list.ghostListKey` (drilled child) each failed the gate with exit 1 and the full prescription; the ledgers were restored and verified byte-identical. The skill and the ledger README carried the now-false claim that this direction never fails; both corrected, with the orphan leg documented as the fix. Gate-only change under `packages/spec/scripts/`, which is not published — empty changeset. Verification: all ten spec gates pass (`check:api-surface` needed a rebuild first — the documented stale-dist false alarm, on a dist predating 2af1988); liveness script suite 98/98; spec `tsc` clean. Claude-Session: https://claude.ai/code/session_01QuViRSR1j6GJjf9qGbnqFX Co-authored-by: Claude <noreply@anthropic.com>
1 parent 956e7f9 commit c9c98c1

6 files changed

Lines changed: 306 additions & 10 deletions

File tree

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
---
2+
---
3+
4+
fix(spec): the liveness gate checks the reverse direction — a ledger row that outlived its schema property now fails CI instead of rotting silently. Gate-only (`packages/spec/scripts/`, not published), so this releases nothing.

.claude/skills/spec-property-retirement/SKILL.md

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -107,13 +107,20 @@ that shape. Therefore:
107107
| `retiredKey()` tombstone | **YES** (`z.never()` is a property) | **STAYS**`status: "dead"`, a `verifiedAt`, and a `note` saying REMOVED + why the entry remains |
108108
| strict removal | no | **DELETED**, along with any CLI advisory-lint expectation |
109109

110-
Get this backwards and: deleting a tombstoned key's entry reports it
111-
**UNCLASSIFIED and fails CI** (14 at once, in the #3896 sweep — the mistake this
112-
section exists to prevent). The other direction is worse in the long run: the
113-
gate never walks entries looking for absent schema keys, so a **leftover entry
114-
after a strict removal rots invisibly** — report `aria`/`performance` sat stale
115-
for a release before a human deleted them as hygiene. One direction fails
116-
loudly; the other never fails at all.
110+
Both directions now fail CI, so getting it backwards is loud either way:
111+
112+
- deleting a **tombstoned** key's row reports it **UNCLASSIFIED** (14 at once in
113+
the #3896 sweep — the mistake this section exists to prevent);
114+
- leaving a **strict-removed** key's row reports it as an **ORPHAN** row.
115+
116+
The orphan leg is new (`scripts/liveness/orphans.mts`). Until it landed, this
117+
direction never failed at all — the gate walked the schema and looked rows up, so
118+
a row whose key had left the shape was simply never asked about, and rotted in
119+
place. That is how the report `aria`/`performance` rows outlived their keys by a
120+
full release, deleted by hand only because someone happened to read the file. If
121+
you hit the orphan error and the property really is still authorable, the fix is
122+
the **walk**, not the row: a property the walk cannot see is a property the
123+
ratchet cannot govern.
117124

118125
Note template for a tombstone entry (verbatim house style, e.g.
119126
`liveness/action.json`):

packages/spec/liveness/README.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,18 @@ This ledger makes that classification **explicit and regression-proof**: every p
1111
of a governed metadata type must declare a liveness status with evidence, or CI fails
1212
(the ratchet — you can't add new undeclared surface).
1313

14+
The gate checks **both directions**, and it took a while to notice the second one was
15+
missing. Schema → ledger catches a property with no row (`UNCLASSIFIED`). Ledger →
16+
schema catches a **row that outlived its property** (`ORPHAN`): when a key is removed
17+
from a `.strict()` schema it leaves the walked shape, so the forward pass simply stops
18+
asking about it and a stale `dead`/`live` claim rots in place — which is exactly what
19+
the report `aria`/`performance` rows did for a full release, deleted by hand only
20+
because someone happened to read the file. Note the asymmetry that makes this
21+
error-prone: a `retiredKey()` tombstone **keeps** the key in the walked shape, so a
22+
tombstoned key's row must **stay**, while a strict-removed key's row must **go**. The
23+
route decides the disposition — see `scripts/liveness/orphans.mts` and
24+
`.claude/skills/spec-property-retirement/SKILL.md` §2.
25+
1426
## Source of truth = the metadata-type registry
1527

1628
The gate reads `BUILTIN_METADATA_TYPE_SCHEMAS` (`packages/spec/src/kernel/metadata-type-schemas.ts`)
@@ -415,6 +427,9 @@ over-share.
415427

416428
- `<type>.json` — the ledger for a governed metadata type.
417429
- `../scripts/liveness/check-liveness.mts` — the gate (tsx; imports the registry).
430+
- `../scripts/liveness/orphans.mts` — the reverse (ledger → schema) scan: rows whose
431+
property is gone. Pure + unit-tested, because the tree was orphan-free when it
432+
landed, so a green gate proves nothing about whether the scan can fire.
418433
- `../scripts/liveness/check-empty-state.mts` — the empty-state gate (above);
419434
`empty-state-registry.mts` is its source of truth.
420435

packages/spec/scripts/liveness/check-liveness.mts

Lines changed: 40 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,11 @@
1919
// (object / record / array-of-object) may be drilled into via `"children"` so e.g.
2020
// `permission.objects.allowCreate` stays distinguishable from a blanket `objects`.
2121
//
22+
// BOTH DIRECTIONS. Schema → ledger catches an undeclared property (UNCLASSIFIED).
23+
// Ledger → schema catches the reverse: a row that outlived its property, which
24+
// went unchecked until #4080 mapped the asymmetry (a strict removal takes the key
25+
// out of the walked shape, so the forward pass just stops asking). See orphans.mts.
26+
//
2227
// Statuses: live | experimental | planned | dead. Resolution per property:
2328
// ledger entry → spec `.describe()` marker ([EXPERIMENTAL — not enforced]) → UNCLASSIFIED
2429
//
@@ -64,6 +69,7 @@ import {
6469
type VerificationReport,
6570
} from './verification.mts';
6671
import { checkEvidence } from './evidence.mts';
72+
import { ORPHAN_GUIDANCE, findOrphanEntries, type Orphan } from './orphans.mts';
6773

6874
const here = dirname(fileURLToPath(import.meta.url));
6975
const specRoot = resolve(here, '../..'); // packages/spec
@@ -202,6 +208,7 @@ const report: any = {
202208
proofErrors: [] as string[], // a `proof` ref that doesn't resolve (missing file / missing tag / malformed)
203209
proofMissing: [] as string[], // a bound high-risk `live` entry with no proof at all
204210
orphanProofs: [] as string[], // a dogfood `@proof:` tag not registered in proof-registry.mts
211+
orphanEntries: [] as string[], // a ledger row whose property is gone from the schema (the reverse direction)
205212
verification: null as VerificationReport | null, // `verifiedAt` ages — the re-verification worklist
206213
evidenceLocal: 0, // repo-rooted evidence paths actually resolved against this checkout
207214
evidenceForeign: 0, // evidence paths attributed to objectui / cloud — not resolvable here
@@ -279,7 +286,24 @@ for (const type of GOVERNED) {
279286
const ledger = loadLedger(type);
280287
const props = ledger.props || {};
281288
const cat = { classified: 0, unclassified: 0, byStatus: {} as Record<string, number> };
282-
for (const { key, node, description } of topProps(type)) {
289+
const walked = topProps(type);
290+
291+
// ── reverse direction: a row whose property is gone (see orphans.mts) ──
292+
// Runs off the SAME walk the forward pass classifies against, so the two
293+
// directions can never disagree about what the schema contains.
294+
const nodeOf = new Map(walked.map((p) => [p.key, p.node]));
295+
const orphans: Orphan[] = findOrphanEntries({
296+
type,
297+
props,
298+
shapeKeys: walked.map((p) => p.key),
299+
childKeysOf: (key) => {
300+
const cs = childShape(nodeOf.get(key));
301+
return cs ? Object.keys(cs) : null;
302+
},
303+
});
304+
for (const o of orphans) report.orphanEntries.push(o.key);
305+
306+
for (const { key, node, description } of walked) {
283307
if (FRAMEWORK_FIELDS.has(key)) { classify(type, key, 'live', null, cat); continue; }
284308
const led = props[key];
285309
if (led?.children) {
@@ -314,7 +338,11 @@ report.verification = buildVerificationReport(verificationEntries, { staleDays }
314338

315339
const totalUnclassified = report.unclassified.length;
316340
const totalProofFailures = report.proofErrors.length + report.proofMissing.length;
317-
const failed = totalUnclassified > 0 || totalProofFailures > 0 || report.verification.errors.length > 0;
341+
const failed =
342+
totalUnclassified > 0 ||
343+
totalProofFailures > 0 ||
344+
report.orphanEntries.length > 0 ||
345+
report.verification.errors.length > 0;
318346
if (asJson) {
319347
process.stdout.write(JSON.stringify(report, null, 2) + '\n');
320348
} else {
@@ -349,6 +377,12 @@ if (asJson) {
349377
console.log(`\n✗ ${totalUnclassified} UNCLASSIFIED — classify in packages/spec/liveness/<type>.json:`);
350378
report.unclassified.forEach((s: string) => console.log(` ${s}`));
351379
}
380+
if (report.orphanEntries.length) {
381+
console.log(`\n✗ ${report.orphanEntries.length} ORPHAN ledger row(s) — the property is gone from the schema:`);
382+
report.orphanEntries.forEach((s: string) => console.log(` ${s}`));
383+
console.log('');
384+
ORPHAN_GUIDANCE.forEach((line) => console.log(line ? ` ${line}` : ''));
385+
}
352386
// ── re-verification clock ──
353387
const v = report.verification!;
354388
if (v.errors.length) {
@@ -373,7 +407,10 @@ if (asJson) {
373407
console.log(' run with --stale-verification[=days] for the worklist.');
374408
}
375409
if (!failed) {
376-
console.log('\n✓ all governed-type properties are classified; all bound high-risk proofs resolve.');
410+
console.log(
411+
'\n✓ all governed-type properties are classified, no ledger row outlives its property, ' +
412+
'and all bound high-risk proofs resolve.',
413+
);
377414
}
378415
}
379416
process.exit(failed ? 1 : 0);
Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// Orphan ledger entries — the silent half of the liveness gate's asymmetry.
4+
//
5+
// WHY THIS EXISTS. The gate walks the SCHEMA and looks each property up in the
6+
// ledger. That direction is well defended: a property with no row reports
7+
// UNCLASSIFIED and fails CI (the ratchet — no new undeclared surface). The
8+
// reverse direction had no check at all, and the two removal routes make that
9+
// gap load-bearing:
10+
//
11+
// - `retiredKey()` tombstone — `z.never()` is still a property, so the key
12+
// STAYS in the walked shape and its row must stay with it. Deleting the row
13+
// reports UNCLASSIFIED and fails loudly (14 at once in the #3896 close-out
14+
// sweep, which is how this asymmetry got mapped in the first place).
15+
// - strict removal — the key leaves the shape entirely, so its row must go.
16+
// Nothing asked. The forward pass simply stops enquiring about a key it can
17+
// no longer see, and the row rots in place: a `dead`/`live` claim about a
18+
// property that does not exist, still read by anyone treating the ledger as
19+
// the capability catalogue it doubles as.
20+
//
21+
// That is not hypothetical. The report `aria`/`performance` rows outlived their
22+
// schema keys by a full release — the keys had left in the report-liveness
23+
// close-out — and were deleted by hand in the #3896 sweep as hygiene, noticed
24+
// only because a human happened to be reading the file. One direction fails
25+
// loudly; the other never failed at all.
26+
//
27+
// So this closes it, and closes it as a FAILURE rather than a warning. The tree
28+
// was clean when the check landed (zero orphans across all sixteen governed
29+
// types), so there is no debt to amortise and nothing to soften: it is a pure
30+
// ratchet on the remaining direction. A warning here would have re-created the
31+
// original defect one layer up — the ledger README's own verdict is that a
32+
// permanently-noisy check is a check nobody reads, the same way a stale row is a
33+
// claim nobody re-tests.
34+
35+
/** One orphaned ledger coordinate, with the cause narrowed as far as data allows. */
36+
export interface Orphan {
37+
/** `<type>/<propPath>` — the ledger coordinate that has no schema property. */
38+
key: string;
39+
/** Whether the row sits at the top level or under a parent's `children`. */
40+
level: 'top' | 'child';
41+
}
42+
43+
export interface OrphanScanInput {
44+
/** The governed metadata type being scanned. */
45+
type: string;
46+
/** The ledger's `props` object (may be empty / absent). */
47+
props: Record<string, any> | undefined;
48+
/** Top-level keys the gate's schema walk produced for this type. */
49+
shapeKeys: readonly string[];
50+
/**
51+
* Child keys of a container property, or `null` when the property is not a
52+
* container. Injected so every Zod-walking detail stays in the gate and this
53+
* module stays pure and testable.
54+
*/
55+
childKeysOf: (key: string) => readonly string[] | null;
56+
}
57+
58+
/**
59+
* Find ledger rows with no corresponding schema property.
60+
*
61+
* Deliberately silent in one case: a row that declares `children` on a property
62+
* that is not a container. The FORWARD pass already reports that as
63+
* UNCLASSIFIED with a more specific message, and reporting it twice under two
64+
* different headings would obscure the single fix.
65+
*/
66+
export function findOrphanEntries({ type, props, shapeKeys, childKeysOf }: OrphanScanInput): Orphan[] {
67+
const orphans: Orphan[] = [];
68+
const shape = new Set(shapeKeys);
69+
70+
for (const key of Object.keys(props ?? {})) {
71+
if (!shape.has(key)) {
72+
orphans.push({ key: `${type}/${key}`, level: 'top' });
73+
continue;
74+
}
75+
const declaredChildren = props![key]?.children;
76+
if (!declaredChildren) continue;
77+
78+
const childKeys = childKeysOf(key);
79+
if (!childKeys) continue; // the forward pass owns this one — see the docblock
80+
const childShape = new Set(childKeys);
81+
for (const childKey of Object.keys(declaredChildren)) {
82+
if (!childShape.has(childKey)) orphans.push({ key: `${type}/${key}.${childKey}`, level: 'child' });
83+
}
84+
}
85+
86+
return orphans;
87+
}
88+
89+
/**
90+
* The prescription printed under the orphan list. Both causes are actionable,
91+
* and which one applies is a judgement the author has to make — so name both
92+
* rather than guessing, and state the asymmetry that makes the wrong guess
93+
* tempting.
94+
*/
95+
export const ORPHAN_GUIDANCE = [
96+
'A ledger row outlives its property when a key is removed from the schema and',
97+
'its row is left behind. That is the STRICT-REMOVAL route: the key leaves the',
98+
'walked shape, so the forward pass stops asking about it and nothing else ever',
99+
'does. Delete the row (and any CLI advisory-lint expectation keyed on it).',
100+
'',
101+
'Mind the asymmetry before you reach for the opposite fix: a `retiredKey()`',
102+
'tombstone KEEPS the key in the walked shape, so a tombstoned key\'s row must',
103+
'STAY — deleting it reports UNCLASSIFIED instead. Route decides disposition.',
104+
'',
105+
'If the property IS still authorable, then the ledger is right and the WALK is',
106+
'wrong — the gate cannot see it (e.g. it lives on a union member `shapeOf`',
107+
'skips). Fix the walk, not the row: a property the walk cannot see is also a',
108+
'property the ratchet cannot govern.',
109+
'',
110+
'See .claude/skills/spec-property-retirement/SKILL.md §2.',
111+
];
Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// Unit tests for the orphan-ledger-entry scan (the reverse direction of the
4+
// liveness gate — see orphans.mts for why it exists).
5+
//
6+
// These matter more than a typical unit test: the tree was ORPHAN-FREE when the
7+
// check landed, so a green `check:liveness` proves nothing about whether the
8+
// scan can fire at all. That proof has to come from here. Real-ledger coverage
9+
// is the gate's own CI job (`Spec property liveness`), which walks the live Zod
10+
// schemas — this file owns the logic, including every case it must stay quiet on.
11+
12+
import { describe, it, expect } from 'vitest';
13+
import { findOrphanEntries, ORPHAN_GUIDANCE } from './orphans.mts';
14+
15+
/** No property is a container unless a test says so. */
16+
const noChildren = () => null;
17+
18+
describe('findOrphanEntries — rows it must catch', () => {
19+
it('catches a top-level row whose key left the schema (the strict-removal shape)', () => {
20+
const orphans = findOrphanEntries({
21+
type: 'tool',
22+
props: { name: { status: 'live' }, category: { status: 'dead' } },
23+
shapeKeys: ['name'], // `category` was deleted from the `.strict()` ToolSchema
24+
childKeysOf: noChildren,
25+
});
26+
expect(orphans).toEqual([{ key: 'tool/category', level: 'top' }]);
27+
});
28+
29+
it('catches a drilled child row whose key left its container', () => {
30+
const orphans = findOrphanEntries({
31+
type: 'report',
32+
props: { layout: { children: { columns: { status: 'live' }, aria: { status: 'dead' } } } },
33+
shapeKeys: ['layout'],
34+
childKeysOf: (key) => (key === 'layout' ? ['columns'] : null),
35+
});
36+
expect(orphans).toEqual([{ key: 'report/layout.aria', level: 'child' }]);
37+
});
38+
39+
it('reports every orphan, not just the first', () => {
40+
const orphans = findOrphanEntries({
41+
type: 'view',
42+
props: {
43+
gone: { status: 'dead' },
44+
alsoGone: { status: 'live' },
45+
list: { children: { type: { status: 'live' }, responsive: { status: 'dead' } } },
46+
},
47+
shapeKeys: ['list'], // `gone` and `alsoGone` both left the schema
48+
childKeysOf: (key) => (key === 'list' ? ['type'] : null),
49+
});
50+
expect(orphans.map((o) => o.key)).toEqual(['view/gone', 'view/alsoGone', 'view/list.responsive']);
51+
});
52+
});
53+
54+
describe('findOrphanEntries — cases it must stay quiet on', () => {
55+
it('does NOT flag a `retiredKey()` tombstone — z.never() is still a property', () => {
56+
// The asymmetry this whole check exists to make legible: a tombstoned key
57+
// stays in the walked shape, so its row is correct and must stay. Flagging
58+
// it would tell the author to delete a row whose deletion fails CI as
59+
// UNCLASSIFIED — pushing them into an infinite loop between two gates.
60+
const orphans = findOrphanEntries({
61+
type: 'action',
62+
props: { shortcut: { status: 'dead', note: 'REMOVED — tombstoned, row stays' } },
63+
shapeKeys: ['shortcut'],
64+
childKeysOf: noChildren,
65+
});
66+
expect(orphans).toEqual([]);
67+
});
68+
69+
it('defers `children` declared on a non-container to the forward pass', () => {
70+
// The forward walk already reports this as UNCLASSIFIED with a more precise
71+
// message ("declared children but property is not a container"). Two
72+
// headings for one fix reads as two problems.
73+
const orphans = findOrphanEntries({
74+
type: 'flow',
75+
props: { name: { children: { nope: { status: 'dead' } } } },
76+
shapeKeys: ['name'],
77+
childKeysOf: noChildren, // `name` is a string, not a container
78+
});
79+
expect(orphans).toEqual([]);
80+
});
81+
82+
it('is quiet on a ledger that matches its schema exactly', () => {
83+
const orphans = findOrphanEntries({
84+
type: 'skill',
85+
props: { name: { status: 'live' }, tools: { status: 'live' } },
86+
shapeKeys: ['name', 'tools', 'label'], // an unclassified extra is the FORWARD pass's job
87+
childKeysOf: noChildren,
88+
});
89+
expect(orphans).toEqual([]);
90+
});
91+
92+
it('handles an absent or empty props block without throwing', () => {
93+
const base = { type: 'page', shapeKeys: ['name'], childKeysOf: noChildren };
94+
expect(findOrphanEntries({ ...base, props: undefined })).toEqual([]);
95+
expect(findOrphanEntries({ ...base, props: {} })).toEqual([]);
96+
});
97+
98+
it('does not confuse an entry FIELD with a child prop', () => {
99+
// `status`/`evidence`/`verifiedAt`/`note`/`authorWarn` live on the row, not
100+
// under `children` — only a declared `children` map is drilled.
101+
const orphans = findOrphanEntries({
102+
type: 'agent',
103+
props: {
104+
role: { status: 'live', evidence: 'x.ts:1', verifiedAt: '2026-07-30', authorWarn: true, note: 'n' },
105+
},
106+
shapeKeys: ['role'],
107+
childKeysOf: () => ['definitely', 'not', 'these'],
108+
});
109+
expect(orphans).toEqual([]);
110+
});
111+
});
112+
113+
describe('ORPHAN_GUIDANCE', () => {
114+
it('names both causes and the asymmetry, so the wrong fix is not the obvious one', () => {
115+
const text = ORPHAN_GUIDANCE.join(' ');
116+
expect(text).toMatch(/STRICT-REMOVAL/);
117+
expect(text).toMatch(/Delete the row/);
118+
expect(text).toMatch(/retiredKey\(\).*KEEPS the key/);
119+
expect(text).toMatch(/UNCLASSIFIED/);
120+
expect(text).toMatch(/Fix the walk, not the row/);
121+
});
122+
});

0 commit comments

Comments
 (0)