Skip to content

Commit ff66262

Browse files
committed
fix(spec): resolve liveness evidence paths properly — 48 stale-evidence warnings were ~100% false
The check assumed every `evidence` string is exactly `path/to/file.ts:123`: const file = String(led.evidence).split(':')[0]; if (/\//.test(file) && !existsSync(join(repoRoot, file))) -> flag Almost none are. Entries carry prose ("packages/spec/src/stack.zod.ts (mergeActionsIntoObjects stable-sorts each group)"), several pointers, or a cross-repo attribution ("objectui: packages/app-shell/..."). Taking everything before the first colon turns the prose into the filename, which never exists — so 48 of 227 entries were flagged and every one was a parse artefact or a deliberate cross-repo pointer. A permanently non-empty, ~100%-false warning is a warning nobody reads, and that is how the one real rot in the list went unnoticed: object.enable.clone cited packages/objectql/src/protocol.ts:2259, but cloneData()'s enable.clone gate had moved to packages/metadata-protocol/src/protocol.ts:2938. The claim stayed true, the pointer rotted, and the check meant to catch exactly this couldn't be heard. Pointer repaired and dated. New evidence.mts extracts repo-rooted paths and honours the realm attribution entries already write in prose: `objectui`/`cloud`/`ee` attribute the paths that follow, up to the next clause boundary, so one string can cite both repos; `framework` switches back. packages/services/service-ai/ is always foreign (the closed cloud runtime — the one sibling absent from this repo's packages/services/). Non-repo-rooted tokens read as prose. The gate now resolves 156 paths, attributes 36 cross-repo, reports ZERO stale, and prints both counts each run so the parser degrading to "extracts nothing" is visible rather than silently green — a unit test asserts that too. README: the advice to write objectui paths "as prose to avoid false stale-flags" was a workaround for this bug; documents the realm-prefix convention instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ajwvrmd1hDC9RBofYBhGuR
1 parent c757854 commit ff66262

6 files changed

Lines changed: 346 additions & 5 deletions

File tree

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
---
2+
"@objectstack/spec": patch
3+
---
4+
5+
fix(spec): the liveness gate's stale-evidence check was ~100% false positives — and it was burying a real one
6+
7+
The check was one line:
8+
9+
```ts
10+
const file = String(led.evidence).split(':')[0];
11+
if (/\//.test(file) && !existsSync(join(repoRoot, file))) → flag
12+
```
13+
14+
i.e. it assumed every `evidence` string is exactly `path/to/file.ts:123`. Almost
15+
none are — they carry prose (`packages/spec/src/stack.zod.ts (mergeActionsIntoObjects
16+
stable-sorts each group)`), multiple pointers, or a cross-repo attribution
17+
(`objectui: packages/app-shell/…`). Taking everything before the first colon
18+
turns that prose into the "filename", which never exists.
19+
20+
Result: **48 of 227 entries flagged, every one a parse artefact or a deliberate
21+
cross-repo pointer.** A permanently non-empty, ~100%-false warning is a warning
22+
nobody reads — which is exactly how the one genuine rot in that list went
23+
unnoticed:
24+
25+
- **`object.enable.clone`** cited `packages/objectql/src/protocol.ts:2259`. That
26+
file no longer exists; `cloneData()`'s `enable.clone` gate moved to
27+
`packages/metadata-protocol/src/protocol.ts:2938`. The claim stayed true, the
28+
pointer rotted, and the check that exists to catch precisely this could not be
29+
heard over the noise. Pointer repaired and dated.
30+
31+
**New `evidence.mts`** extracts repo-rooted paths properly and honours the
32+
cross-repo attribution entries already write in prose:
33+
34+
- a realm marker (`objectui`, `cloud`, `ee`) attributes the paths after it, up to
35+
the next clause boundary, so one string can cite both repos; `framework`
36+
switches back explicitly;
37+
- `packages/services/service-ai/…` is always foreign — the closed cloud runtime,
38+
the one sibling missing from this repo's `packages/services/`;
39+
- non-repo-rooted tokens (`app-shell/MetadataProvider.tsx`,
40+
`action-button/-group`) read as prose, neither resolved nor reported.
41+
42+
The gate now resolves **156 evidence paths** against the checkout, attributes 36
43+
to another repo, and reports **zero** stale — down from 48 warnings that said
44+
nothing. Each run prints the two counts, so the check degrading to "extracts
45+
nothing" is visible rather than silently green (a unit test asserts it too).
46+
47+
Also updates the ledger README, whose advice to write objectui paths "as prose to
48+
avoid false stale-flags" was a workaround for this bug: write the full path with
49+
a realm prefix instead.

packages/spec/liveness/README.md

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,14 +38,46 @@ type back onto the registry and drop the override.
3838

3939
| Status | Meaning |
4040
|---|---|
41-
| `live` | Has a runtime consumer. Cite it in `evidence` (`file:line`; objectui-repo paths as prose to avoid false stale-flags). |
41+
| `live` | Has a runtime consumer. Cite it in `evidence` as `file:line`; for another repo's path, prefix the realm — `objectui: packages/app-shell/…` (see below). |
4242
| `experimental` / `planned` | Declared, intentionally not enforced yet. Also read from a spec `.describe()` marker like `[EXPERIMENTAL — not enforced]`. |
4343
| `dead` | Parsed, no consumer. Tracked for **enforce-or-remove** (ADR-0049). |
4444

4545
Resolution per property: **ledger entry → spec `.describe()` marker → UNCLASSIFIED**.
4646
Framework provenance/lock fields (`_lock*`, `_provenance`, `_packageId/Version`,
4747
`protection` — ADR-0010) are auto-classified `live`.
4848

49+
### Writing `evidence` so the gate can check it
50+
51+
The gate extracts every **repo-rooted** path from an `evidence` string (one
52+
starting with `packages/`, `apps/`, `content/`, …) and resolves it against this
53+
checkout. Prose around the paths is fine and encouraged — `packages/spec/src/stack.zod.ts
54+
(mergeActionsIntoObjects stable-sorts each group)` resolves the path and ignores
55+
the parenthetical.
56+
57+
**A path in another repo must say so**, or the gate will report it as rot:
58+
59+
```jsonc
60+
"evidence": "objectui: packages/app-shell/src/views/RecordDetailView.tsx:573"
61+
"evidence": "registered into ActionEngine (objectui packages/core/…/ActionEngine.ts:150) but no caller"
62+
```
63+
64+
A realm marker (`objectui`, `cloud`, `ee`) attributes the paths that follow it,
65+
up to the next clause boundary (`;` or `)`), so a string can cite both repos —
66+
`framework` switches back explicitly. `packages/services/service-ai/…` is always
67+
treated as foreign: that is the closed cloud runtime, and `packages/services/`
68+
here ships every sibling service except it. A path that is not repo-rooted
69+
(`app-shell/MetadataProvider.tsx`, `action-button/-group`) reads as prose and is
70+
neither resolved nor reported.
71+
72+
> This replaces the old advice to write objectui paths "as prose to avoid false
73+
> stale-flags." That was a workaround for a parser bug — the check took
74+
> `evidence.split(':')[0]` as the filename, so any prose made it fail. It flagged
75+
> **48 of 227 entries and every one was a false positive**, which buried the one
76+
> real rotted pointer in the list (`object.enable.clone`, whose consumer had
77+
> moved from `@objectstack/objectql` to `@objectstack/metadata-protocol`). A
78+
> permanently-noisy check is a check nobody reads — the same way a stale ledger
79+
> entry is a claim nobody re-tests.
80+
4981
### `verifiedAt` — the re-verification clock
5082

5183
An entry may carry `"verifiedAt": "YYYY-MM-DD"`: the date a human last closed the

packages/spec/liveness/object.json

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -179,8 +179,9 @@
179179
},
180180
"clone": {
181181
"status": "live",
182-
"evidence": "packages/objectql/src/protocol.ts:2259",
183-
"note": "cloneData() gates on schema.enable.clone (explicit false ⇒ 403 CLONE_DISABLED); exposed at POST /data/:object/:id/clone (rest-server.ts registerDataActionEndpoints)."
182+
"verifiedAt": "2026-07-28",
183+
"evidence": "packages/metadata-protocol/src/protocol.ts:2938",
184+
"note": "cloneData() gates on schema.enable.clone (explicit false ⇒ 403 CLONE_DISABLED); exposed at POST /data/:object/:id/clone (rest-server.ts registerDataActionEndpoints). POINTER REPAIRED 2026-07 — cited packages/objectql/src/protocol.ts:2259 until the protocol moved to @objectstack/metadata-protocol. The claim never stopped being true, but the stale-evidence check could not surface the rot: 47 parse-artefact false positives were burying it (fixed in the same pass). Behaviour guarded by packages/objectql/src/protocol-clone-real-engine.test.ts."
184185
}
185186
}
186187
},

packages/spec/scripts/liveness/check-liveness.mts

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ import {
6363
type VerificationEntry,
6464
type VerificationReport,
6565
} from './verification.mts';
66+
import { checkEvidence } from './evidence.mts';
6667

6768
const here = dirname(fileURLToPath(import.meta.url));
6869
const specRoot = resolve(here, '../..'); // packages/spec
@@ -202,6 +203,8 @@ const report: any = {
202203
proofMissing: [] as string[], // a bound high-risk `live` entry with no proof at all
203204
orphanProofs: [] as string[], // a dogfood `@proof:` tag not registered in proof-registry.mts
204205
verification: null as VerificationReport | null, // `verifiedAt` ages — the re-verification worklist
206+
evidenceLocal: 0, // repo-rooted evidence paths actually resolved against this checkout
207+
evidenceForeign: 0, // evidence paths attributed to objectui / cloud — not resolvable here
205208
};
206209

207210
// Every classified entry, for the `verifiedAt` fold below. Collected during the
@@ -217,8 +220,14 @@ function classify(type: string, path: string, status: string, led: any, cat: any
217220
// Framework-auto entries (`led === null`) have no ledger row to date-stamp.
218221
if (led !== null) verificationEntries.push({ key: `${type}/${path}`, status, verifiedAt: led?.verifiedAt });
219222
if (status === 'live' && led?.evidence) {
220-
const file = String(led.evidence).split(':')[0];
221-
if (/\//.test(file) && !existsSync(join(repoRoot, file))) report.staleEvidence.push(`${type}/${path}${led.evidence}`);
223+
// Extract every repo-rooted path the evidence claims and resolve the ones
224+
// attributed to THIS repo. Cross-repo pointers (objectui / cloud) are
225+
// counted, not resolved — see evidence.mts for why the old
226+
// `split(':')[0]` heuristic reported ~100% false positives.
227+
const ev = checkEvidence(led.evidence, (p) => existsSync(join(repoRoot, p)));
228+
report.evidenceForeign += ev.foreign.length;
229+
report.evidenceLocal += ev.local.length;
230+
for (const miss of ev.missing) report.staleEvidence.push(`${type}/${path}${miss}`);
222231
}
223232
// ── ADR-0054 prove-it-runs ──
224233
const boundClass = BOUND_PROOF_PATHS.get(`${type}/${path}`);
@@ -316,6 +325,10 @@ if (asJson) {
316325
}
317326
const boundClasses = HIGH_RISK_CLASSES.filter((c) => c.bound).map((c) => c.label);
318327
console.log(`\nprove-it-runs (ADR-0054): proof REQUIRED for bound high-risk classes — ${boundClasses.join(', ') || 'none'}`);
328+
console.log(
329+
`\nevidence paths: ${report.evidenceLocal} resolved against this checkout, ` +
330+
`${report.evidenceForeign} attributed to another repo (objectui / cloud — not resolvable here).`,
331+
);
319332
if (report.staleEvidence.length) {
320333
console.log(`\n⚠ ${report.staleEvidence.length} 'live' entr(ies) cite a missing file:`);
321334
report.staleEvidence.forEach((s: string) => console.log(` ${s}`));
Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// Evidence path resolution for the liveness gate.
4+
//
5+
// WHY THIS EXISTS. The gate's stale-evidence check used to be one line:
6+
//
7+
// const file = String(led.evidence).split(':')[0];
8+
// if (/\//.test(file) && !existsSync(join(repoRoot, file))) → flag
9+
//
10+
// i.e. it assumed every `evidence` string is exactly `path/to/file.ts:123`.
11+
// Almost none are. Real entries look like:
12+
//
13+
// "packages/spec/src/stack.zod.ts (mergeActionsIntoObjects stable-sorts …)"
14+
// "packages/objectql/src/validation/rule-validator.ts (UPDATE strip); packages/…"
15+
// "objectui: packages/app-shell/src/views/RecordDetailView.tsx + utils/…"
16+
//
17+
// Taking everything before the first colon turns the prose into the "filename",
18+
// which never exists — so the check flagged 48 of 227 entries, and **every one
19+
// of the 48 was either a parse artefact or a deliberate cross-repo pointer**. A
20+
// warning list that is permanently non-empty and ~100% false is a warning
21+
// nobody reads: the one genuinely rotted pointer in that list
22+
// (`object.enable.clone` → a file that moved repos) sat there unnoticed.
23+
//
24+
// So: extract path-shaped tokens properly, and honour the cross-repo attribution
25+
// the entries already write ("objectui: …"). What's left is signal.
26+
27+
/** Top-level directories of THIS repo — the anchor for a repo-relative path claim. */
28+
export const REPO_ROOTS = ['apps', 'content', 'docker', 'docs', 'examples', 'packages', 'scripts', 'skills'];
29+
30+
/**
31+
* Realm markers an evidence string may use to attribute a path to another repo.
32+
* `objectui` is the renderer repo; `cloud` is the closed EE runtime. `framework`
33+
* switches back explicitly. These are already the house convention in prose —
34+
* this makes them machine-read instead of decorative.
35+
*/
36+
export const FOREIGN_REALMS = ['objectui', 'cloud', 'ee'];
37+
export const LOCAL_REALM = 'framework';
38+
39+
/**
40+
* Paths that are repo-rooted in shape but never present in the OPEN edition.
41+
* `@objectstack/service-ai` is the closed cloud runtime — `packages/services/`
42+
* here has every sibling service EXCEPT service-ai. Entries cite it because that
43+
* runtime is what consumes the property (see the `_note` in action.json).
44+
*/
45+
export const FOREIGN_PATH_PREFIXES = ['packages/services/service-ai/'];
46+
47+
const PATH_RE = new RegExp(`^(?:${REPO_ROOTS.join('|')})/[\\w.@-]+(?:/[\\w.@-]+)*\\.[a-zA-Z]{1,5}$`);
48+
49+
export interface EvidenceScan {
50+
/** Repo-rooted paths attributed to THIS repo — these must resolve. */
51+
local: string[];
52+
/** Paths attributed to another repo (realm marker or foreign prefix) — not resolved here. */
53+
foreign: string[];
54+
}
55+
56+
/**
57+
* Strip surrounding punctuation and any `:123` / `:12-34` line suffix from a
58+
* token. The trailing class includes `:` so a realm marker written `objectui:`
59+
* reduces to `objectui`; a line suffix (`file.ts:150`) ends in a digit, so it
60+
* survives that pass and is removed by the line-number rule after it.
61+
*/
62+
function bareToken(raw: string): string {
63+
return raw
64+
.replace(/^[([{<"'`,;]+/, '')
65+
.replace(/[)\]}>"'`,;.:]+$/, '')
66+
.replace(/:\d+(?:-\d+)?$/, '');
67+
}
68+
69+
/**
70+
* Split an evidence string into local vs foreign path claims.
71+
*
72+
* A realm marker (`objectui:`, `(objectui`, `cloud:`) attributes the paths that
73+
* FOLLOW it, and its scope ends at the next clause boundary (`;` or a closing
74+
* paren) — so `"objectui X gates … (plugin-audit, packages/plugins/…/y.ts) …"`
75+
* still resolves the framework path in the trailing clause. Anything that is not
76+
* repo-rooted (`app-shell/MetadataProvider.tsx`, `action-button/-group`) is
77+
* prose and is neither resolved nor reported.
78+
*/
79+
export function scanEvidence(evidence: string): EvidenceScan {
80+
const local: string[] = [];
81+
const foreign: string[] = [];
82+
let realm = LOCAL_REALM;
83+
84+
for (const raw of String(evidence).split(/\s+/)) {
85+
const token = bareToken(raw);
86+
const asRealm = token.toLowerCase();
87+
88+
if (FOREIGN_REALMS.includes(asRealm)) { realm = asRealm; continue; }
89+
if (asRealm === LOCAL_REALM) { realm = LOCAL_REALM; continue; }
90+
91+
if (PATH_RE.test(token)) {
92+
const isForeignPath = FOREIGN_PATH_PREFIXES.some((p) => token.startsWith(p));
93+
if (realm !== LOCAL_REALM || isForeignPath) foreign.push(token);
94+
else local.push(token);
95+
}
96+
97+
// A clause boundary ends a realm's scope; the path above is classified first.
98+
if (/[;)]/.test(raw)) realm = LOCAL_REALM;
99+
}
100+
101+
return { local: dedupe(local), foreign: dedupe(foreign) };
102+
}
103+
104+
function dedupe(xs: string[]): string[] {
105+
return [...new Set(xs)];
106+
}
107+
108+
export interface EvidenceCheck extends EvidenceScan {
109+
/** Local paths that do not exist — genuinely rotted pointers. */
110+
missing: string[];
111+
}
112+
113+
/** Scan an evidence string and resolve its local paths against the filesystem. */
114+
export function checkEvidence(evidence: unknown, exists: (path: string) => boolean): EvidenceCheck {
115+
if (typeof evidence !== 'string') return { local: [], foreign: [], missing: [] };
116+
const scan = scanEvidence(evidence);
117+
return { ...scan, missing: scan.local.filter((p) => !exists(p)) };
118+
}

0 commit comments

Comments
 (0)