fix(spec): liveness stale-evidence check was ~100% false positives — and was burying a real one - #3857
Merged
Conversation
…ce warnings were ~100% false
The check assumed every `evidence` string is exactly `path/to/file.ts:123`:
const file = String(led.evidence).split(':')[0];
if (/\//.test(file) && !existsSync(join(repoRoot, file))) -> flag
Almost none are. Entries carry prose ("packages/spec/src/stack.zod.ts
(mergeActionsIntoObjects stable-sorts each group)"), several pointers, or a
cross-repo attribution ("objectui: packages/app-shell/..."). Taking everything
before the first colon turns the prose into the filename, which never exists —
so 48 of 227 entries were flagged and every one was a parse artefact or a
deliberate cross-repo pointer.
A permanently non-empty, ~100%-false warning is a warning nobody reads, and that
is how the one real rot in the list went unnoticed: object.enable.clone cited
packages/objectql/src/protocol.ts:2259, but cloneData()'s enable.clone gate had
moved to packages/metadata-protocol/src/protocol.ts:2938. The claim stayed true,
the pointer rotted, and the check meant to catch exactly this couldn't be heard.
Pointer repaired and dated.
New evidence.mts extracts repo-rooted paths and honours the realm attribution
entries already write in prose: `objectui`/`cloud`/`ee` attribute the paths that
follow, up to the next clause boundary, so one string can cite both repos;
`framework` switches back. packages/services/service-ai/ is always foreign (the
closed cloud runtime — the one sibling absent from this repo's
packages/services/). Non-repo-rooted tokens read as prose.
The gate now resolves 156 paths, attributes 36 cross-repo, reports ZERO stale,
and prints both counts each run so the parser degrading to "extracts nothing" is
visible rather than silently green — a unit test asserts that too.
README: the advice to write objectui paths "as prose to avoid false stale-flags"
was a workaround for this bug; documents the realm-prefix convention instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ajwvrmd1hDC9RBofYBhGuR
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
github-actions
Bot
added
documentation
Contributor
📓 Docs Drift CheckThis PR changes 1 package(s): 104 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
os-zhuang
marked this pull request as ready for review
July 28, 2026 12:52
os-zhuang
added a commit
that referenced
this pull request
Jul 28, 2026
…just skills/ (#3882) check:skill-examples compiles the TypeScript in prose against the built spec, so an example that stops compiling fails CI instead of quietly teaching code that no longer works. It does its job — it caught the broken defineTool example in #3876. But it only ever walked skills/: skills/ 9 files with ts blocks, 9 compiled content/docs/ 124 files with ts blocks, 0 compiled The identical break in a docs page ships. Docs examples are copied verbatim by humans and AI exactly like skill examples — the same shape as the stale-evidence and orphan-proof warnings fixed in #3857 / #3868: a gate covering a fraction of the surface it appears to cover reads as coverage. The walker is now a SOURCE_ROOTS list, and this lands the first batch: 164 docs blocks across 63 pages, taking the gate from 32 to 196 checked examples. content/docs/references/ is excluded — build-docs.ts regenerates it from the schemas, so it cannot drift independently. THE MARKER IS NOW PER-FORMAT. MDX has no HTML comments: `<!-- os:check -->` in a .mdx does not degrade, it fails the fumadocs build outright ("Unexpected character `!`… to create a comment in MDX, use `{/* text */}`"). Found by building the docs site after the first attempt broke 60+ pages; nothing in the type-check gate would have said so, because block extraction is regex-level and never parses MDX. skills/**/*.md keeps `<!-- os:check -->`; content/docs/**/*.mdx uses `{/* os:check */}`. Both spellings are recognised for ORPHAN detection, so a wrong-format marker fails loudly instead of silently checking nothing. The batch was measured, not guessed: marking all 780 docs blocks and compiling showed which are self-contained. One subtlety — a block that "passes" inside a 780-file program can be leaning on globals declared by OTHER blocks (a file with no import/export is a global script), so the first pass's 564 "passing" blocks collapsed to 164 once compiled as the real, smaller set. Converged by recompiling and dropping newly-failing blocks until green. The remaining blocks are mostly fragments, which the opt-in design anticipates. Whether any are genuine rot is now answerable for the first time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #3845. The
verifiedAtclock landed there is only useful if the gate's warnings are trusted — and one of them was pure noise.The bug
The stale-evidence check was one line:
It assumes every
evidencestring is exactlypath/to/file.ts:123. Almost none are:split(':')[0]producedpackages/spec/src/stack.zod.ts (mergeActionsIntoObjects stable-sorts each group)objectui: packages/app-shell/src/views/RecordDetailView.tsx:573objectui— the attribution eaten as a filenamepackages/objectql/…/rule-validator.ts (UPDATE strip); packages/metadata-protocol/…48 of 227 entries flagged, and every one was a parse artefact or a deliberate cross-repo pointer. I checked all 48 by hand: 11 were objectui paths that already said
objectui:, 27 werepackages/services/service-ai/…(the closed cloud runtime —packages/services/here ships every sibling except it), and the rest were paths with a parenthetical after them that do exist.The real one it was burying
object.enable.clonecitedpackages/objectql/src/protocol.ts:2259. That file is gone —cloneData()'senable.clonegate now lives atpackages/metadata-protocol/src/protocol.ts:2938(verified; behaviour still guarded byprotocol-clone-real-engine.test.ts). The claim never stopped being true, the pointer rotted, and the check that exists to catch exactly this could not be heard over 47 false ones. Pointer repaired and dated.That's the same failure this ledger keeps documenting, one level up: a signal nobody trusts is a signal nobody reads.
The fix
New
evidence.mtsextracts repo-rooted paths properly and honours the cross-repo attribution entries were already writing in prose:objectui,cloud,ee) attributes the paths that follow it, up to the next clause boundary (;or)) — so one string can cite both repos, andframeworkswitches back explicitly;packages/services/service-ai/…is always foreign, wherever it appears;app-shell/MetadataProvider.tsx,action-button/-group) read as prose — neither resolved nor reported.Both parenthesised (
(objectui packages/core/…)) and colon (objectui: packages/…) marker forms are handled; the corpus uses both.Down from 48 warnings that said nothing, to zero — with the two counts printed every run, so the parser silently degrading to "extracts nothing" is visible rather than vacuously green. A unit test asserts that too (
expect(local).toBeGreaterThan(100)).The ledger README's advice to write objectui paths "as prose to avoid false stale-flags" was a workaround for this bug; it now documents the realm-prefix convention.
Verification
enable.clonerot). All 11 objectui paths confirmed present inobjectstack-ai/objectui@732b1bf;packages/services/service-ai/confirmed absent from this repo.pnpm --filter @objectstack/spec check:liveness— exits 0, zero stale evidence.pnpm --filter @objectstack/spec test— 262 files, 6796 tests passed (17 new inevidence.test.ts, incl. a contract test over the shipped ledgers).Not in this PR
The gate's other permanently non-empty warning — 13 unregistered dogfood
@proof:tags — is left alone deliberately. Registering them is not clerical: several (attachments-permission-matrix,flow-runas-schedule) look like they should be bound to a ledger entry rather than merely registered as unbound, and deciding that per-proof is its own reviewable change.Generated by Claude Code