feat(spec): the example type-check gate now covers content/docs, not just skills/ - #3882
Merged
Merged
Conversation
…t/docs check:skill-examples compiles the TypeScript in `skills/` against the built spec. It works — it caught the broken `defineTool` example when tool.requiresConfirmation was removed (#3715). But it only ever walked `skills/`: skills/ 9 files with ts blocks, 9 compiled content/docs/ 124 files with ts blocks, 0 compiled So the identical break in a docs page ships. Docs examples are copied verbatim by humans and AI exactly like skill examples; a gate covering a fraction of the surface it appears to cover reads as coverage. Generalises the walker to a SOURCE_ROOTS list and opts in the first batch: 164 docs blocks across 63 pages, taking the gate from 32 to 196 checked examples. content/docs/references/ is excluded — build-docs.ts regenerates it from the schemas, so it cannot drift independently. MARKER IS NOW PER-FORMAT. MDX has no HTML comments: `<!-- os:check -->` in a .mdx fails the fumadocs build outright ("Unexpected character `!`… to create a comment in MDX, use `{/* text */}`"). Verified by building the docs site — the first attempt failed on 60+ pages. skills/*.md keeps `<!-- os:check -->`; content/docs uses `{/* os:check */}`. Both spellings are recognised for ORPHAN detection, so a wrong-format marker fails loudly instead of silently checking nothing. Batch selection was measured, not guessed: marking all 780 docs blocks and compiling showed which are self-contained. Note a subtlety — blocks that "pass" in a 780-file program can be leaning on globals declared by OTHER blocks (a file with no import/export is a global script), so the set was converged by recompiling and dropping newly-failing blocks until green. 164 stand alone. The remaining ~600 are mostly fragments (a `columns: [...]` subtree), which the gate's opt-in design already anticipates; whether any are genuine rot is worth a follow-up pass now that the machinery reaches them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ajwvrmd1hDC9RBofYBhGuR
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ajwvrmd1hDC9RBofYBhGuR
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
github-actions
Bot
added
documentation
Contributor
📓 Docs Drift CheckThis PR changes 1 package(s): 104 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
os-zhuang
marked this pull request as ready for review
July 28, 2026 14:26
os-zhuang
added a commit
that referenced
this pull request
Jul 28, 2026
…a call that throws (#3890) #3882 brought content/docs under the example type-check gate and left the open question: of the ~600 blocks still unmarked, is any GENUINE ROT rather than a fragment? Swept them. One real one. deployment/troubleshooting.mdx answered "How do I extend a built-in schema?" — a direct FAQ, not a sketch — with `FieldSchema.extend({ … })`. FieldSchema carries a .transform() that lowers author-facing sugar at parse time, so the exported value is a ZodPipe, not a ZodObject: `.extend` is undefined and the call throws. The example also used `z` without importing it. Rewritten to COMPOSE — parse with FieldSchema, validate the additions alongside it — verified to both type-check and run, and marked so the gate holds it. Composition also keeps the transform, which .extend() would discard. A divergence found while fixing it: the first attempt used FieldSchema.in.extend() and failed in CI with "Property 'in' does not exist on type 'ZodObject<…>'". The two builds emit DIFFERENT declarations for the same source — locally ZodPipe<ZodObject<…>>, in CI a plain ZodObject — while the runtime is unambiguous (bound ZodPipe, .extend undefined, confirmed after rm -rf dist && pnpm build). So CI's declaration contradicts the value it describes. Likely inference instability in the DTS bundling of these very large zod types; worth its own investigation. The final example uses only .parse(), so it is correct under either declaration. The sweep's METHOD is recorded in the gate docstring, because two traps make a naive pass report a confident "nothing found": 1. tsc stops after syntactic diagnostics and never runs the semantic pass. Marking all 780 blocks at once let ~200 broken fragments suppress type-checking for every other block; that run returned only TS1xxx codes, which reads exactly like "no rot" and proves nothing. Caught by injecting a deliberate type error and watching it go unreported. 2. Unimported type names resolve against the DOM lib — Plugin, Event, Response, Storage all exist there. A block missing its import reports "'version' does not exist in type 'Plugin'" against lib.dom's Plugin: an artefact, not drift. Three of the strongest-looking candidates were exactly this. Everything else is fragments, plus config-resolution.mdx, whose aspirational snippets a callout on the page already labels as design intent. Gate: 196 -> 197 checked examples.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #3876, which is where this gap showed itself.
The gap
check:skill-examplescompiles the TypeScript in prose against the built spec, so an example that stops compiling fails CI instead of quietly teaching code that no longer works. It does its job — it caught the brokendefineToolexample in #3876 whentool.requiresConfirmationwas removed.But it only ever walked
skills/:tsblocksskills/content/docs/(hand-written)The identical break in a docs page ships. Docs examples are copied verbatim by humans and AI exactly like skill examples — same shape as the stale-evidence and orphan-proof warnings fixed in #3857 / #3868: a gate covering a fraction of the surface it appears to cover reads as coverage.
The change
The walker is now a
SOURCE_ROOTSlist, and this lands the first batch: 164 docs blocks across 63 pages, taking the gate from 32 → 196 checked examples.content/docs/references/is excluded —build-docs.tsregenerates it from the schemas, so it cannot drift independently.The marker is now per-format — and this is the part I nearly got wrong
MDX has no HTML comments.
<!-- os:check -->in a.mdxdoesn't degrade, it fails the fumadocs build outright:I found this by building the docs site — the first attempt broke 60+ pages. Nothing in the type-check gate would have told me, because the block extraction is regex-level and never parses MDX.
skills/**/*.md→<!-- os:check -->content/docs/**/*.mdx→{/* os:check */}Both spellings are recognised for orphan detection, so a wrong-format marker fails loudly rather than silently checking nothing. That extends the guard's existing philosophy to the failure mode this change introduces.
The batch was measured, not guessed
Marking all 780 docs blocks and compiling showed which are self-contained. One subtlety worth recording: a block that "passes" inside a 780-file program can be leaning on globals declared by other blocks — a file with no import/export is a global script, so
TaskViewsdeclared in one page's example resolves in another's. Selecting on that first pass gave 564 "passing" blocks, of which only 164 survived once compiled as the real, smaller set. The set was converged by recompiling and dropping newly-failing blocks until green.Verification
pnpm --filter @objectstack/spec check:skill-examples—✅ 196 prose examples type-check(32 skills + 164 docs), re-run against mergedmainafter feat(spec)!: removetool.requiresConfirmation— a safety flag nothing enforced (#3715, ADR-0033 §2) #3876.pnpm --filter @objectstack/docs build— succeeds. This is the check that caught the MDX comment problem; it is the reason the marker is per-format.pnpm check:doc-authoring— 213 files clean. The markers sit on their own line and leave the fence info-string bare, so the sibling scanner that keys on^```(ts|typescript|tsx)$still sees every block (the original script's docstring flags this as a hole to avoid; still avoided).Follow-up
The remaining ~600 blocks are mostly fragments (a
columns: [...]subtree), which the gate's opt-in design already anticipates. Whether any are genuine rot rather than fragments is now answerable for the first time, and worth a pass — that is exactly the class of thing this machinery exists to surface.Generated by Claude Code