feat(dogfood): metadata-driven verifier (objectstack verify) — auto-derive runtime proofs#2027
Merged
Merged
Conversation
…erive runtime proofs The dogfood gate pins the framework's own example apps; it never sees a third party's metadata. But the platform's value is that an AI authors arbitrary metadata and it works at runtime — for a SPECIFIC consumer app the framework CI can't reach. This adds the seed of `objectstack verify`: derive a runtime contract from an app's metadata ALONE (no hand-written tests), then run it against the real in-process stack. v0 derives per-object CRUD round-trip — synthesize a valid create body from each object's field types, POST it, GET it back, assert type fidelity. - src/derive.ts — metadata → CRUD cases. Synthesizes values per field type; reports objects blocked by un-synthesizable required fields (e.g. a required lookup) rather than silently skipping. - src/verify.ts — runs the derived cases over real HTTP, returns a structured report. Classifies outcomes: verified / fidelity-gaps (type leaks) / needs-fixture (the auto-record tripped the app's own validation rules) / create-failed|read-failed (real runtime failures) / skipped. - harness.ts — wire LocalCryptoProvider (as `objectstack dev` does) so apps with a Field.secret are exercisable end-to-end. - test/auto-verify.dogfood.test.ts — runs it on CRM + showcase; gates on "no object fails to create/read". On showcase it auto-found f_progress's type-fidelity gap too — which the hand-written field-zoo matrix had missed. - test/verify-external.dogfood.test.ts — OS_VERIFY_ARTIFACT=<app>/dist/objectstack.json points it at ANY app's built artifact (the consumer entry point; skipped in CI). Proven against hotcrm (15-object third-party app, separate repo, zero dogfood dependency): booted in-process, seeded 162 records, auto-verified — found the platform rating type-leak in crm_case/crm_lead AND a hotcrm-specific write divergence (crm_product.sku normalized to uppercase) that no framework test could reach. Private package — no changeset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📓 Docs Drift CheckNo hand-written docs reference the 1 changed package(s). ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The dogfood gate (#2020, #2022) pins the framework's own example apps. But the platform's value is that an AI authors arbitrary metadata for a third-party app — and framework CI never sees that app's metadata. A bug that only manifests with their objects ships green.
This is the seed of
objectstack verify: derive a runtime contract from an app's metadata alone (no hand-written tests), then run it against the real in-process stack. The consumer (e.g. hotcrm) runs it on their own app.What
v0 derives per-object CRUD round-trip: synthesize a valid create body from each object's field types → POST → GET → assert type fidelity.
src/derive.ts— metadata → CRUD cases; reports objects blocked by an un-synthesizable required field (e.g. required lookup) instead of silently skipping.src/verify.ts— runs over real HTTP; classifiesverified/fidelity-gaps(type leaks) /needs-fixture(auto-record tripped the app's own validation rules) /create-failed·read-failed(real failures) /skipped.harness.ts— wireLocalCryptoProvider(asobjectstack devdoes) soField.secretapps are exercisable.test/auto-verify.dogfood.test.ts— runs on CRM + showcase; gates on "no object fails to create/read". Auto-foundf_progress's fidelity gap that the hand-written field-zoo matrix missed.test/verify-external.dogfood.test.ts—OS_VERIFY_ARTIFACT=<app>/dist/objectstack.jsonpoints it at any app's built artifact (plain JSON, no cross-repo hazard);skipIf-gated, never runs in CI.Proven against hotcrm (the real third-party case)
hotcrm — separate repo, 15 objects, zero dogfood dependency, whose entire runtime verification today is a static smoke test. Pointed at its artifact, the verifier booted it in-process, seeded 162 records, auto-verified:
crm_product.sku→ a hotcrm-specific write divergence (uppercased) that no framework test could reach — only the verifier run on hotcrm's metadata surfaces it. (May be intended; the verifier reports a runtime contract divergence, not a pass/fail oracle.)Notes
27 passed | 3 expected-fail | 1 skipped.objectstack verifyCLI. Scope per ADR-0054 (#2024).🤖 Generated with Claude Code