Skip to content

Commit 43ecc08

Browse files
os-zhuangclaude
andauthored
chore(spec): spec liveness gate — classify-or-fail for authorable properties (#1919)
For a metadata-driven platform the spec IS the product surface; a parsed-but- unenforced property is a silent no-op, and for security props a silent no-op is false compliance (e.g. forceMfa accepted and ignored). The metadata-liveness audits found large dead swaths. This makes the classification explicit and regression-proof. - packages/spec/scripts/liveness/check-liveness.mjs — reads the generated json-schema/<category>/*.json, resolves each authorable property's liveness (ledger entry > spec .describe() marker > UNCLASSIFIED), and exits non-zero on any unclassified property in a GOVERNED category (the ratchet: no new undeclared surface). --dump inventories a category; --json for machines. - packages/spec/liveness/security.json — security ledger seeded from docs/audits/2026-06-security-identity-property-liveness.md (file:line evidence) plus greps for schemas the audit didn't cover. 93 props: 66 dead, 26 live, 1 experimental. The dead set (Policy tree, allow{Transfer,Restore,Purge}, isProfile, contextVariables, SharingRule, Territory, RLSConfig) is the enforce-or-remove worklist. - .github/workflows/spec-liveness-check.yml — runs the gate on PRs touching packages/spec/** (gen:schema then check). - check:liveness npm script + packages/spec/liveness/README.md (how to roll out the next category, highest-risk-first). Governed today: security only. Other categories are added one at a time as their ledgers are seeded from the existing audits. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent f8bfcd6 commit 43ecc08

6 files changed

Lines changed: 341 additions & 1 deletion

File tree

.changeset/spec-liveness-gate.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
---
2+
---
3+
4+
chore(spec): add a spec liveness gate — every authorable property in a governed category must declare a runtime-liveness status (live/experimental/planned/dead) with evidence in `packages/spec/liveness/<category>.json`, enforced by CI on PRs touching `packages/spec/**`. Seeds the `security` category from the liveness audit (93 props: 66 dead, 26 live, 1 experimental); the dead set is the worklist for the security enforce-or-remove ADR. Repo-internal tooling; no package version impact.
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: Spec Liveness Check
2+
3+
# For a metadata-driven platform the spec is the product surface — a parsed-but-
4+
# unenforced property is a silent no-op (and, for security props, false compliance).
5+
# This gate requires every authorable property in a GOVERNED category to declare a
6+
# liveness status with evidence in packages/spec/liveness/<category>.json. A new
7+
# unclassified property fails the check (the ratchet: no new undeclared surface).
8+
# Governed categories are rolled out highest-risk-first; see packages/spec/liveness/README.md.
9+
10+
on:
11+
pull_request:
12+
types: [opened, synchronize, reopened]
13+
paths:
14+
- 'packages/spec/**'
15+
16+
permissions:
17+
contents: read
18+
19+
jobs:
20+
liveness:
21+
name: Spec property liveness
22+
runs-on: ubuntu-latest
23+
steps:
24+
- name: Checkout repository
25+
uses: actions/checkout@v6
26+
27+
- name: Setup Node.js
28+
uses: actions/setup-node@v6
29+
with:
30+
node-version: '20'
31+
32+
- name: Enable Corepack
33+
run: corepack enable
34+
35+
- name: Install dependencies
36+
run: pnpm install --frozen-lockfile
37+
38+
- name: Generate JSON schemas
39+
run: pnpm --filter @objectstack/spec gen:schema
40+
41+
- name: Check spec liveness
42+
run: node packages/spec/scripts/liveness/check-liveness.mjs

packages/spec/liveness/README.md

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Spec liveness ledger
2+
3+
For a metadata-driven platform, **the spec is the product surface**: authors write
4+
metadata against these Zod schemas. A property that is parsed but has no runtime
5+
consumer is a silent no-op — and for a *security* property, a silent no-op is
6+
**false compliance** (e.g. `forceMfa: true` accepted and ignored). The
7+
metadata-liveness audits (`docs/audits/2026-06-*-property-liveness.md`) found that
8+
large swaths of the declared surface are DEAD.
9+
10+
This ledger makes that classification **explicit and regression-proof**: in a
11+
*governed* category, every authorable property must declare a liveness status with
12+
evidence, or CI fails (the ratchet — you can't add new undeclared surface).
13+
14+
## Status vocabulary
15+
16+
| Status | Meaning |
17+
|---|---|
18+
| `live` | Has a runtime consumer. Cite it in `evidence` (`file:line`, or a test). |
19+
| `experimental` / `planned` | Declared, intentionally not enforced yet. Also recognised from a spec `.describe()` marker like `[EXPERIMENTAL — not enforced]`. |
20+
| `dead` | Parsed, no consumer. Tracked for **enforce-or-remove** (cite the audit/grep). |
21+
| `internal` *(schema-level)* | Not authorable metadata (runtime result/DTO, context, enum). Exempt. |
22+
23+
Resolution order per property: **ledger entry → spec `.describe()` marker → UNCLASSIFIED**.
24+
A schema-level `"_schema": "<status>"` applies to all its properties (used for
25+
wholesale-dead subtrees like `PasswordPolicy`, or `internal` runtime types).
26+
Caveat: a `_schema`-classified schema also absorbs *new* properties at that status,
27+
so the ratchet does not flag additions to a wholesale-dead/internal subtree — only
28+
additions to per-property schemas (the mixed ones like `ObjectPermission`,
29+
`PermissionSet`). Use `_schema` only for subtrees that are genuinely all-one-status.
30+
31+
## Files
32+
33+
- `<category>.json` — the ledger for a governed category (currently: `security`).
34+
- `../scripts/liveness/check-liveness.mjs` — the gate. Reads the generated
35+
`packages/spec/json-schema/<category>/*.json`, resolves each authorable
36+
property's status, and exits non-zero on any UNCLASSIFIED property.
37+
38+
## Usage
39+
40+
```bash
41+
pnpm --filter @objectstack/spec gen:schema # produce json-schema/ (the source of truth)
42+
pnpm --filter @objectstack/spec check:liveness # run the gate
43+
node packages/spec/scripts/liveness/check-liveness.mjs --dump security # inventory a category (seeding aid)
44+
```
45+
46+
CI: `.github/workflows/spec-liveness-check.yml` runs the gate on PRs touching
47+
`packages/spec/**`.
48+
49+
## Rolling out the next category
50+
51+
Governed categories are listed in `GOVERNED` at the top of `check-liveness.mjs`,
52+
rolled out **highest-risk-first**. To add one (e.g. `automation`, `data`):
53+
54+
1. `--dump <category>` to inventory its authorable properties.
55+
2. Seed `<category>.json` from that category's liveness audit (file:line evidence)
56+
and targeted greps for anything the audit didn't cover. **Classify only with
57+
evidence**`live` needs a cited consumer; `dead` needs a confirmed absence.
58+
3. Add the category to `GOVERNED` and confirm the gate is green.
59+
60+
## Current state — `security`
61+
62+
93 authorable properties: **66 dead, 26 live, 1 experimental.** ~71% of the
63+
authorable security surface is parsed-but-unenforced. The `dead` entries are the
64+
worklist for the security enforce-or-remove ADR — most urgently the destructive
65+
`ObjectPermission.allow{Transfer,Restore,Purge}` (ungated) and the entirely-dead
66+
`Policy` tree (password/session/`forceMfa`/network/audit).
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
{
2+
"category": "security",
3+
"_note": "Liveness classification for authorable security metadata. Seeded from docs/audits/2026-06-security-identity-property-liveness.md (file:line evidence) plus targeted greps for schemas the audit did not cover. Statuses: live | experimental | planned | dead | (schema-level) internal. 'dead' = parsed, no runtime consumer → enforce-or-remove (security ADR). Security-critical: a 'dead' boundary prop = false compliance.",
4+
"schemas": {
5+
"ObjectPermission": {
6+
"props": {
7+
"allowCreate": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts:8" },
8+
"allowRead": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts:8" },
9+
"allowEdit": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts:8" },
10+
"allowDelete": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts:8" },
11+
"viewAllRecords": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts" },
12+
"modifyAllRecords": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts" },
13+
"allowTransfer": { "status": "dead", "evidence": "permission-evaluator.ts:8-16 (absent from OPERATION_TO_PERMISSION)", "note": "destructive op (ownership transfer) NOT gated by RBAC — enforce-or-remove. Spec marks experimental." },
14+
"allowRestore": { "status": "dead", "evidence": "permission-evaluator.ts:8-16 (absent)", "note": "undelete NOT gated — enforce-or-remove." },
15+
"allowPurge": { "status": "dead", "evidence": "permission-evaluator.ts:8-16 (absent)", "note": "hard-delete/GDPR purge NOT gated — enforce-or-remove." }
16+
}
17+
},
18+
"FieldPermission": {
19+
"props": {
20+
"readable": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts", "note": "FLS read-mask enforced." },
21+
"editable": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts", "note": "FLS write-deny enforced." }
22+
}
23+
},
24+
"PermissionSet": {
25+
"props": {
26+
"name": { "status": "live", "evidence": "permission-evaluator.ts", "note": "registration/assignment key." },
27+
"label": { "status": "live", "note": "display metadata (admin forms), not a security boundary." },
28+
"objects": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts:8" },
29+
"fields": { "status": "live", "evidence": "packages/plugins/plugin-security/src/permission-evaluator.ts" },
30+
"rowLevelSecurity": { "status": "live", "evidence": "packages/plugins/plugin-security/src/rls-compiler.ts", "note": "enforced on find + analytics raw-SQL." },
31+
"systemPermissions": { "status": "live", "evidence": "packages/plugins/plugin-hono-server/src/hono-plugin.ts", "note": "PARTIAL — enforced only for app-entry/nav visibility, NOT as a general capability gate (e.g. manage_users unchecked in data path)." },
32+
"tabPermissions": { "status": "live", "note": "PARTIAL — only 'hidden' is read; default_on/default_off ignored; UI-only, not a boundary." },
33+
"isProfile": { "status": "dead", "evidence": "audit: profile-vs-permset never gates anything", "note": "enforce-or-remove." },
34+
"contextVariables": { "status": "dead", "evidence": "rls-compiler.ts never reads it", "note": "RLS uses only current_user.* built-ins; doc claim of runtime evaluation is false." }
35+
}
36+
},
37+
"RowLevelSecurityPolicy": {
38+
"props": {
39+
"name": { "status": "live", "evidence": "packages/plugins/plugin-security/src/rls-compiler.ts" },
40+
"object": { "status": "live", "evidence": "packages/plugins/plugin-security/src/rls-compiler.ts" },
41+
"operation": { "status": "live", "evidence": "packages/plugins/plugin-security/src/rls-compiler.ts" },
42+
"using": { "status": "live", "evidence": "packages/plugins/plugin-security/src/rls-compiler.ts", "note": "compiled into find + analytics SQL." },
43+
"check": { "status": "live", "evidence": "packages/plugins/plugin-security/src/rls-compiler.ts" },
44+
"roles": { "status": "live", "evidence": "packages/plugins/plugin-security/src/rls-compiler.ts", "note": "flat role match — no subordinate rollup." },
45+
"enabled": { "status": "live", "evidence": "packages/plugins/plugin-security/src/rls-compiler.ts" },
46+
"priority": { "status": "live", "evidence": "packages/plugins/plugin-security/src/rls-compiler.ts", "note": "policy ordering." },
47+
"label": { "status": "live", "note": "display metadata." },
48+
"description": { "status": "live", "note": "display metadata." },
49+
"tags": { "status": "live", "note": "organizational metadata (not a boundary)." }
50+
}
51+
},
52+
"Policy": {
53+
"_note": "PolicySchema is 100% DEAD — not even registered as a metadata type; better-auth runs its own hardcoded config (packages/plugins/plugin-auth/src/index.ts). Authoring a compliance Policy = false compliance.",
54+
"props": {
55+
"name": { "status": "experimental", "evidence": "spec describe(): [EXPERIMENTAL — not enforced]" },
56+
"password": { "status": "dead", "evidence": "auth: better-auth hardcoded scrypt/session", "note": "false compliance — enforce or remove." },
57+
"network": { "status": "dead", "evidence": "no consumer", "note": "IP allow-list/VPN unenforced." },
58+
"session": { "status": "dead", "evidence": "auth: better-auth hardcoded session", "note": "idle/absolute timeout + forceMfa unenforced." },
59+
"audit": { "status": "dead", "evidence": "no consumer", "note": "retention/redaction unenforced." },
60+
"isDefault": { "status": "dead", "evidence": "no consumer" },
61+
"assignedProfiles": { "status": "dead", "evidence": "no consumer" }
62+
}
63+
},
64+
"PasswordPolicy": { "_schema": "dead", "_note": "Policy.password subtree — better-auth hardcoded; unenforced. enforce-or-remove.", "props": {} },
65+
"SessionPolicy": { "_schema": "dead", "_note": "Policy.session subtree — idle/absoluteTimeout/forceMfa all unenforced; better-auth hardcoded. forceMfa=true is a silent no-op (false compliance).", "props": {} },
66+
"NetworkPolicy": { "_schema": "dead", "_note": "Policy.network subtree — trustedRanges/blockUnknown/vpnRequired unenforced.", "props": {} },
67+
"AuditPolicy": { "_schema": "dead", "_note": "Policy.audit subtree — logRetentionDays/sensitiveFields/captureRead unenforced.", "props": {} },
68+
"OwnerSharingRule": { "_schema": "dead", "_note": "Spec SharingRule has NO runtime consumer; runtime enforces a divergent sys_sharing_rule model (packages/plugins/plugin-sharing/src/sharing-plugin.ts). Authoring the spec rule has no effect — reconcile to one contract.", "props": {} },
69+
"RLSConfig": { "_schema": "dead", "_note": "No runtime consumer (grep). RLS enforcement reads RowLevelSecurityPolicy via rls-compiler.ts, not this config object.", "props": {} },
70+
"RLSAuditConfig": { "_schema": "dead", "_note": "No runtime consumer (grep).", "props": {} },
71+
"Territory": { "_schema": "dead", "_note": "Salesforce-style territory management — no runtime consumer (grep).", "props": {} },
72+
"TerritoryModel": { "_schema": "dead", "_note": "No runtime consumer (grep).", "props": {} },
73+
"RLSEvaluationResult": { "_schema": "internal", "_note": "RLS runtime evaluation result — not authorable metadata." },
74+
"RLSAuditEvent": { "_schema": "internal", "_note": "RLS runtime audit event — not authorable metadata." },
75+
"RLSUserContext": { "_schema": "internal", "_note": "RLS runtime user context — not authorable metadata." }
76+
}
77+
}

packages/spec/package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,8 @@
191191
"gen:sbom": "tsx scripts/generate-sbom.ts",
192192
"test": "vitest run",
193193
"test:watch": "vitest",
194-
"test:coverage": "vitest run --coverage"
194+
"test:coverage": "vitest run --coverage",
195+
"check:liveness": "node scripts/liveness/check-liveness.mjs"
195196
},
196197
"keywords": [
197198
"objectstack",

0 commit comments

Comments
 (0)