|
| 1 | +# ADR-0076: objectql is the data engine — relocate metadata management (protocol) out of it; enforce the boundary; defer the engine repo-split |
| 2 | + |
| 3 | +**Status**: Proposed (2026-06-28, rev. 4 — data-driven; centerpiece corrected to protocol relocation) — v12 assessment |
| 4 | +**Deciders**: ObjectStack Protocol Architects |
| 5 | +**Builds on**: [ADR-0005](./0005-metadata-customization-overlay.md) (sys_metadata overlay substrate), [ADR-0025](./0025-plugin-package-distribution.md) (plugin package distribution), [ADR-0033](./0033-ai-assisted-metadata-authoring.md) (open-core boundary), [ADR-0048](./0048-cross-package-metadata-collision.md) (package id is the addressing unit), [ADR-0066](./0066-unified-authorization-model.md) (secure-by-default, posture-gated bypass) |
| 6 | +**Consumers**: **new** `@objectstack/metadata-protocol` (receives `protocol` + `sys-metadata-repository` + `metadata-diagnostics`), `@objectstack/objectql` (loses protocol → becomes a lean data engine; keeps a back-compat re-export), `@objectstack/metadata-core` (gains the `SysMetadataEngine` interface), `@objectstack/plugin-security`, `@objectstack/plugin-sharing`, `@objectstack/spec`, and out-of-tree embedders — notably `../objectbase` (its `gateway`). |
| 7 | + |
| 8 | +**Premise**: objectql was conceived as "a metadata-driven ORM that replaces TypeORM." It has outgrown that framing: it is a metadata-driven **data engine** sitting on top of knex (`driver-sql` → `knex@^3.2.10`). The complexity the team feels is not the engine; it is everything layered above and *beside* it — including `protocol.ts` (268KB of sys_metadata management / draft-publish / package ownership / locks) which lives *inside the objectql package by historical accident*, not by design. |
| 9 | + |
| 10 | +> **Trigger**: `../objectbase`'s `gateway` wants to embed only the objectql engine without the platform. While scoping that, a sharper observation surfaced: **`protocol.ts` is metadata-domain code mis-located in the data-engine package.** Measurement confirms it (below). Relocating it — not a subpath, not a repo split — is the correct, cheap, *now* move that makes objectql lean by construction. |
| 11 | +
|
| 12 | +--- |
| 13 | + |
| 14 | +## TL;DR |
| 15 | + |
| 16 | +1. **knex is a SQL query builder objectql already uses**, not an ORM to switch to. Retreating forfeits the one-library / one-object-model goal and removes none of the felt complexity. |
| 17 | +2. **`protocol.ts` does not belong in the objectql package.** It implements `ObjectStackProtocol` (the contract lives in `@objectstack/spec`), manages sys_metadata, locks, commits, package ownership — pure metadata-domain work. It uses the engine only as storage, through a **5-method `SysMetadataEngine` interface** injected at runtime. |
| 18 | +3. **Measured coupling makes the relocation cheap.** Last month: `protocol.ts` changed 47×; only **3 (6%)** also touched `engine.ts`, while **20 (43%)** touched `metadata-core`/`metadata`/`spec`. Blast radius of the move is **2 source files** (`plugin.ts` wiring, `index.ts` re-export). Its two helpers (`sys-metadata-repository`, `metadata-diagnostics`) depend only on `metadata-core`/`spec`/`types` and move with it. |
| 19 | +4. **Decision (now):** relocate `protocol` + helpers into a **new `@objectstack/metadata-protocol`** package between `metadata-core` (pure contracts) and `metadata` (plugin). objectql becomes a lean data engine **by construction** — the 268KB genuinely leaves the package; the gateway depends on objectql with no protocol. Add a **boundary ratchet** so the engine stays pure. Add the **capability/profile** contract for optional permissions. Formula stays in core. |
| 20 | +5. **Decision (later, separate concern):** extracting the *engine itself* into a standalone repo remains **trigger-gated** on its cross-package commit ratio (currently **88%** for `engine.ts`/`registry.ts`). That is orthogonal to — and unblocked by — the protocol relocation. |
| 21 | + |
| 22 | +## Context: the layers |
| 23 | + |
| 24 | +| Layer | What it is | Where it lives today | Where it should live | |
| 25 | +|---|---|---|---| |
| 26 | +| SQL generation | dialect/pool/binding | `knex` (in `driver-sql`) | unchanged | |
| 27 | +| **Data engine** | QueryAST→driver; CRUD; hooks; validation; formula | `objectql/src/{engine,registry,validation,hook-wrappers}` | unchanged (this is `@objectstack/objectql`) | |
| 28 | +| **Metadata management** | `ObjectStackProtocol` impl; sys_metadata CRUD; draft/publish; locks; ownership; diagnostics | **`objectql/src/protocol.ts` (mis-located)** + `metadata-core` + `metadata` plugin | **new `@objectstack/metadata-protocol`** | |
| 29 | +| Governance | RLS, sharing, field perms | `plugin-security`, `plugin-sharing` (already separate) | unchanged | |
| 30 | + |
| 31 | +## Decision data |
| 32 | + |
| 33 | +``` |
| 34 | +protocol.ts commits last month: 47 |
| 35 | + also touched engine.ts: 3 (6%) ← protocol↔engine ≈ decoupled |
| 36 | + also touched metadata-core/metadata/spec: 20 (43%) ← its real neighbors |
| 37 | +import sites of ObjectStackProtocolImplementation (source): plugin.ts, index.ts (+ co-located helpers) |
| 38 | +engine surface protocol needs: SysMetadataEngine = { find, findOne, insert, update, delete, transaction? } (injected) |
| 39 | +
|
| 40 | +(for contrast) engine.ts/registry.ts commits last month: 50, of which 44 (88%) cross-package → engine itself NOT separable yet |
| 41 | +``` |
| 42 | + |
| 43 | +## Decision |
| 44 | + |
| 45 | +### D1 — Relocate protocol out of objectql into `@objectstack/metadata-protocol` [new — the centerpiece] |
| 46 | + |
| 47 | +- Move `protocol.ts`, `sys-metadata-repository.ts`, `metadata-diagnostics.ts` from `@objectstack/objectql` into a **new `@objectstack/metadata-protocol`** package. |
| 48 | +- Move the `SysMetadataEngine` interface into `@objectstack/metadata-core`; `metadata-protocol` depends on `metadata-core` + `spec` + `core` (for `IDataEngine`) + `types` — **not on the objectql package**. The concrete engine is injected at runtime (as today). |
| 49 | +- `@objectstack/objectql` re-exports `ObjectStackProtocolImplementation` from the new package for back-compat (the only two source importers — `plugin.ts`, `index.ts` — are updated; `plugin.ts` keeps wiring it, now imported from `metadata-protocol`). |
| 50 | +- Result: objectql is a lean data engine **by construction** — 268KB of metadata logic physically leaves the engine package. The gateway depends on `@objectstack/objectql` and never pulls protocol. (This **replaces** the rev.3 `./core` subpath proposal, which was a workaround for the false premise that protocol must stay in objectql. A subpath remains an optional later polish to also exclude the small `ObjectQLPlugin`/`kernel-factory` glue.) |
| 51 | + |
| 52 | +### D2 — Boundary ratchet keeps the engine pure [new — keystone] |
| 53 | + |
| 54 | +A CI test asserts `@objectstack/objectql` does **not** import `@objectstack/metadata-protocol` / `plugin` / `kernel` (the dependency points one way: `metadata-protocol → objectql-public-interface`, never back). This makes the relocation durable and creates backpressure against the engine re-absorbing metadata/platform concerns — the root cause of the original complexity. Fits the team's existing ratchet culture (liveness / api-surface / authz-matrix). |
| 55 | + |
| 56 | +### D3 — Capability + profile contract for optional governance [new, extends ADR-0066] |
| 57 | + |
| 58 | +Required capabilities are **derived** from object declarations (RLS block → `rls`; `requiredPermissions` → `permissions`; sharing → `sharing`). Plugins declare `provides: [...]`. The host validates at boot; **default fail-closed**. A host may run without the authz surface via an explicit `profile: 'trusted' | 'internal'` — **two-key**: build-time plugin absence + explicit runtime assertion; production env-gated. Lifts ADR-0066 posture-gated bypass to the assembly layer. |
| 59 | + |
| 60 | +### D4 — Formula stays in core [ruled] |
| 61 | + |
| 62 | +`@objectstack/formula` is used by `engine.ts` (formula fields), `validation/rule-validator.ts`, `hook-wrappers.ts`, `seed-loader.ts` — a hard dependency of the engine. `cel-js` is small and not the complexity driver. |
| 63 | + |
| 64 | +### D5 — One object model, authored once [existing → ruled] |
| 65 | + |
| 66 | +Definitions live in `@objectstack/spec` (zero runtime deps); gateway and backend import the same `*.object.ts` and call the same engine; only the installed capability set differs. |
| 67 | + |
| 68 | +### D6 — Capabilities attach uniformly across assembly modes [existing → ruled] |
| 69 | + |
| 70 | +RLS/permissions/sharing attach via engine **middleware** (`ql.registerMiddleware(...)`) + hooks — kernel-independent. Ratify this as the public, supported capability-attachment API. |
| 71 | + |
| 72 | +### D7 — Engine repo-split is a separate, trigger-gated future phase [sequencing] |
| 73 | + |
| 74 | +Extracting the *engine* into a standalone repo (the `objectui` sibling-link, no-publish model) is gated on the engine's cross-package commit ratio (currently **88%**) falling. This ADR's relocation (D1) is independent of and unblocked by D7. `objectui` separates cleanly because it is a stable downstream leaf; the engine is an upstream foundation still in 88% cross-cutting co-evolution — separating it now would reproduce the overhead that caused the earlier `@objectql/core` merge-back. |
| 75 | + |
| 76 | +## Feasibility (verified against current source) |
| 77 | + |
| 78 | +| Claim | Status | Evidence | |
| 79 | +|---|---|---| |
| 80 | +| protocol is metadata-domain, not engine | ✅ | implements `ObjectStackProtocol` (`spec/api`); manages sys_metadata/locks/commits/ownership | |
| 81 | +| protocol decoupled from the engine | ✅ | 6% commit co-change with `engine.ts`; needs only the 5-method `SysMetadataEngine` (injected) | |
| 82 | +| relocation blast radius is tiny | ✅ | only `plugin.ts` + `index.ts` import the impl; helpers depend only on metadata-core/spec/types | |
| 83 | +| splitting protocol saves engine deps | ✅ (for the engine package) | protocol's deps (spec/*, core, metadata-core) leave with it; engine keeps only its own | |
| 84 | +| engine itself separable now | ❌ | 88% of engine/registry commits cross-package → D7 deferred | |
| 85 | + |
| 86 | +## Worked example |
| 87 | + |
| 88 | +```ts |
| 89 | +// Gateway (objectbase) — depends on the (now lean) engine; protocol is simply not in the package |
| 90 | +import { ObjectQL } from '@objectstack/objectql'; |
| 91 | +import { MemoryDriver } from '@objectstack/driver-memory'; |
| 92 | +import { Account } from '@shared/objects/account.object'; // SAME definition as the backend |
| 93 | + |
| 94 | +const engine = new ObjectQL({ profile: 'trusted' }); |
| 95 | +engine.registerDriver(new MemoryDriver(), true); |
| 96 | +await engine.init(); |
| 97 | +engine.registry.registerObject(Account); |
| 98 | +await engine.find('account', { where: { active: true } }); |
| 99 | + |
| 100 | +// Backend — adds metadata management + governance on top of the same engine |
| 101 | +import { createObjectQLKernel } from '@objectstack/objectql'; |
| 102 | +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; |
| 103 | +import { SecurityPlugin } from '@objectstack/plugin-security'; |
| 104 | +``` |
| 105 | + |
| 106 | +## Phasing |
| 107 | + |
| 108 | +1. **P1 — Relocate protocol** to `@objectstack/metadata-protocol` (move 3 files; move `SysMetadataEngine` into metadata-core; update `plugin.ts` import + `index.ts` re-export). Behavior unchanged. |
| 109 | +2. **P2 — Boundary ratchet** (objectql must not import metadata-protocol/plugin/kernel) + export-surface ratchet. |
| 110 | +3. **P3 — Capability/profile contract** (the only substantial new code). |
| 111 | +4. **P4 — Standalone embed example + smoke** under `examples/`. |
| 112 | +5. **Later (D7)** — engine repo-split when the cross-package ratio drops. |
| 113 | + |
| 114 | +## Consequences |
| 115 | + |
| 116 | +- **+** objectql is lean by construction; protocol lands in its proper domain; real enforced package boundary (not a convention); gateway unblocked; all without breaking downstream (re-export shim). |
| 117 | +- **+** Cheap and low-risk now (6% coupling, 2-file blast radius, narrow injected interface). |
| 118 | +- **−** One new package in the fixed version group; a major-version bump for the moved export (mitigated by the re-export shim). |
| 119 | +- **Risk (highest)**: capability/profile under-specified → silent authz bypass. Mitigation: default fail-closed; `trusted` two-key + prod env-gate; authz-matrix ratchet. |
| 120 | +- **Risk**: tsup DTS stricter than tsc / turbo build-order during the package move. Mitigation: incremental, watch DTS. |
| 121 | + |
| 122 | +## Open questions (with recommended positions) |
| 123 | + |
| 124 | +1. **protocol's new home** — resolved: a dedicated **`@objectstack/metadata-protocol`** between `metadata-core` (kept pure) and the `metadata` plugin (avoids dragging chokidar/fs deps onto pure runtime protocol logic). |
| 125 | +2. **`SysMetadataEngine` interface home** — recommend `@objectstack/metadata-core` (it is a contract; metadata-protocol depends on metadata-core anyway). |
| 126 | +3. **Capability granularity** — derive from object declarations; do not overload per-user `requiredPermissions`. |
| 127 | +4. **`trusted` profile** — two-key (build-time absence + explicit runtime assertion), prod env-allowlisted. |
| 128 | +5. **D7 trigger threshold** — what cross-package ratio (from 88%) over what window signals "extract the engine"? Track in CI; set on first review. |
0 commit comments