Merge remote-tracking branch 'origin/main' into claude/d1-d2-strict-default-flip-vbly5z

# Conflicts:
#	content/docs/releases/v17.mdx

Author: claude

.changeset/adr-0104-d3w2-pr5b-enable-collection.md

@@ -0,0 +1,36 @@
+---
+"@objectstack/service-storage": minor
+---
+
+feat(storage): released field files enter collection on deployments that verified their file migration — ADR-0104 D3 wave 2 PR-5b (#3459)
+
+The gated, final step of the file-as-reference sequence. On a deployment whose
+`adr-0104-file-references` flag is verified (`os migrate files-to-references
+--apply`, #3617), releasing a field file's ownership — clearing the field, or
+deleting the owning record — now also tombstones the file
+(`status='deleted'` + `deleted_at`), which starts the `sys_file` lifecycle's
+declared 30-day grace window and, at its end, hands the row to the reap sweep.
+Re-referencing the id inside the window revives it, exactly like re-attaching
+an attachment.
+
+**The two halves ship together, deliberately.** The same change extends the
+reap guard's sweep-time re-verify beyond `sys_attachment` join rows to the
+ownership columns: a tombstoned file whose `ref_*` columns name a current
+owner (re-claimed in the window, or a release/claim race) is un-tombstoned and
+vetoed. Tombstoning released files without that re-verify would have turned
+every release into a *guaranteed* byte delete — the guard's old check consults
+a table that is always empty for field files. This pairing was the standing
+hard constraint on #3459, locked by regression tests on both halves.
+
+**Nothing changes for a deployment that has not migrated.** Release keeps
+clearing the ownership columns only, and released files are retained forever.
+Every way of not knowing — no flag row, an unreadable table, an engine that
+cannot be asked — reads as "not verified": the gate fails closed, toward
+retention. And the guard re-reads the flag *fresh* at sweep time (not the
+release path's memoized read), so a later failing migration run — a database
+that has drifted — closes the gate for already-written tombstones too, without
+a restart. Attachments-scope collection is unchanged and needs no flag.
+
+The irreversible moment is therefore per deployment: day 30 after *that*
+deployment verified its migration and released a file — never the upgrade
+itself.

.changeset/adr-0104-d3w2-pr5b-engine-flag-read.md

@@ -0,0 +1,14 @@
+---
+"@objectstack/objectql": minor
+---
+
+feat(objectql): `engine.isFileReferencesMigrationVerified()` is public — one memoized flag read for both in-process consumers (#3459 PR-5b)
+
+The memoized per-deployment read of the `adr-0104-file-references` migration
+flag was private to the engine's media value-shape enforcement. The storage
+service's release path now asks the same question — may a released field file
+be tombstoned? — so the method is public and the release hooks reach it as an
+optional duck-typed member (an older engine or a test fake reads as "not
+verified", failing closed). One read, one invalidation
+(`invalidateDataMigrationFlags()`), no way for the two consumers to see
+different answers.

.changeset/adr-0116-followups.md

@@ -0,0 +1,42 @@
+---
+"@objectstack/plugin-auth": patch
+"@objectstack/plugin-webhooks": patch
+---
+
+fix(plugin-auth,plugin-webhooks): retire a dead degrade branch and an implicit transitive dependency (ADR-0116 follow-ups, #4187)
+
+Two concrete findings from the ADR-0116 consumer-side audit, plus the
+authoring rule that would have prevented both.
+
+**`plugin-auth` claimed a fallback it did not have.** `init()` ran
+`const dataEngine = ctx.getService('data'); if (!dataEngine) { warn('No data
+engine service found - auth will use in-memory storage') }`. That branch could
+never execute: `getService` **throws** for an unregistered service rather than
+returning `undefined`, and this plugin declares a hard dependency on ObjectQL
+(which registers `data` unconditionally), so a kernel without the engine fails
+even earlier with `Dependency … not found`. The branch is removed and the real
+contract is declared — `requiresServices: ['data', 'manifest']` — which also
+replaces a trailing `// manifest service required` comment with the
+machine-checked form of the same claim. `AuthManager` keeps its own optional
+`dataEngine` guards: it is usable outside the plugin.
+
+**`plugin-webhook-outbox` was protected only transitively.** It resolves
+`manifest` in `init()` with no fallback while depending on
+`com.objectstack.service.messaging`, which in turn depends on ObjectQL, the
+actual provider. That works today and would have broken silently the day
+messaging stopped depending on the engine — surfacing as a crash inside an
+unrelated plugin's init. It now declares `requiresServices: ['manifest']`
+directly.
+
+Neither change alters ordering or boot outcomes on any current composition:
+both plugins were already ordered correctly. What changes is what a broken
+composition *says*, and that the guarantees are now checked rather than
+inherited.
+
+Docs: `content/docs/plugins/anatomy.mdx` gains the three ADR-0116 fields and
+the decision rule for resolving a service inside `init()` (hard dependency vs
+`optionalDependencies` + `requiresServices`), including the two traps behind
+these fixes — don't rely on a transitive provider, and don't write an
+`if (!svc)` fallback after a bare `getService`. The api-registry example
+declares the contract on all seven of its plugins instead of relying on
+`kernel.use()` order.

.changeset/discovery-remedy-names-a-real-package.md

@@ -0,0 +1,21 @@
+---
+'@objectstack/spec': minor
+'@objectstack/runtime': patch
+'@objectstack/metadata-protocol': patch
+---
+
+Discovery's "install this to enable" now names a package that exists (#4093 follow-up).
+
+Discovery tells a consumer two things about an absent capability: that it is absent, and what to do about it. The first has been carefully honest since #2462/#4000. The second was invented from the slot name.
+
+The dispatcher templated `Install a ${slot} plugin to enable` across twelve slots, and `metadata-protocol` carried a hand-written table in which **ten of fifteen entries named a package that does not exist** — `plugin-redis`, `plugin-bullmq`, `job-scheduler`, `plugin-notifications`, `plugin-storage`, `plugin-automation`, `ui-plugin`, plus `plugin-ai`, `plugin-search` and `plugin-workflow` for slots nothing implements at all. That value is also surfaced as discovery's `provider`.
+
+A remedy naming a package that cannot be installed is a dead end handed to someone at the exact moment they are trying to fix their stack — and an agent reading discovery cannot tell it apart from a package it should install. It is the same `declared ≠ enforced` failure this lineage has been closing, one level over: not "does the capability exist" but "is the fix real".
+
+`CORE_SERVICE_PROVIDER` and `serviceUnavailableMessage()` in `@objectstack/spec/system` are now the one place that sentence is written, and both discovery builders read them, so the two hosts cannot tell a consumer to install different things (the drift #4089 and #4130 closed for the `metadata` and `data` entries). Entries were verified against what actually calls `registerService` for each slot rather than against name similarity — which is how `notification` turned out to be filled by `@objectstack/service-messaging`, the one slot whose package shares no word with its name.
+
+Four slots — `ai`, `search`, `workflow`, `graphql` — have no implementation anywhere, so they now say so instead of naming a plausible package. `ui` keeps the fuller sentence it got in #4146 (`/ui` is served by the `protocol` service; nothing registers the `ui` slot), and that sentence now reaches both builders instead of one.
+
+`scripts/check-service-providers.mjs` (wired into the lint workflow as `check:service-providers`) fails CI when a named package is not a real workspace package, or when a `CoreServiceName` slot has no entry — so a rename or a deletion cannot leave a stale instruction behind.
+
+FROM → TO: `services.<slot>.message` and `services.<slot>.provider` change text for most unavailable slots. Anything matching on the old `Install a <slot> plugin to enable` wording should match on `status: 'unavailable'` instead — the status field is the contract; the message is prose for humans and agents.

.changeset/exported-types-no-longer-resolve-to-any.md

@@ -0,0 +1,84 @@
+---
+"@objectstack/spec": minor
+"@objectstack/platform-objects": patch
+"@objectstack/metadata-protocol": patch
+"@objectstack/driver-mongodb": patch
+---
+
+fix(spec): five exported symbols resolved to `any` — type the recursive schemas and gate it in CI (#4171)
+
+A recursive Zod schema needs an explicit annotation to break its circular
+inference, and five of them took the cheapest one available:
+
+```ts
+export const NavigationItemSchema: z.ZodType<any> = z.lazy(() => …);
+export type NavigationItem = z.infer<typeof NavigationItemSchema>;   // → any
+```
+
+It compiles, it validates correctly at runtime, and it silently throws the type
+away. `NavigationItem`, `FormField`, `JoinNode` and `NormalizedFilter` were all
+`any` on the published surface, plus `FieldNodeSchema` — which had no exported
+type alias yet, so `z.infer<typeof FieldNodeSchema>` was `any` and
+`QueryAST['fields']` with it.
+
+That is worse than a missing export. #4115 tells every consumer that a local
+declaration under a spec export's name must be replaced by a binding to the
+spec — and for these, obeying it **replaced a precise type with `any`**.
+objectui's `NavigationItem` is a 118-line documented interface (`recordId`
+template variables, `requiresObject` / `requiresService` capability gates,
+`filters` precedence); every key of it exists in the spec's version, so by every
+available signal it read as a redundant fork safe to delete. Deleting it swapped
+a fully-typed interface for `any`, with no compile error anywhere to say so.
+
+It is hard to catch by inspection because `any` is mutually assignable with
+everything, so the natural "are these the same type?" check answers *yes* in both
+directions and recommends precisely the wrong action. Same failure family as
+#4075's `[key: string]: any` on `ActionDef`: a type that agrees with everything
+reads as agreement.
+
+**Now annotated with the real type**, using the pattern `QueryAST` already
+follows in `data/query.zod.ts` — infer the non-recursive part, tie the recursive
+knot in the type, so the keys stay derived from the schema instead of being
+hand-maintained beside it:
+
+```ts
+const BaseXSchema = z.object({ …every non-recursive key });
+export type X = z.infer<typeof BaseXSchema> & { children?: X[] };
+export const XSchema: z.ZodType<X> = z.lazy(() => BaseXSchema.extend({
+  children: z.array(XSchema).optional(),
+}));
+```
+
+`z.infer` now resolves to the type it should always have been: `NavigationItem`
+is the nine-branch discriminated union, `FormField` the 30-key form-field
+contract (with `visibleOn` absent by construction — ADR-0089 D2 folds it into
+`visibleWhen` at the boundary), `JoinNode` and the newly exported `FieldNode`
+the query AST nodes, `NormalizedFilter` the normalized filter AST. Runtime
+validation is unchanged: every schema parses exactly what it parsed before.
+
+**What the types immediately caught**, none of it visible while they were `any`:
+
+- `account.app.ts` set `defaultOpen` on three nav groups — a key the spec has
+  never declared. It worked only because objectui's `NavigationRenderer` still
+  falls back to that legacy alias. Fixed at the producer per Prime Directive
+  #12: the canonical key is `expanded`.
+- The MongoDB driver built its projection with `projection[field] = 1` over
+  `query.fields`, so a relationship `FieldNode` would have keyed the projection
+  on `"[object Object]"`. It now reads the node's field name.
+- `setup.app.ts`, `studio.app.ts` and `setup-nav.contributions.ts` are annotated
+  with the PARSED `App` / `NavigationContribution` types but omitted
+  `.default()`ed keys (`expanded`, `target`), as did the form fields
+  `metadata-protocol` synthesizes for `getUiView` (`span`). Each now states the
+  default it was relying on, matching what the surrounding literals already do
+  for `active` / `isDefault` / `collapsible` / `collapsed` / `columns`.
+
+**Gated, not just fixed** (`check:exported-any`, wired into the required
+`TypeScript Type Check` job). `api-surface.json` records that an export *exists*
+and never what it *resolves to*, which is how these survived a whole major with
+every gate green. The new scan reads the built `.d.ts` a consumer's import
+actually resolves to and fails on any exported type that resolves to `any` — or
+any exported schema whose output is `any`, the root cause, and the only reason
+`FieldNodeSchema` was visible at all. Its `KNOWN_ANY` ledger is shrink-only and
+currently empty. It self-tests against the real zod first, so if the internals it
+reads are ever renamed the gate fails loudly instead of quietly passing
+everything forever.

.changeset/stack-storage-not-an-authoring-key.md

@@ -0,0 +1,71 @@
+---
+'@objectstack/spec': minor
+'@objectstack/cli': patch
+---
+
+**`config.storage` is not a stack key, and an undeclared top-level key now says
+so instead of vanishing (#4167).**
+
+`os serve` read `config.storage` and forwarded it to `StorageServicePlugin`.
+It could almost never arrive: `ObjectStackDefinitionSchema` does not declare
+`storage`, and is not `.strict()`, so `defineStack` — which every documented
+authoring path and every compiled artifact goes through — strips the key before
+`serve` runs. The one combination that reached the branch (a bare-object config
+on the config-boot path) then carried the `driver`/`root` spelling the plugin
+does not read either, so it did nothing there too.
+
+The result was one authoring key that worked on a single unreachable-in-practice
+path and disappeared silently everywhere else. A host writing
+`storage: { driver: 's3', … }` believed it had configured S3 and got local disk.
+
+- **`serve` no longer reads it.** `resolveStorageCapabilityArg` takes only the
+  env root; the production warning stops naming `config.storage` and names the
+  two channels that work — `OS_STORAGE_*` and Setup → Settings, the latter being
+  the one with proper credential handling.
+- **The undeclared-key lint now covers the stack's own top-level keys.** New
+  `lintUnknownStackKeys(rawStack, stackSchema)`, wired into `defineStack`,
+  `os validate` and `os compile` beside the existing walker. `storage` gets a
+  prescriptive entry naming both channels and why a stack definition is the
+  wrong home for a credential — it would commit it to git and to any published
+  artifact. An ordinary misspelling still gets the edit-distance suggestion
+  (`datasource` → `datasources`).
+- **`os migrate files-to-references` shares the resolver.** It built
+  `{ driver: 'local', root }` — the same dead keys — so its adapter used
+  `./storage` while the server writes under `.objectstack/data/uploads` since
+  #4096. That command reconciles what records claim against what storage holds,
+  so a disagreeing root reconciled against the wrong tree.
+
+**`onEnable` is exempt, and the exemption has one owner.** `onEnable` is a
+function, so `ObjectStackDefinitionSchema` cannot declare it and
+`dist/objectstack.json` cannot carry it — but it is not lost: `AppPlugin` calls
+it off the authored bundle, and the artifact-boot path grafts it back (#4095).
+"Not declared" and "dropped at load" are different claims, and this is the
+surface where they come apart. New `STACK_RUNTIME_MEMBERS` in `@objectstack/spec`
+names the members the runtime honours off the bundle; the lint treats them as
+declared, and the CLI's `GRAFTABLE_RUNTIME_MEMBERS` is now **derived** from it
+rather than restating it, so the list that decides what gets grafted and the
+list that decides what the lint stays quiet about cannot drift. `onDisable` is
+deliberately not on it — nothing calls it, so a value written there really does
+go nowhere and the lint should say so.
+
+Additive: `lintUnknownAuthoringKeys` keeps its signature. The new pass is a
+separate export rather than a fold into that walker for two reasons. The walker
+iterates metadata COLLECTIONS, so a stack whose only mistake is at the envelope
+level — no objects, no pages, nothing to iterate — walks clean; and the stack
+schema has to be INJECTED, because `stack.zod.ts` imports the lint module and
+importing back would close a cycle. A separate export keeps that requirement
+visible: a call site either asks for the coverage or does not, and its absence
+shows up in a diff. An optional parameter would be the same silent-loss shape
+this rule family exists to report. It follows the walker's posture rule — only a
+schema that STRIPS unknown keys is linted, so if the stack schema ever graduates
+to `.strict()` the parse takes over and this goes quiet.
+
+Verified end to end: authoring `storage:` through `defineStack` warns at load,
+and `os compile` reports it for configs that skip `defineStack`.
+
+Nothing is being taken away that worked. `storage` was never in the schema, is
+not documented anywhere, and has no consumer in `objectstack-ai/cloud` (checked).
+Whether the platform should eventually grow a real in-stack storage declaration
+is a separate question — if so it should follow `datasources`, which solves
+credentials by referencing `sys_secret` rather than inlining them, and that
+deserves an ADR rather than a resurrected undeclared key.

.changeset/temporal-storage-form-axis-tests.md

@@ -0,0 +1,7 @@
+---
+---
+
+Test-only: extend the ADR-0053 D-A3 storage-form axis to `Field.time` and the
+wasm driver (#4191) — legacy-storage conformance sweeps in `driver-sql` and
+`driver-sqlite-wasm`, plus doc-comment formalization of the axis in
+`spec/data/temporal-conformance.ts`. Releases nothing.

.changeset/unknown-key-strictness-data-step.md

@@ -0,0 +1,44 @@
+---
+"@objectstack/spec": minor
+---
+
+Hooks and datasources reject unknown keys (#4001 data step).
+
+Closes the last two entries in the strictness ledger that still carried a
+provisional classification. Both were confirmed authorable the same way: they sit
+in `BUILTIN_METADATA_TYPE_SCHEMAS`, so one shape backs `defineStack()` parsing,
+`/api/v1/meta/types/:type`, and the Studio form.
+
+Now strict:
+
+- `HookSchema` + its `retryPolicy`, and both hook-body branches
+  (`ExpressionBodySchema`, `ScriptBodySchema`). A misspelt `capabilities`
+  stripped to the empty default and the sandbox threw at invocation time instead
+  of at parse; a misspelt `timeoutMs`/`memoryMb` silently downgraded the body to
+  the enclosing hook's limits.
+- `DatasourceSchema` + `pool` / `healthCheck` / `ssl` / `retryPolicy`,
+  `ExternalDatasourceSettingsSchema` + its `validation` block,
+  `DatasourceCapabilities`, and `DriverDefinitionSchema`.
+
+Deliberately still tolerant:
+
+- `HookContextSchema` and its `session` / `provenance` / `user` blocks — the
+  runtime shape the engine hands a handler. Strictness there would turn an
+  engine-internal enrichment (as `provenance` was in #3712) into a breaking
+  change for anyone parsing a context they were given.
+- `datasource.config` and `readReplicas` — per-driver by construction; the
+  driver's own `configSchema` validates them.
+
+Errors are self-fixing: connection keys written one level too high (`host`,
+`port`, `filename`, `url`, …) are prescribed into `config`; a top-level
+`password` is pointed at `external.credentialsRef` rather than merely relocated;
+and the two near-miss spellings that cross between shapes carry aliases
+(hook-level `timeout` vs body-level `timeoutMs`; hook `retryPolicy.backoffMs` vs
+datasource `retryPolicy.baseDelayMs`).
+
+Note for anyone reading the earlier steps: strictness does not change the
+published JSON Schema. `build-schemas.ts` converts with `io: 'output'`, where zod
+emits `additionalProperties: false` for `.strip()` objects too — verified by
+regenerating both ways (`Datasource.json` is byte-identical). The JSON Schema was
+already advertising `additionalProperties: false` while the parse silently
+dropped keys; this aligns the parse with the published contract.