feat(datasource): fail-closed credential resolution at connect (ADR-0062 Phase 2, D3) (#2199)

* feat(datasource): fail-closed credential resolution at connect (ADR-0062 Phase 2, D3)

A declared external.credentialsRef MUST resolve to a cleartext secret before the
driver is built — an absent secret store or an unresolvable/undecryptable ref now
fails closed (clear message, datasource left unconnected) instead of silently
building a driver without the credential. Follows the same fail-fast (declared
external + onMismatch:fail) vs degrade policy as connect failures. Converges with
the runtime-admin secret path (same SecretBinder threaded through the shared
connection service).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(analytics): analytics over federated objects honors the remote table (ADR-0062 Phase 3, D6) (#2200)

* feat(datasource): reject field.columnName on external + drop showcase onEnable bridge (ADR-0062 Phase 4, D7/D8) (#2203)

---------

Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: 3 people

.changeset/adr-0062-analytics-external.md

@@ -0,0 +1,20 @@
+---
+"@objectstack/spec": minor
+"@objectstack/service-analytics": minor
+---
+
+feat(analytics): correct analytics over federated objects (ADR-0062 Phase 3, D6)
+
+Analytics over an external (federated) object now aggregates against the
+**correct** remote table instead of silently querying the wrong one. The
+`NativeSQLStrategy` hand-compiles `FROM "<object>"` and bare column references,
+which bypass the driver's physical-table resolution (`external.remoteName` /
+`remoteSchema` / `columnMap`). It now **declines** any query whose base or joined
+object is federated, routing it to the `ObjectQLStrategy` — whose
+`engine.aggregate()` goes through the driver's `getBuilder` and already honours
+`remoteName`/`remoteSchema` (#2138/#2149). This "reuses the driver's resolution"
+(D6) rather than re-implementing it.
+
+Adds an optional `StrategyContext.isExternalObject(objectName)` hook (reported by
+the analytics plugin from the object's `external` block). Purely additive — with
+no hook, behavior is unchanged for managed objects.

.changeset/adr-0062-credentials-fail-closed.md

@@ -0,0 +1,20 @@
+---
+"@objectstack/service-datasource": minor
+---
+
+feat(datasource): fail-closed credential resolution at connect (ADR-0062 Phase 2, D3)
+
+`DatasourceConnectionService` now treats a declared `external.credentialsRef` as
+**fail-closed**: the credential must resolve to a cleartext secret (via the
+host's `SecretBinder` over `ICryptoProvider`) *before* the driver is built. An
+absent secret store, or a ref that cannot be resolved/decrypted (missing
+`sys_secret` row, rotated key, or a throwing resolver), leaves the datasource
+**unconnected with a clear message** — never a silent build-without-secret that
+would connect with no/wrong auth or fail later with a confusing driver error.
+
+The same policy as connect failures applies: a code-defined `external` datasource
+with `validation.onMismatch: 'fail'` auto-connected at boot fails fast (bricks
+boot); runtime-admin create/update + boot rehydration degrade-with-warning. Code-
+and runtime-origin secrets converge on the one connection path (the same
+`SecretBinder` is threaded through the shared service). New `failed-credentials`
+connect status.

.changeset/adr-0062-reconcile-cleanup.md

@@ -0,0 +1,22 @@
+---
+"@objectstack/cli": minor
+"@objectstack/verify": minor
+---
+
+feat(datasource): reject field.columnName on external objects + drop showcase onEnable bridge (ADR-0062 Phase 4, D7/D8)
+
+**D7 — reconcile column mapping.** `os compile`/`build` (`validateStackExpressions`)
+now rejects `field.columnName` on a federated (external) object with a corrective
+message: the driver's query pipeline ignores `field.columnName` for external
+objects, so `external.columnMap` is the single authoritative mechanism. Managed
+objects are untouched.
+
+**D8 — drop the canonical example's driver bridge.** `examples/app-showcase`
+declares its external datasource with **no** `onEnable` driver registration — the
+declared datasource auto-connects at boot (ADR-0062 D1). `onEnable` now only
+provisions the "remote" fixture tables. To cover this end-to-end, the
+`@objectstack/verify` harness wires the datasource-admin plugin (registering the
+`'datasource-connection'` service) when an app declares datasources, so it mirrors
+`objectstack dev`/serve; a new dogfood test reads the federated objects through the
+real REST stack (incl. the `remoteName` remap). `onEnable` + `ctx.drivers.register`
+remains supported as an escape hatch for drivers built dynamically at runtime.

docs/adr/0062-external-datasource-runtime.md

@@ -77,10 +77,14 @@ Code-defined datasources surface in `GET /api/v1/datasources`, `GET /api/v1/meta
 
 The analytics native-SQL strategy compiles its own `FROM "<table>"` / column references outside the driver (ADR-0015 §18 noted this). It must resolve an external object's physical table (`remoteName`/`remoteSchema`) and columns (`columnMap`) the same way `SqlDriver` now does — reusing the driver's resolution (e.g. an exposed `physicalTableFor(object)` / `physicalColumnFor(object, field)`), not a second copy. Until then, analytics over external objects stays disabled rather than silently querying the wrong table.
 
+> **Phase 3 implementation note (#2163) — "reuse the driver's resolution" = route external objects to the ObjectQL path.** Rather than re-implement `remoteName`/`remoteSchema`/`columnMap` resolution inside the native-SQL strategy (a second copy — explicitly rejected), `NativeSQLStrategy.canHandle` now **declines** any query whose base or joined object is federated (new optional `StrategyContext.isExternalObject` hook, reported by the analytics plugin from the object's `external` block). Declining routes the query to the lower-priority `ObjectQLStrategy`, whose `engine.aggregate()` already goes through the driver's `getBuilder` — which honours `remoteName`/`remoteSchema` (#2138/#2149). So external analytics aggregates against the **correct** remote table via the single source of truth (the driver), and native-SQL never queries the wrong table. (A native-SQL fast path for external objects can be added later by exposing `physicalTableFor`/`physicalColumnFor` on the driver; deeper `columnMap`-in-`GROUP BY` support is a separate driver concern.)
+
 ### D7 — `columnMap` is the external mechanism; reconcile `field.columnName`
 
 `external.columnMap` ({ remoteColumn → localField }) is the supported way to map external columns (shipped #2149). `field.columnName` (localField → physicalColumn) is its inverse and is **not** applied by the driver's query pipeline for external objects. Decision: for external objects, `columnMap` is authoritative; `field.columnName` on an external object is rejected at validation (no silent dual-source) until a unified column-resolution model is designed. Managed objects' `field.columnName` semantics are untouched.
 
+> **Phase 4 implementation note (#2163).** *D7* is enforced at build time in `validateStackExpressions` (`os compile`/`build`): any object that declares an `external` binding and a field with `columnName` is an error with a corrective message (use `external.columnMap`). Managed objects are untouched. *D8*: `examples/app-showcase` now declares its external datasource with **no** `onEnable` driver registration — `onEnable` only provisions the "remote" fixture tables; the declared datasource auto-connects (D1). To exercise this in the dogfood gate, the `@objectstack/verify` harness now wires the datasource-admin plugin (hence the `'datasource-connection'` service) when an app declares datasources — so the harness matches `objectstack dev`/serve and the federated read is covered end-to-end (a new dogfood test reads `showcase_ext_customer`/`_order`, including the `remoteName` remap). `onEnable` + `ctx.drivers.register` remains documented as the escape hatch for drivers built dynamically at runtime.
+
 ### D8 — Drop the `onEnable` bridge from the canonical example; keep it as an escape hatch
 
 Once D1 lands, `examples/app-showcase` declares its external datasource with **no** `onEnable` driver-registration code (fixture provisioning may remain). `onEnable` + `ctx.drivers.register` stays supported for advanced/dynamic cases (drivers built at runtime from external config).

examples/app-showcase/objectstack.config.ts

@@ -129,11 +129,13 @@ export default defineStack({
   // AI-authored metadata survive restarts (a `:memory:` datasource would wipe
   // everything on every restart, which makes local app-building unusable).
   //
-  // External-datasource federation demo (ADR-0015): a second, read-only SQLite
-  // file declared as a code-defined external datasource. It appears in
-  // Setup → Datasources and its federated objects (below) are queryable via
-  // REST. The fixture file + live driver are wired by `onEnable` (bottom of
-  // this file). `os dev` needs no extra setup.
+  // External-datasource federation demo (ADR-0015 / ADR-0062): a second,
+  // read-only SQLite file declared as a code-defined external datasource. It
+  // appears in Setup → Datasources and its federated objects (below) are
+  // queryable via REST — with NO driver wiring: the declared `external`
+  // datasource AUTO-CONNECTS at boot (ADR-0062 D1/D8). `onEnable` (bottom of
+  // this file) only provisions the "remote" fixture file's tables + seed data;
+  // `os dev` needs no extra setup.
   datasources: [ShowcaseExternalDatasource],
 
   // i18n
@@ -185,11 +187,14 @@ export default defineStack({
 });
 
 /**
- * Runtime wiring for the external-datasource federation demo (ADR-0015).
+ * Provisions the "remote" fixture database for the external-datasource
+ * federation demo (ADR-0015 / ADR-0062). Creating + seeding the remote tables is
+ * CODE (DDL on a separate SQLite file), so it can't live in the declarative
+ * artifact — it runs here. The AppPlugin invokes `onEnable` at boot.
  *
- * Code (fixture provisioning + live external-driver registration) can't live in
- * the declarative artifact, so it runs here. The AppPlugin invokes `onEnable` at
- * boot (Phase 2), before the external-validation gate fires on `kernel:ready`.
+ * NOTE (ADR-0062 D8): this no longer registers the external driver. The declared
+ * `external` datasource auto-connects at boot, so the federated objects are
+ * queryable with no driver wiring.
  */
 export const onEnable = async (ctx: unknown): Promise<void> => {
   await setupShowcaseExternalDatasource(ctx as Parameters<typeof setupShowcaseExternalDatasource>[0]);

examples/app-showcase/src/datasources/external-fixture.ts

@@ -3,22 +3,22 @@
 import { SqlDriver } from '@objectstack/driver-sql';
 
 /**
- * Runtime wiring for the external-datasource federation demo (ADR-0015).
+ * Fixture provisioning for the external-datasource federation demo
+ * (ADR-0015 / ADR-0062).
  *
- * This is CODE, so it can't live in the declarative artifact — it runs from the
- * stack's `onEnable` hook, which the AppPlugin invokes at boot (Phase 2), before
- * the external-validation gate fires on `kernel:ready` (Phase 3).
+ * This is the "remote database" stand-in: it idempotently creates the separate
+ * SQLite file with `customers` / `orders` tables + a little seed data (a MANAGED
+ * driver — DDL allowed) so `os dev` needs no external server. It runs from the
+ * stack's `onEnable` hook at boot.
  *
- * It does three things, all zero-config so `os dev` "just works":
- *   1. Idempotently provisions the "remote" SQLite file with `customers` /
- *      `orders` tables + a little seed data (a MANAGED driver — DDL allowed).
- *   2. Registers a read-only `schemaMode: 'external'` driver for that same file
- *      under the datasource name `showcase_external`, so ObjectQL can route
- *      queries to it. (Declared datasources are surfaced in the metadata
- *      registry for Setup → Datasources, but a live driver must be registered
- *      for the federated objects to be queryable in standalone mode.)
- *   3. Registers the federated objects' read metadata (DDL-free) so coercion +
- *      the remote-table mapping exist immediately.
+ * **It no longer registers a driver or syncs object schemas (ADR-0062 D8).** The
+ * declared `external` datasource (see `showcase-external.datasource.ts`) now
+ * AUTO-CONNECTS: at boot the runtime's `DatasourceConnectionService` builds the
+ * read-only `schemaMode:'external'` driver, registers it under the datasource
+ * name, and registers the federated objects' read metadata — with zero app code.
+ * The old `onEnable` + `ctx.drivers.register` bridge is gone; `onEnable` +
+ * `ctx.drivers.register` remains *supported* only as an advanced escape hatch
+ * (drivers built dynamically at runtime).
  */
 
 // Same relative path as the datasource config — resolved against the project
@@ -60,15 +60,17 @@ const ORDER_ROWS = [
   { id: 'o4', customer_id: 'c3', amount: 3300, status: 'paid', placed_on: '2026-03-01' },
 ];
 
-/** Stack `onEnable` payload (subset we use). */
+/** Stack `onEnable` payload (subset we use — provisioning only logs). */
 interface OnEnableContext {
-  ql: { syncObjectSchema?: (name: string) => Promise<void> };
-  drivers: { register: (driver: unknown) => void };
   logger?: { info?: (msg: string, meta?: unknown) => void; warn?: (msg: string, meta?: unknown) => void };
 }
 
+/**
+ * Provision the "remote" fixture database. Idempotent: creates the tables and
+ * seeds them only when empty. Does NOT register a driver — the declared external
+ * datasource auto-connects at boot (ADR-0062 D1/D8).
+ */
 export async function setupShowcaseExternalDatasource(ctx: OnEnableContext): Promise<void> {
-  // 1. Provision the remote fixture with a MANAGED driver (DDL allowed). Idempotent.
   const fixture = new SqlDriver({
     client: 'better-sqlite3',
     connection: { filename: EXTERNAL_DB_FILE },
@@ -93,21 +95,7 @@ export async function setupShowcaseExternalDatasource(ctx: OnEnableContext): Pro
     await fixture.disconnect();
   }
 
-  // 2. Register the read-only EXTERNAL driver under the datasource name.
-  const ext = new SqlDriver({
-    client: 'better-sqlite3',
-    connection: { filename: EXTERNAL_DB_FILE },
-    useNullAsDefault: true,
-    schemaMode: 'external',
-  } as never) as unknown as { name: string; connect: () => Promise<void> };
-  ext.name = 'showcase_external'; // MUST equal the datasource name (ObjectQL routes by driver name).
-  await ext.connect();
-  ctx.drivers.register(ext);
-
-  // 3. Register read metadata for the federated objects (DDL-free; ADR-0015).
-  //    Needed because the driver is registered after the boot schema-sync.
-  await ctx.ql.syncObjectSchema?.('showcase_ext_customer');
-  await ctx.ql.syncObjectSchema?.('showcase_ext_order');
-
-  ctx.logger?.info?.('[showcase] external datasource "showcase_external" ready — federation demo (ADR-0015)');
+  ctx.logger?.info?.(
+    '[showcase] external fixture provisioned — datasource "showcase_external" auto-connects (ADR-0062)',
+  );
 }

packages/cli/src/utils/validate-expressions.test.ts

@@ -343,4 +343,55 @@ describe('validateStackExpressions (ADR-0032 build-time)', () => {
       expect(issues.some(i => i.where.includes('when') && /bare reference `tier`/.test(i.message))).toBe(true);
     });
   });
+
+  describe('ADR-0062 D7 — field.columnName on external objects', () => {
+    it('rejects field.columnName on a federated (external) object', () => {
+      const issues = validateStackExpressions({
+        objects: [{
+          name: 'ext_customer',
+          datasource: 'warehouse',
+          external: { remoteName: 'customers' },
+          fields: { region: { type: 'text', columnName: 'cust_region' } },
+        }],
+      });
+      const issue = issues.find(i => /columnName/.test(i.message));
+      expect(issue).toBeDefined();
+      expect(issue!.severity).toBe('error');
+      expect(issue!.where).toContain("object 'ext_customer'");
+      expect(issue!.where).toContain("field 'region'");
+      expect(issue!.message).toMatch(/external\.columnMap|ADR-0062 D7/);
+    });
+
+    it('rejects field.columnName declared as a map-shaped field too', () => {
+      const issues = validateStackExpressions({
+        objects: [{
+          name: 'ext_order',
+          external: {},
+          fields: { amount: { type: 'number', columnName: 'amt' } },
+        }],
+      });
+      expect(issues.some(i => /columnName/.test(i.message) && i.where.includes("field 'amount'"))).toBe(true);
+    });
+
+    it('leaves field.columnName on a MANAGED object untouched (no issue)', () => {
+      const issues = validateStackExpressions({
+        objects: [{
+          name: 'crm_account',
+          fields: { region: { type: 'text', columnName: 'acct_region' } },
+        }],
+      });
+      expect(issues.some(i => /columnName/.test(i.message))).toBe(false);
+    });
+
+    it('passes an external object that uses no columnName', () => {
+      const issues = validateStackExpressions({
+        objects: [{
+          name: 'ext_customer',
+          external: { remoteName: 'customers' },
+          fields: { region: { type: 'text' } },
+        }],
+      });
+      expect(issues.some(i => /columnName/.test(i.message))).toBe(false);
+    });
+  });
 });

packages/cli/src/utils/validate-expressions.ts

@@ -150,6 +150,35 @@ export function validateStackExpressions(stack: AnyRec): ExprIssue[] {
     const fieldList = Array.isArray(fields)
       ? (fields as AnyRec[])
       : (fields && typeof fields === 'object' ? Object.values(fields as AnyRec) as AnyRec[] : []);
+
+    // ── ADR-0062 D7 — reject `field.columnName` on external objects ──────
+    // `field.columnName` (localField → physicalColumn) is the managed-object
+    // mechanism; it is NOT applied by the driver's query pipeline for federated
+    // objects, where `external.columnMap` (remoteColumn → localField) is the
+    // authoritative — and inverse — mapping. Allowing both would be a silent
+    // dual-source ambiguity, so reject `columnName` on any object that declares
+    // an `external` binding (a federated object). Managed objects are untouched.
+    if (obj.external != null) {
+      const fieldEntries: Array<[string, AnyRec]> = Array.isArray(fields)
+        ? (fields as AnyRec[]).map((f) => [((f as AnyRec)?.name as string) ?? '?', f as AnyRec])
+        : (fields && typeof fields === 'object'
+            ? (Object.entries(fields as AnyRec) as Array<[string, AnyRec]>)
+            : []);
+      for (const [fname, fdef] of fieldEntries) {
+        if (fdef && typeof fdef === 'object' && (fdef as AnyRec).columnName != null) {
+          issues.push({
+            where: `object '${objectName}' · field '${fname}'`,
+            message:
+              `external object '${objectName}': field '${fname}' sets columnName='${String((fdef as AnyRec).columnName)}', ` +
+              `which is not supported on federated objects (ADR-0062 D7). The driver's query pipeline ignores ` +
+              `field.columnName for external objects; map remote columns via the datasource's external.columnMap instead.`,
+            source: `columnName='${String((fdef as AnyRec).columnName)}'`,
+            severity: 'error',
+          });
+        }
+      }
+    }
+
     for (const f of fieldList) {
       // Field-level conditional rules are server-enforced (rule-validator) and
       // record-scoped — a bare ref silently fails the rule (required/readonly