|
| 1 | +# ADR-0062: External Datasource Runtime — connection lifecycle, credentials, visibility & query completeness |
| 2 | + |
| 3 | +**Status**: Proposed — recommended for acceptance; consolidates the runtime gaps surfaced while implementing ADR-0015 federation. Some requirements are already shipped (marked ✅ below); the open decisions are D1–D8 (2026-06-22). |
| 4 | + |
| 5 | +**Supersedes the runtime portions of**: ADR-0015 §18 addendum (kept as the historical record). ADR-0015 remains the canonical spec/binding decision; this ADR is the canonical *runtime* decision. |
| 6 | + |
| 7 | +## TL;DR |
| 8 | + |
| 9 | +ADR-0015 declared external datasource federation. The declaration surface (`schemaMode`, `external.{remoteName,remoteSchema,columnMap,writable}`), the boot validation gate, the write gate, introspection + the runtime "Sync objects" wizard all shipped — and the query path now honors `remoteName`/`remoteSchema` (#2138), `columnMap` (#2149), the federated objects are visible in the datasource admin/meta REST (#2157), and there is a runnable showcase example + dogfood coverage (#2139). |
| 10 | + |
| 11 | +**But "declare a datasource and it just works" is still not true.** A declared, non-`default` datasource is registered as *metadata* (so it is now *visible*) but is **not connected as a queryable ObjectQL engine driver** — querying its federated objects requires the app to hand-register a live driver via an `onEnable` bridge. Three further runtime concerns are unowned: credential resolution at connect, native-analytics SQL over external objects, and the `columnMap`↔`field.columnName` duplication. |
| 12 | + |
| 13 | +This ADR defines the **complete external-datasource runtime contract** and the decisions to close those gaps with **one connection mechanism for code- and runtime-origin datasources**, a **backward-compatible, opt-in rollout** (existing managed apps unchanged), and an explicit phase plan. |
| 14 | + |
| 15 | +## Context |
| 16 | + |
| 17 | +### What ADR-0015 promised vs. what is live |
| 18 | + |
| 19 | +| # | Requirement | Status | Where | |
| 20 | +|---|---|---|---| |
| 21 | +| R1 | Declaration + per-object binding (`schemaMode`, `external.*`) | ✅ shipped | ADR-0015; `spec/data/{datasource,object}.zod.ts` | |
| 22 | +| R2 | **Declared datasource → queryable engine driver (no app boilerplate)** | ❌ **gap** | needs `app-plugin`/`standalone` + a connection service | |
| 23 | +| R3 | Credentials/secrets resolved at connect (`credentialsRef`) | ⚠️ partial | `SecretBinder`/`ICryptoProvider` exist; not wired into a generic connect path | |
| 24 | +| R4 | Visibility in `GET /api/v1/datasources` + `/meta/datasource` + Setup | ✅ shipped | #2157 (in-memory `registerInMemory` fallback) | |
| 25 | +| R5a | Read path honors `remoteName`/`remoteSchema` | ✅ shipped | #2138 | |
| 26 | +| R5b | Read path honors `columnMap` (remote col ≠ local field) | ✅ shipped | #2149 | |
| 27 | +| R5c | Write gate (double opt-in) | ✅ shipped | ADR-0015 §5.3 | |
| 28 | +| R5d | Read coercion for external (best-effort, no DDL) | ✅ shipped | #2138 `registerExternalObject` | |
| 29 | +| R5e | **Native-analytics SQL over external objects** | ❌ **gap** | analytics service compiles its own `FROM` | |
| 30 | +| R6 | Boot schema-validation gate + background drift | ✅ shipped | ADR-0015 §5.2 `external-validation-plugin` | |
| 31 | +| R7 | Introspection (`remote-tables`, `object-draft`) + runtime Sync wizard | ✅ shipped | ADR-0015 addendum; `service-datasource` | |
| 32 | +| R8 | **Connection lifecycle/health/pooling for N datasources** | ❌ **gap** | only the single `default` driver's lifecycle is managed today | |
| 33 | +| R9 | dogfood/verify handles read-only external; canonical example | ✅ shipped | #2139 (`verify` skip + `app-showcase`) | |
| 34 | +| R10 | **`columnMap` ↔ `field.columnName` single source of truth** | ❌ **gap** | two inverse mechanisms coexist | |
| 35 | + |
| 36 | +### The structural gap, precisely (R2) |
| 37 | + |
| 38 | +`os dev` (standalone) connects exactly **one** driver — the `default` library (`.objectstack/data/standalone.db`), built in `packages/runtime/src/standalone-stack.ts` by detecting the driver kind from the DB URL. A `defineStack({ datasources: [...] })` entry is only registered as metadata by `packages/runtime/src/app-plugin.ts` (`registerInMemory('datasource', …)`). ObjectQL routes queries by **driver name** (`engine.getDriver(object) → this.drivers.get(object.datasource)`) and throws `Datasource 'x' is not registered` when absent. So the only way to make a federated object queryable today is the app's `onEnable` hook calling `ctx.drivers.register(driver)` (= `engine.registerDriver`) — see `examples/app-showcase/src/datasources/external-fixture.ts`. That bridge is framework plumbing leaking into every federation app. |
| 39 | + |
| 40 | +### Why one consolidating ADR (not N small PRs) |
| 41 | + |
| 42 | +R2/R3/R8 are the same change seen from three angles — you cannot "auto-connect" without owning **connection lifecycle** (R8) and **credential resolution** (R3). R4 (just shipped) and R2 are two faces of "declared datasource as a runtime first-class citizen" — visible *and* usable. And the connection mechanism must **converge** the code-origin path (this ADR) with the runtime-origin path (the Sync wizard already builds drivers via the injected factory for test/introspection) so we don't ship two divergent connect paths. These decisions share contracts (the injectable driver factory, the `SecretBinder`, boot ordering vs. the `kernel:ready` validation gate); deciding them piecemeal risks incoherent seams. |
| 43 | + |
| 44 | +## Goals / Non-goals |
| 45 | + |
| 46 | +**Goals.** A declared external datasource is, with no app code: visible (R4 ✅), connected + queryable (R2), credential-resolved (R3), validated at boot (R6 ✅), and lifecycle-managed (R8) — through **one** mechanism shared by code- and runtime-origin datasources, **without changing the behavior of existing managed apps**. |
| 47 | + |
| 48 | +**Non-goals.** New driver dialects; replacing `DataSync`/`ExternalLookup` (ADR-0015 §9 coexistence stands); cross-datasource JOINs in one query (federated objects are queried per-datasource); multi-tenant credential vaulting beyond the existing `SecretBinder` seam. |
| 49 | + |
| 50 | +## Decision |
| 51 | + |
| 52 | +### D1 — One `DatasourceConnectionService`; declared datasources auto-connect |
| 53 | + |
| 54 | +Introduce a single service that, given a datasource definition, builds a driver via the **injected** driver factory (the same `createDefaultDatasourceDriverFactory` used by the runtime-admin path), connects it, and registers it into the ObjectQL engine under the datasource name. `app-plugin` calls it for every declared datasource (in addition to today's `registerInMemory` for visibility); `standalone-stack`'s single-`default`-driver bootstrap is refactored to go through the same service so there is exactly one "definition → live driver" code path. `engine.registerDriver` remains the sink. The `onEnable` bridge becomes unnecessary for the common case (kept as an escape hatch — D8). |
| 55 | + |
| 56 | +### D2 — Connect is opt-in-safe: existing managed apps are byte-for-byte unchanged |
| 57 | + |
| 58 | +Auto-connect must not change apps that today declare datasources that are *decorative* or routed via `datasourceMapping` (e.g. `examples/app-crm`'s `crm_primary`/`crm_analytics`). Gate auto-connect so a declared datasource is only connected when it is meaningfully addressed: **(a)** it is `external` (`schemaMode !== 'managed'`), or **(b)** an object/`datasourceMapping` actually routes to it, or **(c)** it sets an explicit `autoConnect: true`. A managed datasource that nothing routes to stays metadata-only (today's behavior). The `default` datasource keeps its current dedicated bootstrap. This is the load-bearing backward-compat decision. |
| 59 | + |
| 60 | +### D3 — Credentials resolved at connect via `SecretBinder`/`ICryptoProvider` |
| 61 | + |
| 62 | +`DatasourceConnectionService` resolves `external.credentialsRef` (and any `secret` config fields) through the host-provided `SecretBinder` over `ICryptoProvider` **before** building the driver. Open-core default is `InMemoryCryptoProvider`; a datasource that needs a secret the host cannot decrypt **fails closed** (clear boot error, datasource left unconnected — not a silent skip). This reuses the exact mechanism the runtime-admin "Add Datasource" wizard already uses, so code- and runtime-origin secrets converge. |
| 63 | + |
| 64 | +### D4 — Visibility converges on the metadata registry (shipped, ratified here) |
| 65 | + |
| 66 | +Code-defined datasources surface in `GET /api/v1/datasources`, `GET /api/v1/meta/datasource` and **Setup → Datasources** via the single in-memory metadata registry, stamped `origin: 'code'` (read-only in the admin UI). Shipped in #2157 (the in-memory metadata fallback was missing `registerInMemory`, silently skipping registration on the host-config boot path). This ADR ratifies that the metadata registry is the single source for datasource listing across all boot paths; the admin `listDatasources` and `protocol.getMetaItems` both read it. |
| 67 | + |
| 68 | +### D5 — Lifecycle, health & ordering for N datasources |
| 69 | + |
| 70 | +`DatasourceConnectionService` owns connect/disconnect (graceful shutdown), pool config per datasource, and an optional health probe surfaced in the admin list (`status`). **Ordering**: all declared datasources connect **before** the `kernel:ready` external-validation gate (ADR-0015 §5.2) and before first query — i.e. during plugin init/start, not in a `kernel:ready` handler. Connect failure policy is **fail-fast for `external` with `validation.onMismatch: 'fail'`**, **degrade-with-warning** otherwise (a connectivity blip on an optional analytics replica should not brick boot). |
| 71 | + |
| 72 | +### D6 — Native-analytics SQL honors the remote table/columns |
| 73 | + |
| 74 | +The analytics native-SQL strategy compiles its own `FROM "<table>"` / column references outside the driver (ADR-0015 §18 noted this). It must resolve an external object's physical table (`remoteName`/`remoteSchema`) and columns (`columnMap`) the same way `SqlDriver` now does — reusing the driver's resolution (e.g. an exposed `physicalTableFor(object)` / `physicalColumnFor(object, field)`), not a second copy. Until then, analytics over external objects stays disabled rather than silently querying the wrong table. |
| 75 | + |
| 76 | +### D7 — `columnMap` is the external mechanism; reconcile `field.columnName` |
| 77 | + |
| 78 | +`external.columnMap` ({ remoteColumn → localField }) is the supported way to map external columns (shipped #2149). `field.columnName` (localField → physicalColumn) is its inverse and is **not** applied by the driver's query pipeline for external objects. Decision: for external objects, `columnMap` is authoritative; `field.columnName` on an external object is rejected at validation (no silent dual-source) until a unified column-resolution model is designed. Managed objects' `field.columnName` semantics are untouched. |
| 79 | + |
| 80 | +### D8 — Drop the `onEnable` bridge from the canonical example; keep it as an escape hatch |
| 81 | + |
| 82 | +Once D1 lands, `examples/app-showcase` declares its external datasource with **no** `onEnable` driver-registration code (fixture provisioning may remain). `onEnable` + `ctx.drivers.register` stays supported for advanced/dynamic cases (drivers built at runtime from external config). |
| 83 | + |
| 84 | +### Default behavior (normative) |
| 85 | + |
| 86 | +- A declared `external` datasource auto-connects (read-only unless write double-opt-in), is visible, validated at boot, and its federated objects are queryable via REST/ObjectQL — **zero app code**. |
| 87 | +- A declared `managed` datasource that nothing routes to remains metadata-only (unchanged). |
| 88 | +- Missing/undecryptable credentials → fail-closed with a clear error. |
| 89 | +- `onEnable` registration still wins if present (escape hatch). |
| 90 | + |
| 91 | +## Phasing |
| 92 | + |
| 93 | +- **Phase 0 — shipped**: read-path `remoteName`/`remoteSchema` (#2138), `columnMap` (#2149), showcase example + `verify` skip (#2139), visibility (#2157), ADR-0015 §18. |
| 94 | +- **Phase 1 — `DatasourceConnectionService` + auto-connect (D1, D2, D5)**: the core. Extract "definition → live driver" from `standalone-stack`, call from `app-plugin`, gate per D2. Refactor `default` onto the same service last (highest-risk; do behind tests). |
| 95 | +- **Phase 2 — credentials at connect (D3)**: wire `SecretBinder`; converge with runtime-admin secret handling; fail-closed. |
| 96 | +- **Phase 3 — analytics SQL (D6)**: expose driver physical-table/column resolution; route the native-SQL strategy through it. |
| 97 | +- **Phase 4 — reconciliation + cleanup (D7, D8)**: validation for `field.columnName` on external; drop the showcase `onEnable` bridge. |
| 98 | + |
| 99 | +Each phase is its own PR with its own changeset; Phase 1 lands behind the full dogfood gate (every example app boots) before Phase 4 removes the bridge. |
| 100 | + |
| 101 | +## Backward compatibility / migration / blast radius |
| 102 | + |
| 103 | +- `app-plugin` runs for **every** stack, so D1 is high-blast — D2's opt-in gate is what keeps `app-crm` and any current datasource-declaring app unchanged (managed + unrouted → still metadata-only). |
| 104 | +- `onEnable` + `ctx.drivers.register` remains supported (no migration forced). |
| 105 | +- Connecting N datasources introduces N connection lifecycles (pools, disconnect on shutdown, connect-error policy) where only one existed — covered by D5 and the dogfood gate. |
| 106 | +- Code- and runtime-origin datasources converge on one connect path (D1/D3), removing a latent divergence rather than adding one. |
| 107 | + |
| 108 | +## Rejected alternatives |
| 109 | + |
| 110 | +1. **Status quo — `onEnable` is the only way to connect.** Rejected: leaks framework plumbing into every federation app; the canonical example needed ~40 lines of driver wiring just to be queryable. |
| 111 | +2. **Auto-connect every declared datasource by default.** Rejected: changes the runtime behavior of existing managed apps (e.g. `app-crm`'s `:memory:` datasources) — D2's gate is required. |
| 112 | +3. **Separate connect mechanisms for code vs. runtime datasources.** Rejected: two divergent paths for lifecycle + secrets; converge on one service (D1). |
| 113 | +4. **A second copy of table/column resolution inside the analytics service.** Rejected (D6): drift risk; reuse the driver's resolution. |
| 114 | +5. **Support both `columnMap` and `field.columnName` for external objects.** Rejected (D7): silent dual-source ambiguity; pick one until a unified model is designed. |
| 115 | + |
| 116 | +## Consequences |
| 117 | + |
| 118 | +**Positive.** "Declare a datasource → it's visible, connected, validated, queryable" with no app code; one connection + secret path for code and runtime origins; analytics works over external objects; the canonical example shows the real flow without plumbing. |
| 119 | + |
| 120 | +**Negative / cost.** A new connection service + lifecycle ownership; broad regression surface (every example app boots through `app-plugin`); credential plumbing pulled into the boot path. |
| 121 | + |
| 122 | +**Risk.** Phase 1's `default`-driver refactor is the riskiest single step (the hot boot path for all apps) — mitigated by landing auto-connect for declared datasources first, refactoring `default` onto the shared service last, behind the dogfood gate. Connect-error policy (D5) must be conservative so an optional replica's blip never bricks boot. |
| 123 | + |
| 124 | +## References |
| 125 | + |
| 126 | +- ADR-0015 — External Datasource Federation (+ §18 runtime addendum). |
| 127 | +- ADR-0008 — Metadata repository (datasource is a first-class metadata type). |
| 128 | +- PRs: #2138 (read path), #2149 (columnMap), #2139 (showcase example + `verify` skip), #2157 (visibility). |
| 129 | +- `packages/runtime/src/standalone-stack.ts`, `packages/runtime/src/app-plugin.ts` (connection + registration sites). |
| 130 | +- `packages/objectql/src/engine.ts` (`getDriver`/`registerDriver` routing). |
| 131 | +- `packages/services/service-datasource/src/*` (driver factory, admin, external service, `SecretBinder`). |
| 132 | +- `packages/plugins/driver-sql/src/sql-driver.ts` (`registerExternalObject`, physical table/column resolution). |
0 commit comments