Epic: External Datasource Runtime — implement ADR-0062 (auto-connect, credentials, analytics, cleanup)

[xuyushun441-sys]

Tracking/epic issue to drive ADR-0062 — External Datasource Runtime to completion.

• ADR: docs/adr/0062-external-datasource-runtime.md (merged in docs(adr): ADR-0062 — External Datasource Runtime (connection, credentials, visibility, query completeness) #2162)
• Builds on ADR-0015 (federation). Goal: declare an external datasource → it's visible, auto-connected, validated, and queryable with zero app boilerplate, through one connection mechanism shared by code- and runtime-origin datasources, backward-compatible (existing managed apps unchanged).

Phase 0 — shipped ✅

• Read path honors remoteName / remoteSchema — fix(driver-sql,objectql): honor external.remoteName/remoteSchema in the federation read path (ADR-0015) #2138
• Read path honors external.columnMap (remote col ≠ local field) — feat(driver-sql): honor external.columnMap on federated objects (ADR-0015) #2149
• Canonical showcase example + objectstack verify skip for read-only external — feat(app-showcase): add an end-to-end external datasource + federated object example (#2111) #2139
• Code-defined datasources visible in /api/v1/datasources + /meta/datasource + Setup — fix(runtime): surface code-defined datasources on the host-config boot path (ADR-0015 §18) #2157

Phase 1 — DatasourceConnectionService + auto-connect (D1, D2, D5) — core

• D1 New DatasourceConnectionService: given a datasource def, build a driver via the injected factory, connect, and engine.registerDriver under the datasource name. One "definition → live driver" path (extract from standalone-stack.ts, call from app-plugin.ts).
• D2 Opt-in-safe gating: connect only when the datasource is external, or an object/datasourceMapping routes to it, or autoConnect: true. A managed+unrouted datasource stays metadata-only. examples/app-crm and other existing apps must be byte-for-byte unchanged.
• D5 Lifecycle: connect/disconnect (graceful shutdown), per-datasource pool, optional health probe in the admin list; connect before the kernel:ready validation gate; fail-fast for external + onMismatch:'fail', degrade-with-warning otherwise.
• Refactor the single default-driver bootstrap onto the same service last (highest-risk; behind tests).
• Acceptance: a stack that only declares an external datasource (no onEnable) can query its federated objects via REST; full dogfood gate green (every example app boots); app-crm behavior unchanged.

Phase 2 — credentials at connect (D3)

• Resolve external.credentialsRef / secret config via SecretBinder over ICryptoProvider before building the driver; converge with the runtime-admin "Add Datasource" secret path.
• Fail-closed (clear boot error, datasource left unconnected) when the host can't decrypt; open-core default InMemoryCryptoProvider.
• Acceptance: a Postgres external datasource with credentialsRef connects using the decrypted secret; missing/undecryptable secret fails closed with a clear message (no silent skip).

Phase 3 — native-analytics SQL over external objects (D6)

• Route the analytics native-SQL strategy's FROM/column references through the driver's physical-table/column resolution (remoteName/remoteSchema/columnMap) — reuse, don't re-implement.
• Until done, analytics over external objects stays disabled rather than querying the wrong table.
• Acceptance: a dashboard/report over a federated object aggregates against the correct remote table/columns.

Phase 4 — reconciliation + cleanup (D7, D8)

• D7 For external objects, columnMap is authoritative; reject field.columnName on external objects at validation (no silent dual-source) until a unified column-resolution model exists. Managed field.columnName untouched.
• D8 Drop the onEnable driver-registration bridge from examples/app-showcase (fixture provisioning may remain); keep onEnable + ctx.drivers.register documented as the escape hatch.
• Acceptance: showcase declares its external datasource with no onEnable driver wiring and still works end-to-end.

Notes

• Each phase = its own PR + changeset (published packages are in the fixed group).
• Phase 1's default-driver refactor is the riskiest single step (hot boot path for all apps) — land declared-datasource auto-connect first, refactor default last, behind the dogfood gate.
• Coordinate with the already-shipped visibility work (fix(runtime): surface code-defined datasources on the host-config boot path (ADR-0015 §18) #2157) — both are "declared datasource as a runtime first-class citizen".

References

• ADR-0062 (docs(adr): ADR-0062 — External Datasource Runtime (connection, credentials, visibility, query completeness) #2162), ADR-0015 (+ §18 addendum)
• Sites: packages/runtime/src/{standalone-stack,app-plugin}.ts, packages/objectql/src/engine.ts, packages/services/service-datasource/*, packages/plugins/driver-sql/src/sql-driver.ts

[xuyushun441-sys]

Phase 1 note — leave an open, de-plan-ized connect-policy seam (don't fork for cloud)

Recording a design constraint surfaced while reviewing the open/closed boundary for this epic.

Conclusion: everything in this epic (D1–D8, all phases) is open-core runtime mechanism — it all lands in framework, nothing is EE/closed. ADR-0062's Non-goals already scope out "multi-tenant credential vaulting beyond the SecretBinder seam", so the closed pieces (KMS SecretBinder impl, AI introspection, multi-tenant egress/quota governance) live in the cloud repo on top of the seams this epic exposes — not in here.

But the multi-tenant host (shared-container cloud) needs to be able to disable external-datasource connect (SSRF / egress isolation) while self-hosted single-env runs it on by default. So Phase 1's DatasourceConnectionService must consult an injectable connect policy before building/registering a driver — not hard-code "always connect".

Concretely, when implementing D1/D5:

• Add an injectable DatasourceConnectPolicy (or fold it into the D5 connect-error/lifecycle hook) consulted before connect, alongside the existing D2 gate (external / routed / autoConnect:true).
• Open-core default impl = allow (subject to D2) → self-host works out of the box, de-plan-ized, no plan coupling in framework.
• The host can bind a stricter policy. Cloud will bind one that checks sys_environment.plan + egress allow-list + per-tenant quota and fail-closes on the shared runtime — that impl stays in the cloud repo.

Net: framework gains one open policy interface + a permissive default. Same single connect path for code- and runtime-origin (D1) — the host injects a stricter policy, we do not fork a second connect path for cloud. No cloud-side ADR needed; cloud's policy reuses the existing plan-gating mechanism (the same place that gates the AI authoring assistant today).