Skip to content

external-datasources.mdx

Latest commit

 

History

History
260 lines (208 loc) · 11.4 KB

external-datasources.mdx

File metadata and controls

260 lines (208 loc) · 11.4 KB
Raw
title External Datasources (Federation)
description Declare an external database as a datasource and query its tables as ObjectStack objects — visible, auto-connected, and queryable with zero app code.

External Datasources (Federation)

ObjectStack can treat a mature external database — one it does not own — as a read-only (or, with explicit opt-in, writable) datasource, and expose its tables as normal objects. This is federation: the data stays in the remote database; ObjectStack queries it live through the same ObjectQL / REST surface as native objects.

The headline guarantee: declare an external datasource and it is visible, auto-connected, validated at boot, and queryable — with no application code.

This guide is about reading/writing the *live* remote tables (federation). To *copy* rows into ObjectStack-owned tables instead, see seed data / data sync. For the plain multi-datasource routing of **managed** databases ObjectStack owns, see [Database Drivers](/docs/data-modeling/drivers).

1. Declare the datasource

Use defineDatasource with schemaMode: 'external'. The external block carries the federation policy (write gate, boot validation, credentials).

{/* os:check */}

import { defineDatasource } from '@objectstack/spec/data';

export const Warehouse = defineDatasource({
  name: 'warehouse',
  label: 'Analytics Warehouse (Postgres)',
  driver: 'postgres',
  schemaMode: 'external',          // ObjectStack never runs DDL here
  config: { host: 'db.internal', port: 5432, database: 'analytics', user: 'readonly' },
  external: {
    allowWrites: false,            // read-only (the default)
    credentialsRef: 'secret:warehouse/password', // resolved from the secret store
    validation: { onMismatch: 'fail', checkOnBoot: true },
  },
  active: true,
});

Register it on the stack (array or name-keyed map both work):

export default defineStack({
  datasources: [Warehouse],
  // ...
});

schemaMode:

Mode Meaning
managed (default) ObjectStack owns the schema — DDL + migrations allowed.
external A mature external DB — DDL forbidden; a schema mismatch fails boot.
validate-only Like external, but a mismatch warns instead of failing.

2. Bind objects to the remote tables

A federated object sets datasource to the external datasource and declares its remote binding in external. When the remote table or column names differ from your object/field names, map them with external.remoteName / external.remoteSchema and external.columnMap.

export const Customer = ObjectSchema.create({
  name: 'ext_customer',
  datasource: 'warehouse',
  external: {
    remoteName: 'customers',       // remote TABLE name (object name may differ)
    // remoteSchema: 'public',     // optional schema/namespace (pg/mysql)
    // columnMap: { cust_region: 'region' }, // remoteColumn → localField
  },
  fields: {
    id: { type: 'text' },
    name: { type: 'text' },
    region: { type: 'text' },
  },
});

That's it. GET /api/v1/data/ext_customer now returns live rows from the remote customers table; filters (?region=EU) push down to the remote query.

**Do not set `field.columnName` on an external object.** For federated objects the driver's query pipeline ignores `field.columnName`; `external.columnMap` (`remoteColumn → localField`) is the single, authoritative column mapping. `os build` / `os validate` rejects `field.columnName` on an external object with a clear error (ADR-0062 D7). Managed objects are unaffected.

3. Auto-connect — no onEnable needed

At boot the runtime builds a live driver for the datasource, connects it, and registers its federated objects' read metadata — automatically. You do not need an onEnable hook or ctx.drivers.register(...).

A declared datasource auto-connects when it is meaningfully addressed:

  1. it is external (schemaMode !== 'managed'), or
  2. an object explicitly binds to it via object.datasource === <name>, or
  3. it sets autoConnect: true.

A managed datasource that nothing explicitly binds to (for example one that is only referenced by a datasourceMapping rule) stays metadata-only — visible in Setup, but not connected — so existing apps are unchanged. Use autoConnect: true to opt such a datasource into a live connection at boot.

**Escape hatch.** An `onEnable` hook calling `ctx.drivers.register(driver)` is still supported for advanced cases — e.g. a driver built dynamically at runtime from external configuration. Auto-connect is idempotent with it (whichever registers the datasource name first wins), so the two never conflict.

A connected external datasource is also visible in Setup → Datasources (stamped origin: code, read-only in the UI) and via GET /api/v1/datasources and GET /api/v1/meta/datasource, where an admin can run the "Sync objects" wizard.

When auto-connect fails

An object that binds explicitly via datasource: '…' has no fallback — it never falls through to the default driver, so an unconnected datasource means every read and write of that object fails. The boot therefore refuses to start when a datasource in that position cannot be connected (#3758), rather than leaving a server that looks healthy and errors only on the affected pages:

Gate Connect fails at boot
external with validation.onMismatch: 'fail' refuses the boot
objects bind explicitly via object.datasource refuses the boot — the error names them
autoConnect: true with nothing bound warning; left unconnected

This covers every reason a connect can fail — unreachable database, unresolvable credentialsRef, unsupported driver — because the bound objects are equally dead in each case. Every gated datasource is attempted before the boot aborts, so one failed start reports all the misconfigured ones.

`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway — the same flag as the [engine-level driver-connect guard](/docs/data-modeling/drivers#startup-a-driver-that-cannot-connect-aborts-the-boot) — in an explicitly degraded state announced by a `DEGRADED BOOT` banner. The datasource stays unconnected for the process lifetime: nothing re-runs the connect, so queries against its objects keep failing until the server is restarted. Do not set it in production.

Seeing the state of a datasource

Every connect attempt's verdict is retained, so a datasource that is down no longer has to be diagnosed by restarting the server and re-reading boot logs (#3827).

Setup → Datasources and GET /api/v1/datasources report a status per datasource:

status Meaning
ok A live driver is registered and routable.
error A connect was attempted and failed. statusReason carries the cause.
blocked The host's connect policy refused it — a decision, not a fault. It will not clear on its own.
unvalidated No connect attempted: a managed datasource left metadata-only by the gate above, or a runtime row nobody has tested.

statusReason is operator-facing and may name hosts, ports, or internal plans — this surface is already admin-gated.

Note that checkDriversHealth() (and therefore /ready) cannot see these: a datasource that never connected was never registered as a driver, so it is absent from the health probe rather than reported unhealthy. Readiness is deliberately not gated on them — an optional datasource being down must not pull an otherwise-working replica out of the load balancer.

What a query against an unusable datasource says

An object bound to a datasource with no live driver fails with ERR_DATASOURCE_UNAVAILABLE (HTTP 503), and the message says which situation it is (#3828): refused by policy, or a connect that failed under OS_ALLOW_DRIVER_CONNECT_FAILURE. A datasource name that was never declared still gets the original is not registered — that one really is an authoring bug, and there is nothing to add.

The error never carries the underlying cause: connect errors routinely contain hosts, ports and DSNs, and a policy's reason is written for operators. Both stay in the logs and the admin list. A host that wants to tell tenants something specific sets publicReason on its connect decision — opt-in, and the only string that reaches an end user:

const policy: DatasourceConnectPolicy = {
  canConnect: (ds) =>
    allowedFor(tenant, ds)
      ? { allow: true }
      : {
          allow: false,
          // operator-facing: logs + Setup → Datasources only
          reason: `egress allow-list miss for ${ds.name} (${tenant.id}, plan=${tenant.plan})`,
          // tenant-facing: appended to the query-time error
          publicReason: 'External datasources require the Scale plan.',
        },
};

4. Credentials

Never inline a password. Put a reference in external.credentialsRef and store the secret in the secret store (the same SecretBinder / ICryptoProvider the runtime-admin "Add Datasource" wizard uses). The credential is resolved to cleartext at connect, before the driver is built.

Resolution is fail-closed: if a credentialsRef is declared but no secret store is configured, or the secret cannot be resolved/decrypted, the datasource is left unconnected with a clear error — never connected without the credential. An unresolvable credential is a connect failure like any other, so it fails the boot for the datasources listed under When auto-connect fails and degrades with a warning otherwise. (Credential-less drivers such as SQLite simply have no credentialsRef.)

5. Writes (double opt-in)

Federation is read-only by default. To allow writes, both the datasource and the object must opt in:

defineDatasource({ /* ... */ external: { allowWrites: true } });   // datasource gate
ObjectSchema.create({ /* ... */ external: { remoteName: 'orders', writable: true } }); // object gate

With either gate off, insert/update/delete on the federated object is rejected.

6. Analytics over external objects

Dashboards and reports over a federated object aggregate against the correct remote table/columns: the analytics layer routes such queries through the driver's physical-table resolution (honoring remoteName / remoteSchema) rather than the object name, so they never hit the wrong table.

Multi-tenant hosts

A self-hosted single-environment runtime connects external datasources out of the box. A multi-tenant host can bind a stricter connect policy (egress allow-list / per-tenant quota) that is consulted before any connection is opened — the same single connect path, no fork. This is a host-composition concern; the open-core default allows all connects (subject to the gating above).

See also

  • Database Drivers — managed multi-datasource routing.
  • Datasource reference — every defineDatasource field.
  • The examples/app-showcase showcase_external datasource — a runnable end-to-end demo.
  • ADR-0015 (federation spec) and ADR-0062 (external-datasource runtime).