Skip to content

Commit b6b168c

Browse files
xuyushun441-sysos-zhuang
xuyushun441-sys
and
os-zhuang
authored
docs(datasource): add external-datasource federation guide + objectstack-data skill rule (ADR-0062) (#2211)
The ADR-0062 federation runtime ("declare an external datasource → visible, auto-connected, queryable with zero app code") had no narrative docs. Adds: - content/docs/guides/external-datasources.mdx — how-to: defineDatasource + schemaMode, object remoteName/remoteSchema/columnMap binding, auto-connect (D2 gate: external / explicit object.datasource / autoConnect; mapping-only managed stays metadata-only), onEnable only as a dynamic-driver escape hatch, credentialsRef fail-closed (D3), no field.columnName on external → use columnMap (D7), writes double opt-in, analytics over external. Listed in the Integration nav after Database Drivers. Browser-verified rendering. - skills/objectstack-data/rules/datasources.md (+ SKILL.md link) — same rules in the authoring skill, emphasizing the columnMap-not-columnName guardrail. The autoConnect field is already auto-documented in the generated references/data/datasource.mdx (from the spec describe()); not hand-edited. Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com>
1 parent bc83378 commit b6b168c

4 files changed

Lines changed: 251 additions & 0 deletions

File tree

‎content/docs/guides/external-datasources.mdx‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
---
2+
title: External Datasources (Federation)
3+
description: Declare an external database as a datasource and query its tables as ObjectStack objects — visible, auto-connected, and queryable with zero app code.
4+
---
5+
6+
# External Datasources (Federation)
7+
8+
ObjectStack can treat a **mature external database** — one it does not own — as a
9+
read-only (or, with explicit opt-in, writable) datasource, and expose its tables
10+
as normal objects. This is *federation*: the data stays in the remote database;
11+
ObjectStack queries it live through the same ObjectQL / REST surface as native
12+
objects.
13+
14+
The headline guarantee: **declare an external datasource and it is visible,
15+
auto-connected, validated at boot, and queryable — with no application code.**
16+
17+
<Callout type="info">
18+
This guide is about reading/writing the *live* remote tables (federation). To
19+
*copy* rows into ObjectStack-owned tables instead, see seed data / data sync. For
20+
the plain multi-datasource routing of **managed** databases ObjectStack owns, see
21+
[Database Drivers](/docs/guides/driver-configuration).
22+
</Callout>
23+
24+
## 1. Declare the datasource
25+
26+
Use `defineDatasource` with `schemaMode: 'external'`. The `external` block carries
27+
the federation policy (write gate, boot validation, credentials).
28+
29+
```typescript
30+
import { defineDatasource } from '@objectstack/spec/data';
31+
32+
export const Warehouse = defineDatasource({
33+
name: 'warehouse',
34+
label: 'Analytics Warehouse (Postgres)',
35+
driver: 'postgres',
36+
schemaMode: 'external', // ObjectStack never runs DDL here
37+
config: { host: 'db.internal', port: 5432, database: 'analytics', user: 'readonly' },
38+
external: {
39+
allowWrites: false, // read-only (the default)
40+
credentialsRef: 'secret:warehouse/password', // resolved from the secret store
41+
validation: { onMismatch: 'fail', checkOnBoot: true },
42+
},
43+
active: true,
44+
});
45+
```
46+
47+
Register it on the stack (array or name-keyed map both work):
48+
49+
```typescript
50+
export default defineStack({
51+
datasources: [Warehouse],
52+
// ...
53+
});
54+
```
55+
56+
`schemaMode`:
57+
58+
| Mode | Meaning |
59+
| --- | --- |
60+
| `managed` (default) | ObjectStack owns the schema — DDL + migrations allowed. |
61+
| `external` | A mature external DB — **DDL forbidden**; a schema mismatch **fails** boot. |
62+
| `validate-only` | Like `external`, but a mismatch **warns** instead of failing. |
63+
64+
## 2. Bind objects to the remote tables
65+
66+
A federated object sets `datasource` to the external datasource and declares its
67+
remote binding in `external`. When the remote table or column names differ from
68+
your object/field names, map them with `external.remoteName` / `external.remoteSchema`
69+
and `external.columnMap`.
70+
71+
```typescript
72+
export const Customer = ObjectSchema.create({
73+
name: 'ext_customer',
74+
datasource: 'warehouse',
75+
external: {
76+
remoteName: 'customers', // remote TABLE name (object name may differ)
77+
// remoteSchema: 'public', // optional schema/namespace (pg/mysql)
78+
// columnMap: { cust_region: 'region' }, // remoteColumn → localField
79+
},
80+
fields: {
81+
id: { type: 'text' },
82+
name: { type: 'text' },
83+
region: { type: 'text' },
84+
},
85+
});
86+
```
87+
88+
That's it. `GET /api/v1/data/ext_customer` now returns live rows from the remote
89+
`customers` table; filters (`?region=EU`) push down to the remote query.
90+
91+
<Callout type="warn">
92+
**Do not set `field.columnName` on an external object.** For federated objects the
93+
driver's query pipeline ignores `field.columnName`; `external.columnMap`
94+
(`remoteColumn → localField`) is the single, authoritative column mapping.
95+
`os build` / `os validate` rejects `field.columnName` on an external object with a
96+
clear error (ADR-0062 D7). Managed objects are unaffected.
97+
</Callout>
98+
99+
## 3. Auto-connect — no `onEnable` needed
100+
101+
At boot the runtime builds a live driver for the datasource, connects it, and
102+
registers its federated objects' read metadata — automatically. You do **not**
103+
need an `onEnable` hook or `ctx.drivers.register(...)`.
104+
105+
A declared datasource auto-connects when it is **meaningfully addressed**:
106+
107+
1. it is **external** (`schemaMode !== 'managed'`), **or**
108+
2. an object **explicitly** binds to it via `object.datasource === <name>`, **or**
109+
3. it sets **`autoConnect: true`**.
110+
111+
A `managed` datasource that nothing explicitly binds to (for example one that is
112+
only referenced by a `datasourceMapping` rule) stays *metadata-only* — visible in
113+
Setup, but not connected — so existing apps are unchanged. Use `autoConnect: true`
114+
to opt such a datasource into a live connection at boot.
115+
116+
<Callout type="info">
117+
**Escape hatch.** An `onEnable` hook calling `ctx.drivers.register(driver)` is
118+
still supported for advanced cases — e.g. a driver built dynamically at runtime
119+
from external configuration. Auto-connect is idempotent with it (whichever
120+
registers the datasource name first wins), so the two never conflict.
121+
</Callout>
122+
123+
A connected external datasource is also visible in **Setup → Datasources** (stamped
124+
`origin: code`, read-only in the UI) and via `GET /api/v1/datasources` and
125+
`GET /api/v1/meta/datasource`, where an admin can run the "Sync objects" wizard.
126+
127+
## 4. Credentials
128+
129+
Never inline a password. Put a reference in `external.credentialsRef` and store the
130+
secret in the secret store (the same `SecretBinder` / `ICryptoProvider` the
131+
runtime-admin "Add Datasource" wizard uses). The credential is resolved to
132+
cleartext **at connect, before the driver is built**.
133+
134+
Resolution is **fail-closed**: if a `credentialsRef` is declared but no secret store
135+
is configured, or the secret cannot be resolved/decrypted, the datasource is left
136+
**unconnected with a clear error** — never connected without the credential. For a
137+
code-defined external datasource with `validation.onMismatch: 'fail'` this fails
138+
the boot; otherwise it degrades with a warning. (Credential-less drivers such as
139+
SQLite simply have no `credentialsRef`.)
140+
141+
## 5. Writes (double opt-in)
142+
143+
Federation is read-only by default. To allow writes, **both** the datasource and the
144+
object must opt in:
145+
146+
```typescript
147+
defineDatasource({ /* ... */ external: { allowWrites: true } }); // datasource gate
148+
ObjectSchema.create({ /* ... */ external: { remoteName: 'orders', writable: true } }); // object gate
149+
```
150+
151+
With either gate off, insert/update/delete on the federated object is rejected.
152+
153+
## 6. Analytics over external objects
154+
155+
Dashboards and reports over a federated object aggregate against the **correct**
156+
remote table/columns: the analytics layer routes such queries through the driver's
157+
physical-table resolution (honoring `remoteName` / `remoteSchema`) rather than the
158+
object name, so they never hit the wrong table.
159+
160+
## Multi-tenant hosts
161+
162+
A self-hosted single-environment runtime connects external datasources out of the
163+
box. A multi-tenant host can bind a stricter **connect policy** (egress allow-list /
164+
per-tenant quota) that is consulted before any connection is opened — the same
165+
single connect path, no fork. This is a host-composition concern; the open-core
166+
default allows all connects (subject to the gating above).
167+
168+
## See also
169+
170+
- [Database Drivers](/docs/guides/driver-configuration) — managed multi-datasource routing.
171+
- [Datasource reference](/docs/references/data/datasource) — every `defineDatasource` field.
172+
- The `examples/app-showcase` `showcase_external` datasource — a runnable end-to-end demo.
173+
- ADR-0015 (federation spec) and ADR-0062 (external-datasource runtime).

‎content/docs/guides/meta.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
"contracts",
3434
"project-scoping",
3535
"driver-configuration",
36+
"external-datasources",
3637
"kernel-services",
3738
"runtime-services",
3839
"data-flow",

‎skills/objectstack-data/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -198,6 +198,7 @@ For comprehensive documentation with incorrect/correct examples:
198198
- **[Validation Rules](./rules/validation.md)** — All validation types, script inversion, severity levels
199199
- **[Index Strategy](./rules/indexing.md)** — btree/gin/gist/fulltext, composite indexes, partial indexes
200200
- **[Lifecycle Hooks](./rules/hooks.md)** — Hook quick reference (→ see [references/data-hooks.md](./references/data-hooks.md) for the full 14-event guide)
201+
- **[Datasources & Federation](./rules/datasources.md)** — `defineDatasource`, external/federated objects (`remoteName`/`columnMap`), auto-connect gating, credentials; ❌ no `field.columnName` on external objects
201202

202203
---
203204

‎skills/objectstack-data/rules/datasources.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Datasources & Federation
2+
3+
A **datasource** (`defineDatasource`, a `*.datasource.ts` metadata file) is a
4+
connection to a data store. Objects route to one via their `datasource` field
5+
(default: `'default'`). Most apps need only the `default` datasource; declare more
6+
to read/write a **separate** or **external** database.
7+
8+
Full field reference: `node_modules/@objectstack/spec/src/data/datasource.zod.ts`.
9+
Narrative guide: `content/docs/guides/external-datasources.mdx`.
10+
11+
## `schemaMode` — who owns the schema
12+
13+
| Mode | Meaning |
14+
|:--|:--|
15+
| `managed` (default) | ObjectStack owns the schema; DDL + migrations allowed. |
16+
| `external` | A mature external DB ObjectStack does **not** own; DDL forbidden; boot mismatch **fails**. |
17+
| `validate-only` | Like `external`, but a mismatch **warns** instead of failing boot. |
18+
19+
`external` settings are required iff `schemaMode !== 'managed'` (and forbidden otherwise).
20+
21+
## Federated (external) objects
22+
23+
An object on an external datasource binds to its remote table via `external`:
24+
25+
```typescript
26+
ObjectSchema.create({
27+
name: 'ext_customer',
28+
datasource: 'warehouse',
29+
external: {
30+
remoteName: 'customers', // remote TABLE (object name may differ)
31+
// remoteSchema: 'public', // optional schema/namespace (pg/mysql)
32+
// columnMap: { cust_region: 'region' }, // remoteColumn → localField
33+
// writable: true, // per-object write opt-in (see below)
34+
},
35+
fields: { id: { type: 'text' }, name: { type: 'text' }, region: { type: 'text' } },
36+
});
37+
```
38+
39+
### ✅ / ❌ Column mapping (ADR-0062 D7)
40+
41+
- ✅ Map remote columns with **`external.columnMap`** (`remoteColumn → localField`).
42+
- ❌ **Never set `field.columnName` on an external object.** The driver's query
43+
pipeline ignores it for federated objects, so it is a silent dual-source trap.
44+
`os build` / `os validate` **rejects** it with a clear error. (`field.columnName`
45+
on **managed** objects is unaffected.)
46+
47+
## Auto-connect (no `onEnable`)
48+
49+
A declared datasource is built into a live driver, connected, and its federated
50+
objects' read metadata registered **automatically at boot** — no `onEnable` /
51+
`ctx.drivers.register`. It auto-connects when **meaningfully addressed**:
52+
53+
1. it is **external** (`schemaMode !== 'managed'`), **or**
54+
2. an object **explicitly** binds via `object.datasource === <name>`, **or**
55+
3. it sets **`autoConnect: true`**.
56+
57+
A `managed` datasource that nothing explicitly binds (e.g. only referenced by a
58+
`datasourceMapping` rule) stays **metadata-only** — visible but not connected — so
59+
existing apps are unchanged. Set `autoConnect: true` to force a live connection.
60+
61+
> `onEnable` + `ctx.drivers.register(driver)` remains supported only as an escape
62+
> hatch for drivers built dynamically at runtime; it is idempotent with auto-connect.
63+
64+
## Credentials — fail-closed
65+
66+
Never inline a password. Use `external.credentialsRef` and store the secret in the
67+
secret store; it is resolved **at connect, before the driver is built**. A declared
68+
`credentialsRef` that cannot be resolved/decrypted (or no secret store configured)
69+
leaves the datasource **unconnected with a clear error** — never connected without
70+
the credential.
71+
72+
## Writes — double opt-in
73+
74+
Federation is read-only by default. To write, **both** gates must be on:
75+
`datasource.external.allowWrites: true` **and** the object's `external.writable: true`.
76+
With either off, insert/update/delete on the federated object is rejected.

0 commit comments

Comments
 (0)