Skip to content

Commit 19e3e6e

Browse files
authored
feat(runtime): the standalone default datasource is a declaration, connected through the one datasource path (#3826) (#3869)
ADR-0062 D1 asked for exactly one "definition → live driver" path; the standalone `default` was the holdout, pre-built and connected by `ObjectQLEngine.init()` with its own failure verdict. - `createStandaloneStack` emits a datasource DEFINITION; the new `DefaultDatasourcePlugin` connects it through the shared `DatasourceConnectionService` in init() (phase separation beats topology — the kernel orders BOTH phases by dependency graph, not registration), with a hard dependency on ObjectQLPlugin. - `bootCritical`: third D5 fail-fast cause, sharing OS_ALLOW_DRIVER_CONNECT_FAILURE. The default bypasses the connect policy (byte-for-byte with the old boot). - `connect(record, { asDefault: true })`: natural driver name + isDefault, guarded by engine.getDefaultDriverName(). Primary DB shows real status in Setup → Datasources (#3827). - `sqlite-wasm` joined the shared factory; `default` is host-reserved (rejected in app bundles and runtime-admin create); `driver.<name>` kernel service kept for os migrate / storage detection. Verified: dev:crm --fresh table-for-table identical with unmodified main (71 = 71), authenticated data reads, artifact-serve boots with status ok. Remaining second site (CLI config-load fallback, mysql/turso + telemetry coupling) stays tracked in #3826.
1 parent 5b79a34 commit 19e3e6e

15 files changed

Lines changed: 751 additions & 116 deletions
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
"@objectstack/runtime": minor
3+
"@objectstack/service-datasource": minor
4+
"@objectstack/cli": patch
5+
---
6+
7+
feat(runtime)!: the standalone `default` datasource is a declaration, connected through the one datasource path (#3826)
8+
9+
ADR-0062 D1 asked for exactly one "definition → live driver" path. Construction
10+
converged earlier; the *connect + failure verdict* half did not — the standalone
11+
`default` driver was pre-built and smuggled into the engine as a `driver.*`
12+
kernel service, so "what if it cannot connect" lived in `ObjectQLEngine.init()`,
13+
a second implementation of the policy `DatasourceConnectionService` owns for
14+
every other datasource. #3741#3758 showed what two copies cost: a fix to one
15+
missed the other for three months.
16+
17+
- **`createStandaloneStack` now emits a datasource DEFINITION**, not a driver.
18+
URL→config translation and `mkdir` stay host concerns; the new
19+
**`DefaultDatasourcePlugin`** (exported from `@objectstack/runtime`) connects
20+
the definition at boot through the shared `DatasourceConnectionService`
21+
same driver factory, same failure verdict, same retained state. It must be
22+
registered before `ObjectQLPlugin` (boot schema-sync needs the driver);
23+
`createStandaloneStack` orders it correctly.
24+
- **`sqlite-wasm` joined the shared driver factory** (`sqlite-wasm` /
25+
`wasm-sqlite` ids) — it was the last bespoke construction site.
26+
- **`bootCritical` on `ConnectableDatasource`**: the host declares a datasource
27+
the platform cannot run without; a boot connect failure is then fatal
28+
regardless of object bindings, sharing `OS_ALLOW_DRIVER_CONNECT_FAILURE` and
29+
the `DEGRADED BOOT` banner with the engine-level guard. A connect policy that
30+
denies a boot-critical datasource fails the boot loudly — the #3828 "denial is
31+
not a failure" boundary was drawn for optional datasources.
32+
- **`connect(record, { asDefault: true })`**: registers the built driver as the
33+
engine's default under its natural name (no `'default'` stamping — routing to
34+
`default` goes through the engine's default-driver fallback, and the natural
35+
name keeps logs/lookups byte-for-byte with the previous boot).
36+
- **`default` is a host-reserved name**: an app bundle declaring a datasource
37+
named `default` is rejected at load (`AppPlugin`), and the runtime-admin
38+
create rejects it too. It would shadow the host's primary datasource and, if
39+
it passed the auto-connect gate, silently divert every unbound object.
40+
- The primary DB now shows a REAL `status` in Setup → Datasources (#3827) —
41+
`ok` when connected, `error` + reason when the operator boots degraded.
42+
- `ObjectQLEngine.init()` is unchanged and keeps its fail-fast: it re-connects
43+
the already-connected default (every open-core driver's `connect()` is
44+
idempotent), which is exactly the boot verification #3741 wants.
45+
- `DriverPlugin` remains the escape hatch for tests and pre-built/proxy drivers
46+
(e.g. the CLI's `telemetry` datasource) — no longer how the standalone
47+
default boots. The CLI serve config-load fallback (`createStorageDriver`,
48+
incl. mysql/turso) still constructs directly; tracked in #3826.
49+
50+
**Migration.** Boots through `createStandaloneStack` (CLI `serve`/`dev`
51+
artifact path, quickstarts, embedders using the stack factory) change shape but
52+
not behavior: same driver kinds, same URLs, same fail-fast semantics, same
53+
escape hatch. Embedders that composed `DriverPlugin` manually are unaffected.
54+
An app that declared a datasource literally named `default` now fails to load
55+
with a rename instruction — that name never routed correctly to begin with.

content/docs/data-modeling/drivers.mdx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,14 @@ The same guard covers **declared datasources** whose objects have no fallback
9595
see [When auto-connect fails](/docs/data-modeling/external-datasources#when-auto-connect-fails)
9696
([#3758](https://github.com/objectstack-ai/objectstack/issues/3758)).
9797

98+
The standalone `default` datasource itself is now a **declared definition**
99+
([#3826](https://github.com/objectstack-ai/objectstack/issues/3826)): the stack
100+
translates `OS_DATABASE_URL` into `{ driver, config }` and connects it at boot
101+
through the same datasource connection path — one failure verdict, one escape
102+
hatch, and a real `status` for the primary DB in **Setup → Datasources**. The
103+
name `default` is host-reserved: an app bundle declaring a datasource with that
104+
name is rejected at load.
105+
98106
<Callout type="warn">
99107
`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway, in an explicitly degraded
100108
state announced by a `DEGRADED BOOT` banner. Queries to a failed driver fail

docs/adr/0062-external-datasource-runtime.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,9 +64,9 @@ Introduce a single service that, given a datasource definition, builds a driver
6464
> | pool teardown | kernel shutdown via `DriverPlugin` | `DatasourceConnectionService.disconnect()` |
6565
> | connect policy | not consulted | `DatasourceConnectPolicy` |
6666
>
67-
> **What actually blocks the merge is an input-shape mismatch, not ordering.** The kernel's init-all-then-start-all means the connection service *does* exist by `ObjectQLPlugin.start()`, so timing is available. The obstacles are: (1) `DatasourceConnectionService.connect()` takes a datasource *definition* and **builds** the driver, while `default` arrives as an already-constructed driver instance published as a `driver.*` kernel service — there is no "adopt this driver" entry point; and (2) routing `default` through the service would make `ObjectQLPlugin`'s boot depend on an **optional service from a higher layer** (`service-datasource`), inverting the layering for the one driver every app needs. Closing this means either adding an `adoptDriver()` seam to the connection service, or making the standalone `default` a real declared datasource definition — a design decision, not a mechanical move, and still the riskiest single step per §Risk.
67+
> **Resolution (#3826, second pass) — the standalone `default` is now a declared definition.** The input-shape mismatch was resolved by making the definition the input: `createStandaloneStack` translates the database URL into a `{ driver, config }` definition (URL→config translation and `mkdir` stay host concerns) and the runtime's **`DefaultDatasourcePlugin`** — registered before `ObjectQLPlugin`, so the driver exists before boot schema-sync — connects it through `DatasourceConnectionService.connect(record, { asDefault: true })`. The definition is marked **`bootCritical`**, which adds a third fail-fast cause to D5 (the platform cannot run without it; every unbound object routes to it), sharing `OS_ALLOW_DRIVER_CONNECT_FAILURE` and the `DEGRADED BOOT` banner with the engine guard. `asDefault` keeps the driver's **natural name** (routing to `default` uses the engine's default-driver fallback, never `drivers.get('default')`) and registers with `isDefault: true`. The presumed layering inversion did not materialize: the *runtime host* orchestrates (runtime already depends on `service-datasource`); `ObjectQLPlugin` learned nothing. When the datasource-admin plugin is present its shared connection service is used (so `default` shows a real `status` in Setup → Datasources, #3827); a lite kernel instantiates the same class locally — one implementation either way. `sqlite-wasm` joined the shared factory (the last bespoke construction site), `default` became a host-reserved name (rejected in app bundles at load and in runtime-admin create), and `ObjectQLEngine.init()` keeps its #3741 fail-fast unchanged — it re-connects the already-connected default (all open-core drivers' `connect()` is idempotent), which is precisely the boot *verification* role D1 leaves it.
6868
>
69-
> Until then the divergence is guarded rather than assumed: `packages/runtime/src/degraded-boot-parity.test.ts` pins both paths to the same operator-visible contract (fail-fast by default, identical `OS_ALLOW_DRIVER_CONNECT_FAILURE` parsing, `DEGRADED BOOT` on stderr), so a change to one that forgets the other fails CI instead of shipping. #3741#3758 was exactly that miss, and it cost three months and a second bug report.
69+
> **Remaining second sites, tracked in #3826:** the CLI serve **config-load fallback** (`createStorageDriver` + `DriverPlugin`, used when a host `objectstack.config.ts` supplies no driver — it also carries mysql/turso kinds the shared factory does not build, and the `telemetry` sibling-datasource provisioning is coupled to its resolution result), and the cloud stack's own composition. Until those converge, `packages/runtime/src/degraded-boot-parity.test.ts` remains load-bearing: it pins both connect paths to the same operator-visible contract (fail-fast by default, identical `OS_ALLOW_DRIVER_CONNECT_FAILURE` parsing, `DEGRADED BOOT` on stderr), so a change to one that forgets the other fails CI instead of shipping. #3741#3758 was exactly that miss.
7070
7171
### D2 — Connect is opt-in-safe: existing managed apps are byte-for-byte unchanged
7272

packages/cli/src/commands/serve.ts

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -872,7 +872,16 @@ export default class Serve extends Command {
872872
// whole dispatch is unit-testable (storage-driver.test.ts). #3276: the
873873
// `memory` kind now maps to the mingo InMemoryDriver instead of silently
874874
// falling through to the dev SQLite `:memory:` default.
875-
const hasDriver = plugins.some((p: any) => p.name?.includes('driver') || p.constructor?.name?.includes('Driver'));
875+
// A DefaultDatasourcePlugin counts as a driver provider (#3826): the
876+
// standalone stack now DECLARES its `default` datasource and connects it
877+
// at boot through the datasource connection service, so building a
878+
// storage driver here would construct a duplicate pool the engine then
879+
// discards as already-registered.
880+
const hasDriver = plugins.some((p: any) =>
881+
p.name?.includes('driver') ||
882+
p.constructor?.name?.includes('Driver') ||
883+
p.name === 'com.objectstack.runtime.default-datasource' ||
884+
p.constructor?.name === 'DefaultDatasourcePlugin');
876885
if (!hasDriver && config.objects) {
877886
const databaseUrl = process.env.OS_DATABASE_URL;
878887
const driverType = resolveDriverType(process.env.OS_DATABASE_DRIVER, databaseUrl);

packages/objectql/src/engine.ts

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1519,6 +1519,16 @@ export class ObjectQL implements IDataEngine {
15191519
this.unavailableDatasources.delete(name);
15201520
}
15211521

1522+
/**
1523+
* Name of the DEFAULT driver, when one is registered (#3826). The default
1524+
* driver keeps its natural name (`registerDriver(driver, true)` — nothing
1525+
* routes by `drivers.get('default')`), so the datasource connection layer's
1526+
* `asDefault` idempotency guard needs this rather than a name lookup.
1527+
*/
1528+
getDefaultDriverName(): string | undefined {
1529+
return this.defaultDriver ?? undefined;
1530+
}
1531+
15221532
/**
15231533
* Datasources that were declared but are NOT usable, with the reason class.
15241534
*

packages/runtime/src/app-plugin.ts

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -372,6 +372,31 @@ export class AppPlugin implements Plugin {
372372
// ONLY — never persisted to the runtime DB store — and stamped
373373
// `origin:'code'` so the admin service enforces them as read-only.
374374
// The engine already indexed them for the write gate via registerApp().
375+
//
376+
// `default` is a HOST-owned reserved name (#3826): the runtime declares
377+
// and connects it (DefaultDatasourcePlugin). An app declaring it would
378+
// shadow the host's metadata row and — if it passed the D2 gate —
379+
// divert every unbound object to a fresh connection. Contract-first:
380+
// reject at load, loudly (outside the lenient catch below), instead of
381+
// letting the collision produce undefined routing.
382+
{
383+
const dsDefs = this.bundle.datasources;
384+
const declared = Array.isArray(dsDefs)
385+
? dsDefs
386+
: dsDefs && typeof dsDefs === 'object'
387+
? Object.values(dsDefs as Record<string, unknown>)
388+
: [];
389+
const names = Array.isArray(dsDefs)
390+
? declared.map((d: any) => d?.name)
391+
: Object.keys((dsDefs as Record<string, unknown>) ?? {});
392+
if (declared.some((d: any) => d?.name === 'default') || names.includes('default')) {
393+
throw new Error(
394+
`[AppPlugin] app '${appId}' declares a datasource named 'default' — that name is ` +
395+
`reserved for the host's primary datasource. Rename it (e.g. '${appId.split('.').pop()}_primary') ` +
396+
`and route objects to it explicitly, or omit it to use the host default.`,
397+
);
398+
}
399+
}
375400
try {
376401
const dsDefs = this.bundle.datasources;
377402
const dsList = Array.isArray(dsDefs)
Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// ADR-0062 D1 (#3826): the standalone `default` datasource is a DECLARATION,
4+
// connected at boot by DefaultDatasourcePlugin through the same
5+
// DatasourceConnectionService as every declared/runtime datasource — one
6+
// connect path, one failure verdict, one escape hatch. These boots exercise
7+
// the real kernel (init-all → start-all) with the real driver factory.
8+
9+
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
10+
import { Runtime } from './runtime.js';
11+
import { DefaultDatasourcePlugin } from './default-datasource-plugin.js';
12+
import { AppPlugin } from './app-plugin.js';
13+
14+
const BOOT_TIMEOUT = 60_000;
15+
const ENV = 'OS_ALLOW_DRIVER_CONNECT_FAILURE';
16+
17+
async function assemble(opts: {
18+
driver?: string;
19+
withAdminPlugin?: boolean;
20+
connectPolicy?: any;
21+
bundle?: any;
22+
} = {}) {
23+
const { ObjectQLPlugin } = await import('@objectstack/objectql');
24+
const runtime = new Runtime({ cluster: false });
25+
const kernel = runtime.getKernel();
26+
// Order matters for START: the default datasource must connect before
27+
// ObjectQLPlugin.start() runs boot schema-sync.
28+
await kernel.use(new DefaultDatasourcePlugin({ driver: opts.driver ?? 'memory' }));
29+
await kernel.use(new ObjectQLPlugin());
30+
if (opts.bundle) await kernel.use(new AppPlugin(opts.bundle));
31+
if (opts.withAdminPlugin !== false) {
32+
const { DatasourceAdminServicePlugin, createDefaultDatasourceDriverFactory } = await import(
33+
'@objectstack/service-datasource'
34+
);
35+
await kernel.use(
36+
new DatasourceAdminServicePlugin({
37+
driverFactory: createDefaultDatasourceDriverFactory(),
38+
connectPolicy: opts.connectPolicy,
39+
}),
40+
);
41+
}
42+
return kernel;
43+
}
44+
45+
describe('DefaultDatasourcePlugin — the default datasource as a declaration (#3826)', () => {
46+
let saved: string | undefined;
47+
beforeEach(() => { saved = process.env[ENV]; delete process.env[ENV]; });
48+
afterEach(() => {
49+
if (saved === undefined) delete process.env[ENV];
50+
else process.env[ENV] = saved;
51+
});
52+
53+
it('boots, registers the driver as DEFAULT, and serves reads/writes end to end', async () => {
54+
const kernel = await assemble({
55+
bundle: {
56+
manifest: { id: 'com.test.default-ds', name: 'Default DS', version: '1.0.0' },
57+
objects: [{ name: 'note', label: 'Note', fields: { title: { type: 'text' } } }],
58+
},
59+
});
60+
try {
61+
await kernel.bootstrap();
62+
const engine = kernel.getService<any>('data');
63+
// The driver keeps its NATURAL name (no 'default' stamping) — routing to
64+
// `default` goes through the engine's default-driver fallback.
65+
expect(engine.getDriverByName('default')).toBeUndefined();
66+
await engine.insert('note', { title: 'through-the-default' });
67+
const rows = await engine.find('note');
68+
expect(rows.map((r: any) => r.title)).toContain('through-the-default');
69+
} finally {
70+
try { await (kernel as any)?.stop?.(); } catch { /* noop */ }
71+
}
72+
}, BOOT_TIMEOUT);
73+
74+
it('shows the primary DB in the datasource-admin list with a REAL status (#3827)', async () => {
75+
const kernel = await assemble({});
76+
try {
77+
await kernel.bootstrap();
78+
const admin = kernel.getService<{ listDatasources(): Promise<any[]> }>('datasource-admin');
79+
const def = (await admin.listDatasources()).find((d) => d.name === 'default');
80+
expect(def).toBeDefined();
81+
expect(def!.status).toBe('ok');
82+
} finally {
83+
try { await (kernel as any)?.stop?.(); } catch { /* noop */ }
84+
}
85+
}, BOOT_TIMEOUT);
86+
87+
it('works without the datasource-admin plugin — same class, locally instantiated', async () => {
88+
const kernel = await assemble({ withAdminPlugin: false });
89+
try {
90+
await kernel.bootstrap();
91+
const engine = kernel.getService<any>('data');
92+
await engine.insert('sys_metadata', undefined as never).catch(() => { /* shape probe only */ });
93+
// The default driver exists and the engine can answer a trivial query path.
94+
expect(typeof engine.find).toBe('function');
95+
} finally {
96+
try { await (kernel as any)?.stop?.(); } catch { /* noop */ }
97+
}
98+
}, BOOT_TIMEOUT);
99+
100+
it('refuses the boot when the default cannot be built/connected (bootCritical ⇒ fail-fast)', async () => {
101+
const kernel = await assemble({ driver: 'not-a-real-driver' });
102+
const err = await kernel.bootstrap().then(
103+
() => { throw new Error('bootstrap() resolved but should have thrown'); },
104+
(e: unknown) => e as Error,
105+
);
106+
expect(err.message).toMatch(/default/);
107+
expect(err.message).toMatch(/boot-critical/);
108+
expect(err.message).toContain('OS_ALLOW_DRIVER_CONNECT_FAILURE');
109+
try { await (kernel as any)?.stop?.(); } catch { /* noop */ }
110+
}, BOOT_TIMEOUT);
111+
112+
it('boots degraded under OS_ALLOW_DRIVER_CONNECT_FAILURE — same escape hatch as the engine guard', async () => {
113+
process.env[ENV] = '1';
114+
const kernel = await assemble({ driver: 'not-a-real-driver' });
115+
try {
116+
await expect(kernel.bootstrap()).resolves.not.toThrow();
117+
} finally {
118+
try { await (kernel as any)?.stop?.(); } catch { /* noop */ }
119+
}
120+
}, BOOT_TIMEOUT);
121+
122+
it('is NOT gated by the host connect policy — a deny-all policy cannot block the primary DB', async () => {
123+
// Byte-for-byte with the pre-#3826 boot: the default never consulted a
124+
// DatasourceConnectPolicy (that gate exists for optional/external
125+
// datasources). A multi-tenant host's deny-all must not brick every boot.
126+
const kernel = await assemble({
127+
connectPolicy: { canConnect: () => ({ allow: false, reason: 'egress blocked' }) },
128+
});
129+
try {
130+
await expect(kernel.bootstrap()).resolves.not.toThrow();
131+
const engine = kernel.getService<any>('data');
132+
expect(engine.getDefaultDriverName()).toBeDefined();
133+
} finally {
134+
try { await (kernel as any)?.stop?.(); } catch { /* noop */ }
135+
}
136+
}, BOOT_TIMEOUT);
137+
138+
it("rejects an app bundle that declares a datasource named 'default' (host-reserved name)", async () => {
139+
const kernel = await assemble({
140+
bundle: {
141+
manifest: { id: 'com.test.reserved', name: 'Reserved', version: '1.0.0' },
142+
objects: [{ name: 'note', label: 'Note', fields: { title: { type: 'text' } } }],
143+
datasources: [{ name: 'default', driver: 'memory', config: {} }],
144+
},
145+
});
146+
const err = await kernel.bootstrap().then(
147+
() => { throw new Error('bootstrap() resolved but should have thrown'); },
148+
(e: unknown) => e as Error,
149+
);
150+
expect(err.message).toMatch(/reserved for the host's primary datasource/);
151+
try { await (kernel as any)?.stop?.(); } catch { /* noop */ }
152+
}, BOOT_TIMEOUT);
153+
});

0 commit comments

Comments
 (0)