Skip to content

Commit 87aca93

Browse files
authored
fix(datasource): an explicitly-bound datasource that cannot connect refuses the boot (#3758) (#3816)
`DatasourceConnectionService.handleFailure()` fail-fasted only for an `external` datasource with `validation.onMismatch: 'fail'`; everything else degraded to one `warn` line — including the case the D2 auto-connect gate itself identifies as having no fallback path: a datasource that objects bind to explicitly via `object.datasource`. Those objects never fall through to the `default` driver; `engine.getDriver` throws `Datasource 'x' is not registered` for them. - Fail-fast is keyed on "no fallback path", not on `onMismatch` alone. - Every failure mode counts (unresolvable credentialsRef, unsupported driver); a `DatasourceConnectPolicy` denial is NOT a failure and stays metadata-only. - The error names the bound objects; `connectDeclared()` aggregates. - Escape hatch shared with the engine guard: `OS_ALLOW_DRIVER_CONNECT_FAILURE=1`. ADR-0062 D5 amended; deployment + federation docs updated to match.
1 parent 57a3bb3 commit 87aca93

16 files changed

Lines changed: 587 additions & 74 deletions

File tree

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
"@objectstack/service-datasource": minor
3+
"@objectstack/types": patch
4+
"@objectstack/objectql": patch
5+
---
6+
7+
fix(datasource)!: a declared datasource that objects bind to must connect, or the boot fails (#3758)
8+
9+
`DatasourceConnectionService.handleFailure()` fail-fasted only for an `external`
10+
datasource with `validation.onMismatch: 'fail'`. Everything else degraded to one
11+
`warn` line — including the case the D2 auto-connect gate itself flags as having
12+
**no fallback path**: a datasource that objects bind to explicitly via
13+
`object.datasource`. Those objects never fall through to the `default` driver;
14+
`engine.getDriver` throws `Datasource 'x' is not registered` for them.
15+
16+
So an app declaring `datasource: 'analytics'` with 20 objects bound to it, booted
17+
against a wrong `ANALYTICS_URL`, started clean and exited zero — and then failed
18+
every read and write of those 20 objects with an error that reads nothing like
19+
*the analytics database is unreachable*. The rest of the app worked, which made it
20+
**harder** to locate than a total outage: it looks like "some pages are broken",
21+
not like a misconfigured datasource. This is the same decision #3741/#3751 fixed
22+
one layer up in `ObjectQLEngine.init()`; the boundary here was still drawn in the
23+
old place.
24+
25+
- **Fail-fast is now keyed on "no fallback path", not on `onMismatch` alone.** At
26+
the `declared-auto` (boot) trigger, a connect failure aborts the boot when the
27+
datasource is `external` + `onMismatch: 'fail'` **or** when ≥1 object binds to
28+
it explicitly. `autoConnect: true` with nothing bound stays lenient — that is
29+
"connect it if you can", and nothing declares a dependency on it. The
30+
runtime-admin create/update and boot-rehydration triggers are unchanged and
31+
still always degrade: a UI action must never brick a running server.
32+
- **Every failure mode counts**, not just an unreachable socket: an unresolvable
33+
`external.credentialsRef` (D3) and an unsupported `driver` leave the bound
34+
objects exactly as dead, so they take the same verdict.
35+
- **The error names the bound objects** (up to 10, then `+N more`) alongside the
36+
underlying cause, so the message points at the real problem instead of just the
37+
datasource name. The service already receives the list for post-connect
38+
`syncObjectSchema`.
39+
- **`connectDeclared()` attempts every gated datasource before throwing**, and
40+
aggregates, so one failed boot reports all the misconfigured ones rather than
41+
one per restart — the same shape as `ObjectQLEngine.init()`'s
42+
`DriverConnectError`.
43+
- **The escape hatch is shared with the engine guard**:
44+
`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` now also covers this path (and covers
45+
`onMismatch: 'fail'`, which previously had no opt-out). The operator intent is
46+
identical — "I know the database is unreachable, boot anyway" — and two flags
47+
would only guarantee one of them gets missed. When set, boot continues and a
48+
`DEGRADED BOOT` banner goes to stderr as well as the logger, because `os serve`
49+
swallows stdout during boot. `emitDegradedBootBanner` moved to
50+
`@objectstack/types` so both call sites share one implementation;
51+
`@objectstack/objectql` re-exports it unchanged.
52+
53+
ADR-0062 D5 is amended with the new criterion and the shared flag.
54+
55+
**Migration.** No change for a correctly configured deployment — a datasource that
56+
connected before still connects. A deployment that was *silently* booting with a
57+
dead, explicitly-bound datasource now fails the boot instead, naming the
58+
datasource, the cause, and the objects that depend on it; fix the datasource
59+
configuration. To keep booting without it — deliberately, knowing every request
60+
touching those objects will fail — set `OS_ALLOW_DRIVER_CONNECT_FAILURE=1`.

content/docs/data-modeling/drivers.mdx

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,10 @@ comes back is the **boot sequence that was skipped**: nothing re-runs the schema
9191
sync, so those objects can be left with no tables even after the database
9292
returns.
9393

94+
The same guard covers **declared datasources** whose objects have no fallback —
95+
see [When auto-connect fails](/docs/data-modeling/external-datasources#when-auto-connect-fails)
96+
([#3758](https://github.com/objectstack-ai/objectstack/issues/3758)).
97+
9498
<Callout type="warn">
9599
`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway, in an explicitly degraded
96100
state announced by a `DEGRADED BOOT` banner. Queries to a failed driver fail

content/docs/data-modeling/external-datasources.mdx

Lines changed: 35 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,36 @@ A connected external datasource is also visible in **Setup → Datasources** (st
124124
`origin: code`, read-only in the UI) and via `GET /api/v1/datasources` and
125125
`GET /api/v1/meta/datasource`, where an admin can run the "Sync objects" wizard.
126126

127+
### When auto-connect fails
128+
129+
An object that binds explicitly via `datasource: '…'` has **no fallback** — it
130+
never falls through to the `default` driver, so an unconnected datasource means
131+
every read and write of that object fails. The boot therefore **refuses to start**
132+
when a datasource in that position cannot be connected
133+
([#3758](https://github.com/objectstack-ai/objectstack/issues/3758)), rather than
134+
leaving a server that looks healthy and errors only on the affected pages:
135+
136+
| Gate | Connect fails at boot |
137+
|:---|:---|
138+
| **external** with `validation.onMismatch: 'fail'` | refuses the boot |
139+
| **objects bind explicitly** via `object.datasource` | refuses the boot — the error names them |
140+
| **`autoConnect: true`** with nothing bound | warning; left unconnected |
141+
142+
This covers every reason a connect can fail — unreachable database, unresolvable
143+
`credentialsRef`, unsupported `driver` — because the bound objects are equally
144+
dead in each case. Every gated datasource is attempted before the boot aborts, so
145+
one failed start reports *all* the misconfigured ones.
146+
147+
<Callout type="warn">
148+
`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway — the same flag as the
149+
[engine-level driver-connect guard](/docs/data-modeling/drivers#startup-a-driver-that-cannot-connect-aborts-the-boot)
150+
in an explicitly degraded state announced by a `DEGRADED BOOT` banner. The
151+
datasource stays unconnected for the process lifetime: nothing re-runs the
152+
connect, so queries against its objects keep failing with
153+
`Datasource 'x' is not registered` even after the database comes back. Do not set
154+
it in production.
155+
</Callout>
156+
127157
## 4. Credentials
128158

129159
Never inline a password. Put a reference in `external.credentialsRef` and store the
@@ -133,10 +163,11 @@ cleartext **at connect, before the driver is built**.
133163

134164
Resolution is **fail-closed**: if a `credentialsRef` is declared but no secret store
135165
is configured, or the secret cannot be resolved/decrypted, the datasource is left
136-
**unconnected with a clear error** — never connected without the credential. For a
137-
code-defined external datasource with `validation.onMismatch: 'fail'` this fails
138-
the boot; otherwise it degrades with a warning. (Credential-less drivers such as
139-
SQLite simply have no `credentialsRef`.)
166+
**unconnected with a clear error** — never connected without the credential. An
167+
unresolvable credential is a connect failure like any other, so it fails the boot
168+
for the datasources listed under [When auto-connect fails](#when-auto-connect-fails)
169+
and degrades with a warning otherwise. (Credential-less drivers such as SQLite
170+
simply have no `credentialsRef`.)
140171

141172
## 5. Writes (double opt-in)
142173

content/docs/deployment/environment-variables.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ read at startup unless noted otherwise. Boolean variables accept `true` / `false
4949
|:---|:---|:---|:---|
5050
| `OS_DATABASE_URL` | url || Database connection string (e.g. `file:./data.sqlite`, `postgres://…`, `mongodb://…`, `memory://`). `libsql://` (Turso) is not supported. |
5151
| `OS_DATABASE_DRIVER` | enum | inferred | Force a specific driver when the URL is ambiguous. `memory` \| `sqlite` \| `sqlite-wasm` \| `postgres` \| `mongodb`. |
52-
| `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: the drivers that failed stay dead for the process lifetime and every query and schema sync routed to them fails. |
52+
| `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. The same guard covers a **declared datasource** that objects bind to via `datasource: '…'`, or an `external` one with `validation.onMismatch: 'fail'`: those objects have no fallback datasource, so an unconnected one means they are all dead. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: whatever failed stays dead for the process lifetime and every query and schema sync routed to it fails. |
5353
| `OS_STORAGE_ROOT` | path | `./.objectstack/data/uploads` | Root directory for the local file storage adapter, relative to the process cwd (used by `os serve`'s default `storage` capability wiring). |
5454
| `OS_ARTIFACT_PATH` | path || Path or `http(s)://` URL to a compiled `objectstack.json` artifact to boot the kernel from. |
5555

content/docs/deployment/production-readiness.mdx

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,8 +90,12 @@ the [HARDENING.md recipes](https://github.com/objectstack-ai/objectstack/blob/ma
9090
- [ ] `OS_ALLOW_DRIVER_CONNECT_FAILURE` is **unset**. A data driver that cannot
9191
connect at startup refuses the boot, so a bad `OS_DATABASE_URL`, a rotated
9292
password, or a closed network path surfaces as a failed deploy instead of
93-
a "started" server that 500s every request. Setting the flag boots without
94-
the database and never reconnects; never set it in production.
93+
a "started" server that 500s every request. The same applies to a declared
94+
**datasource** that objects bind to explicitly (`datasource: 'analytics'`):
95+
those objects have no fallback, so a misconfigured `analytics` URL fails
96+
the deploy instead of leaving a server where most pages work and that one
97+
subset errors. Setting the flag boots without the database and never
98+
reconnects; never set it in production.
9599
- [ ] Backup / restore drill documented and tested.
96100
- [ ] Data-retention windows reviewed (ADR-0057): the platform's default
97101
`lifecycle` declarations bound telemetry (activity 14d, job runs 30d,

docs/adr/0062-external-datasource-runtime.md

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,10 +69,26 @@ Code-defined datasources surface in `GET /api/v1/datasources`, `GET /api/v1/meta
6969

7070
### D5 — Lifecycle, health & ordering for N datasources
7171

72-
`DatasourceConnectionService` owns connect/disconnect (graceful shutdown), pool config per datasource, and an optional health probe surfaced in the admin list (`status`). **Ordering**: all declared datasources connect **before** the `kernel:ready` external-validation gate (ADR-0015 §5.2) and before first query — i.e. during plugin init/start, not in a `kernel:ready` handler. Connect failure policy is **fail-fast for `external` with `validation.onMismatch: 'fail'`**, **degrade-with-warning** otherwise (a connectivity blip on an optional analytics replica should not brick boot).
72+
`DatasourceConnectionService` owns connect/disconnect (graceful shutdown), pool config per datasource, and an optional health probe surfaced in the admin list (`status`). **Ordering**: all declared datasources connect **before** the `kernel:ready` external-validation gate (ADR-0015 §5.2) and before first query — i.e. during plugin init/start, not in a `kernel:ready` handler. Connect failure policy is **fail-fast when the datasource has no fallback path**, **degrade-with-warning** otherwise (a connectivity blip on an optional analytics replica should not brick boot).
7373

7474
> **Phase 1 implementation notes (#2163).** *Ordering* is satisfied by the kernel's two-phase boot (init-all → start-all): the connection service is registered as the `'datasource-connection'` kernel service during the datasource-admin plugin's `init()`, and declared datasources are auto-connected from `AppPlugin.start()` — which runs before the `kernel:ready` validation gate. Because boot schema-sync runs before the external driver exists, `connect()` calls `engine.syncObjectSchema()` for each bound federated object (DDL-free), so they are queryable with zero app code. *Fail-fast* is scoped to the **declared-auto** trigger; the runtime-admin create/update + boot-rehydration triggers always degrade-with-warning, preserving the pre-ADR-0062 admin behavior (a UI action never bricks the running server). *Connect policy* (the epic #2163 seam): a host-injectable `DatasourceConnectPolicy` is consulted before every connect; the open-core default allows (subject to the D2 gate), and a multi-tenant host binds a stricter, fail-closed policy for egress isolation — one connect path, no cloud fork.
7575

76+
> **Amendment (#3758) — "no fallback path" is the fail-fast criterion, not `onMismatch:'fail'` alone.** The original wording scoped fail-fast to `external` + `validation.onMismatch: 'fail'`, which drew the line in the wrong place: **an explicit `object.datasource` binding is a hard dependency too**, and it is one the D2 note above already identifies as having *no fallback whatsoever* — such objects never resolve to the `default` driver, `engine.getDriver` throws `Datasource 'x' is not registered` for them. So a declared datasource that 20 objects bind to, failing to connect at boot, produced the worst possible shape: a server that starts clean, serves most of the app, and fails every read/write of those 20 objects with an error that reads nothing like "the analytics database is unreachable" — harder to locate than a total outage. The **declared-auto** fail-fast set is therefore:
77+
>
78+
> | | Gate | Boot connect failure |
79+
> |:---|:---|:---|
80+
> | (a) | `external` (`schemaMode !== 'managed'`) with `validation.onMismatch: 'fail'` | **fail-fast** |
81+
> | (b) | ≥1 object binds explicitly via `object.datasource` | **fail-fast** — no fallback path |
82+
> | (c) | `autoConnect: true`, nothing bound | degrade-with-warning — "connect it if you can"; nothing declares a dependency |
83+
>
84+
> This covers every reason a connect can fail, not just an unreachable socket: an unresolvable `external.credentialsRef` (D3) and an unsupported `driver` leave the bound objects exactly as dead, so they take the same verdict. `onMismatch` keeps its own meaning (what to do about a *schema* mismatch) and is no longer overloaded as the only way to say "this datasource is required".
85+
>
86+
> The escape hatch is **shared with the engine-level guard**, `OS_ALLOW_DRIVER_CONNECT_FAILURE=1` (framework#3741/#3751): the operator intent is identical ("I know the database is unreachable — boot anyway"), and two flags would only guarantee one of them gets missed. It now covers (a) as well, which previously had no opt-out. When set, the boot continues and the degraded state is announced through `emitDegradedBootBanner` (stderr as well as the logger, because `os serve`'s boot-quiet capture swallows stdout).
87+
>
88+
> The error message names the **bound objects** (up to 10, then `+N more`), not just the datasource — the service already receives them for post-connect `syncObjectSchema`. And `connectDeclared` attempts **every** gated datasource before throwing an aggregate, so one boot reports every misconfiguration rather than one per restart — the same shape as `ObjectQLEngine.init()`'s `DriverConnectError`.
89+
>
90+
> **A `DatasourceConnectPolicy` denial is not a connect failure** and stays metadata-only, unchanged. A multi-tenant host that blocks egress for a tenant's plan is making a deliberate decision about a datasource it knows it is refusing; fail-fasting there would turn a policy verdict into a boot outage for every tenant on the shared runtime. The fail-fast set covers *failures* — unreachable, unauthenticated, unsupported — not *refusals*.
91+
7692
### D6 — Native-analytics SQL honors the remote table/columns
7793

7894
The analytics native-SQL strategy compiles its own `FROM "<table>"` / column references outside the driver (ADR-0015 §18 noted this). It must resolve an external object's physical table (`remoteName`/`remoteSchema`) and columns (`columnMap`) the same way `SqlDriver` now does — reusing the driver's resolution (e.g. an exposed `physicalTableFor(object)` / `physicalColumnFor(object, field)`), not a second copy. Until then, analytics over external objects stays disabled rather than silently querying the wrong table.

examples/app-showcase/src/system/datasources/showcase-external.datasource.ts

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,15 @@ import { defineDatasource } from '@objectstack/spec/data';
1111
* queryable through the normal ObjectQL/REST surface.
1212
*
1313
* `schemaMode: 'external'` ⇒ ObjectStack never runs DDL here (it's a guest in a
14-
* database it does not own). `onMismatch: 'warn'` keeps a fixture hiccup from
15-
* bricking the whole showcase boot — the rest of the demo still loads.
14+
* database it does not own). `onMismatch: 'warn'` keeps a *schema drift* in the
15+
* fixture (a renamed column, an extra table) from bricking the showcase boot —
16+
* the rest of the demo still loads.
17+
*
18+
* It does **not** make a failed *connection* survivable, and shouldn't: the
19+
* `customer` / `order` objects bind to this datasource explicitly, so they have
20+
* no fallback driver and every query against them would fail (#3758). If the
21+
* fixture file cannot be opened at all, the boot stops with that as the reason
22+
* rather than serving a showcase whose federation pages are quietly dead.
1623
*
1724
* It also shows up in **Setup → Integrations → Datasources** and via
1825
* `GET /api/v1/meta/datasource`, where an admin can run the runtime

packages/objectql/src/driver-connect-errors.ts

Lines changed: 6 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -85,36 +85,10 @@ export class DriverConnectError extends Error {
8585
}
8686

8787
/**
88-
* Emit the degraded-boot banner on a channel the host cannot accidentally
89-
* silence.
90-
*
91-
* `OS_ALLOW_DRIVER_CONNECT_FAILURE` only justifies itself if the state it opts
92-
* into is impossible to miss — and a logger-only banner is missable: `os serve`
93-
* swallows ALL of stdout while the kernel boots (its "boot-quiet" capture), and
94-
* `Logger` routes `warn` to stdout, so the one message that matters would be
95-
* invisible in exactly the situation it exists for. Writing to stderr as well
96-
* is the same belt-and-braces the kernel already uses for plugin startup
97-
* failures.
98-
*
99-
* Best-effort and never throws: falls back to `console.error`, then to silence
100-
* on runtimes that have neither (the logger still carries the structured
101-
* record either way).
88+
* The degraded-boot banner now lives in `@objectstack/types` because
89+
* `DatasourceConnectionService` owes the operator the same banner for the same
90+
* flag (framework#3758) and must not depend on the whole query engine to print
91+
* it. Re-exported here so this module stays the single import site for
92+
* engine-side driver-connect failure reporting.
10293
*/
103-
export function emitDegradedBootBanner(message: string): void {
104-
const proc = (globalThis as {
105-
process?: { stderr?: { write?: (chunk: string) => unknown } };
106-
}).process;
107-
try {
108-
if (typeof proc?.stderr?.write === 'function') {
109-
proc.stderr.write(`${message}\n`);
110-
return;
111-
}
112-
} catch {
113-
/* stderr unavailable / closed — fall through to console */
114-
}
115-
try {
116-
(globalThis as { console?: { error?: (msg: string) => void } }).console?.error?.(message);
117-
} catch {
118-
/* no output channel at all — the logger record is the remaining trace */
119-
}
120-
}
94+
export { emitDegradedBootBanner } from '@objectstack/types';

0 commit comments

Comments
 (0)