feat(automation): ADR-0096 runtime re-materialization of declarative connectors (#2977)

Provider-bound declarative `connectors:` instances materialized only at boot, so
a connector published from Studio (or a dev reload) did not become dispatchable
until a restart. `materializeDeclaredConnectors` is now a reconcile against the
declared set, run at boot (fatal) and on `metadata:reloaded` (soft):

- Adds newly-declared instances, tears down removed / newly-`enabled:false` ones
  (calling their `close`, e.g. an MCP connection), and re-materializes only those
  whose signature — a stable hash of provider + providerConfig + auth + identity —
  changed. An unchanged instance is left untouched (no needless MCP reconnect on
  an unrelated metadata reload).
- Boot keeps the "fail loudly" contract (throws → fatal bootstrap). Reload is
  soft: unknown provider / unresolvable credentialRef / factory failure / name
  conflict is logged and the entry skipped, never crashing the live server; a
  changed instance's old connector keeps serving until its replacement
  materializes successfully.

Tracking (`materializedConnectorClosers` array → `materializedConnectors` Map)
carries the signature + close per name. 6 new reload tests (add/remove/disable/
unchanged/changed/soft-failure); service-automation suite green (285 tests).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pbmw3pMqNJfPbkvwh9Fkjs

Author: claude

.changeset/adr-0096-runtime-rematerialization.md

@@ -0,0 +1,21 @@
+---
+'@objectstack/service-automation': minor
+---
+
+feat(connectors): ADR-0096 runtime re-materialization of declarative connectors (#2977 follow-up)
+
+Provider-bound declarative `connectors:` instances (ADR-0096) previously
+materialized only at boot — a connector published from Studio while the server
+ran did not become dispatchable until a restart. `materializeDeclaredConnectors`
+is now a **reconcile** run both at boot and on `metadata:reloaded`:
+
+- **Add** newly-declared instances, **tear down** removed / newly-`enabled:false`
+  ones (calling their `close`, e.g. an MCP connection), and **re-materialize**
+  only instances whose signature — a stable hash of `provider` + `providerConfig`
+  + `auth` + identity — changed. An unchanged MCP instance is never needlessly
+  reconnected on an unrelated metadata reload.
+- **Boot stays fatal** ("fail loudly"): unknown provider / invalid providerConfig
+  / unresolvable credentialRef / name conflict aborts startup. **Reload is soft**:
+  the same problems are logged and the offending entry skipped, so a bad publish
+  never crashes a running server; a changed instance's old connector keeps
+  serving until its replacement materializes successfully.

docs/adr/0096-declarative-connector-instances.md

@@ -118,8 +118,22 @@ If a provider-bound instance and a plugin-registered connector share a `name`, b
 - **Open source:** the `rest` / `openapi` / `mcp` provider factories; **static** auth (`none` / `api-key` / `basic` / `bearer`); `credentialRef` resolved from **environment variables** (`defaultEnvCredentialResolver`) — the degraded-but-honest story for environments with no managed secrets service.
 - **Enterprise:** managed credential **vaulting** and OAuth2 authorization-code/refresh lifecycle (inject a vault-backed `CredentialResolver` via `AutomationServicePluginOptions.credentialResolver` — no change to the materialization path), plus per-tenant connection lifecycle. The open tier deliberately omits OAuth2 from `ConnectorInstanceAuthSchema`.
 
+### Runtime reconcile (follow-up, landed)
+
+Materialization is no longer boot-only. `materializeDeclaredConnectors` is a
+**reconcile** against the declared set, run both at boot (`fatal: true`) and on
+`metadata:reloaded` (`fatal: false` — a Studio publish / dev reload). The reconcile
+adds newly-declared instances, tears down removed / newly-`enabled:false` ones
+(calling their `close`), and re-materializes only those whose signature (a stable
+hash of `provider` + `providerConfig` + `auth` + identity) changed — an unchanged
+MCP instance is never needlessly reconnected. On reload a bad entry (unknown
+provider, unresolvable credentialRef, factory failure, name conflict) is **logged
+and skipped**, never crashing the live server; a changed instance's old connector
+keeps serving until its replacement materializes successfully. Boot keeps the
+"fail loudly" contract.
+
 ### Deliberate scope boundaries
 
-- **Boot-time only.** Materialization runs once at boot. Re-materializing a provider-bound instance published at runtime (Studio) is a follow-up; the descriptor audit still re-runs on `metadata:reloaded`.
 - **`providerConfig.spec` (openapi)** accepts an inline document or an http(s) URL; resolving a `./file.json` ref relative to the stack is the stack loader's job, not the connector's.
 - **MCP credentials** ride the transport (ADR-0024); for an http transport a resolved `auth` is folded into the request headers.
+- **MCP at boot** connects during materialization, so an unreachable server fails boot; a fail-soft "optional" marker for boot-time materialization is a possible future refinement (runtime reloads are already soft).

packages/services/service-automation/src/connector-materialization.test.ts

@@ -127,6 +127,70 @@ function automationOf(kernel: LiteKernel): AutomationEngine {
     return kernel.getService('automation') as AutomationEngine;
 }
 
+/**
+ * A fake provider whose materialized connectors carry a `close`, so reload tests
+ * can assert teardown. Records the ctx of every invocation (`calls`) and the
+ * names whose `close()` ran (`closed`).
+ */
+function makeClosableProvider() {
+    const calls: ConnectorProviderContext[] = [];
+    const closed: string[] = [];
+    const factory: ConnectorProviderFactory = (ctx) => {
+        calls.push(ctx);
+        return {
+            def: {
+                name: ctx.name,
+                label: ctx.label,
+                type: 'api',
+                authentication: { type: 'none' },
+                actions: [{ key: 'ping', label: 'Ping' }],
+            },
+            handlers: { ping: async () => ({ ok: true }) },
+            close: async () => { closed.push(ctx.name); },
+        };
+    };
+    return { factory, calls, closed };
+}
+
+/**
+ * Boot with a MUTABLE declared set and expose a `reload(next)` that swaps the
+ * registry contents and fires `metadata:reloaded` — the runtime reconcile path
+ * (ADR-0096 F1). The harness's ctx is the shared kernel context, so
+ * `ctx.trigger('metadata:reloaded')` invokes the automation plugin's hook.
+ */
+async function bootReloadable(
+    initial: unknown[],
+    opts: { providerFactory: ConnectorProviderFactory; credentialResolver?: CredentialResolver },
+) {
+    const state = { declared: initial };
+    let captured: any;
+    const kernel = new LiteKernel({ logger: { level: 'silent' } } as never);
+    kernel.use(new AutomationServicePlugin({ credentialResolver: opts.credentialResolver }));
+    const harness = {
+        name: 'test.harness',
+        type: 'standard' as const,
+        version: '1.0.0',
+        dependencies: ['com.objectstack.service-automation'],
+        async init(ctx: any) {
+            captured = ctx;
+            ctx.registerService('objectql', {
+                registry: { listItems: (type: string) => (type === 'connector' ? state.declared : []) },
+            });
+            ctx.getService('automation').registerConnectorProvider('fake', opts.providerFactory);
+        },
+        async start() {},
+    };
+    kernel.use(harness as never);
+    await kernel.bootstrap();
+    await flush();
+    const reload = async (next: unknown[]) => {
+        state.declared = next;
+        await captured.trigger('metadata:reloaded');
+        await flush();
+    };
+    return { kernel, engine: automationOf(kernel), reload };
+}
+
 describe('ADR-0096 — declarative connector materialization', () => {
     it('materializes a provider-bound instance into a live, listed connector', async () => {
         const { factory, calls } = makeFakeProvider();
@@ -282,3 +346,82 @@ describe('ADR-0096 — declarative connector materialization', () => {
         ).rejects.toThrow(/duplicate declarative connector instance name 'dup'/);
     });
 });
+
+describe('ADR-0096 — runtime re-materialization on metadata:reloaded (F1)', () => {
+    it('materializes an instance published after boot', async () => {
+        const { factory, calls } = makeClosableProvider();
+        const { engine, reload, kernel } = await bootReloadable([], { providerFactory: factory });
+        expect(engine.getRegisteredConnectors()).not.toContain('billing');
+
+        await reload([providerConnector('billing')]);
+        expect(engine.getRegisteredConnectors()).toContain('billing');
+        expect(engine.getConnectorOrigin('billing')).toBe('declarative');
+        expect(calls).toHaveLength(1);
+        await kernel.shutdown();
+    });
+
+    it('unregisters and tears down an instance removed after boot', async () => {
+        const { factory, closed } = makeClosableProvider();
+        const { engine, reload, kernel } = await bootReloadable([providerConnector('billing')], { providerFactory: factory });
+        expect(engine.getRegisteredConnectors()).toContain('billing');
+
+        await reload([]); // billing deleted from the stack
+        expect(engine.getRegisteredConnectors()).not.toContain('billing');
+        expect(closed).toEqual(['billing']); // close() ran on teardown
+        await kernel.shutdown();
+    });
+
+    it('tears down an instance newly marked enabled:false', async () => {
+        const { factory, closed } = makeClosableProvider();
+        const { engine, reload, kernel } = await bootReloadable([providerConnector('billing')], { providerFactory: factory });
+        await reload([providerConnector('billing', { enabled: false })]);
+        expect(engine.getRegisteredConnectors()).not.toContain('billing');
+        expect(closed).toEqual(['billing']);
+        await kernel.shutdown();
+    });
+
+    it('leaves an UNCHANGED instance untouched (no reconnect on unrelated reload)', async () => {
+        const { factory, calls, closed } = makeClosableProvider();
+        const { reload, kernel } = await bootReloadable(
+            [providerConnector('billing', { providerConfig: { baseUrl: 'https://x' } })],
+            { providerFactory: factory },
+        );
+        expect(calls).toHaveLength(1);
+
+        // A reload that does not change billing's inputs must not re-invoke the factory.
+        await reload([providerConnector('billing', { providerConfig: { baseUrl: 'https://x' } })]);
+        expect(calls).toHaveLength(1);
+        expect(closed).toEqual([]);
+        await kernel.shutdown();
+    });
+
+    it('re-materializes a CHANGED instance and tears down the old connection', async () => {
+        const { factory, calls, closed } = makeClosableProvider();
+        const { engine, reload, kernel } = await bootReloadable(
+            [providerConnector('billing', { providerConfig: { baseUrl: 'https://old' } })],
+            { providerFactory: factory },
+        );
+        expect(calls).toHaveLength(1);
+
+        await reload([providerConnector('billing', { providerConfig: { baseUrl: 'https://new' } })]);
+        expect(calls).toHaveLength(2); // re-materialized
+        expect(closed).toEqual(['billing']); // old connection torn down
+        expect(engine.getRegisteredConnectors()).toContain('billing');
+        expect((calls[1].providerConfig as { baseUrl?: string }).baseUrl).toBe('https://new');
+        await kernel.shutdown();
+    });
+
+    it('reload is soft: an unknown-provider entry is skipped, not fatal, and other connectors survive', async () => {
+        const { factory } = makeClosableProvider();
+        const { engine, reload, kernel } = await bootReloadable([providerConnector('billing')], { providerFactory: factory });
+
+        // A bad publish must NOT crash the running server (no throw), and the
+        // healthy connector keeps serving.
+        await expect(
+            reload([providerConnector('billing'), providerConnector('ghost', { provider: 'nonexistent' })]),
+        ).resolves.toBeUndefined();
+        expect(engine.getRegisteredConnectors()).toContain('billing');
+        expect(engine.getRegisteredConnectors()).not.toContain('ghost');
+        await kernel.shutdown();
+    });
+});