diff --git a/.claude/agents/phoenix-elixir-expert.md b/.claude/agents/phoenix-elixir-expert.md
index f342a5bb02c..d32d4e8e205 100644
--- a/.claude/agents/phoenix-elixir-expert.md
+++ b/.claude/agents/phoenix-elixir-expert.md
@@ -29,6 +29,15 @@ You are a **battle-tested Elixir/Phoenix architect** with deep expertise in the
- For transaction and prelim-type rules when touching y_ex from Elixir, see `.claude/guidelines/yex-guidelines.md §Transaction Deadlock Rules` and `§Prelim Types`.
+## OTP / supervision trees
+
+- When writing or reviewing a supervisor, GenServer, or named process, see
+ `.claude/guidelines/testable-supervision-trees.md §0. The principle` and
+ `§4. Anti-patterns checklist`. Names and collaborators are *parameters*, not
+ constants — don't bake `name: __MODULE__` into a process or resolve
+ dependencies from global state, or you force the suite serial (breaks
+ `async: true`).
+
## Lightning Project Context
**Architecture Awareness:**
diff --git a/.claude/guidelines/testable-supervision-trees.md b/.claude/guidelines/testable-supervision-trees.md
new file mode 100644
index 00000000000..f567f017e96
--- /dev/null
+++ b/.claude/guidelines/testable-supervision-trees.md
@@ -0,0 +1,323 @@
+---
+type: reference
+status: active
+date: 2026-05-19
+related:
+ - "[[elixir]]"
+tags:
+ - elixir
+ - otp
+ - testing
+---
+
+# Testable Supervision Trees & Named Processes
+
+Guidance for building supervision trees, supervisors, GenServers and named
+processes that are **uniquely addressable and isolated in tests** (so the suite
+runs `async: true`) but **need no name in normal production use**.
+
+This exists because AI harnesses (and people in a hurry) reliably reach for the
+shortcut — `name: __MODULE__` baked into the process, dependencies fished out
+of global state — which works in dev, looks idiomatic, and quietly forces the
+whole test suite serial. Point Claude at this doc when generating or reviewing
+OTP code.
+
+---
+
+## 0. The principle
+
+**A process's name, and its dependencies, are *parameters* — not constants.**
+
+Every failure mode here traces to one shortcut: hardcoding the name inside the
+process, or resolving a collaborator from global state at call time, instead of
+threading it through structure (supervision wiring, `start_link` opts, process
+state, the caller signature).
+
+Useful framing from Gray & Tate, *Designing Elixir Systems with OTP* ("Do Fun
+Things with Big, Loud Worker-Bees"): push logic into a pure functional core
+that needs no processes to test, so the GenServer is a thin shell. Everything
+below is damage control for the thin shell that remains.
+
+> **Scope guard.** Often the cleanest fix is that a thing never needed to be a
+> process or a stored value at all. That's worth one sentence at the design
+> review, then move on. The objective of this document is the **caller
+> signature and how information reaches child processes** — not relitigating
+> whether something should be a GenServer.
+
+---
+
+## 1. The 101 case: fixed children, no Registry
+
+A `Registry` is a lookup table from a **domain key → pid**. You need one only
+when *all three* hold:
+
+1. an **arbitrary / open-ended** number of the process exists,
+2. keyed by **runtime data** (a workflow id, a session id), and
+3. the code that must talk to one **does not already hold its pid**.
+
+A supervisor with a **fixed, known set of children** — one of each — fails all
+three. It needs nothing. The realisation:
+
+> For a constant set of children, **the registered module-atom name *is* your
+> registry, and it's free.** A registered name already survives restarts — the
+> supervisor brings the child back and it re-registers the same atom. That is
+> the one feature people reach to `Registry` for.
+
+The whole 101 pattern, no Registry, fully async-isolatable:
+
+```elixir
+defmodule MyApp.Cache do
+ use GenServer
+
+ # name is an OPTION, defaulted — never hardcoded inside the module
+ def start_link(opts) do
+ {name, opts} = Keyword.pop(opts, :name, __MODULE__)
+ GenServer.start_link(__MODULE__, opts, name: name)
+ end
+
+ # API takes the server ref FIRST, defaulted to the singleton
+ def fetch(server \\ __MODULE__, key), do: GenServer.call(server, {:fetch, key})
+
+ @impl true
+ def init(opts) do
+ # dependencies are injected, with prod defaults — never read from a global
+ {:ok, %{store: %{}, http: Keyword.get(opts, :http, MyApp.HTTP)}}
+ end
+end
+```
+
+Production (in the app supervisor's fixed child list) gets the singleton for
+free; callers write `MyApp.Cache.fetch(key)` and never name anything. The test
+never *looks anything up* — it **holds the pid it just started**:
+
+```elixir
+test "expires entries" do
+ pid = start_supervised!({MyApp.Cache, name: nil, http: HTTPMock})
+ assert MyApp.Cache.fetch(pid, :missing) == nil
+end
+```
+
+`name: nil` is the trick: it starts an anonymous, isolated instance even when
+the app already booted a global `MyApp.Cache`, so there is no
+`{:already_started, _}` clash, no need to gut `application.ex` in test config,
+and the test stays `async: true`. (Only reach for config-driven "don't boot it
+in `:test`" when something you *cannot* hand a pid — a Plug, a distant caller —
+calls the API with the default name. That is the exception, not the default.)
+
+### Argument order convention: server ref first, with a default
+
+This is the OTP-wide convention and it is near-universal:
+`GenServer.call(server, …)`, `Agent.get(agent, …)`,
+`Registry.lookup(registry, …)`, `Phoenix.PubSub.broadcast(pubsub, …)`,
+`Oban.insert(name \\ Oban, changeset)`, `Mox.allow(mock, …)`,
+`Ecto.Adapters.SQL.Sandbox.allow(repo, …)` — the addressed thing is the
+**subject**, so it leads.
+
+Critically, **first-with-default is the idiom that produces the `/1` + `/2`
+pair cleanly**:
+
+```elixir
+def fetch(server \\ __MODULE__, key)
+# fetch(key) -> arity 1, server defaults to __MODULE__ (production)
+# fetch(pid, key) -> arity 2, explicit instance (test)
+```
+
+Production never names anything; the test injects a pid through the *same*
+function. **The signature is the injection seam.** That is why this convention
+is load-bearing, not stylistic.
+
+The one principled exception: **pipeline-first APIs put the instance last** —
+`Finch.build(:get, url) |> Finch.request(MyFinch)` — because the data being
+transformed is the subject and the instance is configuration, so it reads in a
+pipe. Rule of thumb: *callers piping data through it → instance last; callers
+addressing a process → instance first (the common case).*
+
+---
+
+## 2. The two axes (the centrepiece)
+
+People — and AI harnesses — reliably weld together two **independent** problems.
+Welding them is what makes refactors thrash. Keep them apart:
+
+1. **Axis 1 — where does per-instance config / identity live?**
+2. **Axis 2 — which process actually invokes the injected dependency?**
+
+`:persistent_term` (and runtime `Application.put_env`, and naked global ETS) is
+the wrong answer to **Axis 1**. The `set_mox_global` / `async: false` /
+`Sandbox.allow` pain is entirely **Axis 2**. Fixing Axis 1 does not fix Axis 2
+— but it makes Axis 2 *legible*, which is the precondition for fixing it.
+
+### `:persistent_term` is a smell
+
+Treat `:persistent_term`, runtime `Application.put_env`, and naked global ETS
+for per-instance config as a **smell requiring explicit justification**. Its
+presence means someone reached for *stored state* when *structure* is the
+functional answer. The legitimacy litmus:
+
+> **Does this value ever need to differ between two tests running at the same
+> time?** If yes, it cannot live in any global store — inject it. If no, and
+> it is genuinely hot-read and fixed at boot, `:persistent_term` is fine
+> (Phoenix/Ecto use it internally for exactly that).
+
+### Worked example: `Lightning.Adaptors.Supervisor`
+
+The adaptors supervisor is otherwise *exemplary* — its moduledoc states the
+principle verbatim, it derives every child name via `Module.concat(name, …)`
+(a fixed child set with full multi-instance async isolation and **zero
+Registry**), and it injects the `:strategy` as an explicit opt with a
+production default. One wart:
+
+```elixir
+# supervisor.ex init/1 — strategy & source are LOCALS here…
+strategy = Keyword.get(opts, :strategy, Config.strategy())
+:persistent_term.put(meta_key(name), %{strategy: strategy, source: source_for(strategy)})
+# …then the same init/1 injects cache/tasks/source_topic into child specs
+# explicitly, two lines down — but routes strategy/source through a global.
+```
+
+`Scheduler` then re-fetches it from that global *at call time*:
+
+```elixir
+# scheduler.ex — strategy materialises from nowhere, with no traceable owner
+strategy = AdaptorsSupervisor.strategy(state.sup) # :persistent_term.get/1
+strategy.fetch_adaptor(name)
+```
+
+An investigation of every call site classified this **case (b): avoidable**.
+Nothing reaching `Store` lacks the strategy/source at a point where it is
+knowable; there is exactly one hardcoded production instance, so even the
+stateless facade-from-web path collapses to "boot config for the one instance,"
+not a dynamic lookup. The generalisable tell:
+
+> **Global storage smuggling a value past a structural boundary that was
+> already open two lines away and already carrying its siblings across.**
+
+The fix is Axis 1: inject `strategy`/`source` into the child specs the
+supervisor is *already building* (it has them in scope), and let `Scheduler`
+hold them in its state — exactly as it already does for `source` at `init/1`.
+
+### Why that makes Axis 2 legible
+
+Before: you cannot tell *which process* will call `StrategyMock`, because the
+value appears from a global with no owner — so you reach for `set_mox_global`
+and the suite goes serial.
+
+After injection: `Scheduler` visibly owns the strategy in its state, so the
+mock's caller is obvious and you can scope the allowance:
+
+```elixir
+# Axis 2 recipe — explicit allowance, async-safe
+pid = start_supervised!({Lightning.Adaptors.Supervisor,
+ name: name, strategy: StrategyMock})
+Mox.allow(StrategyMock, self(), Process.whereis(scheduler_name(name)))
+```
+
+When the pid does not exist yet at setup time (leader election, lazy start),
+use the **deferred-resolver form** of `Mox.allow/3` — the trick people forget:
+
+```elixir
+Mox.allow(StrategyMock, self(), fn ->
+ :global.whereis_name(scheduler_name(name))
+end)
+```
+
+Mox resolves the pid lazily on first mock invocation, sidestepping the race.
+Tasks started via `Task.Supervisor.async/async_nolink` carry `$callers`, so
+Mox walks back to the allowed parent automatically; `start_child`
+(fire-and-forget) does not propagate and needs its own allowance.
+
+`Mox.allow/3` and `Ecto.Adapters.SQL.Sandbox.allow/3` are **one concept** —
+same signature, same ownership model (a test owns a resource and explicitly
+lends it to processes it spawns). Mox's was modelled on Ecto's.
+
+> **When `set_mox_global` is legitimate, not a cop-out:** explicit allow is the
+> default; its cost scales with how many *process hops* the mocked call
+> traverses and how *dynamic* those pids are. `set_mox_global` (forcing
+> `async: false` for that case) is the correct escape hatch when the hop graph
+> is dynamic and deep — `HighlanderPG`-wrapped leader election is the textbook
+> case. Try the deferred-resolver form *first*; most cases are not Highlander.
+
+---
+
+## 3. Dynamic populations: when a Registry *is* earned
+
+When the population is genuinely open-ended and keyed by runtime data, and the
+caller does not hold the pid — `DynamicSupervisor` + `Registry` + `:via`.
+`Lightning.Collaborate` is the worked example: N sessions/documents keyed by
+`document_name`, looked up by a LiveView that did not start them.
+
+```elixir
+SessionSupervisor.start_child(
+ {Session, workflow: workflow, user: user,
+ name: Registry.via({:session, "#{document_name}:#{session_id}", user.id})}
+)
+```
+
+Two valid shapes, document the trade-off:
+
+- **One Registry per top-level instance** (Oban): perfect isolation, but the
+ instance name threads through every public function
+ (`Oban.insert(MyApp.Oban, …)`).
+- **One global Registry, key-namespaced** (Lightning collaboration): the public
+ API stays clean (`Collaborate.start/1` takes no name); isolation depends on
+ keys being genuinely unique. Correct when identity is naturally unique
+ (workflow + session); risky if a test can reuse a key.
+
+### The "earned name → config lookup" reference
+
+If a **stateless synchronous** caller genuinely holds only a name *and* the
+population is dynamic, a name → config lookup is justified — and even then it
+is **ETS-per-instance, never `:persistent_term`**. `commanded/eventstore` hit
+this exact requirement and chose: one named ETS table (`read_concurrency:
+true`) owned by a GenServer, holding `{name, pid, ref, store, config}`, the
+owner pid monitored so the row **self-deletes on shutdown**. It explicitly
+rejected `:persistent_term` because instances churn under async tests and
+`:persistent_term` writes/erases trigger a global GC scan. The bonus: an
+owner-monitored ETS table needs no manual cleanup function (cf. the adaptors
+`forget/1` wart and its "we don't call this automatically because GC is
+expensive" comment).
+
+---
+
+## 4. Anti-patterns checklist (point the harness here)
+
+- ❌ `name: __MODULE__` hardcoded *inside* `start_link` on a worker. → Make
+ `:name` an option defaulting to `__MODULE__`.
+- ❌ Public API calling `GenServer.call(__MODULE__, …)` with no server arg. →
+ `def fn(server \\ __MODULE__, …)`, server first.
+- ❌ Dependencies / per-instance config read from `:persistent_term` /
+ runtime `Application.put_env` / global ETS at call time. → Inject via
+ `start_link` opts → `init/1` → process state, or thread through the child
+ spec. Litmus: *does it vary per concurrent test?*
+- ❌ Global storage used to pass a value the supervisor already had in scope
+ one child-spec away. → Inject it into the child spec.
+- ❌ `set_mox_global` reached for reflexively to "fix" a boundary. → Try
+ `Mox.allow(mock, self(), pid)`, then the deferred-resolver form; reserve
+ global for genuinely dynamic/deep hop graphs.
+- ❌ A `Registry` added to a *fixed* child set "to name things." → The
+ registered atom name already does this, free.
+- ❌ Testing GenServer internals via raw `:"$gen_call"` / poking state. →
+ Test through the public API; test logic in a pure core module.
+- ❌ App supervisor that cannot be started piecemeal in tests. → Children
+ startable independently via `start_supervised!`.
+
+---
+
+## References
+
+- Gray & Tate, *Designing Elixir Systems with OTP* (Pragmatic Bookshelf) —
+ functional core / boundary layering.
+- Saša Jurić, *Elixir in Action* — process registration as a parameter; `:via`.
+- [Oban — instance & DB isolation](https://hexdocs.pm/oban/isolation.html) ·
+ [`Oban.Registry`](https://hexdocs.pm/oban/Oban.Registry.html)
+- [`Ecto.Adapters.SQL.Sandbox`](https://hexdocs.pm/ecto_sql/Ecto.Adapters.SQL.Sandbox.html)
+ — ownership / `allow/3`.
+- [Mox docs](https://hexdocs.pm/mox/Mox.html) — `allow/3`, deferred resolver,
+ `set_mox_from_context`.
+- [`commanded/eventstore`](https://github.com/commanded/eventstore) —
+ `lib/event_store/config/store.ex` (ETS-per-instance, owner-monitored).
+- [`Registry`](https://hexdocs.pm/elixir/Registry.html) ·
+ [Thoughtbot — dynamic process names](https://thoughtbot.com/blog/how-to-start-processes-with-dynamic-names-in-elixir)
+- In-repo: `lib/lightning/adaptors/supervisor.ex`,
+ `lib/lightning/collaboration.ex`,
+ `lib/lightning/collaboration/registry.ex`.
diff --git a/.gitignore b/.gitignore
index f32ebcacfa4..fbf1f782dd7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -75,6 +75,7 @@ priv/openfn
.elixir_ls
.elixir-tools
+.expert
.env
.env.dev
diff --git a/CLAUDE.md b/CLAUDE.md
index fea632d0f8a..dc872c1d71f 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -132,6 +132,11 @@ in development.
- Use `{}` brace syntax in HEEx templates
- `warnings_as_errors: true` - code must compile without warnings
+> Before writing or reviewing any supervisor, GenServer, or named process, see
+> `.claude/guidelines/testable-supervision-trees.md`. Don't bake `name:
+> __MODULE__` into a process or resolve collaborators from global state — it
+> forces the test suite serial.
+
### React/TypeScript
- Props from LiveView are **underscore_cased** (not camelCase)
@@ -209,5 +214,6 @@ Detailed guidelines in `.claude/guidelines/`:
- `testing-essentials.md` - Unit testing patterns and anti-patterns
- `e2e-testing.md` - Playwright E2E testing
- `yex-guidelines.md` - Critical Yex (Yjs/Elixir) usage rules
+- `testable-supervision-trees.md` - OTP processes addressable in tests (async), nameless in prod
- `toast-notifications.md` - Notification patterns
- `ui-patterns.md` - Button variants, disabled states, Tailwind conventions
\ No newline at end of file
diff --git a/assets/js/collaborative-editor/components/AdaptorIcon.tsx b/assets/js/collaborative-editor/components/AdaptorIcon.tsx
index 51df8928377..918c157dbe3 100644
--- a/assets/js/collaborative-editor/components/AdaptorIcon.tsx
+++ b/assets/js/collaborative-editor/components/AdaptorIcon.tsx
@@ -1,6 +1,7 @@
-import useAdaptorIcons from '#/workflow-diagram/useAdaptorIcons';
+import { useContext, useMemo, useSyncExternalStore } from 'react';
-import { extractAdaptorName } from '../utils/adaptorUtils';
+import { StoreContext } from '../contexts/StoreProvider';
+import { extractAdaptorName, extractPackageName } from '../utils/adaptorUtils';
interface AdaptorIconProps {
name: string;
@@ -13,11 +14,35 @@ const sizeClasses = {
lg: 'h-12 w-12',
};
+// Reads the square icon URL for `name` directly from StoreContext, so callers
+// that mock the `hooks/useAdaptors` module (e.g. FullScreenIDE tests) still get
+// the existing placeholder fallback instead of crashing on a missing mock.
+function useStoreIconUrl(name: string): string | null {
+ const context = useContext(StoreContext);
+ const adaptorStore = context?.adaptorStore ?? null;
+ const packageName = extractPackageName(name);
+
+ const selectIconUrl = useMemo(() => {
+ if (!adaptorStore) return () => null;
+ return adaptorStore.withSelector(state => {
+ const found = state.adaptors.find(a => a.name === packageName);
+ return found?.icon_urls?.square ?? null;
+ });
+ }, [adaptorStore, packageName]);
+
+ const noopSubscribe = useMemo(() => () => () => {}, []);
+
+ return useSyncExternalStore(
+ adaptorStore?.subscribe ?? noopSubscribe,
+ selectIconUrl
+ );
+}
+
export function AdaptorIcon({ name, size = 'md' }: AdaptorIconProps) {
- const adaptorIconsData = useAdaptorIcons();
const displayName = extractAdaptorName(name) ?? null;
+ const iconUrl = useStoreIconUrl(name);
- if (!adaptorIconsData || !displayName) {
+ if (!displayName) {
return (
diff --git a/assets/js/collaborative-editor/components/AdaptorSelectionModal.tsx b/assets/js/collaborative-editor/components/AdaptorSelectionModal.tsx
index 9a18635fc9a..8ade25c2726 100644
--- a/assets/js/collaborative-editor/components/AdaptorSelectionModal.tsx
+++ b/assets/js/collaborative-editor/components/AdaptorSelectionModal.tsx
@@ -14,7 +14,7 @@ interface AdaptorSelectionModalProps {
isOpen: boolean;
onClose: () => void;
onSelect: (adaptorSpec: string) => void;
- projectAdaptors?: Adaptor[];
+ adaptorsInUse?: Adaptor[];
}
interface AdaptorWithDisplayName extends Adaptor {
@@ -25,7 +25,7 @@ export function AdaptorSelectionModal({
isOpen,
onClose,
onSelect,
- projectAdaptors = [],
+ adaptorsInUse = [],
}: AdaptorSelectionModalProps) {
const allAdaptors = useAdaptors();
const [searchQuery, setSearchQuery] = useState('');
@@ -60,17 +60,17 @@ export function AdaptorSelectionModal({
[allAdaptors]
);
- const projectAdaptorsWithDisplayNames = useMemo<
+ const adaptorsInUseWithDisplayNames = useMemo<
AdaptorWithDisplayName[]
>(() => {
- return projectAdaptors.map(adaptor => ({
+ return adaptorsInUse.map(adaptor => ({
...adaptor,
displayName: getAdaptorDisplayName(adaptor.name, {
titleCase: true,
fallback: adaptor.name,
}),
}));
- }, [projectAdaptors]);
+ }, [adaptorsInUse]);
const allAdaptorsWithDisplayNames = useMemo(() => {
return allAdaptors.map(adaptor => ({
@@ -89,10 +89,10 @@ export function AdaptorSelectionModal({
// Filter project adaptors
const filteredProject = searchQuery
- ? projectAdaptorsWithDisplayNames.filter(adaptor =>
+ ? adaptorsInUseWithDisplayNames.filter(adaptor =>
adaptor.displayName.toLowerCase().includes(lowerQuery)
)
- : projectAdaptorsWithDisplayNames;
+ : adaptorsInUseWithDisplayNames;
// Filter all adaptors and exclude duplicates from project adaptors
const projectAdaptorNames = new Set(filteredProject.map(a => a.name));
@@ -130,7 +130,7 @@ export function AdaptorSelectionModal({
};
}, [
searchQuery,
- projectAdaptorsWithDisplayNames,
+ adaptorsInUseWithDisplayNames,
allAdaptorsWithDisplayNames,
httpAdaptor,
]);
diff --git a/assets/js/collaborative-editor/components/AdaptorSelector.tsx b/assets/js/collaborative-editor/components/AdaptorSelector.tsx
index ab63dd35159..3f34af656f6 100644
--- a/assets/js/collaborative-editor/components/AdaptorSelector.tsx
+++ b/assets/js/collaborative-editor/components/AdaptorSelector.tsx
@@ -21,7 +21,7 @@ interface AdaptorSelectorProps {
/** Setter to control configure modal state */
setIsConfigureModalOpen: (open: boolean) => void;
/** Available project adaptors */
- projectAdaptors: Adaptor[];
+ adaptorsInUse: Adaptor[];
/** Optional callback before adaptor change (for form sync in JobForm) */
onAdaptorChangeStart?: () => void;
}
@@ -45,7 +45,7 @@ export function AdaptorSelector({
job,
updateJob,
setIsConfigureModalOpen,
- projectAdaptors,
+ adaptorsInUse,
onAdaptorChangeStart,
}: AdaptorSelectorProps) {
const {
@@ -90,7 +90,7 @@ export function AdaptorSelector({
isOpen={isOpen}
onClose={handlePickerCloseGuarded}
onSelect={handleAdaptorSelect}
- projectAdaptors={projectAdaptors}
+ adaptorsInUse={adaptorsInUse}
/>
(null);
// Fetch project adaptors for modal
- const { projectAdaptors } = useProjectAdaptors();
+ const { adaptorsInUse } = useAdaptorsInUse();
const updateSelection = useCallback(
(id?: string | null) => {
@@ -1000,7 +1000,7 @@ export default function WorkflowDiagram(props: WorkflowDiagramProps) {
isOpen={pendingPlaceholder !== null}
onClose={handleAdaptorModalClose}
onSelect={handleAdaptorSelect}
- projectAdaptors={projectAdaptors}
+ adaptorsInUse={adaptorsInUse}
/>
>
);
diff --git a/assets/js/collaborative-editor/components/ide/FullScreenIDE.tsx b/assets/js/collaborative-editor/components/ide/FullScreenIDE.tsx
index f3eb4db900e..74999f523e5 100644
--- a/assets/js/collaborative-editor/components/ide/FullScreenIDE.tsx
+++ b/assets/js/collaborative-editor/components/ide/FullScreenIDE.tsx
@@ -26,7 +26,7 @@ import * as dataclipApi from '../../api/dataclips';
import { RENDER_MODES } from '../../constants/panel';
import { useCredentialModal } from '../../contexts/CredentialModalContext';
import { useMonacoRef } from '../../contexts/MonacoRefContext';
-import { useProjectAdaptors } from '../../hooks/useAdaptors';
+import { useAdaptorsInUse } from '../../hooks/useAdaptors';
import {
useCredentials,
useCredentialsCommands,
@@ -395,7 +395,7 @@ export function FullScreenIDE({
const { projectCredentials, keychainCredentials } = useCredentials();
const { requestCredentials } = useCredentialsCommands();
- const { projectAdaptors, allAdaptors } = useProjectAdaptors();
+ const { adaptorsInUse, allAdaptors } = useAdaptorsInUse();
const { updateJob } = useWorkflowActions();
// Credential modal is managed by the context
@@ -410,19 +410,19 @@ export function FullScreenIDE({
// to be used by components that can't make use of 'latest'
const currJobAdaptor = useMemo(() => {
if (!currentJob?.adaptor) {
- const latestCommon = projectAdaptors.find(
+ const latestCommon = adaptorsInUse.find(
a => a.name === '@openfn/language-common'
)?.versions?.[0]?.version;
return `@openfn/language-common@${latestCommon || 'latest'}`;
}
const resolved = resolveAdaptor(currentJob.adaptor);
if (resolved.version !== 'latest') return currentJob?.adaptor;
- const latestVersion = projectAdaptors.find(a => a.name === resolved.package)
+ const latestVersion = adaptorsInUse.find(a => a.name === resolved.package)
?.versions?.[0]?.version;
// If version not found, return original adaptor string
if (!latestVersion) return currentJob.adaptor;
return `${resolved.package}@${latestVersion}`;
- }, [projectAdaptors, currentJob?.adaptor]);
+ }, [adaptorsInUse, currentJob?.adaptor]);
// Run/Retry functionality for IDE Header
const { canRun: canRunSnapshot, tooltipMessage: runTooltipMessage } =
@@ -1356,7 +1356,7 @@ export function FullScreenIDE({
job={currentJob}
updateJob={updateJob}
setIsConfigureModalOpen={setIsConfigureModalOpen}
- projectAdaptors={projectAdaptors}
+ adaptorsInUse={adaptorsInUse}
/>
>
)}
diff --git a/assets/js/collaborative-editor/components/inspector/JobForm.tsx b/assets/js/collaborative-editor/components/inspector/JobForm.tsx
index 201ed6593d3..53863e9db79 100644
--- a/assets/js/collaborative-editor/components/inspector/JobForm.tsx
+++ b/assets/js/collaborative-editor/components/inspector/JobForm.tsx
@@ -3,7 +3,7 @@ import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
import { useAppForm } from '#/collaborative-editor/components/form';
import { useCredentialModal } from '#/collaborative-editor/contexts/CredentialModalContext';
-import { useProjectAdaptors } from '#/collaborative-editor/hooks/useAdaptors';
+import { useAdaptorsInUse } from '#/collaborative-editor/hooks/useAdaptors';
import {
useCredentials,
useCredentialsCommands,
@@ -55,7 +55,7 @@ export function JobForm({ job }: JobFormProps) {
const { updateJob } = useWorkflowActions();
const { projectCredentials, keychainCredentials } = useCredentials();
const { requestCredentials } = useCredentialsCommands();
- const { projectAdaptors, allAdaptors } = useProjectAdaptors();
+ const { adaptorsInUse, allAdaptors } = useAdaptorsInUse();
const { isReadOnly } = useWorkflowReadOnly();
// Modal state for adaptor configuration
@@ -395,7 +395,7 @@ export function JobForm({ job }: JobFormProps) {
job={job}
updateJob={updateJob}
setIsConfigureModalOpen={setIsConfigureModalOpen}
- projectAdaptors={projectAdaptors}
+ adaptorsInUse={adaptorsInUse}
onAdaptorChangeStart={syncAdaptorToForm}
/>
diff --git a/assets/js/collaborative-editor/hooks/useAdaptors.ts b/assets/js/collaborative-editor/hooks/useAdaptors.ts
index 208bdb790c3..265fa2ada3f 100644
--- a/assets/js/collaborative-editor/hooks/useAdaptors.ts
+++ b/assets/js/collaborative-editor/hooks/useAdaptors.ts
@@ -10,7 +10,8 @@ import { useSyncExternalStore, useContext, useMemo } from 'react';
import { StoreContext } from '../contexts/StoreProvider';
import type { AdaptorStoreInstance } from '../stores/createAdaptorStore';
import type { Adaptor } from '../types/adaptor';
-import type { Job } from '../types/workflow';
+import type { Workflow } from '../types/workflow';
+import { extractPackageName } from '../utils/adaptorUtils';
/**
* Main hook for accessing the AdaptorStore instance
@@ -86,88 +87,88 @@ export const useAdaptor = (name: string): Adaptor | null => {
};
/**
- * Extracts adaptor package name from a full adaptor specifier
- * e.g., "@openfn/language-common@1.0.0" -> "@openfn/language-common"
+ * Hook to read an adaptor's square-shape icon URL from the AdaptorStore.
+ *
+ * Accepts a full adaptor specifier (with or without version suffix). When no
+ * StoreProvider is mounted (e.g. the LiveView workflow-editor path), returns
+ * `null` rather than throwing so consumers fall back to their string label.
*/
-const getAdaptorPackageName = (adaptor: string | undefined): string | null => {
- if (!adaptor) return null;
- const match = adaptor.match(/^(@[^@]+)@/);
- return match ? match[1] : null;
+export const useAdaptorIconUrl = (
+ adaptor: string | null | undefined
+): string | null => {
+ const context = useContext(StoreContext);
+ const adaptorStore = context?.adaptorStore ?? null;
+
+ const packageName = adaptor ? extractPackageName(adaptor) : null;
+
+ const selectIconUrl = useMemo(() => {
+ if (!adaptorStore) return () => null;
+ return adaptorStore.withSelector(state => {
+ if (!packageName) return null;
+ const found = state.adaptors.find(a => a.name === packageName);
+ return found?.icon_urls?.square ?? null;
+ });
+ }, [adaptorStore, packageName]);
+
+ const noopSubscribe = useMemo(() => () => () => {}, []);
+
+ return useSyncExternalStore(
+ adaptorStore?.subscribe ?? noopSubscribe,
+ selectIconUrl
+ );
};
/**
- * Hook to get project-specific adaptors and all adaptors
- * Returns both project adaptors and all adaptors from backend endpoint
- *
- * Project adaptors are merged from two sources:
- * 1. Backend DB (saved jobs)
- * 2. Y.Doc state (unsaved jobs in collaborative editor)
- *
- * This ensures newly added adaptors appear in projectAdaptors before saving.
+ * Hook to derive the subset of the adaptor catalogue that is referenced by jobs
+ * in the current Y.Doc workflow. Pure selector — the catalogue comes from
+ * `request_adaptors` and the jobs come from the collaborative workflow store.
*/
-export const useProjectAdaptors = (): {
- projectAdaptors: Adaptor[];
+export const useAdaptorsInUse = (): {
+ adaptorsInUse: Adaptor[];
allAdaptors: Adaptor[];
isLoading: boolean;
} => {
const context = useContext(StoreContext);
if (!context) {
- throw new Error('useProjectAdaptors must be used within a StoreProvider');
+ throw new Error('useAdaptorsInUse must be used within a StoreProvider');
}
const { adaptorStore, workflowStore } = context;
- // Get adaptor state from adaptor store
- const selectAdaptorData = adaptorStore.withSelector(state => ({
- backendProjectAdaptors: state.projectAdaptors || [],
- allAdaptors: state.adaptors,
- isLoading: state.isLoading,
- }));
+ const selectAdaptors = adaptorStore.withSelector(state => state.adaptors);
+ const selectIsLoading = adaptorStore.withSelector(state => state.isLoading);
+ const selectJobs = workflowStore.withSelector(state => state.jobs);
- const adaptorData = useSyncExternalStore(
+ const allAdaptors = useSyncExternalStore(
adaptorStore.subscribe,
- selectAdaptorData
+ selectAdaptors
+ );
+ const isLoading = useSyncExternalStore(
+ adaptorStore.subscribe,
+ selectIsLoading
+ );
+ const jobs: Workflow.Job[] = useSyncExternalStore(
+ workflowStore.subscribe,
+ selectJobs
);
- // Get jobs from workflow store (Y.Doc state)
- const selectJobs = workflowStore.withSelector(state => state.jobs);
- const jobs: Job[] = useSyncExternalStore(workflowStore.subscribe, selectJobs);
-
- // Merge backend project adaptors with Y.Doc job adaptors
- const projectAdaptors = useMemo(() => {
- const { backendProjectAdaptors, allAdaptors } = adaptorData;
-
- // Get adaptor names already in backend list
- const backendAdaptorNames = new Set(
- backendProjectAdaptors.map(a => a.name)
- );
+ const adaptorsInUse = useMemo(() => {
+ if (jobs.length === 0) return [];
- // Find adaptors used in Y.Doc jobs that aren't in backend list
- const ydocAdaptorNames = new Set();
+ const names = new Set();
for (const job of jobs) {
- const packageName = getAdaptorPackageName(job.adaptor);
- if (packageName && !backendAdaptorNames.has(packageName)) {
- ydocAdaptorNames.add(packageName);
- }
+ if (!job.adaptor) continue;
+ names.add(extractPackageName(job.adaptor));
}
- // If no new adaptors from Y.Doc, return backend list as-is
- if (ydocAdaptorNames.size === 0) {
- return backendProjectAdaptors;
- }
-
- // Find full adaptor objects from allAdaptors for Y.Doc adaptors
- const ydocAdaptors = allAdaptors.filter(a => ydocAdaptorNames.has(a.name));
-
- // Merge and sort
- return [...backendProjectAdaptors, ...ydocAdaptors].sort((a, b) =>
- a.name.localeCompare(b.name)
- );
- }, [adaptorData, jobs]);
+ return allAdaptors
+ .filter(a => names.has(a.name))
+ .sort((a, b) => a.name.localeCompare(b.name));
+ }, [allAdaptors, jobs]);
return {
- projectAdaptors,
- allAdaptors: adaptorData.allAdaptors,
- isLoading: adaptorData.isLoading,
+ adaptorsInUse,
+ allAdaptors,
+ isLoading,
};
};
diff --git a/assets/js/collaborative-editor/stores/createAdaptorStore.ts b/assets/js/collaborative-editor/stores/createAdaptorStore.ts
index 562147b5680..3712599a005 100644
--- a/assets/js/collaborative-editor/stores/createAdaptorStore.ts
+++ b/assets/js/collaborative-editor/stores/createAdaptorStore.ts
@@ -96,6 +96,29 @@ import { wrapStoreWithDevTools } from './devtools';
const logger = _logger.ns('AdaptorStore').seal();
+// Deep-equality check tailored to the Adaptor shape so referential identity is
+// preserved across no-op `adaptors_updated` pushes.
+function adaptorsEqual(a: Adaptor, b: Adaptor): boolean {
+ if (a === b) return true;
+ if (
+ a.name !== b.name ||
+ a.repo !== b.repo ||
+ a.latest !== b.latest ||
+ a.icon_urls.square !== b.icon_urls.square ||
+ a.icon_urls.rectangle !== b.icon_urls.rectangle ||
+ a.versions.length !== b.versions.length
+ ) {
+ return false;
+ }
+ for (let i = 0; i < a.versions.length; i++) {
+ const aVer = a.versions[i];
+ const bVer = b.versions[i];
+ if (!aVer || !bVer) return false;
+ if (aVer.version !== bVer.version) return false;
+ }
+ return true;
+}
+
// sorts adaptors coming into the adaptor store
// 1. sorts the versions for every adaptor
// 2. sorts the adaptors themselves by name
@@ -119,7 +142,6 @@ export const createAdaptorStore = (): AdaptorStore => {
let state: AdaptorState = produce(
{
adaptors: [],
- projectAdaptors: [],
isLoading: false,
error: null,
lastUpdated: null,
@@ -170,14 +192,36 @@ export const createAdaptorStore = (): AdaptorStore => {
const result = AdaptorsListSchema.safeParse(rawData);
if (result.success) {
- const adaptors = sortAdaptors(result.data);
-
- state = produce(state, draft => {
- draft.adaptors = adaptors;
- draft.isLoading = false;
- draft.error = null;
- draft.lastUpdated = Date.now();
+ const incoming = sortAdaptors(result.data);
+ const existing = state.adaptors;
+ const existingByName = new Map(existing.map(a => [a.name, a]));
+
+ // Merge by name to preserve referential identity of unchanged adaptors so
+ // `withSelector` consumers don't re-render on no-op `adaptors_updated`
+ // pushes.
+ const merged: Adaptor[] = incoming.map(next => {
+ const prev = existingByName.get(next.name);
+ return prev && adaptorsEqual(prev, next) ? prev : next;
});
+
+ const arrayUnchanged =
+ merged.length === existing.length &&
+ merged.every((a, i) => a === existing[i]);
+
+ if (arrayUnchanged) {
+ state = produce(state, draft => {
+ draft.isLoading = false;
+ draft.error = null;
+ draft.lastUpdated = Date.now();
+ });
+ } else {
+ state = produce(state, draft => {
+ draft.adaptors = merged;
+ draft.isLoading = false;
+ draft.error = null;
+ draft.lastUpdated = Date.now();
+ });
+ }
notify('handleAdaptorsReceived');
} else {
const errorMessage = `Invalid adaptors data: ${result.error.message}`;
@@ -267,7 +311,6 @@ export const createAdaptorStore = (): AdaptorStore => {
devtools.connect();
void requestAdaptors();
- void requestProjectAdaptors();
return () => {
devtools.disconnect();
@@ -310,56 +353,6 @@ export const createAdaptorStore = (): AdaptorStore => {
}
};
- /**
- * Request project adaptors from server via channel
- */
- const requestProjectAdaptors = async (): Promise => {
- if (!channelProvider?.channel) {
- logger.warn('Cannot request project adaptors - no channel connected');
- setError('No connection available');
- return;
- }
-
- setLoading(true);
- clearError();
-
- try {
- logger.debug('Requesting project adaptors');
- const response = await channelRequest(
- channelProvider.channel,
- 'request_project_adaptors',
- {}
- );
-
- if (response && typeof response === 'object') {
- const { project_adaptors, all_adaptors } = response as {
- project_adaptors: unknown;
- all_adaptors: unknown;
- };
-
- const projectResult = AdaptorsListSchema.safeParse(project_adaptors);
- const allResult = AdaptorsListSchema.safeParse(all_adaptors);
-
- if (projectResult.success && allResult.success) {
- state = produce(state, draft => {
- draft.projectAdaptors = sortAdaptors(projectResult.data);
- draft.adaptors = sortAdaptors(allResult.data);
- draft.isLoading = false;
- draft.error = null;
- });
- notify('requestProjectAdaptors');
- } else {
- const errorMessage = 'Invalid project adaptors data';
- logger.error(errorMessage, { projectResult, allResult });
- setError(errorMessage);
- }
- }
- } catch (error) {
- logger.error('Project adaptors request failed', error);
- setError('Failed to request project adaptors');
- }
- };
-
// =============================================================================
// QUERY HELPERS
// =============================================================================
@@ -390,7 +383,6 @@ export const createAdaptorStore = (): AdaptorStore => {
// Commands (CQS pattern)
requestAdaptors,
- requestProjectAdaptors,
setAdaptors,
setLoading,
setError,
@@ -403,13 +395,6 @@ export const createAdaptorStore = (): AdaptorStore => {
// Internal methods (not part of public AdaptorStore interface)
_connectChannel: connectChannel,
- // Test helper to set project adaptors directly
- _setProjectAdaptors: (adaptors: Adaptor[]) => {
- state = produce(state, draft => {
- draft.projectAdaptors = sortAdaptors(adaptors);
- });
- notify('_setProjectAdaptors');
- },
};
};
diff --git a/assets/js/collaborative-editor/types/adaptor.ts b/assets/js/collaborative-editor/types/adaptor.ts
index 58ff2a42efe..27291f19751 100644
--- a/assets/js/collaborative-editor/types/adaptor.ts
+++ b/assets/js/collaborative-editor/types/adaptor.ts
@@ -22,11 +22,17 @@ export const AdaptorVersionSchema = z.object({
/**
* Single adaptor schema with all its versions
*/
+export const AdaptorIconUrlsSchema = z.object({
+ square: z.string().nullable(),
+ rectangle: z.string().nullable(),
+});
+
export const AdaptorSchema = z.object({
name: z.string(),
versions: z.array(AdaptorVersionSchema),
repo: z.string(),
latest: z.string(),
+ icon_urls: AdaptorIconUrlsSchema,
});
/**
@@ -60,9 +66,6 @@ export interface AdaptorState {
/** Current list of available adaptors */
adaptors: AdaptorsList;
- /** Project-specific adaptors used across workflows */
- projectAdaptors: AdaptorsList;
-
/** Loading state for initial fetch */
isLoading: boolean;
@@ -80,9 +83,6 @@ export interface AdaptorCommands {
/** Request adaptors list from server */
requestAdaptors: () => Promise;
- /** Request project-specific adaptors from server */
- requestProjectAdaptors: () => Promise;
-
/** Manually set adaptors (for testing/fallback) */
setAdaptors: (adaptors: AdaptorsList) => void;
diff --git a/assets/js/workflow-diagram/components/MiniMapNode.tsx b/assets/js/workflow-diagram/components/MiniMapNode.tsx
index 1e08df865ca..d667bc9e783 100644
--- a/assets/js/workflow-diagram/components/MiniMapNode.tsx
+++ b/assets/js/workflow-diagram/components/MiniMapNode.tsx
@@ -2,9 +2,9 @@ import { ClockIcon, GlobeAltIcon } from '@heroicons/react/24/outline';
import type { MiniMapNodeProps } from '@xyflow/react';
import { memo } from 'react';
+import { useAdaptorIconUrl } from '#/collaborative-editor/hooks/useAdaptors';
+
import { useWorkflowStore } from '../../workflow-store/store';
-import useAdaptorIcons from '../useAdaptorIcons';
-import getAdaptorName from '../util/get-adaptor-name';
type Trigger = {
id: string;
@@ -54,11 +54,12 @@ const MiniMapNode = ({
const storeData = useWorkflowStore();
const jobs = propJobs ?? storeData.jobs;
const triggers = propTriggers ?? storeData.triggers;
- const adaptorIconsData = useAdaptorIcons();
// Check if this node is a trigger by looking it up in the triggers array
const trigger = triggers.find((trigger: Trigger) => trigger.id === id);
const isTrigger = !!trigger;
+ const job = jobs.find((job: Job) => job.id === id);
+ const icon = useAdaptorIconUrl(job?.adaptor);
// For triggers, we'll use the appropriate icon
if (isTrigger) {
@@ -87,14 +88,6 @@ const MiniMapNode = ({
);
}
- // For jobs, we'll use the adaptor icon if available
- const job = jobs.find((job: Job) => job.id === id);
- const adaptor = job?.adaptor ? getAdaptorName(job.adaptor) : null;
- const icon =
- adaptor && adaptorIconsData && adaptor in adaptorIconsData
- ? adaptorIconsData[adaptor]?.square
- : null;
-
// Fallback to rectangle if no icon is available
return (
diff --git a/assets/js/workflow-diagram/nodes/Job.tsx b/assets/js/workflow-diagram/nodes/Job.tsx
index 1e71c0a7b17..9ce572a6a7b 100644
--- a/assets/js/workflow-diagram/nodes/Job.tsx
+++ b/assets/js/workflow-diagram/nodes/Job.tsx
@@ -1,8 +1,9 @@
import { Position, type NodeProps } from '@xyflow/react';
import { memo } from 'react';
+import { useAdaptorIconUrl } from '#/collaborative-editor/hooks/useAdaptors';
+
import PathButton from '../components/PathButton';
-import useAdaptorIcons, { type AdaptorIconData } from '../useAdaptorIcons';
import getAdaptorName from '../util/get-adaptor-name';
import Node from './Node';
@@ -22,10 +23,9 @@ const JobNode = ({
],
];
- const adaptorIconsData = useAdaptorIcons();
-
const adaptor = getAdaptorName(props.data?.adaptor);
- const icon = getAdaptorIcon(adaptor, adaptorIconsData);
+ const iconUrl = useAdaptorIconUrl(props.data?.adaptor);
+ const icon = iconUrl ? : adaptor;
return (
;
- } else {
- return adaptor;
- }
- } catch {
- return adaptor;
- }
-}
-
export default memo(JobNode);
diff --git a/assets/test/collaborative-editor/components/AdaptorIcon.test.tsx b/assets/test/collaborative-editor/components/AdaptorIcon.test.tsx
new file mode 100644
index 00000000000..1a078ba67b5
--- /dev/null
+++ b/assets/test/collaborative-editor/components/AdaptorIcon.test.tsx
@@ -0,0 +1,87 @@
+/**
+ * Tests for AdaptorIcon component
+ *
+ * Verifies that icon URLs are read from the AdaptorStore (icon_urls.square)
+ * with the existing first-letter placeholder fallback when no URL is present
+ * or the adaptor is not in the store.
+ */
+
+import { render, screen } from '@testing-library/react';
+import { describe, expect, it } from 'vitest';
+
+import { AdaptorIcon } from '../../../js/collaborative-editor/components/AdaptorIcon';
+import {
+ StoreContext,
+ type StoreContextValue,
+} from '../../../js/collaborative-editor/contexts/StoreProvider';
+import { createAdaptorStore } from '../../../js/collaborative-editor/stores/createAdaptorStore';
+import type { Adaptor } from '../../../js/collaborative-editor/types/adaptor';
+
+function renderWithAdaptors(ui: React.ReactElement, adaptors: Adaptor[]) {
+ const adaptorStore = createAdaptorStore();
+ adaptorStore.setAdaptors(adaptors);
+
+ const stores = {
+ adaptorStore,
+ credentialStore: {} as StoreContextValue['credentialStore'],
+ metadataStore: {} as StoreContextValue['metadataStore'],
+ awarenessStore: {} as StoreContextValue['awarenessStore'],
+ workflowStore: {} as StoreContextValue['workflowStore'],
+ sessionContextStore: {} as StoreContextValue['sessionContextStore'],
+ historyStore: {} as StoreContextValue['historyStore'],
+ uiStore: {} as StoreContextValue['uiStore'],
+ editorPreferencesStore: {} as StoreContextValue['editorPreferencesStore'],
+ aiAssistantStore: {} as StoreContextValue['aiAssistantStore'],
+ } satisfies StoreContextValue;
+
+ return render(
+ {ui}
+ );
+}
+
+describe('AdaptorIcon', () => {
+ it('renders icon_urls.square as an when populated in the store', () => {
+ const url = '/adaptor-icons/salesforce/square-abc.png';
+ renderWithAdaptors(
+ ,
+ [
+ {
+ name: '@openfn/language-salesforce',
+ versions: [{ version: '2.0.0' }],
+ repo: 'https://example.com',
+ latest: '2.0.0',
+ icon_urls: { square: url, rectangle: null },
+ },
+ ]
+ );
+
+ const img = screen.getByAltText('salesforce');
+ expect(img.tagName).toBe('IMG');
+ expect(img.getAttribute('src')).toContain(url);
+ });
+
+ it('renders the first-letter placeholder when icon_urls.square is null', () => {
+ renderWithAdaptors(
+ ,
+ [
+ {
+ name: '@openfn/language-salesforce',
+ versions: [{ version: '2.0.0' }],
+ repo: 'https://example.com',
+ latest: '2.0.0',
+ icon_urls: { square: null, rectangle: '/some-rectangle.png' },
+ },
+ ]
+ );
+
+ expect(screen.queryByRole('img')).toBeNull();
+ expect(screen.getByText('S')).toBeInTheDocument();
+ });
+
+ it('renders the first-letter placeholder when the adaptor is not in the store', () => {
+ renderWithAdaptors(, []);
+
+ expect(screen.queryByRole('img')).toBeNull();
+ expect(screen.getByText('H')).toBeInTheDocument();
+ });
+});
diff --git a/assets/test/collaborative-editor/components/AdaptorSelectionModal.test.tsx b/assets/test/collaborative-editor/components/AdaptorSelectionModal.test.tsx
index b01e538bb81..8bc7933967c 100644
--- a/assets/test/collaborative-editor/components/AdaptorSelectionModal.test.tsx
+++ b/assets/test/collaborative-editor/components/AdaptorSelectionModal.test.tsx
@@ -125,7 +125,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
);
@@ -156,7 +156,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
);
@@ -172,7 +172,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
);
@@ -187,7 +187,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={[]}
+ adaptorsInUse={[]}
/>
);
@@ -203,7 +203,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
);
@@ -220,7 +220,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
);
@@ -243,7 +243,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
);
@@ -263,7 +263,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
);
@@ -287,7 +287,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={isOpen}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
@@ -329,7 +329,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
);
@@ -349,7 +349,7 @@ describe('AdaptorSelectionModal', () => {
isOpen={true}
onClose={onClose}
onSelect={onSelect}
- projectAdaptors={mockProjectAdaptors}
+ adaptorsInUse={mockProjectAdaptors}
/>
);
diff --git a/assets/test/collaborative-editor/components/ide/FullScreenIDE.docs-panel.test.tsx b/assets/test/collaborative-editor/components/ide/FullScreenIDE.docs-panel.test.tsx
index 2e8bccb3848..141d0450591 100644
--- a/assets/test/collaborative-editor/components/ide/FullScreenIDE.docs-panel.test.tsx
+++ b/assets/test/collaborative-editor/components/ide/FullScreenIDE.docs-panel.test.tsx
@@ -269,8 +269,8 @@ vi.mock('../../../../js/collaborative-editor/hooks/useCredentials', () => ({
// Mock adaptor hooks
vi.mock('../../../../js/collaborative-editor/hooks/useAdaptors', () => ({
- useProjectAdaptors: () => ({
- projectAdaptors: [],
+ useAdaptorsInUse: () => ({
+ adaptorsInUse: [],
allAdaptors: [],
}),
}));
diff --git a/assets/test/collaborative-editor/components/ide/FullScreenIDE.keyboard.test.tsx b/assets/test/collaborative-editor/components/ide/FullScreenIDE.keyboard.test.tsx
index feb809b882e..5700c39ce19 100644
--- a/assets/test/collaborative-editor/components/ide/FullScreenIDE.keyboard.test.tsx
+++ b/assets/test/collaborative-editor/components/ide/FullScreenIDE.keyboard.test.tsx
@@ -366,8 +366,8 @@ vi.mock('../../../../js/collaborative-editor/hooks/useCredentials', () => ({
// Mock adaptor hooks
vi.mock('../../../../js/collaborative-editor/hooks/useAdaptors', () => ({
- useProjectAdaptors: () => ({
- projectAdaptors: [],
+ useAdaptorsInUse: () => ({
+ adaptorsInUse: [],
allAdaptors: [],
}),
useAdaptors: () => ({
diff --git a/assets/test/collaborative-editor/components/ide/FullScreenIDE.test.tsx b/assets/test/collaborative-editor/components/ide/FullScreenIDE.test.tsx
index b4ad650d57c..1152945f0ca 100644
--- a/assets/test/collaborative-editor/components/ide/FullScreenIDE.test.tsx
+++ b/assets/test/collaborative-editor/components/ide/FullScreenIDE.test.tsx
@@ -317,8 +317,8 @@ vi.mock('../../../../js/collaborative-editor/hooks/useAdaptors', () => ({
loading: false,
error: null,
}),
- useProjectAdaptors: () => ({
- projectAdaptors: [],
+ useAdaptorsInUse: () => ({
+ adaptorsInUse: [],
allAdaptors: [],
}),
useAdaptors: () => [],
diff --git a/assets/test/collaborative-editor/createAdaptorStore.test.ts b/assets/test/collaborative-editor/createAdaptorStore.test.ts
index e976b221e17..5ade4e9c68f 100644
--- a/assets/test/collaborative-editor/createAdaptorStore.test.ts
+++ b/assets/test/collaborative-editor/createAdaptorStore.test.ts
@@ -407,4 +407,103 @@ describe('createAdaptorStore', () => {
expect(notFound).toHaveLength(0);
});
});
+
+ describe('handleAdaptorsReceived merge-by-name', () => {
+ adaptorTest(
+ 'preserves referential identity across identical pushes',
+ async ({ mockChannel, mockProvider }) => {
+ const store = createAdaptorStore();
+ mockChannel.push = createMockChannelPushOk({
+ adaptors: mockAdaptorsList,
+ });
+ store._connectChannel(mockProvider as any);
+ await store.requestAdaptors();
+
+ const selectAdaptors = store.withSelector(state => state.adaptors);
+ const firstRef = selectAdaptors();
+ const firstItems = firstRef.map(a => a);
+
+ const mockChannelWithTest = mockChannel as typeof mockChannel & {
+ _test: { emit: (event: string, message: unknown) => void };
+ };
+ mockChannelWithTest._test.emit('adaptors_updated', mockAdaptorsList);
+
+ await waitForCondition(() => store.getSnapshot().lastUpdated !== null);
+
+ const secondRef = selectAdaptors();
+ expect(secondRef).toBe(firstRef);
+ secondRef.forEach((adaptor, i) => {
+ expect(adaptor).toBe(firstItems[i]);
+ });
+ }
+ );
+
+ adaptorTest(
+ 'replaces only the changed entry when one adaptor mutates',
+ async ({ mockChannel, mockProvider }) => {
+ const store = createAdaptorStore();
+ mockChannel.push = createMockChannelPushOk({
+ adaptors: mockAdaptorsList,
+ });
+ store._connectChannel(mockProvider as any);
+ await store.requestAdaptors();
+
+ const beforeItems = store
+ .getSnapshot()
+ .adaptors.reduce>((acc, a) => {
+ acc[a.name] = a;
+ return acc;
+ }, {});
+
+ const target = mockAdaptorsList[0]!;
+ const mutated = mockAdaptorsList.map(a =>
+ a.name === target.name
+ ? {
+ ...a,
+ versions: [{ version: '99.0.0' }, ...a.versions],
+ latest: '99.0.0',
+ }
+ : a
+ );
+
+ const mockChannelWithTest = mockChannel as typeof mockChannel & {
+ _test: { emit: (event: string, message: unknown) => void };
+ };
+ mockChannelWithTest._test.emit('adaptors_updated', mutated);
+
+ await waitForCondition(
+ () => store.findAdaptorByName(target.name)?.latest === '99.0.0'
+ );
+
+ const after = store.getSnapshot().adaptors;
+ for (const adaptor of after) {
+ if (adaptor.name === target.name) {
+ expect(adaptor).not.toBe(beforeItems[adaptor.name]);
+ } else {
+ expect(adaptor).toBe(beforeItems[adaptor.name]);
+ }
+ }
+ }
+ );
+
+ adaptorTest(
+ 'connectChannel does not request project adaptors',
+ async ({ mockChannel, mockProvider }) => {
+ const pushes: string[] = [];
+ const originalPush = mockChannel.push.bind(mockChannel);
+ mockChannel.push = ((event: string, payload: unknown) => {
+ pushes.push(event);
+ return originalPush(event, payload);
+ }) as typeof mockChannel.push;
+
+ const store = createAdaptorStore();
+ store._connectChannel(mockProvider as any);
+
+ await waitForCondition(() => pushes.includes('request_adaptors'));
+
+ expect(pushes).toContain('request_adaptors');
+ expect(pushes).not.toContain('request_project_adaptors');
+ }
+ );
+ });
});
diff --git a/assets/test/collaborative-editor/fixtures/adaptorData.ts b/assets/test/collaborative-editor/fixtures/adaptorData.ts
index e12adb3e1e5..6405f86281a 100644
--- a/assets/test/collaborative-editor/fixtures/adaptorData.ts
+++ b/assets/test/collaborative-editor/fixtures/adaptorData.ts
@@ -31,6 +31,7 @@ export const mockAdaptor: Adaptor = {
versions: mockAdaptorVersions,
repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/http',
latest: '2.1.0',
+ icon_urls: { square: null, rectangle: null },
};
/**
@@ -41,6 +42,7 @@ export const mockAdaptorDhis2: Adaptor = {
versions: [{ version: '4.2.1' }, { version: '4.2.0' }, { version: '4.1.3' }],
repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/dhis2',
latest: '4.2.1',
+ icon_urls: { square: null, rectangle: null },
};
export const mockAdaptorSalesforce: Adaptor = {
@@ -53,6 +55,7 @@ export const mockAdaptorSalesforce: Adaptor = {
],
repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/salesforce',
latest: '3.5.2',
+ icon_urls: { square: null, rectangle: null },
};
export const mockAdaptorGmail: Adaptor = {
@@ -60,6 +63,7 @@ export const mockAdaptorGmail: Adaptor = {
versions: [{ version: '1.2.0' }, { version: '1.1.0' }, { version: '1.0.0' }],
repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/gmail',
latest: '1.2.0',
+ icon_urls: { square: null, rectangle: null },
};
export const mockAdaptorCommon: Adaptor = {
@@ -71,6 +75,7 @@ export const mockAdaptorCommon: Adaptor = {
],
repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/common',
latest: '2.0.0',
+ icon_urls: { square: null, rectangle: null },
};
/**
@@ -143,6 +148,7 @@ export function createMockAdaptorsList(count: number): AdaptorsList {
versions: [{ version: `${i}.1.0` }, { version: `${i}.0.0` }],
repo: `https://github.com/test/adaptor-${i}`,
latest: `${i}.1.0`,
+ icon_urls: { square: null, rectangle: null },
}));
}
/* eslint-enable @typescript-eslint/restrict-template-expressions */
diff --git a/assets/test/collaborative-editor/useAdaptors.test.tsx b/assets/test/collaborative-editor/useAdaptors.test.tsx
index 8727805a33c..d5a1716a895 100644
--- a/assets/test/collaborative-editor/useAdaptors.test.tsx
+++ b/assets/test/collaborative-editor/useAdaptors.test.tsx
@@ -14,8 +14,8 @@ import {
useAdaptorCommands,
useAdaptors,
useAdaptorsError,
+ useAdaptorsInUse,
useAdaptorsLoading,
- useProjectAdaptors,
} from '../../js/collaborative-editor/hooks/useAdaptors';
import { createSessionStore } from '../../js/collaborative-editor/stores/createSessionStore';
@@ -376,162 +376,129 @@ describe('useAdaptors hooks', () => {
});
});
- describe('useProjectAdaptors', () => {
+ describe('useAdaptorsInUse', () => {
test('requires StoreProvider context', () => {
- expect(() => renderHook(() => useProjectAdaptors())).toThrow(
- 'useProjectAdaptors must be used within a StoreProvider'
+ expect(() => renderHook(() => useAdaptorsInUse())).toThrow(
+ 'useAdaptorsInUse must be used within a StoreProvider'
);
});
- test('returns projectAdaptors, allAdaptors, and isLoading', async () => {
+ test('derives adaptors in use from Y.Doc jobs', async () => {
const { wrapper, stores } = createWrapper();
- const { result } = renderHook(() => useProjectAdaptors(), { wrapper });
+ const { result } = renderHook(() => useAdaptorsInUse(), { wrapper });
- // Initially empty
- expect(result.current.projectAdaptors).toEqual([]);
- expect(result.current.allAdaptors).toEqual([]);
- expect(result.current.isLoading).toBe(false);
-
- // Set all adaptors (simulates backend response)
- const allAdaptors = [mockAdaptor, mockAdaptorGmail, mockAdaptorCommon];
- act(() => {
- stores.adaptorStore.setAdaptors(allAdaptors);
- });
-
- await waitFor(() => {
- expect(result.current.allAdaptors).toHaveLength(3);
- });
- });
-
- test('merges Y.Doc job adaptors into projectAdaptors', async () => {
- const { wrapper, stores } = createWrapper();
- const { result } = renderHook(() => useProjectAdaptors(), { wrapper });
-
- // Set all available adaptors
const allAdaptors = [mockAdaptor, mockAdaptorGmail, mockAdaptorCommon];
act(() => {
stores.adaptorStore.setAdaptors(allAdaptors);
});
- // Add a job to the workflow store that uses Gmail adaptor
act(() => {
stores.workflowStore._setJobsForTesting([
{
id: 'job-1',
- name: 'Test Job',
- adaptor: '@openfn/language-gmail@latest',
- body: 'fn(state => state)',
+ name: 'HTTP Job',
+ adaptor: '@openfn/language-http@2.1.0',
+ body: '',
+ },
+ {
+ id: 'job-2',
+ name: 'Common Job',
+ adaptor: '@openfn/language-common@2.0.0',
+ body: '',
},
]);
});
- // projectAdaptors should now include Gmail from Y.Doc
await waitFor(() => {
- const projectAdaptorNames = result.current.projectAdaptors.map(
- a => a.name
- );
- expect(projectAdaptorNames).toContain('@openfn/language-gmail');
- });
+ expect(result.current.adaptorsInUse).toHaveLength(2);
+ });
+
+ const names = result.current.adaptorsInUse.map(a => a.name);
+ expect(names).toEqual([
+ '@openfn/language-common',
+ '@openfn/language-http',
+ ]);
+
+ // Per-entry referential identity: each item in adaptorsInUse is the same
+ // reference as the corresponding catalogue entry.
+ const catalogue = result.current.allAdaptors;
+ for (const a of result.current.adaptorsInUse) {
+ const fromCatalogue = catalogue.find(c => c.name === a.name);
+ expect(a).toBe(fromCatalogue);
+ }
});
- test('does not duplicate adaptors already in backend projectAdaptors', async () => {
+ test('returns empty list when workflow has no jobs', async () => {
const { wrapper, stores } = createWrapper();
- const { result } = renderHook(() => useProjectAdaptors(), { wrapper });
+ const { result } = renderHook(() => useAdaptorsInUse(), { wrapper });
- // Set all available adaptors and project adaptors (simulates backend where HTTP is already saved)
const allAdaptors = [mockAdaptor, mockAdaptorGmail, mockAdaptorCommon];
act(() => {
stores.adaptorStore.setAdaptors(allAdaptors);
- // Directly set projectAdaptors via internal state modification
- stores.adaptorStore._setProjectAdaptors([mockAdaptor]);
- });
-
- // Add a job that uses HTTP (already in projectAdaptors from backend)
- act(() => {
- stores.workflowStore._setJobsForTesting([
- {
- id: 'job-1',
- name: 'Test Job',
- adaptor: '@openfn/language-http@2.1.0',
- body: 'fn(state => state)',
- },
- ]);
});
await waitFor(() => {
- // Should only have HTTP once, not duplicated
- const httpCount = result.current.projectAdaptors.filter(
- a => a.name === '@openfn/language-http'
- ).length;
- expect(httpCount).toBe(1);
+ expect(result.current.allAdaptors).toHaveLength(3);
});
+
+ expect(result.current.adaptorsInUse).toEqual([]);
});
- test('handles jobs without adaptor field', async () => {
+ test('ignores jobs referencing adaptors absent from the catalogue', async () => {
const { wrapper, stores } = createWrapper();
- const { result } = renderHook(() => useProjectAdaptors(), { wrapper });
+ const { result } = renderHook(() => useAdaptorsInUse(), { wrapper });
- // Set all available adaptors
- const allAdaptors = [mockAdaptor, mockAdaptorGmail];
act(() => {
- stores.adaptorStore.setAdaptors(allAdaptors);
+ stores.adaptorStore.setAdaptors([mockAdaptor]);
});
- // Add a job without an adaptor field (should not crash)
act(() => {
stores.workflowStore._setJobsForTesting([
{
id: 'job-1',
- name: 'Test Job',
- adaptor: undefined as any,
- body: 'fn(state => state)',
+ name: 'Unknown',
+ adaptor: '@openfn/language-unknown@1.0.0',
+ body: '',
+ },
+ {
+ id: 'job-2',
+ name: 'HTTP',
+ adaptor: '@openfn/language-http@2.0.0',
+ body: '',
},
]);
});
- // Should not throw and projectAdaptors should be empty (no backend project adaptors set)
await waitFor(() => {
- expect(result.current.projectAdaptors).toEqual([]);
+ expect(result.current.adaptorsInUse).toHaveLength(1);
});
+
+ expect(result.current.adaptorsInUse[0]?.name).toBe(
+ '@openfn/language-http'
+ );
});
- test('sorts merged projectAdaptors alphabetically', async () => {
+ test('matches version-suffixed adaptor specs against catalogue package names', async () => {
const { wrapper, stores } = createWrapper();
- const { result } = renderHook(() => useProjectAdaptors(), { wrapper });
+ const { result } = renderHook(() => useAdaptorsInUse(), { wrapper });
- // Set all available adaptors
- const allAdaptors = [mockAdaptor, mockAdaptorGmail, mockAdaptorCommon];
act(() => {
- stores.adaptorStore.setAdaptors(allAdaptors);
- // Set HTTP as already in project from backend
- stores.adaptorStore._setProjectAdaptors([mockAdaptor]);
+ stores.adaptorStore.setAdaptors([mockAdaptor]);
});
- // Add jobs using Gmail and Common (not in backend projectAdaptors)
act(() => {
stores.workflowStore._setJobsForTesting([
{
id: 'job-1',
- name: 'Gmail Job',
- adaptor: '@openfn/language-gmail@latest',
- body: '',
- },
- {
- id: 'job-2',
- name: 'Common Job',
- adaptor: '@openfn/language-common@2.0.0',
+ name: 'HTTP',
+ adaptor: '@openfn/language-http@2.0.0',
body: '',
},
]);
});
await waitFor(() => {
- expect(result.current.projectAdaptors).toHaveLength(3);
- const names = result.current.projectAdaptors.map(a => a.name);
- // Should be sorted: common, gmail, http
- expect(names).toEqual([
- '@openfn/language-common',
- '@openfn/language-gmail',
+ expect(result.current.adaptorsInUse.map(a => a.name)).toEqual([
'@openfn/language-http',
]);
});
diff --git a/assets/test/workflow-diagram/components/MiniMapNode.test.tsx b/assets/test/workflow-diagram/components/MiniMapNode.test.tsx
new file mode 100644
index 00000000000..4acdf6453e9
--- /dev/null
+++ b/assets/test/workflow-diagram/components/MiniMapNode.test.tsx
@@ -0,0 +1,120 @@
+/**
+ * MiniMapNode Component Tests
+ *
+ * Verifies that the minimap renders job icons sourced from the AdaptorStore
+ * via useAdaptorIconUrl, with the rect-only placeholder fallback when the
+ * URL is null. Trigger rendering must remain untouched.
+ */
+
+import { render } from '@testing-library/react';
+import type { MiniMapNodeProps } from '@xyflow/react';
+import { describe, expect, test } from 'vitest';
+
+import {
+ StoreContext,
+ type StoreContextValue,
+} from '../../../js/collaborative-editor/contexts/StoreProvider';
+import { createAdaptorStore } from '../../../js/collaborative-editor/stores/createAdaptorStore';
+import type { Adaptor } from '../../../js/collaborative-editor/types/adaptor';
+import MiniMapNode from '../../../js/workflow-diagram/components/MiniMapNode';
+
+type Job = { id: string; adaptor?: string };
+type Trigger = { id: string; type: 'webhook' | 'cron' | 'kafka' };
+
+function renderInSvg(
+ nodeProps: MiniMapNodeProps,
+ jobs: Job[],
+ triggers: Trigger[],
+ adaptors: Adaptor[]
+) {
+ const adaptorStore = createAdaptorStore();
+ adaptorStore.setAdaptors(adaptors);
+
+ const stores = {
+ adaptorStore,
+ credentialStore: {} as StoreContextValue['credentialStore'],
+ metadataStore: {} as StoreContextValue['metadataStore'],
+ awarenessStore: {} as StoreContextValue['awarenessStore'],
+ workflowStore: {} as StoreContextValue['workflowStore'],
+ sessionContextStore: {} as StoreContextValue['sessionContextStore'],
+ historyStore: {} as StoreContextValue['historyStore'],
+ uiStore: {} as StoreContextValue['uiStore'],
+ editorPreferencesStore: {} as StoreContextValue['editorPreferencesStore'],
+ aiAssistantStore: {} as StoreContextValue['aiAssistantStore'],
+ } satisfies StoreContextValue;
+
+ return render(
+
+
+
+ );
+}
+
+describe('MiniMapNode - job icon', () => {
+ const baseNodeProps: MiniMapNodeProps = {
+ x: 0,
+ y: 0,
+ width: 120,
+ height: 120,
+ selected: false,
+ borderRadius: 0,
+ className: '',
+ shapeRendering: 'auto',
+ };
+
+ test('renders an when the adaptor is seeded with icon_urls.square', () => {
+ const url = '/adaptor-icons/http/square-1.png';
+ const { container } = renderInSvg(
+ { ...baseNodeProps, id: 'job-1' },
+ [{ id: 'job-1', adaptor: '@openfn/language-http@1.0.0' }],
+ [],
+ [
+ {
+ name: '@openfn/language-http',
+ versions: [{ version: '1.0.0' }],
+ repo: 'https://example.com',
+ latest: '1.0.0',
+ icon_urls: { square: url, rectangle: null },
+ },
+ ]
+ );
+
+ const image = container.querySelector('image');
+ expect(image).not.toBeNull();
+ expect(image!.getAttribute('href')).toBe(url);
+ });
+
+ test('renders no when icon_urls.square is null', () => {
+ const { container } = renderInSvg(
+ { ...baseNodeProps, id: 'job-1' },
+ [{ id: 'job-1', adaptor: '@openfn/language-http@1.0.0' }],
+ [],
+ [
+ {
+ name: '@openfn/language-http',
+ versions: [{ version: '1.0.0' }],
+ repo: 'https://example.com',
+ latest: '1.0.0',
+ icon_urls: { square: null, rectangle: null },
+ },
+ ]
+ );
+
+ expect(container.querySelector('image')).toBeNull();
+ expect(container.querySelector('rect')).not.toBeNull();
+ });
+
+ test('webhook trigger still renders the GlobeAltIcon (regression guard)', () => {
+ const { container } = renderInSvg(
+ { ...baseNodeProps, id: 'trigger-1' },
+ [],
+ [{ id: 'trigger-1', type: 'webhook' }],
+ []
+ );
+
+ // GlobeAltIcon renders an svg inside a foreignObject for triggers
+ expect(container.querySelector('foreignObject svg')).not.toBeNull();
+ });
+});
diff --git a/assets/test/workflow-diagram/nodes/Job.test.tsx b/assets/test/workflow-diagram/nodes/Job.test.tsx
new file mode 100644
index 00000000000..08eff8edb4c
--- /dev/null
+++ b/assets/test/workflow-diagram/nodes/Job.test.tsx
@@ -0,0 +1,95 @@
+/**
+ * Job Node Component Tests
+ *
+ * Verifies that job nodes read their adaptor icons from the AdaptorStore
+ * via useAdaptorIconUrl, with graceful string-label fallback when the URL
+ * is null OR no StoreProvider is mounted (LiveView workflow-editor path).
+ */
+
+import { render } from '@testing-library/react';
+import { ReactFlowProvider } from '@xyflow/react';
+import { describe, expect, test } from 'vitest';
+
+import {
+ StoreContext,
+ type StoreContextValue,
+} from '../../../js/collaborative-editor/contexts/StoreProvider';
+import { createAdaptorStore } from '../../../js/collaborative-editor/stores/createAdaptorStore';
+import type { Adaptor } from '../../../js/collaborative-editor/types/adaptor';
+import JobNode from '../../../js/workflow-diagram/nodes/Job';
+
+function renderJob(adaptor: string, adaptors: Adaptor[] | null) {
+ const data = { name: 'My Job', adaptor };
+
+ const tree = (
+
+
+
+ );
+
+ if (adaptors === null) {
+ return render(tree);
+ }
+
+ const adaptorStore = createAdaptorStore();
+ adaptorStore.setAdaptors(adaptors);
+
+ const stores = {
+ adaptorStore,
+ credentialStore: {} as StoreContextValue['credentialStore'],
+ metadataStore: {} as StoreContextValue['metadataStore'],
+ awarenessStore: {} as StoreContextValue['awarenessStore'],
+ workflowStore: {} as StoreContextValue['workflowStore'],
+ sessionContextStore: {} as StoreContextValue['sessionContextStore'],
+ historyStore: {} as StoreContextValue['historyStore'],
+ uiStore: {} as StoreContextValue['uiStore'],
+ editorPreferencesStore: {} as StoreContextValue['editorPreferencesStore'],
+ aiAssistantStore: {} as StoreContextValue['aiAssistantStore'],
+ } satisfies StoreContextValue;
+
+ return render(
+ {tree}
+ );
+}
+
+describe('JobNode - adaptor icon', () => {
+ test('renders an with icon_urls.square when the adaptor is seeded', () => {
+ const url = '/adaptor-icons/http/square-deadbeef.png';
+ const { container } = renderJob('@openfn/language-http@1.0.0', [
+ {
+ name: '@openfn/language-http',
+ versions: [{ version: '1.0.0' }],
+ repo: 'https://example.com',
+ latest: '1.0.0',
+ icon_urls: { square: url, rectangle: null },
+ },
+ ]);
+
+ const img = container.querySelector('img');
+ expect(img).not.toBeNull();
+ expect(img?.getAttribute('src')).toContain(url);
+ expect(img?.getAttribute('alt')).toBe('http');
+ });
+
+ test('falls back to the adaptor string label when icon_urls.square is null', () => {
+ const { container } = renderJob('@openfn/language-http@1.0.0', [
+ {
+ name: '@openfn/language-http',
+ versions: [{ version: '1.0.0' }],
+ repo: 'https://example.com',
+ latest: '1.0.0',
+ icon_urls: { square: null, rectangle: '/rect.png' },
+ },
+ ]);
+
+ expect(container.querySelector('img')).toBeNull();
+ expect(container.textContent).toContain('http');
+ });
+
+ test('does not throw and falls back to label when no StoreProvider is mounted', () => {
+ const { container } = renderJob('@openfn/language-http@1.0.0', null);
+
+ expect(container.querySelector('img')).toBeNull();
+ expect(container.textContent).toContain('http');
+ });
+});
diff --git a/config/dev.exs b/config/dev.exs
index ad4124a0307..f86bf15564b 100644
--- a/config/dev.exs
+++ b/config/dev.exs
@@ -131,6 +131,18 @@ config :lightning, :is_resettable_demo, true
config :lightning, :apollo, endpoint: "http://localhost:3000", timeout: 30_000
+# Lightning.Adaptors.NPM upstream URLs — explicit override for clarity in dev.
+# Each key is read by a single sub-module:
+# * registry_url → NPM.Registry (npm search + packument)
+# * jsdelivr_url → NPM.Schema (configuration-schema.json fetch)
+# * github_url → NPM.GitHub (raw icon GETs)
+# * github_ref → NPM.GitHub (git ref under OpenFn/adaptors)
+config :lightning, Lightning.Adaptors.NPM,
+ registry_url: "https://registry.npmjs.org",
+ github_url: "https://raw.githubusercontent.com",
+ github_ref: "main",
+ jsdelivr_url: "https://cdn.jsdelivr.net"
+
config :git_hooks,
# In local dev (with a real .git repo) we auto-install hooks.
# In Docker builds the .git directory is not present (or incomplete),
diff --git a/config/test.exs b/config/test.exs
index 9641f9e9a60..8ce0f8d6b5b 100644
--- a/config/test.exs
+++ b/config/test.exs
@@ -96,6 +96,20 @@ config :lightning, Lightning.Mailer, adapter: Swoosh.Adapters.Test
config :lightning, Lightning.AdaptorRegistry,
use_cache: "test/fixtures/adaptor_registry_cache.json"
+# Phase A Adaptors.Supervisor config for test boot.
+#
+# - `:strategy` — the production `Lightning.Adaptors.Supervisor` mounted in
+# `application.ex` would default to `Lightning.Adaptors.NPM` and try to
+# hit the network on the first Scheduler tick. Replace it with the
+# Mox-backed `StrategyMock` so the application-level supervisor (under
+# the production name `Lightning.Adaptors`) is a no-op for tests that
+# exercise the facade directly.
+# - `:refresh_interval` — `0` disables Scheduler tick scheduling entirely.
+# Per-test isolated supervisors set their own interval as needed.
+config :lightning, Lightning.Adaptors,
+ strategy: Lightning.Adaptors.StrategyMock,
+ refresh_interval: 0
+
config :hammer,
backend:
{Hammer.Backend.ETS,
diff --git a/lib/lightning/adaptors.ex b/lib/lightning/adaptors.ex
new file mode 100644
index 00000000000..fd69d2eb9bb
--- /dev/null
+++ b/lib/lightning/adaptors.ex
@@ -0,0 +1,100 @@
+defmodule Lightning.Adaptors do
+ @moduledoc """
+ Public facade for all adaptor metadata.
+
+ Delegates reads to `Lightning.Adaptors.Store`, refresh calls to
+ `Lightning.Adaptors.Scheduler`, and version resolution to
+ `Lightning.Adaptors.Repo`. No logic lives here.
+
+ All functions come in a dual-arity shape: the zero-/single-arg form
+ passes the compile-time default supervisor name `@sup`; the extra-arity
+ form accepts an explicit supervisor name for test isolation.
+ `resolve_version/2` is the single exception — it has no sup arity because
+ it reads the global Repo directly.
+ """
+
+ alias Lightning.Adaptors.Config
+ alias Lightning.Adaptors.Repo
+ alias Lightning.Adaptors.Scheduler
+ alias Lightning.Adaptors.Store
+ alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor
+
+ @sup Lightning.Adaptors
+
+ @type package_meta :: Store.package_meta()
+ @type version_meta :: Store.version_meta()
+
+ @spec packages() :: {:ok, [package_meta()]} | {:error, :timeout | term()}
+ def packages, do: packages(@sup)
+
+ @spec packages(atom()) :: {:ok, [package_meta()]} | {:error, :timeout | term()}
+ def packages(sup), do: Store.packages(sup)
+
+ @spec versions(String.t()) :: {:ok, [version_meta()]} | {:error, term()}
+ def versions(pkg), do: versions(@sup, pkg)
+
+ @spec versions(atom(), String.t()) ::
+ {:ok, [version_meta()]} | {:error, term()}
+ def versions(sup, pkg), do: Store.versions(sup, pkg)
+
+ @spec schema(String.t()) :: {:ok, String.t()} | {:error, term()}
+ def schema(pkg), do: schema(@sup, pkg)
+
+ @spec schema(atom(), String.t()) :: {:ok, String.t()} | {:error, term()}
+ def schema(sup, pkg), do: Store.schema(sup, pkg)
+
+ @spec icon(String.t(), :square | :rectangle) ::
+ {:ok, Path.t()} | {:error, term()}
+ def icon(pkg, shape), do: icon(@sup, pkg, shape)
+
+ @spec icon(atom(), String.t(), :square | :rectangle) ::
+ {:ok, Path.t()} | {:error, term()}
+ def icon(sup, pkg, shape), do: Store.icon(sup, pkg, shape)
+
+ @spec resolve_version(String.t(), String.t()) ::
+ {:ok, String.t()} | {:error, :not_found}
+ def resolve_version(name, requested) when requested in ["latest", "local"] do
+ case Repo.get_adaptor(name, Config.current_source()) do
+ %{latest_version: v} -> {:ok, v}
+ nil -> {:error, :not_found}
+ end
+ end
+
+ def resolve_version(_name, version), do: {:ok, version}
+
+ @spec refresh_now() :: :ok | {:error, term()}
+ def refresh_now, do: refresh_now(@sup)
+
+ @spec refresh_now(atom()) :: :ok | {:error, term()}
+ def refresh_now(sup),
+ do: Scheduler.refresh_now(AdaptorsSupervisor.global_scheduler_name(sup))
+
+ @spec refresh_package(String.t()) :: :ok | {:error, :not_found | term()}
+ def refresh_package(name) when is_binary(name), do: refresh_package(@sup, name)
+
+ @spec refresh_package(atom(), String.t()) ::
+ :ok | {:error, :not_found | term()}
+ def refresh_package(sup, name) when is_binary(name),
+ do:
+ Scheduler.refresh_package(
+ AdaptorsSupervisor.global_scheduler_name(sup),
+ name
+ )
+
+ @spec refresh_icons() ::
+ {:ok, %{updated: non_neg_integer(), unchanged: non_neg_integer()}}
+ | {:error, term()}
+ def refresh_icons, do: refresh_icons(@sup)
+
+ @spec refresh_icons(atom()) ::
+ {:ok, %{updated: non_neg_integer(), unchanged: non_neg_integer()}}
+ | {:error, term()}
+ def refresh_icons(sup),
+ do: Scheduler.refresh_icons(AdaptorsSupervisor.global_scheduler_name(sup))
+
+ @doc false
+ def icon_meta(name), do: icon_meta(@sup, name)
+
+ @doc false
+ def icon_meta(sup, name), do: Store.icon_meta(sup, name)
+end
diff --git a/lib/lightning/adaptors/channel_broadcaster.ex b/lib/lightning/adaptors/channel_broadcaster.ex
new file mode 100644
index 00000000000..85b598374c0
--- /dev/null
+++ b/lib/lightning/adaptors/channel_broadcaster.ex
@@ -0,0 +1,91 @@
+defmodule Lightning.Adaptors.ChannelBroadcaster do
+ @moduledoc """
+ Burst-coalesced fan-out of adaptor changes to connected sessions.
+
+ Subscribes to `:source_topic` (the cache-coherence topic shared with
+ `Lightning.Adaptors.Invalidator`) and republishes a single pre-rendered
+ envelope to `:client_topic` at most once per 250ms leading-edge window.
+
+ Two-topic separation: the source topic is the cache-coherence audience;
+ the client topic is the display-freshness audience (`WorkflowChannel`
+ subscribers). This bridges them: `Lightning.Adaptors.packages/1` is
+ rendered once per burst and fanned out by PubSub rather than once per
+ session (§6.5c).
+
+ No within-callback fan-out in `:flush` — `Phoenix.PubSub.broadcast/3`
+ is a single call that reaches all subscribers in one hop (§10 #19).
+ """
+
+ use GenServer
+
+ @debounce_ms 250
+
+ @doc """
+ Leading-edge coalesce window in milliseconds.
+
+ Exposed so integration tests can compute receive timeouts off the
+ authoritative value rather than hard-coding a duplicate.
+ """
+ @spec debounce_ms() :: pos_integer()
+ def debounce_ms, do: @debounce_ms
+
+ @doc """
+ Start the ChannelBroadcaster linked to the calling process.
+
+ Required opts:
+ * `:name` — registered GenServer name.
+ * `:source_topic` — PubSub topic to subscribe to (cache-coherence).
+ * `:client_topic` — PubSub topic to broadcast the rendered envelope to.
+ * `:sup` — supervisor instance name; forwarded to
+ `Lightning.Adaptors.packages/1` for per-instance isolation.
+ """
+ @spec start_link(keyword()) :: GenServer.on_start()
+ def start_link(opts) do
+ name = Keyword.fetch!(opts, :name)
+ GenServer.start_link(__MODULE__, opts, name: name)
+ end
+
+ @impl true
+ def init(opts) do
+ :ok =
+ Phoenix.PubSub.subscribe(
+ Lightning.PubSub,
+ Keyword.fetch!(opts, :source_topic)
+ )
+
+ {:ok,
+ %{
+ client_topic: Keyword.fetch!(opts, :client_topic),
+ sup: Keyword.fetch!(opts, :sup),
+ timer: nil
+ }}
+ end
+
+ @impl true
+ # First message of a burst: arm the leading-edge timer.
+ def handle_info({:changed, _name, _source}, %{timer: nil} = state) do
+ timer = Process.send_after(self(), :flush, @debounce_ms)
+ {:noreply, %{state | timer: timer}}
+ end
+
+ # Subsequent messages within the debounce window: drop on the floor.
+ def handle_info({:changed, _name, _source}, state) do
+ {:noreply, state}
+ end
+
+ def handle_info(:flush, %{client_topic: topic, sup: sup} = state) do
+ case Lightning.Adaptors.packages(sup) do
+ {:ok, pkgs} ->
+ Phoenix.PubSub.broadcast(
+ Lightning.PubSub,
+ topic,
+ %{event: "adaptors_updated", payload: %{adaptors: pkgs}}
+ )
+
+ {:error, _} ->
+ :ok
+ end
+
+ {:noreply, %{state | timer: nil}}
+ end
+end
diff --git a/lib/lightning/adaptors/config.ex b/lib/lightning/adaptors/config.ex
new file mode 100644
index 00000000000..8f45fd8cfcf
--- /dev/null
+++ b/lib/lightning/adaptors/config.ex
@@ -0,0 +1,102 @@
+defmodule Lightning.Adaptors.Config do
+ @moduledoc """
+ Stateless runtime configuration for the `Lightning.Adaptors.*` subsystem.
+
+ Every helper is a thin wrapper around `Application.get_env/3`. It is the
+ single runtime source of truth for which strategy is active, how often
+ the scheduler ticks, the per-call cache fetch deadline, the icon cache
+ root, and per-strategy opt blocks.
+
+ ## Application key layout
+
+ Two-tier:
+
+ * `:lightning, Lightning.Adaptors` — subsystem-wide knobs
+ (`:strategy`, `:refresh_interval`, `:cache_timeout_ms`, `:icon_path`).
+ * `:lightning, ` — each strategy owns its own
+ Application key for its own knobs; read via `strategy_opts/1`.
+
+ No GenServer, no ETS, no `:persistent_term` — every call is a fresh
+ `Application.get_env/3`.
+ """
+
+ @parent_key Lightning.Adaptors
+
+ @default_strategy Lightning.Adaptors.NPM
+ @default_refresh_interval :timer.hours(1)
+ @default_cache_timeout_ms 15_000
+ @default_icon_path {:tmp, "lightning/adaptor_icons"}
+
+ @doc """
+ The active strategy module. Defaults to `Lightning.Adaptors.NPM`.
+ """
+ @spec strategy() :: module()
+ def strategy do
+ get(:strategy, @default_strategy)
+ end
+
+ @doc """
+ Atom mapping of `strategy/0`: `:local` for `Lightning.Adaptors.Local`,
+ `:npm` for any other strategy module.
+ """
+ @spec current_source() :: :local | :npm
+ def current_source do
+ case strategy() do
+ Lightning.Adaptors.Local -> :local
+ _other -> :npm
+ end
+ end
+
+ @doc """
+ Scheduler tick interval in milliseconds. Defaults to one hour.
+ """
+ @spec refresh_interval() :: non_neg_integer()
+ def refresh_interval do
+ get(:refresh_interval, @default_refresh_interval)
+ end
+
+ @doc """
+ Per-`Cachex.fetch` courier deadline in milliseconds. Defaults to 15s.
+ """
+ @spec cache_timeout_ms() :: non_neg_integer()
+ def cache_timeout_ms do
+ get(:cache_timeout_ms, @default_cache_timeout_ms)
+ end
+
+ @doc """
+ Resolved filesystem path for the icon cache.
+
+ Accepts either:
+
+ * `{:tmp, suffix}` — resolved against `System.tmp_dir!/0` at call
+ time so the default does not bake a container-specific tmp path
+ into a compiled release.
+ * a plain binary path — returned verbatim.
+
+ Defaults to `{:tmp, "lightning/adaptor_icons"}`.
+ """
+ @spec icon_path() :: Path.t()
+ def icon_path do
+ case get(:icon_path, @default_icon_path) do
+ {:tmp, suffix} -> Path.join(System.tmp_dir!(), suffix)
+ path when is_binary(path) -> path
+ end
+ end
+
+ @doc """
+ Per-strategy keyword opts. Parameterised on the strategy module — each
+ strategy is its own Application key, not nested under the parent.
+ Returns `[]` when the strategy's Application key is unset.
+ """
+ @spec strategy_opts(module()) :: keyword()
+ def strategy_opts(strategy_mod) when is_atom(strategy_mod) do
+ Application.get_env(:lightning, strategy_mod, [])
+ end
+
+ @spec get(atom(), term()) :: term()
+ defp get(key, default) do
+ :lightning
+ |> Application.get_env(@parent_key, [])
+ |> Keyword.get(key, default)
+ end
+end
diff --git a/lib/lightning/adaptors/icon_cache.ex b/lib/lightning/adaptors/icon_cache.ex
new file mode 100644
index 00000000000..22147f68139
--- /dev/null
+++ b/lib/lightning/adaptors/icon_cache.ex
@@ -0,0 +1,96 @@
+defmodule Lightning.Adaptors.IconCache do
+ @moduledoc """
+ Pure filesystem helper owning the on-disk adaptor icon cache.
+
+ Not a GenServer. Three stateless functions over
+ `Lightning.Adaptors.Config.icon_path/0`, which resolves the
+ `{:tmp, suffix}` default at call time.
+
+ Disk layout is **source-partitioned** and **latest-only**:
+
+ ///.
+
+ Source partitioning means flipping `LOCAL_ADAPTORS` between restarts
+ cannot accidentally serve `:npm` bytes from a row that's now resolved
+ via `:local` (or vice versa). Latest-only means a subsequent
+ `write!/5` for the same key overwrites — content-addressable URLs
+ carry the sha8 prefix, so cache invalidation is intrinsic and we
+ don't need to keep old versions on disk.
+
+ Concurrent first-request fetchers are coalesced upstream by Cachex's
+ courier on `{:icon_bytes, source, name, shape}` inside
+ `Lightning.Adaptors.Store.icon/3` — the courier returns `{:ignore, _}`
+ so no entry is committed, but all in-flight peers receive the courier's
+ result for free. The temp-then-rename in `write!/5` is the belt-and-
+ braces guarantee for the file-write step itself: readers never observe
+ a half-written file.
+ """
+
+ alias Lightning.Adaptors.Config
+
+ @type source :: :npm | :local
+ @type name :: String.t()
+ @type shape :: :square | :rectangle
+ @type ext :: String.t()
+
+ @doc """
+ Disk path for an icon. Pure — nothing is checked or created.
+
+ `name` may contain a `/` (scoped npm packages like
+ `@openfn/language-foo`); `Path.join/1` preserves the slash so the
+ scope becomes a real subdirectory.
+ """
+ @spec path(source(), name(), shape(), ext()) :: Path.t()
+ def path(source, name, shape, ext) do
+ Path.join([Config.icon_path(), to_string(source), name, "#{shape}.#{ext}"])
+ end
+
+ @doc """
+ Whether the icon at `path(source, name, shape, ext)` exists on disk.
+ """
+ @spec cached?(source(), name(), shape(), ext()) :: boolean()
+ def cached?(source, name, shape, ext) do
+ File.exists?(path(source, name, shape, ext))
+ end
+
+ @doc """
+ Atomically write `bytes` to `path(source, name, shape, ext)` and
+ return the sha256 of the supplied bytes as a 32-byte binary.
+
+ The write is staged in a sibling temp file and then renamed into
+ place, so concurrent readers never observe a half-written file.
+
+ The caller (Strategy / Scheduler) persists the returned sha on the
+ adaptor row and is responsible for verifying it matches the expected
+ sha from the upstream `adaptor_record` — defence against tarball or
+ filesystem corruption.
+ """
+ @spec write!(source(), name(), shape(), ext(), binary()) ::
+ {:ok, binary()}
+ def write!(source, name, shape, ext, bytes) when is_binary(bytes) do
+ final_path = path(source, name, shape, ext)
+ dir = Path.dirname(final_path)
+ File.mkdir_p!(dir)
+
+ sha = :crypto.hash(:sha256, bytes)
+
+ temp_path =
+ Path.join(dir, ".#{Path.basename(final_path)}.#{random_suffix()}.tmp")
+
+ try do
+ File.write!(temp_path, bytes)
+ File.rename!(temp_path, final_path)
+ rescue
+ e ->
+ _ = File.rm(temp_path)
+ reraise e, __STACKTRACE__
+ end
+
+ {:ok, sha}
+ end
+
+ @spec random_suffix() :: String.t()
+ defp random_suffix do
+ 8 |> :crypto.strong_rand_bytes() |> Base.encode16(case: :lower)
+ end
+end
diff --git a/lib/lightning/adaptors/invalidator.ex b/lib/lightning/adaptors/invalidator.ex
new file mode 100644
index 00000000000..a548c23ee13
--- /dev/null
+++ b/lib/lightning/adaptors/invalidator.ex
@@ -0,0 +1,45 @@
+defmodule Lightning.Adaptors.Invalidator do
+ @moduledoc """
+ Subscribes to cluster adaptor-change broadcasts and evicts matching
+ local Cachex entries, keeping each node coherent with Postgres.
+
+ Subscribes to `opts[:source_topic]` on `Lightning.PubSub` at init.
+ On `{:changed, name, source}`, deletes the four per-adaptor cache keys
+ written by `Lightning.Adaptors.Store`. No source filtering on the hot
+ path — a broadcast for a source that isn't active on this node is a
+ no-op because those keys simply don't exist in Cachex.
+ """
+
+ use GenServer
+
+ @doc """
+ Start the Invalidator linked to the calling process.
+
+ Required opts:
+ * `:name` — registered process name.
+ * `:source_topic` — `Phoenix.PubSub` topic to subscribe to.
+ * `:cache` — Cachex table atom (from `Lightning.Adaptors.Supervisor.cache_name/1`).
+ """
+ @spec start_link(keyword()) :: GenServer.on_start()
+ def start_link(opts) do
+ name = Keyword.fetch!(opts, :name)
+ GenServer.start_link(__MODULE__, opts, name: name)
+ end
+
+ @impl true
+ def init(opts) do
+ topic = Keyword.fetch!(opts, :source_topic)
+ cache = Keyword.fetch!(opts, :cache)
+ :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, topic)
+ {:ok, %{cache: cache}}
+ end
+
+ @impl true
+ def handle_info({:changed, name, source}, state) do
+ Cachex.del(state.cache, {:schema, name, source})
+ Cachex.del(state.cache, {:versions, name, source})
+ Cachex.del(state.cache, {:icon_meta, name, source})
+ Cachex.del(state.cache, {:packages, source})
+ {:noreply, state}
+ end
+end
diff --git a/lib/lightning/adaptors/local.ex b/lib/lightning/adaptors/local.ex
new file mode 100644
index 00000000000..c47a942af6f
--- /dev/null
+++ b/lib/lightning/adaptors/local.ex
@@ -0,0 +1,229 @@
+defmodule Lightning.Adaptors.Local do
+ @moduledoc """
+ Filesystem implementation of `Lightning.Adaptors.Strategy`.
+
+ Serves adaptor metadata, schemas, and icons from an on-disk OpenFn
+ adaptors monorepo checkout. Gated by `LOCAL_ADAPTORS=true` and
+ `OPENFN_ADAPTORS_REPO=/path/to/adaptors` at the runtime-config layer;
+ this module only reads the resolved path via
+ `Lightning.Adaptors.Config.strategy_opts(__MODULE__)[:path]`.
+
+ Each callback walks the filesystem afresh — caching is the Store's
+ responsibility. The module is stateless; no GenServer, no ETS.
+
+ ## Layout
+
+ Walks `$path/packages/*/`, reads each subdirectory's `package.json`
+ for the authoritative `name` and `version`. Directories with missing
+ or unparseable `package.json` are skipped with `Logger.warning` so a
+ malformed entry never crashes boot. Multiple directories sharing the
+ same `name` are collapsed into one record: `latest_version` is the
+ highest semver and `versions` lists every on-disk path.
+
+ `source: :local` is **not** set here — the Store stamps it before
+ upsert. No network calls anywhere in this module.
+ """
+
+ @behaviour Lightning.Adaptors.Strategy
+
+ alias Lightning.Adaptors.Config
+
+ require Logger
+
+ @schema_filename "configuration-schema.json"
+ @icon_exts ~w(png svg)
+
+ @impl Lightning.Adaptors.Strategy
+ def list_adaptors do
+ with {:ok, records} <- discover() do
+ {:ok,
+ Enum.map(records, fn %{name: name, latest_version: v} ->
+ %{name: name, latest_version: v}
+ end)}
+ end
+ end
+
+ @impl Lightning.Adaptors.Strategy
+ def fetch_adaptor(name) when is_binary(name) do
+ with {:ok, records} <- discover() do
+ case Enum.find(records, &(&1.name == name)) do
+ nil -> {:error, :not_found}
+ record -> {:ok, build_adaptor_record(record)}
+ end
+ end
+ end
+
+ @impl Lightning.Adaptors.Strategy
+ def fetch_icon(name, shape)
+ when is_binary(name) and shape in [:square, :rectangle] do
+ with {:ok, records} <- discover() do
+ case Enum.find(records, &(&1.name == name)) do
+ nil -> {:error, :not_found}
+ %{latest_path: path} -> read_icon(path, shape)
+ end
+ end
+ end
+
+ @impl Lightning.Adaptors.Strategy
+ def fetch_icons(_opts \\ []) do
+ with {:ok, records} <- discover() do
+ icons =
+ Enum.reduce(records, %{}, fn record, acc ->
+ Enum.reduce([:square, :rectangle], acc, fn shape, inner ->
+ case read_icon(record.latest_path, shape) do
+ {:ok, %{data: bytes, ext: ext}} ->
+ entry = %{
+ data: bytes,
+ ext: ext,
+ sha256: :crypto.hash(:sha256, bytes)
+ }
+
+ Map.update(
+ inner,
+ record.name,
+ %{shape => entry},
+ &Map.put(&1, shape, entry)
+ )
+
+ {:error, _} ->
+ inner
+ end
+ end)
+ end)
+
+ {:ok, icons}
+ end
+ end
+
+ defp discover do
+ case Config.strategy_opts(__MODULE__)[:path] do
+ nil ->
+ Logger.warning(
+ "Lightning.Adaptors.Local: :path is not configured " <>
+ "(set OPENFN_ADAPTORS_REPO or :lightning, Lightning.Adaptors.Local, path:)"
+ )
+
+ {:error, :no_repo_path}
+
+ path ->
+ records =
+ path
+ |> Path.join("packages")
+ |> Path.join("*")
+ |> Path.wildcard()
+ |> Enum.filter(&File.dir?/1)
+ |> Enum.flat_map(&read_package_dir/1)
+ |> group_by_name()
+
+ {:ok, records}
+ end
+ end
+
+ defp read_package_dir(dir) do
+ pkg_json_path = Path.join(dir, "package.json")
+
+ with {:ok, body} <- File.read(pkg_json_path),
+ {:ok, %{"name" => name, "version" => version} = parsed}
+ when is_binary(name) and is_binary(version) <- Jason.decode(body) do
+ [%{name: name, version: version, path: dir, package_json: parsed}]
+ else
+ other ->
+ Logger.warning(
+ "Lightning.Adaptors.Local: skipping #{inspect(dir)}: " <>
+ "missing or unparseable package.json (#{inspect(other)})"
+ )
+
+ []
+ end
+ end
+
+ defp group_by_name(entries) do
+ entries
+ |> Enum.group_by(& &1.name)
+ |> Enum.map(fn {name, versions} ->
+ sorted = Enum.sort_by(versions, & &1.version, &version_descending/2)
+ latest = List.first(sorted)
+
+ %{
+ name: name,
+ latest_version: latest.version,
+ latest_path: latest.path,
+ latest_package_json: latest.package_json,
+ versions: sorted
+ }
+ end)
+ end
+
+ defp version_descending(a, b) do
+ case {Version.parse(a), Version.parse(b)} do
+ {{:ok, va}, {:ok, vb}} -> Version.compare(va, vb) != :lt
+ _ -> a >= b
+ end
+ end
+
+ defp build_adaptor_record(record) do
+ pkg = record.latest_package_json
+ {schema_data, schema_sha256} = read_schema(record.latest_path)
+
+ %{
+ name: record.name,
+ description: pkg["description"],
+ homepage: pkg["homepage"],
+ repository: extract_repository(pkg["repository"]),
+ license: pkg["license"],
+ latest_version: record.latest_version,
+ deprecated: false,
+ schema_data: schema_data,
+ schema_sha256: schema_sha256,
+ versions: Enum.map(record.versions, &build_version_record/1)
+ }
+ end
+
+ defp build_version_record(%{version: v, package_json: pkg}) do
+ %{
+ version: v,
+ integrity: nil,
+ tarball_url: nil,
+ size_bytes: nil,
+ dependencies: Map.get(pkg, "dependencies", %{}),
+ peer_dependencies: Map.get(pkg, "peerDependencies", %{}),
+ published_at: nil,
+ deprecated: false
+ }
+ end
+
+ defp read_schema(dir) do
+ case File.read(Path.join(dir, @schema_filename)) do
+ {:ok, body} ->
+ # Validate JSON, but keep the raw binary so credential-form
+ # rendering can re-engage ordered_objects decoding downstream.
+ case Jason.decode(body) do
+ {:ok, _data} ->
+ sha = :sha256 |> :crypto.hash(body) |> Base.encode16(case: :lower)
+ {body, sha}
+
+ {:error, _} ->
+ {nil, nil}
+ end
+
+ {:error, _} ->
+ {nil, nil}
+ end
+ end
+
+ defp read_icon(dir, shape) do
+ Enum.find_value(@icon_exts, {:error, :not_found}, fn ext ->
+ case File.read(icon_path(dir, shape, ext)) do
+ {:ok, bytes} -> {:ok, %{data: bytes, ext: ext}}
+ {:error, _} -> nil
+ end
+ end)
+ end
+
+ defp icon_path(dir, shape, ext),
+ do: Path.join([dir, "assets", "#{shape}.#{ext}"])
+
+ defp extract_repository(repo) when is_binary(repo), do: repo
+ defp extract_repository(%{"url" => url}) when is_binary(url), do: url
+ defp extract_repository(_), do: nil
+end
diff --git a/lib/lightning/adaptors/node_monitor.ex b/lib/lightning/adaptors/node_monitor.ex
new file mode 100644
index 00000000000..3ecc0d3ab8f
--- /dev/null
+++ b/lib/lightning/adaptors/node_monitor.ex
@@ -0,0 +1,48 @@
+defmodule Lightning.Adaptors.NodeMonitor do
+ @moduledoc """
+ Partition-recovery companion to `Lightning.Adaptors.Invalidator`.
+
+ On `:nodeup`, re-warms the Cachex table from Postgres so a reconnecting
+ peer never serves stale data until the 24-hour TTL expires. Steady-state
+ invalidation belongs to `Lightning.Adaptors.Invalidator`.
+
+ `:nodedown` is a deliberate no-op. The worst case on a silent departure is
+ one stale-URL redirect per client, backstopped by 302-on-stale-sha.
+ """
+
+ use GenServer
+
+ alias Lightning.Adaptors.Store
+
+ @doc """
+ Start a NodeMonitor for the given supervisor instance.
+
+ Required opts:
+ * `:name` — registered GenServer name (§6.11 async-test rule).
+ * `:sup` — supervisor instance name, forwarded to `Store.warm_from_repo/1`.
+ """
+ @spec start_link(keyword()) :: GenServer.on_start()
+ def start_link(opts) do
+ name = Keyword.fetch!(opts, :name)
+ GenServer.start_link(__MODULE__, opts, name: name)
+ end
+
+ @impl true
+ def init(opts) do
+ sup = Keyword.fetch!(opts, :sup)
+ :net_kernel.monitor_nodes(true, node_type: :visible)
+ {:ok, %{sup: sup}}
+ end
+
+ @impl true
+ def handle_info({:nodeup, _node, _info}, state) do
+ Store.warm_from_repo(state.sup)
+ {:noreply, state}
+ end
+
+ # Deliberate no-op: nodedown does not trigger a re-warm. The 24h Cachex TTL
+ # backstops any staleness; 302-on-stale-sha handles already-issued URLs.
+ def handle_info({:nodedown, _node, _info}, state) do
+ {:noreply, state}
+ end
+end
diff --git a/lib/lightning/adaptors/npm.ex b/lib/lightning/adaptors/npm.ex
new file mode 100644
index 00000000000..d5ed5e44c29
--- /dev/null
+++ b/lib/lightning/adaptors/npm.ex
@@ -0,0 +1,108 @@
+defmodule Lightning.Adaptors.NPM do
+ @moduledoc """
+ Production implementation of `Lightning.Adaptors.Strategy` that talks
+ to the public NPM registry and the OpenFn adaptors monorepo on GitHub.
+
+ Consolidates the legacy `Lightning.AdaptorRegistry`,
+ `Mix.Tasks.Lightning.InstallSchemas`, and
+ `Mix.Tasks.Lightning.InstallAdaptorIcons` into one stateless module:
+
+ * `c:list_adaptors/0` — single search-API call returning
+ `name + latest_version` for every `@openfn/language-*` package.
+ * `c:fetch_adaptor/1` — packument fetch + per-version decode and
+ latest-version schema retrieval via jsDelivr. Icon fields are
+ **not** stamped here; the Scheduler joins them on after a bulk
+ `c:fetch_icons/1` pass.
+ * `c:fetch_icon/2` — single icon raw GET against
+ `raw.githubusercontent.com`, used by the Store's rare lazy-miss
+ fallback.
+ * `c:fetch_icons/1` — bulk fan-out over the search listing, one
+ HTTP request per `(name, shape)`. Threads `:prior_etags` from
+ the caller down into the per-request `If-None-Match` headers.
+
+ ## HTTP
+
+ This module is a thin orchestrator. The actual HTTP work is delegated
+ to three sub-modules, each of which owns its own Tesla client and
+ upstream base URL:
+
+ * `Lightning.Adaptors.NPM.Registry` — npm registry search + packument.
+ * `Lightning.Adaptors.NPM.Schema` — jsDelivr `configuration-schema.json`.
+ * `Lightning.Adaptors.NPM.GitHub` — `raw.githubusercontent.com`
+ icon fetches (one GET per `(name, shape)`).
+
+ Each sub-module issues at most a handful of single-shot Tesla requests
+ bounded by `http_timeout`. No retry, no backoff, no circuit-breaker —
+ transient failures (5xx, timeout, nxdomain) of the *primary* request
+ (`packument` for `fetch_adaptor/1`, `search` for `list_adaptors/0` and
+ `fetch_icons/1`) surface as `{:error, term()}` unchanged. Schema and
+ icon fetches inside `fetch_adaptor/1` and `fetch_icons/1` are
+ best-effort: a single icon miss degrades that entry to absence rather
+ than failing the whole record.
+
+ ## Configuration
+
+ Each sub-module reads `:registry_url`, `:jsdelivr_url`, `:github_url`,
+ `:github_ref`, and `:http_timeout` via
+ `Lightning.Adaptors.Config.strategy_opts(__MODULE__)`, with defaults
+ baked in so the module works even when no Application env block is
+ set.
+ """
+
+ @behaviour Lightning.Adaptors.Strategy
+
+ alias Lightning.Adaptors.NPM.GitHub
+ alias Lightning.Adaptors.NPM.Registry
+ alias Lightning.Adaptors.NPM.Schema
+
+ @impl Lightning.Adaptors.Strategy
+ def list_adaptors, do: Registry.list_adaptors()
+
+ @impl Lightning.Adaptors.Strategy
+ def fetch_adaptor(name) when is_binary(name) do
+ with {:ok, packument} <- Registry.get_packument(name),
+ {:ok, latest_version} <- Registry.latest_version(packument) do
+ {schema_data, schema_sha} = Schema.schema(name, latest_version)
+
+ {:ok,
+ %{
+ name: Map.get(packument, "name", name),
+ description: Map.get(packument, "description"),
+ homepage: Map.get(packument, "homepage"),
+ repository: Registry.repository_url(Map.get(packument, "repository")),
+ license: Map.get(packument, "license"),
+ latest_version: latest_version,
+ deprecated: Registry.deprecated?(packument, latest_version),
+ schema_data: encode_schema(schema_data),
+ schema_sha256: schema_sha,
+ versions: Registry.build_versions(packument)
+ }}
+ end
+ end
+
+ # Strategy boundary: re-encode the decoded schema map to a JSON binary
+ # so the row is persisted as text and `Jason.decode!(_,
+ # objects: :ordered_objects)` re-engages downstream. NPM's upstream
+ # Schema sub-module already decoded into a regular map, so field order
+ # is whatever map iteration yields — the round-trip preserves it for
+ # the Local strategy (raw binary in) and is a no-op for NPM data.
+ defp encode_schema(nil), do: nil
+ defp encode_schema(data) when is_binary(data), do: data
+ defp encode_schema(data) when is_map(data), do: Jason.encode!(data)
+
+ @impl Lightning.Adaptors.Strategy
+ def fetch_icon(name, shape)
+ when is_binary(name) and shape in [:square, :rectangle] do
+ GitHub.fetch_one(name, shape)
+ end
+
+ @impl Lightning.Adaptors.Strategy
+ def fetch_icons(opts \\ []) when is_list(opts) do
+ prior_etags = Keyword.get(opts, :prior_etags, %{})
+
+ with {:ok, listing} <- Registry.list_adaptors() do
+ names = Enum.map(listing, & &1.name)
+ GitHub.fetch_all(names, prior_etags)
+ end
+ end
+end
diff --git a/lib/lightning/adaptors/npm/github.ex b/lib/lightning/adaptors/npm/github.ex
new file mode 100644
index 00000000000..1e2a8096cd4
--- /dev/null
+++ b/lib/lightning/adaptors/npm/github.ex
@@ -0,0 +1,302 @@
+defmodule Lightning.Adaptors.NPM.GitHub do
+ @moduledoc """
+ Raw `raw.githubusercontent.com` client for adaptor icons.
+
+ Icons aren't published inside npm tarballs — they live in the
+ `OpenFn/adaptors` monorepo. This module fetches them directly via the
+ raw GitHub content host, one icon per HTTP GET, no tarball walking.
+
+ ## URL pattern
+
+ /OpenFn/adaptors//packages//assets/.
+
+ where `` strips the `@openfn/` scope from the package name.
+ Each `(name, shape)` is probed `png` first then `svg` — matching the
+ ext order used by `Lightning.Adaptors.Local`.
+
+ ## Configuration
+
+ Both `:github_url` (default `https://raw.githubusercontent.com`) and
+ `:github_ref` (default `main`) are read via
+ `Lightning.Adaptors.Config.strategy_opts(Lightning.Adaptors.NPM)`,
+ symmetric with the existing `:registry_url`, `:jsdelivr_url`, and
+ `:http_timeout` keys.
+ """
+
+ alias Lightning.Adaptors.Config
+
+ require Logger
+
+ @default_github_url "https://raw.githubusercontent.com"
+ @default_github_ref "main"
+ @default_http_timeout :timer.seconds(30)
+
+ @default_max_concurrency 20
+
+ @icon_exts ~w(png svg)
+ @scope_prefix "@openfn/"
+ @language_prefix "language-"
+
+ @doc """
+ Fetch a single icon for `(name, shape)`.
+
+ Tries `png` then `svg`. No conditional GET — this entry point is used
+ by the Store's lazy-miss fallback, where no prior etag is in scope.
+ Returns:
+
+ * `{:ok, %{data: binary(), ext: String.t(), etag: String.t() | nil}}`
+ on success.
+ * `{:error, :not_found}` when neither ext yields a 200.
+ * `{:error, term()}` on transport-level failure (timeout, nxdomain).
+ """
+ @spec fetch_one(String.t(), :square | :rectangle) ::
+ {:ok, %{data: binary(), ext: String.t(), etag: String.t() | nil}}
+ | {:error, :not_found | term()}
+ def fetch_one(name, shape)
+ when is_binary(name) and shape in [:square, :rectangle] do
+ client = raw_client()
+ do_fetch_one(client, name, shape, nil)
+ end
+
+ @doc """
+ Fetch icons for every `(name, shape)` pair across `names`.
+
+ Returns `{:ok, partial_map}` where each entry is keyed by the
+ package name and contains zero, one, or two shape keys. Absence
+ is **not** an error — packages with no upstream icon simply do not
+ appear (or appear with a missing shape).
+
+ When `prior_etags` is supplied as `%{name => %{shape => etag}}`, the
+ corresponding `If-None-Match` header is sent per `(name, shape)`. A
+ 304 response is surfaced as a `:not_modified` sentinel in the
+ per-shape slot — distinct from "absent" which means upstream had no
+ such shape at all.
+
+ Fans out via `Task.async_stream` with a bounded concurrency. Transport
+ failures for a single `(name, shape)` are dropped silently — the whole
+ pipeline only fails if every fetch crashes the supervisor, which is
+ not surfaced here.
+ """
+ @spec fetch_all([String.t()], %{
+ optional(String.t()) => %{
+ optional(:square) => String.t(),
+ optional(:rectangle) => String.t()
+ }
+ }) ::
+ {:ok,
+ %{
+ required(String.t()) => %{
+ optional(:square) =>
+ %{
+ data: binary(),
+ ext: String.t(),
+ sha256: binary(),
+ etag: String.t() | nil
+ }
+ | :not_modified,
+ optional(:rectangle) =>
+ %{
+ data: binary(),
+ ext: String.t(),
+ sha256: binary(),
+ etag: String.t() | nil
+ }
+ | :not_modified
+ }
+ }}
+ def fetch_all(names, prior_etags)
+ when is_list(names) and is_map(prior_etags) do
+ client = raw_client()
+
+ work =
+ for name <- names, shape <- [:square, :rectangle], do: {name, shape}
+
+ {results, counts} =
+ work
+ |> Task.async_stream(
+ fn {name, shape} ->
+ prior = prior_etag_for(prior_etags, name, shape)
+
+ case do_fetch_one(client, name, shape, prior) do
+ {:ok, %{data: bytes, ext: ext, etag: etag}} ->
+ {name, shape,
+ {:ok,
+ %{
+ data: bytes,
+ ext: ext,
+ sha256: :crypto.hash(:sha256, bytes),
+ etag: etag
+ }}}
+
+ :not_modified ->
+ {name, shape, :not_modified}
+
+ {:error, reason} ->
+ {name, shape, {:error, reason}}
+ end
+ end,
+ max_concurrency: max_concurrency(),
+ timeout: max(http_timeout() * 2, 5_000),
+ on_timeout: :kill_task,
+ ordered: false
+ )
+ |> Enum.reduce(
+ {%{}, %{fetched: 0, not_modified: 0, not_found: 0, error: 0}},
+ fn
+ {:ok, {name, shape, {:ok, entry}}}, {acc, c} ->
+ {put_entry(acc, name, shape, entry),
+ Map.update!(c, :fetched, &(&1 + 1))}
+
+ {:ok, {name, shape, :not_modified}}, {acc, c} ->
+ {put_entry(acc, name, shape, :not_modified),
+ Map.update!(c, :not_modified, &(&1 + 1))}
+
+ {:ok, {_name, _shape, {:error, :not_found}}}, {acc, c} ->
+ {acc, Map.update!(c, :not_found, &(&1 + 1))}
+
+ {:ok, {_name, _shape, {:error, _reason}}}, {acc, c} ->
+ {acc, Map.update!(c, :error, &(&1 + 1))}
+
+ {:exit, _reason}, {acc, c} ->
+ {acc, Map.update!(c, :error, &(&1 + 1))}
+ end
+ )
+
+ Logger.info(
+ "NPM.GitHub: fetch_all names=#{length(names)} pairs=#{length(work)} " <>
+ "fetched=#{counts.fetched} not_modified=#{counts.not_modified} " <>
+ "not_found=#{counts.not_found} errors=#{counts.error}"
+ )
+
+ {:ok, results}
+ end
+
+ defp prior_etag_for(prior_etags, name, shape) do
+ case Map.get(prior_etags, name) do
+ %{} = shapes -> Map.get(shapes, shape)
+ _ -> nil
+ end
+ end
+
+ defp put_entry(acc, name, shape, entry) do
+ Map.update(acc, name, %{shape => entry}, &Map.put(&1, shape, entry))
+ end
+
+ # GitHub raw does not gzip-compress PNG/SVG bodies, so no
+ # `Tesla.Middleware.DecompressResponse` is in play. The sha256 computed
+ # in `fetch_all/2` is therefore over the raw response bytes (which equals
+ # the decompressed bytes in our case — there is no compression layer to
+ # speak of, and no `accept-encoding` middleware is configured). If GitHub
+ # ever starts compressing 200 bodies, Tesla/Finch would need an
+ # accept-encoding/decompress middleware to preserve this invariant.
+ defp do_fetch_one(client, name, shape, prior_etag) do
+ suffix = strip_scope(name)
+ cond_get? = is_binary(prior_etag)
+
+ headers =
+ if cond_get?, do: [{"if-none-match", prior_etag}], else: []
+
+ Enum.reduce_while(@icon_exts, {:error, :not_found}, fn ext, _acc ->
+ path = build_path(suffix, shape, ext)
+
+ case Tesla.get(client, path, headers: headers) do
+ {:ok, %Tesla.Env{status: 200, body: body} = env} when is_binary(body) ->
+ etag = response_etag(env)
+
+ Logger.debug(fn ->
+ "NPM.GitHub: GET #{path} → 200 (#{byte_size(body)}B) " <>
+ "name=#{name} shape=#{shape} cond_get=#{cond_get?}"
+ end)
+
+ {:halt, {:ok, %{data: body, ext: ext, etag: etag}}}
+
+ {:ok, %Tesla.Env{status: 304}} ->
+ Logger.debug(fn ->
+ "NPM.GitHub: GET #{path} → 304 name=#{name} shape=#{shape} " <>
+ "cond_get=#{cond_get?}"
+ end)
+
+ {:halt, :not_modified}
+
+ {:ok, %Tesla.Env{status: 404}} ->
+ Logger.debug(fn ->
+ "NPM.GitHub: GET #{path} → 404 name=#{name} shape=#{shape} " <>
+ "cond_get=#{cond_get?}"
+ end)
+
+ {:cont, {:error, :not_found}}
+
+ {:ok, %Tesla.Env{status: status}} ->
+ Logger.debug(fn ->
+ "NPM.GitHub: GET #{path} → #{status} name=#{name} shape=#{shape} " <>
+ "cond_get=#{cond_get?}"
+ end)
+
+ {:halt, {:error, {:http_status, status}}}
+
+ {:error, reason} ->
+ Logger.debug(fn ->
+ "NPM.GitHub: GET #{path} → transport error #{inspect(reason)} " <>
+ "name=#{name} shape=#{shape} cond_get=#{cond_get?}"
+ end)
+
+ {:halt, {:error, reason}}
+ end
+ end)
+ end
+
+ defp response_etag(%Tesla.Env{headers: headers}) do
+ Enum.find_value(headers, fn {k, v} ->
+ if is_binary(k) and String.downcase(k) == "etag", do: v
+ end)
+ end
+
+ defp build_path(name_suffix, shape, ext) do
+ "/OpenFn/adaptors/#{github_ref()}/packages/#{name_suffix}/assets/#{shape}.#{ext}"
+ end
+
+ defp strip_scope(@scope_prefix <> @language_prefix <> rest), do: rest
+ defp strip_scope(@scope_prefix <> rest), do: rest
+ defp strip_scope(name), do: name
+
+ defp raw_client do
+ build_client([
+ {Tesla.Middleware.BaseUrl, github_url()},
+ Tesla.Middleware.FollowRedirects
+ ])
+ end
+
+ defp build_client(middleware) do
+ case Application.get_env(:tesla, :adapter) do
+ {Tesla.Adapter.Finch, _opts} ->
+ Tesla.client(
+ middleware,
+ {Tesla.Adapter.Finch,
+ name: Lightning.Finch, receive_timeout: http_timeout()}
+ )
+
+ _other ->
+ Tesla.client(middleware)
+ end
+ end
+
+ defp github_url do
+ Config.strategy_opts(Lightning.Adaptors.NPM)[:github_url] ||
+ @default_github_url
+ end
+
+ defp github_ref do
+ Config.strategy_opts(Lightning.Adaptors.NPM)[:github_ref] ||
+ @default_github_ref
+ end
+
+ defp http_timeout do
+ Config.strategy_opts(Lightning.Adaptors.NPM)[:http_timeout] ||
+ @default_http_timeout
+ end
+
+ defp max_concurrency do
+ Config.strategy_opts(Lightning.Adaptors.NPM)[:icon_max_concurrency] ||
+ @default_max_concurrency
+ end
+end
diff --git a/lib/lightning/adaptors/npm/registry.ex b/lib/lightning/adaptors/npm/registry.ex
new file mode 100644
index 00000000000..a05e93dfb02
--- /dev/null
+++ b/lib/lightning/adaptors/npm/registry.ex
@@ -0,0 +1,195 @@
+defmodule Lightning.Adaptors.NPM.Registry do
+ @moduledoc """
+ NPM registry HTTP client for `Lightning.Adaptors.NPM`.
+
+ Talks to `registry.npmjs.org`. Responsible for the `search` endpoint
+ used by `c:Lightning.Adaptors.Strategy.list_adaptors/0` and the
+ `packument` endpoint used by `fetch_adaptor/1` and `fetch_icon/2`.
+
+ Base URL via `Lightning.Adaptors.Config.strategy_opts(Lightning.Adaptors.NPM)[:registry_url]`,
+ default `https://registry.npmjs.org`.
+
+ Search results are filtered down to `@openfn/language-*` packages,
+ matching the legacy `AdaptorRegistry` semantics; non-language packages
+ in the `@openfn/` scope (e.g. `@openfn/cli`) are rejected.
+ """
+
+ alias Lightning.Adaptors.Config
+
+ @default_registry_url "https://registry.npmjs.org"
+ @default_http_timeout :timer.seconds(30)
+
+ @search_scope "openfn"
+ @search_size 250
+
+ @language_prefix "@openfn/language-"
+
+ @doc """
+ Single `/-/v1/search` call returning `name + latest_version` for every
+ `@openfn/language-*` package.
+ """
+ @spec list_adaptors() ::
+ {:ok, [%{name: String.t(), latest_version: String.t()}]}
+ | {:error, term()}
+ def list_adaptors do
+ case Tesla.get(json_client(), "/-/v1/search",
+ query: [text: "@" <> @search_scope, size: @search_size]
+ ) do
+ {:ok, %Tesla.Env{status: 200, body: body}} when is_map(body) ->
+ listing =
+ body
+ |> Map.get("objects", [])
+ |> Enum.map(&extract_listing_entry/1)
+ |> Enum.reject(&is_nil/1)
+
+ {:ok, listing}
+
+ {:ok, %Tesla.Env{status: status}} ->
+ {:error, {:http_status, status}}
+
+ {:error, reason} ->
+ {:error, reason}
+ end
+ end
+
+ @doc """
+ Fetch the full packument for a package.
+ """
+ @spec get_packument(String.t()) ::
+ {:ok, map()} | {:error, :not_found} | {:error, term()}
+ def get_packument(name) do
+ case Tesla.get(json_client(), "/" <> name) do
+ {:ok, %Tesla.Env{status: 200, body: body}} when is_map(body) ->
+ {:ok, body}
+
+ {:ok, %Tesla.Env{status: 404}} ->
+ {:error, :not_found}
+
+ {:ok, %Tesla.Env{status: status}} ->
+ {:error, {:http_status, status}}
+
+ {:error, reason} ->
+ {:error, reason}
+ end
+ end
+
+ @doc """
+ Extract the `dist-tags.latest` version from a packument.
+ """
+ @spec latest_version(map()) ::
+ {:ok, String.t()} | {:error, :no_latest_version}
+ def latest_version(packument) do
+ case get_in(packument, ["dist-tags", "latest"]) do
+ v when is_binary(v) -> {:ok, v}
+ _ -> {:error, :no_latest_version}
+ end
+ end
+
+ @doc """
+ Resolve the tarball URL for `version` in `packument`.
+ """
+ @spec require_tarball_url(map(), String.t()) ::
+ {:ok, String.t()} | {:error, :no_tarball_url}
+ def require_tarball_url(packument, version) do
+ case get_in(packument, ["versions", version, "dist", "tarball"]) do
+ url when is_binary(url) -> {:ok, url}
+ _ -> {:error, :no_tarball_url}
+ end
+ end
+
+ @doc """
+ Build the per-version `version_record` list from a packument.
+ """
+ @spec build_versions(map()) :: [map()]
+ def build_versions(packument) do
+ versions = Map.get(packument, "versions", %{})
+ times = Map.get(packument, "time", %{})
+
+ Enum.map(versions, fn {version, info} ->
+ %{
+ version: version,
+ integrity: get_in(info, ["dist", "integrity"]),
+ tarball_url: get_in(info, ["dist", "tarball"]),
+ size_bytes: get_in(info, ["dist", "unpackedSize"]),
+ dependencies: Map.get(info, "dependencies", %{}),
+ peer_dependencies: Map.get(info, "peerDependencies", %{}),
+ published_at: parse_time(Map.get(times, version)),
+ deprecated: deprecated_marker?(info)
+ }
+ end)
+ end
+
+ @doc """
+ Is the given `version` in `packument` flagged as deprecated?
+ """
+ @spec deprecated?(map(), String.t()) :: boolean()
+ def deprecated?(packument, version) do
+ deprecated_marker?(get_in(packument, ["versions", version]) || %{})
+ end
+
+ @doc """
+ Normalise the packument's `repository` field to a plain URL string.
+ """
+ @spec repository_url(term()) :: String.t() | nil
+ def repository_url(%{"url" => url}) when is_binary(url), do: url
+ def repository_url(url) when is_binary(url), do: url
+ def repository_url(_), do: nil
+
+ defp extract_listing_entry(%{
+ "package" => %{"name" => name, "version" => version}
+ })
+ when is_binary(name) and is_binary(version) do
+ if String.starts_with?(name, @language_prefix) do
+ %{name: name, latest_version: version}
+ end
+ end
+
+ defp extract_listing_entry(_), do: nil
+
+ defp deprecated_marker?(%{"deprecated" => v}) when is_binary(v) and v != "",
+ do: true
+
+ defp deprecated_marker?(%{"deprecated" => true}), do: true
+ defp deprecated_marker?(_), do: false
+
+ defp parse_time(time) when is_binary(time) do
+ case DateTime.from_iso8601(time) do
+ {:ok, dt, _offset} -> dt
+ _ -> nil
+ end
+ end
+
+ defp parse_time(_), do: nil
+
+ defp json_client do
+ build_client([
+ {Tesla.Middleware.BaseUrl, registry_url()},
+ Tesla.Middleware.JSON,
+ Tesla.Middleware.FollowRedirects
+ ])
+ end
+
+ defp build_client(middleware) do
+ case Application.get_env(:tesla, :adapter) do
+ {Tesla.Adapter.Finch, _opts} ->
+ Tesla.client(
+ middleware,
+ {Tesla.Adapter.Finch,
+ name: Lightning.Finch, receive_timeout: http_timeout()}
+ )
+
+ _other ->
+ Tesla.client(middleware)
+ end
+ end
+
+ defp registry_url do
+ Config.strategy_opts(Lightning.Adaptors.NPM)[:registry_url] ||
+ @default_registry_url
+ end
+
+ defp http_timeout do
+ Config.strategy_opts(Lightning.Adaptors.NPM)[:http_timeout] ||
+ @default_http_timeout
+ end
+end
diff --git a/lib/lightning/adaptors/npm/schema.ex b/lib/lightning/adaptors/npm/schema.ex
new file mode 100644
index 00000000000..8975888e559
--- /dev/null
+++ b/lib/lightning/adaptors/npm/schema.ex
@@ -0,0 +1,83 @@
+defmodule Lightning.Adaptors.NPM.Schema do
+ @moduledoc """
+ jsDelivr CDN client for adaptor configuration schemas.
+
+ Fetches `/npm/@/configuration-schema.json` from
+ `cdn.jsdelivr.net`, decodes it, and returns
+ `{schema_data, schema_sha256}` (or `{nil, nil}` on any failure —
+ schema fetch is best-effort).
+
+ Base URL via `Lightning.Adaptors.Config.strategy_opts(Lightning.Adaptors.NPM)[:jsdelivr_url]`,
+ default `https://cdn.jsdelivr.net`.
+ """
+
+ alias Lightning.Adaptors.Config
+
+ @default_jsdelivr_url "https://cdn.jsdelivr.net"
+ @default_http_timeout :timer.seconds(30)
+
+ @doc """
+ Fetch the configuration schema for `name@version` from jsDelivr.
+
+ Returns `{schema_data, schema_sha256}` on success, `{nil, nil}` on
+ any failure (best-effort — schema absence must not fail the adaptor
+ record assembly).
+ """
+ @spec schema(String.t(), String.t()) ::
+ {map() | nil, String.t() | nil}
+ def schema(name, version) do
+ with {:ok, body} <- fetch_schema_bytes(name, version),
+ {:ok, data} <- Jason.decode(body) do
+ sha = :sha256 |> :crypto.hash(body) |> Base.encode16(case: :lower)
+ {data, sha}
+ else
+ _ -> {nil, nil}
+ end
+ end
+
+ defp fetch_schema_bytes(name, version) do
+ url = "/npm/#{name}@#{version}/configuration-schema.json"
+
+ case Tesla.get(jsdelivr_client(), url) do
+ {:ok, %Tesla.Env{status: 200, body: body}} when is_binary(body) ->
+ {:ok, body}
+
+ {:ok, %Tesla.Env{status: status}} ->
+ {:error, {:http_status, status}}
+
+ {:error, reason} ->
+ {:error, reason}
+ end
+ end
+
+ defp jsdelivr_client do
+ build_client([
+ {Tesla.Middleware.BaseUrl, jsdelivr_url()},
+ Tesla.Middleware.FollowRedirects
+ ])
+ end
+
+ defp build_client(middleware) do
+ case Application.get_env(:tesla, :adapter) do
+ {Tesla.Adapter.Finch, _opts} ->
+ Tesla.client(
+ middleware,
+ {Tesla.Adapter.Finch,
+ name: Lightning.Finch, receive_timeout: http_timeout()}
+ )
+
+ _other ->
+ Tesla.client(middleware)
+ end
+ end
+
+ defp jsdelivr_url do
+ Config.strategy_opts(Lightning.Adaptors.NPM)[:jsdelivr_url] ||
+ @default_jsdelivr_url
+ end
+
+ defp http_timeout do
+ Config.strategy_opts(Lightning.Adaptors.NPM)[:http_timeout] ||
+ @default_http_timeout
+ end
+end
diff --git a/lib/lightning/adaptors/package_name.ex b/lib/lightning/adaptors/package_name.ex
new file mode 100644
index 00000000000..a629fa74c9d
--- /dev/null
+++ b/lib/lightning/adaptors/package_name.ex
@@ -0,0 +1,64 @@
+defmodule Lightning.Adaptors.PackageName do
+ @moduledoc """
+ NPM-style package-name parsing and worker wire-shape recomposition for
+ the `Lightning.Adaptors.*` subsystem.
+
+ This module is the single source of truth for the legacy
+ `AdaptorRegistry.resolve_adaptor/1` and `resolve_package_name/1`
+ contracts, ported to read through the `Lightning.Adaptors` facade.
+
+ `parse/1` splits `"name@version"` strings; `to_wire/1` resolves the
+ `latest` literal through `Lightning.Adaptors.resolve_version/2`,
+ preserves `"name@local"` as a literal regardless of source, and emits
+ `"name@local"` under a `:local` strategy source.
+ """
+
+ alias Lightning.Adaptors
+ alias Lightning.Adaptors.Config
+
+ @package_name_regex ~r/(@?[\/\d\n\w-]+)(?:@([\d\.\w-]+))?$/
+
+ @spec parse(nil) :: {nil, nil}
+ def parse(nil), do: {nil, nil}
+
+ @spec parse(String.t()) :: {String.t() | nil, String.t() | nil}
+ def parse(package_name) when is_binary(package_name) do
+ case Regex.run(@package_name_regex, package_name) do
+ [_, name, version] -> {name, version}
+ [_, _name] -> {package_name, nil}
+ _ -> {nil, nil}
+ end
+ end
+
+ @spec to_wire(String.t() | nil) :: String.t()
+ def to_wire(adaptor) do
+ case parse(adaptor) do
+ {nil, nil} -> ""
+ {name, version} -> recompose(name, version, adaptor)
+ end
+ end
+
+ defp recompose(name, "local", _original), do: "#{name}@local"
+
+ defp recompose(name, version, original) do
+ case Config.current_source() do
+ :local ->
+ "#{name}@local"
+
+ _ ->
+ case version do
+ "latest" ->
+ case Adaptors.resolve_version(name, "latest") do
+ {:ok, resolved} -> "#{name}@#{resolved}"
+ {:error, _} -> "#{name}@latest"
+ end
+
+ nil ->
+ original
+
+ _concrete ->
+ original
+ end
+ end
+ end
+end
diff --git a/lib/lightning/adaptors/repo.ex b/lib/lightning/adaptors/repo.ex
new file mode 100644
index 00000000000..f15f2ad690d
--- /dev/null
+++ b/lib/lightning/adaptors/repo.ex
@@ -0,0 +1,339 @@
+defmodule Lightning.Adaptors.Repo do
+ @moduledoc """
+ Query and write helpers over the `adaptors` and `adaptor_versions` tables.
+
+ Despite the name, this is **not** an `Ecto.Repo` — it is a thin
+ data-access module that wraps `Lightning.Repo` (the real
+ `Ecto.Repo`). The two schemas it targets live as siblings:
+ `Lightning.Adaptors.Repo.Adaptor` and
+ `Lightning.Adaptors.Repo.AdaptorVersion`.
+
+ Every read helper takes the desired `:source` (`:npm | :local`)
+ explicitly; the module itself stays source-agnostic. Callers resolve
+ the active source via `Lightning.Adaptors.Config.current_source/0`
+ (see §4.4 source-tagging invariant and §6.4 in
+ `.context/lightning/adaptors/REWRITE-2026-05.md`).
+
+ `upsert_adaptor/1` is the only writer the Scheduler uses. It is
+ idempotent, transactional, and diff-aware: `checked_at` advances on
+ every call, while `updated_at` only advances when the row's
+ meaningful fields differ from what was already in the DB. Version
+ rows are replaced inside the same transaction so a partial failure
+ cannot leave the table half-rewritten.
+ """
+
+ import Ecto.Query
+
+ alias Ecto.Multi
+ alias Lightning.Adaptors.Repo.Adaptor
+ alias Lightning.Adaptors.Repo.AdaptorVersion
+
+ @type source :: :npm | :local
+
+ @type package_meta :: %{
+ name: String.t(),
+ latest_version: String.t(),
+ description: String.t() | nil,
+ deprecated: boolean(),
+ updated_at: DateTime.t(),
+ icon_square_ext: String.t() | nil,
+ icon_rectangle_ext: String.t() | nil,
+ icon_square_sha256: binary() | nil,
+ icon_rectangle_sha256: binary() | nil
+ }
+
+ @version_row_fields ~w(adaptor_id version integrity tarball_url
+ size_bytes dependencies peer_dependencies
+ published_at deprecated)a
+
+ @doc """
+ Picker-facing lean projection for a source. Avoids the heavy JSONB
+ columns (`schema_data`, `dependencies`, `peer_dependencies`).
+ """
+ @spec list_package_metas(source()) :: [package_meta()]
+ def list_package_metas(source) do
+ Lightning.Repo.all(
+ from a in Adaptor,
+ where: a.source == ^source,
+ select: %{
+ name: a.name,
+ latest_version: a.latest_version,
+ description: a.description,
+ deprecated: a.deprecated,
+ updated_at: a.updated_at,
+ icon_square_ext: a.icon_square_ext,
+ icon_rectangle_ext: a.icon_rectangle_ext,
+ icon_square_sha256: a.icon_square_sha256,
+ icon_rectangle_sha256: a.icon_rectangle_sha256
+ }
+ )
+ end
+
+ @doc """
+ Full structs for a source. Rare — used by debug tools and admin
+ views. Picker traffic goes through `list_package_metas/1`.
+ """
+ @spec list_adaptors(source()) :: [Adaptor.t()]
+ def list_adaptors(source) do
+ Lightning.Repo.all(from a in Adaptor, where: a.source == ^source)
+ end
+
+ @doc """
+ Fetch a single adaptor by `name` within a `source`. Returns `nil`
+ when no row matches.
+ """
+ @spec get_adaptor(String.t(), source()) :: Adaptor.t() | nil
+ def get_adaptor(name, source) do
+ Lightning.Repo.get_by(Adaptor, name: name, source: source)
+ end
+
+ @doc """
+ All versions of an adaptor (`name`, `source`), in insertion order.
+ """
+ @spec list_versions(String.t(), source()) :: [AdaptorVersion.t()]
+ def list_versions(name, source) do
+ Lightning.Repo.all(
+ from v in AdaptorVersion,
+ join: a in Adaptor,
+ on: v.adaptor_id == a.id,
+ where: a.name == ^name and a.source == ^source,
+ order_by: [asc: v.inserted_at]
+ )
+ end
+
+ @doc """
+ Idempotent, transactional, diff-aware upsert of one adaptor record
+ plus its version rows. The `:source` is read from the record.
+
+ Behaviour:
+
+ * On every call, `checked_at` is advanced to "now".
+ * `updated_at` only advances when at least one non-`checked_at`
+ field of the adaptor row actually differs from the existing row.
+ * Version rows are replaced (delete + insert) inside the same
+ transaction.
+ * Every row is run through its schema changeset before write, so a
+ corrupt Strategy response cannot poison the DB.
+
+ Raises if the underlying transaction fails (e.g. invalid input from
+ a misbehaving strategy) — the success type is the only contract the
+ Scheduler relies on.
+ """
+ @spec upsert_adaptor(map()) :: {:ok, Adaptor.t()}
+ def upsert_adaptor(record) when is_map(record) do
+ now = DateTime.utc_now()
+
+ {versions, adaptor_attrs} =
+ record
+ |> Map.put(:checked_at, now)
+ |> Map.pop(:versions, [])
+
+ name = Map.fetch!(adaptor_attrs, :name)
+ source = Map.fetch!(adaptor_attrs, :source)
+
+ multi =
+ Multi.new()
+ |> Multi.run(:existing, fn repo, _ ->
+ {:ok, repo.get_by(Adaptor, name: name, source: source)}
+ end)
+ |> Multi.run(:adaptor, fn repo, %{existing: existing} ->
+ upsert_adaptor_row(repo, existing, adaptor_attrs, now)
+ end)
+ |> Multi.run(:delete_versions, fn repo, %{adaptor: adaptor} ->
+ {count, _} =
+ repo.delete_all(
+ from v in AdaptorVersion, where: v.adaptor_id == ^adaptor.id
+ )
+
+ {:ok, count}
+ end)
+ |> Multi.run(:insert_versions, fn repo, %{adaptor: adaptor} ->
+ insert_version_rows(repo, adaptor.id, versions, now)
+ end)
+
+ case Lightning.Repo.transaction(multi) do
+ {:ok, %{adaptor: adaptor}} ->
+ {:ok, adaptor}
+
+ {:error, step, reason, _changes} ->
+ raise ArgumentError,
+ "Lightning.Adaptors.Repo.upsert_adaptor/1 failed at #{inspect(step)}: " <>
+ inspect(reason)
+ end
+ end
+
+ @doc """
+ Advance `checked_at` for a known `(name, source)` row without
+ loading it. No-op when no row matches.
+
+ Used by the Scheduler's "polled NPM, nothing changed" path —
+ cheaper than a full upsert and never bumps `updated_at`.
+ """
+ @spec touch_checked_at(String.t(), source()) :: :ok
+ def touch_checked_at(name, source) do
+ now = DateTime.utc_now()
+
+ Lightning.Repo.update_all(
+ from(a in Adaptor, where: a.name == ^name and a.source == ^source),
+ set: [checked_at: now]
+ )
+
+ :ok
+ end
+
+ @doc """
+ The `limit` rows for a given `source` whose `checked_at` is oldest
+ first. Backs the Scheduler's per-tick work list.
+ """
+ @spec stalest(pos_integer(), source()) :: [Adaptor.t()]
+ def stalest(limit, source) when is_integer(limit) and limit > 0 do
+ Lightning.Repo.all(
+ from a in Adaptor,
+ where: a.source == ^source,
+ order_by: [asc: a.checked_at],
+ limit: ^limit
+ )
+ end
+
+ @doc """
+ Lean list of source-scoped adaptors that are missing at least one icon
+ shape. Returns only the fields the Scheduler needs to decide whether to
+ re-apply the bulk icon fetch result.
+ """
+ @spec list_missing_icons(source()) :: [
+ %{
+ name: String.t(),
+ icon_square_sha256: binary() | nil,
+ icon_rectangle_sha256: binary() | nil
+ }
+ ]
+ def list_missing_icons(source) do
+ Lightning.Repo.all(
+ from a in Adaptor,
+ where:
+ a.source == ^source and
+ (is_nil(a.icon_square_sha256) or is_nil(a.icon_rectangle_sha256)),
+ select: %{
+ name: a.name,
+ icon_square_sha256: a.icon_square_sha256,
+ icon_rectangle_sha256: a.icon_rectangle_sha256
+ }
+ )
+ end
+
+ @doc """
+ Update only the icon columns for a single `(name, source)` row.
+
+ `attrs` may include any subset of `:icon_square_ext`,
+ `:icon_square_sha256`, `:icon_rectangle_ext`, `:icon_rectangle_sha256`,
+ `:icon_square_etag`, `:icon_rectangle_etag`.
+ `updated_at` is advanced so callers can observe the change.
+
+ Sidesteps `upsert_adaptor/1` deliberately: that helper rewrites the
+ `adaptor_versions` rows in the same transaction, which is the wrong
+ thing to do for an icon-only fix-up.
+ """
+ @spec update_icons(String.t(), source(), map()) :: {integer(), nil}
+ def update_icons(name, source, attrs) when is_map(attrs) do
+ allowed =
+ attrs
+ |> Map.take([
+ :icon_square_ext,
+ :icon_square_sha256,
+ :icon_rectangle_ext,
+ :icon_rectangle_sha256,
+ :icon_square_etag,
+ :icon_rectangle_etag
+ ])
+ |> Map.put(:updated_at, DateTime.utc_now())
+ |> Enum.into([])
+
+ Lightning.Repo.update_all(
+ from(a in Adaptor, where: a.name == ^name and a.source == ^source),
+ set: allowed
+ )
+ end
+
+ @doc """
+ Maximum `checked_at` seen for `source`, or `nil` when the table is
+ empty for that source. Backs the Scheduler's smart-init timing.
+ """
+ @spec max_checked_at(source()) :: DateTime.t() | nil
+ def max_checked_at(source) do
+ Lightning.Repo.one(
+ from a in Adaptor,
+ where: a.source == ^source,
+ select: max(a.checked_at)
+ )
+ end
+
+ defp upsert_adaptor_row(repo, nil, attrs, _now) do
+ %Adaptor{}
+ |> Adaptor.changeset(attrs)
+ |> repo.insert()
+ end
+
+ defp upsert_adaptor_row(repo, %Adaptor{} = existing, attrs, now) do
+ changeset = Adaptor.changeset(existing, attrs)
+
+ # `Ecto.Changeset.cast/3` only records a change when the cast value
+ # differs from the underlying struct, so the set of "real" changes
+ # is `:changes` minus the `:checked_at` tick we apply on every call.
+ meaningful_changes? =
+ changeset.changes
+ |> Map.delete(:checked_at)
+ |> map_size() > 0
+
+ if meaningful_changes? do
+ repo.update(changeset)
+ else
+ {1, _} =
+ repo.update_all(
+ from(a in Adaptor, where: a.id == ^existing.id),
+ set: [checked_at: now]
+ )
+
+ {:ok, %{existing | checked_at: now}}
+ end
+ end
+
+ defp insert_version_rows(_repo, _adaptor_id, [], _now), do: {:ok, 0}
+
+ defp insert_version_rows(repo, adaptor_id, records, now) do
+ case build_version_rows(adaptor_id, records, now) do
+ {:ok, rows} ->
+ {count, _} = repo.insert_all(AdaptorVersion, rows)
+ {:ok, count}
+
+ {:error, changeset} ->
+ {:error, changeset}
+ end
+ end
+
+ defp build_version_rows(adaptor_id, records, now) do
+ records
+ |> Enum.reduce_while({:ok, []}, fn record, {:ok, acc} ->
+ attrs = Map.put(record, :adaptor_id, adaptor_id)
+ changeset = AdaptorVersion.changeset(%AdaptorVersion{}, attrs)
+
+ if changeset.valid? do
+ {:cont, {:ok, [version_row_from_changeset(changeset, now) | acc]}}
+ else
+ {:halt, {:error, changeset}}
+ end
+ end)
+ |> case do
+ {:ok, rows} -> {:ok, Enum.reverse(rows)}
+ err -> err
+ end
+ end
+
+ defp version_row_from_changeset(changeset, now) do
+ changeset
+ |> Ecto.Changeset.apply_changes()
+ |> Map.from_struct()
+ |> Map.take(@version_row_fields)
+ |> Map.put(:id, Ecto.UUID.generate())
+ |> Map.put(:inserted_at, now)
+ end
+end
diff --git a/lib/lightning/adaptors/repo_adaptor.ex b/lib/lightning/adaptors/repo_adaptor.ex
new file mode 100644
index 00000000000..469620df5fe
--- /dev/null
+++ b/lib/lightning/adaptors/repo_adaptor.ex
@@ -0,0 +1,163 @@
+defmodule Lightning.Adaptors.Repo.Adaptor do
+ @moduledoc """
+ Ecto schema for one row of the `adaptors` table — the per-package
+ metadata projection used by the picker and Scheduler.
+
+ Source-tagged via `:source` (`:npm | :local`) so the same package
+ name can coexist across sources; the unique index is `[:name, :source]`
+ (see §4.4 source-tagging invariant in
+ `.context/lightning/adaptors/REWRITE-2026-05.md`).
+
+ Mirrors `Lightning.Adaptors.Strategy.adaptor_record` minus `:versions`,
+ which lives on `Lightning.Adaptors.Repo.AdaptorVersion`.
+ """
+
+ use Ecto.Schema
+
+ import Ecto.Changeset
+
+ defmodule JSONBinary do
+ @moduledoc """
+ Ecto type backing `schema_data` with a text column while preserving
+ JSON field order on read.
+
+ Storage is a JSON binary in a `text` column. Inputs may be either a
+ binary or a map — maps are encoded with `Jason.encode!/1` at the
+ dumper to keep `Lightning.Factories.adaptor/2` and other direct
+ struct inserts compatible without forcing every caller to encode
+ up-front. Loads always return a binary so credential-form rendering
+ can re-engage `Jason.decode!(_, objects: :ordered_objects)`.
+ """
+
+ use Ecto.Type
+
+ @impl true
+ def type, do: :string
+
+ @impl true
+ def cast(nil), do: {:ok, nil}
+ def cast(value) when is_binary(value), do: {:ok, value}
+ def cast(value) when is_map(value), do: {:ok, Jason.encode!(value)}
+ def cast(_), do: :error
+
+ @impl true
+ def load(nil), do: {:ok, nil}
+ def load(value) when is_binary(value), do: {:ok, value}
+
+ @impl true
+ def dump(nil), do: {:ok, nil}
+ def dump(value) when is_binary(value), do: {:ok, value}
+ def dump(value) when is_map(value), do: {:ok, Jason.encode!(value)}
+ def dump(_), do: :error
+ end
+
+ @type t :: %__MODULE__{
+ id: Ecto.UUID.t() | nil,
+ name: String.t() | nil,
+ source: :npm | :local | nil,
+ description: String.t() | nil,
+ homepage: String.t() | nil,
+ repository: String.t() | nil,
+ license: String.t() | nil,
+ latest_version: String.t() | nil,
+ deprecated: boolean(),
+ schema_data: String.t() | nil,
+ schema_sha256: String.t() | nil,
+ icon_square_ext: String.t() | nil,
+ icon_rectangle_ext: String.t() | nil,
+ icon_square_sha256: binary() | nil,
+ icon_rectangle_sha256: binary() | nil,
+ icon_square_etag: String.t() | nil,
+ icon_rectangle_etag: String.t() | nil,
+ checked_at: DateTime.t() | nil,
+ inserted_at: DateTime.t() | nil,
+ updated_at: DateTime.t() | nil
+ }
+
+ @primary_key {:id, :binary_id, autogenerate: true}
+ @foreign_key_type :binary_id
+ @timestamps_opts [type: :utc_datetime_usec]
+
+ schema "adaptors" do
+ field :name, :string
+ field :source, Ecto.Enum, values: [:npm, :local]
+ field :description, :string
+ field :homepage, :string
+ field :repository, :string
+ field :license, :string
+ field :latest_version, :string
+ field :deprecated, :boolean, default: false
+ field :schema_data, Lightning.Adaptors.Repo.Adaptor.JSONBinary
+ field :schema_sha256, :string
+ field :icon_square_ext, :string
+ field :icon_rectangle_ext, :string
+ field :icon_square_sha256, :binary
+ field :icon_rectangle_sha256, :binary
+ field :icon_square_etag, :string
+ field :icon_rectangle_etag, :string
+ field :checked_at, :utc_datetime_usec
+
+ timestamps()
+ end
+
+ @required ~w(name source latest_version checked_at)a
+ @optional ~w(description homepage repository license deprecated
+ schema_data schema_sha256
+ icon_square_ext icon_rectangle_ext
+ icon_square_sha256 icon_rectangle_sha256
+ icon_square_etag icon_rectangle_etag)a
+
+ @doc """
+ Build a changeset for upserting a single adaptor row.
+
+ This is the single clause used by every write path on
+ `Lightning.Adaptors.Repo` — there is no separate update path because
+ the writer always rewrites the full row.
+ """
+ @spec changeset(t(), map()) :: Ecto.Changeset.t()
+ def changeset(struct, attrs) do
+ struct
+ |> cast(attrs, @required ++ @optional)
+ |> validate_required(@required)
+ |> validate_length(:name, max: 214)
+ |> validate_inclusion(:icon_square_ext, ~w(png svg))
+ |> validate_inclusion(:icon_rectangle_ext, ~w(png svg))
+ |> validate_icon_sha256_pair(:icon_square)
+ |> validate_icon_sha256_pair(:icon_rectangle)
+ |> unique_constraint([:name, :source])
+ end
+
+ # Enforces the §6.4 invariant: a non-nil `icon__ext` requires a
+ # non-nil `icon__sha256`, and vice versa. Either both fields
+ # are set or both are nil — half-populated pairs fail the changeset.
+ @spec validate_icon_sha256_pair(
+ Ecto.Changeset.t(),
+ :icon_square | :icon_rectangle
+ ) :: Ecto.Changeset.t()
+ defp validate_icon_sha256_pair(changeset, shape) do
+ ext_field = :"#{shape}_ext"
+ sha_field = :"#{shape}_sha256"
+
+ case {get_field(changeset, ext_field), get_field(changeset, sha_field)} do
+ {nil, nil} ->
+ changeset
+
+ {nil, _sha} ->
+ add_error(
+ changeset,
+ sha_field,
+ "must be nil when #{ext_field} is nil"
+ )
+
+ {_ext, nil} ->
+ add_error(
+ changeset,
+ sha_field,
+ "must not be nil when #{ext_field} is set"
+ )
+
+ {_ext, _sha} ->
+ changeset
+ end
+ end
+end
diff --git a/lib/lightning/adaptors/repo_adaptor_version.ex b/lib/lightning/adaptors/repo_adaptor_version.ex
new file mode 100644
index 00000000000..ec048b81806
--- /dev/null
+++ b/lib/lightning/adaptors/repo_adaptor_version.ex
@@ -0,0 +1,73 @@
+defmodule Lightning.Adaptors.Repo.AdaptorVersion do
+ @moduledoc """
+ Ecto schema for one row of the `adaptor_versions` table — per-version
+ metadata for an adaptor package (`integrity`, `tarball_url`,
+ `size_bytes`, `dependencies`, `peer_dependencies`, `published_at`,
+ `deprecated`).
+
+ Belongs to `Lightning.Adaptors.Repo.Adaptor` and cascade-deletes with
+ its parent. Mirrors `Lightning.Adaptors.Strategy.version_record` (see
+ §6.1 and §6.4 in `.context/lightning/adaptors/REWRITE-2026-05.md`).
+ """
+
+ use Ecto.Schema
+
+ import Ecto.Changeset
+
+ alias Lightning.Adaptors.Repo.Adaptor
+
+ @type t :: %__MODULE__{
+ id: Ecto.UUID.t() | nil,
+ adaptor_id: Ecto.UUID.t() | nil,
+ adaptor: Adaptor.t() | Ecto.Association.NotLoaded.t() | nil,
+ version: String.t() | nil,
+ integrity: String.t() | nil,
+ tarball_url: String.t() | nil,
+ size_bytes: integer() | nil,
+ dependencies: map() | nil,
+ peer_dependencies: map() | nil,
+ published_at: DateTime.t() | nil,
+ deprecated: boolean(),
+ inserted_at: DateTime.t() | nil
+ }
+
+ @primary_key {:id, :binary_id, autogenerate: true}
+ @foreign_key_type :binary_id
+ @timestamps_opts [type: :utc_datetime_usec]
+
+ schema "adaptor_versions" do
+ field :version, :string
+ field :integrity, :string
+ field :tarball_url, :string
+ field :size_bytes, :integer
+ field :dependencies, :map
+ field :peer_dependencies, :map
+ field :published_at, :utc_datetime_usec
+ field :deprecated, :boolean, default: false
+
+ belongs_to :adaptor, Adaptor
+
+ timestamps(updated_at: false)
+ end
+
+ @required ~w(adaptor_id version)a
+ @optional ~w(integrity tarball_url size_bytes
+ dependencies peer_dependencies
+ published_at deprecated)a
+
+ @doc """
+ Build a changeset for inserting an `adaptor_versions` row.
+
+ `Lightning.Adaptors.Repo.upsert_adaptor/1` replaces version rows with
+ a delete-then-insert inside a transaction, so there is no separate
+ update path.
+ """
+ @spec changeset(t(), map()) :: Ecto.Changeset.t()
+ def changeset(struct, attrs) do
+ struct
+ |> cast(attrs, @required ++ @optional)
+ |> validate_required(@required)
+ |> unique_constraint([:adaptor_id, :version])
+ |> assoc_constraint(:adaptor)
+ end
+end
diff --git a/lib/lightning/adaptors/scheduler.ex b/lib/lightning/adaptors/scheduler.ex
new file mode 100644
index 00000000000..6077fc48d2a
--- /dev/null
+++ b/lib/lightning/adaptors/scheduler.ex
@@ -0,0 +1,597 @@
+defmodule Lightning.Adaptors.Scheduler do
+ @moduledoc """
+ Cluster-singleton GenServer that periodically refreshes the active
+ source's ledger via the configured strategy, persists through
+ `Lightning.Adaptors.Repo`, and broadcasts `{:changed, name, source}`.
+
+ Wrapped by `HighlanderPG` so only one node in the cluster runs the
+ Scheduler at a time. The inner GenServer registers under
+ `{:global, Lightning.Adaptors.Supervisor.global_scheduler_name(name)}`,
+ so callers on any node reach the leader transparently via Erlang
+ distribution. Peer nodes react to refreshes via
+ `Lightning.Adaptors.Invalidator` and
+ `Lightning.Adaptors.ChannelBroadcaster`.
+
+ Smart-init timing: the first tick is scheduled at
+ `max(0, last_checked_at + interval - now)` to avoid double-refreshing
+ shortly after a deploy. An empty table or an overdue schedule fires
+ immediately (`delay = 0`). Interval `0` disables scheduling entirely.
+
+ ## Two-pipeline refresh
+
+ A tick runs two parallel pipelines under the per-instance
+ `Task.Supervisor`:
+
+ * **Pipeline A** — `strategy.fetch_icons/1` for every adaptor.
+ * **Pipeline B** — `strategy.list_adaptors/0` followed by a bounded
+ per-adaptor fan-out (`async_stream_nolink`) calling
+ `strategy.fetch_adaptor/1` only for names whose `latest_version`
+ changed since the last tick.
+
+ Once both pipelines complete the join step merges the icons map into
+ each fetched record, writes the icon bytes to disk via
+ `Lightning.Adaptors.IconCache.write!/5`, and upserts each adaptor in
+ one go. `refresh_package/2` deliberately bypasses the icon pipeline —
+ on-demand single-package refreshes do not refetch icons.
+ """
+
+ use GenServer
+
+ alias Lightning.Adaptors.Config
+ alias Lightning.Adaptors.IconCache
+ alias Lightning.Adaptors.Repo, as: AdaptorsRepo
+ alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor
+
+ require Logger
+
+ @fetch_max_concurrency 8
+ @icons_task_timeout :timer.seconds(60)
+
+ @doc """
+ Start the Scheduler for the given supervisor instance.
+
+ Required opts: `:name`, `:sup`, `:lock_key`, `:cache`, `:tasks`,
+ `:source_topic`.
+ """
+ @spec start_link(keyword()) :: GenServer.on_start()
+ def start_link(opts) do
+ name = Keyword.fetch!(opts, :name)
+ _ = Keyword.fetch!(opts, :sup)
+ _ = Keyword.fetch!(opts, :lock_key)
+ _ = Keyword.fetch!(opts, :cache)
+ _ = Keyword.fetch!(opts, :tasks)
+ _ = Keyword.fetch!(opts, :source_topic)
+ GenServer.start_link(__MODULE__, opts, name: name)
+ end
+
+ @doc """
+ Trigger an immediate refresh tick.
+
+ Routes via `:global` to the leader-held GenServer.
+ """
+ @spec refresh_now(GenServer.server()) :: :ok | {:error, term()}
+ def refresh_now(scheduler_name) do
+ GenServer.call(scheduler_name, :refresh_now)
+ end
+
+ @doc """
+ Force a single-adaptor refresh, bypassing the diff. 30-second timeout.
+
+ Returns `{:error, :not_found}` or `{:error, term()}` from a failed
+ strategy fetch.
+ """
+ @spec refresh_package(GenServer.server(), String.t()) ::
+ :ok | {:error, :not_found | term()}
+ def refresh_package(scheduler_name, name) do
+ GenServer.call(scheduler_name, {:refresh_package, name}, 30_000)
+ end
+
+ @doc """
+ Refresh icons only, against every source-scoped adaptor row.
+
+ Runs `strategy.fetch_icons/1` and re-applies any shape whose `sha256`
+ differs from what is on the row. Adaptor metadata and version rows
+ are not touched. Returns `{:ok, %{updated: n, unchanged: m}}` on
+ success or `{:error, reason}` if the bulk fetch fails.
+ """
+ @spec refresh_icons(GenServer.server()) ::
+ {:ok, %{updated: non_neg_integer(), unchanged: non_neg_integer()}}
+ | {:error, term()}
+ def refresh_icons(scheduler_name) do
+ GenServer.call(scheduler_name, :refresh_icons, 120_000)
+ end
+
+ @impl true
+ def init(opts) do
+ sup = Keyword.fetch!(opts, :sup)
+ source_topic = Keyword.fetch!(opts, :source_topic)
+ cache = Keyword.fetch!(opts, :cache)
+ tasks = Keyword.fetch!(opts, :tasks)
+
+ source = AdaptorsSupervisor.source(sup)
+ interval_ms = Config.refresh_interval()
+
+ if interval_ms > 0 do
+ delay =
+ time_until_next_ms(AdaptorsRepo.max_checked_at(source), interval_ms)
+
+ Process.send_after(self(), :tick, delay)
+
+ Logger.info(
+ "Adaptors[#{source}]: scheduler started interval=#{interval_ms}ms next_tick_in=#{delay}ms"
+ )
+ else
+ Logger.info("Adaptors[#{source}]: scheduler started interval=0 (disabled)")
+ end
+
+ {:ok,
+ %{
+ sup: sup,
+ source: source,
+ interval_ms: interval_ms,
+ source_topic: source_topic,
+ cache: cache,
+ tasks: tasks
+ }}
+ end
+
+ @impl true
+ def handle_info(:tick, state) do
+ if state.interval_ms > 0 do
+ Process.send_after(self(), :tick, state.interval_ms)
+ end
+
+ Task.Supervisor.start_child(state.tasks, fn -> do_refresh(state) end)
+
+ {:noreply, state}
+ end
+
+ @impl true
+ def handle_call(:refresh_now, _from, state) do
+ Logger.info("Adaptors[#{state.source}]: refresh_now requested")
+ send(self(), :tick)
+ {:reply, :ok, state}
+ end
+
+ def handle_call({:refresh_package, name}, _from, state) do
+ Logger.info("Adaptors[#{state.source}]: refresh_package(#{name}) requested")
+
+ strategy = AdaptorsSupervisor.strategy(state.sup)
+ result = force_refresh_one(strategy, name, state)
+ {:reply, result, state}
+ end
+
+ def handle_call(:refresh_icons, _from, state) do
+ Logger.info("Adaptors[#{state.source}]: refresh_icons requested")
+ strategy = AdaptorsSupervisor.strategy(state.sup)
+
+ existing = AdaptorsRepo.list_adaptors(state.source)
+ prior_etags = prior_etags_from_rows(existing)
+
+ case strategy.fetch_icons(prior_etags: prior_etags) do
+ {:ok, icons} ->
+ result = reapply_icons(existing, icons, state)
+
+ Logger.info(
+ "Adaptors[#{state.source}]: refresh_icons done " <>
+ "rows=#{length(existing)} icons=#{map_size(icons)} " <>
+ "updated=#{result.updated} unchanged=#{result.unchanged}"
+ )
+
+ {:reply, {:ok, result}, state}
+
+ {:error, reason} ->
+ Logger.warning(
+ "Adaptors[#{state.source}]: refresh_icons strategy fetch failed: #{inspect(reason)}"
+ )
+
+ {:reply, {:error, reason}, state}
+ end
+ end
+
+ defp do_refresh(state) do
+ started_at = System.monotonic_time(:millisecond)
+ strategy = AdaptorsSupervisor.strategy(state.sup)
+
+ # Single DB round-trip serves both the icons-task input (prior etags)
+ # and the version diff used below to decide which adaptors to fetch.
+ existing_rows = AdaptorsRepo.list_adaptors(state.source)
+ prior_etags = prior_etags_from_rows(existing_rows)
+
+ existing_by_name =
+ Map.new(existing_rows, fn a -> {a.name, a.latest_version} end)
+
+ icons_task =
+ Task.Supervisor.async_nolink(state.tasks, fn ->
+ strategy.fetch_icons(prior_etags: prior_etags)
+ end)
+
+ case strategy.list_adaptors() do
+ {:ok, upstream} ->
+ {fetched, changed, errors} =
+ state.tasks
+ |> Task.Supervisor.async_stream_nolink(
+ upstream,
+ &fetch_if_changed(strategy, &1, existing_by_name, state),
+ max_concurrency: @fetch_max_concurrency,
+ ordered: false,
+ on_timeout: :kill_task
+ )
+ |> Enum.reduce({[], 0, 0}, fn
+ {:ok, {:fetched, record}}, {acc, c, e} -> {[record | acc], c + 1, e}
+ {:ok, :touched}, {acc, c, e} -> {acc, c, e}
+ {:ok, {:error, _reason}}, {acc, c, e} -> {acc, c, e + 1}
+ {:exit, _reason}, {acc, c, e} -> {acc, c, e + 1}
+ end)
+
+ icons = await_icons(icons_task)
+
+ persisted =
+ fetched
+ |> Enum.map(fn record -> persist_with_icons(record, icons, state) end)
+ |> Enum.count(&(&1 == :ok))
+
+ healed = heal_missing_icons(icons, state)
+ not_modified = count_not_modified(icons)
+
+ listed = length(upstream)
+ touched = listed - changed - errors
+ duration_ms = System.monotonic_time(:millisecond) - started_at
+
+ Logger.info(
+ "Adaptors[#{state.source}]: refresh tick listed=#{listed} " <>
+ "changed=#{changed} touched=#{touched} fetched=#{persisted} " <>
+ "icons=#{map_size(icons)} healed=#{healed} " <>
+ "not_modified=#{not_modified} " <>
+ "errors=#{errors} duration=#{duration_ms}ms"
+ )
+
+ {:error, reason} ->
+ Logger.warning("Scheduler: list_adaptors failed: #{inspect(reason)}")
+ _ = await_icons(icons_task)
+ duration_ms = System.monotonic_time(:millisecond) - started_at
+
+ Logger.info(
+ "Adaptors[#{state.source}]: refresh tick listed=0 changed=0 " <>
+ "touched=0 fetched=0 icons=0 errors=1 duration=#{duration_ms}ms"
+ )
+
+ :ok
+ end
+ end
+
+ defp fetch_if_changed(
+ strategy,
+ %{name: name, latest_version: version},
+ existing_by_name,
+ state
+ ) do
+ if Map.get(existing_by_name, name) == version do
+ AdaptorsRepo.touch_checked_at(name, state.source)
+ :touched
+ else
+ case strategy.fetch_adaptor(name) do
+ {:ok, record} ->
+ Logger.debug(
+ "Adaptors[#{state.source}]: fetched #{name}@#{record.version}"
+ )
+
+ {:fetched, record}
+
+ {:error, reason} ->
+ Logger.warning(
+ "Scheduler: fetch_adaptor(#{name}) failed: #{inspect(reason)}"
+ )
+
+ {:error, reason}
+ end
+ end
+ end
+
+ defp await_icons(task) do
+ case Task.yield(task, @icons_task_timeout) || Task.shutdown(task) do
+ {:ok, {:ok, map}} when is_map(map) ->
+ map
+
+ {:ok, {:error, reason}} ->
+ Logger.warning(
+ "Scheduler: fetch_icons failed: #{inspect(reason)} — persisting records without icons"
+ )
+
+ %{}
+
+ {:exit, reason} ->
+ Logger.warning(
+ "Scheduler: fetch_icons crashed: #{inspect(reason)} — persisting records without icons"
+ )
+
+ %{}
+
+ nil ->
+ Logger.warning(
+ "Scheduler: fetch_icons timed out — persisting records without icons"
+ )
+
+ %{}
+ end
+ end
+
+ defp persist_with_icons(record, icons, state) do
+ name = record.name
+ package_icons = Map.get(icons, name, %{})
+
+ record_with_icons =
+ record
+ |> Map.put(:source, state.source)
+ |> merge_icon(:square, package_icons, state.source)
+ |> merge_icon(:rectangle, package_icons, state.source)
+
+ try do
+ {:ok, _} = AdaptorsRepo.upsert_adaptor(record_with_icons)
+
+ Phoenix.PubSub.broadcast(
+ Lightning.PubSub,
+ state.source_topic,
+ {:changed, name, state.source}
+ )
+
+ Logger.debug("Adaptors[#{state.source}]: persisted #{name}")
+ :ok
+ rescue
+ e ->
+ Logger.error(
+ "Scheduler: upsert_adaptor(#{name}) failed: #{Exception.message(e)}"
+ )
+
+ :error
+ end
+ end
+
+ defp merge_icon(record, shape, package_icons, source) do
+ case Map.get(package_icons, shape) do
+ %{data: bytes, ext: ext, sha256: sha} = entry when is_binary(bytes) ->
+ try do
+ {:ok, ^sha} = IconCache.write!(source, record.name, shape, ext, bytes)
+
+ record
+ |> Map.put(:"icon_#{shape}_ext", ext)
+ |> Map.put(:"icon_#{shape}_sha256", sha)
+ |> maybe_put_etag(shape, Map.get(entry, :etag))
+ rescue
+ e ->
+ Logger.warning(
+ "Scheduler: IconCache.write!(#{record.name}, #{shape}) failed: #{Exception.message(e)}"
+ )
+
+ record
+ end
+
+ :not_modified ->
+ # Upstream confirmed unchanged — leave row's existing icon and
+ # etag in place. Counted in the tick summary via
+ # count_not_modified/1.
+ record
+
+ _ ->
+ record
+ end
+ end
+
+ # Stamp the etag onto the record only when the strategy supplied one
+ # (NPM 200 entries always have the key; Local omits it). A nil etag is
+ # not stamped — we preserve whatever was already on the row.
+ defp maybe_put_etag(record, _shape, nil), do: record
+
+ defp maybe_put_etag(record, shape, etag) when is_binary(etag) do
+ Map.put(record, :"icon_#{shape}_etag", etag)
+ end
+
+ # Top up icons on rows that currently have NULL on at least one shape.
+ # Runs after the main upsert pass on every tick — cheap, scoped to
+ # rows with gaps, and self-correcting after a strategy outage or a
+ # past bug like the one that left every row iconless.
+ defp heal_missing_icons(icons, _state) when map_size(icons) == 0, do: 0
+
+ defp heal_missing_icons(icons, state) do
+ state.source
+ |> AdaptorsRepo.list_missing_icons()
+ |> Enum.reduce(0, fn row, acc ->
+ package_icons = Map.get(icons, row.name, %{})
+
+ case apply_icons_to_existing(row, package_icons, state) do
+ :updated -> acc + 1
+ :unchanged -> acc
+ end
+ end)
+ end
+
+ defp reapply_icons(existing_rows, icons, state) do
+ Enum.reduce(existing_rows, %{updated: 0, unchanged: 0}, fn row, acc ->
+ package_icons = Map.get(icons, row.name, %{})
+
+ case apply_icons_to_existing(row, package_icons, state) do
+ :updated -> %{acc | updated: acc.updated + 1}
+ :unchanged -> %{acc | unchanged: acc.unchanged + 1}
+ end
+ end)
+ end
+
+ # `row` is either an Adaptor struct (from list_adaptors/1) or a lean
+ # map (from list_missing_icons/1) — both expose :name and the icon
+ # sha256 fields, which is all we need.
+ defp apply_icons_to_existing(_row, package_icons, _state)
+ when map_size(package_icons) == 0,
+ do: :unchanged
+
+ defp apply_icons_to_existing(row, package_icons, state) do
+ changes =
+ [:square, :rectangle]
+ |> Enum.reduce(%{}, fn shape, acc ->
+ accumulate_icon_change(acc, shape, row, package_icons, state)
+ end)
+
+ if map_size(changes) > 0 do
+ {1, _} = AdaptorsRepo.update_icons(row.name, state.source, changes)
+
+ Phoenix.PubSub.broadcast(
+ Lightning.PubSub,
+ state.source_topic,
+ {:changed, row.name, state.source}
+ )
+
+ :updated
+ else
+ :unchanged
+ end
+ end
+
+ defp accumulate_icon_change(acc, shape, row, package_icons, state) do
+ sha_key = :"icon_#{shape}_sha256"
+ ext_key = :"icon_#{shape}_ext"
+ etag_key = :"icon_#{shape}_etag"
+
+ case Map.get(package_icons, shape) do
+ %{data: bytes, ext: ext, sha256: sha} = entry when is_binary(bytes) ->
+ if Map.get(row, sha_key) == sha do
+ # Same bytes already on disk; the etag may still need
+ # refreshing if the strategy gave us a new (non-nil) value
+ # that differs from what we have. nil never clobbers.
+ maybe_accumulate_etag(acc, etag_key, row, Map.get(entry, :etag))
+ else
+ accumulate_fetched_icon(acc, shape, row, entry, ext, sha, bytes, state,
+ sha_key: sha_key,
+ ext_key: ext_key,
+ etag_key: etag_key
+ )
+ end
+
+ :not_modified ->
+ # 304 confirmed — nothing to write, etag already current.
+ acc
+
+ _ ->
+ acc
+ end
+ end
+
+ defp accumulate_fetched_icon(acc, shape, row, entry, ext, sha, bytes, state,
+ sha_key: sha_key,
+ ext_key: ext_key,
+ etag_key: etag_key
+ ) do
+ try do
+ {:ok, ^sha} = IconCache.write!(state.source, row.name, shape, ext, bytes)
+
+ acc
+ |> Map.put(ext_key, ext)
+ |> Map.put(sha_key, sha)
+ |> maybe_accumulate_etag(etag_key, row, Map.get(entry, :etag))
+ rescue
+ e ->
+ Logger.warning(
+ "Scheduler: IconCache.write!(#{row.name}, #{shape}) failed: " <>
+ Exception.message(e)
+ )
+
+ acc
+ end
+ end
+
+ # nil → preserve existing etag on the row (do not clobber).
+ # value matching the row's current etag → no-op (avoid no-op write).
+ # value differing → emit the change.
+ defp maybe_accumulate_etag(acc, _etag_key, _row, nil), do: acc
+
+ defp maybe_accumulate_etag(acc, etag_key, row, etag) when is_binary(etag) do
+ if Map.get(row, etag_key) == etag do
+ acc
+ else
+ Map.put(acc, etag_key, etag)
+ end
+ end
+
+ defp force_refresh_one(strategy, name, state) do
+ case strategy.fetch_adaptor(name) do
+ {:ok, record} ->
+ record_with_source = Map.put(record, :source, state.source)
+
+ try do
+ {:ok, _} = AdaptorsRepo.upsert_adaptor(record_with_source)
+
+ Phoenix.PubSub.broadcast(
+ Lightning.PubSub,
+ state.source_topic,
+ {:changed, name, state.source}
+ )
+
+ Logger.info(
+ "Adaptors[#{state.source}]: refresh_package(#{name}) ok version=#{record.version}"
+ )
+
+ :ok
+ rescue
+ e ->
+ Logger.error(
+ "Scheduler: upsert_adaptor(#{name}) failed: #{Exception.message(e)}"
+ )
+
+ {:error, {:upsert_failed, Exception.message(e)}}
+ end
+
+ {:error, reason} ->
+ Logger.warning(
+ "Scheduler: refresh_package(#{name}) strategy fetch failed: #{inspect(reason)}"
+ )
+
+ {:error, reason}
+ end
+ end
+
+ # Project a list of adaptor rows to the prior-etag map shape expected
+ # by `Strategy.fetch_icons/1`: `%{name => %{shape => etag}}`. Rows
+ # whose etags are both nil are skipped entirely (no empty inner map);
+ # within a row, only shapes with a non-nil etag are kept. The consumer
+ # treats absence as "no prior etag, send no If-None-Match", so an empty
+ # entry would be wasteful but harmless — we drop it for clarity.
+ @spec prior_etags_from_rows([map()]) :: %{
+ String.t() => %{optional(:square | :rectangle) => String.t()}
+ }
+ defp prior_etags_from_rows(rows) do
+ Enum.reduce(rows, %{}, fn row, acc ->
+ inner =
+ %{}
+ |> maybe_put_shape_etag(:square, Map.get(row, :icon_square_etag))
+ |> maybe_put_shape_etag(:rectangle, Map.get(row, :icon_rectangle_etag))
+
+ if map_size(inner) == 0 do
+ acc
+ else
+ Map.put(acc, row.name, inner)
+ end
+ end)
+ end
+
+ defp maybe_put_shape_etag(map, _shape, nil), do: map
+
+ defp maybe_put_shape_etag(map, shape, etag) when is_binary(etag),
+ do: Map.put(map, shape, etag)
+
+ # Count :not_modified sentinels across all shapes in the icons map.
+ # Used in the tick summary log.
+ defp count_not_modified(icons) do
+ Enum.reduce(icons, 0, fn {_name, shapes}, acc ->
+ Enum.reduce(shapes, acc, fn
+ {_shape, :not_modified}, n -> n + 1
+ {_shape, _}, n -> n
+ end)
+ end)
+ end
+
+ defp time_until_next_ms(nil, _interval_ms), do: 0
+
+ defp time_until_next_ms(%DateTime{} = last, interval_ms) do
+ next = DateTime.add(last, interval_ms, :millisecond)
+ diff = DateTime.diff(next, DateTime.utc_now(), :millisecond)
+ max(0, diff)
+ end
+end
diff --git a/lib/lightning/adaptors/store.ex b/lib/lightning/adaptors/store.ex
new file mode 100644
index 00000000000..a1d23c781d9
--- /dev/null
+++ b/lib/lightning/adaptors/store.ex
@@ -0,0 +1,347 @@
+defmodule Lightning.Adaptors.Store do
+ @moduledoc """
+ Stateless read facade over `Cachex`, `Lightning.Adaptors.Repo`, and the
+ active `Lightning.Adaptors.Strategy`.
+
+ Every public read helper wraps a `Cachex.fetch/4` whose fallback first
+ consults the local Postgres projection (`Lightning.Adaptors.Repo`) and
+ only invokes the Strategy as a last resort. Cachex's courier supplies
+ blocking semantics and per-key coalescing of concurrent first-callers
+ for free — there is no GenServer mailbox in front of the reads.
+
+ ## Source tagging
+
+ Each cache key carries the active `:source` (`:npm | :local`) read via
+ `Lightning.Adaptors.Supervisor.source/1`, so the same package name can
+ coexist across deployment modes without manual scrubbing (see §4.4 of
+ `.context/adaptors/REWRITE-2026-05.md`).
+
+ ## Commit vs ignore
+
+ Successful Strategy/Repo lookups commit their projected value to the
+ cache. Failures — empty `packages/1` results, unknown adaptors for
+ `icon_meta/2`, Strategy errors — return `:ignore`, so a subsequent
+ caller retries fresh rather than seeing a poisoned cache entry.
+
+ ## Icons
+
+ `icon/3` returns a `Path.t/0` the controller serves via `send_file/3`
+ — no binary on the BEAM heap. The on-disk `Lightning.Adaptors.IconCache`
+ is the primary cache: a `cached?/4` hit short-circuits before Cachex
+ is touched. On a disk miss the lazy Strategy fetch is wrapped in
+ `Cachex.fetch/4` on `{:icon_bytes, source, name, shape}` so that
+ concurrent first-callers coalesce onto a single courier; the courier
+ returns `{:ignore, _}` so no entry is committed, and subsequent
+ callers re-read the now-populated file from disk.
+ """
+
+ alias Lightning.Adaptors.Config
+ alias Lightning.Adaptors.IconCache
+ alias Lightning.Adaptors.Repo, as: AdaptorsRepo
+ alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor
+
+ # Strategy and source are scoped to the supervisor instance — every
+ # `Store` call resolves both from the per-instance `:persistent_term`
+ # entry the supervisor populated at boot. No `Application.get_env`
+ # reads in the hot path; no global mutable state in tests.
+
+ @type sup :: atom()
+
+ @type version_meta :: %{
+ version: String.t(),
+ integrity: String.t() | nil,
+ size_bytes: integer() | nil,
+ published_at: DateTime.t() | nil,
+ deprecated: boolean()
+ }
+
+ @type icon_meta :: %{
+ icon_square_ext: String.t() | nil,
+ icon_rectangle_ext: String.t() | nil,
+ icon_square_sha256: binary() | nil,
+ icon_rectangle_sha256: binary() | nil
+ }
+
+ @type package_meta :: AdaptorsRepo.package_meta()
+
+ @doc """
+ Read the `schema_data` JSON blob for a single adaptor.
+
+ Cache-then-Repo-then-Strategy. On Strategy success the full adaptor
+ record is upserted into Postgres and the projected schema blob is
+ committed to the cache.
+
+ Returns the schema as a JSON binary so that ordered-objects decoding
+ can re-engage at the credential-form renderer.
+ """
+ @spec schema(sup(), String.t()) :: {:ok, String.t()} | {:error, term()}
+ def schema(sup, name) do
+ cache = AdaptorsSupervisor.cache_name(sup)
+ source = AdaptorsSupervisor.source(sup)
+
+ cache
+ |> Cachex.fetch(
+ {:schema, name, source},
+ fn _key ->
+ case AdaptorsRepo.get_adaptor(name, source) do
+ %{schema_data: data} when not is_nil(data) ->
+ {:commit, {:ok, data}}
+
+ _ ->
+ fetch_and_persist(sup, name, source, :schema_data)
+ end
+ end,
+ timeout: Config.cache_timeout_ms()
+ )
+ |> unwrap()
+ end
+
+ @doc """
+ Read the version history for a single adaptor as a list of lean
+ per-version maps. See `t:version_meta/0` for the projected shape.
+ """
+ @spec versions(sup(), String.t()) ::
+ {:ok, [version_meta()]} | {:error, term()}
+ def versions(sup, name) do
+ cache = AdaptorsSupervisor.cache_name(sup)
+ source = AdaptorsSupervisor.source(sup)
+
+ cache
+ |> Cachex.fetch(
+ {:versions, name, source},
+ fn _key ->
+ case AdaptorsRepo.list_versions(name, source) do
+ [] -> fetch_and_persist(sup, name, source, :versions)
+ rows -> {:commit, {:ok, project_versions(rows)}}
+ end
+ end,
+ timeout: Config.cache_timeout_ms()
+ )
+ |> unwrap()
+ end
+
+ @doc """
+ Resolve the on-disk path of one icon variant for an adaptor.
+
+ Disk is the cache: a cache-hit on `IconCache.cached?/4` returns the
+ path immediately. A cache-miss is routed through `Cachex.fetch/4` on
+ `{:icon_bytes, source, name, shape}` so concurrent first-callers
+ coalesce onto one in-flight Strategy fetch — the courier returns
+ `{:ignore, _}` so no cache entry is committed and the next miss reads
+ the freshly-written file from disk.
+
+ Returns `{:error, :not_found}` when the icon variant is absent from
+ the adaptor row (the row is the source of truth).
+ """
+ @spec icon(sup(), String.t(), :square | :rectangle) ::
+ {:ok, Path.t()} | {:error, :not_found | term()}
+ def icon(sup, name, shape) when shape in [:square, :rectangle] do
+ cache = AdaptorsSupervisor.cache_name(sup)
+ source = AdaptorsSupervisor.source(sup)
+ strategy = AdaptorsSupervisor.strategy(sup)
+
+ with {:ok, meta} <- icon_meta(sup, name),
+ {:ok, ext} <- ext_for_shape(meta, shape),
+ {:ok, _sha256} <- sha256_for_shape(meta, shape) do
+ if IconCache.cached?(source, name, shape, ext) do
+ {:ok, IconCache.path(source, name, shape, ext)}
+ else
+ cache
+ |> Cachex.fetch(
+ {:icon_bytes, source, name, shape},
+ fn _key -> fetch_icon_bytes(strategy, source, name, shape, ext) end,
+ timeout: Config.cache_timeout_ms()
+ )
+ |> unwrap()
+ end
+ end
+ end
+
+ defp fetch_icon_bytes(strategy, source, name, shape, ext) do
+ case strategy.fetch_icon(name, shape) do
+ {:ok, %{data: bytes, ext: ^ext}} ->
+ {:ok, _sha} = IconCache.write!(source, name, shape, ext, bytes)
+ {:ignore, {:ok, IconCache.path(source, name, shape, ext)}}
+
+ {:ok, %{ext: other_ext}} ->
+ {:ignore, {:error, {:ext_mismatch, expected: ext, got: other_ext}}}
+
+ {:error, _} = err ->
+ {:ignore, err}
+ end
+ end
+
+ @doc """
+ Picker-facing lean projection: every adaptor row for the active
+ source, minus heavy JSONB columns (`schema_data`, `dependencies`,
+ `peer_dependencies`).
+
+ An empty Repo result returns `{:ok, []}` but is **not** committed to
+ the cache — during cold-start the Scheduler will fill the table on
+ its next tick, and the next call will pick that up automatically.
+ """
+ @spec packages(sup()) :: {:ok, [package_meta()]} | {:error, term()}
+ def packages(sup) do
+ cache = AdaptorsSupervisor.cache_name(sup)
+ source = AdaptorsSupervisor.source(sup)
+
+ cache
+ |> Cachex.fetch(
+ {:packages, source},
+ fn _key ->
+ case AdaptorsRepo.list_package_metas(source) do
+ [] -> {:ignore, {:ok, []}}
+ metas -> {:commit, {:ok, metas}}
+ end
+ end,
+ timeout: Config.cache_timeout_ms()
+ )
+ |> unwrap()
+ end
+
+ @doc """
+ Cheap `{icon__ext, icon__sha256}` projection for the
+ icon controller's sha-validation path. Pure metadata — no disk I/O.
+
+ Unknown adaptors return `{:error, :not_found}` and are **not**
+ cached, so a subsequent insert by the Scheduler becomes visible on
+ the very next call.
+ """
+ @spec icon_meta(sup(), String.t()) ::
+ {:ok, icon_meta()} | {:error, :not_found}
+ def icon_meta(sup, name) do
+ cache = AdaptorsSupervisor.cache_name(sup)
+ source = AdaptorsSupervisor.source(sup)
+
+ cache
+ |> Cachex.fetch(
+ {:icon_meta, name, source},
+ fn _key ->
+ case AdaptorsRepo.get_adaptor(name, source) do
+ nil -> {:ignore, {:error, :not_found}}
+ adaptor -> {:commit, {:ok, project_icon_meta(adaptor)}}
+ end
+ end,
+ timeout: Config.cache_timeout_ms()
+ )
+ |> unwrap()
+ end
+
+ @doc """
+ Re-warm Cachex from Postgres for the active source.
+
+ Called by `Lightning.Adaptors.NodeMonitor` on `:nodeup` — a peer
+ rejoining after a partition can't know which `{:changed, name, source}`
+ broadcasts it missed, so it treats its entire local Cachex as
+ suspect and overwrites from the DB.
+
+ Uses `Cachex.put_many/2` (never `Cachex.clear/1`-then-fill) so
+ concurrent callers never observe an empty cache and never trigger a
+ spurious cold-miss Strategy fetch during the warm.
+ """
+ @spec warm_from_repo(sup()) :: :ok
+ def warm_from_repo(sup) do
+ cache = AdaptorsSupervisor.cache_name(sup)
+ source = AdaptorsSupervisor.source(sup)
+
+ metas = AdaptorsRepo.list_package_metas(source)
+
+ icon_metas =
+ Enum.map(metas, fn m ->
+ {{:icon_meta, m.name, source}, {:ok, project_icon_meta(m)}}
+ end)
+
+ Cachex.put_many(
+ cache,
+ [{{:packages, source}, {:ok, metas}} | icon_metas]
+ )
+
+ :ok
+ end
+
+ @spec fetch_and_persist(atom(), String.t(), :npm | :local, atom()) ::
+ {:commit, {:ok, term()}} | {:ignore, {:error, term()}}
+ defp fetch_and_persist(sup, name, source, field) do
+ case AdaptorsSupervisor.strategy(sup).fetch_adaptor(name) do
+ {:ok, record} ->
+ record =
+ record
+ |> Map.put(:source, source)
+ |> normalize_schema_data()
+
+ {:ok, _} = AdaptorsRepo.upsert_adaptor(record)
+ {:commit, {:ok, Map.get(record, field)}}
+
+ {:error, reason} ->
+ {:ignore, {:error, reason}}
+ end
+ end
+
+ # Strategies should emit `schema_data` as a JSON binary, but legacy
+ # call paths (and tests) may still hand us a map. Normalize here so
+ # the cached value matches what subsequent DB-backed reads return.
+ defp normalize_schema_data(%{schema_data: data} = record)
+ when is_map(data) and not is_struct(data) do
+ %{record | schema_data: Jason.encode!(data)}
+ end
+
+ defp normalize_schema_data(record), do: record
+
+ @spec project_icon_meta(map()) :: icon_meta()
+ defp project_icon_meta(adaptor) do
+ Map.take(adaptor, [
+ :icon_square_ext,
+ :icon_rectangle_ext,
+ :icon_square_sha256,
+ :icon_rectangle_sha256
+ ])
+ end
+
+ @spec project_versions([map()]) :: [version_meta()]
+ defp project_versions(rows) do
+ Enum.map(
+ rows,
+ &Map.take(&1, [
+ :version,
+ :integrity,
+ :size_bytes,
+ :published_at,
+ :deprecated
+ ])
+ )
+ end
+
+ @spec ext_for_shape(icon_meta(), :square | :rectangle) ::
+ {:ok, String.t()} | {:error, :not_found}
+ defp ext_for_shape(meta, shape) do
+ case Map.get(meta, :"icon_#{shape}_ext") do
+ nil -> {:error, :not_found}
+ ext -> {:ok, ext}
+ end
+ end
+
+ @spec sha256_for_shape(icon_meta(), :square | :rectangle) ::
+ {:ok, binary()} | {:error, :not_found}
+ defp sha256_for_shape(meta, shape) do
+ case Map.get(meta, :"icon_#{shape}_sha256") do
+ nil -> {:error, :not_found}
+ sha -> {:ok, sha}
+ end
+ end
+
+ # `Cachex.fetch/4` returns one of:
+ # * `{:ok, value}` — cache hit (or coalesced peer of a `:commit`)
+ # * `{:commit, value}` — fallback ran and committed
+ # * `{:ignore, value}` — fallback ran and chose not to cache
+ # * `{:error, term}` — Cachex-side failure (fallback raised, etc.)
+ #
+ # Our fallbacks return `{:commit, {:ok, _}}` / `{:ignore, {:error, _}}`,
+ # so the wrapper tuple's second element is itself the public
+ # `{:ok, _} | {:error, _}` we want to return. Cachex-side `{:error, _}`
+ # passes through unchanged.
+ @spec unwrap(tuple()) :: {:ok, term()} | {:error, term()}
+ defp unwrap({:ok, inner}), do: inner
+ defp unwrap({:commit, inner}), do: inner
+ defp unwrap({:ignore, inner}), do: inner
+ defp unwrap({:error, _} = error), do: error
+end
diff --git a/lib/lightning/adaptors/strategy.ex b/lib/lightning/adaptors/strategy.ex
new file mode 100644
index 00000000000..1a82a73c1f0
--- /dev/null
+++ b/lib/lightning/adaptors/strategy.ex
@@ -0,0 +1,148 @@
+defmodule Lightning.Adaptors.Strategy do
+ @moduledoc """
+ Behaviour shared by every adaptor strategy (NPM, Local, and the test
+ mock).
+
+ A strategy is the sole boundary between the `Lightning.Adaptors.*`
+ subsystem and the outside world. It defines four callbacks:
+
+ * `c:fetch_adaptor/1` — given a package name, return a structured
+ `t:adaptor_record/0` covering version history, integrity hashes,
+ and dependency metadata. Icon fields are **not** part of this
+ record any more; the Scheduler stamps them on after joining the
+ bulk icon pipeline.
+ * `c:fetch_icon/2` — given a package name and an icon variant,
+ return the raw bytes plus extension. Used by the Store's rare
+ lazy-miss fallback.
+ * `c:fetch_icons/1` — bulk icon fetch for every adaptor known to
+ the strategy. The Scheduler invokes this once per tick in parallel
+ with its per-adaptor fan-out. Accepts a keyword list of options;
+ see the callback docs for `:prior_etags`.
+ * `c:list_adaptors/0` — the cheap change-signal: one call returning
+ `name + latest_version` for every `@openfn/*` package, used by
+ the scheduler to diff against the `adaptors` table.
+
+ The active strategy module is resolved at runtime via
+ `Lightning.Adaptors.Config.strategy/0`. Implementations must surface
+ transient failures (5xx, timeout, nxdomain) as `{:error, term()}`;
+ retry policy lives at the scheduler/store layer, not here.
+ """
+
+ @typedoc """
+ Per-version metadata extracted from an upstream packument or local
+ `package.json`.
+ """
+ @type version_record :: %{
+ version: String.t(),
+ integrity: String.t() | nil,
+ tarball_url: String.t() | nil,
+ size_bytes: integer() | nil,
+ dependencies: map(),
+ peer_dependencies: map(),
+ published_at: DateTime.t() | nil,
+ deprecated: boolean()
+ }
+
+ @typedoc """
+ The structured adaptor record returned by `c:fetch_adaptor/1`. Icon
+ fields are persisted separately by the Scheduler after joining
+ `c:fetch_icons/1` — they are not stamped onto this record.
+ """
+ @type adaptor_record :: %{
+ name: String.t(),
+ description: String.t() | nil,
+ homepage: String.t() | nil,
+ repository: String.t() | nil,
+ license: String.t() | nil,
+ latest_version: String.t(),
+ deprecated: boolean(),
+ schema_data: map() | nil,
+ schema_sha256: String.t() | nil,
+ versions: [version_record()]
+ }
+
+ @typedoc """
+ Fresh-fetch icon entry inside the `c:fetch_icons/1` result map. The
+ optional `:etag` field carries the upstream-provided cache validator
+ (verbatim from the HTTP response) and is `nil` when the upstream
+ didn't supply one — strategies without a transport-level validator
+ (e.g. `Lightning.Adaptors.Local`) omit the key entirely.
+ """
+ @type icon_entry :: %{
+ required(:data) => binary(),
+ required(:ext) => String.t(),
+ required(:sha256) => binary(),
+ optional(:etag) => String.t() | nil
+ }
+
+ @typedoc """
+ Per-shape value inside the `c:fetch_icons/1` result map. Either a
+ fresh `t:icon_entry/0` (200 response) or the `:not_modified` sentinel
+ (304 response — upstream confirmed unchanged; only ever returned when
+ the caller supplied a prior etag via the `:prior_etags` option).
+ """
+ @type icon_shape_value :: icon_entry() | :not_modified
+
+ @typedoc """
+ Bulk icon map returned by `c:fetch_icons/1`. Three branches matter:
+
+ * shape **entirely absent** — upstream had no such icon for this
+ package;
+ * shape present as `:not_modified` — upstream confirmed the icon
+ is unchanged since the prior etag was issued;
+ * shape present as a map — apply the bytes (a fresh fetch).
+ """
+ @type icons_map :: %{
+ required(String.t()) => %{
+ optional(:square) => icon_shape_value(),
+ optional(:rectangle) => icon_shape_value()
+ }
+ }
+
+ @doc """
+ Fetch the full structured record for a single adaptor package.
+ """
+ @callback fetch_adaptor(name :: String.t()) ::
+ {:ok, adaptor_record()} | {:error, term()}
+
+ @doc """
+ Fetch the raw bytes for one icon variant (`:square` or `:rectangle`)
+ of an adaptor package, together with the file extension.
+ """
+ @callback fetch_icon(name :: String.t(), :square | :rectangle) ::
+ {:ok, %{data: binary(), ext: String.t()}}
+ | {:error, term()}
+
+ @doc """
+ Bulk fetch every available icon for every adaptor known to the
+ strategy.
+
+ Returns `{:ok, partial_map}` where each per-shape slot is either
+ absent (no icon upstream), a fresh `t:icon_entry/0` (200), or the
+ `:not_modified` sentinel (304 — only when a prior etag was sent).
+ A top-level `{:error, term()}` is only returned when the whole
+ pipeline can't proceed (e.g. an upstream `list_adaptors/0` call
+ inside the bulk implementation fails).
+
+ ## Options
+
+ * `:prior_etags` — a map of the form
+ `%{name => %{optional(:square | :rectangle) => etag_string}}`
+ whose values are sent as `If-None-Match` per `(name, shape)`.
+ Defaults to `%{}`. Unknown keys in the keyword list are
+ ignored. Strategies without a transport-level cache validator
+ (e.g. `Lightning.Adaptors.Local`) ignore this option entirely
+ and never return `:not_modified`.
+ """
+ @callback fetch_icons(opts :: keyword()) ::
+ {:ok, icons_map()} | {:error, term()}
+
+ @doc """
+ Cheap change-signal listing: `name + latest_version` for every
+ `@openfn/*` package known to the strategy. The scheduler diffs this
+ against the `adaptors` table to compute its work list.
+ """
+ @callback list_adaptors() ::
+ {:ok, [%{name: String.t(), latest_version: String.t()}]}
+ | {:error, term()}
+end
diff --git a/lib/lightning/adaptors/supervisor.ex b/lib/lightning/adaptors/supervisor.ex
new file mode 100644
index 00000000000..c1691630854
--- /dev/null
+++ b/lib/lightning/adaptors/supervisor.ex
@@ -0,0 +1,242 @@
+defmodule Lightning.Adaptors.Supervisor do
+ @moduledoc """
+ Per-instance supervisor for the `Lightning.Adaptors.*` subsystem.
+
+ The entire subsystem boots, crashes, and is supervised as a unit
+ under `:rest_for_one`. `Cachex` is the load-bearing root: if it
+ crashes, the supervisor restarts it and cascades to its dependents
+ (`Task.Supervisor`, plus the broadcaster/scheduler children added in
+ later phases) so they re-bind to the fresh Cachex name on the way
+ back up.
+
+ No registered name, Cachex table name, PubSub topic, `Task.Supervisor`
+ name, or `HighlanderPG` lock key is hardcoded. Every name is derived
+ from a single `:name` opt — which is what lets the integration suite
+ spin up multiple isolated instances inside one BEAM for
+ `async: true` tests. Production starts exactly one instance under
+ `name: Lightning.Adaptors`.
+
+ ## Cluster-singleton Scheduler
+
+ The `Lightning.Adaptors.Scheduler` is wrapped in `HighlanderPG`
+ (`pg_try_advisory_lock` on `lock_key/1`) so exactly one node in a
+ multi-node deployment runs the refresh tick. The inner Scheduler
+ registers under `{:global, global_scheduler_name(name)}`; callers on
+ any node hit the leader transparently via Erlang distribution.
+
+ ## Strategy injection
+
+ The active `Lightning.Adaptors.Strategy` implementation is passed in
+ explicitly via the `:strategy` opt. Tests instantiate an isolated
+ supervisor with `strategy: Lightning.Adaptors.StrategyMock` — no
+ `Application.put_env` mutation, no shared mutable state. The
+ production caller in `lib/lightning/application.ex` passes the
+ default from `Lightning.Adaptors.Config.strategy/0` (resolved from
+ Application env at boot time).
+
+ `strategy/1` and `source/1` expose the per-instance values back to
+ the stateless `Lightning.Adaptors.Store` callers.
+ """
+
+ use Supervisor
+
+ alias Lightning.Adaptors.Config
+
+ @doc """
+ Start a supervisor instance.
+
+ Required opts:
+
+ * `:name` — supervisor instance name (atom). Derives every child
+ name via `Module.concat/2`.
+
+ Optional opts:
+
+ * `:strategy` — `Lightning.Adaptors.Strategy` implementation.
+ Defaults to `Lightning.Adaptors.Config.strategy/0`.
+
+ * `:lock_key` — explicit `HighlanderPG` advisory-lock key. Defaults
+ to `lock_key(name)`. Override only in integration tests where
+ multiple supervisor instances must compete for the same lock.
+ """
+ @spec start_link(keyword()) :: Supervisor.on_start()
+ def start_link(opts) do
+ name = Keyword.fetch!(opts, :name)
+ Supervisor.start_link(__MODULE__, opts, name: name)
+ end
+
+ @impl true
+ def init(opts) do
+ name = Keyword.fetch!(opts, :name)
+ strategy = Keyword.get(opts, :strategy, Config.strategy())
+ lock_key = Keyword.get(opts, :lock_key, lock_key(name))
+
+ :persistent_term.put(meta_key(name), %{
+ strategy: strategy,
+ source: source_for(strategy)
+ })
+
+ cache = cache_name(name)
+ tasks = tasks_name(name)
+ source_topic = source_topic(name)
+ client_topic = client_topic(name)
+
+ scheduler_child =
+ %{
+ id: Lightning.Adaptors.Scheduler,
+ start:
+ {Lightning.Adaptors.Scheduler, :start_link,
+ [
+ [
+ name: global_scheduler_name(name),
+ sup: name,
+ lock_key: lock_key,
+ cache: cache,
+ tasks: tasks,
+ source_topic: source_topic
+ ]
+ ]}
+ }
+
+ children = [
+ {Cachex, name: cache},
+ {Task.Supervisor, name: tasks},
+ {Lightning.Adaptors.Invalidator,
+ name: invalidator_name(name), source_topic: source_topic, cache: cache},
+ {Lightning.Adaptors.NodeMonitor, name: node_monitor_name(name), sup: name},
+ {Lightning.Adaptors.ChannelBroadcaster,
+ name: channel_broadcaster_name(name),
+ source_topic: source_topic,
+ client_topic: client_topic,
+ sup: name},
+ Supervisor.child_spec(
+ {HighlanderPG,
+ child: scheduler_child,
+ repo: Lightning.Repo,
+ name: lock_key,
+ sup_name: highlander_name(name)},
+ id: highlander_name(name)
+ )
+ ]
+
+ Supervisor.init(children, strategy: :rest_for_one)
+ end
+
+ @doc """
+ The active strategy for the supervisor instance named `name`.
+
+ Reads from `:persistent_term` populated at `init/1`. Raises if the
+ supervisor has not been started under that name.
+ """
+ @spec strategy(atom()) :: module()
+ def strategy(name) do
+ :persistent_term.get(meta_key(name)).strategy
+ end
+
+ @doc """
+ The active source (`:npm | :local`) for the supervisor instance
+ named `name`.
+ """
+ @spec source(atom()) :: :npm | :local
+ def source(name) do
+ :persistent_term.get(meta_key(name)).source
+ end
+
+ @doc """
+ Best-effort cleanup of the per-instance `:persistent_term` entry.
+
+ Not called automatically — `:persistent_term.erase/1` triggers a
+ global GC and is expensive enough that we leave it to deliberate
+ teardown paths (e.g. release shutdown).
+ """
+ @spec forget(atom()) :: boolean()
+ def forget(name) do
+ :persistent_term.erase(meta_key(name))
+ end
+
+ @doc "Cachex table name for the supervisor named `name`."
+ @spec cache_name(atom()) :: atom()
+ def cache_name(name), do: Module.concat(name, Cache)
+
+ @doc "`Task.Supervisor` name for the supervisor named `name`."
+ @spec tasks_name(atom()) :: atom()
+ def tasks_name(name), do: Module.concat(name, Tasks)
+
+ @doc "`Invalidator` GenServer name for the supervisor named `name`."
+ @spec invalidator_name(atom()) :: atom()
+ def invalidator_name(name), do: Module.concat(name, Invalidator)
+
+ @doc "`ChannelBroadcaster` GenServer name for the supervisor named `name`."
+ @spec channel_broadcaster_name(atom()) :: atom()
+ def channel_broadcaster_name(name),
+ do: Module.concat(name, ChannelBroadcaster)
+
+ @doc "`NodeMonitor` GenServer name for the supervisor named `name`."
+ @spec node_monitor_name(atom()) :: atom()
+ def node_monitor_name(name), do: Module.concat(name, NodeMonitor)
+
+ @doc """
+ Local `Scheduler` GenServer name for the supervisor named `name`.
+
+ The inner Scheduler is actually registered globally — see
+ `global_scheduler_name/1`. This atom form is retained as the
+ child-spec `id` and for derived module names.
+ """
+ @spec scheduler_name(atom()) :: atom()
+ def scheduler_name(name), do: Module.concat(name, Scheduler)
+
+ @doc """
+ `:global`-registered Scheduler name for the supervisor named `name`.
+
+ The HighlanderPG-wrapped Scheduler registers itself under this name so
+ callers on any node reach the leader via Erlang distribution. Pass the
+ return value to `GenServer.call/3` directly.
+ """
+ @spec global_scheduler_name(atom()) :: {:global, atom()}
+ def global_scheduler_name(name), do: {:global, scheduler_name(name)}
+
+ @doc """
+ `HighlanderPG` supervisor name for the supervisor named `name`.
+
+ Used as the child-spec id and the `:sup_name` for introspection
+ (`HighlanderPG.which_children/1`, etc.).
+ """
+ @spec highlander_name(atom()) :: atom()
+ def highlander_name(name), do: Module.concat(name, HighlanderPG)
+
+ @doc """
+ Source-side PubSub topic for the supervisor named `name`.
+
+ Used by the `Scheduler` and `Invalidator` to broadcast and receive
+ `{:changed, name, source}` style events.
+ """
+ @spec source_topic(atom()) :: String.t()
+ def source_topic(name), do: "adaptors:#{inspect(name)}"
+
+ @doc """
+ Client-side PubSub topic for the supervisor named `name`.
+
+ The `ChannelBroadcaster` republishes throttled updates from
+ `source_topic/1` onto this topic for `WorkflowChannel` subscribers.
+ """
+ @spec client_topic(atom()) :: String.t()
+ def client_topic(name), do: "adaptors:client_update:#{inspect(name)}"
+
+ @doc """
+ Postgres advisory-lock key for the supervisor named `name`.
+
+ Derived as `:erlang.phash2({:adaptors, name})` so each supervisor
+ instance leases its `HighlanderPG`-wrapped `Scheduler` against a
+ distinct `int4` key — two concurrent test supervisors with different
+ names cannot collide on advisory locks. The §12.7 integration test
+ overrides `:lock_key` on `start_link/1` to force two supervisors to
+ compete for the same lock.
+ """
+ @spec lock_key(atom()) :: non_neg_integer()
+ def lock_key(name), do: :erlang.phash2({:adaptors, name})
+
+ defp meta_key(name), do: {__MODULE__, name}
+
+ defp source_for(Lightning.Adaptors.Local), do: :local
+ defp source_for(_other), do: :npm
+end
diff --git a/lib/lightning/ai_assistant/ai_assistant.ex b/lib/lightning/ai_assistant/ai_assistant.ex
index b95886e1a4f..b6ad226feeb 100644
--- a/lib/lightning/ai_assistant/ai_assistant.ex
+++ b/lib/lightning/ai_assistant/ai_assistant.ex
@@ -614,7 +614,7 @@ defmodule Lightning.AiAssistant do
%{
session
| expression: expression,
- adaptor: Lightning.AdaptorRegistry.resolve_adaptor(adaptor)
+ adaptor: Lightning.Adaptors.PackageName.to_wire(adaptor)
}
end
diff --git a/lib/lightning/application.ex b/lib/lightning/application.ex
index 2ab64a218a0..e915ae1c2fc 100644
--- a/lib/lightning/application.ex
+++ b/lib/lightning/application.ex
@@ -148,6 +148,7 @@ defmodule Lightning.Application do
LightningWeb.WorkerPresence,
adaptor_registry_childspec,
adaptor_service_childspec,
+ {Lightning.Adaptors.Supervisor, name: Lightning.Adaptors},
{Lightning.TaskWorker, name: :cli_task_worker},
{Lightning.Runtime.RuntimeManager,
worker_secret: Lightning.Config.worker_secret(),
diff --git a/lib/lightning/credentials.ex b/lib/lightning/credentials.ex
index ed237b5d75c..f1ded9a6ed5 100644
--- a/lib/lightning/credentials.ex
+++ b/lib/lightning/credentials.ex
@@ -575,15 +575,12 @@ defmodule Lightning.Credentials do
"""
@spec get_schema(String.t()) :: Credentials.Schema.t()
def get_schema(schema_name) do
- {:ok, schemas_path} = Application.fetch_env(:lightning, :schemas_path)
-
- File.read("#{schemas_path}/#{schema_name}.json")
- |> case do
- {:ok, raw_json} ->
- Credentials.Schema.new(raw_json, schema_name)
+ case Lightning.Adaptors.schema(schema_name) do
+ {:ok, schema_body} ->
+ Credentials.Schema.new(schema_body, schema_name)
{:error, reason} ->
- raise "Error reading credential schema. Got: #{reason |> inspect()}"
+ raise "Error reading credential schema. Got: #{inspect(reason)}"
end
end
diff --git a/lib/lightning_web/channels/run_with_options.ex b/lib/lightning_web/channels/run_with_options.ex
index 643466ee291..18aaff4941a 100644
--- a/lib/lightning_web/channels/run_with_options.ex
+++ b/lib/lightning_web/channels/run_with_options.ex
@@ -1,7 +1,7 @@
defmodule LightningWeb.RunWithOptions do
@moduledoc false
- alias Lightning.AdaptorRegistry
+ alias Lightning.Adaptors.PackageName
alias Lightning.Run
alias Lightning.Workflows.Snapshot.Edge
alias Lightning.Workflows.Snapshot.Job
@@ -40,7 +40,7 @@ defmodule LightningWeb.RunWithOptions do
def render(%Job{} = job) do
%{
"id" => job.id,
- "adaptor" => AdaptorRegistry.resolve_adaptor(job.adaptor),
+ "adaptor" => PackageName.to_wire(job.adaptor),
"credential_id" => get_credential_id(job),
"body" => job.body,
"name" => job.name
diff --git a/lib/lightning_web/channels/workflow_channel.ex b/lib/lightning_web/channels/workflow_channel.ex
index fe2283f8141..48fe065bd19 100644
--- a/lib/lightning_web/channels/workflow_channel.ex
+++ b/lib/lightning_web/channels/workflow_channel.ex
@@ -16,7 +16,6 @@ defmodule LightningWeb.WorkflowChannel do
alias Lightning.Repo
alias Lightning.VersionControl
alias Lightning.VersionControl.VersionControlUsageLimiter
- alias Lightning.Workflows.Job
alias Lightning.Workflows.Snapshot
alias Lightning.Workflows.Workflow
alias Lightning.Workflows.WorkflowUsageLimiter
@@ -72,6 +71,11 @@ defmodule LightningWeb.WorkflowChannel do
"workflow:collaborate:#{workflow_id}"
)
+ Phoenix.PubSub.subscribe(
+ Lightning.PubSub,
+ Lightning.Adaptors.Supervisor.client_topic(Lightning.Adaptors)
+ )
+
{:ok,
assign(socket,
workflow_id: workflow_id,
@@ -96,40 +100,11 @@ defmodule LightningWeb.WorkflowChannel do
@impl true
def handle_in("request_adaptors", _payload, socket) do
async_task(socket, "request_adaptors", fn ->
- adaptors = Lightning.AdaptorRegistry.all()
- %{adaptors: adaptors}
- end)
- end
+ adaptors =
+ list_all_packages()
+ |> Enum.map(&with_icon_urls/1)
- @impl true
- def handle_in("request_project_adaptors", _payload, socket) do
- project = socket.assigns.project
-
- async_task(socket, "request_project_adaptors", fn ->
- project_adaptor_names =
- from(j in Job,
- join: w in assoc(j, :workflow),
- where: w.project_id == ^project.id,
- select: j.adaptor,
- distinct: true
- )
- |> Lightning.Repo.all()
- |> Enum.sort()
-
- all_adaptors = Lightning.AdaptorRegistry.all()
-
- project_adaptors =
- all_adaptors
- |> Enum.filter(fn adaptor ->
- Enum.any?(project_adaptor_names, fn used_adaptor ->
- String.starts_with?(used_adaptor, adaptor.name)
- end)
- end)
-
- %{
- project_adaptors: project_adaptors,
- all_adaptors: all_adaptors
- }
+ %{adaptors: adaptors}
end)
end
@@ -698,6 +673,12 @@ defmodule LightningWeb.WorkflowChannel do
{:noreply, socket}
end
+ @impl true
+ def handle_info(%{event: "adaptors_updated", payload: payload}, socket) do
+ push(socket, "adaptors_updated", payload)
+ {:noreply, socket}
+ end
+
@impl true
def handle_info(
%{event: "webhook_auth_methods_updated", payload: webhook_auth_methods},
@@ -831,6 +812,30 @@ defmodule LightningWeb.WorkflowChannel do
{:noreply, socket}
end
+ defp list_all_packages do
+ case Lightning.Adaptors.packages() do
+ {:ok, pkgs} -> pkgs
+ {:error, _} -> []
+ end
+ end
+
+ defp with_icon_urls(adaptor) do
+ Map.put(adaptor, :icon_urls, icon_urls_for(adaptor.name))
+ end
+
+ defp icon_urls_for(name) do
+ case Lightning.Adaptors.icon_meta(name) do
+ {:ok, meta} ->
+ %{
+ square: LightningWeb.AdaptorIconURL.build(name, meta, :square),
+ rectangle: LightningWeb.AdaptorIconURL.build(name, meta, :rectangle)
+ }
+
+ {:error, :not_found} ->
+ %{square: nil, rectangle: nil}
+ end
+ end
+
defp handle_async_event("request_run_steps", socket_ref, reply) do
unwrapped_reply = unwrap_run_steps_reply(reply)
reply(socket_ref, unwrapped_reply)
@@ -839,7 +844,6 @@ defmodule LightningWeb.WorkflowChannel do
defp handle_async_event(event, socket_ref, reply)
when event in [
"request_adaptors",
- "request_project_adaptors",
"request_credentials",
"request_metadata",
"request_current_user",
diff --git a/lib/lightning_web/components/layouts/settings.html.heex b/lib/lightning_web/components/layouts/settings.html.heex
index 2fabf7de552..4b8226ab2f2 100644
--- a/lib/lightning_web/components/layouts/settings.html.heex
+++ b/lib/lightning_web/components/layouts/settings.html.heex
@@ -73,6 +73,13 @@
<.icon name="hero-circle-stack" class="h-5 w-5 shrink-0" />
Collections
+
+ <.icon name="hero-wrench-screwdriver" class="h-5 w-5 shrink-0" />
+ Maintenance
+ binary_part(0, 4) |> Base.encode16(case: :lower)
+
+ "/adaptors/icons/#{URI.encode(name, &URI.char_unreserved?/1)}/" <>
+ "#{shape}-#{sha8}.#{ext}"
+ else
+ _ -> nil
+ end
+ end
+end
+
+defmodule LightningWeb.AdaptorIconController do
+ @moduledoc """
+ Serves content-addressable adaptor icons.
+
+ Route: `/adaptors/icons/:name/:shape-:sha8.:ext`
+
+ `sha8` is the first 4 raw bytes of the stored sha256 hex-encoded to
+ 8 lowercase characters. The controller compares `sha8` against the
+ DB-projected metadata and responds with one of:
+
+ - **200** — sha matches; serves bytes with a 1-year immutable cache.
+ - **302** — sha is stale but the adaptor still has an icon; redirects
+ to the canonical (current-sha) URL with `Cache-Control: no-store`
+ on the redirect itself.
+ - **404** — adaptor unknown, ext mismatch, bad shape, or no icon.
+ """
+
+ use LightningWeb, :controller
+
+ alias Lightning.Adaptors
+
+ @immutable_cache "public, max-age=31536000, immutable"
+
+ # Router-shaped params: a single `:filename` segment of the form
+ # `-.` because Phoenix path matchers permit only one
+ # dynamic segment per path component. We split here and delegate to the
+ # 4-key clause below, which is also what the unit tests call directly.
+ @filename_regex ~r/\A(?[a-z]+)-(?[A-Fa-f0-9]+)\.(?[A-Za-z0-9]+)\z/
+
+ @doc false
+ def show(conn, %{"name" => name, "filename" => filename}) do
+ case Regex.named_captures(@filename_regex, filename) do
+ %{"shape" => shape, "sha8" => sha8, "ext" => ext} ->
+ show(conn, %{
+ "name" => name,
+ "shape" => shape,
+ "sha8" => sha8,
+ "ext" => ext
+ })
+
+ _ ->
+ send_resp(conn, 404, "")
+ end
+ end
+
+ def show(
+ conn,
+ %{"name" => name, "shape" => shape, "sha8" => sha8, "ext" => ext}
+ )
+ when shape in ~w(square rectangle) do
+ case Adaptors.icon_meta(name) do
+ {:error, :not_found} ->
+ send_resp(conn, 404, "")
+
+ {:ok, meta} ->
+ cond do
+ ext_for_shape(meta, shape) != ext ->
+ send_resp(conn, 404, "")
+
+ not has_icon?(meta, shape) ->
+ send_resp(conn, 404, "")
+
+ sha_matches?(meta, shape, sha8) ->
+ serve_bytes(conn, name, shape, ext)
+
+ true ->
+ redirect_to_current(conn, name, meta, shape)
+ end
+ end
+ end
+
+ def show(conn, _params), do: send_resp(conn, 404, "")
+
+ defp serve_bytes(conn, name, shape, ext) do
+ case Adaptors.icon(name, String.to_existing_atom(shape)) do
+ {:ok, path} ->
+ conn
+ |> put_resp_content_type(content_type_for(ext))
+ |> put_resp_header("cache-control", @immutable_cache)
+ |> send_file(200, path)
+
+ {:error, _} ->
+ send_resp(conn, 404, "")
+ end
+ end
+
+ defp redirect_to_current(conn, name, meta, shape) do
+ url =
+ LightningWeb.AdaptorIconURL.build(
+ name,
+ meta,
+ String.to_existing_atom(shape)
+ )
+
+ conn
+ |> put_resp_header("cache-control", "no-store")
+ |> put_resp_header("location", url)
+ |> send_resp(302, "")
+ end
+
+ defp has_icon?(meta, shape), do: not is_nil(ext_for_shape(meta, shape))
+
+ defp ext_for_shape(meta, shape), do: Map.get(meta, :"icon_#{shape}_ext")
+
+ defp sha_matches?(meta, shape, sha8) do
+ case Map.get(meta, :"icon_#{shape}_sha256") do
+ <> ->
+ Base.encode16(prefix, case: :lower) == String.downcase(sha8)
+
+ _ ->
+ false
+ end
+ end
+
+ defp content_type_for("png"), do: "image/png"
+ defp content_type_for("svg"), do: "image/svg+xml"
+end
diff --git a/lib/lightning_web/live/job_live/adaptor_picker.ex b/lib/lightning_web/live/job_live/adaptor_picker.ex
index 6a6460c92c1..d4ace5565f3 100644
--- a/lib/lightning_web/live/job_live/adaptor_picker.ex
+++ b/lib/lightning_web/live/job_live/adaptor_picker.ex
@@ -109,7 +109,7 @@ defmodule LightningWeb.JobLive.AdaptorPicker do
|> assign_new(
:local_adaptors_enabled?,
fn ->
- Lightning.AdaptorRegistry.local_adaptors_enabled?()
+ Lightning.Adaptors.Config.current_source() == :local
end
)}
end
@@ -133,19 +133,19 @@ defmodule LightningWeb.JobLive.AdaptorPicker do
def get_adaptor_version_options(adaptor) do
adaptor_names =
- Lightning.AdaptorRegistry.all()
+ list_all_adaptors()
|> Enum.map(&display_name_for_adaptor(&1.name))
|> Enum.sort()
{module_name, version, versions} =
if adaptor do
{module_name, version} =
- Lightning.AdaptorRegistry.resolve_package_name(adaptor)
+ Lightning.Adaptors.PackageName.parse(adaptor)
- latest = Lightning.AdaptorRegistry.latest_for(module_name)
+ latest = latest_for(module_name)
versions =
- Lightning.AdaptorRegistry.versions_for(module_name)
+ versions_for(module_name)
|> List.wrap()
|> Enum.map(&Map.get(&1, :version))
|> sort_versions_desc()
@@ -177,6 +177,31 @@ defmodule LightningWeb.JobLive.AdaptorPicker do
[key: version, value: "#{module_name}@#{version}"]
end
+ defp list_all_adaptors do
+ case Lightning.Adaptors.packages() do
+ {:ok, pkgs} -> pkgs
+ {:error, _} -> []
+ end
+ end
+
+ defp latest_for(nil), do: nil
+
+ defp latest_for(name) do
+ case Lightning.Adaptors.resolve_version(name, "latest") do
+ {:ok, version} -> version
+ {:error, _} -> nil
+ end
+ end
+
+ defp versions_for(nil), do: []
+
+ defp versions_for(name) do
+ case Lightning.Adaptors.versions(name) do
+ {:ok, versions} -> versions
+ {:error, _} -> []
+ end
+ end
+
@doc "Sort version strings in descending semver order."
@spec sort_versions_desc([String.t()]) :: [String.t()]
def sort_versions_desc(versions) when is_list(versions) do
@@ -192,7 +217,7 @@ defmodule LightningWeb.JobLive.AdaptorPicker do
socket
) do
# Get the latest specific version instead of using @latest
- latest_version = Lightning.AdaptorRegistry.latest_for(value)
+ latest_version = latest_for(value)
adaptor_value =
if latest_version do
diff --git a/lib/lightning_web/live/maintenance_live/index.ex b/lib/lightning_web/live/maintenance_live/index.ex
new file mode 100644
index 00000000000..4a6ab504b6b
--- /dev/null
+++ b/lib/lightning_web/live/maintenance_live/index.ex
@@ -0,0 +1,85 @@
+defmodule LightningWeb.MaintenanceLive.Index do
+ @moduledoc """
+ Superuser-only maintenance page exposing on-demand operations against the
+ `Lightning.Adaptors` supervisor (and, eventually, other facades).
+
+ Currently exposes a single action: "Refresh Adaptor Registry", which calls
+ `Lightning.Adaptors.refresh_now/0`. The call is fire-and-forget; the user
+ receives a flash and the actual work happens asynchronously on the leader
+ node.
+ """
+ use LightningWeb, :live_view
+
+ alias Lightning.Policies.Permissions
+ alias Lightning.Policies.Users
+
+ @impl true
+ def mount(_params, _session, socket) do
+ if superuser?(socket) do
+ {:ok,
+ assign(socket,
+ active_menu_item: :maintenance,
+ page_title: "Maintenance"
+ ), layout: {LightningWeb.Layouts, :settings}}
+ else
+ {:ok,
+ socket
+ |> put_flash(:nav, :no_access)
+ |> push_navigate(to: "/projects")}
+ end
+ end
+
+ @impl true
+ def handle_event("refresh_adaptors", _params, socket) do
+ if superuser?(socket) do
+ socket =
+ case Lightning.Adaptors.refresh_now() do
+ :ok ->
+ put_flash(socket, :info, "Adaptor refresh queued.")
+
+ {:error, reason} ->
+ put_flash(socket, :error, "Refresh failed: #{inspect(reason)}")
+ end
+
+ {:noreply, socket}
+ else
+ {:noreply,
+ socket
+ |> put_flash(:nav, :no_access)
+ |> push_navigate(to: "/projects")}
+ end
+ end
+
+ def handle_event("refresh_icons", _params, socket) do
+ if superuser?(socket) do
+ socket =
+ case Lightning.Adaptors.refresh_icons() do
+ {:ok, %{updated: updated, unchanged: unchanged}} ->
+ put_flash(
+ socket,
+ :info,
+ "Icon refresh complete — #{updated} updated, #{unchanged} unchanged."
+ )
+
+ {:error, reason} ->
+ put_flash(socket, :error, "Icon refresh failed: #{inspect(reason)}")
+ end
+
+ {:noreply, socket}
+ else
+ {:noreply,
+ socket
+ |> put_flash(:nav, :no_access)
+ |> push_navigate(to: "/projects")}
+ end
+ end
+
+ defp superuser?(socket) do
+ Permissions.can?(
+ Users,
+ :access_admin_space,
+ socket.assigns.current_user,
+ {}
+ )
+ end
+end
diff --git a/lib/lightning_web/live/maintenance_live/index.html.heex b/lib/lightning_web/live/maintenance_live/index.html.heex
new file mode 100644
index 00000000000..eb018839448
--- /dev/null
+++ b/lib/lightning_web/live/maintenance_live/index.html.heex
@@ -0,0 +1,53 @@
+
+ <:header>
+
+ <:title>Maintenance
+
+
+
+
+
+
+
+ Refresh Adaptor Registry
+
+
+ Re-fetch the list of available adaptors and their versions.
+