diff --git a/.claude/agents/phoenix-elixir-expert.md b/.claude/agents/phoenix-elixir-expert.md index f342a5bb02c..d32d4e8e205 100644 --- a/.claude/agents/phoenix-elixir-expert.md +++ b/.claude/agents/phoenix-elixir-expert.md @@ -29,6 +29,15 @@ You are a **battle-tested Elixir/Phoenix architect** with deep expertise in the - For transaction and prelim-type rules when touching y_ex from Elixir, see `.claude/guidelines/yex-guidelines.md §Transaction Deadlock Rules` and `§Prelim Types`. +## OTP / supervision trees + +- When writing or reviewing a supervisor, GenServer, or named process, see + `.claude/guidelines/testable-supervision-trees.md §0. The principle` and + `§4. Anti-patterns checklist`. Names and collaborators are *parameters*, not + constants — don't bake `name: __MODULE__` into a process or resolve + dependencies from global state, or you force the suite serial (breaks + `async: true`). + ## Lightning Project Context **Architecture Awareness:** diff --git a/.claude/guidelines/testable-supervision-trees.md b/.claude/guidelines/testable-supervision-trees.md new file mode 100644 index 00000000000..f567f017e96 --- /dev/null +++ b/.claude/guidelines/testable-supervision-trees.md @@ -0,0 +1,323 @@ +--- +type: reference +status: active +date: 2026-05-19 +related: + - "[[elixir]]" +tags: + - elixir + - otp + - testing +--- + +# Testable Supervision Trees & Named Processes + +Guidance for building supervision trees, supervisors, GenServers and named +processes that are **uniquely addressable and isolated in tests** (so the suite +runs `async: true`) but **need no name in normal production use**. + +This exists because AI harnesses (and people in a hurry) reliably reach for the +shortcut — `name: __MODULE__` baked into the process, dependencies fished out +of global state — which works in dev, looks idiomatic, and quietly forces the +whole test suite serial. Point Claude at this doc when generating or reviewing +OTP code. + +--- + +## 0. The principle + +**A process's name, and its dependencies, are *parameters* — not constants.** + +Every failure mode here traces to one shortcut: hardcoding the name inside the +process, or resolving a collaborator from global state at call time, instead of +threading it through structure (supervision wiring, `start_link` opts, process +state, the caller signature). + +Useful framing from Gray & Tate, *Designing Elixir Systems with OTP* ("Do Fun +Things with Big, Loud Worker-Bees"): push logic into a pure functional core +that needs no processes to test, so the GenServer is a thin shell. Everything +below is damage control for the thin shell that remains. + +> **Scope guard.** Often the cleanest fix is that a thing never needed to be a +> process or a stored value at all. That's worth one sentence at the design +> review, then move on. The objective of this document is the **caller +> signature and how information reaches child processes** — not relitigating +> whether something should be a GenServer. + +--- + +## 1. The 101 case: fixed children, no Registry + +A `Registry` is a lookup table from a **domain key → pid**. You need one only +when *all three* hold: + +1. an **arbitrary / open-ended** number of the process exists, +2. keyed by **runtime data** (a workflow id, a session id), and +3. the code that must talk to one **does not already hold its pid**. + +A supervisor with a **fixed, known set of children** — one of each — fails all +three. It needs nothing. The realisation: + +> For a constant set of children, **the registered module-atom name *is* your +> registry, and it's free.** A registered name already survives restarts — the +> supervisor brings the child back and it re-registers the same atom. That is +> the one feature people reach to `Registry` for. + +The whole 101 pattern, no Registry, fully async-isolatable: + +```elixir +defmodule MyApp.Cache do + use GenServer + + # name is an OPTION, defaulted — never hardcoded inside the module + def start_link(opts) do + {name, opts} = Keyword.pop(opts, :name, __MODULE__) + GenServer.start_link(__MODULE__, opts, name: name) + end + + # API takes the server ref FIRST, defaulted to the singleton + def fetch(server \\ __MODULE__, key), do: GenServer.call(server, {:fetch, key}) + + @impl true + def init(opts) do + # dependencies are injected, with prod defaults — never read from a global + {:ok, %{store: %{}, http: Keyword.get(opts, :http, MyApp.HTTP)}} + end +end +``` + +Production (in the app supervisor's fixed child list) gets the singleton for +free; callers write `MyApp.Cache.fetch(key)` and never name anything. The test +never *looks anything up* — it **holds the pid it just started**: + +```elixir +test "expires entries" do + pid = start_supervised!({MyApp.Cache, name: nil, http: HTTPMock}) + assert MyApp.Cache.fetch(pid, :missing) == nil +end +``` + +`name: nil` is the trick: it starts an anonymous, isolated instance even when +the app already booted a global `MyApp.Cache`, so there is no +`{:already_started, _}` clash, no need to gut `application.ex` in test config, +and the test stays `async: true`. (Only reach for config-driven "don't boot it +in `:test`" when something you *cannot* hand a pid — a Plug, a distant caller — +calls the API with the default name. That is the exception, not the default.) + +### Argument order convention: server ref first, with a default + +This is the OTP-wide convention and it is near-universal: +`GenServer.call(server, …)`, `Agent.get(agent, …)`, +`Registry.lookup(registry, …)`, `Phoenix.PubSub.broadcast(pubsub, …)`, +`Oban.insert(name \\ Oban, changeset)`, `Mox.allow(mock, …)`, +`Ecto.Adapters.SQL.Sandbox.allow(repo, …)` — the addressed thing is the +**subject**, so it leads. + +Critically, **first-with-default is the idiom that produces the `/1` + `/2` +pair cleanly**: + +```elixir +def fetch(server \\ __MODULE__, key) +# fetch(key) -> arity 1, server defaults to __MODULE__ (production) +# fetch(pid, key) -> arity 2, explicit instance (test) +``` + +Production never names anything; the test injects a pid through the *same* +function. **The signature is the injection seam.** That is why this convention +is load-bearing, not stylistic. + +The one principled exception: **pipeline-first APIs put the instance last** — +`Finch.build(:get, url) |> Finch.request(MyFinch)` — because the data being +transformed is the subject and the instance is configuration, so it reads in a +pipe. Rule of thumb: *callers piping data through it → instance last; callers +addressing a process → instance first (the common case).* + +--- + +## 2. The two axes (the centrepiece) + +People — and AI harnesses — reliably weld together two **independent** problems. +Welding them is what makes refactors thrash. Keep them apart: + +1. **Axis 1 — where does per-instance config / identity live?** +2. **Axis 2 — which process actually invokes the injected dependency?** + +`:persistent_term` (and runtime `Application.put_env`, and naked global ETS) is +the wrong answer to **Axis 1**. The `set_mox_global` / `async: false` / +`Sandbox.allow` pain is entirely **Axis 2**. Fixing Axis 1 does not fix Axis 2 +— but it makes Axis 2 *legible*, which is the precondition for fixing it. + +### `:persistent_term` is a smell + +Treat `:persistent_term`, runtime `Application.put_env`, and naked global ETS +for per-instance config as a **smell requiring explicit justification**. Its +presence means someone reached for *stored state* when *structure* is the +functional answer. The legitimacy litmus: + +> **Does this value ever need to differ between two tests running at the same +> time?** If yes, it cannot live in any global store — inject it. If no, and +> it is genuinely hot-read and fixed at boot, `:persistent_term` is fine +> (Phoenix/Ecto use it internally for exactly that). + +### Worked example: `Lightning.Adaptors.Supervisor` + +The adaptors supervisor is otherwise *exemplary* — its moduledoc states the +principle verbatim, it derives every child name via `Module.concat(name, …)` +(a fixed child set with full multi-instance async isolation and **zero +Registry**), and it injects the `:strategy` as an explicit opt with a +production default. One wart: + +```elixir +# supervisor.ex init/1 — strategy & source are LOCALS here… +strategy = Keyword.get(opts, :strategy, Config.strategy()) +:persistent_term.put(meta_key(name), %{strategy: strategy, source: source_for(strategy)}) +# …then the same init/1 injects cache/tasks/source_topic into child specs +# explicitly, two lines down — but routes strategy/source through a global. +``` + +`Scheduler` then re-fetches it from that global *at call time*: + +```elixir +# scheduler.ex — strategy materialises from nowhere, with no traceable owner +strategy = AdaptorsSupervisor.strategy(state.sup) # :persistent_term.get/1 +strategy.fetch_adaptor(name) +``` + +An investigation of every call site classified this **case (b): avoidable**. +Nothing reaching `Store` lacks the strategy/source at a point where it is +knowable; there is exactly one hardcoded production instance, so even the +stateless facade-from-web path collapses to "boot config for the one instance," +not a dynamic lookup. The generalisable tell: + +> **Global storage smuggling a value past a structural boundary that was +> already open two lines away and already carrying its siblings across.** + +The fix is Axis 1: inject `strategy`/`source` into the child specs the +supervisor is *already building* (it has them in scope), and let `Scheduler` +hold them in its state — exactly as it already does for `source` at `init/1`. + +### Why that makes Axis 2 legible + +Before: you cannot tell *which process* will call `StrategyMock`, because the +value appears from a global with no owner — so you reach for `set_mox_global` +and the suite goes serial. + +After injection: `Scheduler` visibly owns the strategy in its state, so the +mock's caller is obvious and you can scope the allowance: + +```elixir +# Axis 2 recipe — explicit allowance, async-safe +pid = start_supervised!({Lightning.Adaptors.Supervisor, + name: name, strategy: StrategyMock}) +Mox.allow(StrategyMock, self(), Process.whereis(scheduler_name(name))) +``` + +When the pid does not exist yet at setup time (leader election, lazy start), +use the **deferred-resolver form** of `Mox.allow/3` — the trick people forget: + +```elixir +Mox.allow(StrategyMock, self(), fn -> + :global.whereis_name(scheduler_name(name)) +end) +``` + +Mox resolves the pid lazily on first mock invocation, sidestepping the race. +Tasks started via `Task.Supervisor.async/async_nolink` carry `$callers`, so +Mox walks back to the allowed parent automatically; `start_child` +(fire-and-forget) does not propagate and needs its own allowance. + +`Mox.allow/3` and `Ecto.Adapters.SQL.Sandbox.allow/3` are **one concept** — +same signature, same ownership model (a test owns a resource and explicitly +lends it to processes it spawns). Mox's was modelled on Ecto's. + +> **When `set_mox_global` is legitimate, not a cop-out:** explicit allow is the +> default; its cost scales with how many *process hops* the mocked call +> traverses and how *dynamic* those pids are. `set_mox_global` (forcing +> `async: false` for that case) is the correct escape hatch when the hop graph +> is dynamic and deep — `HighlanderPG`-wrapped leader election is the textbook +> case. Try the deferred-resolver form *first*; most cases are not Highlander. + +--- + +## 3. Dynamic populations: when a Registry *is* earned + +When the population is genuinely open-ended and keyed by runtime data, and the +caller does not hold the pid — `DynamicSupervisor` + `Registry` + `:via`. +`Lightning.Collaborate` is the worked example: N sessions/documents keyed by +`document_name`, looked up by a LiveView that did not start them. + +```elixir +SessionSupervisor.start_child( + {Session, workflow: workflow, user: user, + name: Registry.via({:session, "#{document_name}:#{session_id}", user.id})} +) +``` + +Two valid shapes, document the trade-off: + +- **One Registry per top-level instance** (Oban): perfect isolation, but the + instance name threads through every public function + (`Oban.insert(MyApp.Oban, …)`). +- **One global Registry, key-namespaced** (Lightning collaboration): the public + API stays clean (`Collaborate.start/1` takes no name); isolation depends on + keys being genuinely unique. Correct when identity is naturally unique + (workflow + session); risky if a test can reuse a key. + +### The "earned name → config lookup" reference + +If a **stateless synchronous** caller genuinely holds only a name *and* the +population is dynamic, a name → config lookup is justified — and even then it +is **ETS-per-instance, never `:persistent_term`**. `commanded/eventstore` hit +this exact requirement and chose: one named ETS table (`read_concurrency: +true`) owned by a GenServer, holding `{name, pid, ref, store, config}`, the +owner pid monitored so the row **self-deletes on shutdown**. It explicitly +rejected `:persistent_term` because instances churn under async tests and +`:persistent_term` writes/erases trigger a global GC scan. The bonus: an +owner-monitored ETS table needs no manual cleanup function (cf. the adaptors +`forget/1` wart and its "we don't call this automatically because GC is +expensive" comment). + +--- + +## 4. Anti-patterns checklist (point the harness here) + +- ❌ `name: __MODULE__` hardcoded *inside* `start_link` on a worker. → Make + `:name` an option defaulting to `__MODULE__`. +- ❌ Public API calling `GenServer.call(__MODULE__, …)` with no server arg. → + `def fn(server \\ __MODULE__, …)`, server first. +- ❌ Dependencies / per-instance config read from `:persistent_term` / + runtime `Application.put_env` / global ETS at call time. → Inject via + `start_link` opts → `init/1` → process state, or thread through the child + spec. Litmus: *does it vary per concurrent test?* +- ❌ Global storage used to pass a value the supervisor already had in scope + one child-spec away. → Inject it into the child spec. +- ❌ `set_mox_global` reached for reflexively to "fix" a boundary. → Try + `Mox.allow(mock, self(), pid)`, then the deferred-resolver form; reserve + global for genuinely dynamic/deep hop graphs. +- ❌ A `Registry` added to a *fixed* child set "to name things." → The + registered atom name already does this, free. +- ❌ Testing GenServer internals via raw `:"$gen_call"` / poking state. → + Test through the public API; test logic in a pure core module. +- ❌ App supervisor that cannot be started piecemeal in tests. → Children + startable independently via `start_supervised!`. + +--- + +## References + +- Gray & Tate, *Designing Elixir Systems with OTP* (Pragmatic Bookshelf) — + functional core / boundary layering. +- Saša Jurić, *Elixir in Action* — process registration as a parameter; `:via`. +- [Oban — instance & DB isolation](https://hexdocs.pm/oban/isolation.html) · + [`Oban.Registry`](https://hexdocs.pm/oban/Oban.Registry.html) +- [`Ecto.Adapters.SQL.Sandbox`](https://hexdocs.pm/ecto_sql/Ecto.Adapters.SQL.Sandbox.html) + — ownership / `allow/3`. +- [Mox docs](https://hexdocs.pm/mox/Mox.html) — `allow/3`, deferred resolver, + `set_mox_from_context`. +- [`commanded/eventstore`](https://github.com/commanded/eventstore) — + `lib/event_store/config/store.ex` (ETS-per-instance, owner-monitored). +- [`Registry`](https://hexdocs.pm/elixir/Registry.html) · + [Thoughtbot — dynamic process names](https://thoughtbot.com/blog/how-to-start-processes-with-dynamic-names-in-elixir) +- In-repo: `lib/lightning/adaptors/supervisor.ex`, + `lib/lightning/collaboration.ex`, + `lib/lightning/collaboration/registry.ex`. diff --git a/.gitignore b/.gitignore index f32ebcacfa4..fbf1f782dd7 100644 --- a/.gitignore +++ b/.gitignore @@ -75,6 +75,7 @@ priv/openfn .elixir_ls .elixir-tools +.expert .env .env.dev diff --git a/CLAUDE.md b/CLAUDE.md index fea632d0f8a..dc872c1d71f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -132,6 +132,11 @@ in development. - Use `{}` brace syntax in HEEx templates - `warnings_as_errors: true` - code must compile without warnings +> Before writing or reviewing any supervisor, GenServer, or named process, see +> `.claude/guidelines/testable-supervision-trees.md`. Don't bake `name: +> __MODULE__` into a process or resolve collaborators from global state — it +> forces the test suite serial. + ### React/TypeScript - Props from LiveView are **underscore_cased** (not camelCase) @@ -209,5 +214,6 @@ Detailed guidelines in `.claude/guidelines/`: - `testing-essentials.md` - Unit testing patterns and anti-patterns - `e2e-testing.md` - Playwright E2E testing - `yex-guidelines.md` - Critical Yex (Yjs/Elixir) usage rules +- `testable-supervision-trees.md` - OTP processes addressable in tests (async), nameless in prod - `toast-notifications.md` - Notification patterns - `ui-patterns.md` - Button variants, disabled states, Tailwind conventions \ No newline at end of file diff --git a/assets/js/collaborative-editor/components/AdaptorIcon.tsx b/assets/js/collaborative-editor/components/AdaptorIcon.tsx index 51df8928377..918c157dbe3 100644 --- a/assets/js/collaborative-editor/components/AdaptorIcon.tsx +++ b/assets/js/collaborative-editor/components/AdaptorIcon.tsx @@ -1,6 +1,7 @@ -import useAdaptorIcons from '#/workflow-diagram/useAdaptorIcons'; +import { useContext, useMemo, useSyncExternalStore } from 'react'; -import { extractAdaptorName } from '../utils/adaptorUtils'; +import { StoreContext } from '../contexts/StoreProvider'; +import { extractAdaptorName, extractPackageName } from '../utils/adaptorUtils'; interface AdaptorIconProps { name: string; @@ -13,11 +14,35 @@ const sizeClasses = { lg: 'h-12 w-12', }; +// Reads the square icon URL for `name` directly from StoreContext, so callers +// that mock the `hooks/useAdaptors` module (e.g. FullScreenIDE tests) still get +// the existing placeholder fallback instead of crashing on a missing mock. +function useStoreIconUrl(name: string): string | null { + const context = useContext(StoreContext); + const adaptorStore = context?.adaptorStore ?? null; + const packageName = extractPackageName(name); + + const selectIconUrl = useMemo(() => { + if (!adaptorStore) return () => null; + return adaptorStore.withSelector(state => { + const found = state.adaptors.find(a => a.name === packageName); + return found?.icon_urls?.square ?? null; + }); + }, [adaptorStore, packageName]); + + const noopSubscribe = useMemo(() => () => () => {}, []); + + return useSyncExternalStore( + adaptorStore?.subscribe ?? noopSubscribe, + selectIconUrl + ); +} + export function AdaptorIcon({ name, size = 'md' }: AdaptorIconProps) { - const adaptorIconsData = useAdaptorIcons(); const displayName = extractAdaptorName(name) ?? null; + const iconUrl = useStoreIconUrl(name); - if (!adaptorIconsData || !displayName) { + if (!displayName) { return (
diff --git a/assets/js/collaborative-editor/components/AdaptorSelectionModal.tsx b/assets/js/collaborative-editor/components/AdaptorSelectionModal.tsx index 9a18635fc9a..8ade25c2726 100644 --- a/assets/js/collaborative-editor/components/AdaptorSelectionModal.tsx +++ b/assets/js/collaborative-editor/components/AdaptorSelectionModal.tsx @@ -14,7 +14,7 @@ interface AdaptorSelectionModalProps { isOpen: boolean; onClose: () => void; onSelect: (adaptorSpec: string) => void; - projectAdaptors?: Adaptor[]; + adaptorsInUse?: Adaptor[]; } interface AdaptorWithDisplayName extends Adaptor { @@ -25,7 +25,7 @@ export function AdaptorSelectionModal({ isOpen, onClose, onSelect, - projectAdaptors = [], + adaptorsInUse = [], }: AdaptorSelectionModalProps) { const allAdaptors = useAdaptors(); const [searchQuery, setSearchQuery] = useState(''); @@ -60,17 +60,17 @@ export function AdaptorSelectionModal({ [allAdaptors] ); - const projectAdaptorsWithDisplayNames = useMemo< + const adaptorsInUseWithDisplayNames = useMemo< AdaptorWithDisplayName[] >(() => { - return projectAdaptors.map(adaptor => ({ + return adaptorsInUse.map(adaptor => ({ ...adaptor, displayName: getAdaptorDisplayName(adaptor.name, { titleCase: true, fallback: adaptor.name, }), })); - }, [projectAdaptors]); + }, [adaptorsInUse]); const allAdaptorsWithDisplayNames = useMemo(() => { return allAdaptors.map(adaptor => ({ @@ -89,10 +89,10 @@ export function AdaptorSelectionModal({ // Filter project adaptors const filteredProject = searchQuery - ? projectAdaptorsWithDisplayNames.filter(adaptor => + ? adaptorsInUseWithDisplayNames.filter(adaptor => adaptor.displayName.toLowerCase().includes(lowerQuery) ) - : projectAdaptorsWithDisplayNames; + : adaptorsInUseWithDisplayNames; // Filter all adaptors and exclude duplicates from project adaptors const projectAdaptorNames = new Set(filteredProject.map(a => a.name)); @@ -130,7 +130,7 @@ export function AdaptorSelectionModal({ }; }, [ searchQuery, - projectAdaptorsWithDisplayNames, + adaptorsInUseWithDisplayNames, allAdaptorsWithDisplayNames, httpAdaptor, ]); diff --git a/assets/js/collaborative-editor/components/AdaptorSelector.tsx b/assets/js/collaborative-editor/components/AdaptorSelector.tsx index ab63dd35159..3f34af656f6 100644 --- a/assets/js/collaborative-editor/components/AdaptorSelector.tsx +++ b/assets/js/collaborative-editor/components/AdaptorSelector.tsx @@ -21,7 +21,7 @@ interface AdaptorSelectorProps { /** Setter to control configure modal state */ setIsConfigureModalOpen: (open: boolean) => void; /** Available project adaptors */ - projectAdaptors: Adaptor[]; + adaptorsInUse: Adaptor[]; /** Optional callback before adaptor change (for form sync in JobForm) */ onAdaptorChangeStart?: () => void; } @@ -45,7 +45,7 @@ export function AdaptorSelector({ job, updateJob, setIsConfigureModalOpen, - projectAdaptors, + adaptorsInUse, onAdaptorChangeStart, }: AdaptorSelectorProps) { const { @@ -90,7 +90,7 @@ export function AdaptorSelector({ isOpen={isOpen} onClose={handlePickerCloseGuarded} onSelect={handleAdaptorSelect} - projectAdaptors={projectAdaptors} + adaptorsInUse={adaptorsInUse} /> (null); // Fetch project adaptors for modal - const { projectAdaptors } = useProjectAdaptors(); + const { adaptorsInUse } = useAdaptorsInUse(); const updateSelection = useCallback( (id?: string | null) => { @@ -1000,7 +1000,7 @@ export default function WorkflowDiagram(props: WorkflowDiagramProps) { isOpen={pendingPlaceholder !== null} onClose={handleAdaptorModalClose} onSelect={handleAdaptorSelect} - projectAdaptors={projectAdaptors} + adaptorsInUse={adaptorsInUse} /> ); diff --git a/assets/js/collaborative-editor/components/ide/FullScreenIDE.tsx b/assets/js/collaborative-editor/components/ide/FullScreenIDE.tsx index f3eb4db900e..74999f523e5 100644 --- a/assets/js/collaborative-editor/components/ide/FullScreenIDE.tsx +++ b/assets/js/collaborative-editor/components/ide/FullScreenIDE.tsx @@ -26,7 +26,7 @@ import * as dataclipApi from '../../api/dataclips'; import { RENDER_MODES } from '../../constants/panel'; import { useCredentialModal } from '../../contexts/CredentialModalContext'; import { useMonacoRef } from '../../contexts/MonacoRefContext'; -import { useProjectAdaptors } from '../../hooks/useAdaptors'; +import { useAdaptorsInUse } from '../../hooks/useAdaptors'; import { useCredentials, useCredentialsCommands, @@ -395,7 +395,7 @@ export function FullScreenIDE({ const { projectCredentials, keychainCredentials } = useCredentials(); const { requestCredentials } = useCredentialsCommands(); - const { projectAdaptors, allAdaptors } = useProjectAdaptors(); + const { adaptorsInUse, allAdaptors } = useAdaptorsInUse(); const { updateJob } = useWorkflowActions(); // Credential modal is managed by the context @@ -410,19 +410,19 @@ export function FullScreenIDE({ // to be used by components that can't make use of 'latest' const currJobAdaptor = useMemo(() => { if (!currentJob?.adaptor) { - const latestCommon = projectAdaptors.find( + const latestCommon = adaptorsInUse.find( a => a.name === '@openfn/language-common' )?.versions?.[0]?.version; return `@openfn/language-common@${latestCommon || 'latest'}`; } const resolved = resolveAdaptor(currentJob.adaptor); if (resolved.version !== 'latest') return currentJob?.adaptor; - const latestVersion = projectAdaptors.find(a => a.name === resolved.package) + const latestVersion = adaptorsInUse.find(a => a.name === resolved.package) ?.versions?.[0]?.version; // If version not found, return original adaptor string if (!latestVersion) return currentJob.adaptor; return `${resolved.package}@${latestVersion}`; - }, [projectAdaptors, currentJob?.adaptor]); + }, [adaptorsInUse, currentJob?.adaptor]); // Run/Retry functionality for IDE Header const { canRun: canRunSnapshot, tooltipMessage: runTooltipMessage } = @@ -1356,7 +1356,7 @@ export function FullScreenIDE({ job={currentJob} updateJob={updateJob} setIsConfigureModalOpen={setIsConfigureModalOpen} - projectAdaptors={projectAdaptors} + adaptorsInUse={adaptorsInUse} /> )} diff --git a/assets/js/collaborative-editor/components/inspector/JobForm.tsx b/assets/js/collaborative-editor/components/inspector/JobForm.tsx index 201ed6593d3..53863e9db79 100644 --- a/assets/js/collaborative-editor/components/inspector/JobForm.tsx +++ b/assets/js/collaborative-editor/components/inspector/JobForm.tsx @@ -3,7 +3,7 @@ import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; import { useAppForm } from '#/collaborative-editor/components/form'; import { useCredentialModal } from '#/collaborative-editor/contexts/CredentialModalContext'; -import { useProjectAdaptors } from '#/collaborative-editor/hooks/useAdaptors'; +import { useAdaptorsInUse } from '#/collaborative-editor/hooks/useAdaptors'; import { useCredentials, useCredentialsCommands, @@ -55,7 +55,7 @@ export function JobForm({ job }: JobFormProps) { const { updateJob } = useWorkflowActions(); const { projectCredentials, keychainCredentials } = useCredentials(); const { requestCredentials } = useCredentialsCommands(); - const { projectAdaptors, allAdaptors } = useProjectAdaptors(); + const { adaptorsInUse, allAdaptors } = useAdaptorsInUse(); const { isReadOnly } = useWorkflowReadOnly(); // Modal state for adaptor configuration @@ -395,7 +395,7 @@ export function JobForm({ job }: JobFormProps) { job={job} updateJob={updateJob} setIsConfigureModalOpen={setIsConfigureModalOpen} - projectAdaptors={projectAdaptors} + adaptorsInUse={adaptorsInUse} onAdaptorChangeStart={syncAdaptorToForm} />
diff --git a/assets/js/collaborative-editor/hooks/useAdaptors.ts b/assets/js/collaborative-editor/hooks/useAdaptors.ts index 208bdb790c3..265fa2ada3f 100644 --- a/assets/js/collaborative-editor/hooks/useAdaptors.ts +++ b/assets/js/collaborative-editor/hooks/useAdaptors.ts @@ -10,7 +10,8 @@ import { useSyncExternalStore, useContext, useMemo } from 'react'; import { StoreContext } from '../contexts/StoreProvider'; import type { AdaptorStoreInstance } from '../stores/createAdaptorStore'; import type { Adaptor } from '../types/adaptor'; -import type { Job } from '../types/workflow'; +import type { Workflow } from '../types/workflow'; +import { extractPackageName } from '../utils/adaptorUtils'; /** * Main hook for accessing the AdaptorStore instance @@ -86,88 +87,88 @@ export const useAdaptor = (name: string): Adaptor | null => { }; /** - * Extracts adaptor package name from a full adaptor specifier - * e.g., "@openfn/language-common@1.0.0" -> "@openfn/language-common" + * Hook to read an adaptor's square-shape icon URL from the AdaptorStore. + * + * Accepts a full adaptor specifier (with or without version suffix). When no + * StoreProvider is mounted (e.g. the LiveView workflow-editor path), returns + * `null` rather than throwing so consumers fall back to their string label. */ -const getAdaptorPackageName = (adaptor: string | undefined): string | null => { - if (!adaptor) return null; - const match = adaptor.match(/^(@[^@]+)@/); - return match ? match[1] : null; +export const useAdaptorIconUrl = ( + adaptor: string | null | undefined +): string | null => { + const context = useContext(StoreContext); + const adaptorStore = context?.adaptorStore ?? null; + + const packageName = adaptor ? extractPackageName(adaptor) : null; + + const selectIconUrl = useMemo(() => { + if (!adaptorStore) return () => null; + return adaptorStore.withSelector(state => { + if (!packageName) return null; + const found = state.adaptors.find(a => a.name === packageName); + return found?.icon_urls?.square ?? null; + }); + }, [adaptorStore, packageName]); + + const noopSubscribe = useMemo(() => () => () => {}, []); + + return useSyncExternalStore( + adaptorStore?.subscribe ?? noopSubscribe, + selectIconUrl + ); }; /** - * Hook to get project-specific adaptors and all adaptors - * Returns both project adaptors and all adaptors from backend endpoint - * - * Project adaptors are merged from two sources: - * 1. Backend DB (saved jobs) - * 2. Y.Doc state (unsaved jobs in collaborative editor) - * - * This ensures newly added adaptors appear in projectAdaptors before saving. + * Hook to derive the subset of the adaptor catalogue that is referenced by jobs + * in the current Y.Doc workflow. Pure selector — the catalogue comes from + * `request_adaptors` and the jobs come from the collaborative workflow store. */ -export const useProjectAdaptors = (): { - projectAdaptors: Adaptor[]; +export const useAdaptorsInUse = (): { + adaptorsInUse: Adaptor[]; allAdaptors: Adaptor[]; isLoading: boolean; } => { const context = useContext(StoreContext); if (!context) { - throw new Error('useProjectAdaptors must be used within a StoreProvider'); + throw new Error('useAdaptorsInUse must be used within a StoreProvider'); } const { adaptorStore, workflowStore } = context; - // Get adaptor state from adaptor store - const selectAdaptorData = adaptorStore.withSelector(state => ({ - backendProjectAdaptors: state.projectAdaptors || [], - allAdaptors: state.adaptors, - isLoading: state.isLoading, - })); + const selectAdaptors = adaptorStore.withSelector(state => state.adaptors); + const selectIsLoading = adaptorStore.withSelector(state => state.isLoading); + const selectJobs = workflowStore.withSelector(state => state.jobs); - const adaptorData = useSyncExternalStore( + const allAdaptors = useSyncExternalStore( adaptorStore.subscribe, - selectAdaptorData + selectAdaptors + ); + const isLoading = useSyncExternalStore( + adaptorStore.subscribe, + selectIsLoading + ); + const jobs: Workflow.Job[] = useSyncExternalStore( + workflowStore.subscribe, + selectJobs ); - // Get jobs from workflow store (Y.Doc state) - const selectJobs = workflowStore.withSelector(state => state.jobs); - const jobs: Job[] = useSyncExternalStore(workflowStore.subscribe, selectJobs); - - // Merge backend project adaptors with Y.Doc job adaptors - const projectAdaptors = useMemo(() => { - const { backendProjectAdaptors, allAdaptors } = adaptorData; - - // Get adaptor names already in backend list - const backendAdaptorNames = new Set( - backendProjectAdaptors.map(a => a.name) - ); + const adaptorsInUse = useMemo(() => { + if (jobs.length === 0) return []; - // Find adaptors used in Y.Doc jobs that aren't in backend list - const ydocAdaptorNames = new Set(); + const names = new Set(); for (const job of jobs) { - const packageName = getAdaptorPackageName(job.adaptor); - if (packageName && !backendAdaptorNames.has(packageName)) { - ydocAdaptorNames.add(packageName); - } + if (!job.adaptor) continue; + names.add(extractPackageName(job.adaptor)); } - // If no new adaptors from Y.Doc, return backend list as-is - if (ydocAdaptorNames.size === 0) { - return backendProjectAdaptors; - } - - // Find full adaptor objects from allAdaptors for Y.Doc adaptors - const ydocAdaptors = allAdaptors.filter(a => ydocAdaptorNames.has(a.name)); - - // Merge and sort - return [...backendProjectAdaptors, ...ydocAdaptors].sort((a, b) => - a.name.localeCompare(b.name) - ); - }, [adaptorData, jobs]); + return allAdaptors + .filter(a => names.has(a.name)) + .sort((a, b) => a.name.localeCompare(b.name)); + }, [allAdaptors, jobs]); return { - projectAdaptors, - allAdaptors: adaptorData.allAdaptors, - isLoading: adaptorData.isLoading, + adaptorsInUse, + allAdaptors, + isLoading, }; }; diff --git a/assets/js/collaborative-editor/stores/createAdaptorStore.ts b/assets/js/collaborative-editor/stores/createAdaptorStore.ts index 562147b5680..3712599a005 100644 --- a/assets/js/collaborative-editor/stores/createAdaptorStore.ts +++ b/assets/js/collaborative-editor/stores/createAdaptorStore.ts @@ -96,6 +96,29 @@ import { wrapStoreWithDevTools } from './devtools'; const logger = _logger.ns('AdaptorStore').seal(); +// Deep-equality check tailored to the Adaptor shape so referential identity is +// preserved across no-op `adaptors_updated` pushes. +function adaptorsEqual(a: Adaptor, b: Adaptor): boolean { + if (a === b) return true; + if ( + a.name !== b.name || + a.repo !== b.repo || + a.latest !== b.latest || + a.icon_urls.square !== b.icon_urls.square || + a.icon_urls.rectangle !== b.icon_urls.rectangle || + a.versions.length !== b.versions.length + ) { + return false; + } + for (let i = 0; i < a.versions.length; i++) { + const aVer = a.versions[i]; + const bVer = b.versions[i]; + if (!aVer || !bVer) return false; + if (aVer.version !== bVer.version) return false; + } + return true; +} + // sorts adaptors coming into the adaptor store // 1. sorts the versions for every adaptor // 2. sorts the adaptors themselves by name @@ -119,7 +142,6 @@ export const createAdaptorStore = (): AdaptorStore => { let state: AdaptorState = produce( { adaptors: [], - projectAdaptors: [], isLoading: false, error: null, lastUpdated: null, @@ -170,14 +192,36 @@ export const createAdaptorStore = (): AdaptorStore => { const result = AdaptorsListSchema.safeParse(rawData); if (result.success) { - const adaptors = sortAdaptors(result.data); - - state = produce(state, draft => { - draft.adaptors = adaptors; - draft.isLoading = false; - draft.error = null; - draft.lastUpdated = Date.now(); + const incoming = sortAdaptors(result.data); + const existing = state.adaptors; + const existingByName = new Map(existing.map(a => [a.name, a])); + + // Merge by name to preserve referential identity of unchanged adaptors so + // `withSelector` consumers don't re-render on no-op `adaptors_updated` + // pushes. + const merged: Adaptor[] = incoming.map(next => { + const prev = existingByName.get(next.name); + return prev && adaptorsEqual(prev, next) ? prev : next; }); + + const arrayUnchanged = + merged.length === existing.length && + merged.every((a, i) => a === existing[i]); + + if (arrayUnchanged) { + state = produce(state, draft => { + draft.isLoading = false; + draft.error = null; + draft.lastUpdated = Date.now(); + }); + } else { + state = produce(state, draft => { + draft.adaptors = merged; + draft.isLoading = false; + draft.error = null; + draft.lastUpdated = Date.now(); + }); + } notify('handleAdaptorsReceived'); } else { const errorMessage = `Invalid adaptors data: ${result.error.message}`; @@ -267,7 +311,6 @@ export const createAdaptorStore = (): AdaptorStore => { devtools.connect(); void requestAdaptors(); - void requestProjectAdaptors(); return () => { devtools.disconnect(); @@ -310,56 +353,6 @@ export const createAdaptorStore = (): AdaptorStore => { } }; - /** - * Request project adaptors from server via channel - */ - const requestProjectAdaptors = async (): Promise => { - if (!channelProvider?.channel) { - logger.warn('Cannot request project adaptors - no channel connected'); - setError('No connection available'); - return; - } - - setLoading(true); - clearError(); - - try { - logger.debug('Requesting project adaptors'); - const response = await channelRequest( - channelProvider.channel, - 'request_project_adaptors', - {} - ); - - if (response && typeof response === 'object') { - const { project_adaptors, all_adaptors } = response as { - project_adaptors: unknown; - all_adaptors: unknown; - }; - - const projectResult = AdaptorsListSchema.safeParse(project_adaptors); - const allResult = AdaptorsListSchema.safeParse(all_adaptors); - - if (projectResult.success && allResult.success) { - state = produce(state, draft => { - draft.projectAdaptors = sortAdaptors(projectResult.data); - draft.adaptors = sortAdaptors(allResult.data); - draft.isLoading = false; - draft.error = null; - }); - notify('requestProjectAdaptors'); - } else { - const errorMessage = 'Invalid project adaptors data'; - logger.error(errorMessage, { projectResult, allResult }); - setError(errorMessage); - } - } - } catch (error) { - logger.error('Project adaptors request failed', error); - setError('Failed to request project adaptors'); - } - }; - // ============================================================================= // QUERY HELPERS // ============================================================================= @@ -390,7 +383,6 @@ export const createAdaptorStore = (): AdaptorStore => { // Commands (CQS pattern) requestAdaptors, - requestProjectAdaptors, setAdaptors, setLoading, setError, @@ -403,13 +395,6 @@ export const createAdaptorStore = (): AdaptorStore => { // Internal methods (not part of public AdaptorStore interface) _connectChannel: connectChannel, - // Test helper to set project adaptors directly - _setProjectAdaptors: (adaptors: Adaptor[]) => { - state = produce(state, draft => { - draft.projectAdaptors = sortAdaptors(adaptors); - }); - notify('_setProjectAdaptors'); - }, }; }; diff --git a/assets/js/collaborative-editor/types/adaptor.ts b/assets/js/collaborative-editor/types/adaptor.ts index 58ff2a42efe..27291f19751 100644 --- a/assets/js/collaborative-editor/types/adaptor.ts +++ b/assets/js/collaborative-editor/types/adaptor.ts @@ -22,11 +22,17 @@ export const AdaptorVersionSchema = z.object({ /** * Single adaptor schema with all its versions */ +export const AdaptorIconUrlsSchema = z.object({ + square: z.string().nullable(), + rectangle: z.string().nullable(), +}); + export const AdaptorSchema = z.object({ name: z.string(), versions: z.array(AdaptorVersionSchema), repo: z.string(), latest: z.string(), + icon_urls: AdaptorIconUrlsSchema, }); /** @@ -60,9 +66,6 @@ export interface AdaptorState { /** Current list of available adaptors */ adaptors: AdaptorsList; - /** Project-specific adaptors used across workflows */ - projectAdaptors: AdaptorsList; - /** Loading state for initial fetch */ isLoading: boolean; @@ -80,9 +83,6 @@ export interface AdaptorCommands { /** Request adaptors list from server */ requestAdaptors: () => Promise; - /** Request project-specific adaptors from server */ - requestProjectAdaptors: () => Promise; - /** Manually set adaptors (for testing/fallback) */ setAdaptors: (adaptors: AdaptorsList) => void; diff --git a/assets/js/workflow-diagram/components/MiniMapNode.tsx b/assets/js/workflow-diagram/components/MiniMapNode.tsx index 1e08df865ca..d667bc9e783 100644 --- a/assets/js/workflow-diagram/components/MiniMapNode.tsx +++ b/assets/js/workflow-diagram/components/MiniMapNode.tsx @@ -2,9 +2,9 @@ import { ClockIcon, GlobeAltIcon } from '@heroicons/react/24/outline'; import type { MiniMapNodeProps } from '@xyflow/react'; import { memo } from 'react'; +import { useAdaptorIconUrl } from '#/collaborative-editor/hooks/useAdaptors'; + import { useWorkflowStore } from '../../workflow-store/store'; -import useAdaptorIcons from '../useAdaptorIcons'; -import getAdaptorName from '../util/get-adaptor-name'; type Trigger = { id: string; @@ -54,11 +54,12 @@ const MiniMapNode = ({ const storeData = useWorkflowStore(); const jobs = propJobs ?? storeData.jobs; const triggers = propTriggers ?? storeData.triggers; - const adaptorIconsData = useAdaptorIcons(); // Check if this node is a trigger by looking it up in the triggers array const trigger = triggers.find((trigger: Trigger) => trigger.id === id); const isTrigger = !!trigger; + const job = jobs.find((job: Job) => job.id === id); + const icon = useAdaptorIconUrl(job?.adaptor); // For triggers, we'll use the appropriate icon if (isTrigger) { @@ -87,14 +88,6 @@ const MiniMapNode = ({ ); } - // For jobs, we'll use the adaptor icon if available - const job = jobs.find((job: Job) => job.id === id); - const adaptor = job?.adaptor ? getAdaptorName(job.adaptor) : null; - const icon = - adaptor && adaptorIconsData && adaptor in adaptorIconsData - ? adaptorIconsData[adaptor]?.square - : null; - // Fallback to rectangle if no icon is available return ( diff --git a/assets/js/workflow-diagram/nodes/Job.tsx b/assets/js/workflow-diagram/nodes/Job.tsx index 1e71c0a7b17..9ce572a6a7b 100644 --- a/assets/js/workflow-diagram/nodes/Job.tsx +++ b/assets/js/workflow-diagram/nodes/Job.tsx @@ -1,8 +1,9 @@ import { Position, type NodeProps } from '@xyflow/react'; import { memo } from 'react'; +import { useAdaptorIconUrl } from '#/collaborative-editor/hooks/useAdaptors'; + import PathButton from '../components/PathButton'; -import useAdaptorIcons, { type AdaptorIconData } from '../useAdaptorIcons'; import getAdaptorName from '../util/get-adaptor-name'; import Node from './Node'; @@ -22,10 +23,9 @@ const JobNode = ({ ], ]; - const adaptorIconsData = useAdaptorIcons(); - const adaptor = getAdaptorName(props.data?.adaptor); - const icon = getAdaptorIcon(adaptor, adaptorIconsData); + const iconUrl = useAdaptorIconUrl(props.data?.adaptor); + const icon = iconUrl ? {adaptor} : adaptor; return ( ; - } else { - return adaptor; - } - } catch { - return adaptor; - } -} - export default memo(JobNode); diff --git a/assets/test/collaborative-editor/components/AdaptorIcon.test.tsx b/assets/test/collaborative-editor/components/AdaptorIcon.test.tsx new file mode 100644 index 00000000000..1a078ba67b5 --- /dev/null +++ b/assets/test/collaborative-editor/components/AdaptorIcon.test.tsx @@ -0,0 +1,87 @@ +/** + * Tests for AdaptorIcon component + * + * Verifies that icon URLs are read from the AdaptorStore (icon_urls.square) + * with the existing first-letter placeholder fallback when no URL is present + * or the adaptor is not in the store. + */ + +import { render, screen } from '@testing-library/react'; +import { describe, expect, it } from 'vitest'; + +import { AdaptorIcon } from '../../../js/collaborative-editor/components/AdaptorIcon'; +import { + StoreContext, + type StoreContextValue, +} from '../../../js/collaborative-editor/contexts/StoreProvider'; +import { createAdaptorStore } from '../../../js/collaborative-editor/stores/createAdaptorStore'; +import type { Adaptor } from '../../../js/collaborative-editor/types/adaptor'; + +function renderWithAdaptors(ui: React.ReactElement, adaptors: Adaptor[]) { + const adaptorStore = createAdaptorStore(); + adaptorStore.setAdaptors(adaptors); + + const stores = { + adaptorStore, + credentialStore: {} as StoreContextValue['credentialStore'], + metadataStore: {} as StoreContextValue['metadataStore'], + awarenessStore: {} as StoreContextValue['awarenessStore'], + workflowStore: {} as StoreContextValue['workflowStore'], + sessionContextStore: {} as StoreContextValue['sessionContextStore'], + historyStore: {} as StoreContextValue['historyStore'], + uiStore: {} as StoreContextValue['uiStore'], + editorPreferencesStore: {} as StoreContextValue['editorPreferencesStore'], + aiAssistantStore: {} as StoreContextValue['aiAssistantStore'], + } satisfies StoreContextValue; + + return render( + {ui} + ); +} + +describe('AdaptorIcon', () => { + it('renders icon_urls.square as an when populated in the store', () => { + const url = '/adaptor-icons/salesforce/square-abc.png'; + renderWithAdaptors( + , + [ + { + name: '@openfn/language-salesforce', + versions: [{ version: '2.0.0' }], + repo: 'https://example.com', + latest: '2.0.0', + icon_urls: { square: url, rectangle: null }, + }, + ] + ); + + const img = screen.getByAltText('salesforce'); + expect(img.tagName).toBe('IMG'); + expect(img.getAttribute('src')).toContain(url); + }); + + it('renders the first-letter placeholder when icon_urls.square is null', () => { + renderWithAdaptors( + , + [ + { + name: '@openfn/language-salesforce', + versions: [{ version: '2.0.0' }], + repo: 'https://example.com', + latest: '2.0.0', + icon_urls: { square: null, rectangle: '/some-rectangle.png' }, + }, + ] + ); + + expect(screen.queryByRole('img')).toBeNull(); + expect(screen.getByText('S')).toBeInTheDocument(); + }); + + it('renders the first-letter placeholder when the adaptor is not in the store', () => { + renderWithAdaptors(, []); + + expect(screen.queryByRole('img')).toBeNull(); + expect(screen.getByText('H')).toBeInTheDocument(); + }); +}); diff --git a/assets/test/collaborative-editor/components/AdaptorSelectionModal.test.tsx b/assets/test/collaborative-editor/components/AdaptorSelectionModal.test.tsx index b01e538bb81..8bc7933967c 100644 --- a/assets/test/collaborative-editor/components/AdaptorSelectionModal.test.tsx +++ b/assets/test/collaborative-editor/components/AdaptorSelectionModal.test.tsx @@ -125,7 +125,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> ); @@ -156,7 +156,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> ); @@ -172,7 +172,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> ); @@ -187,7 +187,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={[]} + adaptorsInUse={[]} /> ); @@ -203,7 +203,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> ); @@ -220,7 +220,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> ); @@ -243,7 +243,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> ); @@ -263,7 +263,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> ); @@ -287,7 +287,7 @@ describe('AdaptorSelectionModal', () => { isOpen={isOpen} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> @@ -329,7 +329,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> ); @@ -349,7 +349,7 @@ describe('AdaptorSelectionModal', () => { isOpen={true} onClose={onClose} onSelect={onSelect} - projectAdaptors={mockProjectAdaptors} + adaptorsInUse={mockProjectAdaptors} /> ); diff --git a/assets/test/collaborative-editor/components/ide/FullScreenIDE.docs-panel.test.tsx b/assets/test/collaborative-editor/components/ide/FullScreenIDE.docs-panel.test.tsx index 2e8bccb3848..141d0450591 100644 --- a/assets/test/collaborative-editor/components/ide/FullScreenIDE.docs-panel.test.tsx +++ b/assets/test/collaborative-editor/components/ide/FullScreenIDE.docs-panel.test.tsx @@ -269,8 +269,8 @@ vi.mock('../../../../js/collaborative-editor/hooks/useCredentials', () => ({ // Mock adaptor hooks vi.mock('../../../../js/collaborative-editor/hooks/useAdaptors', () => ({ - useProjectAdaptors: () => ({ - projectAdaptors: [], + useAdaptorsInUse: () => ({ + adaptorsInUse: [], allAdaptors: [], }), })); diff --git a/assets/test/collaborative-editor/components/ide/FullScreenIDE.keyboard.test.tsx b/assets/test/collaborative-editor/components/ide/FullScreenIDE.keyboard.test.tsx index feb809b882e..5700c39ce19 100644 --- a/assets/test/collaborative-editor/components/ide/FullScreenIDE.keyboard.test.tsx +++ b/assets/test/collaborative-editor/components/ide/FullScreenIDE.keyboard.test.tsx @@ -366,8 +366,8 @@ vi.mock('../../../../js/collaborative-editor/hooks/useCredentials', () => ({ // Mock adaptor hooks vi.mock('../../../../js/collaborative-editor/hooks/useAdaptors', () => ({ - useProjectAdaptors: () => ({ - projectAdaptors: [], + useAdaptorsInUse: () => ({ + adaptorsInUse: [], allAdaptors: [], }), useAdaptors: () => ({ diff --git a/assets/test/collaborative-editor/components/ide/FullScreenIDE.test.tsx b/assets/test/collaborative-editor/components/ide/FullScreenIDE.test.tsx index b4ad650d57c..1152945f0ca 100644 --- a/assets/test/collaborative-editor/components/ide/FullScreenIDE.test.tsx +++ b/assets/test/collaborative-editor/components/ide/FullScreenIDE.test.tsx @@ -317,8 +317,8 @@ vi.mock('../../../../js/collaborative-editor/hooks/useAdaptors', () => ({ loading: false, error: null, }), - useProjectAdaptors: () => ({ - projectAdaptors: [], + useAdaptorsInUse: () => ({ + adaptorsInUse: [], allAdaptors: [], }), useAdaptors: () => [], diff --git a/assets/test/collaborative-editor/createAdaptorStore.test.ts b/assets/test/collaborative-editor/createAdaptorStore.test.ts index e976b221e17..5ade4e9c68f 100644 --- a/assets/test/collaborative-editor/createAdaptorStore.test.ts +++ b/assets/test/collaborative-editor/createAdaptorStore.test.ts @@ -407,4 +407,103 @@ describe('createAdaptorStore', () => { expect(notFound).toHaveLength(0); }); }); + + describe('handleAdaptorsReceived merge-by-name', () => { + adaptorTest( + 'preserves referential identity across identical pushes', + async ({ mockChannel, mockProvider }) => { + const store = createAdaptorStore(); + mockChannel.push = createMockChannelPushOk({ + adaptors: mockAdaptorsList, + }); + store._connectChannel(mockProvider as any); + await store.requestAdaptors(); + + const selectAdaptors = store.withSelector(state => state.adaptors); + const firstRef = selectAdaptors(); + const firstItems = firstRef.map(a => a); + + const mockChannelWithTest = mockChannel as typeof mockChannel & { + _test: { emit: (event: string, message: unknown) => void }; + }; + mockChannelWithTest._test.emit('adaptors_updated', mockAdaptorsList); + + await waitForCondition(() => store.getSnapshot().lastUpdated !== null); + + const secondRef = selectAdaptors(); + expect(secondRef).toBe(firstRef); + secondRef.forEach((adaptor, i) => { + expect(adaptor).toBe(firstItems[i]); + }); + } + ); + + adaptorTest( + 'replaces only the changed entry when one adaptor mutates', + async ({ mockChannel, mockProvider }) => { + const store = createAdaptorStore(); + mockChannel.push = createMockChannelPushOk({ + adaptors: mockAdaptorsList, + }); + store._connectChannel(mockProvider as any); + await store.requestAdaptors(); + + const beforeItems = store + .getSnapshot() + .adaptors.reduce>((acc, a) => { + acc[a.name] = a; + return acc; + }, {}); + + const target = mockAdaptorsList[0]!; + const mutated = mockAdaptorsList.map(a => + a.name === target.name + ? { + ...a, + versions: [{ version: '99.0.0' }, ...a.versions], + latest: '99.0.0', + } + : a + ); + + const mockChannelWithTest = mockChannel as typeof mockChannel & { + _test: { emit: (event: string, message: unknown) => void }; + }; + mockChannelWithTest._test.emit('adaptors_updated', mutated); + + await waitForCondition( + () => store.findAdaptorByName(target.name)?.latest === '99.0.0' + ); + + const after = store.getSnapshot().adaptors; + for (const adaptor of after) { + if (adaptor.name === target.name) { + expect(adaptor).not.toBe(beforeItems[adaptor.name]); + } else { + expect(adaptor).toBe(beforeItems[adaptor.name]); + } + } + } + ); + + adaptorTest( + 'connectChannel does not request project adaptors', + async ({ mockChannel, mockProvider }) => { + const pushes: string[] = []; + const originalPush = mockChannel.push.bind(mockChannel); + mockChannel.push = ((event: string, payload: unknown) => { + pushes.push(event); + return originalPush(event, payload); + }) as typeof mockChannel.push; + + const store = createAdaptorStore(); + store._connectChannel(mockProvider as any); + + await waitForCondition(() => pushes.includes('request_adaptors')); + + expect(pushes).toContain('request_adaptors'); + expect(pushes).not.toContain('request_project_adaptors'); + } + ); + }); }); diff --git a/assets/test/collaborative-editor/fixtures/adaptorData.ts b/assets/test/collaborative-editor/fixtures/adaptorData.ts index e12adb3e1e5..6405f86281a 100644 --- a/assets/test/collaborative-editor/fixtures/adaptorData.ts +++ b/assets/test/collaborative-editor/fixtures/adaptorData.ts @@ -31,6 +31,7 @@ export const mockAdaptor: Adaptor = { versions: mockAdaptorVersions, repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/http', latest: '2.1.0', + icon_urls: { square: null, rectangle: null }, }; /** @@ -41,6 +42,7 @@ export const mockAdaptorDhis2: Adaptor = { versions: [{ version: '4.2.1' }, { version: '4.2.0' }, { version: '4.1.3' }], repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/dhis2', latest: '4.2.1', + icon_urls: { square: null, rectangle: null }, }; export const mockAdaptorSalesforce: Adaptor = { @@ -53,6 +55,7 @@ export const mockAdaptorSalesforce: Adaptor = { ], repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/salesforce', latest: '3.5.2', + icon_urls: { square: null, rectangle: null }, }; export const mockAdaptorGmail: Adaptor = { @@ -60,6 +63,7 @@ export const mockAdaptorGmail: Adaptor = { versions: [{ version: '1.2.0' }, { version: '1.1.0' }, { version: '1.0.0' }], repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/gmail', latest: '1.2.0', + icon_urls: { square: null, rectangle: null }, }; export const mockAdaptorCommon: Adaptor = { @@ -71,6 +75,7 @@ export const mockAdaptorCommon: Adaptor = { ], repo: 'https://github.com/OpenFn/adaptors/tree/main/packages/common', latest: '2.0.0', + icon_urls: { square: null, rectangle: null }, }; /** @@ -143,6 +148,7 @@ export function createMockAdaptorsList(count: number): AdaptorsList { versions: [{ version: `${i}.1.0` }, { version: `${i}.0.0` }], repo: `https://github.com/test/adaptor-${i}`, latest: `${i}.1.0`, + icon_urls: { square: null, rectangle: null }, })); } /* eslint-enable @typescript-eslint/restrict-template-expressions */ diff --git a/assets/test/collaborative-editor/useAdaptors.test.tsx b/assets/test/collaborative-editor/useAdaptors.test.tsx index 8727805a33c..d5a1716a895 100644 --- a/assets/test/collaborative-editor/useAdaptors.test.tsx +++ b/assets/test/collaborative-editor/useAdaptors.test.tsx @@ -14,8 +14,8 @@ import { useAdaptorCommands, useAdaptors, useAdaptorsError, + useAdaptorsInUse, useAdaptorsLoading, - useProjectAdaptors, } from '../../js/collaborative-editor/hooks/useAdaptors'; import { createSessionStore } from '../../js/collaborative-editor/stores/createSessionStore'; @@ -376,162 +376,129 @@ describe('useAdaptors hooks', () => { }); }); - describe('useProjectAdaptors', () => { + describe('useAdaptorsInUse', () => { test('requires StoreProvider context', () => { - expect(() => renderHook(() => useProjectAdaptors())).toThrow( - 'useProjectAdaptors must be used within a StoreProvider' + expect(() => renderHook(() => useAdaptorsInUse())).toThrow( + 'useAdaptorsInUse must be used within a StoreProvider' ); }); - test('returns projectAdaptors, allAdaptors, and isLoading', async () => { + test('derives adaptors in use from Y.Doc jobs', async () => { const { wrapper, stores } = createWrapper(); - const { result } = renderHook(() => useProjectAdaptors(), { wrapper }); + const { result } = renderHook(() => useAdaptorsInUse(), { wrapper }); - // Initially empty - expect(result.current.projectAdaptors).toEqual([]); - expect(result.current.allAdaptors).toEqual([]); - expect(result.current.isLoading).toBe(false); - - // Set all adaptors (simulates backend response) - const allAdaptors = [mockAdaptor, mockAdaptorGmail, mockAdaptorCommon]; - act(() => { - stores.adaptorStore.setAdaptors(allAdaptors); - }); - - await waitFor(() => { - expect(result.current.allAdaptors).toHaveLength(3); - }); - }); - - test('merges Y.Doc job adaptors into projectAdaptors', async () => { - const { wrapper, stores } = createWrapper(); - const { result } = renderHook(() => useProjectAdaptors(), { wrapper }); - - // Set all available adaptors const allAdaptors = [mockAdaptor, mockAdaptorGmail, mockAdaptorCommon]; act(() => { stores.adaptorStore.setAdaptors(allAdaptors); }); - // Add a job to the workflow store that uses Gmail adaptor act(() => { stores.workflowStore._setJobsForTesting([ { id: 'job-1', - name: 'Test Job', - adaptor: '@openfn/language-gmail@latest', - body: 'fn(state => state)', + name: 'HTTP Job', + adaptor: '@openfn/language-http@2.1.0', + body: '', + }, + { + id: 'job-2', + name: 'Common Job', + adaptor: '@openfn/language-common@2.0.0', + body: '', }, ]); }); - // projectAdaptors should now include Gmail from Y.Doc await waitFor(() => { - const projectAdaptorNames = result.current.projectAdaptors.map( - a => a.name - ); - expect(projectAdaptorNames).toContain('@openfn/language-gmail'); - }); + expect(result.current.adaptorsInUse).toHaveLength(2); + }); + + const names = result.current.adaptorsInUse.map(a => a.name); + expect(names).toEqual([ + '@openfn/language-common', + '@openfn/language-http', + ]); + + // Per-entry referential identity: each item in adaptorsInUse is the same + // reference as the corresponding catalogue entry. + const catalogue = result.current.allAdaptors; + for (const a of result.current.adaptorsInUse) { + const fromCatalogue = catalogue.find(c => c.name === a.name); + expect(a).toBe(fromCatalogue); + } }); - test('does not duplicate adaptors already in backend projectAdaptors', async () => { + test('returns empty list when workflow has no jobs', async () => { const { wrapper, stores } = createWrapper(); - const { result } = renderHook(() => useProjectAdaptors(), { wrapper }); + const { result } = renderHook(() => useAdaptorsInUse(), { wrapper }); - // Set all available adaptors and project adaptors (simulates backend where HTTP is already saved) const allAdaptors = [mockAdaptor, mockAdaptorGmail, mockAdaptorCommon]; act(() => { stores.adaptorStore.setAdaptors(allAdaptors); - // Directly set projectAdaptors via internal state modification - stores.adaptorStore._setProjectAdaptors([mockAdaptor]); - }); - - // Add a job that uses HTTP (already in projectAdaptors from backend) - act(() => { - stores.workflowStore._setJobsForTesting([ - { - id: 'job-1', - name: 'Test Job', - adaptor: '@openfn/language-http@2.1.0', - body: 'fn(state => state)', - }, - ]); }); await waitFor(() => { - // Should only have HTTP once, not duplicated - const httpCount = result.current.projectAdaptors.filter( - a => a.name === '@openfn/language-http' - ).length; - expect(httpCount).toBe(1); + expect(result.current.allAdaptors).toHaveLength(3); }); + + expect(result.current.adaptorsInUse).toEqual([]); }); - test('handles jobs without adaptor field', async () => { + test('ignores jobs referencing adaptors absent from the catalogue', async () => { const { wrapper, stores } = createWrapper(); - const { result } = renderHook(() => useProjectAdaptors(), { wrapper }); + const { result } = renderHook(() => useAdaptorsInUse(), { wrapper }); - // Set all available adaptors - const allAdaptors = [mockAdaptor, mockAdaptorGmail]; act(() => { - stores.adaptorStore.setAdaptors(allAdaptors); + stores.adaptorStore.setAdaptors([mockAdaptor]); }); - // Add a job without an adaptor field (should not crash) act(() => { stores.workflowStore._setJobsForTesting([ { id: 'job-1', - name: 'Test Job', - adaptor: undefined as any, - body: 'fn(state => state)', + name: 'Unknown', + adaptor: '@openfn/language-unknown@1.0.0', + body: '', + }, + { + id: 'job-2', + name: 'HTTP', + adaptor: '@openfn/language-http@2.0.0', + body: '', }, ]); }); - // Should not throw and projectAdaptors should be empty (no backend project adaptors set) await waitFor(() => { - expect(result.current.projectAdaptors).toEqual([]); + expect(result.current.adaptorsInUse).toHaveLength(1); }); + + expect(result.current.adaptorsInUse[0]?.name).toBe( + '@openfn/language-http' + ); }); - test('sorts merged projectAdaptors alphabetically', async () => { + test('matches version-suffixed adaptor specs against catalogue package names', async () => { const { wrapper, stores } = createWrapper(); - const { result } = renderHook(() => useProjectAdaptors(), { wrapper }); + const { result } = renderHook(() => useAdaptorsInUse(), { wrapper }); - // Set all available adaptors - const allAdaptors = [mockAdaptor, mockAdaptorGmail, mockAdaptorCommon]; act(() => { - stores.adaptorStore.setAdaptors(allAdaptors); - // Set HTTP as already in project from backend - stores.adaptorStore._setProjectAdaptors([mockAdaptor]); + stores.adaptorStore.setAdaptors([mockAdaptor]); }); - // Add jobs using Gmail and Common (not in backend projectAdaptors) act(() => { stores.workflowStore._setJobsForTesting([ { id: 'job-1', - name: 'Gmail Job', - adaptor: '@openfn/language-gmail@latest', - body: '', - }, - { - id: 'job-2', - name: 'Common Job', - adaptor: '@openfn/language-common@2.0.0', + name: 'HTTP', + adaptor: '@openfn/language-http@2.0.0', body: '', }, ]); }); await waitFor(() => { - expect(result.current.projectAdaptors).toHaveLength(3); - const names = result.current.projectAdaptors.map(a => a.name); - // Should be sorted: common, gmail, http - expect(names).toEqual([ - '@openfn/language-common', - '@openfn/language-gmail', + expect(result.current.adaptorsInUse.map(a => a.name)).toEqual([ '@openfn/language-http', ]); }); diff --git a/assets/test/workflow-diagram/components/MiniMapNode.test.tsx b/assets/test/workflow-diagram/components/MiniMapNode.test.tsx new file mode 100644 index 00000000000..4acdf6453e9 --- /dev/null +++ b/assets/test/workflow-diagram/components/MiniMapNode.test.tsx @@ -0,0 +1,120 @@ +/** + * MiniMapNode Component Tests + * + * Verifies that the minimap renders job icons sourced from the AdaptorStore + * via useAdaptorIconUrl, with the rect-only placeholder fallback when the + * URL is null. Trigger rendering must remain untouched. + */ + +import { render } from '@testing-library/react'; +import type { MiniMapNodeProps } from '@xyflow/react'; +import { describe, expect, test } from 'vitest'; + +import { + StoreContext, + type StoreContextValue, +} from '../../../js/collaborative-editor/contexts/StoreProvider'; +import { createAdaptorStore } from '../../../js/collaborative-editor/stores/createAdaptorStore'; +import type { Adaptor } from '../../../js/collaborative-editor/types/adaptor'; +import MiniMapNode from '../../../js/workflow-diagram/components/MiniMapNode'; + +type Job = { id: string; adaptor?: string }; +type Trigger = { id: string; type: 'webhook' | 'cron' | 'kafka' }; + +function renderInSvg( + nodeProps: MiniMapNodeProps, + jobs: Job[], + triggers: Trigger[], + adaptors: Adaptor[] +) { + const adaptorStore = createAdaptorStore(); + adaptorStore.setAdaptors(adaptors); + + const stores = { + adaptorStore, + credentialStore: {} as StoreContextValue['credentialStore'], + metadataStore: {} as StoreContextValue['metadataStore'], + awarenessStore: {} as StoreContextValue['awarenessStore'], + workflowStore: {} as StoreContextValue['workflowStore'], + sessionContextStore: {} as StoreContextValue['sessionContextStore'], + historyStore: {} as StoreContextValue['historyStore'], + uiStore: {} as StoreContextValue['uiStore'], + editorPreferencesStore: {} as StoreContextValue['editorPreferencesStore'], + aiAssistantStore: {} as StoreContextValue['aiAssistantStore'], + } satisfies StoreContextValue; + + return render( + + + + + + ); +} + +describe('MiniMapNode - job icon', () => { + const baseNodeProps: MiniMapNodeProps = { + x: 0, + y: 0, + width: 120, + height: 120, + selected: false, + borderRadius: 0, + className: '', + shapeRendering: 'auto', + }; + + test('renders an when the adaptor is seeded with icon_urls.square', () => { + const url = '/adaptor-icons/http/square-1.png'; + const { container } = renderInSvg( + { ...baseNodeProps, id: 'job-1' }, + [{ id: 'job-1', adaptor: '@openfn/language-http@1.0.0' }], + [], + [ + { + name: '@openfn/language-http', + versions: [{ version: '1.0.0' }], + repo: 'https://example.com', + latest: '1.0.0', + icon_urls: { square: url, rectangle: null }, + }, + ] + ); + + const image = container.querySelector('image'); + expect(image).not.toBeNull(); + expect(image!.getAttribute('href')).toBe(url); + }); + + test('renders no when icon_urls.square is null', () => { + const { container } = renderInSvg( + { ...baseNodeProps, id: 'job-1' }, + [{ id: 'job-1', adaptor: '@openfn/language-http@1.0.0' }], + [], + [ + { + name: '@openfn/language-http', + versions: [{ version: '1.0.0' }], + repo: 'https://example.com', + latest: '1.0.0', + icon_urls: { square: null, rectangle: null }, + }, + ] + ); + + expect(container.querySelector('image')).toBeNull(); + expect(container.querySelector('rect')).not.toBeNull(); + }); + + test('webhook trigger still renders the GlobeAltIcon (regression guard)', () => { + const { container } = renderInSvg( + { ...baseNodeProps, id: 'trigger-1' }, + [], + [{ id: 'trigger-1', type: 'webhook' }], + [] + ); + + // GlobeAltIcon renders an svg inside a foreignObject for triggers + expect(container.querySelector('foreignObject svg')).not.toBeNull(); + }); +}); diff --git a/assets/test/workflow-diagram/nodes/Job.test.tsx b/assets/test/workflow-diagram/nodes/Job.test.tsx new file mode 100644 index 00000000000..08eff8edb4c --- /dev/null +++ b/assets/test/workflow-diagram/nodes/Job.test.tsx @@ -0,0 +1,95 @@ +/** + * Job Node Component Tests + * + * Verifies that job nodes read their adaptor icons from the AdaptorStore + * via useAdaptorIconUrl, with graceful string-label fallback when the URL + * is null OR no StoreProvider is mounted (LiveView workflow-editor path). + */ + +import { render } from '@testing-library/react'; +import { ReactFlowProvider } from '@xyflow/react'; +import { describe, expect, test } from 'vitest'; + +import { + StoreContext, + type StoreContextValue, +} from '../../../js/collaborative-editor/contexts/StoreProvider'; +import { createAdaptorStore } from '../../../js/collaborative-editor/stores/createAdaptorStore'; +import type { Adaptor } from '../../../js/collaborative-editor/types/adaptor'; +import JobNode from '../../../js/workflow-diagram/nodes/Job'; + +function renderJob(adaptor: string, adaptors: Adaptor[] | null) { + const data = { name: 'My Job', adaptor }; + + const tree = ( + + + + ); + + if (adaptors === null) { + return render(tree); + } + + const adaptorStore = createAdaptorStore(); + adaptorStore.setAdaptors(adaptors); + + const stores = { + adaptorStore, + credentialStore: {} as StoreContextValue['credentialStore'], + metadataStore: {} as StoreContextValue['metadataStore'], + awarenessStore: {} as StoreContextValue['awarenessStore'], + workflowStore: {} as StoreContextValue['workflowStore'], + sessionContextStore: {} as StoreContextValue['sessionContextStore'], + historyStore: {} as StoreContextValue['historyStore'], + uiStore: {} as StoreContextValue['uiStore'], + editorPreferencesStore: {} as StoreContextValue['editorPreferencesStore'], + aiAssistantStore: {} as StoreContextValue['aiAssistantStore'], + } satisfies StoreContextValue; + + return render( + {tree} + ); +} + +describe('JobNode - adaptor icon', () => { + test('renders an with icon_urls.square when the adaptor is seeded', () => { + const url = '/adaptor-icons/http/square-deadbeef.png'; + const { container } = renderJob('@openfn/language-http@1.0.0', [ + { + name: '@openfn/language-http', + versions: [{ version: '1.0.0' }], + repo: 'https://example.com', + latest: '1.0.0', + icon_urls: { square: url, rectangle: null }, + }, + ]); + + const img = container.querySelector('img'); + expect(img).not.toBeNull(); + expect(img?.getAttribute('src')).toContain(url); + expect(img?.getAttribute('alt')).toBe('http'); + }); + + test('falls back to the adaptor string label when icon_urls.square is null', () => { + const { container } = renderJob('@openfn/language-http@1.0.0', [ + { + name: '@openfn/language-http', + versions: [{ version: '1.0.0' }], + repo: 'https://example.com', + latest: '1.0.0', + icon_urls: { square: null, rectangle: '/rect.png' }, + }, + ]); + + expect(container.querySelector('img')).toBeNull(); + expect(container.textContent).toContain('http'); + }); + + test('does not throw and falls back to label when no StoreProvider is mounted', () => { + const { container } = renderJob('@openfn/language-http@1.0.0', null); + + expect(container.querySelector('img')).toBeNull(); + expect(container.textContent).toContain('http'); + }); +}); diff --git a/config/dev.exs b/config/dev.exs index ad4124a0307..f86bf15564b 100644 --- a/config/dev.exs +++ b/config/dev.exs @@ -131,6 +131,18 @@ config :lightning, :is_resettable_demo, true config :lightning, :apollo, endpoint: "http://localhost:3000", timeout: 30_000 +# Lightning.Adaptors.NPM upstream URLs — explicit override for clarity in dev. +# Each key is read by a single sub-module: +# * registry_url → NPM.Registry (npm search + packument) +# * jsdelivr_url → NPM.Schema (configuration-schema.json fetch) +# * github_url → NPM.GitHub (raw icon GETs) +# * github_ref → NPM.GitHub (git ref under OpenFn/adaptors) +config :lightning, Lightning.Adaptors.NPM, + registry_url: "https://registry.npmjs.org", + github_url: "https://raw.githubusercontent.com", + github_ref: "main", + jsdelivr_url: "https://cdn.jsdelivr.net" + config :git_hooks, # In local dev (with a real .git repo) we auto-install hooks. # In Docker builds the .git directory is not present (or incomplete), diff --git a/config/test.exs b/config/test.exs index 9641f9e9a60..8ce0f8d6b5b 100644 --- a/config/test.exs +++ b/config/test.exs @@ -96,6 +96,20 @@ config :lightning, Lightning.Mailer, adapter: Swoosh.Adapters.Test config :lightning, Lightning.AdaptorRegistry, use_cache: "test/fixtures/adaptor_registry_cache.json" +# Phase A Adaptors.Supervisor config for test boot. +# +# - `:strategy` — the production `Lightning.Adaptors.Supervisor` mounted in +# `application.ex` would default to `Lightning.Adaptors.NPM` and try to +# hit the network on the first Scheduler tick. Replace it with the +# Mox-backed `StrategyMock` so the application-level supervisor (under +# the production name `Lightning.Adaptors`) is a no-op for tests that +# exercise the facade directly. +# - `:refresh_interval` — `0` disables Scheduler tick scheduling entirely. +# Per-test isolated supervisors set their own interval as needed. +config :lightning, Lightning.Adaptors, + strategy: Lightning.Adaptors.StrategyMock, + refresh_interval: 0 + config :hammer, backend: {Hammer.Backend.ETS, diff --git a/lib/lightning/adaptors.ex b/lib/lightning/adaptors.ex new file mode 100644 index 00000000000..fd69d2eb9bb --- /dev/null +++ b/lib/lightning/adaptors.ex @@ -0,0 +1,100 @@ +defmodule Lightning.Adaptors do + @moduledoc """ + Public facade for all adaptor metadata. + + Delegates reads to `Lightning.Adaptors.Store`, refresh calls to + `Lightning.Adaptors.Scheduler`, and version resolution to + `Lightning.Adaptors.Repo`. No logic lives here. + + All functions come in a dual-arity shape: the zero-/single-arg form + passes the compile-time default supervisor name `@sup`; the extra-arity + form accepts an explicit supervisor name for test isolation. + `resolve_version/2` is the single exception — it has no sup arity because + it reads the global Repo directly. + """ + + alias Lightning.Adaptors.Config + alias Lightning.Adaptors.Repo + alias Lightning.Adaptors.Scheduler + alias Lightning.Adaptors.Store + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + @sup Lightning.Adaptors + + @type package_meta :: Store.package_meta() + @type version_meta :: Store.version_meta() + + @spec packages() :: {:ok, [package_meta()]} | {:error, :timeout | term()} + def packages, do: packages(@sup) + + @spec packages(atom()) :: {:ok, [package_meta()]} | {:error, :timeout | term()} + def packages(sup), do: Store.packages(sup) + + @spec versions(String.t()) :: {:ok, [version_meta()]} | {:error, term()} + def versions(pkg), do: versions(@sup, pkg) + + @spec versions(atom(), String.t()) :: + {:ok, [version_meta()]} | {:error, term()} + def versions(sup, pkg), do: Store.versions(sup, pkg) + + @spec schema(String.t()) :: {:ok, String.t()} | {:error, term()} + def schema(pkg), do: schema(@sup, pkg) + + @spec schema(atom(), String.t()) :: {:ok, String.t()} | {:error, term()} + def schema(sup, pkg), do: Store.schema(sup, pkg) + + @spec icon(String.t(), :square | :rectangle) :: + {:ok, Path.t()} | {:error, term()} + def icon(pkg, shape), do: icon(@sup, pkg, shape) + + @spec icon(atom(), String.t(), :square | :rectangle) :: + {:ok, Path.t()} | {:error, term()} + def icon(sup, pkg, shape), do: Store.icon(sup, pkg, shape) + + @spec resolve_version(String.t(), String.t()) :: + {:ok, String.t()} | {:error, :not_found} + def resolve_version(name, requested) when requested in ["latest", "local"] do + case Repo.get_adaptor(name, Config.current_source()) do + %{latest_version: v} -> {:ok, v} + nil -> {:error, :not_found} + end + end + + def resolve_version(_name, version), do: {:ok, version} + + @spec refresh_now() :: :ok | {:error, term()} + def refresh_now, do: refresh_now(@sup) + + @spec refresh_now(atom()) :: :ok | {:error, term()} + def refresh_now(sup), + do: Scheduler.refresh_now(AdaptorsSupervisor.global_scheduler_name(sup)) + + @spec refresh_package(String.t()) :: :ok | {:error, :not_found | term()} + def refresh_package(name) when is_binary(name), do: refresh_package(@sup, name) + + @spec refresh_package(atom(), String.t()) :: + :ok | {:error, :not_found | term()} + def refresh_package(sup, name) when is_binary(name), + do: + Scheduler.refresh_package( + AdaptorsSupervisor.global_scheduler_name(sup), + name + ) + + @spec refresh_icons() :: + {:ok, %{updated: non_neg_integer(), unchanged: non_neg_integer()}} + | {:error, term()} + def refresh_icons, do: refresh_icons(@sup) + + @spec refresh_icons(atom()) :: + {:ok, %{updated: non_neg_integer(), unchanged: non_neg_integer()}} + | {:error, term()} + def refresh_icons(sup), + do: Scheduler.refresh_icons(AdaptorsSupervisor.global_scheduler_name(sup)) + + @doc false + def icon_meta(name), do: icon_meta(@sup, name) + + @doc false + def icon_meta(sup, name), do: Store.icon_meta(sup, name) +end diff --git a/lib/lightning/adaptors/channel_broadcaster.ex b/lib/lightning/adaptors/channel_broadcaster.ex new file mode 100644 index 00000000000..85b598374c0 --- /dev/null +++ b/lib/lightning/adaptors/channel_broadcaster.ex @@ -0,0 +1,91 @@ +defmodule Lightning.Adaptors.ChannelBroadcaster do + @moduledoc """ + Burst-coalesced fan-out of adaptor changes to connected sessions. + + Subscribes to `:source_topic` (the cache-coherence topic shared with + `Lightning.Adaptors.Invalidator`) and republishes a single pre-rendered + envelope to `:client_topic` at most once per 250ms leading-edge window. + + Two-topic separation: the source topic is the cache-coherence audience; + the client topic is the display-freshness audience (`WorkflowChannel` + subscribers). This bridges them: `Lightning.Adaptors.packages/1` is + rendered once per burst and fanned out by PubSub rather than once per + session (§6.5c). + + No within-callback fan-out in `:flush` — `Phoenix.PubSub.broadcast/3` + is a single call that reaches all subscribers in one hop (§10 #19). + """ + + use GenServer + + @debounce_ms 250 + + @doc """ + Leading-edge coalesce window in milliseconds. + + Exposed so integration tests can compute receive timeouts off the + authoritative value rather than hard-coding a duplicate. + """ + @spec debounce_ms() :: pos_integer() + def debounce_ms, do: @debounce_ms + + @doc """ + Start the ChannelBroadcaster linked to the calling process. + + Required opts: + * `:name` — registered GenServer name. + * `:source_topic` — PubSub topic to subscribe to (cache-coherence). + * `:client_topic` — PubSub topic to broadcast the rendered envelope to. + * `:sup` — supervisor instance name; forwarded to + `Lightning.Adaptors.packages/1` for per-instance isolation. + """ + @spec start_link(keyword()) :: GenServer.on_start() + def start_link(opts) do + name = Keyword.fetch!(opts, :name) + GenServer.start_link(__MODULE__, opts, name: name) + end + + @impl true + def init(opts) do + :ok = + Phoenix.PubSub.subscribe( + Lightning.PubSub, + Keyword.fetch!(opts, :source_topic) + ) + + {:ok, + %{ + client_topic: Keyword.fetch!(opts, :client_topic), + sup: Keyword.fetch!(opts, :sup), + timer: nil + }} + end + + @impl true + # First message of a burst: arm the leading-edge timer. + def handle_info({:changed, _name, _source}, %{timer: nil} = state) do + timer = Process.send_after(self(), :flush, @debounce_ms) + {:noreply, %{state | timer: timer}} + end + + # Subsequent messages within the debounce window: drop on the floor. + def handle_info({:changed, _name, _source}, state) do + {:noreply, state} + end + + def handle_info(:flush, %{client_topic: topic, sup: sup} = state) do + case Lightning.Adaptors.packages(sup) do + {:ok, pkgs} -> + Phoenix.PubSub.broadcast( + Lightning.PubSub, + topic, + %{event: "adaptors_updated", payload: %{adaptors: pkgs}} + ) + + {:error, _} -> + :ok + end + + {:noreply, %{state | timer: nil}} + end +end diff --git a/lib/lightning/adaptors/config.ex b/lib/lightning/adaptors/config.ex new file mode 100644 index 00000000000..8f45fd8cfcf --- /dev/null +++ b/lib/lightning/adaptors/config.ex @@ -0,0 +1,102 @@ +defmodule Lightning.Adaptors.Config do + @moduledoc """ + Stateless runtime configuration for the `Lightning.Adaptors.*` subsystem. + + Every helper is a thin wrapper around `Application.get_env/3`. It is the + single runtime source of truth for which strategy is active, how often + the scheduler ticks, the per-call cache fetch deadline, the icon cache + root, and per-strategy opt blocks. + + ## Application key layout + + Two-tier: + + * `:lightning, Lightning.Adaptors` — subsystem-wide knobs + (`:strategy`, `:refresh_interval`, `:cache_timeout_ms`, `:icon_path`). + * `:lightning, ` — each strategy owns its own + Application key for its own knobs; read via `strategy_opts/1`. + + No GenServer, no ETS, no `:persistent_term` — every call is a fresh + `Application.get_env/3`. + """ + + @parent_key Lightning.Adaptors + + @default_strategy Lightning.Adaptors.NPM + @default_refresh_interval :timer.hours(1) + @default_cache_timeout_ms 15_000 + @default_icon_path {:tmp, "lightning/adaptor_icons"} + + @doc """ + The active strategy module. Defaults to `Lightning.Adaptors.NPM`. + """ + @spec strategy() :: module() + def strategy do + get(:strategy, @default_strategy) + end + + @doc """ + Atom mapping of `strategy/0`: `:local` for `Lightning.Adaptors.Local`, + `:npm` for any other strategy module. + """ + @spec current_source() :: :local | :npm + def current_source do + case strategy() do + Lightning.Adaptors.Local -> :local + _other -> :npm + end + end + + @doc """ + Scheduler tick interval in milliseconds. Defaults to one hour. + """ + @spec refresh_interval() :: non_neg_integer() + def refresh_interval do + get(:refresh_interval, @default_refresh_interval) + end + + @doc """ + Per-`Cachex.fetch` courier deadline in milliseconds. Defaults to 15s. + """ + @spec cache_timeout_ms() :: non_neg_integer() + def cache_timeout_ms do + get(:cache_timeout_ms, @default_cache_timeout_ms) + end + + @doc """ + Resolved filesystem path for the icon cache. + + Accepts either: + + * `{:tmp, suffix}` — resolved against `System.tmp_dir!/0` at call + time so the default does not bake a container-specific tmp path + into a compiled release. + * a plain binary path — returned verbatim. + + Defaults to `{:tmp, "lightning/adaptor_icons"}`. + """ + @spec icon_path() :: Path.t() + def icon_path do + case get(:icon_path, @default_icon_path) do + {:tmp, suffix} -> Path.join(System.tmp_dir!(), suffix) + path when is_binary(path) -> path + end + end + + @doc """ + Per-strategy keyword opts. Parameterised on the strategy module — each + strategy is its own Application key, not nested under the parent. + Returns `[]` when the strategy's Application key is unset. + """ + @spec strategy_opts(module()) :: keyword() + def strategy_opts(strategy_mod) when is_atom(strategy_mod) do + Application.get_env(:lightning, strategy_mod, []) + end + + @spec get(atom(), term()) :: term() + defp get(key, default) do + :lightning + |> Application.get_env(@parent_key, []) + |> Keyword.get(key, default) + end +end diff --git a/lib/lightning/adaptors/icon_cache.ex b/lib/lightning/adaptors/icon_cache.ex new file mode 100644 index 00000000000..22147f68139 --- /dev/null +++ b/lib/lightning/adaptors/icon_cache.ex @@ -0,0 +1,96 @@ +defmodule Lightning.Adaptors.IconCache do + @moduledoc """ + Pure filesystem helper owning the on-disk adaptor icon cache. + + Not a GenServer. Three stateless functions over + `Lightning.Adaptors.Config.icon_path/0`, which resolves the + `{:tmp, suffix}` default at call time. + + Disk layout is **source-partitioned** and **latest-only**: + + ///. + + Source partitioning means flipping `LOCAL_ADAPTORS` between restarts + cannot accidentally serve `:npm` bytes from a row that's now resolved + via `:local` (or vice versa). Latest-only means a subsequent + `write!/5` for the same key overwrites — content-addressable URLs + carry the sha8 prefix, so cache invalidation is intrinsic and we + don't need to keep old versions on disk. + + Concurrent first-request fetchers are coalesced upstream by Cachex's + courier on `{:icon_bytes, source, name, shape}` inside + `Lightning.Adaptors.Store.icon/3` — the courier returns `{:ignore, _}` + so no entry is committed, but all in-flight peers receive the courier's + result for free. The temp-then-rename in `write!/5` is the belt-and- + braces guarantee for the file-write step itself: readers never observe + a half-written file. + """ + + alias Lightning.Adaptors.Config + + @type source :: :npm | :local + @type name :: String.t() + @type shape :: :square | :rectangle + @type ext :: String.t() + + @doc """ + Disk path for an icon. Pure — nothing is checked or created. + + `name` may contain a `/` (scoped npm packages like + `@openfn/language-foo`); `Path.join/1` preserves the slash so the + scope becomes a real subdirectory. + """ + @spec path(source(), name(), shape(), ext()) :: Path.t() + def path(source, name, shape, ext) do + Path.join([Config.icon_path(), to_string(source), name, "#{shape}.#{ext}"]) + end + + @doc """ + Whether the icon at `path(source, name, shape, ext)` exists on disk. + """ + @spec cached?(source(), name(), shape(), ext()) :: boolean() + def cached?(source, name, shape, ext) do + File.exists?(path(source, name, shape, ext)) + end + + @doc """ + Atomically write `bytes` to `path(source, name, shape, ext)` and + return the sha256 of the supplied bytes as a 32-byte binary. + + The write is staged in a sibling temp file and then renamed into + place, so concurrent readers never observe a half-written file. + + The caller (Strategy / Scheduler) persists the returned sha on the + adaptor row and is responsible for verifying it matches the expected + sha from the upstream `adaptor_record` — defence against tarball or + filesystem corruption. + """ + @spec write!(source(), name(), shape(), ext(), binary()) :: + {:ok, binary()} + def write!(source, name, shape, ext, bytes) when is_binary(bytes) do + final_path = path(source, name, shape, ext) + dir = Path.dirname(final_path) + File.mkdir_p!(dir) + + sha = :crypto.hash(:sha256, bytes) + + temp_path = + Path.join(dir, ".#{Path.basename(final_path)}.#{random_suffix()}.tmp") + + try do + File.write!(temp_path, bytes) + File.rename!(temp_path, final_path) + rescue + e -> + _ = File.rm(temp_path) + reraise e, __STACKTRACE__ + end + + {:ok, sha} + end + + @spec random_suffix() :: String.t() + defp random_suffix do + 8 |> :crypto.strong_rand_bytes() |> Base.encode16(case: :lower) + end +end diff --git a/lib/lightning/adaptors/invalidator.ex b/lib/lightning/adaptors/invalidator.ex new file mode 100644 index 00000000000..a548c23ee13 --- /dev/null +++ b/lib/lightning/adaptors/invalidator.ex @@ -0,0 +1,45 @@ +defmodule Lightning.Adaptors.Invalidator do + @moduledoc """ + Subscribes to cluster adaptor-change broadcasts and evicts matching + local Cachex entries, keeping each node coherent with Postgres. + + Subscribes to `opts[:source_topic]` on `Lightning.PubSub` at init. + On `{:changed, name, source}`, deletes the four per-adaptor cache keys + written by `Lightning.Adaptors.Store`. No source filtering on the hot + path — a broadcast for a source that isn't active on this node is a + no-op because those keys simply don't exist in Cachex. + """ + + use GenServer + + @doc """ + Start the Invalidator linked to the calling process. + + Required opts: + * `:name` — registered process name. + * `:source_topic` — `Phoenix.PubSub` topic to subscribe to. + * `:cache` — Cachex table atom (from `Lightning.Adaptors.Supervisor.cache_name/1`). + """ + @spec start_link(keyword()) :: GenServer.on_start() + def start_link(opts) do + name = Keyword.fetch!(opts, :name) + GenServer.start_link(__MODULE__, opts, name: name) + end + + @impl true + def init(opts) do + topic = Keyword.fetch!(opts, :source_topic) + cache = Keyword.fetch!(opts, :cache) + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, topic) + {:ok, %{cache: cache}} + end + + @impl true + def handle_info({:changed, name, source}, state) do + Cachex.del(state.cache, {:schema, name, source}) + Cachex.del(state.cache, {:versions, name, source}) + Cachex.del(state.cache, {:icon_meta, name, source}) + Cachex.del(state.cache, {:packages, source}) + {:noreply, state} + end +end diff --git a/lib/lightning/adaptors/local.ex b/lib/lightning/adaptors/local.ex new file mode 100644 index 00000000000..c47a942af6f --- /dev/null +++ b/lib/lightning/adaptors/local.ex @@ -0,0 +1,229 @@ +defmodule Lightning.Adaptors.Local do + @moduledoc """ + Filesystem implementation of `Lightning.Adaptors.Strategy`. + + Serves adaptor metadata, schemas, and icons from an on-disk OpenFn + adaptors monorepo checkout. Gated by `LOCAL_ADAPTORS=true` and + `OPENFN_ADAPTORS_REPO=/path/to/adaptors` at the runtime-config layer; + this module only reads the resolved path via + `Lightning.Adaptors.Config.strategy_opts(__MODULE__)[:path]`. + + Each callback walks the filesystem afresh — caching is the Store's + responsibility. The module is stateless; no GenServer, no ETS. + + ## Layout + + Walks `$path/packages/*/`, reads each subdirectory's `package.json` + for the authoritative `name` and `version`. Directories with missing + or unparseable `package.json` are skipped with `Logger.warning` so a + malformed entry never crashes boot. Multiple directories sharing the + same `name` are collapsed into one record: `latest_version` is the + highest semver and `versions` lists every on-disk path. + + `source: :local` is **not** set here — the Store stamps it before + upsert. No network calls anywhere in this module. + """ + + @behaviour Lightning.Adaptors.Strategy + + alias Lightning.Adaptors.Config + + require Logger + + @schema_filename "configuration-schema.json" + @icon_exts ~w(png svg) + + @impl Lightning.Adaptors.Strategy + def list_adaptors do + with {:ok, records} <- discover() do + {:ok, + Enum.map(records, fn %{name: name, latest_version: v} -> + %{name: name, latest_version: v} + end)} + end + end + + @impl Lightning.Adaptors.Strategy + def fetch_adaptor(name) when is_binary(name) do + with {:ok, records} <- discover() do + case Enum.find(records, &(&1.name == name)) do + nil -> {:error, :not_found} + record -> {:ok, build_adaptor_record(record)} + end + end + end + + @impl Lightning.Adaptors.Strategy + def fetch_icon(name, shape) + when is_binary(name) and shape in [:square, :rectangle] do + with {:ok, records} <- discover() do + case Enum.find(records, &(&1.name == name)) do + nil -> {:error, :not_found} + %{latest_path: path} -> read_icon(path, shape) + end + end + end + + @impl Lightning.Adaptors.Strategy + def fetch_icons(_opts \\ []) do + with {:ok, records} <- discover() do + icons = + Enum.reduce(records, %{}, fn record, acc -> + Enum.reduce([:square, :rectangle], acc, fn shape, inner -> + case read_icon(record.latest_path, shape) do + {:ok, %{data: bytes, ext: ext}} -> + entry = %{ + data: bytes, + ext: ext, + sha256: :crypto.hash(:sha256, bytes) + } + + Map.update( + inner, + record.name, + %{shape => entry}, + &Map.put(&1, shape, entry) + ) + + {:error, _} -> + inner + end + end) + end) + + {:ok, icons} + end + end + + defp discover do + case Config.strategy_opts(__MODULE__)[:path] do + nil -> + Logger.warning( + "Lightning.Adaptors.Local: :path is not configured " <> + "(set OPENFN_ADAPTORS_REPO or :lightning, Lightning.Adaptors.Local, path:)" + ) + + {:error, :no_repo_path} + + path -> + records = + path + |> Path.join("packages") + |> Path.join("*") + |> Path.wildcard() + |> Enum.filter(&File.dir?/1) + |> Enum.flat_map(&read_package_dir/1) + |> group_by_name() + + {:ok, records} + end + end + + defp read_package_dir(dir) do + pkg_json_path = Path.join(dir, "package.json") + + with {:ok, body} <- File.read(pkg_json_path), + {:ok, %{"name" => name, "version" => version} = parsed} + when is_binary(name) and is_binary(version) <- Jason.decode(body) do + [%{name: name, version: version, path: dir, package_json: parsed}] + else + other -> + Logger.warning( + "Lightning.Adaptors.Local: skipping #{inspect(dir)}: " <> + "missing or unparseable package.json (#{inspect(other)})" + ) + + [] + end + end + + defp group_by_name(entries) do + entries + |> Enum.group_by(& &1.name) + |> Enum.map(fn {name, versions} -> + sorted = Enum.sort_by(versions, & &1.version, &version_descending/2) + latest = List.first(sorted) + + %{ + name: name, + latest_version: latest.version, + latest_path: latest.path, + latest_package_json: latest.package_json, + versions: sorted + } + end) + end + + defp version_descending(a, b) do + case {Version.parse(a), Version.parse(b)} do + {{:ok, va}, {:ok, vb}} -> Version.compare(va, vb) != :lt + _ -> a >= b + end + end + + defp build_adaptor_record(record) do + pkg = record.latest_package_json + {schema_data, schema_sha256} = read_schema(record.latest_path) + + %{ + name: record.name, + description: pkg["description"], + homepage: pkg["homepage"], + repository: extract_repository(pkg["repository"]), + license: pkg["license"], + latest_version: record.latest_version, + deprecated: false, + schema_data: schema_data, + schema_sha256: schema_sha256, + versions: Enum.map(record.versions, &build_version_record/1) + } + end + + defp build_version_record(%{version: v, package_json: pkg}) do + %{ + version: v, + integrity: nil, + tarball_url: nil, + size_bytes: nil, + dependencies: Map.get(pkg, "dependencies", %{}), + peer_dependencies: Map.get(pkg, "peerDependencies", %{}), + published_at: nil, + deprecated: false + } + end + + defp read_schema(dir) do + case File.read(Path.join(dir, @schema_filename)) do + {:ok, body} -> + # Validate JSON, but keep the raw binary so credential-form + # rendering can re-engage ordered_objects decoding downstream. + case Jason.decode(body) do + {:ok, _data} -> + sha = :sha256 |> :crypto.hash(body) |> Base.encode16(case: :lower) + {body, sha} + + {:error, _} -> + {nil, nil} + end + + {:error, _} -> + {nil, nil} + end + end + + defp read_icon(dir, shape) do + Enum.find_value(@icon_exts, {:error, :not_found}, fn ext -> + case File.read(icon_path(dir, shape, ext)) do + {:ok, bytes} -> {:ok, %{data: bytes, ext: ext}} + {:error, _} -> nil + end + end) + end + + defp icon_path(dir, shape, ext), + do: Path.join([dir, "assets", "#{shape}.#{ext}"]) + + defp extract_repository(repo) when is_binary(repo), do: repo + defp extract_repository(%{"url" => url}) when is_binary(url), do: url + defp extract_repository(_), do: nil +end diff --git a/lib/lightning/adaptors/node_monitor.ex b/lib/lightning/adaptors/node_monitor.ex new file mode 100644 index 00000000000..3ecc0d3ab8f --- /dev/null +++ b/lib/lightning/adaptors/node_monitor.ex @@ -0,0 +1,48 @@ +defmodule Lightning.Adaptors.NodeMonitor do + @moduledoc """ + Partition-recovery companion to `Lightning.Adaptors.Invalidator`. + + On `:nodeup`, re-warms the Cachex table from Postgres so a reconnecting + peer never serves stale data until the 24-hour TTL expires. Steady-state + invalidation belongs to `Lightning.Adaptors.Invalidator`. + + `:nodedown` is a deliberate no-op. The worst case on a silent departure is + one stale-URL redirect per client, backstopped by 302-on-stale-sha. + """ + + use GenServer + + alias Lightning.Adaptors.Store + + @doc """ + Start a NodeMonitor for the given supervisor instance. + + Required opts: + * `:name` — registered GenServer name (§6.11 async-test rule). + * `:sup` — supervisor instance name, forwarded to `Store.warm_from_repo/1`. + """ + @spec start_link(keyword()) :: GenServer.on_start() + def start_link(opts) do + name = Keyword.fetch!(opts, :name) + GenServer.start_link(__MODULE__, opts, name: name) + end + + @impl true + def init(opts) do + sup = Keyword.fetch!(opts, :sup) + :net_kernel.monitor_nodes(true, node_type: :visible) + {:ok, %{sup: sup}} + end + + @impl true + def handle_info({:nodeup, _node, _info}, state) do + Store.warm_from_repo(state.sup) + {:noreply, state} + end + + # Deliberate no-op: nodedown does not trigger a re-warm. The 24h Cachex TTL + # backstops any staleness; 302-on-stale-sha handles already-issued URLs. + def handle_info({:nodedown, _node, _info}, state) do + {:noreply, state} + end +end diff --git a/lib/lightning/adaptors/npm.ex b/lib/lightning/adaptors/npm.ex new file mode 100644 index 00000000000..d5ed5e44c29 --- /dev/null +++ b/lib/lightning/adaptors/npm.ex @@ -0,0 +1,108 @@ +defmodule Lightning.Adaptors.NPM do + @moduledoc """ + Production implementation of `Lightning.Adaptors.Strategy` that talks + to the public NPM registry and the OpenFn adaptors monorepo on GitHub. + + Consolidates the legacy `Lightning.AdaptorRegistry`, + `Mix.Tasks.Lightning.InstallSchemas`, and + `Mix.Tasks.Lightning.InstallAdaptorIcons` into one stateless module: + + * `c:list_adaptors/0` — single search-API call returning + `name + latest_version` for every `@openfn/language-*` package. + * `c:fetch_adaptor/1` — packument fetch + per-version decode and + latest-version schema retrieval via jsDelivr. Icon fields are + **not** stamped here; the Scheduler joins them on after a bulk + `c:fetch_icons/1` pass. + * `c:fetch_icon/2` — single icon raw GET against + `raw.githubusercontent.com`, used by the Store's rare lazy-miss + fallback. + * `c:fetch_icons/1` — bulk fan-out over the search listing, one + HTTP request per `(name, shape)`. Threads `:prior_etags` from + the caller down into the per-request `If-None-Match` headers. + + ## HTTP + + This module is a thin orchestrator. The actual HTTP work is delegated + to three sub-modules, each of which owns its own Tesla client and + upstream base URL: + + * `Lightning.Adaptors.NPM.Registry` — npm registry search + packument. + * `Lightning.Adaptors.NPM.Schema` — jsDelivr `configuration-schema.json`. + * `Lightning.Adaptors.NPM.GitHub` — `raw.githubusercontent.com` + icon fetches (one GET per `(name, shape)`). + + Each sub-module issues at most a handful of single-shot Tesla requests + bounded by `http_timeout`. No retry, no backoff, no circuit-breaker — + transient failures (5xx, timeout, nxdomain) of the *primary* request + (`packument` for `fetch_adaptor/1`, `search` for `list_adaptors/0` and + `fetch_icons/1`) surface as `{:error, term()}` unchanged. Schema and + icon fetches inside `fetch_adaptor/1` and `fetch_icons/1` are + best-effort: a single icon miss degrades that entry to absence rather + than failing the whole record. + + ## Configuration + + Each sub-module reads `:registry_url`, `:jsdelivr_url`, `:github_url`, + `:github_ref`, and `:http_timeout` via + `Lightning.Adaptors.Config.strategy_opts(__MODULE__)`, with defaults + baked in so the module works even when no Application env block is + set. + """ + + @behaviour Lightning.Adaptors.Strategy + + alias Lightning.Adaptors.NPM.GitHub + alias Lightning.Adaptors.NPM.Registry + alias Lightning.Adaptors.NPM.Schema + + @impl Lightning.Adaptors.Strategy + def list_adaptors, do: Registry.list_adaptors() + + @impl Lightning.Adaptors.Strategy + def fetch_adaptor(name) when is_binary(name) do + with {:ok, packument} <- Registry.get_packument(name), + {:ok, latest_version} <- Registry.latest_version(packument) do + {schema_data, schema_sha} = Schema.schema(name, latest_version) + + {:ok, + %{ + name: Map.get(packument, "name", name), + description: Map.get(packument, "description"), + homepage: Map.get(packument, "homepage"), + repository: Registry.repository_url(Map.get(packument, "repository")), + license: Map.get(packument, "license"), + latest_version: latest_version, + deprecated: Registry.deprecated?(packument, latest_version), + schema_data: encode_schema(schema_data), + schema_sha256: schema_sha, + versions: Registry.build_versions(packument) + }} + end + end + + # Strategy boundary: re-encode the decoded schema map to a JSON binary + # so the row is persisted as text and `Jason.decode!(_, + # objects: :ordered_objects)` re-engages downstream. NPM's upstream + # Schema sub-module already decoded into a regular map, so field order + # is whatever map iteration yields — the round-trip preserves it for + # the Local strategy (raw binary in) and is a no-op for NPM data. + defp encode_schema(nil), do: nil + defp encode_schema(data) when is_binary(data), do: data + defp encode_schema(data) when is_map(data), do: Jason.encode!(data) + + @impl Lightning.Adaptors.Strategy + def fetch_icon(name, shape) + when is_binary(name) and shape in [:square, :rectangle] do + GitHub.fetch_one(name, shape) + end + + @impl Lightning.Adaptors.Strategy + def fetch_icons(opts \\ []) when is_list(opts) do + prior_etags = Keyword.get(opts, :prior_etags, %{}) + + with {:ok, listing} <- Registry.list_adaptors() do + names = Enum.map(listing, & &1.name) + GitHub.fetch_all(names, prior_etags) + end + end +end diff --git a/lib/lightning/adaptors/npm/github.ex b/lib/lightning/adaptors/npm/github.ex new file mode 100644 index 00000000000..1e2a8096cd4 --- /dev/null +++ b/lib/lightning/adaptors/npm/github.ex @@ -0,0 +1,302 @@ +defmodule Lightning.Adaptors.NPM.GitHub do + @moduledoc """ + Raw `raw.githubusercontent.com` client for adaptor icons. + + Icons aren't published inside npm tarballs — they live in the + `OpenFn/adaptors` monorepo. This module fetches them directly via the + raw GitHub content host, one icon per HTTP GET, no tarball walking. + + ## URL pattern + + /OpenFn/adaptors//packages//assets/. + + where `` strips the `@openfn/` scope from the package name. + Each `(name, shape)` is probed `png` first then `svg` — matching the + ext order used by `Lightning.Adaptors.Local`. + + ## Configuration + + Both `:github_url` (default `https://raw.githubusercontent.com`) and + `:github_ref` (default `main`) are read via + `Lightning.Adaptors.Config.strategy_opts(Lightning.Adaptors.NPM)`, + symmetric with the existing `:registry_url`, `:jsdelivr_url`, and + `:http_timeout` keys. + """ + + alias Lightning.Adaptors.Config + + require Logger + + @default_github_url "https://raw.githubusercontent.com" + @default_github_ref "main" + @default_http_timeout :timer.seconds(30) + + @default_max_concurrency 20 + + @icon_exts ~w(png svg) + @scope_prefix "@openfn/" + @language_prefix "language-" + + @doc """ + Fetch a single icon for `(name, shape)`. + + Tries `png` then `svg`. No conditional GET — this entry point is used + by the Store's lazy-miss fallback, where no prior etag is in scope. + Returns: + + * `{:ok, %{data: binary(), ext: String.t(), etag: String.t() | nil}}` + on success. + * `{:error, :not_found}` when neither ext yields a 200. + * `{:error, term()}` on transport-level failure (timeout, nxdomain). + """ + @spec fetch_one(String.t(), :square | :rectangle) :: + {:ok, %{data: binary(), ext: String.t(), etag: String.t() | nil}} + | {:error, :not_found | term()} + def fetch_one(name, shape) + when is_binary(name) and shape in [:square, :rectangle] do + client = raw_client() + do_fetch_one(client, name, shape, nil) + end + + @doc """ + Fetch icons for every `(name, shape)` pair across `names`. + + Returns `{:ok, partial_map}` where each entry is keyed by the + package name and contains zero, one, or two shape keys. Absence + is **not** an error — packages with no upstream icon simply do not + appear (or appear with a missing shape). + + When `prior_etags` is supplied as `%{name => %{shape => etag}}`, the + corresponding `If-None-Match` header is sent per `(name, shape)`. A + 304 response is surfaced as a `:not_modified` sentinel in the + per-shape slot — distinct from "absent" which means upstream had no + such shape at all. + + Fans out via `Task.async_stream` with a bounded concurrency. Transport + failures for a single `(name, shape)` are dropped silently — the whole + pipeline only fails if every fetch crashes the supervisor, which is + not surfaced here. + """ + @spec fetch_all([String.t()], %{ + optional(String.t()) => %{ + optional(:square) => String.t(), + optional(:rectangle) => String.t() + } + }) :: + {:ok, + %{ + required(String.t()) => %{ + optional(:square) => + %{ + data: binary(), + ext: String.t(), + sha256: binary(), + etag: String.t() | nil + } + | :not_modified, + optional(:rectangle) => + %{ + data: binary(), + ext: String.t(), + sha256: binary(), + etag: String.t() | nil + } + | :not_modified + } + }} + def fetch_all(names, prior_etags) + when is_list(names) and is_map(prior_etags) do + client = raw_client() + + work = + for name <- names, shape <- [:square, :rectangle], do: {name, shape} + + {results, counts} = + work + |> Task.async_stream( + fn {name, shape} -> + prior = prior_etag_for(prior_etags, name, shape) + + case do_fetch_one(client, name, shape, prior) do + {:ok, %{data: bytes, ext: ext, etag: etag}} -> + {name, shape, + {:ok, + %{ + data: bytes, + ext: ext, + sha256: :crypto.hash(:sha256, bytes), + etag: etag + }}} + + :not_modified -> + {name, shape, :not_modified} + + {:error, reason} -> + {name, shape, {:error, reason}} + end + end, + max_concurrency: max_concurrency(), + timeout: max(http_timeout() * 2, 5_000), + on_timeout: :kill_task, + ordered: false + ) + |> Enum.reduce( + {%{}, %{fetched: 0, not_modified: 0, not_found: 0, error: 0}}, + fn + {:ok, {name, shape, {:ok, entry}}}, {acc, c} -> + {put_entry(acc, name, shape, entry), + Map.update!(c, :fetched, &(&1 + 1))} + + {:ok, {name, shape, :not_modified}}, {acc, c} -> + {put_entry(acc, name, shape, :not_modified), + Map.update!(c, :not_modified, &(&1 + 1))} + + {:ok, {_name, _shape, {:error, :not_found}}}, {acc, c} -> + {acc, Map.update!(c, :not_found, &(&1 + 1))} + + {:ok, {_name, _shape, {:error, _reason}}}, {acc, c} -> + {acc, Map.update!(c, :error, &(&1 + 1))} + + {:exit, _reason}, {acc, c} -> + {acc, Map.update!(c, :error, &(&1 + 1))} + end + ) + + Logger.info( + "NPM.GitHub: fetch_all names=#{length(names)} pairs=#{length(work)} " <> + "fetched=#{counts.fetched} not_modified=#{counts.not_modified} " <> + "not_found=#{counts.not_found} errors=#{counts.error}" + ) + + {:ok, results} + end + + defp prior_etag_for(prior_etags, name, shape) do + case Map.get(prior_etags, name) do + %{} = shapes -> Map.get(shapes, shape) + _ -> nil + end + end + + defp put_entry(acc, name, shape, entry) do + Map.update(acc, name, %{shape => entry}, &Map.put(&1, shape, entry)) + end + + # GitHub raw does not gzip-compress PNG/SVG bodies, so no + # `Tesla.Middleware.DecompressResponse` is in play. The sha256 computed + # in `fetch_all/2` is therefore over the raw response bytes (which equals + # the decompressed bytes in our case — there is no compression layer to + # speak of, and no `accept-encoding` middleware is configured). If GitHub + # ever starts compressing 200 bodies, Tesla/Finch would need an + # accept-encoding/decompress middleware to preserve this invariant. + defp do_fetch_one(client, name, shape, prior_etag) do + suffix = strip_scope(name) + cond_get? = is_binary(prior_etag) + + headers = + if cond_get?, do: [{"if-none-match", prior_etag}], else: [] + + Enum.reduce_while(@icon_exts, {:error, :not_found}, fn ext, _acc -> + path = build_path(suffix, shape, ext) + + case Tesla.get(client, path, headers: headers) do + {:ok, %Tesla.Env{status: 200, body: body} = env} when is_binary(body) -> + etag = response_etag(env) + + Logger.debug(fn -> + "NPM.GitHub: GET #{path} → 200 (#{byte_size(body)}B) " <> + "name=#{name} shape=#{shape} cond_get=#{cond_get?}" + end) + + {:halt, {:ok, %{data: body, ext: ext, etag: etag}}} + + {:ok, %Tesla.Env{status: 304}} -> + Logger.debug(fn -> + "NPM.GitHub: GET #{path} → 304 name=#{name} shape=#{shape} " <> + "cond_get=#{cond_get?}" + end) + + {:halt, :not_modified} + + {:ok, %Tesla.Env{status: 404}} -> + Logger.debug(fn -> + "NPM.GitHub: GET #{path} → 404 name=#{name} shape=#{shape} " <> + "cond_get=#{cond_get?}" + end) + + {:cont, {:error, :not_found}} + + {:ok, %Tesla.Env{status: status}} -> + Logger.debug(fn -> + "NPM.GitHub: GET #{path} → #{status} name=#{name} shape=#{shape} " <> + "cond_get=#{cond_get?}" + end) + + {:halt, {:error, {:http_status, status}}} + + {:error, reason} -> + Logger.debug(fn -> + "NPM.GitHub: GET #{path} → transport error #{inspect(reason)} " <> + "name=#{name} shape=#{shape} cond_get=#{cond_get?}" + end) + + {:halt, {:error, reason}} + end + end) + end + + defp response_etag(%Tesla.Env{headers: headers}) do + Enum.find_value(headers, fn {k, v} -> + if is_binary(k) and String.downcase(k) == "etag", do: v + end) + end + + defp build_path(name_suffix, shape, ext) do + "/OpenFn/adaptors/#{github_ref()}/packages/#{name_suffix}/assets/#{shape}.#{ext}" + end + + defp strip_scope(@scope_prefix <> @language_prefix <> rest), do: rest + defp strip_scope(@scope_prefix <> rest), do: rest + defp strip_scope(name), do: name + + defp raw_client do + build_client([ + {Tesla.Middleware.BaseUrl, github_url()}, + Tesla.Middleware.FollowRedirects + ]) + end + + defp build_client(middleware) do + case Application.get_env(:tesla, :adapter) do + {Tesla.Adapter.Finch, _opts} -> + Tesla.client( + middleware, + {Tesla.Adapter.Finch, + name: Lightning.Finch, receive_timeout: http_timeout()} + ) + + _other -> + Tesla.client(middleware) + end + end + + defp github_url do + Config.strategy_opts(Lightning.Adaptors.NPM)[:github_url] || + @default_github_url + end + + defp github_ref do + Config.strategy_opts(Lightning.Adaptors.NPM)[:github_ref] || + @default_github_ref + end + + defp http_timeout do + Config.strategy_opts(Lightning.Adaptors.NPM)[:http_timeout] || + @default_http_timeout + end + + defp max_concurrency do + Config.strategy_opts(Lightning.Adaptors.NPM)[:icon_max_concurrency] || + @default_max_concurrency + end +end diff --git a/lib/lightning/adaptors/npm/registry.ex b/lib/lightning/adaptors/npm/registry.ex new file mode 100644 index 00000000000..a05e93dfb02 --- /dev/null +++ b/lib/lightning/adaptors/npm/registry.ex @@ -0,0 +1,195 @@ +defmodule Lightning.Adaptors.NPM.Registry do + @moduledoc """ + NPM registry HTTP client for `Lightning.Adaptors.NPM`. + + Talks to `registry.npmjs.org`. Responsible for the `search` endpoint + used by `c:Lightning.Adaptors.Strategy.list_adaptors/0` and the + `packument` endpoint used by `fetch_adaptor/1` and `fetch_icon/2`. + + Base URL via `Lightning.Adaptors.Config.strategy_opts(Lightning.Adaptors.NPM)[:registry_url]`, + default `https://registry.npmjs.org`. + + Search results are filtered down to `@openfn/language-*` packages, + matching the legacy `AdaptorRegistry` semantics; non-language packages + in the `@openfn/` scope (e.g. `@openfn/cli`) are rejected. + """ + + alias Lightning.Adaptors.Config + + @default_registry_url "https://registry.npmjs.org" + @default_http_timeout :timer.seconds(30) + + @search_scope "openfn" + @search_size 250 + + @language_prefix "@openfn/language-" + + @doc """ + Single `/-/v1/search` call returning `name + latest_version` for every + `@openfn/language-*` package. + """ + @spec list_adaptors() :: + {:ok, [%{name: String.t(), latest_version: String.t()}]} + | {:error, term()} + def list_adaptors do + case Tesla.get(json_client(), "/-/v1/search", + query: [text: "@" <> @search_scope, size: @search_size] + ) do + {:ok, %Tesla.Env{status: 200, body: body}} when is_map(body) -> + listing = + body + |> Map.get("objects", []) + |> Enum.map(&extract_listing_entry/1) + |> Enum.reject(&is_nil/1) + + {:ok, listing} + + {:ok, %Tesla.Env{status: status}} -> + {:error, {:http_status, status}} + + {:error, reason} -> + {:error, reason} + end + end + + @doc """ + Fetch the full packument for a package. + """ + @spec get_packument(String.t()) :: + {:ok, map()} | {:error, :not_found} | {:error, term()} + def get_packument(name) do + case Tesla.get(json_client(), "/" <> name) do + {:ok, %Tesla.Env{status: 200, body: body}} when is_map(body) -> + {:ok, body} + + {:ok, %Tesla.Env{status: 404}} -> + {:error, :not_found} + + {:ok, %Tesla.Env{status: status}} -> + {:error, {:http_status, status}} + + {:error, reason} -> + {:error, reason} + end + end + + @doc """ + Extract the `dist-tags.latest` version from a packument. + """ + @spec latest_version(map()) :: + {:ok, String.t()} | {:error, :no_latest_version} + def latest_version(packument) do + case get_in(packument, ["dist-tags", "latest"]) do + v when is_binary(v) -> {:ok, v} + _ -> {:error, :no_latest_version} + end + end + + @doc """ + Resolve the tarball URL for `version` in `packument`. + """ + @spec require_tarball_url(map(), String.t()) :: + {:ok, String.t()} | {:error, :no_tarball_url} + def require_tarball_url(packument, version) do + case get_in(packument, ["versions", version, "dist", "tarball"]) do + url when is_binary(url) -> {:ok, url} + _ -> {:error, :no_tarball_url} + end + end + + @doc """ + Build the per-version `version_record` list from a packument. + """ + @spec build_versions(map()) :: [map()] + def build_versions(packument) do + versions = Map.get(packument, "versions", %{}) + times = Map.get(packument, "time", %{}) + + Enum.map(versions, fn {version, info} -> + %{ + version: version, + integrity: get_in(info, ["dist", "integrity"]), + tarball_url: get_in(info, ["dist", "tarball"]), + size_bytes: get_in(info, ["dist", "unpackedSize"]), + dependencies: Map.get(info, "dependencies", %{}), + peer_dependencies: Map.get(info, "peerDependencies", %{}), + published_at: parse_time(Map.get(times, version)), + deprecated: deprecated_marker?(info) + } + end) + end + + @doc """ + Is the given `version` in `packument` flagged as deprecated? + """ + @spec deprecated?(map(), String.t()) :: boolean() + def deprecated?(packument, version) do + deprecated_marker?(get_in(packument, ["versions", version]) || %{}) + end + + @doc """ + Normalise the packument's `repository` field to a plain URL string. + """ + @spec repository_url(term()) :: String.t() | nil + def repository_url(%{"url" => url}) when is_binary(url), do: url + def repository_url(url) when is_binary(url), do: url + def repository_url(_), do: nil + + defp extract_listing_entry(%{ + "package" => %{"name" => name, "version" => version} + }) + when is_binary(name) and is_binary(version) do + if String.starts_with?(name, @language_prefix) do + %{name: name, latest_version: version} + end + end + + defp extract_listing_entry(_), do: nil + + defp deprecated_marker?(%{"deprecated" => v}) when is_binary(v) and v != "", + do: true + + defp deprecated_marker?(%{"deprecated" => true}), do: true + defp deprecated_marker?(_), do: false + + defp parse_time(time) when is_binary(time) do + case DateTime.from_iso8601(time) do + {:ok, dt, _offset} -> dt + _ -> nil + end + end + + defp parse_time(_), do: nil + + defp json_client do + build_client([ + {Tesla.Middleware.BaseUrl, registry_url()}, + Tesla.Middleware.JSON, + Tesla.Middleware.FollowRedirects + ]) + end + + defp build_client(middleware) do + case Application.get_env(:tesla, :adapter) do + {Tesla.Adapter.Finch, _opts} -> + Tesla.client( + middleware, + {Tesla.Adapter.Finch, + name: Lightning.Finch, receive_timeout: http_timeout()} + ) + + _other -> + Tesla.client(middleware) + end + end + + defp registry_url do + Config.strategy_opts(Lightning.Adaptors.NPM)[:registry_url] || + @default_registry_url + end + + defp http_timeout do + Config.strategy_opts(Lightning.Adaptors.NPM)[:http_timeout] || + @default_http_timeout + end +end diff --git a/lib/lightning/adaptors/npm/schema.ex b/lib/lightning/adaptors/npm/schema.ex new file mode 100644 index 00000000000..8975888e559 --- /dev/null +++ b/lib/lightning/adaptors/npm/schema.ex @@ -0,0 +1,83 @@ +defmodule Lightning.Adaptors.NPM.Schema do + @moduledoc """ + jsDelivr CDN client for adaptor configuration schemas. + + Fetches `/npm/@/configuration-schema.json` from + `cdn.jsdelivr.net`, decodes it, and returns + `{schema_data, schema_sha256}` (or `{nil, nil}` on any failure — + schema fetch is best-effort). + + Base URL via `Lightning.Adaptors.Config.strategy_opts(Lightning.Adaptors.NPM)[:jsdelivr_url]`, + default `https://cdn.jsdelivr.net`. + """ + + alias Lightning.Adaptors.Config + + @default_jsdelivr_url "https://cdn.jsdelivr.net" + @default_http_timeout :timer.seconds(30) + + @doc """ + Fetch the configuration schema for `name@version` from jsDelivr. + + Returns `{schema_data, schema_sha256}` on success, `{nil, nil}` on + any failure (best-effort — schema absence must not fail the adaptor + record assembly). + """ + @spec schema(String.t(), String.t()) :: + {map() | nil, String.t() | nil} + def schema(name, version) do + with {:ok, body} <- fetch_schema_bytes(name, version), + {:ok, data} <- Jason.decode(body) do + sha = :sha256 |> :crypto.hash(body) |> Base.encode16(case: :lower) + {data, sha} + else + _ -> {nil, nil} + end + end + + defp fetch_schema_bytes(name, version) do + url = "/npm/#{name}@#{version}/configuration-schema.json" + + case Tesla.get(jsdelivr_client(), url) do + {:ok, %Tesla.Env{status: 200, body: body}} when is_binary(body) -> + {:ok, body} + + {:ok, %Tesla.Env{status: status}} -> + {:error, {:http_status, status}} + + {:error, reason} -> + {:error, reason} + end + end + + defp jsdelivr_client do + build_client([ + {Tesla.Middleware.BaseUrl, jsdelivr_url()}, + Tesla.Middleware.FollowRedirects + ]) + end + + defp build_client(middleware) do + case Application.get_env(:tesla, :adapter) do + {Tesla.Adapter.Finch, _opts} -> + Tesla.client( + middleware, + {Tesla.Adapter.Finch, + name: Lightning.Finch, receive_timeout: http_timeout()} + ) + + _other -> + Tesla.client(middleware) + end + end + + defp jsdelivr_url do + Config.strategy_opts(Lightning.Adaptors.NPM)[:jsdelivr_url] || + @default_jsdelivr_url + end + + defp http_timeout do + Config.strategy_opts(Lightning.Adaptors.NPM)[:http_timeout] || + @default_http_timeout + end +end diff --git a/lib/lightning/adaptors/package_name.ex b/lib/lightning/adaptors/package_name.ex new file mode 100644 index 00000000000..a629fa74c9d --- /dev/null +++ b/lib/lightning/adaptors/package_name.ex @@ -0,0 +1,64 @@ +defmodule Lightning.Adaptors.PackageName do + @moduledoc """ + NPM-style package-name parsing and worker wire-shape recomposition for + the `Lightning.Adaptors.*` subsystem. + + This module is the single source of truth for the legacy + `AdaptorRegistry.resolve_adaptor/1` and `resolve_package_name/1` + contracts, ported to read through the `Lightning.Adaptors` facade. + + `parse/1` splits `"name@version"` strings; `to_wire/1` resolves the + `latest` literal through `Lightning.Adaptors.resolve_version/2`, + preserves `"name@local"` as a literal regardless of source, and emits + `"name@local"` under a `:local` strategy source. + """ + + alias Lightning.Adaptors + alias Lightning.Adaptors.Config + + @package_name_regex ~r/(@?[\/\d\n\w-]+)(?:@([\d\.\w-]+))?$/ + + @spec parse(nil) :: {nil, nil} + def parse(nil), do: {nil, nil} + + @spec parse(String.t()) :: {String.t() | nil, String.t() | nil} + def parse(package_name) when is_binary(package_name) do + case Regex.run(@package_name_regex, package_name) do + [_, name, version] -> {name, version} + [_, _name] -> {package_name, nil} + _ -> {nil, nil} + end + end + + @spec to_wire(String.t() | nil) :: String.t() + def to_wire(adaptor) do + case parse(adaptor) do + {nil, nil} -> "" + {name, version} -> recompose(name, version, adaptor) + end + end + + defp recompose(name, "local", _original), do: "#{name}@local" + + defp recompose(name, version, original) do + case Config.current_source() do + :local -> + "#{name}@local" + + _ -> + case version do + "latest" -> + case Adaptors.resolve_version(name, "latest") do + {:ok, resolved} -> "#{name}@#{resolved}" + {:error, _} -> "#{name}@latest" + end + + nil -> + original + + _concrete -> + original + end + end + end +end diff --git a/lib/lightning/adaptors/repo.ex b/lib/lightning/adaptors/repo.ex new file mode 100644 index 00000000000..f15f2ad690d --- /dev/null +++ b/lib/lightning/adaptors/repo.ex @@ -0,0 +1,339 @@ +defmodule Lightning.Adaptors.Repo do + @moduledoc """ + Query and write helpers over the `adaptors` and `adaptor_versions` tables. + + Despite the name, this is **not** an `Ecto.Repo` — it is a thin + data-access module that wraps `Lightning.Repo` (the real + `Ecto.Repo`). The two schemas it targets live as siblings: + `Lightning.Adaptors.Repo.Adaptor` and + `Lightning.Adaptors.Repo.AdaptorVersion`. + + Every read helper takes the desired `:source` (`:npm | :local`) + explicitly; the module itself stays source-agnostic. Callers resolve + the active source via `Lightning.Adaptors.Config.current_source/0` + (see §4.4 source-tagging invariant and §6.4 in + `.context/lightning/adaptors/REWRITE-2026-05.md`). + + `upsert_adaptor/1` is the only writer the Scheduler uses. It is + idempotent, transactional, and diff-aware: `checked_at` advances on + every call, while `updated_at` only advances when the row's + meaningful fields differ from what was already in the DB. Version + rows are replaced inside the same transaction so a partial failure + cannot leave the table half-rewritten. + """ + + import Ecto.Query + + alias Ecto.Multi + alias Lightning.Adaptors.Repo.Adaptor + alias Lightning.Adaptors.Repo.AdaptorVersion + + @type source :: :npm | :local + + @type package_meta :: %{ + name: String.t(), + latest_version: String.t(), + description: String.t() | nil, + deprecated: boolean(), + updated_at: DateTime.t(), + icon_square_ext: String.t() | nil, + icon_rectangle_ext: String.t() | nil, + icon_square_sha256: binary() | nil, + icon_rectangle_sha256: binary() | nil + } + + @version_row_fields ~w(adaptor_id version integrity tarball_url + size_bytes dependencies peer_dependencies + published_at deprecated)a + + @doc """ + Picker-facing lean projection for a source. Avoids the heavy JSONB + columns (`schema_data`, `dependencies`, `peer_dependencies`). + """ + @spec list_package_metas(source()) :: [package_meta()] + def list_package_metas(source) do + Lightning.Repo.all( + from a in Adaptor, + where: a.source == ^source, + select: %{ + name: a.name, + latest_version: a.latest_version, + description: a.description, + deprecated: a.deprecated, + updated_at: a.updated_at, + icon_square_ext: a.icon_square_ext, + icon_rectangle_ext: a.icon_rectangle_ext, + icon_square_sha256: a.icon_square_sha256, + icon_rectangle_sha256: a.icon_rectangle_sha256 + } + ) + end + + @doc """ + Full structs for a source. Rare — used by debug tools and admin + views. Picker traffic goes through `list_package_metas/1`. + """ + @spec list_adaptors(source()) :: [Adaptor.t()] + def list_adaptors(source) do + Lightning.Repo.all(from a in Adaptor, where: a.source == ^source) + end + + @doc """ + Fetch a single adaptor by `name` within a `source`. Returns `nil` + when no row matches. + """ + @spec get_adaptor(String.t(), source()) :: Adaptor.t() | nil + def get_adaptor(name, source) do + Lightning.Repo.get_by(Adaptor, name: name, source: source) + end + + @doc """ + All versions of an adaptor (`name`, `source`), in insertion order. + """ + @spec list_versions(String.t(), source()) :: [AdaptorVersion.t()] + def list_versions(name, source) do + Lightning.Repo.all( + from v in AdaptorVersion, + join: a in Adaptor, + on: v.adaptor_id == a.id, + where: a.name == ^name and a.source == ^source, + order_by: [asc: v.inserted_at] + ) + end + + @doc """ + Idempotent, transactional, diff-aware upsert of one adaptor record + plus its version rows. The `:source` is read from the record. + + Behaviour: + + * On every call, `checked_at` is advanced to "now". + * `updated_at` only advances when at least one non-`checked_at` + field of the adaptor row actually differs from the existing row. + * Version rows are replaced (delete + insert) inside the same + transaction. + * Every row is run through its schema changeset before write, so a + corrupt Strategy response cannot poison the DB. + + Raises if the underlying transaction fails (e.g. invalid input from + a misbehaving strategy) — the success type is the only contract the + Scheduler relies on. + """ + @spec upsert_adaptor(map()) :: {:ok, Adaptor.t()} + def upsert_adaptor(record) when is_map(record) do + now = DateTime.utc_now() + + {versions, adaptor_attrs} = + record + |> Map.put(:checked_at, now) + |> Map.pop(:versions, []) + + name = Map.fetch!(adaptor_attrs, :name) + source = Map.fetch!(adaptor_attrs, :source) + + multi = + Multi.new() + |> Multi.run(:existing, fn repo, _ -> + {:ok, repo.get_by(Adaptor, name: name, source: source)} + end) + |> Multi.run(:adaptor, fn repo, %{existing: existing} -> + upsert_adaptor_row(repo, existing, adaptor_attrs, now) + end) + |> Multi.run(:delete_versions, fn repo, %{adaptor: adaptor} -> + {count, _} = + repo.delete_all( + from v in AdaptorVersion, where: v.adaptor_id == ^adaptor.id + ) + + {:ok, count} + end) + |> Multi.run(:insert_versions, fn repo, %{adaptor: adaptor} -> + insert_version_rows(repo, adaptor.id, versions, now) + end) + + case Lightning.Repo.transaction(multi) do + {:ok, %{adaptor: adaptor}} -> + {:ok, adaptor} + + {:error, step, reason, _changes} -> + raise ArgumentError, + "Lightning.Adaptors.Repo.upsert_adaptor/1 failed at #{inspect(step)}: " <> + inspect(reason) + end + end + + @doc """ + Advance `checked_at` for a known `(name, source)` row without + loading it. No-op when no row matches. + + Used by the Scheduler's "polled NPM, nothing changed" path — + cheaper than a full upsert and never bumps `updated_at`. + """ + @spec touch_checked_at(String.t(), source()) :: :ok + def touch_checked_at(name, source) do + now = DateTime.utc_now() + + Lightning.Repo.update_all( + from(a in Adaptor, where: a.name == ^name and a.source == ^source), + set: [checked_at: now] + ) + + :ok + end + + @doc """ + The `limit` rows for a given `source` whose `checked_at` is oldest + first. Backs the Scheduler's per-tick work list. + """ + @spec stalest(pos_integer(), source()) :: [Adaptor.t()] + def stalest(limit, source) when is_integer(limit) and limit > 0 do + Lightning.Repo.all( + from a in Adaptor, + where: a.source == ^source, + order_by: [asc: a.checked_at], + limit: ^limit + ) + end + + @doc """ + Lean list of source-scoped adaptors that are missing at least one icon + shape. Returns only the fields the Scheduler needs to decide whether to + re-apply the bulk icon fetch result. + """ + @spec list_missing_icons(source()) :: [ + %{ + name: String.t(), + icon_square_sha256: binary() | nil, + icon_rectangle_sha256: binary() | nil + } + ] + def list_missing_icons(source) do + Lightning.Repo.all( + from a in Adaptor, + where: + a.source == ^source and + (is_nil(a.icon_square_sha256) or is_nil(a.icon_rectangle_sha256)), + select: %{ + name: a.name, + icon_square_sha256: a.icon_square_sha256, + icon_rectangle_sha256: a.icon_rectangle_sha256 + } + ) + end + + @doc """ + Update only the icon columns for a single `(name, source)` row. + + `attrs` may include any subset of `:icon_square_ext`, + `:icon_square_sha256`, `:icon_rectangle_ext`, `:icon_rectangle_sha256`, + `:icon_square_etag`, `:icon_rectangle_etag`. + `updated_at` is advanced so callers can observe the change. + + Sidesteps `upsert_adaptor/1` deliberately: that helper rewrites the + `adaptor_versions` rows in the same transaction, which is the wrong + thing to do for an icon-only fix-up. + """ + @spec update_icons(String.t(), source(), map()) :: {integer(), nil} + def update_icons(name, source, attrs) when is_map(attrs) do + allowed = + attrs + |> Map.take([ + :icon_square_ext, + :icon_square_sha256, + :icon_rectangle_ext, + :icon_rectangle_sha256, + :icon_square_etag, + :icon_rectangle_etag + ]) + |> Map.put(:updated_at, DateTime.utc_now()) + |> Enum.into([]) + + Lightning.Repo.update_all( + from(a in Adaptor, where: a.name == ^name and a.source == ^source), + set: allowed + ) + end + + @doc """ + Maximum `checked_at` seen for `source`, or `nil` when the table is + empty for that source. Backs the Scheduler's smart-init timing. + """ + @spec max_checked_at(source()) :: DateTime.t() | nil + def max_checked_at(source) do + Lightning.Repo.one( + from a in Adaptor, + where: a.source == ^source, + select: max(a.checked_at) + ) + end + + defp upsert_adaptor_row(repo, nil, attrs, _now) do + %Adaptor{} + |> Adaptor.changeset(attrs) + |> repo.insert() + end + + defp upsert_adaptor_row(repo, %Adaptor{} = existing, attrs, now) do + changeset = Adaptor.changeset(existing, attrs) + + # `Ecto.Changeset.cast/3` only records a change when the cast value + # differs from the underlying struct, so the set of "real" changes + # is `:changes` minus the `:checked_at` tick we apply on every call. + meaningful_changes? = + changeset.changes + |> Map.delete(:checked_at) + |> map_size() > 0 + + if meaningful_changes? do + repo.update(changeset) + else + {1, _} = + repo.update_all( + from(a in Adaptor, where: a.id == ^existing.id), + set: [checked_at: now] + ) + + {:ok, %{existing | checked_at: now}} + end + end + + defp insert_version_rows(_repo, _adaptor_id, [], _now), do: {:ok, 0} + + defp insert_version_rows(repo, adaptor_id, records, now) do + case build_version_rows(adaptor_id, records, now) do + {:ok, rows} -> + {count, _} = repo.insert_all(AdaptorVersion, rows) + {:ok, count} + + {:error, changeset} -> + {:error, changeset} + end + end + + defp build_version_rows(adaptor_id, records, now) do + records + |> Enum.reduce_while({:ok, []}, fn record, {:ok, acc} -> + attrs = Map.put(record, :adaptor_id, adaptor_id) + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, attrs) + + if changeset.valid? do + {:cont, {:ok, [version_row_from_changeset(changeset, now) | acc]}} + else + {:halt, {:error, changeset}} + end + end) + |> case do + {:ok, rows} -> {:ok, Enum.reverse(rows)} + err -> err + end + end + + defp version_row_from_changeset(changeset, now) do + changeset + |> Ecto.Changeset.apply_changes() + |> Map.from_struct() + |> Map.take(@version_row_fields) + |> Map.put(:id, Ecto.UUID.generate()) + |> Map.put(:inserted_at, now) + end +end diff --git a/lib/lightning/adaptors/repo_adaptor.ex b/lib/lightning/adaptors/repo_adaptor.ex new file mode 100644 index 00000000000..469620df5fe --- /dev/null +++ b/lib/lightning/adaptors/repo_adaptor.ex @@ -0,0 +1,163 @@ +defmodule Lightning.Adaptors.Repo.Adaptor do + @moduledoc """ + Ecto schema for one row of the `adaptors` table — the per-package + metadata projection used by the picker and Scheduler. + + Source-tagged via `:source` (`:npm | :local`) so the same package + name can coexist across sources; the unique index is `[:name, :source]` + (see §4.4 source-tagging invariant in + `.context/lightning/adaptors/REWRITE-2026-05.md`). + + Mirrors `Lightning.Adaptors.Strategy.adaptor_record` minus `:versions`, + which lives on `Lightning.Adaptors.Repo.AdaptorVersion`. + """ + + use Ecto.Schema + + import Ecto.Changeset + + defmodule JSONBinary do + @moduledoc """ + Ecto type backing `schema_data` with a text column while preserving + JSON field order on read. + + Storage is a JSON binary in a `text` column. Inputs may be either a + binary or a map — maps are encoded with `Jason.encode!/1` at the + dumper to keep `Lightning.Factories.adaptor/2` and other direct + struct inserts compatible without forcing every caller to encode + up-front. Loads always return a binary so credential-form rendering + can re-engage `Jason.decode!(_, objects: :ordered_objects)`. + """ + + use Ecto.Type + + @impl true + def type, do: :string + + @impl true + def cast(nil), do: {:ok, nil} + def cast(value) when is_binary(value), do: {:ok, value} + def cast(value) when is_map(value), do: {:ok, Jason.encode!(value)} + def cast(_), do: :error + + @impl true + def load(nil), do: {:ok, nil} + def load(value) when is_binary(value), do: {:ok, value} + + @impl true + def dump(nil), do: {:ok, nil} + def dump(value) when is_binary(value), do: {:ok, value} + def dump(value) when is_map(value), do: {:ok, Jason.encode!(value)} + def dump(_), do: :error + end + + @type t :: %__MODULE__{ + id: Ecto.UUID.t() | nil, + name: String.t() | nil, + source: :npm | :local | nil, + description: String.t() | nil, + homepage: String.t() | nil, + repository: String.t() | nil, + license: String.t() | nil, + latest_version: String.t() | nil, + deprecated: boolean(), + schema_data: String.t() | nil, + schema_sha256: String.t() | nil, + icon_square_ext: String.t() | nil, + icon_rectangle_ext: String.t() | nil, + icon_square_sha256: binary() | nil, + icon_rectangle_sha256: binary() | nil, + icon_square_etag: String.t() | nil, + icon_rectangle_etag: String.t() | nil, + checked_at: DateTime.t() | nil, + inserted_at: DateTime.t() | nil, + updated_at: DateTime.t() | nil + } + + @primary_key {:id, :binary_id, autogenerate: true} + @foreign_key_type :binary_id + @timestamps_opts [type: :utc_datetime_usec] + + schema "adaptors" do + field :name, :string + field :source, Ecto.Enum, values: [:npm, :local] + field :description, :string + field :homepage, :string + field :repository, :string + field :license, :string + field :latest_version, :string + field :deprecated, :boolean, default: false + field :schema_data, Lightning.Adaptors.Repo.Adaptor.JSONBinary + field :schema_sha256, :string + field :icon_square_ext, :string + field :icon_rectangle_ext, :string + field :icon_square_sha256, :binary + field :icon_rectangle_sha256, :binary + field :icon_square_etag, :string + field :icon_rectangle_etag, :string + field :checked_at, :utc_datetime_usec + + timestamps() + end + + @required ~w(name source latest_version checked_at)a + @optional ~w(description homepage repository license deprecated + schema_data schema_sha256 + icon_square_ext icon_rectangle_ext + icon_square_sha256 icon_rectangle_sha256 + icon_square_etag icon_rectangle_etag)a + + @doc """ + Build a changeset for upserting a single adaptor row. + + This is the single clause used by every write path on + `Lightning.Adaptors.Repo` — there is no separate update path because + the writer always rewrites the full row. + """ + @spec changeset(t(), map()) :: Ecto.Changeset.t() + def changeset(struct, attrs) do + struct + |> cast(attrs, @required ++ @optional) + |> validate_required(@required) + |> validate_length(:name, max: 214) + |> validate_inclusion(:icon_square_ext, ~w(png svg)) + |> validate_inclusion(:icon_rectangle_ext, ~w(png svg)) + |> validate_icon_sha256_pair(:icon_square) + |> validate_icon_sha256_pair(:icon_rectangle) + |> unique_constraint([:name, :source]) + end + + # Enforces the §6.4 invariant: a non-nil `icon__ext` requires a + # non-nil `icon__sha256`, and vice versa. Either both fields + # are set or both are nil — half-populated pairs fail the changeset. + @spec validate_icon_sha256_pair( + Ecto.Changeset.t(), + :icon_square | :icon_rectangle + ) :: Ecto.Changeset.t() + defp validate_icon_sha256_pair(changeset, shape) do + ext_field = :"#{shape}_ext" + sha_field = :"#{shape}_sha256" + + case {get_field(changeset, ext_field), get_field(changeset, sha_field)} do + {nil, nil} -> + changeset + + {nil, _sha} -> + add_error( + changeset, + sha_field, + "must be nil when #{ext_field} is nil" + ) + + {_ext, nil} -> + add_error( + changeset, + sha_field, + "must not be nil when #{ext_field} is set" + ) + + {_ext, _sha} -> + changeset + end + end +end diff --git a/lib/lightning/adaptors/repo_adaptor_version.ex b/lib/lightning/adaptors/repo_adaptor_version.ex new file mode 100644 index 00000000000..ec048b81806 --- /dev/null +++ b/lib/lightning/adaptors/repo_adaptor_version.ex @@ -0,0 +1,73 @@ +defmodule Lightning.Adaptors.Repo.AdaptorVersion do + @moduledoc """ + Ecto schema for one row of the `adaptor_versions` table — per-version + metadata for an adaptor package (`integrity`, `tarball_url`, + `size_bytes`, `dependencies`, `peer_dependencies`, `published_at`, + `deprecated`). + + Belongs to `Lightning.Adaptors.Repo.Adaptor` and cascade-deletes with + its parent. Mirrors `Lightning.Adaptors.Strategy.version_record` (see + §6.1 and §6.4 in `.context/lightning/adaptors/REWRITE-2026-05.md`). + """ + + use Ecto.Schema + + import Ecto.Changeset + + alias Lightning.Adaptors.Repo.Adaptor + + @type t :: %__MODULE__{ + id: Ecto.UUID.t() | nil, + adaptor_id: Ecto.UUID.t() | nil, + adaptor: Adaptor.t() | Ecto.Association.NotLoaded.t() | nil, + version: String.t() | nil, + integrity: String.t() | nil, + tarball_url: String.t() | nil, + size_bytes: integer() | nil, + dependencies: map() | nil, + peer_dependencies: map() | nil, + published_at: DateTime.t() | nil, + deprecated: boolean(), + inserted_at: DateTime.t() | nil + } + + @primary_key {:id, :binary_id, autogenerate: true} + @foreign_key_type :binary_id + @timestamps_opts [type: :utc_datetime_usec] + + schema "adaptor_versions" do + field :version, :string + field :integrity, :string + field :tarball_url, :string + field :size_bytes, :integer + field :dependencies, :map + field :peer_dependencies, :map + field :published_at, :utc_datetime_usec + field :deprecated, :boolean, default: false + + belongs_to :adaptor, Adaptor + + timestamps(updated_at: false) + end + + @required ~w(adaptor_id version)a + @optional ~w(integrity tarball_url size_bytes + dependencies peer_dependencies + published_at deprecated)a + + @doc """ + Build a changeset for inserting an `adaptor_versions` row. + + `Lightning.Adaptors.Repo.upsert_adaptor/1` replaces version rows with + a delete-then-insert inside a transaction, so there is no separate + update path. + """ + @spec changeset(t(), map()) :: Ecto.Changeset.t() + def changeset(struct, attrs) do + struct + |> cast(attrs, @required ++ @optional) + |> validate_required(@required) + |> unique_constraint([:adaptor_id, :version]) + |> assoc_constraint(:adaptor) + end +end diff --git a/lib/lightning/adaptors/scheduler.ex b/lib/lightning/adaptors/scheduler.ex new file mode 100644 index 00000000000..6077fc48d2a --- /dev/null +++ b/lib/lightning/adaptors/scheduler.ex @@ -0,0 +1,597 @@ +defmodule Lightning.Adaptors.Scheduler do + @moduledoc """ + Cluster-singleton GenServer that periodically refreshes the active + source's ledger via the configured strategy, persists through + `Lightning.Adaptors.Repo`, and broadcasts `{:changed, name, source}`. + + Wrapped by `HighlanderPG` so only one node in the cluster runs the + Scheduler at a time. The inner GenServer registers under + `{:global, Lightning.Adaptors.Supervisor.global_scheduler_name(name)}`, + so callers on any node reach the leader transparently via Erlang + distribution. Peer nodes react to refreshes via + `Lightning.Adaptors.Invalidator` and + `Lightning.Adaptors.ChannelBroadcaster`. + + Smart-init timing: the first tick is scheduled at + `max(0, last_checked_at + interval - now)` to avoid double-refreshing + shortly after a deploy. An empty table or an overdue schedule fires + immediately (`delay = 0`). Interval `0` disables scheduling entirely. + + ## Two-pipeline refresh + + A tick runs two parallel pipelines under the per-instance + `Task.Supervisor`: + + * **Pipeline A** — `strategy.fetch_icons/1` for every adaptor. + * **Pipeline B** — `strategy.list_adaptors/0` followed by a bounded + per-adaptor fan-out (`async_stream_nolink`) calling + `strategy.fetch_adaptor/1` only for names whose `latest_version` + changed since the last tick. + + Once both pipelines complete the join step merges the icons map into + each fetched record, writes the icon bytes to disk via + `Lightning.Adaptors.IconCache.write!/5`, and upserts each adaptor in + one go. `refresh_package/2` deliberately bypasses the icon pipeline — + on-demand single-package refreshes do not refetch icons. + """ + + use GenServer + + alias Lightning.Adaptors.Config + alias Lightning.Adaptors.IconCache + alias Lightning.Adaptors.Repo, as: AdaptorsRepo + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + require Logger + + @fetch_max_concurrency 8 + @icons_task_timeout :timer.seconds(60) + + @doc """ + Start the Scheduler for the given supervisor instance. + + Required opts: `:name`, `:sup`, `:lock_key`, `:cache`, `:tasks`, + `:source_topic`. + """ + @spec start_link(keyword()) :: GenServer.on_start() + def start_link(opts) do + name = Keyword.fetch!(opts, :name) + _ = Keyword.fetch!(opts, :sup) + _ = Keyword.fetch!(opts, :lock_key) + _ = Keyword.fetch!(opts, :cache) + _ = Keyword.fetch!(opts, :tasks) + _ = Keyword.fetch!(opts, :source_topic) + GenServer.start_link(__MODULE__, opts, name: name) + end + + @doc """ + Trigger an immediate refresh tick. + + Routes via `:global` to the leader-held GenServer. + """ + @spec refresh_now(GenServer.server()) :: :ok | {:error, term()} + def refresh_now(scheduler_name) do + GenServer.call(scheduler_name, :refresh_now) + end + + @doc """ + Force a single-adaptor refresh, bypassing the diff. 30-second timeout. + + Returns `{:error, :not_found}` or `{:error, term()}` from a failed + strategy fetch. + """ + @spec refresh_package(GenServer.server(), String.t()) :: + :ok | {:error, :not_found | term()} + def refresh_package(scheduler_name, name) do + GenServer.call(scheduler_name, {:refresh_package, name}, 30_000) + end + + @doc """ + Refresh icons only, against every source-scoped adaptor row. + + Runs `strategy.fetch_icons/1` and re-applies any shape whose `sha256` + differs from what is on the row. Adaptor metadata and version rows + are not touched. Returns `{:ok, %{updated: n, unchanged: m}}` on + success or `{:error, reason}` if the bulk fetch fails. + """ + @spec refresh_icons(GenServer.server()) :: + {:ok, %{updated: non_neg_integer(), unchanged: non_neg_integer()}} + | {:error, term()} + def refresh_icons(scheduler_name) do + GenServer.call(scheduler_name, :refresh_icons, 120_000) + end + + @impl true + def init(opts) do + sup = Keyword.fetch!(opts, :sup) + source_topic = Keyword.fetch!(opts, :source_topic) + cache = Keyword.fetch!(opts, :cache) + tasks = Keyword.fetch!(opts, :tasks) + + source = AdaptorsSupervisor.source(sup) + interval_ms = Config.refresh_interval() + + if interval_ms > 0 do + delay = + time_until_next_ms(AdaptorsRepo.max_checked_at(source), interval_ms) + + Process.send_after(self(), :tick, delay) + + Logger.info( + "Adaptors[#{source}]: scheduler started interval=#{interval_ms}ms next_tick_in=#{delay}ms" + ) + else + Logger.info("Adaptors[#{source}]: scheduler started interval=0 (disabled)") + end + + {:ok, + %{ + sup: sup, + source: source, + interval_ms: interval_ms, + source_topic: source_topic, + cache: cache, + tasks: tasks + }} + end + + @impl true + def handle_info(:tick, state) do + if state.interval_ms > 0 do + Process.send_after(self(), :tick, state.interval_ms) + end + + Task.Supervisor.start_child(state.tasks, fn -> do_refresh(state) end) + + {:noreply, state} + end + + @impl true + def handle_call(:refresh_now, _from, state) do + Logger.info("Adaptors[#{state.source}]: refresh_now requested") + send(self(), :tick) + {:reply, :ok, state} + end + + def handle_call({:refresh_package, name}, _from, state) do + Logger.info("Adaptors[#{state.source}]: refresh_package(#{name}) requested") + + strategy = AdaptorsSupervisor.strategy(state.sup) + result = force_refresh_one(strategy, name, state) + {:reply, result, state} + end + + def handle_call(:refresh_icons, _from, state) do + Logger.info("Adaptors[#{state.source}]: refresh_icons requested") + strategy = AdaptorsSupervisor.strategy(state.sup) + + existing = AdaptorsRepo.list_adaptors(state.source) + prior_etags = prior_etags_from_rows(existing) + + case strategy.fetch_icons(prior_etags: prior_etags) do + {:ok, icons} -> + result = reapply_icons(existing, icons, state) + + Logger.info( + "Adaptors[#{state.source}]: refresh_icons done " <> + "rows=#{length(existing)} icons=#{map_size(icons)} " <> + "updated=#{result.updated} unchanged=#{result.unchanged}" + ) + + {:reply, {:ok, result}, state} + + {:error, reason} -> + Logger.warning( + "Adaptors[#{state.source}]: refresh_icons strategy fetch failed: #{inspect(reason)}" + ) + + {:reply, {:error, reason}, state} + end + end + + defp do_refresh(state) do + started_at = System.monotonic_time(:millisecond) + strategy = AdaptorsSupervisor.strategy(state.sup) + + # Single DB round-trip serves both the icons-task input (prior etags) + # and the version diff used below to decide which adaptors to fetch. + existing_rows = AdaptorsRepo.list_adaptors(state.source) + prior_etags = prior_etags_from_rows(existing_rows) + + existing_by_name = + Map.new(existing_rows, fn a -> {a.name, a.latest_version} end) + + icons_task = + Task.Supervisor.async_nolink(state.tasks, fn -> + strategy.fetch_icons(prior_etags: prior_etags) + end) + + case strategy.list_adaptors() do + {:ok, upstream} -> + {fetched, changed, errors} = + state.tasks + |> Task.Supervisor.async_stream_nolink( + upstream, + &fetch_if_changed(strategy, &1, existing_by_name, state), + max_concurrency: @fetch_max_concurrency, + ordered: false, + on_timeout: :kill_task + ) + |> Enum.reduce({[], 0, 0}, fn + {:ok, {:fetched, record}}, {acc, c, e} -> {[record | acc], c + 1, e} + {:ok, :touched}, {acc, c, e} -> {acc, c, e} + {:ok, {:error, _reason}}, {acc, c, e} -> {acc, c, e + 1} + {:exit, _reason}, {acc, c, e} -> {acc, c, e + 1} + end) + + icons = await_icons(icons_task) + + persisted = + fetched + |> Enum.map(fn record -> persist_with_icons(record, icons, state) end) + |> Enum.count(&(&1 == :ok)) + + healed = heal_missing_icons(icons, state) + not_modified = count_not_modified(icons) + + listed = length(upstream) + touched = listed - changed - errors + duration_ms = System.monotonic_time(:millisecond) - started_at + + Logger.info( + "Adaptors[#{state.source}]: refresh tick listed=#{listed} " <> + "changed=#{changed} touched=#{touched} fetched=#{persisted} " <> + "icons=#{map_size(icons)} healed=#{healed} " <> + "not_modified=#{not_modified} " <> + "errors=#{errors} duration=#{duration_ms}ms" + ) + + {:error, reason} -> + Logger.warning("Scheduler: list_adaptors failed: #{inspect(reason)}") + _ = await_icons(icons_task) + duration_ms = System.monotonic_time(:millisecond) - started_at + + Logger.info( + "Adaptors[#{state.source}]: refresh tick listed=0 changed=0 " <> + "touched=0 fetched=0 icons=0 errors=1 duration=#{duration_ms}ms" + ) + + :ok + end + end + + defp fetch_if_changed( + strategy, + %{name: name, latest_version: version}, + existing_by_name, + state + ) do + if Map.get(existing_by_name, name) == version do + AdaptorsRepo.touch_checked_at(name, state.source) + :touched + else + case strategy.fetch_adaptor(name) do + {:ok, record} -> + Logger.debug( + "Adaptors[#{state.source}]: fetched #{name}@#{record.version}" + ) + + {:fetched, record} + + {:error, reason} -> + Logger.warning( + "Scheduler: fetch_adaptor(#{name}) failed: #{inspect(reason)}" + ) + + {:error, reason} + end + end + end + + defp await_icons(task) do + case Task.yield(task, @icons_task_timeout) || Task.shutdown(task) do + {:ok, {:ok, map}} when is_map(map) -> + map + + {:ok, {:error, reason}} -> + Logger.warning( + "Scheduler: fetch_icons failed: #{inspect(reason)} — persisting records without icons" + ) + + %{} + + {:exit, reason} -> + Logger.warning( + "Scheduler: fetch_icons crashed: #{inspect(reason)} — persisting records without icons" + ) + + %{} + + nil -> + Logger.warning( + "Scheduler: fetch_icons timed out — persisting records without icons" + ) + + %{} + end + end + + defp persist_with_icons(record, icons, state) do + name = record.name + package_icons = Map.get(icons, name, %{}) + + record_with_icons = + record + |> Map.put(:source, state.source) + |> merge_icon(:square, package_icons, state.source) + |> merge_icon(:rectangle, package_icons, state.source) + + try do + {:ok, _} = AdaptorsRepo.upsert_adaptor(record_with_icons) + + Phoenix.PubSub.broadcast( + Lightning.PubSub, + state.source_topic, + {:changed, name, state.source} + ) + + Logger.debug("Adaptors[#{state.source}]: persisted #{name}") + :ok + rescue + e -> + Logger.error( + "Scheduler: upsert_adaptor(#{name}) failed: #{Exception.message(e)}" + ) + + :error + end + end + + defp merge_icon(record, shape, package_icons, source) do + case Map.get(package_icons, shape) do + %{data: bytes, ext: ext, sha256: sha} = entry when is_binary(bytes) -> + try do + {:ok, ^sha} = IconCache.write!(source, record.name, shape, ext, bytes) + + record + |> Map.put(:"icon_#{shape}_ext", ext) + |> Map.put(:"icon_#{shape}_sha256", sha) + |> maybe_put_etag(shape, Map.get(entry, :etag)) + rescue + e -> + Logger.warning( + "Scheduler: IconCache.write!(#{record.name}, #{shape}) failed: #{Exception.message(e)}" + ) + + record + end + + :not_modified -> + # Upstream confirmed unchanged — leave row's existing icon and + # etag in place. Counted in the tick summary via + # count_not_modified/1. + record + + _ -> + record + end + end + + # Stamp the etag onto the record only when the strategy supplied one + # (NPM 200 entries always have the key; Local omits it). A nil etag is + # not stamped — we preserve whatever was already on the row. + defp maybe_put_etag(record, _shape, nil), do: record + + defp maybe_put_etag(record, shape, etag) when is_binary(etag) do + Map.put(record, :"icon_#{shape}_etag", etag) + end + + # Top up icons on rows that currently have NULL on at least one shape. + # Runs after the main upsert pass on every tick — cheap, scoped to + # rows with gaps, and self-correcting after a strategy outage or a + # past bug like the one that left every row iconless. + defp heal_missing_icons(icons, _state) when map_size(icons) == 0, do: 0 + + defp heal_missing_icons(icons, state) do + state.source + |> AdaptorsRepo.list_missing_icons() + |> Enum.reduce(0, fn row, acc -> + package_icons = Map.get(icons, row.name, %{}) + + case apply_icons_to_existing(row, package_icons, state) do + :updated -> acc + 1 + :unchanged -> acc + end + end) + end + + defp reapply_icons(existing_rows, icons, state) do + Enum.reduce(existing_rows, %{updated: 0, unchanged: 0}, fn row, acc -> + package_icons = Map.get(icons, row.name, %{}) + + case apply_icons_to_existing(row, package_icons, state) do + :updated -> %{acc | updated: acc.updated + 1} + :unchanged -> %{acc | unchanged: acc.unchanged + 1} + end + end) + end + + # `row` is either an Adaptor struct (from list_adaptors/1) or a lean + # map (from list_missing_icons/1) — both expose :name and the icon + # sha256 fields, which is all we need. + defp apply_icons_to_existing(_row, package_icons, _state) + when map_size(package_icons) == 0, + do: :unchanged + + defp apply_icons_to_existing(row, package_icons, state) do + changes = + [:square, :rectangle] + |> Enum.reduce(%{}, fn shape, acc -> + accumulate_icon_change(acc, shape, row, package_icons, state) + end) + + if map_size(changes) > 0 do + {1, _} = AdaptorsRepo.update_icons(row.name, state.source, changes) + + Phoenix.PubSub.broadcast( + Lightning.PubSub, + state.source_topic, + {:changed, row.name, state.source} + ) + + :updated + else + :unchanged + end + end + + defp accumulate_icon_change(acc, shape, row, package_icons, state) do + sha_key = :"icon_#{shape}_sha256" + ext_key = :"icon_#{shape}_ext" + etag_key = :"icon_#{shape}_etag" + + case Map.get(package_icons, shape) do + %{data: bytes, ext: ext, sha256: sha} = entry when is_binary(bytes) -> + if Map.get(row, sha_key) == sha do + # Same bytes already on disk; the etag may still need + # refreshing if the strategy gave us a new (non-nil) value + # that differs from what we have. nil never clobbers. + maybe_accumulate_etag(acc, etag_key, row, Map.get(entry, :etag)) + else + accumulate_fetched_icon(acc, shape, row, entry, ext, sha, bytes, state, + sha_key: sha_key, + ext_key: ext_key, + etag_key: etag_key + ) + end + + :not_modified -> + # 304 confirmed — nothing to write, etag already current. + acc + + _ -> + acc + end + end + + defp accumulate_fetched_icon(acc, shape, row, entry, ext, sha, bytes, state, + sha_key: sha_key, + ext_key: ext_key, + etag_key: etag_key + ) do + try do + {:ok, ^sha} = IconCache.write!(state.source, row.name, shape, ext, bytes) + + acc + |> Map.put(ext_key, ext) + |> Map.put(sha_key, sha) + |> maybe_accumulate_etag(etag_key, row, Map.get(entry, :etag)) + rescue + e -> + Logger.warning( + "Scheduler: IconCache.write!(#{row.name}, #{shape}) failed: " <> + Exception.message(e) + ) + + acc + end + end + + # nil → preserve existing etag on the row (do not clobber). + # value matching the row's current etag → no-op (avoid no-op write). + # value differing → emit the change. + defp maybe_accumulate_etag(acc, _etag_key, _row, nil), do: acc + + defp maybe_accumulate_etag(acc, etag_key, row, etag) when is_binary(etag) do + if Map.get(row, etag_key) == etag do + acc + else + Map.put(acc, etag_key, etag) + end + end + + defp force_refresh_one(strategy, name, state) do + case strategy.fetch_adaptor(name) do + {:ok, record} -> + record_with_source = Map.put(record, :source, state.source) + + try do + {:ok, _} = AdaptorsRepo.upsert_adaptor(record_with_source) + + Phoenix.PubSub.broadcast( + Lightning.PubSub, + state.source_topic, + {:changed, name, state.source} + ) + + Logger.info( + "Adaptors[#{state.source}]: refresh_package(#{name}) ok version=#{record.version}" + ) + + :ok + rescue + e -> + Logger.error( + "Scheduler: upsert_adaptor(#{name}) failed: #{Exception.message(e)}" + ) + + {:error, {:upsert_failed, Exception.message(e)}} + end + + {:error, reason} -> + Logger.warning( + "Scheduler: refresh_package(#{name}) strategy fetch failed: #{inspect(reason)}" + ) + + {:error, reason} + end + end + + # Project a list of adaptor rows to the prior-etag map shape expected + # by `Strategy.fetch_icons/1`: `%{name => %{shape => etag}}`. Rows + # whose etags are both nil are skipped entirely (no empty inner map); + # within a row, only shapes with a non-nil etag are kept. The consumer + # treats absence as "no prior etag, send no If-None-Match", so an empty + # entry would be wasteful but harmless — we drop it for clarity. + @spec prior_etags_from_rows([map()]) :: %{ + String.t() => %{optional(:square | :rectangle) => String.t()} + } + defp prior_etags_from_rows(rows) do + Enum.reduce(rows, %{}, fn row, acc -> + inner = + %{} + |> maybe_put_shape_etag(:square, Map.get(row, :icon_square_etag)) + |> maybe_put_shape_etag(:rectangle, Map.get(row, :icon_rectangle_etag)) + + if map_size(inner) == 0 do + acc + else + Map.put(acc, row.name, inner) + end + end) + end + + defp maybe_put_shape_etag(map, _shape, nil), do: map + + defp maybe_put_shape_etag(map, shape, etag) when is_binary(etag), + do: Map.put(map, shape, etag) + + # Count :not_modified sentinels across all shapes in the icons map. + # Used in the tick summary log. + defp count_not_modified(icons) do + Enum.reduce(icons, 0, fn {_name, shapes}, acc -> + Enum.reduce(shapes, acc, fn + {_shape, :not_modified}, n -> n + 1 + {_shape, _}, n -> n + end) + end) + end + + defp time_until_next_ms(nil, _interval_ms), do: 0 + + defp time_until_next_ms(%DateTime{} = last, interval_ms) do + next = DateTime.add(last, interval_ms, :millisecond) + diff = DateTime.diff(next, DateTime.utc_now(), :millisecond) + max(0, diff) + end +end diff --git a/lib/lightning/adaptors/store.ex b/lib/lightning/adaptors/store.ex new file mode 100644 index 00000000000..a1d23c781d9 --- /dev/null +++ b/lib/lightning/adaptors/store.ex @@ -0,0 +1,347 @@ +defmodule Lightning.Adaptors.Store do + @moduledoc """ + Stateless read facade over `Cachex`, `Lightning.Adaptors.Repo`, and the + active `Lightning.Adaptors.Strategy`. + + Every public read helper wraps a `Cachex.fetch/4` whose fallback first + consults the local Postgres projection (`Lightning.Adaptors.Repo`) and + only invokes the Strategy as a last resort. Cachex's courier supplies + blocking semantics and per-key coalescing of concurrent first-callers + for free — there is no GenServer mailbox in front of the reads. + + ## Source tagging + + Each cache key carries the active `:source` (`:npm | :local`) read via + `Lightning.Adaptors.Supervisor.source/1`, so the same package name can + coexist across deployment modes without manual scrubbing (see §4.4 of + `.context/adaptors/REWRITE-2026-05.md`). + + ## Commit vs ignore + + Successful Strategy/Repo lookups commit their projected value to the + cache. Failures — empty `packages/1` results, unknown adaptors for + `icon_meta/2`, Strategy errors — return `:ignore`, so a subsequent + caller retries fresh rather than seeing a poisoned cache entry. + + ## Icons + + `icon/3` returns a `Path.t/0` the controller serves via `send_file/3` + — no binary on the BEAM heap. The on-disk `Lightning.Adaptors.IconCache` + is the primary cache: a `cached?/4` hit short-circuits before Cachex + is touched. On a disk miss the lazy Strategy fetch is wrapped in + `Cachex.fetch/4` on `{:icon_bytes, source, name, shape}` so that + concurrent first-callers coalesce onto a single courier; the courier + returns `{:ignore, _}` so no entry is committed, and subsequent + callers re-read the now-populated file from disk. + """ + + alias Lightning.Adaptors.Config + alias Lightning.Adaptors.IconCache + alias Lightning.Adaptors.Repo, as: AdaptorsRepo + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + # Strategy and source are scoped to the supervisor instance — every + # `Store` call resolves both from the per-instance `:persistent_term` + # entry the supervisor populated at boot. No `Application.get_env` + # reads in the hot path; no global mutable state in tests. + + @type sup :: atom() + + @type version_meta :: %{ + version: String.t(), + integrity: String.t() | nil, + size_bytes: integer() | nil, + published_at: DateTime.t() | nil, + deprecated: boolean() + } + + @type icon_meta :: %{ + icon_square_ext: String.t() | nil, + icon_rectangle_ext: String.t() | nil, + icon_square_sha256: binary() | nil, + icon_rectangle_sha256: binary() | nil + } + + @type package_meta :: AdaptorsRepo.package_meta() + + @doc """ + Read the `schema_data` JSON blob for a single adaptor. + + Cache-then-Repo-then-Strategy. On Strategy success the full adaptor + record is upserted into Postgres and the projected schema blob is + committed to the cache. + + Returns the schema as a JSON binary so that ordered-objects decoding + can re-engage at the credential-form renderer. + """ + @spec schema(sup(), String.t()) :: {:ok, String.t()} | {:error, term()} + def schema(sup, name) do + cache = AdaptorsSupervisor.cache_name(sup) + source = AdaptorsSupervisor.source(sup) + + cache + |> Cachex.fetch( + {:schema, name, source}, + fn _key -> + case AdaptorsRepo.get_adaptor(name, source) do + %{schema_data: data} when not is_nil(data) -> + {:commit, {:ok, data}} + + _ -> + fetch_and_persist(sup, name, source, :schema_data) + end + end, + timeout: Config.cache_timeout_ms() + ) + |> unwrap() + end + + @doc """ + Read the version history for a single adaptor as a list of lean + per-version maps. See `t:version_meta/0` for the projected shape. + """ + @spec versions(sup(), String.t()) :: + {:ok, [version_meta()]} | {:error, term()} + def versions(sup, name) do + cache = AdaptorsSupervisor.cache_name(sup) + source = AdaptorsSupervisor.source(sup) + + cache + |> Cachex.fetch( + {:versions, name, source}, + fn _key -> + case AdaptorsRepo.list_versions(name, source) do + [] -> fetch_and_persist(sup, name, source, :versions) + rows -> {:commit, {:ok, project_versions(rows)}} + end + end, + timeout: Config.cache_timeout_ms() + ) + |> unwrap() + end + + @doc """ + Resolve the on-disk path of one icon variant for an adaptor. + + Disk is the cache: a cache-hit on `IconCache.cached?/4` returns the + path immediately. A cache-miss is routed through `Cachex.fetch/4` on + `{:icon_bytes, source, name, shape}` so concurrent first-callers + coalesce onto one in-flight Strategy fetch — the courier returns + `{:ignore, _}` so no cache entry is committed and the next miss reads + the freshly-written file from disk. + + Returns `{:error, :not_found}` when the icon variant is absent from + the adaptor row (the row is the source of truth). + """ + @spec icon(sup(), String.t(), :square | :rectangle) :: + {:ok, Path.t()} | {:error, :not_found | term()} + def icon(sup, name, shape) when shape in [:square, :rectangle] do + cache = AdaptorsSupervisor.cache_name(sup) + source = AdaptorsSupervisor.source(sup) + strategy = AdaptorsSupervisor.strategy(sup) + + with {:ok, meta} <- icon_meta(sup, name), + {:ok, ext} <- ext_for_shape(meta, shape), + {:ok, _sha256} <- sha256_for_shape(meta, shape) do + if IconCache.cached?(source, name, shape, ext) do + {:ok, IconCache.path(source, name, shape, ext)} + else + cache + |> Cachex.fetch( + {:icon_bytes, source, name, shape}, + fn _key -> fetch_icon_bytes(strategy, source, name, shape, ext) end, + timeout: Config.cache_timeout_ms() + ) + |> unwrap() + end + end + end + + defp fetch_icon_bytes(strategy, source, name, shape, ext) do + case strategy.fetch_icon(name, shape) do + {:ok, %{data: bytes, ext: ^ext}} -> + {:ok, _sha} = IconCache.write!(source, name, shape, ext, bytes) + {:ignore, {:ok, IconCache.path(source, name, shape, ext)}} + + {:ok, %{ext: other_ext}} -> + {:ignore, {:error, {:ext_mismatch, expected: ext, got: other_ext}}} + + {:error, _} = err -> + {:ignore, err} + end + end + + @doc """ + Picker-facing lean projection: every adaptor row for the active + source, minus heavy JSONB columns (`schema_data`, `dependencies`, + `peer_dependencies`). + + An empty Repo result returns `{:ok, []}` but is **not** committed to + the cache — during cold-start the Scheduler will fill the table on + its next tick, and the next call will pick that up automatically. + """ + @spec packages(sup()) :: {:ok, [package_meta()]} | {:error, term()} + def packages(sup) do + cache = AdaptorsSupervisor.cache_name(sup) + source = AdaptorsSupervisor.source(sup) + + cache + |> Cachex.fetch( + {:packages, source}, + fn _key -> + case AdaptorsRepo.list_package_metas(source) do + [] -> {:ignore, {:ok, []}} + metas -> {:commit, {:ok, metas}} + end + end, + timeout: Config.cache_timeout_ms() + ) + |> unwrap() + end + + @doc """ + Cheap `{icon__ext, icon__sha256}` projection for the + icon controller's sha-validation path. Pure metadata — no disk I/O. + + Unknown adaptors return `{:error, :not_found}` and are **not** + cached, so a subsequent insert by the Scheduler becomes visible on + the very next call. + """ + @spec icon_meta(sup(), String.t()) :: + {:ok, icon_meta()} | {:error, :not_found} + def icon_meta(sup, name) do + cache = AdaptorsSupervisor.cache_name(sup) + source = AdaptorsSupervisor.source(sup) + + cache + |> Cachex.fetch( + {:icon_meta, name, source}, + fn _key -> + case AdaptorsRepo.get_adaptor(name, source) do + nil -> {:ignore, {:error, :not_found}} + adaptor -> {:commit, {:ok, project_icon_meta(adaptor)}} + end + end, + timeout: Config.cache_timeout_ms() + ) + |> unwrap() + end + + @doc """ + Re-warm Cachex from Postgres for the active source. + + Called by `Lightning.Adaptors.NodeMonitor` on `:nodeup` — a peer + rejoining after a partition can't know which `{:changed, name, source}` + broadcasts it missed, so it treats its entire local Cachex as + suspect and overwrites from the DB. + + Uses `Cachex.put_many/2` (never `Cachex.clear/1`-then-fill) so + concurrent callers never observe an empty cache and never trigger a + spurious cold-miss Strategy fetch during the warm. + """ + @spec warm_from_repo(sup()) :: :ok + def warm_from_repo(sup) do + cache = AdaptorsSupervisor.cache_name(sup) + source = AdaptorsSupervisor.source(sup) + + metas = AdaptorsRepo.list_package_metas(source) + + icon_metas = + Enum.map(metas, fn m -> + {{:icon_meta, m.name, source}, {:ok, project_icon_meta(m)}} + end) + + Cachex.put_many( + cache, + [{{:packages, source}, {:ok, metas}} | icon_metas] + ) + + :ok + end + + @spec fetch_and_persist(atom(), String.t(), :npm | :local, atom()) :: + {:commit, {:ok, term()}} | {:ignore, {:error, term()}} + defp fetch_and_persist(sup, name, source, field) do + case AdaptorsSupervisor.strategy(sup).fetch_adaptor(name) do + {:ok, record} -> + record = + record + |> Map.put(:source, source) + |> normalize_schema_data() + + {:ok, _} = AdaptorsRepo.upsert_adaptor(record) + {:commit, {:ok, Map.get(record, field)}} + + {:error, reason} -> + {:ignore, {:error, reason}} + end + end + + # Strategies should emit `schema_data` as a JSON binary, but legacy + # call paths (and tests) may still hand us a map. Normalize here so + # the cached value matches what subsequent DB-backed reads return. + defp normalize_schema_data(%{schema_data: data} = record) + when is_map(data) and not is_struct(data) do + %{record | schema_data: Jason.encode!(data)} + end + + defp normalize_schema_data(record), do: record + + @spec project_icon_meta(map()) :: icon_meta() + defp project_icon_meta(adaptor) do + Map.take(adaptor, [ + :icon_square_ext, + :icon_rectangle_ext, + :icon_square_sha256, + :icon_rectangle_sha256 + ]) + end + + @spec project_versions([map()]) :: [version_meta()] + defp project_versions(rows) do + Enum.map( + rows, + &Map.take(&1, [ + :version, + :integrity, + :size_bytes, + :published_at, + :deprecated + ]) + ) + end + + @spec ext_for_shape(icon_meta(), :square | :rectangle) :: + {:ok, String.t()} | {:error, :not_found} + defp ext_for_shape(meta, shape) do + case Map.get(meta, :"icon_#{shape}_ext") do + nil -> {:error, :not_found} + ext -> {:ok, ext} + end + end + + @spec sha256_for_shape(icon_meta(), :square | :rectangle) :: + {:ok, binary()} | {:error, :not_found} + defp sha256_for_shape(meta, shape) do + case Map.get(meta, :"icon_#{shape}_sha256") do + nil -> {:error, :not_found} + sha -> {:ok, sha} + end + end + + # `Cachex.fetch/4` returns one of: + # * `{:ok, value}` — cache hit (or coalesced peer of a `:commit`) + # * `{:commit, value}` — fallback ran and committed + # * `{:ignore, value}` — fallback ran and chose not to cache + # * `{:error, term}` — Cachex-side failure (fallback raised, etc.) + # + # Our fallbacks return `{:commit, {:ok, _}}` / `{:ignore, {:error, _}}`, + # so the wrapper tuple's second element is itself the public + # `{:ok, _} | {:error, _}` we want to return. Cachex-side `{:error, _}` + # passes through unchanged. + @spec unwrap(tuple()) :: {:ok, term()} | {:error, term()} + defp unwrap({:ok, inner}), do: inner + defp unwrap({:commit, inner}), do: inner + defp unwrap({:ignore, inner}), do: inner + defp unwrap({:error, _} = error), do: error +end diff --git a/lib/lightning/adaptors/strategy.ex b/lib/lightning/adaptors/strategy.ex new file mode 100644 index 00000000000..1a82a73c1f0 --- /dev/null +++ b/lib/lightning/adaptors/strategy.ex @@ -0,0 +1,148 @@ +defmodule Lightning.Adaptors.Strategy do + @moduledoc """ + Behaviour shared by every adaptor strategy (NPM, Local, and the test + mock). + + A strategy is the sole boundary between the `Lightning.Adaptors.*` + subsystem and the outside world. It defines four callbacks: + + * `c:fetch_adaptor/1` — given a package name, return a structured + `t:adaptor_record/0` covering version history, integrity hashes, + and dependency metadata. Icon fields are **not** part of this + record any more; the Scheduler stamps them on after joining the + bulk icon pipeline. + * `c:fetch_icon/2` — given a package name and an icon variant, + return the raw bytes plus extension. Used by the Store's rare + lazy-miss fallback. + * `c:fetch_icons/1` — bulk icon fetch for every adaptor known to + the strategy. The Scheduler invokes this once per tick in parallel + with its per-adaptor fan-out. Accepts a keyword list of options; + see the callback docs for `:prior_etags`. + * `c:list_adaptors/0` — the cheap change-signal: one call returning + `name + latest_version` for every `@openfn/*` package, used by + the scheduler to diff against the `adaptors` table. + + The active strategy module is resolved at runtime via + `Lightning.Adaptors.Config.strategy/0`. Implementations must surface + transient failures (5xx, timeout, nxdomain) as `{:error, term()}`; + retry policy lives at the scheduler/store layer, not here. + """ + + @typedoc """ + Per-version metadata extracted from an upstream packument or local + `package.json`. + """ + @type version_record :: %{ + version: String.t(), + integrity: String.t() | nil, + tarball_url: String.t() | nil, + size_bytes: integer() | nil, + dependencies: map(), + peer_dependencies: map(), + published_at: DateTime.t() | nil, + deprecated: boolean() + } + + @typedoc """ + The structured adaptor record returned by `c:fetch_adaptor/1`. Icon + fields are persisted separately by the Scheduler after joining + `c:fetch_icons/1` — they are not stamped onto this record. + """ + @type adaptor_record :: %{ + name: String.t(), + description: String.t() | nil, + homepage: String.t() | nil, + repository: String.t() | nil, + license: String.t() | nil, + latest_version: String.t(), + deprecated: boolean(), + schema_data: map() | nil, + schema_sha256: String.t() | nil, + versions: [version_record()] + } + + @typedoc """ + Fresh-fetch icon entry inside the `c:fetch_icons/1` result map. The + optional `:etag` field carries the upstream-provided cache validator + (verbatim from the HTTP response) and is `nil` when the upstream + didn't supply one — strategies without a transport-level validator + (e.g. `Lightning.Adaptors.Local`) omit the key entirely. + """ + @type icon_entry :: %{ + required(:data) => binary(), + required(:ext) => String.t(), + required(:sha256) => binary(), + optional(:etag) => String.t() | nil + } + + @typedoc """ + Per-shape value inside the `c:fetch_icons/1` result map. Either a + fresh `t:icon_entry/0` (200 response) or the `:not_modified` sentinel + (304 response — upstream confirmed unchanged; only ever returned when + the caller supplied a prior etag via the `:prior_etags` option). + """ + @type icon_shape_value :: icon_entry() | :not_modified + + @typedoc """ + Bulk icon map returned by `c:fetch_icons/1`. Three branches matter: + + * shape **entirely absent** — upstream had no such icon for this + package; + * shape present as `:not_modified` — upstream confirmed the icon + is unchanged since the prior etag was issued; + * shape present as a map — apply the bytes (a fresh fetch). + """ + @type icons_map :: %{ + required(String.t()) => %{ + optional(:square) => icon_shape_value(), + optional(:rectangle) => icon_shape_value() + } + } + + @doc """ + Fetch the full structured record for a single adaptor package. + """ + @callback fetch_adaptor(name :: String.t()) :: + {:ok, adaptor_record()} | {:error, term()} + + @doc """ + Fetch the raw bytes for one icon variant (`:square` or `:rectangle`) + of an adaptor package, together with the file extension. + """ + @callback fetch_icon(name :: String.t(), :square | :rectangle) :: + {:ok, %{data: binary(), ext: String.t()}} + | {:error, term()} + + @doc """ + Bulk fetch every available icon for every adaptor known to the + strategy. + + Returns `{:ok, partial_map}` where each per-shape slot is either + absent (no icon upstream), a fresh `t:icon_entry/0` (200), or the + `:not_modified` sentinel (304 — only when a prior etag was sent). + A top-level `{:error, term()}` is only returned when the whole + pipeline can't proceed (e.g. an upstream `list_adaptors/0` call + inside the bulk implementation fails). + + ## Options + + * `:prior_etags` — a map of the form + `%{name => %{optional(:square | :rectangle) => etag_string}}` + whose values are sent as `If-None-Match` per `(name, shape)`. + Defaults to `%{}`. Unknown keys in the keyword list are + ignored. Strategies without a transport-level cache validator + (e.g. `Lightning.Adaptors.Local`) ignore this option entirely + and never return `:not_modified`. + """ + @callback fetch_icons(opts :: keyword()) :: + {:ok, icons_map()} | {:error, term()} + + @doc """ + Cheap change-signal listing: `name + latest_version` for every + `@openfn/*` package known to the strategy. The scheduler diffs this + against the `adaptors` table to compute its work list. + """ + @callback list_adaptors() :: + {:ok, [%{name: String.t(), latest_version: String.t()}]} + | {:error, term()} +end diff --git a/lib/lightning/adaptors/supervisor.ex b/lib/lightning/adaptors/supervisor.ex new file mode 100644 index 00000000000..c1691630854 --- /dev/null +++ b/lib/lightning/adaptors/supervisor.ex @@ -0,0 +1,242 @@ +defmodule Lightning.Adaptors.Supervisor do + @moduledoc """ + Per-instance supervisor for the `Lightning.Adaptors.*` subsystem. + + The entire subsystem boots, crashes, and is supervised as a unit + under `:rest_for_one`. `Cachex` is the load-bearing root: if it + crashes, the supervisor restarts it and cascades to its dependents + (`Task.Supervisor`, plus the broadcaster/scheduler children added in + later phases) so they re-bind to the fresh Cachex name on the way + back up. + + No registered name, Cachex table name, PubSub topic, `Task.Supervisor` + name, or `HighlanderPG` lock key is hardcoded. Every name is derived + from a single `:name` opt — which is what lets the integration suite + spin up multiple isolated instances inside one BEAM for + `async: true` tests. Production starts exactly one instance under + `name: Lightning.Adaptors`. + + ## Cluster-singleton Scheduler + + The `Lightning.Adaptors.Scheduler` is wrapped in `HighlanderPG` + (`pg_try_advisory_lock` on `lock_key/1`) so exactly one node in a + multi-node deployment runs the refresh tick. The inner Scheduler + registers under `{:global, global_scheduler_name(name)}`; callers on + any node hit the leader transparently via Erlang distribution. + + ## Strategy injection + + The active `Lightning.Adaptors.Strategy` implementation is passed in + explicitly via the `:strategy` opt. Tests instantiate an isolated + supervisor with `strategy: Lightning.Adaptors.StrategyMock` — no + `Application.put_env` mutation, no shared mutable state. The + production caller in `lib/lightning/application.ex` passes the + default from `Lightning.Adaptors.Config.strategy/0` (resolved from + Application env at boot time). + + `strategy/1` and `source/1` expose the per-instance values back to + the stateless `Lightning.Adaptors.Store` callers. + """ + + use Supervisor + + alias Lightning.Adaptors.Config + + @doc """ + Start a supervisor instance. + + Required opts: + + * `:name` — supervisor instance name (atom). Derives every child + name via `Module.concat/2`. + + Optional opts: + + * `:strategy` — `Lightning.Adaptors.Strategy` implementation. + Defaults to `Lightning.Adaptors.Config.strategy/0`. + + * `:lock_key` — explicit `HighlanderPG` advisory-lock key. Defaults + to `lock_key(name)`. Override only in integration tests where + multiple supervisor instances must compete for the same lock. + """ + @spec start_link(keyword()) :: Supervisor.on_start() + def start_link(opts) do + name = Keyword.fetch!(opts, :name) + Supervisor.start_link(__MODULE__, opts, name: name) + end + + @impl true + def init(opts) do + name = Keyword.fetch!(opts, :name) + strategy = Keyword.get(opts, :strategy, Config.strategy()) + lock_key = Keyword.get(opts, :lock_key, lock_key(name)) + + :persistent_term.put(meta_key(name), %{ + strategy: strategy, + source: source_for(strategy) + }) + + cache = cache_name(name) + tasks = tasks_name(name) + source_topic = source_topic(name) + client_topic = client_topic(name) + + scheduler_child = + %{ + id: Lightning.Adaptors.Scheduler, + start: + {Lightning.Adaptors.Scheduler, :start_link, + [ + [ + name: global_scheduler_name(name), + sup: name, + lock_key: lock_key, + cache: cache, + tasks: tasks, + source_topic: source_topic + ] + ]} + } + + children = [ + {Cachex, name: cache}, + {Task.Supervisor, name: tasks}, + {Lightning.Adaptors.Invalidator, + name: invalidator_name(name), source_topic: source_topic, cache: cache}, + {Lightning.Adaptors.NodeMonitor, name: node_monitor_name(name), sup: name}, + {Lightning.Adaptors.ChannelBroadcaster, + name: channel_broadcaster_name(name), + source_topic: source_topic, + client_topic: client_topic, + sup: name}, + Supervisor.child_spec( + {HighlanderPG, + child: scheduler_child, + repo: Lightning.Repo, + name: lock_key, + sup_name: highlander_name(name)}, + id: highlander_name(name) + ) + ] + + Supervisor.init(children, strategy: :rest_for_one) + end + + @doc """ + The active strategy for the supervisor instance named `name`. + + Reads from `:persistent_term` populated at `init/1`. Raises if the + supervisor has not been started under that name. + """ + @spec strategy(atom()) :: module() + def strategy(name) do + :persistent_term.get(meta_key(name)).strategy + end + + @doc """ + The active source (`:npm | :local`) for the supervisor instance + named `name`. + """ + @spec source(atom()) :: :npm | :local + def source(name) do + :persistent_term.get(meta_key(name)).source + end + + @doc """ + Best-effort cleanup of the per-instance `:persistent_term` entry. + + Not called automatically — `:persistent_term.erase/1` triggers a + global GC and is expensive enough that we leave it to deliberate + teardown paths (e.g. release shutdown). + """ + @spec forget(atom()) :: boolean() + def forget(name) do + :persistent_term.erase(meta_key(name)) + end + + @doc "Cachex table name for the supervisor named `name`." + @spec cache_name(atom()) :: atom() + def cache_name(name), do: Module.concat(name, Cache) + + @doc "`Task.Supervisor` name for the supervisor named `name`." + @spec tasks_name(atom()) :: atom() + def tasks_name(name), do: Module.concat(name, Tasks) + + @doc "`Invalidator` GenServer name for the supervisor named `name`." + @spec invalidator_name(atom()) :: atom() + def invalidator_name(name), do: Module.concat(name, Invalidator) + + @doc "`ChannelBroadcaster` GenServer name for the supervisor named `name`." + @spec channel_broadcaster_name(atom()) :: atom() + def channel_broadcaster_name(name), + do: Module.concat(name, ChannelBroadcaster) + + @doc "`NodeMonitor` GenServer name for the supervisor named `name`." + @spec node_monitor_name(atom()) :: atom() + def node_monitor_name(name), do: Module.concat(name, NodeMonitor) + + @doc """ + Local `Scheduler` GenServer name for the supervisor named `name`. + + The inner Scheduler is actually registered globally — see + `global_scheduler_name/1`. This atom form is retained as the + child-spec `id` and for derived module names. + """ + @spec scheduler_name(atom()) :: atom() + def scheduler_name(name), do: Module.concat(name, Scheduler) + + @doc """ + `:global`-registered Scheduler name for the supervisor named `name`. + + The HighlanderPG-wrapped Scheduler registers itself under this name so + callers on any node reach the leader via Erlang distribution. Pass the + return value to `GenServer.call/3` directly. + """ + @spec global_scheduler_name(atom()) :: {:global, atom()} + def global_scheduler_name(name), do: {:global, scheduler_name(name)} + + @doc """ + `HighlanderPG` supervisor name for the supervisor named `name`. + + Used as the child-spec id and the `:sup_name` for introspection + (`HighlanderPG.which_children/1`, etc.). + """ + @spec highlander_name(atom()) :: atom() + def highlander_name(name), do: Module.concat(name, HighlanderPG) + + @doc """ + Source-side PubSub topic for the supervisor named `name`. + + Used by the `Scheduler` and `Invalidator` to broadcast and receive + `{:changed, name, source}` style events. + """ + @spec source_topic(atom()) :: String.t() + def source_topic(name), do: "adaptors:#{inspect(name)}" + + @doc """ + Client-side PubSub topic for the supervisor named `name`. + + The `ChannelBroadcaster` republishes throttled updates from + `source_topic/1` onto this topic for `WorkflowChannel` subscribers. + """ + @spec client_topic(atom()) :: String.t() + def client_topic(name), do: "adaptors:client_update:#{inspect(name)}" + + @doc """ + Postgres advisory-lock key for the supervisor named `name`. + + Derived as `:erlang.phash2({:adaptors, name})` so each supervisor + instance leases its `HighlanderPG`-wrapped `Scheduler` against a + distinct `int4` key — two concurrent test supervisors with different + names cannot collide on advisory locks. The §12.7 integration test + overrides `:lock_key` on `start_link/1` to force two supervisors to + compete for the same lock. + """ + @spec lock_key(atom()) :: non_neg_integer() + def lock_key(name), do: :erlang.phash2({:adaptors, name}) + + defp meta_key(name), do: {__MODULE__, name} + + defp source_for(Lightning.Adaptors.Local), do: :local + defp source_for(_other), do: :npm +end diff --git a/lib/lightning/ai_assistant/ai_assistant.ex b/lib/lightning/ai_assistant/ai_assistant.ex index b95886e1a4f..b6ad226feeb 100644 --- a/lib/lightning/ai_assistant/ai_assistant.ex +++ b/lib/lightning/ai_assistant/ai_assistant.ex @@ -614,7 +614,7 @@ defmodule Lightning.AiAssistant do %{ session | expression: expression, - adaptor: Lightning.AdaptorRegistry.resolve_adaptor(adaptor) + adaptor: Lightning.Adaptors.PackageName.to_wire(adaptor) } end diff --git a/lib/lightning/application.ex b/lib/lightning/application.ex index 2ab64a218a0..e915ae1c2fc 100644 --- a/lib/lightning/application.ex +++ b/lib/lightning/application.ex @@ -148,6 +148,7 @@ defmodule Lightning.Application do LightningWeb.WorkerPresence, adaptor_registry_childspec, adaptor_service_childspec, + {Lightning.Adaptors.Supervisor, name: Lightning.Adaptors}, {Lightning.TaskWorker, name: :cli_task_worker}, {Lightning.Runtime.RuntimeManager, worker_secret: Lightning.Config.worker_secret(), diff --git a/lib/lightning/credentials.ex b/lib/lightning/credentials.ex index ed237b5d75c..f1ded9a6ed5 100644 --- a/lib/lightning/credentials.ex +++ b/lib/lightning/credentials.ex @@ -575,15 +575,12 @@ defmodule Lightning.Credentials do """ @spec get_schema(String.t()) :: Credentials.Schema.t() def get_schema(schema_name) do - {:ok, schemas_path} = Application.fetch_env(:lightning, :schemas_path) - - File.read("#{schemas_path}/#{schema_name}.json") - |> case do - {:ok, raw_json} -> - Credentials.Schema.new(raw_json, schema_name) + case Lightning.Adaptors.schema(schema_name) do + {:ok, schema_body} -> + Credentials.Schema.new(schema_body, schema_name) {:error, reason} -> - raise "Error reading credential schema. Got: #{reason |> inspect()}" + raise "Error reading credential schema. Got: #{inspect(reason)}" end end diff --git a/lib/lightning_web/channels/run_with_options.ex b/lib/lightning_web/channels/run_with_options.ex index 643466ee291..18aaff4941a 100644 --- a/lib/lightning_web/channels/run_with_options.ex +++ b/lib/lightning_web/channels/run_with_options.ex @@ -1,7 +1,7 @@ defmodule LightningWeb.RunWithOptions do @moduledoc false - alias Lightning.AdaptorRegistry + alias Lightning.Adaptors.PackageName alias Lightning.Run alias Lightning.Workflows.Snapshot.Edge alias Lightning.Workflows.Snapshot.Job @@ -40,7 +40,7 @@ defmodule LightningWeb.RunWithOptions do def render(%Job{} = job) do %{ "id" => job.id, - "adaptor" => AdaptorRegistry.resolve_adaptor(job.adaptor), + "adaptor" => PackageName.to_wire(job.adaptor), "credential_id" => get_credential_id(job), "body" => job.body, "name" => job.name diff --git a/lib/lightning_web/channels/workflow_channel.ex b/lib/lightning_web/channels/workflow_channel.ex index fe2283f8141..48fe065bd19 100644 --- a/lib/lightning_web/channels/workflow_channel.ex +++ b/lib/lightning_web/channels/workflow_channel.ex @@ -16,7 +16,6 @@ defmodule LightningWeb.WorkflowChannel do alias Lightning.Repo alias Lightning.VersionControl alias Lightning.VersionControl.VersionControlUsageLimiter - alias Lightning.Workflows.Job alias Lightning.Workflows.Snapshot alias Lightning.Workflows.Workflow alias Lightning.Workflows.WorkflowUsageLimiter @@ -72,6 +71,11 @@ defmodule LightningWeb.WorkflowChannel do "workflow:collaborate:#{workflow_id}" ) + Phoenix.PubSub.subscribe( + Lightning.PubSub, + Lightning.Adaptors.Supervisor.client_topic(Lightning.Adaptors) + ) + {:ok, assign(socket, workflow_id: workflow_id, @@ -96,40 +100,11 @@ defmodule LightningWeb.WorkflowChannel do @impl true def handle_in("request_adaptors", _payload, socket) do async_task(socket, "request_adaptors", fn -> - adaptors = Lightning.AdaptorRegistry.all() - %{adaptors: adaptors} - end) - end + adaptors = + list_all_packages() + |> Enum.map(&with_icon_urls/1) - @impl true - def handle_in("request_project_adaptors", _payload, socket) do - project = socket.assigns.project - - async_task(socket, "request_project_adaptors", fn -> - project_adaptor_names = - from(j in Job, - join: w in assoc(j, :workflow), - where: w.project_id == ^project.id, - select: j.adaptor, - distinct: true - ) - |> Lightning.Repo.all() - |> Enum.sort() - - all_adaptors = Lightning.AdaptorRegistry.all() - - project_adaptors = - all_adaptors - |> Enum.filter(fn adaptor -> - Enum.any?(project_adaptor_names, fn used_adaptor -> - String.starts_with?(used_adaptor, adaptor.name) - end) - end) - - %{ - project_adaptors: project_adaptors, - all_adaptors: all_adaptors - } + %{adaptors: adaptors} end) end @@ -698,6 +673,12 @@ defmodule LightningWeb.WorkflowChannel do {:noreply, socket} end + @impl true + def handle_info(%{event: "adaptors_updated", payload: payload}, socket) do + push(socket, "adaptors_updated", payload) + {:noreply, socket} + end + @impl true def handle_info( %{event: "webhook_auth_methods_updated", payload: webhook_auth_methods}, @@ -831,6 +812,30 @@ defmodule LightningWeb.WorkflowChannel do {:noreply, socket} end + defp list_all_packages do + case Lightning.Adaptors.packages() do + {:ok, pkgs} -> pkgs + {:error, _} -> [] + end + end + + defp with_icon_urls(adaptor) do + Map.put(adaptor, :icon_urls, icon_urls_for(adaptor.name)) + end + + defp icon_urls_for(name) do + case Lightning.Adaptors.icon_meta(name) do + {:ok, meta} -> + %{ + square: LightningWeb.AdaptorIconURL.build(name, meta, :square), + rectangle: LightningWeb.AdaptorIconURL.build(name, meta, :rectangle) + } + + {:error, :not_found} -> + %{square: nil, rectangle: nil} + end + end + defp handle_async_event("request_run_steps", socket_ref, reply) do unwrapped_reply = unwrap_run_steps_reply(reply) reply(socket_ref, unwrapped_reply) @@ -839,7 +844,6 @@ defmodule LightningWeb.WorkflowChannel do defp handle_async_event(event, socket_ref, reply) when event in [ "request_adaptors", - "request_project_adaptors", "request_credentials", "request_metadata", "request_current_user", diff --git a/lib/lightning_web/components/layouts/settings.html.heex b/lib/lightning_web/components/layouts/settings.html.heex index 2fabf7de552..4b8226ab2f2 100644 --- a/lib/lightning_web/components/layouts/settings.html.heex +++ b/lib/lightning_web/components/layouts/settings.html.heex @@ -73,6 +73,13 @@ <.icon name="hero-circle-stack" class="h-5 w-5 shrink-0" /> Collections + + <.icon name="hero-wrench-screwdriver" class="h-5 w-5 shrink-0" /> + Maintenance + binary_part(0, 4) |> Base.encode16(case: :lower) + + "/adaptors/icons/#{URI.encode(name, &URI.char_unreserved?/1)}/" <> + "#{shape}-#{sha8}.#{ext}" + else + _ -> nil + end + end +end + +defmodule LightningWeb.AdaptorIconController do + @moduledoc """ + Serves content-addressable adaptor icons. + + Route: `/adaptors/icons/:name/:shape-:sha8.:ext` + + `sha8` is the first 4 raw bytes of the stored sha256 hex-encoded to + 8 lowercase characters. The controller compares `sha8` against the + DB-projected metadata and responds with one of: + + - **200** — sha matches; serves bytes with a 1-year immutable cache. + - **302** — sha is stale but the adaptor still has an icon; redirects + to the canonical (current-sha) URL with `Cache-Control: no-store` + on the redirect itself. + - **404** — adaptor unknown, ext mismatch, bad shape, or no icon. + """ + + use LightningWeb, :controller + + alias Lightning.Adaptors + + @immutable_cache "public, max-age=31536000, immutable" + + # Router-shaped params: a single `:filename` segment of the form + # `-.` because Phoenix path matchers permit only one + # dynamic segment per path component. We split here and delegate to the + # 4-key clause below, which is also what the unit tests call directly. + @filename_regex ~r/\A(?[a-z]+)-(?[A-Fa-f0-9]+)\.(?[A-Za-z0-9]+)\z/ + + @doc false + def show(conn, %{"name" => name, "filename" => filename}) do + case Regex.named_captures(@filename_regex, filename) do + %{"shape" => shape, "sha8" => sha8, "ext" => ext} -> + show(conn, %{ + "name" => name, + "shape" => shape, + "sha8" => sha8, + "ext" => ext + }) + + _ -> + send_resp(conn, 404, "") + end + end + + def show( + conn, + %{"name" => name, "shape" => shape, "sha8" => sha8, "ext" => ext} + ) + when shape in ~w(square rectangle) do + case Adaptors.icon_meta(name) do + {:error, :not_found} -> + send_resp(conn, 404, "") + + {:ok, meta} -> + cond do + ext_for_shape(meta, shape) != ext -> + send_resp(conn, 404, "") + + not has_icon?(meta, shape) -> + send_resp(conn, 404, "") + + sha_matches?(meta, shape, sha8) -> + serve_bytes(conn, name, shape, ext) + + true -> + redirect_to_current(conn, name, meta, shape) + end + end + end + + def show(conn, _params), do: send_resp(conn, 404, "") + + defp serve_bytes(conn, name, shape, ext) do + case Adaptors.icon(name, String.to_existing_atom(shape)) do + {:ok, path} -> + conn + |> put_resp_content_type(content_type_for(ext)) + |> put_resp_header("cache-control", @immutable_cache) + |> send_file(200, path) + + {:error, _} -> + send_resp(conn, 404, "") + end + end + + defp redirect_to_current(conn, name, meta, shape) do + url = + LightningWeb.AdaptorIconURL.build( + name, + meta, + String.to_existing_atom(shape) + ) + + conn + |> put_resp_header("cache-control", "no-store") + |> put_resp_header("location", url) + |> send_resp(302, "") + end + + defp has_icon?(meta, shape), do: not is_nil(ext_for_shape(meta, shape)) + + defp ext_for_shape(meta, shape), do: Map.get(meta, :"icon_#{shape}_ext") + + defp sha_matches?(meta, shape, sha8) do + case Map.get(meta, :"icon_#{shape}_sha256") do + <> -> + Base.encode16(prefix, case: :lower) == String.downcase(sha8) + + _ -> + false + end + end + + defp content_type_for("png"), do: "image/png" + defp content_type_for("svg"), do: "image/svg+xml" +end diff --git a/lib/lightning_web/live/job_live/adaptor_picker.ex b/lib/lightning_web/live/job_live/adaptor_picker.ex index 6a6460c92c1..d4ace5565f3 100644 --- a/lib/lightning_web/live/job_live/adaptor_picker.ex +++ b/lib/lightning_web/live/job_live/adaptor_picker.ex @@ -109,7 +109,7 @@ defmodule LightningWeb.JobLive.AdaptorPicker do |> assign_new( :local_adaptors_enabled?, fn -> - Lightning.AdaptorRegistry.local_adaptors_enabled?() + Lightning.Adaptors.Config.current_source() == :local end )} end @@ -133,19 +133,19 @@ defmodule LightningWeb.JobLive.AdaptorPicker do def get_adaptor_version_options(adaptor) do adaptor_names = - Lightning.AdaptorRegistry.all() + list_all_adaptors() |> Enum.map(&display_name_for_adaptor(&1.name)) |> Enum.sort() {module_name, version, versions} = if adaptor do {module_name, version} = - Lightning.AdaptorRegistry.resolve_package_name(adaptor) + Lightning.Adaptors.PackageName.parse(adaptor) - latest = Lightning.AdaptorRegistry.latest_for(module_name) + latest = latest_for(module_name) versions = - Lightning.AdaptorRegistry.versions_for(module_name) + versions_for(module_name) |> List.wrap() |> Enum.map(&Map.get(&1, :version)) |> sort_versions_desc() @@ -177,6 +177,31 @@ defmodule LightningWeb.JobLive.AdaptorPicker do [key: version, value: "#{module_name}@#{version}"] end + defp list_all_adaptors do + case Lightning.Adaptors.packages() do + {:ok, pkgs} -> pkgs + {:error, _} -> [] + end + end + + defp latest_for(nil), do: nil + + defp latest_for(name) do + case Lightning.Adaptors.resolve_version(name, "latest") do + {:ok, version} -> version + {:error, _} -> nil + end + end + + defp versions_for(nil), do: [] + + defp versions_for(name) do + case Lightning.Adaptors.versions(name) do + {:ok, versions} -> versions + {:error, _} -> [] + end + end + @doc "Sort version strings in descending semver order." @spec sort_versions_desc([String.t()]) :: [String.t()] def sort_versions_desc(versions) when is_list(versions) do @@ -192,7 +217,7 @@ defmodule LightningWeb.JobLive.AdaptorPicker do socket ) do # Get the latest specific version instead of using @latest - latest_version = Lightning.AdaptorRegistry.latest_for(value) + latest_version = latest_for(value) adaptor_value = if latest_version do diff --git a/lib/lightning_web/live/maintenance_live/index.ex b/lib/lightning_web/live/maintenance_live/index.ex new file mode 100644 index 00000000000..4a6ab504b6b --- /dev/null +++ b/lib/lightning_web/live/maintenance_live/index.ex @@ -0,0 +1,85 @@ +defmodule LightningWeb.MaintenanceLive.Index do + @moduledoc """ + Superuser-only maintenance page exposing on-demand operations against the + `Lightning.Adaptors` supervisor (and, eventually, other facades). + + Currently exposes a single action: "Refresh Adaptor Registry", which calls + `Lightning.Adaptors.refresh_now/0`. The call is fire-and-forget; the user + receives a flash and the actual work happens asynchronously on the leader + node. + """ + use LightningWeb, :live_view + + alias Lightning.Policies.Permissions + alias Lightning.Policies.Users + + @impl true + def mount(_params, _session, socket) do + if superuser?(socket) do + {:ok, + assign(socket, + active_menu_item: :maintenance, + page_title: "Maintenance" + ), layout: {LightningWeb.Layouts, :settings}} + else + {:ok, + socket + |> put_flash(:nav, :no_access) + |> push_navigate(to: "/projects")} + end + end + + @impl true + def handle_event("refresh_adaptors", _params, socket) do + if superuser?(socket) do + socket = + case Lightning.Adaptors.refresh_now() do + :ok -> + put_flash(socket, :info, "Adaptor refresh queued.") + + {:error, reason} -> + put_flash(socket, :error, "Refresh failed: #{inspect(reason)}") + end + + {:noreply, socket} + else + {:noreply, + socket + |> put_flash(:nav, :no_access) + |> push_navigate(to: "/projects")} + end + end + + def handle_event("refresh_icons", _params, socket) do + if superuser?(socket) do + socket = + case Lightning.Adaptors.refresh_icons() do + {:ok, %{updated: updated, unchanged: unchanged}} -> + put_flash( + socket, + :info, + "Icon refresh complete — #{updated} updated, #{unchanged} unchanged." + ) + + {:error, reason} -> + put_flash(socket, :error, "Icon refresh failed: #{inspect(reason)}") + end + + {:noreply, socket} + else + {:noreply, + socket + |> put_flash(:nav, :no_access) + |> push_navigate(to: "/projects")} + end + end + + defp superuser?(socket) do + Permissions.can?( + Users, + :access_admin_space, + socket.assigns.current_user, + {} + ) + end +end diff --git a/lib/lightning_web/live/maintenance_live/index.html.heex b/lib/lightning_web/live/maintenance_live/index.html.heex new file mode 100644 index 00000000000..eb018839448 --- /dev/null +++ b/lib/lightning_web/live/maintenance_live/index.html.heex @@ -0,0 +1,53 @@ + + <:header> + + <:title>Maintenance + + + +
+
+
+

+ Refresh Adaptor Registry +

+

+ Re-fetch the list of available adaptors and their versions. +

+
+ <.button + theme="primary" + phx-click="refresh_adaptors" + id="refresh-adaptors-button" + > + Run + +
+ +
+
+

+ Refresh Adaptor Icons +

+

+ Re-fetch icons for every adaptor and replace rows whose icon has + changed. Does not touch adaptor metadata or version history. +

+
+ <.button + theme="primary" + phx-click="refresh_icons" + id="refresh-icons-button" + > + Run + +
+
+
+
diff --git a/lib/lightning_web/live/workflow_live/edit.ex b/lib/lightning_web/live/workflow_live/edit.ex index cabfcf2c9b6..c23ff8ca08b 100644 --- a/lib/lightning_web/live/workflow_live/edit.ex +++ b/lib/lightning_web/live/workflow_live/edit.ex @@ -3837,11 +3837,7 @@ defmodule LightningWeb.WorkflowLive.Edit do defp maybe_add_default_adaptor(job_param) do if Map.keys(job_param) == ["id"] do - job_param - |> Map.put( - "adaptor", - Lightning.AdaptorRegistry.resolve_adaptor(%Job{}.adaptor) - ) + Map.put(job_param, "adaptor", %Job{}.adaptor) else job_param end diff --git a/lib/lightning_web/live/workflow_live/editor_pane.ex b/lib/lightning_web/live/workflow_live/editor_pane.ex index b8023c6b257..169aa61384f 100644 --- a/lib/lightning_web/live/workflow_live/editor_pane.ex +++ b/lib/lightning_web/live/workflow_live/editor_pane.ex @@ -42,7 +42,7 @@ defmodule LightningWeb.WorkflowLive.EditorPane do |> assign( adaptor: form[:adaptor].value - |> Lightning.AdaptorRegistry.resolve_adaptor(), + |> Lightning.Adaptors.PackageName.to_wire(), source: form.source.data.body, job_id: form[:id].value ) diff --git a/lib/lightning_web/live/workflow_live/job_view.ex b/lib/lightning_web/live/workflow_live/job_view.ex index 59e37e929b8..fdae197f729 100644 --- a/lib/lightning_web/live/workflow_live/job_view.ex +++ b/lib/lightning_web/live/workflow_live/job_view.ex @@ -288,7 +288,7 @@ defmodule LightningWeb.WorkflowLive.JobView do defp adaptor_block(assigns) do {package_name, version} = - Lightning.AdaptorRegistry.resolve_package_name(assigns.adaptor) + Lightning.Adaptors.PackageName.parse(assigns.adaptor) assigns = assigns diff --git a/lib/lightning_web/router.ex b/lib/lightning_web/router.ex index e8f64c1fe53..b08ed89edf6 100644 --- a/lib/lightning_web/router.ex +++ b/lib/lightning_web/router.ex @@ -68,6 +68,10 @@ defmodule LightningWeb.Router do get "/authenticate/:provider/callback", OidcController, :new get "/oauth/:provider/callback", OauthController, :new + + get "/adaptors/icons/:name/:filename", + AdaptorIconController, + :show end ## JSON API @@ -225,6 +229,8 @@ defmodule LightningWeb.Router do live "/settings/audit", AuditLive.Index, :index + live "/settings/maintenance", MaintenanceLive.Index, :index + live "/settings/authentication", AuthProvidersLive.Index, :edit live "/settings/authentication/new", AuthProvidersLive.Index, :new diff --git a/lib/mix/tasks/lightning.refresh_adaptors.ex b/lib/mix/tasks/lightning.refresh_adaptors.ex new file mode 100644 index 00000000000..74707fec9b7 --- /dev/null +++ b/lib/mix/tasks/lightning.refresh_adaptors.ex @@ -0,0 +1,59 @@ +defmodule Mix.Tasks.Lightning.RefreshAdaptors do + @shortdoc "On-demand adaptor metadata refresh" + @moduledoc """ + Trigger an immediate adaptor refresh from the command line. + + Use cases: + + * Dev re-scan — force a re-scan after adding local adaptors + * Ops force-pull — pull latest metadata without waiting for the scheduler tick + + ## Usage + + mix lightning.refresh_adaptors + mix lightning.refresh_adaptors --name @openfn/language-http + + The first form calls `Lightning.Adaptors.refresh_now/0`, refreshing all + adaptors. The second form calls `Lightning.Adaptors.refresh_package/1` + to force a single-adaptor refresh, bypassing the ledger diff. + + Both forms block until completion. The Scheduler is wrapped in + `HighlanderPG` and registered globally, so the call routes through + Erlang distribution to whichever node currently holds the lease — the + CLI can be run from any node in the cluster. + + ## Exit codes + + * `0` — success + * `1` — package name not found (possible typo) + * `2` — other error + """ + + use Mix.Task + + @impl Mix.Task + def run(argv) do + Mix.Task.run("app.start") + + {opts, _args} = OptionParser.parse!(argv, strict: [name: :string]) + + result = + case opts[:name] do + nil -> Lightning.Adaptors.refresh_now() + pkg -> Lightning.Adaptors.refresh_package(pkg) + end + + case result do + :ok -> + Mix.shell().info("Adaptors refreshed successfully.") + + {:error, :not_found} -> + Mix.shell().error("Package not found. Check the name and try again.") + exit({:shutdown, 1}) + + {:error, reason} -> + Mix.shell().error("Refresh failed: #{inspect(reason)}") + exit({:shutdown, 2}) + end + end +end diff --git a/mix.exs b/mix.exs index 3bb0167ab35..e35010191ea 100644 --- a/mix.exs +++ b/mix.exs @@ -97,6 +97,7 @@ defmodule Lightning.MixProject do {:google_api_storage, "~> 0.46.0"}, {:hackney, "~> 1.18"}, {:heroicons, "~> 0.5.3"}, + {:highlander_pg, "~> 1.0"}, {:httpoison, "~> 2.0"}, {:jason, "~> 1.4"}, {:joken, "~> 2.6.0"}, diff --git a/mix.lock b/mix.lock index 4b39c1d7619..d8be8ec81a0 100644 --- a/mix.lock +++ b/mix.lock @@ -64,6 +64,7 @@ "hammer": {:hex, :hammer, "6.2.1", "5ae9c33e3dceaeb42de0db46bf505bd9c35f259c8defb03390cd7556fea67ee2", [:mix], [{:poolboy, "~> 1.5", [hex: :poolboy, repo: "hexpm", optional: false]}], "hexpm", "b9476d0c13883d2dc0cc72e786bac6ac28911fba7cc2e04b70ce6a6d9c4b2bdc"}, "hammer_backend_mnesia": {:hex, :hammer_backend_mnesia, "0.6.1", "d10d94fc29cbffbf04ecb3c3127d705ce4cc1cecfb9f3d6b18a554c3cae9af2c", [:mix], [{:hammer, "~> 6.1", [hex: :hammer, repo: "hexpm", optional: false]}], "hexpm", "85ad2ef6ebe035207dd9a03a116dc6a7ee43fbd53e8154cf32a1e33b9200fb62"}, "heroicons": {:hex, :heroicons, "0.5.6", "95d730e7179c633df32d95c1fdaaecdf81b0da11010b89b737b843ac176a7eb5", [:mix], [{:castore, ">= 0.0.0", [hex: :castore, repo: "hexpm", optional: false]}, {:phoenix_live_view, ">= 0.18.2", [hex: :phoenix_live_view, repo: "hexpm", optional: false]}], "hexpm", "ca267f02a5fa695a4178a737b649fb6644a2e399639d4ba7964c18e8a58c2352"}, + "highlander_pg": {:hex, :highlander_pg, "1.0.8", "30c5c2cd23cd48991d4b5f66368997ac711d3e736217e0a9e762051d22b0329a", [:mix], [{:ecto, "~> 3.0", [hex: :ecto, repo: "hexpm", optional: true]}, {:ecto_sql, "~> 3.0", [hex: :ecto_sql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.16.1 or ~> 0.17.0 or ~> 0.18.0 or ~> 0.19.0 or ~> 0.20.0 or ~> 0.21.0", [hex: :postgrex, repo: "hexpm", optional: false]}], "hexpm", "fe03a830971e3d626d776ed1c62e0e1148e953274f1e82eba8a0618dde1c415c"}, "hpax": {:hex, :hpax, "1.0.3", "ed67ef51ad4df91e75cc6a1494f851850c0bd98ebc0be6e81b026e765ee535aa", [:mix], [], "hexpm", "8eab6e1cfa8d5918c2ce4ba43588e894af35dbd8e91e6e55c817bca5847df34a"}, "httpoison": {:hex, :httpoison, "2.2.3", "a599d4b34004cc60678999445da53b5e653630651d4da3d14675fedc9dd34bd6", [:mix], [{:hackney, "~> 1.21", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "fa0f2e3646d3762fdc73edb532104c8619c7636a6997d20af4003da6cfc53e53"}, "idna": {:hex, :idna, "6.1.1", "8a63070e9f7d0c62eb9d9fcb360a7de382448200fbbd1b106cc96d3d8099df8d", [:rebar3], [{:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "92376eb7894412ed19ac475e4a86f7b413c1b9fbb5bd16dccd57934157944cea"}, diff --git a/priv/repo/migrations/20260514150000_create_adaptors.exs b/priv/repo/migrations/20260514150000_create_adaptors.exs new file mode 100644 index 00000000000..4426f342303 --- /dev/null +++ b/priv/repo/migrations/20260514150000_create_adaptors.exs @@ -0,0 +1,50 @@ +defmodule Lightning.Repo.Migrations.CreateAdaptors do + use Ecto.Migration + + def change do + create table(:adaptors, primary_key: false) do + add :id, :binary_id, primary_key: true + add :name, :string, null: false + add :source, :string, null: false, default: "npm" + add :description, :text + add :homepage, :string + add :repository, :string + add :license, :string + add :latest_version, :string, null: false + add :deprecated, :boolean, default: false, null: false + add :schema_data, :map + add :schema_sha256, :string + add :icon_square_ext, :string + add :icon_rectangle_ext, :string + add :icon_square_sha256, :binary + add :icon_rectangle_sha256, :binary + add :checked_at, :utc_datetime_usec, null: false + + timestamps(type: :utc_datetime_usec, null: false) + end + + create unique_index(:adaptors, [:name, :source]) + create index(:adaptors, [:source, :checked_at]) + + create table(:adaptor_versions, primary_key: false) do + add :id, :binary_id, primary_key: true + + add :adaptor_id, + references(:adaptors, type: :binary_id, on_delete: :delete_all), + null: false + + add :version, :string, null: false + add :integrity, :string + add :tarball_url, :string + add :size_bytes, :integer + add :dependencies, :map + add :peer_dependencies, :map + add :published_at, :utc_datetime_usec + add :deprecated, :boolean, default: false, null: false + + timestamps(type: :utc_datetime_usec, updated_at: false, null: false) + end + + create unique_index(:adaptor_versions, [:adaptor_id, :version]) + end +end diff --git a/priv/repo/migrations/20260519090558_add_icon_etags_to_adaptors.exs b/priv/repo/migrations/20260519090558_add_icon_etags_to_adaptors.exs new file mode 100644 index 00000000000..4d5482dceef --- /dev/null +++ b/priv/repo/migrations/20260519090558_add_icon_etags_to_adaptors.exs @@ -0,0 +1,10 @@ +defmodule Lightning.Repo.Migrations.AddIconEtagsToAdaptors do + use Ecto.Migration + + def change do + alter table(:adaptors) do + add :icon_square_etag, :string + add :icon_rectangle_etag, :string + end + end +end diff --git a/priv/repo/migrations/20260520125623_change_adaptors_schema_data_to_text.exs b/priv/repo/migrations/20260520125623_change_adaptors_schema_data_to_text.exs new file mode 100644 index 00000000000..28347505b3a --- /dev/null +++ b/priv/repo/migrations/20260520125623_change_adaptors_schema_data_to_text.exs @@ -0,0 +1,19 @@ +defmodule Lightning.Repo.Migrations.ChangeAdaptorsSchemaDataToText do + use Ecto.Migration + + def up do + execute(""" + ALTER TABLE adaptors + ALTER COLUMN schema_data TYPE text + USING schema_data::text + """) + end + + def down do + execute(""" + ALTER TABLE adaptors + ALTER COLUMN schema_data TYPE jsonb + USING schema_data::jsonb + """) + end +end diff --git a/test/integration/web_and_worker_test.exs b/test/integration/web_and_worker_test.exs index 23a9948906d..fbd83f15c8a 100644 --- a/test/integration/web_and_worker_test.exs +++ b/test/integration/web_and_worker_test.exs @@ -117,6 +117,17 @@ defmodule Lightning.WebAndWorkerTest do @tag :integration @tag timeout: 20_000 test "the whole thing", %{conn: conn, user: user} do + # `RunWithOptions.render/1` resolves `@latest` via + # `Lightning.Adaptors.PackageName.to_wire/1` (was + # `AdaptorRegistry.resolve_adaptor/1` — sourced from the legacy + # `test/fixtures/adaptor_registry_cache.json`). Seed the version + # the legacy fixture used so the worker gets a concrete pin + # instead of resolving `@latest` against live NPM. + Lightning.AdaptorTestHelpers.seed_adaptor_package( + "@openfn/language-http", + "3.1.12" + ) + project = insert(:project) # Create credential with body for main environment diff --git a/test/lightning/adaptors/channel_broadcaster_test.exs b/test/lightning/adaptors/channel_broadcaster_test.exs new file mode 100644 index 00000000000..002f0e19978 --- /dev/null +++ b/test/lightning/adaptors/channel_broadcaster_test.exs @@ -0,0 +1,238 @@ +defmodule Lightning.Adaptors.ChannelBroadcasterTest do + @moduledoc """ + Tests `:flush` via `Lightning.Adaptors.packages/1` (the 2-arity facade), + not `packages/0`, because each test spins up its own isolated supervisor + instance. The Batch 7 review should confirm that `/1` and `/0` are + behaviourally identical in production (both delegate to `Store.packages/1`). + """ + + use ExUnit.Case, async: true + + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + setup do + sup = :"cb_test_#{System.unique_integer([:positive])}" + + # The supervisor's :rest_for_one child list starts the + # ChannelBroadcaster automatically — registered under + # `channel_broadcaster_name(sup)`. + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.StrategyMock} + ) + + # Stop the auto-started Invalidator — these tests pre-populate the + # Cachex `{:packages, source}` key directly to exercise the + # ChannelBroadcaster's `:flush` path in isolation. The Invalidator + # subscribes to the same source_topic and would race the broadcaster + # by deleting the cached entry before the flush window expires. + :ok = Supervisor.terminate_child(sup, Lightning.Adaptors.Invalidator) + + source_topic = AdaptorsSupervisor.source_topic(sup) + client_topic = AdaptorsSupervisor.client_topic(sup) + cb_name = AdaptorsSupervisor.channel_broadcaster_name(sup) + cache = AdaptorsSupervisor.cache_name(sup) + source = AdaptorsSupervisor.source(sup) + + packages = [%{name: "@openfn/language-http", latest_version: "1.0.0"}] + Cachex.put!(cache, {:packages, source}, {:ok, packages}) + + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, client_topic) + + {:ok, + sup: sup, + cb_name: cb_name, + source_topic: source_topic, + cache: cache, + source: source, + packages: packages} + end + + describe "start_link/1" do + test "registers under the :name opt", %{cb_name: cb_name} do + assert is_pid(Process.whereis(cb_name)) + end + end + + describe "handle_info/2 - {:changed, ...}" do + test "first message in idle state arms the 250ms timer", %{ + cb_name: cb_name, + source_topic: source_topic + } do + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, "pkg", :npm} + ) + + %{timer: timer} = :sys.get_state(cb_name) + assert is_reference(timer) + end + + test "subsequent messages within the window are dropped — one broadcast per burst", + %{source_topic: source_topic, packages: packages} do + for _ <- 1..5 do + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, "pkg", :npm} + ) + end + + assert_receive %{ + event: "adaptors_updated", + payload: %{adaptors: ^packages} + }, + 500 + + refute_receive %{event: "adaptors_updated"}, 100 + end + + test "timer resets to nil after :flush fires", %{ + cb_name: cb_name, + source_topic: source_topic + } do + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, "pkg", :npm} + ) + + assert_receive %{event: "adaptors_updated"}, 500 + %{timer: timer} = :sys.get_state(cb_name) + assert timer == nil + end + end + + describe "handle_info/2 - :flush" do + test "broadcasts the envelope to client_topic with the correct shape", %{ + source_topic: source_topic, + packages: packages + } do + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, "pkg", :npm} + ) + + assert_receive %{ + event: "adaptors_updated", + payload: %{adaptors: ^packages} + }, + 500 + end + + test "broadcasts with empty adaptors list when packages returns {:ok, []}", + %{ + cache: cache, + source: source, + source_topic: source_topic + } do + Cachex.put!(cache, {:packages, source}, {:ok, []}) + + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, "pkg", :npm} + ) + + assert_receive %{event: "adaptors_updated", payload: %{adaptors: []}}, 500 + end + end + + describe "crash recovery" do + test "supervisor restarts the GenServer; next {:changed} re-arms cleanly", %{ + cb_name: cb_name, + source_topic: source_topic, + packages: packages + } do + original_pid = Process.whereis(cb_name) + assert is_pid(original_pid) + + ref = Process.monitor(original_pid) + + # Arm the timer, then kill the process mid-burst. + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, "pkg", :npm} + ) + + Process.exit(original_pid, :kill) + + # Confirm death before looking for the restarted process. + assert_receive {:DOWN, ^ref, :process, ^original_pid, :killed}, 500 + + new_pid = await_registered(cb_name) + assert is_pid(new_pid) + assert new_pid != original_pid + + # The new instance starts with timer: nil — one more {:changed} opens a + # fresh 250ms window and produces a clean broadcast. + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, "pkg", :npm} + ) + + assert_receive %{ + event: "adaptors_updated", + payload: %{adaptors: ^packages} + }, + 500 + end + end + + describe "leading-edge throttle invariant" do + test "a 10ms drip over 500ms yields multiple broadcasts, not 0 and not one-per-message", + %{source_topic: source_topic} do + task = + Task.async(fn -> + for _ <- 1..50 do + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, "pkg", :npm} + ) + + Process.sleep(10) + end + end) + + Task.await(task, 3_000) + # Allow time for the final flush window to fire. + Process.sleep(300) + + count = drain_broadcasts() + + # Leading-edge invariant: throttle produces some broadcasts (> 0) + # but far fewer than one per message (< 50). + assert count > 0 and count < 50, + "Expected leading-edge throttling (1..49), got #{count}" + end + end + + defp await_registered(name, deadline \\ nil) do + deadline = deadline || System.monotonic_time(:millisecond) + 500 + + case Process.whereis(name) do + nil -> + if System.monotonic_time(:millisecond) < deadline do + Process.sleep(10) + await_registered(name, deadline) + else + raise "#{inspect(name)} did not restart within 500ms" + end + + pid -> + pid + end + end + + defp drain_broadcasts(acc \\ 0) do + receive do + %{event: "adaptors_updated"} -> drain_broadcasts(acc + 1) + after + 0 -> acc + end + end +end diff --git a/test/lightning/adaptors/config_test.exs b/test/lightning/adaptors/config_test.exs new file mode 100644 index 00000000000..cce00b3d670 --- /dev/null +++ b/test/lightning/adaptors/config_test.exs @@ -0,0 +1,114 @@ +defmodule Lightning.Adaptors.ConfigTest do + use ExUnit.Case, async: true + + alias Lightning.Adaptors.Config + + @parent_key Lightning.Adaptors + + describe "current_source/0" do + test "returns :local when strategy is Lightning.Adaptors.Local" do + put_parent(:strategy, Lightning.Adaptors.Local) + + assert Config.current_source() == :local + end + + test "returns :npm for any other strategy module" do + put_parent(:strategy, Lightning.Adaptors.NPM) + assert Config.current_source() == :npm + + put_parent(:strategy, SomeOther.Strategy) + assert Config.current_source() == :npm + end + end + + describe "icon_path/0" do + test "resolves a {:tmp, suffix} tuple against System.tmp_dir!/0" do + put_parent(:icon_path, {:tmp, "lightning/adaptor_icons_under_test"}) + + assert Config.icon_path() == + Path.join(System.tmp_dir!(), "lightning/adaptor_icons_under_test") + end + + test "returns a plain binary path verbatim" do + put_parent(:icon_path, "/var/lib/lightning/adaptor_icons") + + assert Config.icon_path() == "/var/lib/lightning/adaptor_icons" + end + end + + describe "strategy_opts/1" do + test "reads the strategy module's own Application key" do + put_strategy_opts(SomeStrategy.Module, repo_path: "/tmp/local-adaptors") + + assert Config.strategy_opts(SomeStrategy.Module) == + [repo_path: "/tmp/local-adaptors"] + end + + test "returns [] when the strategy module's Application key is unset" do + clear_strategy_opts(UnsetStrategy.Module) + + assert Config.strategy_opts(UnsetStrategy.Module) == [] + end + end + + describe "defaults when unset" do + test "refresh_interval/0 defaults to :timer.hours(1)" do + delete_parent_key(:refresh_interval) + + assert Config.refresh_interval() == :timer.hours(1) + end + + test "cache_timeout_ms/0 defaults to 15_000" do + delete_parent_key(:cache_timeout_ms) + + assert Config.cache_timeout_ms() == 15_000 + end + end + + defp put_parent(key, value) do + original = Application.get_env(:lightning, @parent_key, []) + + Application.put_env( + :lightning, + @parent_key, + Keyword.put(original, key, value) + ) + + on_exit(fn -> + Application.put_env(:lightning, @parent_key, original) + end) + end + + defp delete_parent_key(key) do + original = Application.get_env(:lightning, @parent_key, []) + Application.put_env(:lightning, @parent_key, Keyword.delete(original, key)) + + on_exit(fn -> + Application.put_env(:lightning, @parent_key, original) + end) + end + + defp put_strategy_opts(mod, value) do + original = Application.get_env(:lightning, mod, :__unset__) + Application.put_env(:lightning, mod, value) + + on_exit(fn -> + restore_app_env(mod, original) + end) + end + + defp clear_strategy_opts(mod) do + original = Application.get_env(:lightning, mod, :__unset__) + Application.delete_env(:lightning, mod) + + on_exit(fn -> + restore_app_env(mod, original) + end) + end + + defp restore_app_env(mod, :__unset__), + do: Application.delete_env(:lightning, mod) + + defp restore_app_env(mod, value), + do: Application.put_env(:lightning, mod, value) +end diff --git a/test/lightning/adaptors/end_to_end_broadcast_test.exs b/test/lightning/adaptors/end_to_end_broadcast_test.exs new file mode 100644 index 00000000000..ed4e0d765c1 --- /dev/null +++ b/test/lightning/adaptors/end_to_end_broadcast_test.exs @@ -0,0 +1,48 @@ +defmodule Lightning.Adaptors.EndToEndBroadcastTest do + @moduledoc """ + Phase A closeout — §6.5c integration smoke. + + A `{:changed, name, source}` broadcast on the per-instance source + topic (the cache-coherence audience that the `Scheduler` and + `Invalidator` share) must traverse the wired stack and arrive on + the per-instance client topic (the display-freshness audience that + `WorkflowChannel` subscribers listen to) as a single coalesced + `adaptors_updated` envelope. + + This is the only assertion that breaks if any of the four newly-wired + Supervisor children (Invalidator, NodeMonitor, ChannelBroadcaster, + Scheduler) is misconfigured for the boot path. + """ + + use Lightning.DataCase, async: false + + alias Lightning.Adaptors.ChannelBroadcaster + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + test "PubSub source-topic broadcast reaches client topic as coalesced envelope" do + sup = :"e2e_#{System.unique_integer([:positive])}" + + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.Local} + ) + + source_topic = AdaptorsSupervisor.source_topic(sup) + client_topic = AdaptorsSupervisor.client_topic(sup) + + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, client_topic) + + :ok = + Phoenix.PubSub.broadcast( + Lightning.PubSub, + source_topic, + {:changed, "@openfn/language-test", :local} + ) + + # The ChannelBroadcaster fans out a map envelope (see + # `Lightning.Adaptors.ChannelBroadcaster.handle_info(:flush, _)`). + # The DB is empty in this case → `Store.packages/1` returns + # `{:ok, []}` → an empty-list envelope is broadcast. + assert_receive %{event: "adaptors_updated", payload: %{adaptors: _}}, + ChannelBroadcaster.debounce_ms() + 200 + end +end diff --git a/test/lightning/adaptors/highlander_integration_test.exs b/test/lightning/adaptors/highlander_integration_test.exs new file mode 100644 index 00000000000..bbcc3472dce --- /dev/null +++ b/test/lightning/adaptors/highlander_integration_test.exs @@ -0,0 +1,107 @@ +defmodule Lightning.Adaptors.HighlanderIntegrationTest do + @moduledoc """ + §12.7 — verifies that the HighlanderPG-wrapped `Lightning.Adaptors.Scheduler` + actually behaves as a cluster singleton when two supervisor instances + compete for the same Postgres advisory lock. + + Both supervisors share an explicit `:lock_key` so they race for the + same `pg_try_advisory_lock` bucket, but each keeps its own derived + `:global` Scheduler name. Exactly one of them — the leader — registers + a Scheduler under its `{:global, …}` name; the other's HighlanderPG + polls and waits. When the leading supervisor stops (releasing its + Postgres session and thus its advisory lock), the surviving instance + must acquire the lock within ~2× the default 300ms polling interval + and register its own Scheduler under its own `{:global, …}` name. + """ + + # async: false — real advisory locks coordinate against the test DB; + # set_mox_global so the StrategyMock is visible to the wrapped child + # processes started by HighlanderPG. + use Lightning.DataCase, async: false + + import Eventually + import Mox + + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + setup :set_mox_global + setup :verify_on_exit! + + # Both supervisors come up with refresh_interval=0 (default in config/test.exs) + # so the inert Scheduler's init does no DB work; the test only cares about + # HighlanderPG's leader election, not refresh behaviour. + + test "two supervisors sharing one lock_key: only one runs the Scheduler at a time; failover on leader shutdown" do + suffix = System.unique_integer([:positive]) + sup_a = :"hl_a_#{suffix}" + sup_b = :"hl_b_#{suffix}" + shared_lock_key = :erlang.phash2({:adaptors_highlander_test, suffix}) + + on_exit(fn -> + AdaptorsSupervisor.forget(sup_a) + AdaptorsSupervisor.forget(sup_b) + end) + + {:ok, _pid_a} = + start_supervised( + Supervisor.child_spec( + {AdaptorsSupervisor, + name: sup_a, + strategy: Lightning.Adaptors.StrategyMock, + lock_key: shared_lock_key}, + id: :sup_a + ) + ) + + {:ok, _pid_b} = + start_supervised( + Supervisor.child_spec( + {AdaptorsSupervisor, + name: sup_b, + strategy: Lightning.Adaptors.StrategyMock, + lock_key: shared_lock_key}, + id: :sup_b + ) + ) + + {:global, gname_a} = AdaptorsSupervisor.global_scheduler_name(sup_a) + {:global, gname_b} = AdaptorsSupervisor.global_scheduler_name(sup_b) + + # The advisory-lock race may go to either supervisor. Allow ~3s for + # the winner's HighlanderPG to acquire the lock and start its child. + assert_eventually( + is_pid(:global.whereis_name(gname_a)) or + is_pid(:global.whereis_name(gname_b)), + 3_000 + ) + + leader_global = + cond do + is_pid(:global.whereis_name(gname_a)) -> gname_a + is_pid(:global.whereis_name(gname_b)) -> gname_b + end + + surviving_global = + if leader_global == gname_a, do: gname_b, else: gname_a + + leader_sup = if leader_global == gname_a, do: sup_a, else: sup_b + + # Singleton invariant: only the leader's :global registration is + # populated cluster-wide. + assert :global.whereis_name(surviving_global) == :undefined, + "expected only the leader to have a globally-registered Scheduler" + + # Stop the leader supervisor. start_supervised gave us a stable id + # (:sup_a / :sup_b), so we can unambiguously target it for teardown. + leader_id = if leader_sup == sup_a, do: :sup_a, else: :sup_b + :ok = stop_supervised(leader_id) + + # The surviving HighlanderPG polls at 300ms by default; give it + # comfortably more than 2× that interval to win the lock and start + # its wrapped Scheduler under its own :global name. + assert_eventually(is_pid(:global.whereis_name(surviving_global)), 3_000) + + # Sanity: the formerly-leading :global name is gone. + assert :global.whereis_name(leader_global) == :undefined + end +end diff --git a/test/lightning/adaptors/icon_cache_test.exs b/test/lightning/adaptors/icon_cache_test.exs new file mode 100644 index 00000000000..849ee02d599 --- /dev/null +++ b/test/lightning/adaptors/icon_cache_test.exs @@ -0,0 +1,156 @@ +defmodule Lightning.Adaptors.IconCacheTest do + use ExUnit.Case, async: false + + alias Lightning.Adaptors.IconCache + + @parent_key Lightning.Adaptors + + setup do + root = + Path.join( + System.tmp_dir!(), + "lightning_icon_cache_test_#{System.unique_integer([:positive])}" + ) + + File.mkdir_p!(root) + + original = Application.get_env(:lightning, @parent_key, []) + + Application.put_env( + :lightning, + @parent_key, + Keyword.put(original, :icon_path, root) + ) + + on_exit(fn -> + Application.put_env(:lightning, @parent_key, original) + File.rm_rf!(root) + end) + + {:ok, root: root} + end + + describe "path/4" do + test "joins Config.icon_path with source/name/shape.ext", %{root: root} do + assert IconCache.path(:npm, "salesforce", :square, "png") == + Path.join([root, "npm", "salesforce", "square.png"]) + end + + test "handles names containing a slash like @openfn/language-foo", %{ + root: root + } do + assert IconCache.path(:npm, "@openfn/language-foo", :square, "png") == + Path.join([ + root, + "npm", + "@openfn", + "language-foo", + "square.png" + ]) + end + + test "source-partitions paths for the same name", %{root: root} do + npm_path = IconCache.path(:npm, "salesforce", :square, "png") + local_path = IconCache.path(:local, "salesforce", :square, "png") + + assert npm_path == Path.join([root, "npm", "salesforce", "square.png"]) + + assert local_path == + Path.join([root, "local", "salesforce", "square.png"]) + + refute npm_path == local_path + end + + test "is pure — nothing is created on disk", %{root: root} do + _ = IconCache.path(:npm, "never-written", :rectangle, "svg") + + assert File.ls!(root) == [] + end + end + + describe "cached?/4" do + test "returns false when the file does not exist" do + refute IconCache.cached?(:npm, "definitely-missing", :square, "png") + end + + test "returns true after write!/5 places the file" do + {:ok, _sha} = IconCache.write!(:npm, "cached-pkg", :square, "png", "x") + + assert IconCache.cached?(:npm, "cached-pkg", :square, "png") + end + + test "stays source-partitioned: a write to :npm doesn't satisfy :local" do + {:ok, _} = IconCache.write!(:npm, "split-pkg", :square, "png", "x") + + assert IconCache.cached?(:npm, "split-pkg", :square, "png") + refute IconCache.cached?(:local, "split-pkg", :square, "png") + end + end + + describe "write!/5" do + test "writes bytes and a round-trip read returns them" do + bytes = :crypto.strong_rand_bytes(2_048) + + {:ok, _sha} = + IconCache.write!(:npm, "round-trip", :square, "png", bytes) + + assert File.read!(IconCache.path(:npm, "round-trip", :square, "png")) == + bytes + end + + test "returns the sha256 of the supplied bytes as a 32-byte binary" do + bytes = "hello, icon" + + {:ok, sha} = IconCache.write!(:npm, "sha-test", :square, "png", bytes) + + assert sha == :crypto.hash(:sha256, bytes) + assert byte_size(sha) == 32 + end + + test "is latest-only: a subsequent write for the same key overwrites" do + {:ok, _} = IconCache.write!(:npm, "overwrite", :square, "png", "first") + {:ok, _} = IconCache.write!(:npm, "overwrite", :square, "png", "second") + + assert File.read!(IconCache.path(:npm, "overwrite", :square, "png")) == + "second" + end + + test "creates intermediate directories for scoped names" do + {:ok, _} = + IconCache.write!(:npm, "@openfn/language-http", :square, "png", "abc") + + assert File.read!( + IconCache.path(:npm, "@openfn/language-http", :square, "png") + ) == "abc" + end + + test "is atomic: concurrent writers produce no half-written file and no leftover temps", + %{root: root} do + payloads = + for i <- 0..49 do + :crypto.strong_rand_bytes(16_384) <> <> + end + + payloads + |> Enum.map(fn bytes -> + Task.async(fn -> + IconCache.write!(:npm, "concurrent", :square, "png", bytes) + end) + end) + |> Task.await_many(10_000) + + final_path = IconCache.path(:npm, "concurrent", :square, "png") + final = File.read!(final_path) + + assert final in payloads, + "final file does not match any written payload — write was not atomic" + + dir = Path.dirname(final_path) + + assert dir |> File.ls!() |> Enum.reject(&(&1 == "square.png")) == [], + "leftover temp files in #{dir}: #{inspect(File.ls!(dir))}" + + _ = root + end + end +end diff --git a/test/lightning/adaptors/invalidator_test.exs b/test/lightning/adaptors/invalidator_test.exs new file mode 100644 index 00000000000..4ac7feac8d1 --- /dev/null +++ b/test/lightning/adaptors/invalidator_test.exs @@ -0,0 +1,122 @@ +defmodule Lightning.Adaptors.InvalidatorTest do + use ExUnit.Case, async: true + + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + setup do + sup = :"inv_test_#{System.unique_integer([:positive])}" + + # The supervisor's :rest_for_one child list starts the Invalidator + # automatically — registered under `invalidator_name(sup)`. + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.StrategyMock} + ) + + cache = AdaptorsSupervisor.cache_name(sup) + inv_name = AdaptorsSupervisor.invalidator_name(sup) + + {:ok, sup: sup, cache: cache, inv_name: inv_name} + end + + describe "start_link/1" do + test "registers under the :name opt", %{inv_name: inv_name} do + assert is_pid(Process.whereis(inv_name)) + end + end + + describe "handle_info/2 - {:changed, name, source}" do + test "evicts all four matching cache keys on broadcast", %{ + sup: sup, + cache: cache, + inv_name: inv_name + } do + source = AdaptorsSupervisor.source(sup) + source_topic = AdaptorsSupervisor.source_topic(sup) + name = "@openfn/language-http" + + Cachex.put!(cache, {:schema, name, source}, {:ok, %{"type" => "object"}}) + Cachex.put!(cache, {:versions, name, source}, {:ok, [%{version: "1.0.0"}]}) + + Cachex.put!( + cache, + {:icon_meta, name, source}, + {:ok, %{icon_square_ext: "svg"}} + ) + + Cachex.put!(cache, {:packages, source}, {:ok, [%{name: name}]}) + + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, name, source} + ) + + :sys.get_state(inv_name) + + assert {:ok, nil} = Cachex.get(cache, {:schema, name, source}) + assert {:ok, nil} = Cachex.get(cache, {:versions, name, source}) + assert {:ok, nil} = Cachex.get(cache, {:icon_meta, name, source}) + assert {:ok, nil} = Cachex.get(cache, {:packages, source}) + end + + test "does not evict name-scoped keys for a different adaptor", %{ + sup: sup, + cache: cache, + inv_name: inv_name + } do + source = AdaptorsSupervisor.source(sup) + source_topic = AdaptorsSupervisor.source_topic(sup) + + target = "@openfn/language-http" + bystander = "@openfn/language-dhis2" + + Cachex.put!( + cache, + {:schema, bystander, source}, + {:ok, %{"type" => "object"}} + ) + + Cachex.put!(cache, {:versions, bystander, source}, {:ok, []}) + Cachex.put!(cache, {:icon_meta, bystander, source}, {:ok, %{}}) + + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, target, source} + ) + + :sys.get_state(inv_name) + + assert {:ok, {:ok, _}} = Cachex.get(cache, {:schema, bystander, source}) + assert {:ok, {:ok, _}} = Cachex.get(cache, {:versions, bystander, source}) + assert {:ok, {:ok, _}} = Cachex.get(cache, {:icon_meta, bystander, source}) + end + + test "{:changed, _, :local} on an npm-mode node is harmless", %{ + sup: sup, + cache: cache, + inv_name: inv_name + } do + source_topic = AdaptorsSupervisor.source_topic(sup) + name = "@openfn/language-http" + npm_source = AdaptorsSupervisor.source(sup) + + Cachex.put!( + cache, + {:schema, name, npm_source}, + {:ok, %{"type" => "object"}} + ) + + Phoenix.PubSub.broadcast!( + Lightning.PubSub, + source_topic, + {:changed, name, :local} + ) + + :sys.get_state(inv_name) + + assert is_pid(Process.whereis(inv_name)), "invalidator must still be alive" + assert {:ok, {:ok, _}} = Cachex.get(cache, {:schema, name, npm_source}) + end + end +end diff --git a/test/lightning/adaptors/local_test.exs b/test/lightning/adaptors/local_test.exs new file mode 100644 index 00000000000..1fc69bb4e91 --- /dev/null +++ b/test/lightning/adaptors/local_test.exs @@ -0,0 +1,360 @@ +defmodule Lightning.Adaptors.LocalTest do + use ExUnit.Case, async: false + + import ExUnit.CaptureLog + + alias Lightning.Adaptors.Local + + setup do + root = + Path.join( + System.tmp_dir!(), + "lightning_adaptors_local_test_#{System.unique_integer([:positive])}" + ) + + File.mkdir_p!(Path.join(root, "packages")) + + original = Application.get_env(:lightning, Local, :__unset__) + Application.put_env(:lightning, Local, path: root) + + on_exit(fn -> + case original do + :__unset__ -> Application.delete_env(:lightning, Local) + value -> Application.put_env(:lightning, Local, value) + end + + File.rm_rf!(root) + end) + + {:ok, root: root} + end + + describe "list_adaptors/0" do + test "returns {:ok, []} when there are no package directories" do + assert Local.list_adaptors() == {:ok, []} + end + + test "returns name + latest_version for each package", %{root: root} do + write_package!(root, "language-http", "@openfn/language-http", "2.1.0") + + write_package!( + root, + "language-salesforce", + "@openfn/language-salesforce", + "4.6.3" + ) + + {:ok, listing} = Local.list_adaptors() + + assert Enum.sort_by(listing, & &1.name) == [ + %{name: "@openfn/language-http", latest_version: "2.1.0"}, + %{name: "@openfn/language-salesforce", latest_version: "4.6.3"} + ] + end + + test "collapses multiple directories sharing a name into one record with the highest semver as latest_version", + %{root: root} do + write_package!(root, "http-1", "@openfn/language-http", "1.0.0") + write_package!(root, "http-2", "@openfn/language-http", "2.3.4") + write_package!(root, "http-3", "@openfn/language-http", "2.3.1") + + assert {:ok, [%{name: "@openfn/language-http", latest_version: "2.3.4"}]} = + Local.list_adaptors() + end + + test "skips a directory with a missing package.json and logs a warning", + %{root: root} do + write_package!(root, "good", "@openfn/language-good", "1.0.0") + File.mkdir_p!(Path.join([root, "packages", "broken"])) + + {result, log} = with_log(fn -> Local.list_adaptors() end) + + assert {:ok, [%{name: "@openfn/language-good"}]} = result + assert log =~ "skipping" + assert log =~ "broken" + end + + test "skips a directory with unparseable JSON and logs a warning", + %{root: root} do + bad_dir = Path.join([root, "packages", "junk"]) + File.mkdir_p!(bad_dir) + File.write!(Path.join(bad_dir, "package.json"), "{not json") + + {result, log} = with_log(fn -> Local.list_adaptors() end) + + assert {:ok, []} = result + assert log =~ "skipping" + assert log =~ "junk" + end + + test "skips package.json that has no name or version", %{root: root} do + dir = Path.join([root, "packages", "incomplete"]) + File.mkdir_p!(dir) + + File.write!( + Path.join(dir, "package.json"), + Jason.encode!(%{"name" => "x"}) + ) + + {result, _log} = with_log(fn -> Local.list_adaptors() end) + + assert {:ok, []} = result + end + + test "returns {:error, :no_repo_path} when :path is unset" do + Application.delete_env(:lightning, Local) + + assert capture_log(fn -> + assert Local.list_adaptors() == {:error, :no_repo_path} + end) =~ "not configured" + end + end + + describe "fetch_adaptor/1" do + test "decodes a realistic on-disk package into the full adaptor_record shape", + %{root: root} do + pkg = %{ + "name" => "@openfn/language-http", + "version" => "2.1.0", + "description" => "HTTP adaptor", + "homepage" => "https://docs.openfn.org/adaptors/http", + "repository" => %{"url" => "git+https://github.com/OpenFn/adaptors.git"}, + "license" => "LGPL-3.0", + "dependencies" => %{"axios" => "^1.5.0"}, + "peerDependencies" => %{"@openfn/language-common" => "^2.0.0"} + } + + schema = %{"type" => "object", "properties" => %{"baseUrl" => %{}}} + + dir = write_package_raw!(root, "language-http", pkg) + + File.write!( + Path.join(dir, "configuration-schema.json"), + Jason.encode!(schema) + ) + + write_icon!(dir, :square, "png", "square-bytes") + write_icon!(dir, :rectangle, "svg", "") + + {:ok, record} = Local.fetch_adaptor("@openfn/language-http") + + assert record.name == "@openfn/language-http" + assert record.description == "HTTP adaptor" + assert record.homepage == "https://docs.openfn.org/adaptors/http" + assert record.repository == "git+https://github.com/OpenFn/adaptors.git" + assert record.license == "LGPL-3.0" + assert record.latest_version == "2.1.0" + assert record.deprecated == false + assert record.schema_data == Jason.encode!(schema) + + assert record.schema_sha256 == + :crypto.hash(:sha256, Jason.encode!(schema)) + |> Base.encode16(case: :lower) + + refute Map.has_key?(record, :icon_square_ext), + "fetch_adaptor/1 no longer carries icon fields — the Scheduler joins them" + + refute Map.has_key?(record, :icon_rectangle_ext) + refute Map.has_key?(record, :icon_square_sha256) + refute Map.has_key?(record, :icon_rectangle_sha256) + + refute Map.has_key?(record, :source), + "the strategy must not stamp :source — the Store owns that field" + + assert [version] = record.versions + assert version.version == "2.1.0" + assert version.dependencies == %{"axios" => "^1.5.0"} + + assert version.peer_dependencies == %{ + "@openfn/language-common" => "^2.0.0" + } + + assert version.integrity == nil + assert version.tarball_url == nil + assert version.size_bytes == nil + assert version.published_at == nil + assert version.deprecated == false + end + + test "reads schema_data from the latest version's directory specifically", + %{root: root} do + old_dir = + write_package_raw!(root, "http-old", %{ + "name" => "@openfn/language-http", + "version" => "1.0.0" + }) + + new_dir = + write_package_raw!(root, "http-new", %{ + "name" => "@openfn/language-http", + "version" => "2.0.0" + }) + + File.write!( + Path.join(old_dir, "configuration-schema.json"), + Jason.encode!(%{"version" => "old"}) + ) + + File.write!( + Path.join(new_dir, "configuration-schema.json"), + Jason.encode!(%{"version" => "new"}) + ) + + {:ok, record} = Local.fetch_adaptor("@openfn/language-http") + + assert record.schema_data == Jason.encode!(%{"version" => "new"}) + assert record.latest_version == "2.0.0" + end + + test "returns nil-shaped schema fields when files are absent", + %{root: root} do + write_package!(root, "bare", "@openfn/language-bare", "1.0.0") + + {:ok, record} = Local.fetch_adaptor("@openfn/language-bare") + + assert record.schema_data == nil + assert record.schema_sha256 == nil + end + + test "handles a plain-string repository field", %{root: root} do + write_package_raw!(root, "p", %{ + "name" => "@openfn/language-p", + "version" => "1.0.0", + "repository" => "https://github.com/example/p" + }) + + {:ok, record} = Local.fetch_adaptor("@openfn/language-p") + assert record.repository == "https://github.com/example/p" + end + + test "lists every on-disk version in :versions", %{root: root} do + write_package!(root, "http-1", "@openfn/language-http", "1.0.0") + write_package!(root, "http-2", "@openfn/language-http", "2.3.4") + write_package!(root, "http-3", "@openfn/language-http", "2.3.1") + + {:ok, record} = Local.fetch_adaptor("@openfn/language-http") + + versions = Enum.map(record.versions, & &1.version) + assert versions == ["2.3.4", "2.3.1", "1.0.0"] + end + + test "returns {:error, :not_found} for an unknown package", %{root: root} do + write_package!(root, "p", "@openfn/language-p", "1.0.0") + + assert Local.fetch_adaptor("@openfn/language-missing") == + {:error, :not_found} + end + end + + describe "fetch_icon/2" do + test "reads an icon from the latest version's assets dir", %{root: root} do + dir = write_package!(root, "http", "@openfn/language-http", "1.0.0") + write_icon!(dir, :square, "png", "PNGDATA") + + assert {:ok, %{data: "PNGDATA", ext: "png"}} = + Local.fetch_icon("@openfn/language-http", :square) + end + + test "prefers the latest version when multiple version dirs exist", + %{root: root} do + old_dir = + write_package_raw!(root, "http-old", %{ + "name" => "@openfn/language-http", + "version" => "1.0.0" + }) + + new_dir = + write_package_raw!(root, "http-new", %{ + "name" => "@openfn/language-http", + "version" => "2.0.0" + }) + + write_icon!(old_dir, :square, "png", "OLD") + write_icon!(new_dir, :square, "png", "NEW") + + assert {:ok, %{data: "NEW", ext: "png"}} = + Local.fetch_icon("@openfn/language-http", :square) + end + + test "falls back to svg when png is absent", %{root: root} do + dir = write_package!(root, "p", "@openfn/language-p", "1.0.0") + write_icon!(dir, :rectangle, "svg", "") + + assert {:ok, %{data: "", ext: "svg"}} = + Local.fetch_icon("@openfn/language-p", :rectangle) + end + + test "returns {:error, :not_found} when no icon variant exists", + %{root: root} do + write_package!(root, "p", "@openfn/language-p", "1.0.0") + + assert Local.fetch_icon("@openfn/language-p", :square) == + {:error, :not_found} + end + + test "returns {:error, :not_found} for an unknown package", %{root: root} do + write_package!(root, "p", "@openfn/language-p", "1.0.0") + + assert Local.fetch_icon("@openfn/language-missing", :square) == + {:error, :not_found} + end + end + + describe "fetch_icons/1" do + test "returns an entry per package per shape including sha256", + %{root: root} do + http = write_package!(root, "http", "@openfn/language-http", "1.0.0") + sf = write_package!(root, "sf", "@openfn/language-salesforce", "2.0.0") + + write_icon!(http, :square, "png", "HTTP_SQ") + write_icon!(http, :rectangle, "svg", "") + write_icon!(sf, :square, "png", "SF_SQ") + + {:ok, map} = Local.fetch_icons([]) + + assert %{ + "@openfn/language-http" => %{ + square: %{data: "HTTP_SQ", ext: "png", sha256: http_sq_sha}, + rectangle: %{data: "", ext: "svg"} + }, + "@openfn/language-salesforce" => %{ + square: %{data: "SF_SQ", ext: "png"} + } + } = map + + assert http_sq_sha == :crypto.hash(:sha256, "HTTP_SQ") + refute Map.has_key?(map["@openfn/language-salesforce"], :rectangle) + end + + test "returns {:ok, %{}} when no packages have icons", %{root: root} do + write_package!(root, "bare", "@openfn/language-bare", "1.0.0") + + assert {:ok, %{}} = Local.fetch_icons([]) + end + + test "returns {:error, :no_repo_path} when :path is unset" do + Application.delete_env(:lightning, Local) + + assert capture_log(fn -> + assert Local.fetch_icons([]) == {:error, :no_repo_path} + end) =~ "not configured" + end + end + + defp write_package!(root, dir_name, name, version) do + write_package_raw!(root, dir_name, %{"name" => name, "version" => version}) + end + + defp write_package_raw!(root, dir_name, package_json) do + dir = Path.join([root, "packages", dir_name]) + File.mkdir_p!(dir) + File.write!(Path.join(dir, "package.json"), Jason.encode!(package_json)) + dir + end + + defp write_icon!(dir, shape, ext, bytes) do + assets = Path.join(dir, "assets") + File.mkdir_p!(assets) + File.write!(Path.join(assets, "#{shape}.#{ext}"), bytes) + end +end diff --git a/test/lightning/adaptors/node_monitor_test.exs b/test/lightning/adaptors/node_monitor_test.exs new file mode 100644 index 00000000000..c0b212550c0 --- /dev/null +++ b/test/lightning/adaptors/node_monitor_test.exs @@ -0,0 +1,160 @@ +defmodule Lightning.Adaptors.NodeMonitorTest do + use Lightning.DataCase, async: true + + import Mox + + alias Lightning.Adaptors.Repo, as: AdaptorsRepo + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + setup :verify_on_exit! + + setup do + sup = :"nm_test_#{System.unique_integer([:positive])}" + + # The supervisor's :rest_for_one child list starts the NodeMonitor + # automatically — registered under `node_monitor_name(sup)`. + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.StrategyMock} + ) + + cache = AdaptorsSupervisor.cache_name(sup) + nm_name = AdaptorsSupervisor.node_monitor_name(sup) + + nm_pid = Process.whereis(nm_name) + Ecto.Adapters.SQL.Sandbox.allow(Lightning.Repo, self(), nm_pid) + + # Scheduler is auto-started too (wrapped in HighlanderPG, registered + # via :global). It may not be up yet at setup time — HighlanderPG + # polls at 300ms — so this is best-effort. + {:global, global_sched_name} = AdaptorsSupervisor.global_scheduler_name(sup) + sched_pid = :global.whereis_name(global_sched_name) + + if is_pid(sched_pid), + do: Ecto.Adapters.SQL.Sandbox.allow(Lightning.Repo, self(), sched_pid) + + {:ok, sup: sup, cache: cache, nm_name: nm_name} + end + + describe "start_link/1" do + test "registers under the :name opt", %{nm_name: nm_name} do + assert is_pid(Process.whereis(nm_name)) + end + end + + describe "handle_info/2 - {:nodeup, ...}" do + test "warms {:packages, source} and {:icon_meta, name, source} from Postgres", + %{ + sup: sup, + cache: cache, + nm_name: nm_name + } do + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 0, fn _ -> + :unreachable + end) + + {:ok, _} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + source = AdaptorsSupervisor.source(sup) + + send(nm_name, {:nodeup, :node@host, %{node_type: :visible}}) + :sys.get_state(nm_name) + + assert {:ok, {:ok, [pkg]}} = Cachex.get(cache, {:packages, source}) + assert pkg.name == "@openfn/language-http" + + assert {:ok, {:ok, icon_meta}} = + Cachex.get(cache, {:icon_meta, "@openfn/language-http", source}) + + assert Map.has_key?(icon_meta, :icon_square_ext) + end + + test "uses put_many; pre-existing schema keys survive the warm", %{ + sup: sup, + cache: cache, + nm_name: nm_name + } do + source = AdaptorsSupervisor.source(sup) + + Cachex.put!( + cache, + {:schema, "pre-existing", source}, + {:ok, %{"kept" => true}} + ) + + {:ok, _} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + + send(nm_name, {:nodeup, :node@host, %{node_type: :visible}}) + :sys.get_state(nm_name) + + assert {:ok, {:ok, %{"kept" => true}}} = + Cachex.get(cache, {:schema, "pre-existing", source}) + end + + test "warm covers only the active source; :local keys are not materialised in npm mode", + %{ + cache: cache, + nm_name: nm_name + } do + {:ok, _} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + + send(nm_name, {:nodeup, :node@host, %{node_type: :visible}}) + :sys.get_state(nm_name) + + assert {:ok, nil} = Cachex.get(cache, {:packages, :local}) + end + end + + describe "handle_info/2 - {:nodedown, ...}" do + test "nodedown is a no-op; cache and state are unchanged", %{ + sup: sup, + cache: cache, + nm_name: nm_name + } do + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 0, fn _ -> + :unreachable + end) + + source = AdaptorsSupervisor.source(sup) + Cachex.put!(cache, {:packages, source}, {:ok, [%{name: "sentinel"}]}) + + send(nm_name, {:nodedown, :node@host, %{node_type: :visible}}) + :sys.get_state(nm_name) + + assert {:ok, {:ok, [%{name: "sentinel"}]}} = + Cachex.get(cache, {:packages, source}) + end + end + + defp adaptor_record(overrides \\ []) do + overrides = Map.new(overrides) + + %{ + name: "@openfn/language-http", + source: :npm, + latest_version: "1.0.0", + description: "HTTP adaptor", + homepage: nil, + repository: nil, + license: "LGPL-3.0", + deprecated: false, + schema_data: nil, + schema_sha256: nil, + icon_square_ext: nil, + icon_rectangle_ext: nil, + icon_square_sha256: nil, + icon_rectangle_sha256: nil, + versions: [ + %{ + version: "1.0.0", + integrity: "sha512-1.0.0", + tarball_url: "https://example.com/x/-/x-1.0.0.tgz", + size_bytes: 1024, + dependencies: %{}, + peer_dependencies: %{}, + published_at: nil, + deprecated: false + } + ] + } + |> Map.merge(overrides) + end +end diff --git a/test/lightning/adaptors/npm/github_test.exs b/test/lightning/adaptors/npm/github_test.exs new file mode 100644 index 00000000000..3f09b0ae966 --- /dev/null +++ b/test/lightning/adaptors/npm/github_test.exs @@ -0,0 +1,390 @@ +defmodule Lightning.Adaptors.NPM.GitHubTest do + use ExUnit.Case, async: false + + alias Lightning.Adaptors.NPM.GitHub + + setup do + bypass = Bypass.open() + + Application.put_env(:lightning, Lightning.Adaptors.NPM, + github_url: "http://localhost:#{bypass.port}", + github_ref: "main", + http_timeout: 1_000 + ) + + prev_adapter = Application.get_env(:tesla, :adapter) + + Application.put_env( + :tesla, + :adapter, + {Tesla.Adapter.Finch, name: Lightning.Finch} + ) + + on_exit(fn -> + Application.delete_env(:lightning, Lightning.Adaptors.NPM) + + if prev_adapter do + Application.put_env(:tesla, :adapter, prev_adapter) + else + Application.delete_env(:tesla, :adapter) + end + end) + + %{bypass: bypass} + end + + describe "fetch_one/2" do + test "returns png bytes on a happy-path 200", %{bypass: bypass} do + Bypass.expect( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/square.png", + fn conn -> Plug.Conn.resp(conn, 200, "SQUARE_PNG_BYTES") end + ) + + assert {:ok, %{data: "SQUARE_PNG_BYTES", ext: "png"}} = + GitHub.fetch_one("@openfn/language-http", :square) + end + + test "falls back to svg when png is missing", %{bypass: bypass} do + Bypass.expect(bypass, fn conn -> + case conn.request_path do + "/OpenFn/adaptors/main/packages/http/assets/rectangle.png" -> + Plug.Conn.resp(conn, 404, "") + + "/OpenFn/adaptors/main/packages/http/assets/rectangle.svg" -> + Plug.Conn.resp(conn, 200, "") + + path -> + raise "unexpected path: #{path}" + end + end) + + assert {:ok, %{data: "", ext: "svg"}} = + GitHub.fetch_one("@openfn/language-http", :rectangle) + end + + test "returns {:error, :not_found} when both exts 404", %{bypass: bypass} do + Bypass.expect(bypass, fn conn -> Plug.Conn.resp(conn, 404, "") end) + + assert {:error, :not_found} = + GitHub.fetch_one("@openfn/language-missing", :square) + end + + test "surfaces 5xx as {:error, {:http_status, status}}", %{bypass: bypass} do + Bypass.expect(bypass, fn conn -> Plug.Conn.resp(conn, 503, "") end) + + assert {:error, {:http_status, 503}} = + GitHub.fetch_one("@openfn/language-http", :square) + end + + test "surfaces network failure as {:error, _}", %{bypass: bypass} do + Bypass.down(bypass) + + assert {:error, _reason} = + GitHub.fetch_one("@openfn/language-http", :square) + end + + test "surfaces :http_timeout expiry as {:error, _}", %{bypass: bypass} do + Bypass.expect(bypass, fn conn -> + Bypass.pass(bypass) + Process.sleep(1_500) + Plug.Conn.resp(conn, 200, "should not arrive") + end) + + assert {:error, _reason} = + GitHub.fetch_one("@openfn/language-http", :square) + end + + test "strips the @openfn/language- prefix from the URL path", %{ + bypass: bypass + } do + Bypass.expect( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/salesforce/assets/square.png", + fn conn -> Plug.Conn.resp(conn, 200, "OK") end + ) + + assert {:ok, %{ext: "png"}} = + GitHub.fetch_one("@openfn/language-salesforce", :square) + end + + test "honours the configured :github_ref", %{bypass: bypass} do + Application.put_env(:lightning, Lightning.Adaptors.NPM, + github_url: "http://localhost:#{bypass.port}", + github_ref: "v2", + http_timeout: 1_000 + ) + + Bypass.expect( + bypass, + "GET", + "/OpenFn/adaptors/v2/packages/http/assets/square.png", + fn conn -> Plug.Conn.resp(conn, 200, "REF_BYTES") end + ) + + assert {:ok, %{data: "REF_BYTES"}} = + GitHub.fetch_one("@openfn/language-http", :square) + end + end + + describe "fetch_all/2" do + test "returns an entry per package per shape including sha256", %{ + bypass: bypass + } do + Bypass.expect(bypass, fn conn -> + case conn.request_path do + "/OpenFn/adaptors/main/packages/http/assets/square.png" -> + Plug.Conn.resp(conn, 200, "HTTP_SQ") + + "/OpenFn/adaptors/main/packages/http/assets/rectangle.png" -> + Plug.Conn.resp(conn, 200, "HTTP_RECT") + + "/OpenFn/adaptors/main/packages/salesforce/assets/square.png" -> + Plug.Conn.resp(conn, 200, "SF_SQ") + + "/OpenFn/adaptors/main/packages/salesforce/assets/rectangle.png" -> + Plug.Conn.resp(conn, 200, "SF_RECT") + + _ -> + Plug.Conn.resp(conn, 404, "") + end + end) + + {:ok, map} = + GitHub.fetch_all( + [ + "@openfn/language-http", + "@openfn/language-salesforce" + ], + %{} + ) + + assert %{ + "@openfn/language-http" => %{ + square: %{data: "HTTP_SQ", ext: "png", sha256: http_sq_sha}, + rectangle: %{data: "HTTP_RECT", ext: "png"} + }, + "@openfn/language-salesforce" => %{ + square: %{data: "SF_SQ", ext: "png"}, + rectangle: %{data: "SF_RECT", ext: "png"} + } + } = map + + assert http_sq_sha == :crypto.hash(:sha256, "HTTP_SQ") + end + + test "absent packages and missing shapes are simply absent from the map", + %{bypass: bypass} do + Bypass.expect(bypass, fn conn -> + case conn.request_path do + "/OpenFn/adaptors/main/packages/http/assets/square.png" -> + Plug.Conn.resp(conn, 200, "ONLY_SQ") + + _ -> + Plug.Conn.resp(conn, 404, "") + end + end) + + {:ok, map} = + GitHub.fetch_all( + [ + "@openfn/language-http", + "@openfn/language-missing" + ], + %{} + ) + + assert Map.keys(map) == ["@openfn/language-http"] + assert Map.keys(map["@openfn/language-http"]) == [:square] + end + + test "returns {:ok, %{}} when the upstream is unreachable", %{ + bypass: bypass + } do + Bypass.down(bypass) + + assert {:ok, map} = + GitHub.fetch_all( + [ + "@openfn/language-http", + "@openfn/language-salesforce" + ], + %{} + ) + + assert map == %{} + end + + test "sends If-None-Match when a prior etag is supplied", %{bypass: bypass} do + etag = ~s(W/"abc") + + Bypass.expect( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/square.png", + fn conn -> + assert Plug.Conn.get_req_header(conn, "if-none-match") == [etag] + Plug.Conn.resp(conn, 304, "") + end + ) + + Bypass.stub( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/rectangle.png", + fn conn -> + Plug.Conn.resp(conn, 404, "") + end + ) + + Bypass.stub( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/rectangle.svg", + fn conn -> + Plug.Conn.resp(conn, 404, "") + end + ) + + {:ok, map} = + GitHub.fetch_all( + ["@openfn/language-http"], + %{"@openfn/language-http" => %{square: etag}} + ) + + assert get_in(map, ["@openfn/language-http", :square]) == :not_modified + end + + test "sends no If-None-Match when no prior etag", %{bypass: bypass} do + Bypass.expect( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/square.png", + fn conn -> + assert Plug.Conn.get_req_header(conn, "if-none-match") == [] + + conn + |> Plug.Conn.put_resp_header("etag", "test-etag-abc") + |> Plug.Conn.resp(200, "SQ_BYTES") + end + ) + + Bypass.stub( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/rectangle.png", + fn conn -> + Plug.Conn.resp(conn, 404, "") + end + ) + + Bypass.stub( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/rectangle.svg", + fn conn -> + Plug.Conn.resp(conn, 404, "") + end + ) + + {:ok, map} = GitHub.fetch_all(["@openfn/language-http"], %{}) + + assert %{ + data: "SQ_BYTES", + ext: "png", + etag: "test-etag-abc" + } = get_in(map, ["@openfn/language-http", :square]) + end + + test "304 short-circuits with the :not_modified sentinel in the slot", %{ + bypass: bypass + } do + etag = ~s("xyz") + + Bypass.expect( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/square.png", + fn conn -> + assert Plug.Conn.get_req_header(conn, "if-none-match") == [etag] + Plug.Conn.resp(conn, 304, "") + end + ) + + Bypass.stub( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/rectangle.png", + fn conn -> + Plug.Conn.resp(conn, 404, "") + end + ) + + Bypass.stub( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/rectangle.svg", + fn conn -> + Plug.Conn.resp(conn, 404, "") + end + ) + + {:ok, map} = + GitHub.fetch_all( + ["@openfn/language-http"], + %{"@openfn/language-http" => %{square: etag}} + ) + + # explicit sentinel — distinct from "absent" (which would mean upstream + # had no such shape at all). + assert map["@openfn/language-http"][:square] == :not_modified + end + + test "200 with a new etag overrides the prior", %{bypass: bypass} do + prior = ~s("old") + fresh = ~s("new") + + Bypass.expect( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/square.png", + fn conn -> + assert Plug.Conn.get_req_header(conn, "if-none-match") == [prior] + + conn + |> Plug.Conn.put_resp_header("etag", fresh) + |> Plug.Conn.resp(200, "FRESH") + end + ) + + Bypass.stub( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/rectangle.png", + fn conn -> + Plug.Conn.resp(conn, 404, "") + end + ) + + Bypass.stub( + bypass, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/rectangle.svg", + fn conn -> + Plug.Conn.resp(conn, 404, "") + end + ) + + {:ok, map} = + GitHub.fetch_all( + ["@openfn/language-http"], + %{"@openfn/language-http" => %{square: prior}} + ) + + assert %{data: "FRESH", ext: "png", etag: ^fresh} = + map["@openfn/language-http"][:square] + end + end +end diff --git a/test/lightning/adaptors/npm/registry_test.exs b/test/lightning/adaptors/npm/registry_test.exs new file mode 100644 index 00000000000..760b4891264 --- /dev/null +++ b/test/lightning/adaptors/npm/registry_test.exs @@ -0,0 +1,199 @@ +defmodule Lightning.Adaptors.NPM.RegistryTest do + use ExUnit.Case, async: false + + alias Lightning.Adaptors.NPM.Registry + + setup do + bypass = Bypass.open() + + Application.put_env(:lightning, Lightning.Adaptors.NPM, + registry_url: "http://localhost:#{bypass.port}", + http_timeout: 1_000 + ) + + # Per-test Tesla adapter override — config/test.exs globally pins + # `Lightning.Tesla.Mock`, but we need the real Finch adapter so that + # Bypass actually receives requests over a socket. + prev_adapter = Application.get_env(:tesla, :adapter) + + Application.put_env( + :tesla, + :adapter, + {Tesla.Adapter.Finch, name: Lightning.Finch} + ) + + on_exit(fn -> + Application.delete_env(:lightning, Lightning.Adaptors.NPM) + + if prev_adapter do + Application.put_env(:tesla, :adapter, prev_adapter) + else + Application.delete_env(:tesla, :adapter) + end + end) + + %{bypass: bypass} + end + + describe "list_adaptors/0" do + test "returns an empty list when the search has no results", %{ + bypass: bypass + } do + Bypass.expect(bypass, "GET", "/-/v1/search", fn conn -> + conn = Plug.Conn.fetch_query_params(conn) + assert conn.query_params["text"] == "@openfn" + assert conn.query_params["size"] == "250" + + json_resp(conn, 200, %{"objects" => []}) + end) + + assert {:ok, []} = Registry.list_adaptors() + end + + test "returns name + latest_version for each search hit", %{bypass: bypass} do + Bypass.expect(bypass, "GET", "/-/v1/search", fn conn -> + body = %{ + "objects" => [ + %{ + "package" => %{ + "name" => "@openfn/language-http", + "version" => "2.1.0" + } + }, + %{ + "package" => %{ + "name" => "@openfn/language-salesforce", + "version" => "4.6.3" + } + } + ] + } + + json_resp(conn, 200, body) + end) + + {:ok, listing} = Registry.list_adaptors() + + assert Enum.sort_by(listing, & &1.name) == [ + %{name: "@openfn/language-http", latest_version: "2.1.0"}, + %{name: "@openfn/language-salesforce", latest_version: "4.6.3"} + ] + end + + test "filters out @openfn/* packages that aren't language-* adaptors and other scopes", + %{bypass: bypass} do + Bypass.expect(bypass, "GET", "/-/v1/search", fn conn -> + body = %{ + "objects" => [ + %{ + "package" => %{ + "name" => "@openfn/language-http", + "version" => "1.0.0" + } + }, + %{ + "package" => %{"name" => "@openfn/cli", "version" => "1.2.3"} + }, + %{ + "package" => %{ + "name" => "@openfn/buildtools", + "version" => "0.9.0" + } + }, + %{ + "package" => %{ + "name" => "@sid-indonesia/language-http", + "version" => "2.0.0" + } + }, + %{ + "package" => %{ + "name" => "language-template", + "version" => "3.0.0" + } + } + ] + } + + json_resp(conn, 200, body) + end) + + assert {:ok, [%{name: "@openfn/language-http", latest_version: "1.0.0"}]} = + Registry.list_adaptors() + end + + test "skips malformed entries that lack name or version", %{bypass: bypass} do + Bypass.expect(bypass, "GET", "/-/v1/search", fn conn -> + body = %{ + "objects" => [ + %{ + "package" => %{ + "name" => "@openfn/language-http", + "version" => "1.0.0" + } + }, + %{"package" => %{"name" => "@openfn/language-no-version"}}, + %{"score" => %{"final" => 0.5}} + ] + } + + json_resp(conn, 200, body) + end) + + assert {:ok, [%{name: "@openfn/language-http", latest_version: "1.0.0"}]} = + Registry.list_adaptors() + end + + test "surfaces 5xx responses as {:error, _}", %{bypass: bypass} do + Bypass.expect(bypass, "GET", "/-/v1/search", fn conn -> + Plug.Conn.resp(conn, 503, "") + end) + + assert {:error, {:http_status, 503}} = Registry.list_adaptors() + end + + test "surfaces network failure as {:error, _}", %{bypass: bypass} do + Bypass.down(bypass) + assert {:error, _reason} = Registry.list_adaptors() + end + end + + describe "get_packument/1" do + test "returns the decoded map on 200", %{bypass: bypass} do + packument = %{ + "name" => "@openfn/language-http", + "dist-tags" => %{"latest" => "2.1.0"} + } + + Bypass.expect(bypass, "GET", "/@openfn/language-http", fn conn -> + json_resp(conn, 200, packument) + end) + + assert {:ok, ^packument} = Registry.get_packument("@openfn/language-http") + end + + test "returns {:error, :not_found} on 404", %{bypass: bypass} do + Bypass.expect(bypass, "GET", "/@openfn/language-missing", fn conn -> + Plug.Conn.resp(conn, 404, "") + end) + + assert {:error, :not_found} = + Registry.get_packument("@openfn/language-missing") + end + + test "surfaces 5xx as {:error, {:http_status, status}}", %{bypass: bypass} do + Bypass.expect(bypass, "GET", "/@openfn/language-http", fn conn -> + Plug.Conn.resp(conn, 502, "") + end) + + assert {:error, {:http_status, 502}} = + Registry.get_packument("@openfn/language-http") + end + end + + defp json_resp(conn, status, body) do + conn + |> Plug.Conn.put_resp_content_type("application/json") + |> Plug.Conn.resp(status, Jason.encode!(body)) + end +end diff --git a/test/lightning/adaptors/npm/schema_test.exs b/test/lightning/adaptors/npm/schema_test.exs new file mode 100644 index 00000000000..958fd953a2c --- /dev/null +++ b/test/lightning/adaptors/npm/schema_test.exs @@ -0,0 +1,86 @@ +defmodule Lightning.Adaptors.NPM.SchemaTest do + use ExUnit.Case, async: false + + alias Lightning.Adaptors.NPM.Schema + + @package "@openfn/language-http" + @version "2.1.0" + @path "/npm/#{@package}@#{@version}/configuration-schema.json" + + setup do + bypass = Bypass.open() + + Application.put_env(:lightning, Lightning.Adaptors.NPM, + jsdelivr_url: "http://localhost:#{bypass.port}", + http_timeout: 1_000 + ) + + prev_adapter = Application.get_env(:tesla, :adapter) + + Application.put_env( + :tesla, + :adapter, + {Tesla.Adapter.Finch, name: Lightning.Finch} + ) + + on_exit(fn -> + Application.delete_env(:lightning, Lightning.Adaptors.NPM) + + if prev_adapter do + Application.put_env(:tesla, :adapter, prev_adapter) + else + Application.delete_env(:tesla, :adapter) + end + end) + + %{bypass: bypass} + end + + describe "schema/2" do + test "returns the decoded schema and a hex sha256 on 200", %{bypass: bypass} do + schema = %{"type" => "object", "properties" => %{"baseUrl" => %{}}} + body = Jason.encode!(schema) + + expected_sha = + :sha256 + |> :crypto.hash(body) + |> Base.encode16(case: :lower) + + Bypass.expect(bypass, "GET", @path, fn conn -> + assert conn.request_path == @path + Plug.Conn.resp(conn, 200, body) + end) + + assert {^schema, ^expected_sha} = Schema.schema(@package, @version) + end + + test "returns {nil, nil} on 404", %{bypass: bypass} do + Bypass.expect(bypass, "GET", @path, fn conn -> + Plug.Conn.resp(conn, 404, "") + end) + + assert {nil, nil} = Schema.schema(@package, @version) + end + + test "returns {nil, nil} on 5xx", %{bypass: bypass} do + Bypass.expect(bypass, "GET", @path, fn conn -> + Plug.Conn.resp(conn, 500, "") + end) + + assert {nil, nil} = Schema.schema(@package, @version) + end + + test "returns {nil, nil} on invalid JSON body", %{bypass: bypass} do + Bypass.expect(bypass, "GET", @path, fn conn -> + Plug.Conn.resp(conn, 200, "this is not json {") + end) + + assert {nil, nil} = Schema.schema(@package, @version) + end + + test "returns {nil, nil} on connection refused", %{bypass: bypass} do + Bypass.down(bypass) + assert {nil, nil} = Schema.schema(@package, @version) + end + end +end diff --git a/test/lightning/adaptors/npm_test.exs b/test/lightning/adaptors/npm_test.exs new file mode 100644 index 00000000000..ad2ea8ded67 --- /dev/null +++ b/test/lightning/adaptors/npm_test.exs @@ -0,0 +1,270 @@ +defmodule Lightning.Adaptors.NPMTest do + use ExUnit.Case, async: false + + alias Lightning.Adaptors.NPM + + @package "@openfn/language-http" + @latest_version "2.1.0" + + # Three Bypass servers: one for the npm registry, one for jsDelivr, + # one for raw.githubusercontent.com. Per-test config installs all three + # URLs onto the strategy_opts block. + setup do + registry = Bypass.open() + jsdelivr = Bypass.open() + github = Bypass.open() + + Application.put_env(:lightning, Lightning.Adaptors.NPM, + registry_url: "http://localhost:#{registry.port}", + jsdelivr_url: "http://localhost:#{jsdelivr.port}", + github_url: "http://localhost:#{github.port}", + github_ref: "main", + http_timeout: 1_000 + ) + + prev_adapter = Application.get_env(:tesla, :adapter) + + Application.put_env( + :tesla, + :adapter, + {Tesla.Adapter.Finch, name: Lightning.Finch} + ) + + on_exit(fn -> + Application.delete_env(:lightning, Lightning.Adaptors.NPM) + + if prev_adapter do + Application.put_env(:tesla, :adapter, prev_adapter) + else + Application.delete_env(:tesla, :adapter) + end + end) + + %{registry: registry, jsdelivr: jsdelivr, github: github} + end + + describe "fetch_adaptor/1" do + test "decodes a packument into the icon-free adaptor_record shape", %{ + registry: registry, + jsdelivr: jsdelivr + } do + schema = %{"type" => "object", "properties" => %{"baseUrl" => %{}}} + schema_bytes = Jason.encode!(schema) + packument = build_packument() + + Bypass.expect(registry, "GET", "/" <> @package, fn conn -> + json_resp(conn, 200, packument) + end) + + Bypass.expect( + jsdelivr, + "GET", + "/npm/#{@package}@#{@latest_version}/configuration-schema.json", + fn conn -> Plug.Conn.resp(conn, 200, schema_bytes) end + ) + + {:ok, record} = NPM.fetch_adaptor(@package) + + expected_schema_sha = + :sha256 |> :crypto.hash(schema_bytes) |> Base.encode16(case: :lower) + + assert %{ + name: @package, + description: "HTTP adaptor", + homepage: "https://docs.openfn.org/adaptors/http", + repository: "git+https://github.com/OpenFn/adaptors.git", + license: "LGPL-3.0", + latest_version: @latest_version, + deprecated: false, + schema_data: ^schema_bytes, + schema_sha256: ^expected_schema_sha + } = record + + refute Map.has_key?(record, :icon_square_ext), + "fetch_adaptor/1 no longer carries icon fields — the Scheduler joins them" + + refute Map.has_key?(record, :icon_rectangle_ext) + refute Map.has_key?(record, :icon_square_sha256) + refute Map.has_key?(record, :icon_rectangle_sha256) + + refute Map.has_key?(record, :source), + "strategy must not stamp :source — the Store owns that field" + + assert length(record.versions) == 2 + + latest = Enum.find(record.versions, &(&1.version == @latest_version)) + + assert %{ + integrity: "sha512-abc", + size_bytes: 12_345, + dependencies: %{"axios" => "^1.5.0"}, + peer_dependencies: %{"@openfn/language-common" => "^2.0.0"}, + deprecated: false + } = latest + + assert %DateTime{} = latest.published_at + assert DateTime.to_iso8601(latest.published_at) =~ "2024-06-01" + + old = Enum.find(record.versions, &(&1.version == "1.0.0")) + assert old.integrity == "sha512-old" + assert old.dependencies == %{} + assert old.deprecated == true + end + + test "degrades to nil schema when jsDelivr returns 5xx", %{ + registry: registry, + jsdelivr: jsdelivr + } do + packument = build_packument() + + Bypass.expect(registry, "GET", "/" <> @package, fn conn -> + json_resp(conn, 200, packument) + end) + + Bypass.expect(jsdelivr, fn conn -> + Plug.Conn.resp(conn, 500, "") + end) + + {:ok, record} = NPM.fetch_adaptor(@package) + + assert record.schema_data == nil + assert record.schema_sha256 == nil + assert record.name == @package + assert record.latest_version == @latest_version + end + end + + describe "fetch_icon/2" do + test "delegates to NPM.GitHub for raw icon bytes", %{github: github} do + Bypass.expect( + github, + "GET", + "/OpenFn/adaptors/main/packages/http/assets/square.png", + fn conn -> Plug.Conn.resp(conn, 200, "PNG_PAYLOAD") end + ) + + assert {:ok, %{data: "PNG_PAYLOAD", ext: "png"}} = + NPM.fetch_icon(@package, :square) + end + + test "returns {:error, :not_found} when both png and svg 404", %{ + github: github + } do + Bypass.expect(github, fn conn -> Plug.Conn.resp(conn, 404, "") end) + + assert {:error, :not_found} = + NPM.fetch_icon("@openfn/language-missing", :square) + end + + test "surfaces transport failure as {:error, _}", %{github: github} do + Bypass.down(github) + + assert {:error, _reason} = NPM.fetch_icon(@package, :square) + end + end + + describe "fetch_icons/1" do + test "lists adaptors then fans out to GitHub raw fetches", %{ + registry: registry, + github: github + } do + Bypass.expect(registry, "GET", "/-/v1/search", fn conn -> + body = %{ + "objects" => [ + %{ + "package" => %{ + "name" => "@openfn/language-http", + "version" => "2.1.0" + } + }, + %{ + "package" => %{ + "name" => "@openfn/language-salesforce", + "version" => "4.6.3" + } + } + ] + } + + json_resp(conn, 200, body) + end) + + Bypass.expect(github, fn conn -> + case conn.request_path do + "/OpenFn/adaptors/main/packages/http/assets/square.png" -> + Plug.Conn.resp(conn, 200, "HTTP_SQ") + + "/OpenFn/adaptors/main/packages/salesforce/assets/square.png" -> + Plug.Conn.resp(conn, 200, "SF_SQ") + + _ -> + Plug.Conn.resp(conn, 404, "") + end + end) + + {:ok, icons} = NPM.fetch_icons([]) + + assert %{ + "@openfn/language-http" => %{ + square: %{data: "HTTP_SQ", ext: "png"} + }, + "@openfn/language-salesforce" => %{ + square: %{data: "SF_SQ", ext: "png"} + } + } = icons + + assert icons["@openfn/language-http"].square.sha256 == + :crypto.hash(:sha256, "HTTP_SQ") + end + + test "surfaces list_adaptors errors as {:error, _}", %{registry: registry} do + Bypass.expect(registry, "GET", "/-/v1/search", fn conn -> + Plug.Conn.resp(conn, 503, "") + end) + + assert {:error, _} = NPM.fetch_icons([]) + end + end + + # ==================== Helpers ==================== + + defp build_packument do + %{ + "name" => @package, + "description" => "HTTP adaptor", + "homepage" => "https://docs.openfn.org/adaptors/http", + "repository" => %{"url" => "git+https://github.com/OpenFn/adaptors.git"}, + "license" => "LGPL-3.0", + "dist-tags" => %{"latest" => @latest_version}, + "time" => %{ + "1.0.0" => "2023-01-01T00:00:00.000Z", + "2.1.0" => "2024-06-01T12:00:00.000Z" + }, + "versions" => %{ + "1.0.0" => %{ + "dependencies" => %{}, + "peerDependencies" => %{}, + "deprecated" => "please upgrade", + "dist" => %{ + "integrity" => "sha512-old", + "unpackedSize" => 5_000 + } + }, + @latest_version => %{ + "dependencies" => %{"axios" => "^1.5.0"}, + "peerDependencies" => %{"@openfn/language-common" => "^2.0.0"}, + "dist" => %{ + "integrity" => "sha512-abc", + "unpackedSize" => 12_345 + } + } + } + } + end + + defp json_resp(conn, status, body) do + conn + |> Plug.Conn.put_resp_content_type("application/json") + |> Plug.Conn.resp(status, Jason.encode!(body)) + end +end diff --git a/test/lightning/adaptors/package_name_test.exs b/test/lightning/adaptors/package_name_test.exs new file mode 100644 index 00000000000..7f01aacc2b6 --- /dev/null +++ b/test/lightning/adaptors/package_name_test.exs @@ -0,0 +1,73 @@ +defmodule Lightning.Adaptors.PackageNameTest do + use Lightning.DataCase, async: false + + import Lightning.Factories + + alias Lightning.Adaptors.PackageName + + describe "parse/1" do + test "splits scoped name and semver version" do + assert PackageName.parse("@openfn/language-common@1.2.3") == + {"@openfn/language-common", "1.2.3"} + end + + test "splits unscoped name and version" do + assert PackageName.parse("foo@2.0.0") == {"foo", "2.0.0"} + end + + test "returns the name with nil version when no @version is given" do + assert PackageName.parse("@openfn/language-common") == + {"@openfn/language-common", nil} + end + + test "treats the @local literal as a version" do + assert PackageName.parse("@openfn/language-common@local") == + {"@openfn/language-common", "local"} + end + + test "treats the @latest literal as a version" do + assert PackageName.parse("@openfn/language-common@latest") == + {"@openfn/language-common", "latest"} + end + + test "returns {nil, nil} for nil input" do + assert PackageName.parse(nil) == {nil, nil} + end + + test "returns {nil, nil} for malformed input" do + assert PackageName.parse("") == {nil, nil} + end + end + + describe "to_wire/1" do + test "passes through concrete semver unchanged" do + assert PackageName.to_wire("@openfn/language-common@1.6.2") == + "@openfn/language-common@1.6.2" + end + + test "returns empty string for nil input" do + assert PackageName.to_wire(nil) == "" + end + + test "preserves @local literal regardless of source" do + assert PackageName.to_wire("@openfn/language-common@local") == + "@openfn/language-common@local" + end + + test "resolves @latest to the concrete latest_version from Adaptors.Repo" do + insert(:adaptor, + name: "@openfn/language-common", + source: :npm, + latest_version: "9.9.9" + ) + + assert PackageName.to_wire("@openfn/language-common@latest") == + "@openfn/language-common@9.9.9" + end + + test "falls back to @latest literal when adaptor is unknown" do + assert PackageName.to_wire("@openfn/never-existed@latest") == + "@openfn/never-existed@latest" + end + end +end diff --git a/test/lightning/adaptors/repo_adaptor_test.exs b/test/lightning/adaptors/repo_adaptor_test.exs new file mode 100644 index 00000000000..e77868fa6cd --- /dev/null +++ b/test/lightning/adaptors/repo_adaptor_test.exs @@ -0,0 +1,315 @@ +defmodule Lightning.Adaptors.Repo.AdaptorTest do + use ExUnit.Case, async: true + + alias Lightning.Adaptors.Repo.Adaptor + + @valid_attrs %{ + name: "@openfn/language-http", + source: :npm, + latest_version: "1.2.3", + checked_at: ~U[2026-05-14 00:00:00.000000Z] + } + + describe "changeset/2 — required fields" do + test "is valid with the minimum required set" do + changeset = Adaptor.changeset(%Adaptor{}, @valid_attrs) + assert changeset.valid? + end + + test "requires :name" do + changeset = + Adaptor.changeset(%Adaptor{}, Map.delete(@valid_attrs, :name)) + + refute changeset.valid? + assert "can't be blank" in errors_on(changeset, :name) + end + + test "requires :source" do + changeset = + Adaptor.changeset(%Adaptor{}, Map.delete(@valid_attrs, :source)) + + refute changeset.valid? + assert "can't be blank" in errors_on(changeset, :source) + end + + test "requires :latest_version" do + changeset = + Adaptor.changeset( + %Adaptor{}, + Map.delete(@valid_attrs, :latest_version) + ) + + refute changeset.valid? + assert "can't be blank" in errors_on(changeset, :latest_version) + end + + test "requires :checked_at" do + changeset = + Adaptor.changeset(%Adaptor{}, Map.delete(@valid_attrs, :checked_at)) + + refute changeset.valid? + assert "can't be blank" in errors_on(changeset, :checked_at) + end + end + + describe "changeset/2 — :name length cap" do + test "rejects :name longer than 214 characters (npm pkg limit)" do + too_long = String.duplicate("a", 215) + + changeset = + Adaptor.changeset(%Adaptor{}, %{@valid_attrs | name: too_long}) + + refute changeset.valid? + + assert Enum.any?( + errors_on(changeset, :name), + &(&1 =~ "should be at most") + ) + end + + test "accepts :name of exactly 214 characters" do + ok = String.duplicate("a", 214) + + assert Adaptor.changeset(%Adaptor{}, %{@valid_attrs | name: ok}).valid? + end + end + + describe "changeset/2 — :source Ecto.Enum cast" do + test "round-trips atom :npm" do + changeset = + Adaptor.changeset(%Adaptor{}, %{@valid_attrs | source: :npm}) + + assert changeset.valid? + assert Ecto.Changeset.get_change(changeset, :source) == :npm + end + + test "round-trips atom :local" do + changeset = + Adaptor.changeset(%Adaptor{}, %{@valid_attrs | source: :local}) + + assert changeset.valid? + assert Ecto.Changeset.get_change(changeset, :source) == :local + end + + test "casts the string form \"npm\" back to the :npm atom" do + changeset = + Adaptor.changeset(%Adaptor{}, %{@valid_attrs | source: "npm"}) + + assert changeset.valid? + assert Ecto.Changeset.get_change(changeset, :source) == :npm + end + + test "casts the string form \"local\" back to the :local atom" do + changeset = + Adaptor.changeset(%Adaptor{}, %{@valid_attrs | source: "local"}) + + assert changeset.valid? + assert Ecto.Changeset.get_change(changeset, :source) == :local + end + + test "rejects an unknown source atom" do + changeset = + Adaptor.changeset(%Adaptor{}, %{@valid_attrs | source: :other}) + + refute changeset.valid? + assert "is invalid" in errors_on(changeset, :source) + end + + test "rejects an unknown source string" do + changeset = + Adaptor.changeset(%Adaptor{}, %{@valid_attrs | source: "other"}) + + refute changeset.valid? + assert "is invalid" in errors_on(changeset, :source) + end + end + + describe "changeset/2 — icon ext inclusion" do + test "accepts \"png\" for :icon_square_ext" do + attrs = + Map.merge(@valid_attrs, %{ + icon_square_ext: "png", + icon_square_sha256: :crypto.strong_rand_bytes(32) + }) + + assert Adaptor.changeset(%Adaptor{}, attrs).valid? + end + + test "accepts \"svg\" for :icon_square_ext" do + attrs = + Map.merge(@valid_attrs, %{ + icon_square_ext: "svg", + icon_square_sha256: :crypto.strong_rand_bytes(32) + }) + + assert Adaptor.changeset(%Adaptor{}, attrs).valid? + end + + test "rejects an unknown :icon_square_ext" do + attrs = + Map.merge(@valid_attrs, %{ + icon_square_ext: "gif", + icon_square_sha256: :crypto.strong_rand_bytes(32) + }) + + changeset = Adaptor.changeset(%Adaptor{}, attrs) + refute changeset.valid? + assert "is invalid" in errors_on(changeset, :icon_square_ext) + end + + test "accepts \"png\" for :icon_rectangle_ext" do + attrs = + Map.merge(@valid_attrs, %{ + icon_rectangle_ext: "png", + icon_rectangle_sha256: :crypto.strong_rand_bytes(32) + }) + + assert Adaptor.changeset(%Adaptor{}, attrs).valid? + end + + test "accepts \"svg\" for :icon_rectangle_ext" do + attrs = + Map.merge(@valid_attrs, %{ + icon_rectangle_ext: "svg", + icon_rectangle_sha256: :crypto.strong_rand_bytes(32) + }) + + assert Adaptor.changeset(%Adaptor{}, attrs).valid? + end + + test "rejects an unknown :icon_rectangle_ext" do + attrs = + Map.merge(@valid_attrs, %{ + icon_rectangle_ext: "jpeg", + icon_rectangle_sha256: :crypto.strong_rand_bytes(32) + }) + + changeset = Adaptor.changeset(%Adaptor{}, attrs) + refute changeset.valid? + assert "is invalid" in errors_on(changeset, :icon_rectangle_ext) + end + end + + describe "changeset/2 — validate_icon_sha256_pair (square)" do + test "accepts both nil (no icon)" do + attrs = + Map.merge(@valid_attrs, %{ + icon_square_ext: nil, + icon_square_sha256: nil + }) + + assert Adaptor.changeset(%Adaptor{}, attrs).valid? + end + + test "accepts both set" do + attrs = + Map.merge(@valid_attrs, %{ + icon_square_ext: "png", + icon_square_sha256: :crypto.strong_rand_bytes(32) + }) + + assert Adaptor.changeset(%Adaptor{}, attrs).valid? + end + + test "rejects ext set without sha256" do + attrs = + Map.merge(@valid_attrs, %{ + icon_square_ext: "png", + icon_square_sha256: nil + }) + + changeset = Adaptor.changeset(%Adaptor{}, attrs) + refute changeset.valid? + + assert Enum.any?( + errors_on(changeset, :icon_square_sha256), + &(&1 =~ "must not be nil") + ) + end + + test "rejects sha256 set without ext" do + attrs = + Map.merge(@valid_attrs, %{ + icon_square_ext: nil, + icon_square_sha256: :crypto.strong_rand_bytes(32) + }) + + changeset = Adaptor.changeset(%Adaptor{}, attrs) + refute changeset.valid? + + assert Enum.any?( + errors_on(changeset, :icon_square_sha256), + &(&1 =~ "must be nil") + ) + end + end + + describe "changeset/2 — validate_icon_sha256_pair (rectangle)" do + test "accepts both nil (no icon)" do + attrs = + Map.merge(@valid_attrs, %{ + icon_rectangle_ext: nil, + icon_rectangle_sha256: nil + }) + + assert Adaptor.changeset(%Adaptor{}, attrs).valid? + end + + test "accepts both set" do + attrs = + Map.merge(@valid_attrs, %{ + icon_rectangle_ext: "svg", + icon_rectangle_sha256: :crypto.strong_rand_bytes(32) + }) + + assert Adaptor.changeset(%Adaptor{}, attrs).valid? + end + + test "rejects ext set without sha256" do + attrs = + Map.merge(@valid_attrs, %{ + icon_rectangle_ext: "svg", + icon_rectangle_sha256: nil + }) + + changeset = Adaptor.changeset(%Adaptor{}, attrs) + refute changeset.valid? + + assert Enum.any?( + errors_on(changeset, :icon_rectangle_sha256), + &(&1 =~ "must not be nil") + ) + end + + test "rejects sha256 set without ext" do + attrs = + Map.merge(@valid_attrs, %{ + icon_rectangle_ext: nil, + icon_rectangle_sha256: :crypto.strong_rand_bytes(32) + }) + + changeset = Adaptor.changeset(%Adaptor{}, attrs) + refute changeset.valid? + + assert Enum.any?( + errors_on(changeset, :icon_rectangle_sha256), + &(&1 =~ "must be nil") + ) + end + end + + describe "changeset/2 — unique_constraint" do + test "registers a unique_constraint on [:name, :source]" do + changeset = Adaptor.changeset(%Adaptor{}, @valid_attrs) + + assert Enum.any?(changeset.constraints, fn c -> + c.type == :unique and + c.constraint == "adaptors_name_source_index" + end) + end + end + + defp errors_on(changeset, field) do + for {f, {msg, _opts}} <- changeset.errors, f == field, do: msg + end +end diff --git a/test/lightning/adaptors/repo_adaptor_version_test.exs b/test/lightning/adaptors/repo_adaptor_version_test.exs new file mode 100644 index 00000000000..b9e9a52167a --- /dev/null +++ b/test/lightning/adaptors/repo_adaptor_version_test.exs @@ -0,0 +1,172 @@ +defmodule Lightning.Adaptors.Repo.AdaptorVersionTest do + use ExUnit.Case, async: true + + alias Lightning.Adaptors.Repo.AdaptorVersion + + @adaptor_id Ecto.UUID.generate() + + @valid_attrs %{ + adaptor_id: @adaptor_id, + version: "1.2.3" + } + + describe "changeset/2 — required fields" do + test "is valid with the minimum required set" do + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, @valid_attrs) + assert changeset.valid? + end + + test "requires :adaptor_id" do + changeset = + AdaptorVersion.changeset( + %AdaptorVersion{}, + Map.delete(@valid_attrs, :adaptor_id) + ) + + refute changeset.valid? + assert "can't be blank" in errors_on(changeset, :adaptor_id) + end + + test "requires :version" do + changeset = + AdaptorVersion.changeset( + %AdaptorVersion{}, + Map.delete(@valid_attrs, :version) + ) + + refute changeset.valid? + assert "can't be blank" in errors_on(changeset, :version) + end + end + + describe "changeset/2 — optional fields round-trip" do + test "casts :integrity" do + attrs = Map.put(@valid_attrs, :integrity, "sha512-abcdef==") + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, attrs) + + assert changeset.valid? + + assert Ecto.Changeset.get_change(changeset, :integrity) == + "sha512-abcdef==" + end + + test "casts :tarball_url" do + attrs = + Map.put( + @valid_attrs, + :tarball_url, + "https://registry.npmjs.org/x/-/x-1.2.3.tgz" + ) + + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, attrs) + + assert changeset.valid? + + assert Ecto.Changeset.get_change(changeset, :tarball_url) == + "https://registry.npmjs.org/x/-/x-1.2.3.tgz" + end + + test "casts :size_bytes" do + attrs = Map.put(@valid_attrs, :size_bytes, 12_345) + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, attrs) + + assert changeset.valid? + assert Ecto.Changeset.get_change(changeset, :size_bytes) == 12_345 + end + + test "casts :dependencies as a map (no structural validation)" do + deps = %{"axios" => "^1.0.0", "lodash" => "4.17.21"} + attrs = Map.put(@valid_attrs, :dependencies, deps) + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, attrs) + + assert changeset.valid? + assert Ecto.Changeset.get_change(changeset, :dependencies) == deps + end + + test "casts :peer_dependencies as a map (no structural validation)" do + peers = %{"react" => "^18.0.0"} + attrs = Map.put(@valid_attrs, :peer_dependencies, peers) + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, attrs) + + assert changeset.valid? + + assert Ecto.Changeset.get_change(changeset, :peer_dependencies) == + peers + end + + test "accepts an arbitrarily-shaped :dependencies map" do + weird = %{"a" => 1, "b" => %{"nested" => true}, "c" => nil} + attrs = Map.put(@valid_attrs, :dependencies, weird) + + assert AdaptorVersion.changeset(%AdaptorVersion{}, attrs).valid? + end + + test "casts :published_at" do + ts = ~U[2026-05-14 12:00:00.000000Z] + attrs = Map.put(@valid_attrs, :published_at, ts) + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, attrs) + + assert changeset.valid? + assert Ecto.Changeset.get_change(changeset, :published_at) == ts + end + + test "casts :deprecated" do + attrs = Map.put(@valid_attrs, :deprecated, true) + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, attrs) + + assert changeset.valid? + assert Ecto.Changeset.get_change(changeset, :deprecated) == true + end + + test "defaults :deprecated to false on a fresh struct" do + assert %AdaptorVersion{}.deprecated == false + end + end + + describe "changeset/2 — unique_constraint" do + test "registers a unique_constraint on [:adaptor_id, :version]" do + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, @valid_attrs) + + assert Enum.any?(changeset.constraints, fn c -> + c.type == :unique and + c.constraint == "adaptor_versions_adaptor_id_version_index" + end) + end + end + + describe "changeset/2 — FK constraint" do + test "registers a foreign_key constraint on the :adaptor association" do + changeset = AdaptorVersion.changeset(%AdaptorVersion{}, @valid_attrs) + + assert Enum.any?(changeset.constraints, fn c -> + c.type == :foreign_key and c.field == :adaptor + end) + end + end + + describe "schema" do + test "belongs_to :adaptor uses binary_id" do + assoc = AdaptorVersion.__schema__(:association, :adaptor) + assert assoc.related == Lightning.Adaptors.Repo.Adaptor + assert assoc.owner_key == :adaptor_id + end + + test ":adaptor_id field is binary_id" do + assert AdaptorVersion.__schema__(:type, :adaptor_id) == :binary_id + end + + test ":id field is binary_id" do + assert AdaptorVersion.__schema__(:type, :id) == :binary_id + end + + test "has :inserted_at but not :updated_at" do + fields = AdaptorVersion.__schema__(:fields) + assert :inserted_at in fields + refute :updated_at in fields + end + end + + defp errors_on(changeset, field) do + for {f, {msg, _opts}} <- changeset.errors, f == field, do: msg + end +end diff --git a/test/lightning/adaptors/repo_test.exs b/test/lightning/adaptors/repo_test.exs new file mode 100644 index 00000000000..c56c9ec8162 --- /dev/null +++ b/test/lightning/adaptors/repo_test.exs @@ -0,0 +1,485 @@ +defmodule Lightning.Adaptors.RepoTest do + use Lightning.DataCase, async: true + + alias Lightning.Adaptors.Repo, as: AdaptorRepo + alias Lightning.Adaptors.Repo.Adaptor + alias Lightning.Adaptors.Repo.AdaptorVersion + + describe "upsert_adaptor/1 — initial insert" do + test "inserts the adaptor row and its versions in one transaction" do + record = + adaptor_record( + versions: [version_record("1.0.0"), version_record("1.1.0")] + ) + + assert {:ok, %Adaptor{} = adaptor} = AdaptorRepo.upsert_adaptor(record) + + assert adaptor.name == "@openfn/language-http" + assert adaptor.source == :npm + assert adaptor.latest_version == "1.0.0" + assert %DateTime{} = adaptor.checked_at + assert %DateTime{} = adaptor.updated_at + + versions = AdaptorRepo.list_versions(adaptor.name, :npm) + + assert versions |> Enum.map(& &1.version) |> Enum.sort() == [ + "1.0.0", + "1.1.0" + ] + + assert Enum.all?(versions, &(&1.adaptor_id == adaptor.id)) + end + + test "accepts a record with no versions" do + record = adaptor_record(versions: []) + + assert {:ok, %Adaptor{} = adaptor} = AdaptorRepo.upsert_adaptor(record) + assert AdaptorRepo.list_versions(adaptor.name, :npm) == [] + end + end + + describe "upsert_adaptor/1 — idempotency (§12.2)" do + test "re-upserting the same record advances :checked_at but not :updated_at" do + {:ok, first} = AdaptorRepo.upsert_adaptor(adaptor_record()) + + Process.sleep(5) + + {:ok, second} = AdaptorRepo.upsert_adaptor(adaptor_record()) + + assert second.id == first.id + assert second.updated_at == first.updated_at + assert DateTime.compare(second.checked_at, first.checked_at) == :gt + end + end + + describe "upsert_adaptor/1 — diff-aware :updated_at" do + test "changing :latest_version bumps :updated_at" do + {:ok, first} = AdaptorRepo.upsert_adaptor(adaptor_record()) + + Process.sleep(5) + + {:ok, second} = + AdaptorRepo.upsert_adaptor(adaptor_record(latest_version: "1.1.0")) + + assert second.latest_version == "1.1.0" + assert DateTime.compare(second.updated_at, first.updated_at) == :gt + end + + test "changing :description bumps :updated_at" do + {:ok, first} = AdaptorRepo.upsert_adaptor(adaptor_record()) + + Process.sleep(5) + + {:ok, second} = + AdaptorRepo.upsert_adaptor(adaptor_record(description: "new copy")) + + assert second.description == "new copy" + assert DateTime.compare(second.updated_at, first.updated_at) == :gt + end + end + + describe "upsert_adaptor/1 — version row replacement (§12.2)" do + test "replaces version rows atomically" do + {:ok, _adaptor} = + AdaptorRepo.upsert_adaptor( + adaptor_record( + versions: [version_record("1.0.0"), version_record("1.1.0")] + ) + ) + + assert AdaptorRepo.list_versions("@openfn/language-http", :npm) + |> Enum.map(& &1.version) + |> Enum.sort() == ["1.0.0", "1.1.0"] + + {:ok, _adaptor} = + AdaptorRepo.upsert_adaptor( + adaptor_record( + versions: [ + version_record("1.1.0"), + version_record("1.2.0"), + version_record("2.0.0") + ] + ) + ) + + assert AdaptorRepo.list_versions("@openfn/language-http", :npm) + |> Enum.map(& &1.version) + |> Enum.sort() == ["1.1.0", "1.2.0", "2.0.0"] + end + + test "shrinking the version set drops the missing rows" do + {:ok, _} = + AdaptorRepo.upsert_adaptor( + adaptor_record( + versions: [version_record("1.0.0"), version_record("1.1.0")] + ) + ) + + {:ok, _} = + AdaptorRepo.upsert_adaptor( + adaptor_record(versions: [version_record("1.1.0")]) + ) + + assert [%AdaptorVersion{version: "1.1.0"}] = + AdaptorRepo.list_versions("@openfn/language-http", :npm) + end + + test "persists version-row payload fields verbatim" do + payload = %{ + version: "1.0.0", + integrity: "sha512-deadbeef==", + tarball_url: "https://registry.npmjs.org/x/-/x-1.0.0.tgz", + size_bytes: 4321, + dependencies: %{"axios" => "^1.0.0"}, + peer_dependencies: %{"react" => "^18"}, + published_at: ~U[2026-05-01 10:00:00.000000Z], + deprecated: false + } + + {:ok, _} = AdaptorRepo.upsert_adaptor(adaptor_record(versions: [payload])) + + assert [version] = + AdaptorRepo.list_versions("@openfn/language-http", :npm) + + assert version.integrity == payload.integrity + assert version.tarball_url == payload.tarball_url + assert version.size_bytes == payload.size_bytes + assert version.dependencies == payload.dependencies + assert version.peer_dependencies == payload.peer_dependencies + assert version.published_at == payload.published_at + assert version.deprecated == payload.deprecated + end + end + + describe "upsert_adaptor/1 — source isolation" do + test "the same name can coexist across sources" do + {:ok, npm_row} = + AdaptorRepo.upsert_adaptor(adaptor_record(source: :npm)) + + {:ok, local_row} = + AdaptorRepo.upsert_adaptor(adaptor_record(source: :local)) + + assert npm_row.id != local_row.id + assert npm_row.source == :npm + assert local_row.source == :local + end + end + + describe "touch_checked_at/2 (§12.2)" do + test "advances :checked_at and leaves :updated_at alone" do + {:ok, original} = AdaptorRepo.upsert_adaptor(adaptor_record()) + + Process.sleep(5) + + assert :ok = AdaptorRepo.touch_checked_at(original.name, :npm) + + reloaded = AdaptorRepo.get_adaptor(original.name, :npm) + assert DateTime.compare(reloaded.checked_at, original.checked_at) == :gt + assert reloaded.updated_at == original.updated_at + end + + test "is a no-op for an unknown (name, source) — does not require loading the row" do + assert :ok = AdaptorRepo.touch_checked_at("@openfn/never-existed", :npm) + assert AdaptorRepo.get_adaptor("@openfn/never-existed", :npm) == nil + end + + test "is source-scoped" do + {:ok, npm_row} = + AdaptorRepo.upsert_adaptor(adaptor_record(source: :npm)) + + {:ok, local_row} = + AdaptorRepo.upsert_adaptor(adaptor_record(source: :local)) + + Process.sleep(5) + + :ok = AdaptorRepo.touch_checked_at(npm_row.name, :npm) + + reloaded_npm = AdaptorRepo.get_adaptor(npm_row.name, :npm) + reloaded_local = AdaptorRepo.get_adaptor(local_row.name, :local) + + assert DateTime.compare(reloaded_npm.checked_at, npm_row.checked_at) == :gt + assert reloaded_local.checked_at == local_row.checked_at + end + end + + describe "stalest/2 (§12.2)" do + test "orders by :checked_at ascending" do + base = DateTime.utc_now() + + seed_adaptor(name: "@openfn/a", checked_at: DateTime.add(base, -300)) + seed_adaptor(name: "@openfn/b", checked_at: DateTime.add(base, -100)) + seed_adaptor(name: "@openfn/c", checked_at: DateTime.add(base, -200)) + + assert AdaptorRepo.stalest(10, :npm) |> Enum.map(& &1.name) == + ["@openfn/a", "@openfn/c", "@openfn/b"] + end + + test "honours the limit" do + base = DateTime.utc_now() + seed_adaptor(name: "@openfn/a", checked_at: DateTime.add(base, -300)) + seed_adaptor(name: "@openfn/b", checked_at: DateTime.add(base, -200)) + seed_adaptor(name: "@openfn/c", checked_at: DateTime.add(base, -100)) + + assert length(AdaptorRepo.stalest(2, :npm)) == 2 + end + + test "filters by source" do + seed_adaptor(name: "@openfn/a", source: :npm) + seed_adaptor(name: "@openfn/a", source: :local) + + assert [%Adaptor{source: :npm}] = AdaptorRepo.stalest(10, :npm) + assert [%Adaptor{source: :local}] = AdaptorRepo.stalest(10, :local) + end + end + + describe "max_checked_at/1" do + test "returns the largest :checked_at for the given source" do + base = DateTime.utc_now() + newest = DateTime.add(base, -100) + seed_adaptor(name: "@openfn/a", checked_at: DateTime.add(base, -300)) + seed_adaptor(name: "@openfn/b", checked_at: newest) + + assert AdaptorRepo.max_checked_at(:npm) == newest + end + + test "returns nil when the source has no rows" do + seed_adaptor(name: "@openfn/a", source: :npm) + assert AdaptorRepo.max_checked_at(:local) == nil + end + end + + describe "get_adaptor/2" do + test "returns the matching adaptor" do + {:ok, inserted} = AdaptorRepo.upsert_adaptor(adaptor_record()) + reloaded = AdaptorRepo.get_adaptor(inserted.name, :npm) + + assert %Adaptor{} = reloaded + assert reloaded.id == inserted.id + end + + test "returns nil when not found" do + assert AdaptorRepo.get_adaptor("@openfn/never-existed", :npm) == nil + end + + test "is source-scoped" do + {:ok, _} = AdaptorRepo.upsert_adaptor(adaptor_record(source: :npm)) + assert AdaptorRepo.get_adaptor("@openfn/language-http", :local) == nil + end + end + + describe "list_package_metas/1" do + test "returns the lean projection without heavy JSONB columns" do + {:ok, _} = + AdaptorRepo.upsert_adaptor( + adaptor_record( + description: "yep", + schema_data: %{"big" => "json", "nested" => %{"more" => "stuff"}} + ) + ) + + assert [meta] = AdaptorRepo.list_package_metas(:npm) + + assert meta.name == "@openfn/language-http" + assert meta.latest_version == "1.0.0" + assert meta.description == "yep" + assert meta.deprecated == false + assert %DateTime{} = meta.updated_at + + refute Map.has_key?(meta, :schema_data) + refute Map.has_key?(meta, :homepage) + end + + test "filters by source" do + {:ok, _} = AdaptorRepo.upsert_adaptor(adaptor_record(source: :npm)) + {:ok, _} = AdaptorRepo.upsert_adaptor(adaptor_record(source: :local)) + + assert [%{name: "@openfn/language-http"}] = + AdaptorRepo.list_package_metas(:npm) + + assert [%{name: "@openfn/language-http"}] = + AdaptorRepo.list_package_metas(:local) + end + end + + describe "list_adaptors/1" do + test "returns full structs filtered by source" do + {:ok, _} = AdaptorRepo.upsert_adaptor(adaptor_record(source: :npm)) + {:ok, _} = AdaptorRepo.upsert_adaptor(adaptor_record(source: :local)) + + assert [%Adaptor{source: :npm}] = AdaptorRepo.list_adaptors(:npm) + assert [%Adaptor{source: :local}] = AdaptorRepo.list_adaptors(:local) + end + end + + describe "list_missing_icons/1" do + test "returns rows where either icon shape sha256 is nil" do + {:ok, _} = AdaptorRepo.upsert_adaptor(adaptor_record(name: "@openfn/a")) + + {:ok, _} = + AdaptorRepo.upsert_adaptor( + adaptor_record( + name: "@openfn/b", + icon_square_ext: "png", + icon_square_sha256: :crypto.hash(:sha256, "x") + ) + ) + + {:ok, _} = + AdaptorRepo.upsert_adaptor( + adaptor_record( + name: "@openfn/c", + icon_square_ext: "png", + icon_square_sha256: :crypto.hash(:sha256, "y"), + icon_rectangle_ext: "png", + icon_rectangle_sha256: :crypto.hash(:sha256, "z") + ) + ) + + names = + AdaptorRepo.list_missing_icons(:npm) + |> Enum.map(& &1.name) + |> Enum.sort() + + assert names == ["@openfn/a", "@openfn/b"] + end + + test "is source-scoped" do + {:ok, _} = + AdaptorRepo.upsert_adaptor( + adaptor_record(name: "@openfn/x", source: :local) + ) + + assert AdaptorRepo.list_missing_icons(:npm) == [] + assert [%{name: "@openfn/x"}] = AdaptorRepo.list_missing_icons(:local) + end + end + + describe "update_icons/3" do + test "writes only icon columns and bumps :updated_at" do + {:ok, before} = AdaptorRepo.upsert_adaptor(adaptor_record()) + Process.sleep(5) + sha = :crypto.hash(:sha256, "PNG") + + assert {1, nil} = + AdaptorRepo.update_icons(before.name, :npm, %{ + icon_square_ext: "png", + icon_square_sha256: sha + }) + + after_row = AdaptorRepo.get_adaptor(before.name, :npm) + + assert after_row.icon_square_ext == "png" + assert after_row.icon_square_sha256 == sha + assert after_row.latest_version == before.latest_version + assert DateTime.compare(after_row.updated_at, before.updated_at) == :gt + end + + test "ignores keys outside the icon set" do + {:ok, before} = AdaptorRepo.upsert_adaptor(adaptor_record()) + + AdaptorRepo.update_icons(before.name, :npm, %{ + latest_version: "9.9.9", + icon_square_ext: "svg", + icon_square_sha256: :crypto.hash(:sha256, "S") + }) + + after_row = AdaptorRepo.get_adaptor(before.name, :npm) + assert after_row.latest_version == before.latest_version + assert after_row.icon_square_ext == "svg" + end + + test "writes icon etag columns alongside ext/sha256" do + {:ok, before} = AdaptorRepo.upsert_adaptor(adaptor_record()) + sha = :crypto.hash(:sha256, "PNG") + + assert {1, nil} = + AdaptorRepo.update_icons(before.name, :npm, %{ + icon_square_ext: "png", + icon_square_sha256: sha, + icon_square_etag: ~s("abc123") + }) + + after_row = AdaptorRepo.get_adaptor(before.name, :npm) + + assert %{ + icon_square_ext: "png", + icon_square_sha256: ^sha, + icon_square_etag: ~s("abc123"), + icon_rectangle_etag: nil + } = after_row + end + + test "leaves version rows untouched" do + {:ok, before} = + AdaptorRepo.upsert_adaptor( + adaptor_record( + versions: [version_record("1.0.0"), version_record("2.0.0")] + ) + ) + + AdaptorRepo.update_icons(before.name, :npm, %{ + icon_square_ext: "png", + icon_square_sha256: :crypto.hash(:sha256, "P") + }) + + versions = AdaptorRepo.list_versions(before.name, :npm) + assert length(versions) == 2 + end + end + + defp adaptor_record(overrides \\ []) do + overrides = Map.new(overrides) + + %{ + name: "@openfn/language-http", + source: :npm, + latest_version: "1.0.0", + description: "HTTP adaptor", + homepage: nil, + repository: nil, + license: "LGPL-3.0", + deprecated: false, + schema_data: nil, + schema_sha256: nil, + icon_square_ext: nil, + icon_rectangle_ext: nil, + icon_square_sha256: nil, + icon_rectangle_sha256: nil, + icon_square_etag: nil, + icon_rectangle_etag: nil, + versions: [version_record("1.0.0")] + } + |> Map.merge(overrides) + end + + defp version_record(version) do + %{ + version: version, + integrity: "sha512-#{version}", + tarball_url: "https://example.com/x/-/x-#{version}.tgz", + size_bytes: 1024, + dependencies: %{}, + peer_dependencies: %{}, + published_at: nil, + deprecated: false + } + end + + defp seed_adaptor(opts) do + attrs = + %{ + name: "@openfn/language-http", + source: :npm, + latest_version: "1.0.0", + checked_at: DateTime.utc_now() + } + |> Map.merge(Map.new(opts)) + + {:ok, adaptor} = + %Adaptor{} + |> Adaptor.changeset(attrs) + |> Lightning.Repo.insert() + + adaptor + end +end diff --git a/test/lightning/adaptors/scheduler_test.exs b/test/lightning/adaptors/scheduler_test.exs new file mode 100644 index 00000000000..bbf148dbffb --- /dev/null +++ b/test/lightning/adaptors/scheduler_test.exs @@ -0,0 +1,959 @@ +defmodule Lightning.Adaptors.SchedulerTest do + # async: false because: + # 1. DataCase uses shared sandbox mode (all processes access DB without allow/3) + # 2. set_mox_global is safe only when tests run serially + use Lightning.DataCase, async: false + + import Mox + + alias Lightning.Adaptors.Repo, as: AdaptorsRepo + alias Lightning.Adaptors.Scheduler + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + # set_mox_global makes expectations visible to tasks spawned by the Scheduler, + # whose $callers chain does not include the test process (only the GenServer). + setup :set_mox_global + setup :verify_on_exit! + + # Each test owns an isolated supervisor. The supervisor starts its own + # Scheduler as part of the :rest_for_one child list, but with the + # test-env `refresh_interval: 0` it's an inert no-op. Individual tests + # call `start_scheduler/2` to replace it with a controlled-interval + # Scheduler under `start_supervised!/1` (so Mox expectations can be + # registered before init fires). + setup do + sup = :"sched_test_#{System.unique_integer([:positive])}" + + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.StrategyMock} + ) + + # Default no-op icons stub for tests that don't care about the icons + # pipeline. Individual tests override via `expect` when they need to + # assert on it. + stub(Lightning.Adaptors.StrategyMock, :fetch_icons, fn _opts -> + {:ok, %{}} + end) + + {:ok, sup: sup} + end + + # Replace the supervisor's inert auto-started (HighlanderPG-wrapped) + # Scheduler with a controlled one under test ownership. Application + # env is restored immediately after start_supervised!/1 returns + # because the Scheduler captures interval_ms in init/1. + # + # The test-owned Scheduler bypasses HighlanderPG entirely: we + # register the GenServer directly under the same `{:global, …}` name + # the production wrapper would, so test code can call it via + # `AdaptorsSupervisor.global_scheduler_name/1` exactly as production + # callers do. + defp start_scheduler(sup, opts \\ []) do + interval = Keyword.get(opts, :interval, 99_999_999) + original_env = Application.get_env(:lightning, Lightning.Adaptors, []) + + Application.put_env( + :lightning, + Lightning.Adaptors, + Keyword.put(original_env, :refresh_interval, interval) + ) + + global_name = AdaptorsSupervisor.global_scheduler_name(sup) + source_topic = AdaptorsSupervisor.source_topic(sup) + + # Stop the supervisor's auto-started HighlanderPG (and its wrapped + # Scheduler) so we can start a replacement under the controlled + # interval without name collision. + :ok = + Supervisor.terminate_child(sup, AdaptorsSupervisor.highlander_name(sup)) + + pid = + start_supervised!({ + Scheduler, + name: global_name, + sup: sup, + lock_key: AdaptorsSupervisor.lock_key(sup), + cache: AdaptorsSupervisor.cache_name(sup), + tasks: AdaptorsSupervisor.tasks_name(sup), + source_topic: source_topic + }) + + Application.put_env(:lightning, Lightning.Adaptors, original_env) + + pid + end + + defp adaptor_record(overrides \\ []) do + overrides = Map.new(overrides) + + %{ + name: "@openfn/language-http", + source: :npm, + latest_version: "1.0.0", + description: "HTTP adaptor", + homepage: nil, + repository: nil, + license: "LGPL-3.0", + deprecated: false, + schema_data: nil, + schema_sha256: nil, + versions: [ + %{ + version: "1.0.0", + integrity: "sha512-abc", + tarball_url: "https://example.com/x-1.0.0.tgz", + size_bytes: 1024, + dependencies: %{}, + peer_dependencies: %{}, + published_at: nil, + deprecated: false + } + ] + } + |> Map.merge(overrides) + end + + describe "start_link/1" do + test "raises when :name is missing", %{sup: sup} do + assert_raise KeyError, ~r/key :name not found/, fn -> + Scheduler.start_link( + sup: sup, + lock_key: 1, + cache: :cache, + tasks: :tasks, + source_topic: "t" + ) + end + end + + test "raises when :sup is missing", %{sup: sup} do + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert_raise KeyError, ~r/key :sup not found/, fn -> + Scheduler.start_link( + name: sched_name, + lock_key: 1, + cache: :cache, + tasks: :tasks, + source_topic: "t" + ) + end + end + + test "raises when :lock_key is missing", %{sup: sup} do + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert_raise KeyError, ~r/key :lock_key not found/, fn -> + Scheduler.start_link( + name: sched_name, + sup: sup, + cache: :cache, + tasks: :tasks, + source_topic: "t" + ) + end + end + + test "registers under :global with global_scheduler_name/1", %{sup: sup} do + stub(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> {:ok, []} end) + start_scheduler(sup) + {:global, global_name} = AdaptorsSupervisor.global_scheduler_name(sup) + assert is_pid(:global.whereis_name(global_name)) + end + end + + describe "tick timing" do + test "tick fires on init when table is empty", %{sup: sup} do + test_pid = self() + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + send(test_pid, :list_adaptors_called) + {:ok, []} + end) + + # Empty table → max_checked_at returns nil → delay 0 → tick fires on init. + start_scheduler(sup) + + assert_receive :list_adaptors_called, 2000 + end + + test "tick re-arms itself", %{sup: sup} do + test_pid = self() + + # Stub allows repeated calls; each fires a message so we can count them. + stub(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + send(test_pid, :tick_ran) + {:ok, []} + end) + + # 30ms interval → two ticks fire well within 2s. + start_scheduler(sup, interval: 30) + + assert_receive :tick_ran, 2000 + assert_receive :tick_ran, 2000 + end + end + + describe "do_refresh/1 diff logic" do + test "unchanged adaptor: touch_checked_at only, no upsert, no broadcast", %{ + sup: sup + } do + test_pid = self() + source = AdaptorsSupervisor.source(sup) + source_topic = AdaptorsSupervisor.source_topic(sup) + + {:ok, existing} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + checked_at_before = existing.checked_at + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + send(test_pid, :list_adaptors_called) + {:ok, [%{name: "@openfn/language-http", latest_version: "1.0.0"}]} + end) + + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 0, fn _ -> + :unreachable + end) + + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + # With a recently-inserted adaptor, max_checked_at is "now", so the smart- + # init delay is ~99,999 seconds. Trigger an explicit tick via refresh_now. + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + Scheduler.refresh_now(sched_name) + + assert_receive :list_adaptors_called, 2000 + + # Allow the spawned task to complete before asserting no broadcast. + refute_receive {:changed, _, _}, 200 + + row = AdaptorsRepo.get_adaptor("@openfn/language-http", source) + assert DateTime.compare(row.checked_at, checked_at_before) == :gt + assert row.latest_version == "1.0.0" + end + + test "changed adaptor: upsert and broadcast per changed name", %{sup: sup} do + test_pid = self() + source = AdaptorsSupervisor.source(sup) + source_topic = AdaptorsSupervisor.source_topic(sup) + + {:ok, _} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + send(test_pid, :list_adaptors_called) + {:ok, [%{name: "@openfn/language-http", latest_version: "2.0.0"}]} + end) + + expect( + Lightning.Adaptors.StrategyMock, + :fetch_adaptor, + 1, + fn "@openfn/language-http" -> + {:ok, adaptor_record(latest_version: "2.0.0")} + end + ) + + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + # Trigger an explicit tick since smart-init delay is large (recent checked_at). + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + Scheduler.refresh_now(sched_name) + + assert_receive :list_adaptors_called, 2000 + assert_receive {:changed, "@openfn/language-http", ^source}, 2000 + + row = AdaptorsRepo.get_adaptor("@openfn/language-http", source) + assert row.latest_version == "2.0.0" + end + + test "new adaptor (not in DB): upsert and broadcast", %{sup: sup} do + source = AdaptorsSupervisor.source(sup) + source_topic = AdaptorsSupervisor.source_topic(sup) + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + {:ok, [%{name: "@openfn/language-new", latest_version: "1.0.0"}]} + end) + + expect( + Lightning.Adaptors.StrategyMock, + :fetch_adaptor, + 1, + fn "@openfn/language-new" -> + {:ok, adaptor_record(name: "@openfn/language-new")} + end + ) + + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + assert_receive {:changed, "@openfn/language-new", ^source}, 2000 + assert AdaptorsRepo.get_adaptor("@openfn/language-new", source) != nil + end + + test "list_adaptors error: no DB writes, no broadcasts", %{sup: sup} do + test_pid = self() + source_topic = AdaptorsSupervisor.source_topic(sup) + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + send(test_pid, :list_adaptors_called) + {:error, :timeout} + end) + + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 0, fn _ -> + :unreachable + end) + + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + assert_receive :list_adaptors_called, 2000 + refute_receive {:changed, _, _}, 200 + end + + test "fetch_adaptor error: logs warning, continues to next adaptor", %{ + sup: sup + } do + source_topic = AdaptorsSupervisor.source_topic(sup) + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + {:ok, + [ + %{name: "@openfn/bad-adaptor", latest_version: "1.0.0"}, + %{name: "@openfn/good-adaptor", latest_version: "1.0.0"} + ]} + end) + + # Single multi-clause expectation — Mox routes by pattern within + # one slot, so Scheduler's async_stream_nolink can fan out to the + # two adaptors in either order. Two separate `expect/4` calls + # would dispatch FIFO and crash with FunctionClauseError when the + # task arrival order doesn't match the expectation insertion order. + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 2, fn + "@openfn/bad-adaptor" -> + {:error, :not_found} + + "@openfn/good-adaptor" -> + {:ok, adaptor_record(name: "@openfn/good-adaptor")} + end) + + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + assert_receive {:changed, "@openfn/good-adaptor", _}, 2000 + refute_receive {:changed, "@openfn/bad-adaptor", _}, 200 + end + end + + describe "refresh_now/1" do + test "triggers an immediate tick on the leader", %{sup: sup} do + test_pid = self() + + # First call: from init tick. Second call: from refresh_now. + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, 2, fn -> + send(test_pid, :tick_ran) + {:ok, []} + end) + + start_scheduler(sup) + + # Wait for init tick. + assert_receive :tick_ran, 2000 + + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + assert :ok = Scheduler.refresh_now(sched_name) + + assert_receive :tick_ran, 2000 + end + end + + describe "icons pipeline" do + test "writes icon bytes to disk and stamps ext+sha256 on the row", %{ + sup: sup + } do + source = AdaptorsSupervisor.source(sup) + + bytes = "ICON_BYTES" + sha = :crypto.hash(:sha256, bytes) + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + {:ok, [%{name: "@openfn/language-http", latest_version: "1.0.0"}]} + end) + + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn _ -> + {:ok, adaptor_record()} + end) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, fn _opts -> + {:ok, + %{ + "@openfn/language-http" => %{ + square: %{data: bytes, ext: "png", sha256: sha} + } + }} + end) + + source_topic = AdaptorsSupervisor.source_topic(sup) + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + assert_receive {:changed, "@openfn/language-http", ^source}, 2000 + + row = AdaptorsRepo.get_adaptor("@openfn/language-http", source) + assert row.icon_square_ext == "png" + assert row.icon_square_sha256 == sha + assert row.icon_rectangle_ext == nil + assert row.icon_rectangle_sha256 == nil + + icon_path = + Lightning.Adaptors.IconCache.path( + source, + "@openfn/language-http", + :square, + "png" + ) + + assert File.exists?(icon_path) + assert File.read!(icon_path) == bytes + File.rm!(icon_path) + end + + test "fetch_icons error: records still persist without icons", %{sup: sup} do + source = AdaptorsSupervisor.source(sup) + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + {:ok, [%{name: "@openfn/language-http", latest_version: "1.0.0"}]} + end) + + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn _ -> + {:ok, adaptor_record()} + end) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, fn _opts -> + {:error, :timeout} + end) + + source_topic = AdaptorsSupervisor.source_topic(sup) + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + assert_receive {:changed, "@openfn/language-http", ^source}, 2000 + + row = AdaptorsRepo.get_adaptor("@openfn/language-http", source) + assert row != nil + assert row.icon_square_ext == nil + assert row.icon_square_sha256 == nil + end + + test "self-heals iconless rows on the periodic tick", %{sup: sup} do + source = AdaptorsSupervisor.source(sup) + + # Pre-seed a row that already matches the listed latest_version + # (so the diff path will :touch instead of :fetch). Without + # self-heal this row would stay iconless forever. + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record(name: "@openfn/language-stale") + ) + + bytes = "STALE_ICON" + sha = :crypto.hash(:sha256, bytes) + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + {:ok, [%{name: "@openfn/language-stale", latest_version: "1.0.0"}]} + end) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, fn opts -> + # Row has no etags pre-seeded, so it is omitted from the + # prior-etags map entirely (no empty inner map). + assert Keyword.get(opts, :prior_etags) == %{} + + {:ok, + %{ + "@openfn/language-stale" => %{ + square: %{data: bytes, ext: "png", sha256: sha} + } + }} + end) + + source_topic = AdaptorsSupervisor.source_topic(sup) + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + # The pre-seeded row pushes max_checked_at to "now", so init + # delay = full interval — drive the tick explicitly. + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + :ok = Scheduler.refresh_now(sched_name) + + assert_receive {:changed, "@openfn/language-stale", ^source}, 2000 + + row = AdaptorsRepo.get_adaptor("@openfn/language-stale", source) + assert row.icon_square_ext == "png" + assert row.icon_square_sha256 == sha + + icon_path = + Lightning.Adaptors.IconCache.path( + source, + "@openfn/language-stale", + :square, + "png" + ) + + File.rm(icon_path) + end + + test "fetches per-adaptor in parallel (multiple concurrent fetch_adaptor calls)", + %{sup: sup} do + test_pid = self() + barrier = :ets.new(:scheduler_test_barrier, [:public, :set]) + :ets.insert(barrier, {:in_flight, 0}) + :ets.insert(barrier, {:max_in_flight, 0}) + + names = + for i <- 1..6, + do: %{name: "@openfn/language-pkg#{i}", latest_version: "1.0.0"} + + expect(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + {:ok, names} + end) + + stub(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn name -> + in_flight = :ets.update_counter(barrier, :in_flight, 1) + + :ets.update_element( + barrier, + :max_in_flight, + {2, max_seen(barrier, in_flight)} + ) + + # Hold long enough that the fan-out has time to overlap. + Process.sleep(80) + :ets.update_counter(barrier, :in_flight, -1) + send(test_pid, {:fetched, name}) + {:ok, adaptor_record(name: name)} + end) + + start_scheduler(sup) + + for _ <- 1..6 do + assert_receive {:fetched, _name}, 5_000 + end + + [{:max_in_flight, max_in_flight}] = :ets.lookup(barrier, :max_in_flight) + :ets.delete(barrier) + + assert max_in_flight > 1, + "expected concurrent fetch_adaptor calls, saw at most 1 in-flight" + end + end + + defp max_seen(barrier, current) do + [{:max_in_flight, prev}] = :ets.lookup(barrier, :max_in_flight) + max(prev, current) + end + + describe "refresh_package/2" do + test "fetches and upserts a single adaptor, bypassing diff", %{sup: sup} do + test_pid = self() + source = AdaptorsSupervisor.source(sup) + source_topic = AdaptorsSupervisor.source_topic(sup) + + stub(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + send(test_pid, :init_tick_done) + {:ok, []} + end) + + expect( + Lightning.Adaptors.StrategyMock, + :fetch_adaptor, + 1, + fn "@openfn/language-http" -> + {:ok, adaptor_record()} + end + ) + + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + # Drain the init tick (table is empty → delay 0 → fires immediately). + assert_receive :init_tick_done, 2000 + + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert :ok = Scheduler.refresh_package(sched_name, "@openfn/language-http") + assert_receive {:changed, "@openfn/language-http", ^source}, 2000 + + assert AdaptorsRepo.get_adaptor("@openfn/language-http", source) != nil + end + + test "returns error tuple when fetch_adaptor fails", %{sup: sup} do + test_pid = self() + + stub(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + send(test_pid, :init_tick_done) + {:ok, []} + end) + + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 1, fn _ -> + {:error, :not_found} + end) + + start_scheduler(sup) + + # Drain init tick before calling refresh_package. + assert_receive :init_tick_done, 2000 + + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert {:error, :not_found} = + Scheduler.refresh_package(sched_name, "@openfn/language-http") + end + + test "does not call fetch_icons (icons only refresh on the periodic tick)", + %{sup: sup} do + test_pid = self() + source_topic = AdaptorsSupervisor.source_topic(sup) + + stub(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + send(test_pid, :init_tick_done) + {:ok, []} + end) + + # Exactly one fetch_icons call — the init tick. If refresh_package + # also fetched icons the count would be 2 and Mox would fail. + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, 1, fn _opts -> + send(test_pid, :icons_called) + {:ok, %{}} + end) + + expect( + Lightning.Adaptors.StrategyMock, + :fetch_adaptor, + 1, + fn "@openfn/language-http" -> {:ok, adaptor_record()} end + ) + + :ok = Phoenix.PubSub.subscribe(Lightning.PubSub, source_topic) + start_scheduler(sup) + + assert_receive :init_tick_done, 2000 + assert_receive :icons_called, 2000 + + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + assert :ok = Scheduler.refresh_package(sched_name, "@openfn/language-http") + assert_receive {:changed, "@openfn/language-http", _}, 2000 + + # Give any (mistaken) extra fetch_icons call time to happen. + Process.sleep(100) + end + end + + describe "refresh_icons/1" do + test "updates rows whose shape sha256 differs from the fetched icon", %{ + sup: sup + } do + source = AdaptorsSupervisor.source(sup) + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record(name: "@openfn/language-empty") + ) + + old_sha = :crypto.hash(:sha256, "OLD") + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: "@openfn/language-current", + icon_square_ext: "png", + icon_square_sha256: old_sha + ) + ) + + new_bytes = "NEW_BYTES" + new_sha = :crypto.hash(:sha256, new_bytes) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, fn _opts -> + {:ok, + %{ + "@openfn/language-empty" => %{ + square: %{data: new_bytes, ext: "png", sha256: new_sha} + }, + "@openfn/language-current" => %{ + square: %{data: new_bytes, ext: "png", sha256: new_sha} + } + }} + end) + + # interval: 0 disables the init tick so refresh_icons is the only + # path that calls fetch_icons. + start_scheduler(sup, interval: 0) + + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert {:ok, %{updated: 2, unchanged: 0}} = + Scheduler.refresh_icons(sched_name) + + empty = AdaptorsRepo.get_adaptor("@openfn/language-empty", source) + assert empty.icon_square_ext == "png" + assert empty.icon_square_sha256 == new_sha + + current = AdaptorsRepo.get_adaptor("@openfn/language-current", source) + assert current.icon_square_sha256 == new_sha + + for name <- ["@openfn/language-empty", "@openfn/language-current"] do + Lightning.Adaptors.IconCache.path(source, name, :square, "png") + |> File.rm() + end + end + + test "leaves rows whose shape sha256 already matches unchanged, passing prior etag", + %{sup: sup} do + source = AdaptorsSupervisor.source(sup) + sha = :crypto.hash(:sha256, "SAME") + etag = ~s("prior-etag-1") + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: "@openfn/language-same", + icon_square_ext: "png", + icon_square_sha256: sha, + icon_square_etag: etag + ) + ) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, fn opts -> + # Strategy receives the prior etag for this row's shape. + assert Keyword.get(opts, :prior_etags) == %{ + "@openfn/language-same" => %{square: etag} + } + + {:ok, %{"@openfn/language-same" => %{square: :not_modified}}} + end) + + start_scheduler(sup, interval: 0) + + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert {:ok, %{updated: 0, unchanged: 1}} = + Scheduler.refresh_icons(sched_name) + + row = AdaptorsRepo.get_adaptor("@openfn/language-same", source) + assert row.icon_square_sha256 == sha + assert row.icon_square_etag == etag + end + + test "applies new etag when shape sha256 changes", %{sup: sup} do + source = AdaptorsSupervisor.source(sup) + old_sha = :crypto.hash(:sha256, "OLD") + new_bytes = "NEW" + new_sha = :crypto.hash(:sha256, new_bytes) + old_etag = ~s("etag-A") + new_etag = ~s("etag-B") + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: "@openfn/language-rotated", + icon_square_ext: "png", + icon_square_sha256: old_sha, + icon_square_etag: old_etag + ) + ) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, fn _opts -> + {:ok, + %{ + "@openfn/language-rotated" => %{ + square: %{ + data: new_bytes, + ext: "png", + sha256: new_sha, + etag: new_etag + } + } + }} + end) + + start_scheduler(sup, interval: 0) + + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert {:ok, %{updated: 1, unchanged: 0}} = + Scheduler.refresh_icons(sched_name) + + row = AdaptorsRepo.get_adaptor("@openfn/language-rotated", source) + assert row.icon_square_sha256 == new_sha + assert row.icon_square_etag == new_etag + + Lightning.Adaptors.IconCache.path( + source, + "@openfn/language-rotated", + :square, + "png" + ) + |> File.rm() + end + + test "preserves existing etag when fetched entry's etag is nil or missing", + %{sup: sup} do + source = AdaptorsSupervisor.source(sup) + old_sha = :crypto.hash(:sha256, "OLD") + new_bytes_a = "NEW_A" + new_sha_a = :crypto.hash(:sha256, new_bytes_a) + new_bytes_b = "NEW_B" + new_sha_b = :crypto.hash(:sha256, new_bytes_b) + prior_etag = ~s("etag-A") + + # Two rows: one returns 200 with etag: nil (NPM-style), the other + # returns 200 with the :etag key entirely absent (Local-style). + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: "@openfn/language-nil-etag", + icon_square_ext: "png", + icon_square_sha256: old_sha, + icon_square_etag: prior_etag + ) + ) + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: "@openfn/language-no-etag-key", + icon_square_ext: "png", + icon_square_sha256: old_sha, + icon_square_etag: prior_etag + ) + ) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, fn _opts -> + {:ok, + %{ + "@openfn/language-nil-etag" => %{ + square: %{ + data: new_bytes_a, + ext: "png", + sha256: new_sha_a, + etag: nil + } + }, + "@openfn/language-no-etag-key" => %{ + square: %{data: new_bytes_b, ext: "png", sha256: new_sha_b} + } + }} + end) + + start_scheduler(sup, interval: 0) + + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert {:ok, %{updated: 2, unchanged: 0}} = + Scheduler.refresh_icons(sched_name) + + row_a = AdaptorsRepo.get_adaptor("@openfn/language-nil-etag", source) + assert row_a.icon_square_sha256 == new_sha_a + assert row_a.icon_square_etag == prior_etag + + row_b = AdaptorsRepo.get_adaptor("@openfn/language-no-etag-key", source) + assert row_b.icon_square_sha256 == new_sha_b + assert row_b.icon_square_etag == prior_etag + + for name <- ["@openfn/language-nil-etag", "@openfn/language-no-etag-key"] do + Lightning.Adaptors.IconCache.path(source, name, :square, "png") + |> File.rm() + end + end + + test "mixed 304 and 200: unchanged row preserves its etag verbatim", + %{sup: sup} do + source = AdaptorsSupervisor.source(sup) + stale_old_sha = :crypto.hash(:sha256, "STALE_OLD") + stale_new_bytes = "STALE_NEW" + stale_new_sha = :crypto.hash(:sha256, stale_new_bytes) + stale_old_etag = ~s("etag-stale-old") + stale_new_etag = ~s("etag-stale-new") + + current_sha = :crypto.hash(:sha256, "CURRENT_BYTES") + current_etag = ~s("etag-current") + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: "@openfn/language-stale-etag", + icon_square_ext: "png", + icon_square_sha256: stale_old_sha, + icon_square_etag: stale_old_etag + ) + ) + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: "@openfn/language-current-etag", + icon_square_ext: "png", + icon_square_sha256: current_sha, + icon_square_etag: current_etag + ) + ) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, fn opts -> + # Both rows contribute prior etags. + assert Keyword.get(opts, :prior_etags) == %{ + "@openfn/language-stale-etag" => %{square: stale_old_etag}, + "@openfn/language-current-etag" => %{square: current_etag} + } + + {:ok, + %{ + "@openfn/language-stale-etag" => %{ + square: %{ + data: stale_new_bytes, + ext: "png", + sha256: stale_new_sha, + etag: stale_new_etag + } + }, + "@openfn/language-current-etag" => %{square: :not_modified} + }} + end) + + start_scheduler(sup, interval: 0) + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert {:ok, %{updated: 1, unchanged: 1}} = + Scheduler.refresh_icons(sched_name) + + stale_row = AdaptorsRepo.get_adaptor("@openfn/language-stale-etag", source) + assert stale_row.icon_square_sha256 == stale_new_sha + assert stale_row.icon_square_etag == stale_new_etag + + current_row = + AdaptorsRepo.get_adaptor("@openfn/language-current-etag", source) + + assert current_row.icon_square_sha256 == current_sha + assert current_row.icon_square_etag == current_etag + + Lightning.Adaptors.IconCache.path( + source, + "@openfn/language-stale-etag", + :square, + "png" + ) + |> File.rm() + end + + test "surfaces a strategy fetch error as {:error, reason}", %{sup: sup} do + expect(Lightning.Adaptors.StrategyMock, :fetch_icons, fn _opts -> + {:error, :upstream_down} + end) + + start_scheduler(sup, interval: 0) + + sched_name = AdaptorsSupervisor.global_scheduler_name(sup) + + assert {:error, :upstream_down} = Scheduler.refresh_icons(sched_name) + end + end +end diff --git a/test/lightning/adaptors/store_test.exs b/test/lightning/adaptors/store_test.exs new file mode 100644 index 00000000000..02f565aa867 --- /dev/null +++ b/test/lightning/adaptors/store_test.exs @@ -0,0 +1,552 @@ +defmodule Lightning.Adaptors.StoreTest do + use Lightning.DataCase, async: true + + import Mox + + alias Lightning.Adaptors.Repo, as: AdaptorsRepo + alias Lightning.Adaptors.Store + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + setup :verify_on_exit! + + setup do + # Each test owns an isolated `Lightning.Adaptors.Supervisor` instance, + # parameterised on a unique `name:` so cache table / persistent_term + # entries don't collide across the async suite. The `:strategy` opt + # is threaded explicitly — no `Application.put_env` mutation. + sup = :"store_test_#{System.unique_integer([:positive])}" + + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.StrategyMock} + ) + + cache = AdaptorsSupervisor.cache_name(sup) + + {:ok, sup: sup, cache: cache} + end + + describe "schema/2" do + test "cache hit returns cached value without touching Strategy or DB", %{ + sup: sup, + cache: cache + } do + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 0, fn _ -> + :unreachable + end) + + source = AdaptorsSupervisor.source(sup) + + Cachex.put!( + cache, + {:schema, "@openfn/language-http", source}, + {:ok, ~s({"type":"object"})} + ) + + assert {:ok, ~s({"type":"object"})} = + Store.schema(sup, "@openfn/language-http") + + assert AdaptorsRepo.get_adaptor("@openfn/language-http", source) == nil + end + + test "cache miss + DB hit returns DB value without calling Strategy", %{ + sup: sup + } do + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 0, fn _ -> + :unreachable + end) + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record(schema_data: ~s({"type":"object"})) + ) + + assert {:ok, ~s({"type":"object"})} = + Store.schema(sup, "@openfn/language-http") + end + + test "cache miss + DB miss calls Strategy once, upserts to DB, caches result", + %{ + sup: sup, + cache: cache + } do + source = AdaptorsSupervisor.source(sup) + + expect( + Lightning.Adaptors.StrategyMock, + :fetch_adaptor, + 1, + fn "@openfn/language-http" -> + {:ok, adaptor_record(schema_data: ~s({"type":"object"}))} + end + ) + + assert {:ok, ~s({"type":"object"})} = + Store.schema(sup, "@openfn/language-http") + + assert %{schema_data: ~s({"type":"object"})} = + AdaptorsRepo.get_adaptor("@openfn/language-http", source) + + assert {:ok, {:ok, ~s({"type":"object"})}} = + Cachex.get(cache, {:schema, "@openfn/language-http", source}) + end + + test "three concurrent calls coalesce to one Strategy call", %{sup: sup} do + name = "@openfn/language-http" + test_pid = self() + + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 1, fn ^name -> + # Brief sleep so the other two tasks queue up in Cachex's courier. + Process.sleep(30) + {:ok, adaptor_record(schema_data: ~s({"type":"object"}))} + end) + + tasks = + Enum.map(1..3, fn _ -> + Task.async(fn -> + receive do + :go -> Store.schema(sup, name) + end + end) + end) + + # Allow all tasks to use the test process's Mox expectations before releasing them. + Enum.each( + tasks, + &Mox.allow(Lightning.Adaptors.StrategyMock, test_pid, &1.pid) + ) + + Enum.each(tasks, &send(&1.pid, :go)) + + results = Task.await_many(tasks, 5_000) + assert Enum.all?(results, &match?({:ok, ~s({"type":"object"})}, &1)) + end + + test "Strategy error returns {:error, _} and is not cached — next call retries", + %{ + sup: sup, + cache: cache + } do + source = AdaptorsSupervisor.source(sup) + + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 1, fn _ -> + {:error, :upstream_error} + end) + + assert {:error, :upstream_error} = + Store.schema(sup, "@openfn/language-http") + + assert {:ok, nil} = + Cachex.get(cache, {:schema, "@openfn/language-http", source}) + + expect( + Lightning.Adaptors.StrategyMock, + :fetch_adaptor, + 1, + fn "@openfn/language-http" -> + {:ok, adaptor_record(schema_data: ~s({"type":"object"}))} + end + ) + + assert {:ok, ~s({"type":"object"})} = + Store.schema(sup, "@openfn/language-http") + end + + test "preserves JSON property order through the persistence round-trip", + %{sup: sup} do + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 0, fn _ -> + :unreachable + end) + + ordered_body = ~s({"a":1,"z":2,"m":3}) + + {:ok, _} = + AdaptorsRepo.upsert_adaptor(adaptor_record(schema_data: ordered_body)) + + assert {:ok, ^ordered_body} = Store.schema(sup, "@openfn/language-http") + end + end + + describe "versions/2" do + test "cache miss + DB hit returns projected versions without calling Strategy", + %{sup: sup} do + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 0, fn _ -> + :unreachable + end) + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + versions: [version_record("1.0.0"), version_record("1.1.0")] + ) + ) + + assert {:ok, versions} = Store.versions(sup, "@openfn/language-http") + assert length(versions) == 2 + assert Enum.all?(versions, &Map.has_key?(&1, :version)) + assert Enum.all?(versions, &Map.has_key?(&1, :deprecated)) + end + + test "cache miss + DB miss calls Strategy and caches projected versions", %{ + sup: sup, + cache: cache + } do + source = AdaptorsSupervisor.source(sup) + + expect( + Lightning.Adaptors.StrategyMock, + :fetch_adaptor, + 1, + fn "@openfn/language-http" -> + {:ok, + adaptor_record( + versions: [version_record("1.0.0"), version_record("2.0.0")] + )} + end + ) + + assert {:ok, versions} = Store.versions(sup, "@openfn/language-http") + assert length(versions) == 2 + + assert {:ok, {:ok, cached_versions}} = + Cachex.get(cache, {:versions, "@openfn/language-http", source}) + + assert length(cached_versions) == 2 + end + end + + describe "packages/1" do + test "empty DB returns {:ok, []} but does NOT cache the empty result", %{ + sup: sup, + cache: cache + } do + expect(Lightning.Adaptors.StrategyMock, :fetch_adaptor, 0, fn _ -> + :unreachable + end) + + assert {:ok, []} = Store.packages(sup) + + source = AdaptorsSupervisor.source(sup) + assert {:ok, nil} = Cachex.get(cache, {:packages, source}) + end + + test "DB with rows returns and caches package metas", %{ + sup: sup, + cache: cache + } do + {:ok, _} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + + assert {:ok, [pkg]} = Store.packages(sup) + assert pkg.name == "@openfn/language-http" + + source = AdaptorsSupervisor.source(sup) + assert {:ok, {:ok, [_]}} = Cachex.get(cache, {:packages, source}) + end + end + + describe "icon/3" do + # Each test uses a unique adaptor name so the on-disk cache (shared + # default {:tmp, "lightning/adaptor_icons"} path) does not collide + # across this `async: true` suite. Directories created here are not + # cleaned up — they live under System.tmp_dir! and are namespaced + # per-name so they cannot collide. + defp unique_name(prefix) do + "@openfn/language-#{prefix}-#{System.unique_integer([:positive])}" + end + + test "disk hit returns path without calling Strategy", %{sup: sup} do + source = AdaptorsSupervisor.source(sup) + name = unique_name("disk-hit") + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: name, + icon_square_ext: "png", + icon_square_sha256: :crypto.hash(:sha256, "PRE_WARMED") + ) + ) + + {:ok, _} = + Lightning.Adaptors.IconCache.write!( + source, + name, + :square, + "png", + "PRE_WARMED" + ) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icon, 0, fn _, _ -> + :unreachable + end) + + assert {:ok, path} = Store.icon(sup, name, :square) + assert File.read!(path) == "PRE_WARMED" + end + + test "disk miss + Strategy success writes to disk and returns path", %{ + sup: sup, + cache: cache + } do + source = AdaptorsSupervisor.source(sup) + name = unique_name("disk-miss") + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: name, + icon_square_ext: "png", + icon_square_sha256: :crypto.hash(:sha256, "LAZY_BYTES") + ) + ) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icon, 1, fn ^name, + :square -> + {:ok, %{data: "LAZY_BYTES", ext: "png"}} + end) + + assert {:ok, path} = Store.icon(sup, name, :square) + assert File.read!(path) == "LAZY_BYTES" + + # Courier returned {:ignore, _} → no committed entry on the bytes key. + assert {:ok, nil} = + Cachex.get(cache, {:icon_bytes, source, name, :square}) + end + + test "Strategy error returns {:error, _} and does not commit", %{ + sup: sup, + cache: cache + } do + source = AdaptorsSupervisor.source(sup) + name = unique_name("err") + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: name, + icon_square_ext: "png", + icon_square_sha256: :crypto.hash(:sha256, "UNUSED") + ) + ) + + expect(Lightning.Adaptors.StrategyMock, :fetch_icon, 1, fn _, _ -> + {:error, :upstream_5xx} + end) + + assert {:error, :upstream_5xx} = Store.icon(sup, name, :square) + + assert {:ok, nil} = + Cachex.get(cache, {:icon_bytes, source, name, :square}) + end + + test "concurrent first-callers coalesce onto one Strategy fetch", %{ + sup: sup + } do + test_pid = self() + name = unique_name("coalesce") + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: name, + icon_square_ext: "png", + icon_square_sha256: :crypto.hash(:sha256, "COALESCED") + ) + ) + + # Single Mox expectation → if both callers reach the strategy + # the second hits "no expectation" and Mox raises. + expect(Lightning.Adaptors.StrategyMock, :fetch_icon, 1, fn ^name, + :square -> + send(test_pid, :fetch_started) + # Block long enough for the second caller to also reach the + # Cachex courier and coalesce onto this call. + Process.sleep(150) + {:ok, %{data: "COALESCED", ext: "png"}} + end) + + t1 = Task.async(fn -> Store.icon(sup, name, :square) end) + assert_receive :fetch_started, 1000 + t2 = Task.async(fn -> Store.icon(sup, name, :square) end) + + assert {:ok, p1} = Task.await(t1, 5000) + assert {:ok, p2} = Task.await(t2, 5000) + assert p1 == p2 + assert File.read!(p1) == "COALESCED" + end + + test "different (name, shape) misses fetch in parallel without false coalescing", + %{sup: sup} do + test_pid = self() + name_a = unique_name("parA") + name_b = unique_name("parB") + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: name_a, + icon_square_ext: "png", + icon_square_sha256: :crypto.hash(:sha256, "A_BYTES") + ) + ) + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + name: name_b, + icon_square_ext: "png", + icon_square_sha256: :crypto.hash(:sha256, "B_BYTES") + ) + ) + + # Single multi-clause expectation with count: 2 — Mox routes by + # pattern within one slot, so the two parallel courier calls can + # arrive in either order. Two separate `expect/3` calls would + # queue FIFO and crash with FunctionClauseError when the task + # arrival order doesn't match the expectation insertion order. + expect(Lightning.Adaptors.StrategyMock, :fetch_icon, 2, fn + ^name_a, :square -> {:ok, %{data: "A_BYTES", ext: "png"}} + ^name_b, :square -> {:ok, %{data: "B_BYTES", ext: "png"}} + end) + + t_a = + Task.async(fn -> + receive do + :go -> Store.icon(sup, name_a, :square) + end + end) + + t_b = + Task.async(fn -> + receive do + :go -> Store.icon(sup, name_b, :square) + end + end) + + Mox.allow(Lightning.Adaptors.StrategyMock, test_pid, t_a.pid) + Mox.allow(Lightning.Adaptors.StrategyMock, test_pid, t_b.pid) + + send(t_a.pid, :go) + send(t_b.pid, :go) + + assert {:ok, p1} = Task.await(t_a, 5000) + assert {:ok, p2} = Task.await(t_b, 5000) + + assert File.read!(p1) == "A_BYTES" + assert File.read!(p2) == "B_BYTES" + end + end + + describe "icon_meta/2" do + test "unknown adaptor returns {:error, :not_found} and is not cached", %{ + sup: sup, + cache: cache + } do + assert {:error, :not_found} = Store.icon_meta(sup, "@openfn/never-existed") + + source = AdaptorsSupervisor.source(sup) + + assert {:ok, nil} = + Cachex.get(cache, {:icon_meta, "@openfn/never-existed", source}) + end + + test "known adaptor returns icon metadata and caches it", %{ + sup: sup, + cache: cache + } do + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + icon_square_ext: "svg", + icon_square_sha256: :crypto.hash(:sha256, "fake-svg-bytes") + ) + ) + + assert {:ok, meta} = Store.icon_meta(sup, "@openfn/language-http") + assert meta.icon_square_ext == "svg" + + source = AdaptorsSupervisor.source(sup) + + assert {:ok, {:ok, cached}} = + Cachex.get(cache, {:icon_meta, "@openfn/language-http", source}) + + assert cached.icon_square_ext == "svg" + end + end + + describe "warm_from_repo/1" do + test "populates {:packages, source} and {:icon_meta, name, source} keys", %{ + sup: sup, + cache: cache + } do + {:ok, _} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + + assert :ok = Store.warm_from_repo(sup) + + source = AdaptorsSupervisor.source(sup) + + assert {:ok, {:ok, [pkg]}} = Cachex.get(cache, {:packages, source}) + assert pkg.name == "@openfn/language-http" + + assert {:ok, {:ok, icon_meta}} = + Cachex.get(cache, {:icon_meta, "@openfn/language-http", source}) + + assert Map.has_key?(icon_meta, :icon_square_ext) + assert Map.has_key?(icon_meta, :icon_rectangle_ext) + end + + test "overwrites existing keys without clearing unrelated ones", %{ + sup: sup, + cache: cache + } do + source = AdaptorsSupervisor.source(sup) + + Cachex.put!( + cache, + {:schema, "pre-existing", source}, + {:ok, %{"kept" => true}} + ) + + {:ok, _} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + assert :ok = Store.warm_from_repo(sup) + + assert {:ok, {:ok, %{"kept" => true}}} = + Cachex.get(cache, {:schema, "pre-existing", source}) + end + end + + defp adaptor_record(overrides \\ []) do + overrides = Map.new(overrides) + + %{ + name: "@openfn/language-http", + source: :npm, + latest_version: "1.0.0", + description: "HTTP adaptor", + homepage: nil, + repository: nil, + license: "LGPL-3.0", + deprecated: false, + schema_data: nil, + schema_sha256: nil, + icon_square_ext: nil, + icon_rectangle_ext: nil, + icon_square_sha256: nil, + icon_rectangle_sha256: nil, + versions: [version_record("1.0.0")] + } + |> Map.merge(overrides) + end + + defp version_record(version) do + %{ + version: version, + integrity: "sha512-#{version}", + tarball_url: "https://example.com/x/-/x-#{version}.tgz", + size_bytes: 1024, + dependencies: %{}, + peer_dependencies: %{}, + published_at: nil, + deprecated: false + } + end +end diff --git a/test/lightning/adaptors/supervisor_integration_test.exs b/test/lightning/adaptors/supervisor_integration_test.exs new file mode 100644 index 00000000000..608466cb860 --- /dev/null +++ b/test/lightning/adaptors/supervisor_integration_test.exs @@ -0,0 +1,191 @@ +defmodule Lightning.Adaptors.SupervisorIntegrationTest do + @moduledoc """ + Integration-level tests for `Lightning.Adaptors.Supervisor`: prove all + Phase A children boot under a single `start_supervised!` call and that + the `:rest_for_one` cascade pins §6.5a (Invalidator subscribes at init; + if Cachex restarts without Invalidator restarting, the cache goes + stale). + """ + + use Lightning.DataCase, async: false + + import Eventually + + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + # Children with their *registered* names. We look up live PIDs by name + # (Process.whereis/1) rather than by child id from which_children/1, + # because module-based child specs share child ids like `Cachex` or + # `Lightning.Adaptors.Invalidator` — those don't carry the per-instance + # name we derive in the Supervisor. The Scheduler is registered via + # `:global` (HighlanderPG-wrapped) so it needs a `:global.whereis_name/1` + # lookup instead. + defp local_named_children(sup) do + %{ + cache: AdaptorsSupervisor.cache_name(sup), + tasks: AdaptorsSupervisor.tasks_name(sup), + invalidator: AdaptorsSupervisor.invalidator_name(sup), + node_monitor: AdaptorsSupervisor.node_monitor_name(sup), + broadcaster: AdaptorsSupervisor.channel_broadcaster_name(sup) + } + end + + defp scheduler_pid(sup) do + {:global, global_name} = AdaptorsSupervisor.global_scheduler_name(sup) + + case :global.whereis_name(global_name) do + :undefined -> nil + pid -> pid + end + end + + defp pids_by_role(sup) do + locals = + sup + |> local_named_children() + |> Enum.map(fn {role, registered_name} -> + {role, Process.whereis(registered_name)} + end) + |> Map.new() + + Map.put(locals, :scheduler, scheduler_pid(sup)) + end + + # HighlanderPG polls every 300ms by default; allow ~3s for the + # wrapped child to acquire the advisory lock and register globally. + @scheduler_wait_ms 3_000 + + setup do + sup = :"test_full_boot_#{System.unique_integer([:positive])}" + on_exit(fn -> AdaptorsSupervisor.forget(sup) end) + {:ok, sup: sup} + end + + describe "child-list boot" do + test "boots the full child list under one start_supervised! call", + %{sup: sup} do + pid = + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.Local} + ) + + children = Supervisor.which_children(pid) + + # Cachex + Task.Supervisor + Invalidator + NodeMonitor + + # ChannelBroadcaster + HighlanderPG(Scheduler) = 6. + assert length(children) == 6 + + ids = Enum.map(children, fn {id, _pid, _type, _mods} -> id end) + assert AdaptorsSupervisor.highlander_name(sup) in ids + + Enum.each(children, fn {_id, child_pid, _type, _mods} -> + assert is_pid(child_pid), + "unexpected child pid shape: #{inspect(child_pid)}" + + assert Process.alive?(child_pid), + "child pid #{inspect(child_pid)} is not alive" + end) + + # Locally-registered children are up under their derived names. + Enum.each(local_named_children(sup), fn {role, registered_name} -> + pid = Process.whereis(registered_name) + assert is_pid(pid), "expected #{role} to be registered and alive" + assert Process.alive?(pid) + end) + + # The HighlanderPG-wrapped Scheduler registers globally once it + # acquires the advisory lock — give it up to ~3s to do so. + assert_eventually(is_pid(scheduler_pid(sup)), @scheduler_wait_ms) + assert Process.alive?(scheduler_pid(sup)) + end + + test "exposes the per-instance strategy and source via :persistent_term", + %{sup: sup} do + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.Local} + ) + + assert AdaptorsSupervisor.strategy(sup) == Lightning.Adaptors.Local + assert AdaptorsSupervisor.source(sup) == :local + end + end + + describe ":rest_for_one strategy" do + test "Cachex crash cascades to Invalidator / ChannelBroadcaster / Scheduler", + %{sup: sup} do + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.Local} + ) + + # Block until the HighlanderPG-wrapped Scheduler has registered + # globally so we have a baseline pid to compare against. + assert_eventually(is_pid(scheduler_pid(sup)), @scheduler_wait_ms) + + before = pids_by_role(sup) + + cachex_pid = Map.fetch!(before, :cache) + assert is_pid(cachex_pid) + + ref = Process.monitor(cachex_pid) + Process.exit(cachex_pid, :kill) + assert_receive {:DOWN, ^ref, :process, ^cachex_pid, _}, 1_000 + + after_pids = wait_for_restart(sup, before) + + # Cachex itself comes back under a fresh pid. + assert Map.fetch!(after_pids, :cache) != cachex_pid + + # §6.5a: under :rest_for_one, all children that depend on Cachex + # (Invalidator, Broadcaster, Scheduler) must restart too so they + # re-bind to the fresh cache. + for role <- [:invalidator, :broadcaster, :scheduler] do + old = Map.fetch!(before, role) + new = Map.fetch!(after_pids, role) + assert is_pid(old) + assert is_pid(new) + + assert new != old, + "expected #{role} to restart after Cachex crash " <> + "(before=#{inspect(old)}, after=#{inspect(new)})" + end + end + end + + # Polls `pids_by_role/1` until the children we expect to be restarted + # show new PIDs, or we hit the deadline. Returns the post-restart map. + # The Scheduler restart goes through HighlanderPG (lock + poll cycle), + # so allow a slightly longer deadline than for the locally-registered + # children alone. + defp wait_for_restart(sup, before, deadline_ms \\ 3_000) do + start = System.monotonic_time(:millisecond) + roles_expected = [:invalidator, :broadcaster, :scheduler] + do_wait_for_restart(sup, before, roles_expected, start, deadline_ms) + end + + defp do_wait_for_restart(sup, before, roles, start, deadline_ms) do + current = pids_by_role(sup) + + changed? = + Enum.all?(roles, fn role -> + case {Map.get(before, role), Map.get(current, role)} do + {old, new} when is_pid(old) and is_pid(new) -> old != new + _ -> false + end + end) + + cond do + changed? -> + current + + System.monotonic_time(:millisecond) - start > deadline_ms -> + flunk( + "supervisor children did not restart within #{deadline_ms}ms; " <> + "before=#{inspect(before)} after=#{inspect(current)}" + ) + + true -> + Process.sleep(20) + do_wait_for_restart(sup, before, roles, start, deadline_ms) + end + end +end diff --git a/test/lightning/adaptors/supervisor_test.exs b/test/lightning/adaptors/supervisor_test.exs new file mode 100644 index 00000000000..a05304c8ea2 --- /dev/null +++ b/test/lightning/adaptors/supervisor_test.exs @@ -0,0 +1,178 @@ +defmodule Lightning.Adaptors.SupervisorTest do + use ExUnit.Case, async: true + + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + describe "start_link/1" do + test "raises KeyError when :name is missing" do + assert_raise KeyError, ~r/key :name not found/, fn -> + AdaptorsSupervisor.start_link([]) + end + end + + test "raises KeyError when opts has no :name key" do + assert_raise KeyError, fn -> + AdaptorsSupervisor.start_link(strategy: :ignored) + end + end + end + + describe "derived-name helpers" do + test "cache_name/1 concatenates `Cache` onto the supervisor name" do + assert AdaptorsSupervisor.cache_name(Lightning.Adaptors) == + Lightning.Adaptors.Cache + + assert AdaptorsSupervisor.cache_name(:MyAdaptors) == + Module.concat(:MyAdaptors, Cache) + end + + test "tasks_name/1 concatenates `Tasks` onto the supervisor name" do + assert AdaptorsSupervisor.tasks_name(Lightning.Adaptors) == + Lightning.Adaptors.Tasks + end + + test "invalidator_name/1 concatenates `Invalidator` onto the supervisor name" do + assert AdaptorsSupervisor.invalidator_name(Lightning.Adaptors) == + Lightning.Adaptors.Invalidator + end + + test "channel_broadcaster_name/1 concatenates `ChannelBroadcaster`" do + assert AdaptorsSupervisor.channel_broadcaster_name(Lightning.Adaptors) == + Lightning.Adaptors.ChannelBroadcaster + end + + test "node_monitor_name/1 concatenates `NodeMonitor`" do + assert AdaptorsSupervisor.node_monitor_name(Lightning.Adaptors) == + Lightning.Adaptors.NodeMonitor + end + + test "scheduler_name/1 concatenates `Scheduler`" do + assert AdaptorsSupervisor.scheduler_name(Lightning.Adaptors) == + Lightning.Adaptors.Scheduler + end + + test "source_topic/1 returns an `adaptors:` string" do + assert AdaptorsSupervisor.source_topic(Lightning.Adaptors) == + "adaptors:Lightning.Adaptors" + end + + test "client_topic/1 returns an `adaptors:client_update:` string" do + assert AdaptorsSupervisor.client_topic(Lightning.Adaptors) == + "adaptors:client_update:Lightning.Adaptors" + end + + test "source_topic/1 and client_topic/1 produce distinct strings" do + name = Lightning.Adaptors + + refute AdaptorsSupervisor.source_topic(name) == + AdaptorsSupervisor.client_topic(name) + end + + test "lock_key/1 derives an int via :erlang.phash2({:adaptors, name})" do + name = Lightning.Adaptors + + assert AdaptorsSupervisor.lock_key(name) == + :erlang.phash2({:adaptors, name}) + end + + test "lock_key/1 of a name differs from phash2 of just the name" do + name = :"Adaptors_#{System.unique_integer([:positive])}" + + assert AdaptorsSupervisor.lock_key(name) != :erlang.phash2(name) + end + end + + describe "two concurrent supervisors do not collide" do + test "derived Cachex / Task.Supervisor / GenServer names differ between instances" do + a = :"AdaptorsA_#{System.unique_integer([:positive])}" + b = :"AdaptorsB_#{System.unique_integer([:positive])}" + + assert AdaptorsSupervisor.cache_name(a) != + AdaptorsSupervisor.cache_name(b) + + assert AdaptorsSupervisor.tasks_name(a) != + AdaptorsSupervisor.tasks_name(b) + + assert AdaptorsSupervisor.invalidator_name(a) != + AdaptorsSupervisor.invalidator_name(b) + + assert AdaptorsSupervisor.channel_broadcaster_name(a) != + AdaptorsSupervisor.channel_broadcaster_name(b) + + assert AdaptorsSupervisor.node_monitor_name(a) != + AdaptorsSupervisor.node_monitor_name(b) + + assert AdaptorsSupervisor.scheduler_name(a) != + AdaptorsSupervisor.scheduler_name(b) + end + + test "PubSub topics differ between instances" do + a = :"AdaptorsA_#{System.unique_integer([:positive])}" + b = :"AdaptorsB_#{System.unique_integer([:positive])}" + + assert AdaptorsSupervisor.source_topic(a) != + AdaptorsSupervisor.source_topic(b) + + assert AdaptorsSupervisor.client_topic(a) != + AdaptorsSupervisor.client_topic(b) + end + + test "HighlanderPG lock keys differ between instances" do + a = :"AdaptorsA_#{System.unique_integer([:positive])}" + b = :"AdaptorsB_#{System.unique_integer([:positive])}" + + assert AdaptorsSupervisor.lock_key(a) != + AdaptorsSupervisor.lock_key(b) + end + + test "production-equivalent lock key is stable across calls" do + first = AdaptorsSupervisor.lock_key(Lightning.Adaptors) + second = AdaptorsSupervisor.lock_key(Lightning.Adaptors) + + assert first == second + assert is_integer(first) + end + end + + describe "init/1 (child spec list)" do + # End-to-end boot (Cachex / Task.Supervisor / Invalidator / + # NodeMonitor / ChannelBroadcaster / HighlanderPG-wrapped Scheduler) + # is covered by `Lightning.Adaptors.SupervisorIntegrationTest`, + # which needs `Lightning.DataCase` and a real Postgres connection + # (HighlanderPG opens its own dedicated connection on `Lightning.Repo`). + + test "init/1 requires the :name opt" do + assert_raise KeyError, ~r/key :name not found/, fn -> + AdaptorsSupervisor.init([]) + end + end + end + + describe "derived names for HighlanderPG-wrapped Scheduler" do + test "global_scheduler_name/1 returns the {:global, atom()} pair" do + assert AdaptorsSupervisor.global_scheduler_name(Lightning.Adaptors) == + {:global, Lightning.Adaptors.Scheduler} + end + + test "global_scheduler_name/1 differs between instances" do + a = :"AdaptorsA_#{System.unique_integer([:positive])}" + b = :"AdaptorsB_#{System.unique_integer([:positive])}" + + assert AdaptorsSupervisor.global_scheduler_name(a) != + AdaptorsSupervisor.global_scheduler_name(b) + end + + test "highlander_name/1 concatenates `HighlanderPG`" do + assert AdaptorsSupervisor.highlander_name(Lightning.Adaptors) == + Lightning.Adaptors.HighlanderPG + end + + test "highlander_name/1 differs between instances" do + a = :"AdaptorsA_#{System.unique_integer([:positive])}" + b = :"AdaptorsB_#{System.unique_integer([:positive])}" + + assert AdaptorsSupervisor.highlander_name(a) != + AdaptorsSupervisor.highlander_name(b) + end + end +end diff --git a/test/lightning/adaptors_test.exs b/test/lightning/adaptors_test.exs new file mode 100644 index 00000000000..7d52228b327 --- /dev/null +++ b/test/lightning/adaptors_test.exs @@ -0,0 +1,279 @@ +defmodule Lightning.AdaptorsTest do + use Lightning.DataCase, async: false + + import Mox + + alias Lightning.Adaptors + alias Lightning.Adaptors.Repo, as: AdaptorsRepo + alias Lightning.Adaptors.Scheduler + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + setup :set_mox_global + setup :verify_on_exit! + + setup do + sup = :"adaptors_test_#{System.unique_integer([:positive])}" + + start_supervised!( + {AdaptorsSupervisor, name: sup, strategy: Lightning.Adaptors.StrategyMock} + ) + + {:ok, sup: sup} + end + + defp adaptor_record(overrides \\ []) do + overrides = Map.new(overrides) + + %{ + name: "@openfn/language-http", + source: :npm, + latest_version: "1.0.0", + description: "HTTP adaptor", + homepage: nil, + repository: nil, + license: "LGPL-3.0", + deprecated: false, + schema_data: nil, + schema_sha256: nil, + icon_square_ext: nil, + icon_rectangle_ext: nil, + icon_square_sha256: nil, + icon_rectangle_sha256: nil, + versions: [version_record("1.0.0")] + } + |> Map.merge(overrides) + end + + defp version_record(version) do + %{ + version: version, + integrity: "sha512-#{version}", + tarball_url: "https://example.com/x/-/x-#{version}.tgz", + size_bytes: 1024, + dependencies: %{}, + peer_dependencies: %{}, + published_at: nil, + deprecated: false + } + end + + defp start_scheduler(sup) do + original_env = Application.get_env(:lightning, Lightning.Adaptors, []) + + Application.put_env( + :lightning, + Lightning.Adaptors, + Keyword.put(original_env, :refresh_interval, 99_999_999) + ) + + # Stop the supervisor's auto-started HighlanderPG (and its wrapped + # Scheduler) so we can start a replacement under the controlled + # interval without name collision. The test-owned Scheduler registers + # directly under the same `{:global, …}` name production callers use. + :ok = + Supervisor.terminate_child(sup, AdaptorsSupervisor.highlander_name(sup)) + + pid = + start_supervised!({ + Scheduler, + name: AdaptorsSupervisor.global_scheduler_name(sup), + sup: sup, + lock_key: AdaptorsSupervisor.lock_key(sup), + cache: AdaptorsSupervisor.cache_name(sup), + tasks: AdaptorsSupervisor.tasks_name(sup), + source_topic: AdaptorsSupervisor.source_topic(sup) + }) + + Application.put_env(:lightning, Lightning.Adaptors, original_env) + + pid + end + + describe "packages/1" do + test "returns packages from DB", %{sup: sup} do + stub(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn _ -> + {:error, :unreachable} + end) + + {:ok, _} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + + assert {:ok, [pkg]} = Adaptors.packages(sup) + assert pkg.name == "@openfn/language-http" + end + + test "returns {:ok, []} when DB is empty", %{sup: sup} do + assert {:ok, []} = Adaptors.packages(sup) + end + end + + describe "packages/0 delegates to packages(Lightning.Adaptors)" do + test "packages/0 and packages(Lightning.Adaptors) return identical results" do + # The production `Lightning.Adaptors.Supervisor` is started under the + # name `Lightning.Adaptors` in `application.ex`; in test it uses + # `Lightning.Adaptors.StrategyMock` per `config/test.exs`. Both forms + # resolve to `Store.packages(Lightning.Adaptors)`; equality is always + # guaranteed regardless of cache state. + assert Adaptors.packages() == Adaptors.packages(Lightning.Adaptors) + end + end + + describe "versions/2" do + test "delegates to Store.versions/2 and returns version list", %{sup: sup} do + stub(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn _ -> + {:error, :unreachable} + end) + + {:ok, _} = AdaptorsRepo.upsert_adaptor(adaptor_record()) + + assert {:ok, [v]} = Adaptors.versions(sup, "@openfn/language-http") + assert v.version == "1.0.0" + end + + test "returns {:error, _} for unknown adaptor when strategy unavailable", %{ + sup: sup + } do + stub(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn _ -> + {:error, :not_found} + end) + + assert {:error, _} = Adaptors.versions(sup, "@openfn/does-not-exist") + end + end + + describe "schema/2" do + test "delegates to Store.schema/2 and returns schema", %{sup: sup} do + stub(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn _ -> + {:error, :unreachable} + end) + + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record(schema_data: ~s({"type":"object"})) + ) + + assert {:ok, ~s({"type":"object"})} = + Adaptors.schema(sup, "@openfn/language-http") + end + + test "preserves JSON property order across the DB round-trip", %{sup: sup} do + stub(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn _ -> + {:error, :unreachable} + end) + + ordered_body = ~s({"a":1,"z":2,"m":3}) + + {:ok, _} = + AdaptorsRepo.upsert_adaptor(adaptor_record(schema_data: ordered_body)) + + assert {:ok, ^ordered_body} = + Adaptors.schema(sup, "@openfn/language-http") + end + end + + describe "resolve_version/2" do + test "\"latest\" resolves from DB and returns latest_version" do + {:ok, _} = + AdaptorsRepo.upsert_adaptor(adaptor_record(latest_version: "2.3.4")) + + assert {:ok, "2.3.4"} = + Adaptors.resolve_version("@openfn/language-http", "latest") + end + + test "\"local\" resolves from DB and returns latest_version" do + {:ok, _} = + AdaptorsRepo.upsert_adaptor(adaptor_record(latest_version: "1.5.0")) + + assert {:ok, "1.5.0"} = + Adaptors.resolve_version("@openfn/language-http", "local") + end + + test "\"latest\" returns {:error, :not_found} when adaptor absent from DB" do + assert {:error, :not_found} = + Adaptors.resolve_version("@openfn/does-not-exist", "latest") + end + + test "concrete semver passes through without any DB lookup" do + # No adaptor in DB: if a lookup occurred the result would be :not_found. + # Pass-through means we get {:ok, version} regardless. + assert {:ok, "3.0.0"} = + Adaptors.resolve_version("@openfn/language-http", "3.0.0") + end + end + + describe "refresh_now/1" do + test "delegates to Scheduler.refresh_now via global_scheduler_name/1", %{ + sup: sup + } do + test_pid = self() + + # list_adaptors is called by the background Task that :tick spawns. + # With an empty DB the scheduler fires an init-tick immediately, so + # we must stub before start_scheduler and drain that first tick before + # calling refresh_now (which triggers a second tick). + stub(Lightning.Adaptors.StrategyMock, :list_adaptors, fn -> + send(test_pid, :tick_ran) + {:ok, []} + end) + + start_scheduler(sup) + assert_receive :tick_ran, 2000 + + assert :ok = Adaptors.refresh_now(sup) + assert_receive :tick_ran, 2000 + end + end + + describe "refresh_package/2" do + test "delegates to Scheduler.refresh_package via global_scheduler_name/1", %{ + sup: sup + } do + stub(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn _name -> + {:ok, adaptor_record(latest_version: "2.0.0")} + end) + + start_scheduler(sup) + + assert :ok = Adaptors.refresh_package(sup, "@openfn/language-http") + end + end + + describe "icon_meta/1,2" do + test "icon_meta is @doc false for all arities" do + {:docs_v1, _, :elixir, _, _, _, docs} = Code.fetch_docs(Lightning.Adaptors) + + icon_meta_docs = + Enum.filter(docs, fn + {{:function, :icon_meta, _}, _, _, _, _} -> true + _ -> false + end) + + refute Enum.empty?(icon_meta_docs) + + Enum.each(icon_meta_docs, fn doc -> + assert {{:function, :icon_meta, _}, _, _, :hidden, _} = doc + end) + end + + test "icon_meta/2 delegates to Store.icon_meta/2 for known adaptor", %{ + sup: sup + } do + {:ok, _} = + AdaptorsRepo.upsert_adaptor( + adaptor_record( + icon_square_ext: "svg", + icon_square_sha256: :crypto.hash(:sha256, "fake-svg-bytes") + ) + ) + + assert {:ok, meta} = Adaptors.icon_meta(sup, "@openfn/language-http") + assert meta.icon_square_ext == "svg" + end + + test "icon_meta/2 returns {:error, :not_found} for unknown adaptor", %{ + sup: sup + } do + assert {:error, :not_found} = + Adaptors.icon_meta(sup, "@openfn/never-existed") + end + end +end diff --git a/test/lightning/ai_assistant/ai_assistant_test.exs b/test/lightning/ai_assistant/ai_assistant_test.exs index d57b1e3f158..07e5aba6ba4 100644 --- a/test/lightning/ai_assistant/ai_assistant_test.exs +++ b/test/lightning/ai_assistant/ai_assistant_test.exs @@ -672,7 +672,7 @@ defmodule Lightning.AiAssistantTest do assert session.expression == job_1.body assert session.adaptor == - Lightning.AdaptorRegistry.resolve_adaptor(job_1.adaptor) + Lightning.Adaptors.PackageName.to_wire(job_1.adaptor) assert length(session.messages) == 1 message = hd(session.messages) @@ -1146,7 +1146,7 @@ defmodule Lightning.AiAssistantTest do assert updated_session.expression == expression assert updated_session.adaptor == - Lightning.AdaptorRegistry.resolve_adaptor(adaptor) + Lightning.Adaptors.PackageName.to_wire(adaptor) end end @@ -1270,7 +1270,7 @@ defmodule Lightning.AiAssistantTest do assert enriched.expression == job.body assert enriched.adaptor == - Lightning.AdaptorRegistry.resolve_adaptor(job.adaptor) + Lightning.Adaptors.PackageName.to_wire(job.adaptor) end test "adds run logs when follow_run_id is in meta", %{ @@ -1436,7 +1436,7 @@ defmodule Lightning.AiAssistantTest do assert enriched.expression == "console.log('test');" assert enriched.adaptor == - Lightning.AdaptorRegistry.resolve_adaptor( + Lightning.Adaptors.PackageName.to_wire( "@openfn/language-http@latest" ) end diff --git a/test/lightning/credentials/schema_test.exs b/test/lightning/credentials/schema_test.exs index 8ac4fd34c2a..0c7ff12be8c 100644 --- a/test/lightning/credentials/schema_test.exs +++ b/test/lightning/credentials/schema_test.exs @@ -2,6 +2,7 @@ defmodule Lightning.Credentials.SchemaTest do use Lightning.DataCase, async: true import ExUnit.CaptureLog + import Lightning.Factories import Mox alias Lightning.Credentials @@ -16,6 +17,17 @@ defmodule Lightning.Credentials.SchemaTest do :ok end + defp seed_adaptor_schema(name) do + # Persist the raw JSON binary so the credential-form renderer can + # re-engage `Jason.decode!(_, objects: :ordered_objects)` and + # preserve the schema author's property order. + schema_body = + Path.join(["test", "fixtures", "schemas", "#{name}.json"]) + |> File.read!() + + insert(:adaptor, name: name, source: :npm, schema_data: schema_body) + end + setup do schema_map = """ @@ -301,7 +313,46 @@ defmodule Lightning.Credentials.SchemaTest do end end + describe "Credentials.get_schema/1" do + setup do + Lightning.AdaptorTestHelpers.clear_global_adaptors_cache() + :ok + end + + test "preserves JSON property order from the persisted schema body" do + ordered_body = ~s({ + "properties": { + "zeta": {"type": "string"}, + "alpha": {"type": "string"}, + "mu": {"type": "string"} + }, + "type": "object" + }) + + insert(:adaptor, + name: "ordered-fixture", + source: :npm, + schema_data: ordered_body + ) + + Lightning.AdaptorTestHelpers.clear_global_adaptors_cache() + + stub(Lightning.Adaptors.StrategyMock, :fetch_adaptor, fn _ -> + {:error, :unreachable} + end) + + schema = Credentials.get_schema("ordered-fixture") + + assert schema.fields == [:zeta, :alpha, :mu] + end + end + describe "validate/2" do + setup do + Enum.each(~w(godata postgresql http dhis2), &seed_adaptor_schema/1) + :ok + end + test "successfully validates field with json schema email format" do schema = Credentials.get_schema("godata") diff --git a/test/lightning/credentials_test.exs b/test/lightning/credentials_test.exs index e58ad8a5127..03aba555361 100644 --- a/test/lightning/credentials_test.exs +++ b/test/lightning/credentials_test.exs @@ -277,6 +277,14 @@ defmodule Lightning.CredentialsTest do end describe "create_credential/1" do + setup do + # `Credentials.create_credential/1` calls `get_schema/1` which is + # now routed through `Lightning.Adaptors.schema/1`. Seed the + # `postgresql` fixture (used by the body-casting test below). + Lightning.AdaptorTestHelpers.seed_credential_schema("postgresql") + :ok + end + test "fails if another cred exists with the same name for the same user" do valid_attrs = %{ body: %{"a" => "test"}, @@ -422,6 +430,11 @@ defmodule Lightning.CredentialsTest do end describe "update_credential/2" do + setup do + Lightning.AdaptorTestHelpers.seed_credential_schema("postgresql") + :ok + end + test "updates an OAuth credential with new scopes" do user = insert(:user) oauth_client = insert(:oauth_client) diff --git a/test/lightning_web/channels/run_channel_test.exs b/test/lightning_web/channels/run_channel_test.exs index 5288b0b3cae..4523dd40ffb 100644 --- a/test/lightning_web/channels/run_channel_test.exs +++ b/test/lightning_web/channels/run_channel_test.exs @@ -117,6 +117,22 @@ defmodule LightningWeb.RunChannelTest do setup :set_google_credential setup :create_socket_and_run + # `RunWithOptions.render/1` now uses + # `Lightning.Adaptors.PackageName.to_wire/1`, which resolves + # `@latest` against `Lightning.Adaptors.resolve_version/2` (a direct + # `Repo.get_adaptor/2` read — async-safe under the Ecto sandbox). + # Seed `language-common` with the version the legacy registry + # fixture used to publish. + setup do + insert(:adaptor, + name: "@openfn/language-common", + source: :npm, + latest_version: "1.6.2" + ) + + :ok + end + test "fetch:plan success", %{ socket: socket, run: run, diff --git a/test/lightning_web/channels/run_with_options_test.exs b/test/lightning_web/channels/run_with_options_test.exs index 8961938f5a5..7c5f99ecbe7 100644 --- a/test/lightning_web/channels/run_with_options_test.exs +++ b/test/lightning_web/channels/run_with_options_test.exs @@ -1,5 +1,5 @@ defmodule LightningWeb.RunWithOptionsTest do - use Lightning.DataCase, async: true + use Lightning.DataCase, async: false import Lightning.Factories @@ -9,6 +9,23 @@ defmodule LightningWeb.RunWithOptionsTest do alias LightningWeb.RunWithOptions describe "rendering a run" do + setup do + # Clear the production Adaptors.Supervisor Cachex so each test's seeded + # rows are visible (Cachex persists across DB-sandbox boundaries). + cache = Lightning.Adaptors.Supervisor.cache_name(Lightning.Adaptors) + Cachex.clear(cache) + + # Seed @openfn/language-common so `@latest` resolves to a concrete + # semver via `Lightning.Adaptors.PackageName.to_wire/1`. + insert(:adaptor, + name: "@openfn/language-common", + source: :npm, + latest_version: "1.6.2" + ) + + :ok + end + test "renders a workflow using a snapshot" do user = insert(:user) @@ -109,14 +126,25 @@ defmodule LightningWeb.RunWithOptionsTest do expected_result end - @tag :tmp_dir - test "renders adaptors with @local when local_daptors_repo is configured", %{ - tmp_dir: tmp_dir - } do - Mox.stub(Lightning.MockConfig, :adaptor_registry, fn -> - [local_adaptors_repo: tmp_dir] + test "renders adaptors with @local when :local strategy source is active" do + prev = Application.get_env(:lightning, Lightning.Adaptors, []) + + Application.put_env( + :lightning, + Lightning.Adaptors, + Keyword.put(prev, :strategy, Lightning.Adaptors.Local) + ) + + on_exit(fn -> + Application.put_env(:lightning, Lightning.Adaptors, prev) end) + insert(:adaptor, + name: "@openfn/language-common", + source: :local, + latest_version: "local" + ) + user = insert(:user) {:ok, %{triggers: [trigger], jobs: [job]} = workflow} = diff --git a/test/lightning_web/channels/workflow_channel_test.exs b/test/lightning_web/channels/workflow_channel_test.exs index 671bcf6c7c0..790838a0a4a 100644 --- a/test/lightning_web/channels/workflow_channel_test.exs +++ b/test/lightning_web/channels/workflow_channel_test.exs @@ -88,89 +88,114 @@ defmodule LightningWeb.WorkflowChannelTest do end describe "request_adaptors and request_credentials" do + setup do + # The production Adaptors.Supervisor's Cachex persists across tests; + # clear it so each test's seeded Adaptors.Repo rows are visible. + cache = Lightning.Adaptors.Supervisor.cache_name(Lightning.Adaptors) + Cachex.clear(cache) + + # Seed Adaptors.Repo rows so packages/0 returns a non-empty list. + # Individual tests insert additional rows for icon-meta assertions. + insert(:adaptor, name: "@openfn/language-salesforce", source: :npm) + insert(:adaptor, name: "@openfn/language-http", source: :npm) + :ok + end + test "handles multiple concurrent requests independently", %{ socket: socket } do ref_adaptors = push(socket, "request_adaptors", %{}) ref_credentials = push(socket, "request_credentials", %{}) - assert_reply ref_adaptors, :ok, %{adaptors: _} + assert_reply ref_adaptors, :ok, %{adaptors: adaptors} assert_reply ref_credentials, :ok, %{credentials: credentials} + assert is_list(adaptors) + assert adaptors != [] + assert Enum.all?(adaptors, &Map.has_key?(&1, :icon_urls)) + + assert Enum.all?(adaptors, fn a -> + Enum.sort(Map.keys(a.icon_urls)) == [:rectangle, :square] + end) + assert Map.has_key?(credentials, :project_credentials) assert Map.has_key?(credentials, :keychain_credentials) assert is_list(credentials.project_credentials) assert is_list(credentials.keychain_credentials) end - test "returns project-specific adaptors", %{socket: socket, project: project} do - # Create jobs with specific adaptors in this project - workflow = insert(:workflow, project: project) - - insert(:job, - workflow: workflow, - adaptor: "@openfn/language-salesforce@latest" + test "request_adaptors enriches records with icon_urls when meta present", + %{socket: socket} do + name = "@openfn/language-common" + square_sha = :crypto.strong_rand_bytes(32) + rectangle_sha = :crypto.strong_rand_bytes(32) + + insert(:adaptor, + name: name, + icon_square_ext: "png", + icon_square_sha256: square_sha, + icon_rectangle_ext: "svg", + icon_rectangle_sha256: rectangle_sha ) - insert(:job, workflow: workflow, adaptor: "@openfn/language-http@2.0.0") + ref = push(socket, "request_adaptors", %{}) + assert_reply ref, :ok, %{adaptors: adaptors} - ref = push(socket, "request_project_adaptors", %{}) + record = Enum.find(adaptors, &(&1.name == name)) + assert record, "expected legacy registry to include #{name}" - assert_reply ref, :ok, %{ - project_adaptors: project_adaptors, - all_adaptors: all_adaptors - } + {:ok, meta} = Lightning.Adaptors.icon_meta(name) - assert is_list(project_adaptors) - assert is_list(all_adaptors) + assert record.icon_urls.square == + LightningWeb.AdaptorIconURL.build(name, meta, :square) - # Verify project_adaptors contains only adaptors used in the project - project_adaptor_names = Enum.map(project_adaptors, & &1.name) - assert "@openfn/language-salesforce" in project_adaptor_names - assert "@openfn/language-http" in project_adaptor_names + assert record.icon_urls.rectangle == + LightningWeb.AdaptorIconURL.build(name, meta, :rectangle) - # Verify all_adaptors contains the full registry - assert length(all_adaptors) > 0 + assert is_binary(record.icon_urls.square) + assert is_binary(record.icon_urls.rectangle) end - test "returns empty project_adaptors for project with no jobs", %{ - socket: socket - } do - ref = push(socket, "request_project_adaptors", %{}) + test "request_adaptors emits nil icon_urls when row has no icon meta", + %{socket: socket} do + name = "@openfn/language-dhis2" + + insert(:adaptor, + name: name, + source: :npm, + icon_square_ext: nil, + icon_square_sha256: nil, + icon_rectangle_ext: nil, + icon_rectangle_sha256: nil + ) - assert_reply ref, :ok, %{ - project_adaptors: project_adaptors, - all_adaptors: all_adaptors - } + ref = push(socket, "request_adaptors", %{}) + assert_reply ref, :ok, %{adaptors: adaptors} - assert project_adaptors == [] - assert is_list(all_adaptors) - assert length(all_adaptors) > 0 + record = Enum.find(adaptors, &(&1.name == name)) + assert record, "expected packages/0 to include #{name}" + assert record.icon_urls == %{square: nil, rectangle: nil} end - test "handles duplicate adaptors in project", %{ - socket: socket, - project: project - } do - workflow = insert(:workflow, project: project) + test "request_adaptors handles half-populated icon meta", %{socket: socket} do + name = "@openfn/language-commcare" + square_sha = :crypto.strong_rand_bytes(32) - # Create multiple jobs with the same adaptor - insert(:job, - workflow: workflow, - adaptor: "@openfn/language-common@latest" + insert(:adaptor, + name: name, + icon_square_ext: "png", + icon_square_sha256: square_sha, + icon_rectangle_ext: nil, + icon_rectangle_sha256: nil ) - insert(:job, workflow: workflow, adaptor: "@openfn/language-common@1.0.0") - - ref = push(socket, "request_project_adaptors", %{}) - - assert_reply ref, :ok, %{project_adaptors: project_adaptors} + ref = push(socket, "request_adaptors", %{}) + assert_reply ref, :ok, %{adaptors: adaptors} - # Should only appear once in project_adaptors - common_adaptors = - Enum.filter(project_adaptors, &(&1.name == "@openfn/language-common")) - - assert length(common_adaptors) <= 1 + record = Enum.find(adaptors, &(&1.name == name)) + assert record, "expected legacy registry to include #{name}" + assert is_binary(record.icon_urls.square) + assert record.icon_urls.rectangle == nil end test "returns correctly structured project credentials", %{ @@ -1577,6 +1602,59 @@ defmodule LightningWeb.WorkflowChannelTest do end end + describe "PubSub subscription and adaptors broadcasting" do + test "forwards adaptors_updated envelope from client topic to socket", %{ + socket: _socket + } do + payload = %{adaptors: [%{name: "a"}]} + + Phoenix.PubSub.broadcast( + Lightning.PubSub, + Lightning.Adaptors.Supervisor.client_topic(Lightning.Adaptors), + %{event: "adaptors_updated", payload: payload} + ) + + assert_push "adaptors_updated", %{adaptors: [%{name: "a"}]} + end + + test "credentials_updated forwarder still pushes after adaptors clause added", + %{workflow: workflow} do + rendered_credentials = %{ + project_credentials: [], + keychain_credentials: [] + } + + Phoenix.PubSub.broadcast( + Lightning.PubSub, + "workflow:collaborate:#{workflow.id}", + %{event: "credentials_updated", payload: rendered_credentials} + ) + + assert_push "credentials_updated", %{ + project_credentials: [], + keychain_credentials: [] + } + end + + test "does not push adaptors_updated for unrelated events on client topic", + %{socket: socket} do + Process.flag(:trap_exit, true) + Process.unlink(socket.channel_pid) + ref = Process.monitor(socket.channel_pid) + + capture_log(fn -> + Phoenix.PubSub.broadcast( + Lightning.PubSub, + Lightning.Adaptors.Supervisor.client_topic(Lightning.Adaptors), + %{event: "something_else", payload: %{}} + ) + + refute_push "adaptors_updated", _, 50 + assert_receive {:DOWN, ^ref, :process, _, _}, 200 + end) + end + end + describe "request_history" do test "returns work orders with runs for workflow", %{ socket: socket, diff --git a/test/lightning_web/controllers/adaptor_icon_controller_test.exs b/test/lightning_web/controllers/adaptor_icon_controller_test.exs new file mode 100644 index 00000000000..3825a97a188 --- /dev/null +++ b/test/lightning_web/controllers/adaptor_icon_controller_test.exs @@ -0,0 +1,591 @@ +defmodule LightningWeb.AdaptorIconControllerTest do + # async: false — all tests share the Lightning.Adaptors supervisor name. + use LightningWeb.ConnCase, async: false + + import Mox + + alias Lightning.Adaptors.IconCache + alias Lightning.Adaptors.Repo, as: AdaptorsRepo + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + alias LightningWeb.AdaptorIconController + alias LightningWeb.AdaptorIconURL + + setup :verify_on_exit! + + # The production `Lightning.Adaptors.Supervisor` is started in + # `application.ex` under the name `Lightning.Adaptors` and — in test — + # uses `Lightning.Adaptors.StrategyMock` (see `config/test.exs`). No + # per-test supervisor start is needed. + + # --------------------------------------------------------------------------- + # Helpers + # --------------------------------------------------------------------------- + + defp sha8_from_bytes(bytes) do + :crypto.hash(:sha256, bytes) + |> binary_part(0, 4) + |> Base.encode16(case: :lower) + end + + defp source, do: AdaptorsSupervisor.source(Lightning.Adaptors) + + defp unique_adaptor_name do + "@openfn/language-test-#{System.unique_integer([:positive])}" + end + + defp insert_adaptor(name, overrides \\ %{}) do + attrs = + Map.merge( + %{ + name: name, + source: :npm, + latest_version: "1.0.0", + deprecated: false + }, + overrides + ) + + {:ok, _adaptor} = AdaptorsRepo.upsert_adaptor(attrs) + end + + defp write_icon(name, shape, ext, bytes) do + {:ok, _sha} = IconCache.write!(source(), name, shape, ext, bytes) + :ok + end + + # --------------------------------------------------------------------------- + # 200 — sha matches, disk warm + # --------------------------------------------------------------------------- + + describe "show/2 — match + warm disk" do + test "returns 200 with immutable Cache-Control", %{conn: conn} do + name = unique_adaptor_name() + bytes = "png icon bytes" + sha256 = :crypto.hash(:sha256, bytes) + sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_square_ext: "png", + icon_square_sha256: sha256 + }) + + write_icon(name, :square, "png", bytes) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => sha8, + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 200 + + assert get_resp_header(result, "cache-control") == [ + "public, max-age=31536000, immutable" + ] + + assert result.resp_body == bytes + end + + test "serves correct Content-Type for png", %{conn: conn} do + name = unique_adaptor_name() + bytes = "png bytes" + sha256 = :crypto.hash(:sha256, bytes) + sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_square_ext: "png", + icon_square_sha256: sha256 + }) + + write_icon(name, :square, "png", bytes) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => sha8, + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 200 + [ct] = get_resp_header(result, "content-type") + assert ct =~ "image/png" + end + + test "serves correct Content-Type for svg", %{conn: conn} do + name = unique_adaptor_name() + bytes = "" + sha256 = :crypto.hash(:sha256, bytes) + sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_square_ext: "svg", + icon_square_sha256: sha256 + }) + + write_icon(name, :square, "svg", bytes) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => sha8, + "ext" => "svg" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 200 + [ct] = get_resp_header(result, "content-type") + assert ct =~ "image/svg+xml" + end + + test "sha8 is case-insensitive on input", %{conn: conn} do + name = unique_adaptor_name() + bytes = "case test bytes" + sha256 = :crypto.hash(:sha256, bytes) + sha8_lower = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + sha8_upper = String.upcase(sha8_lower) + + insert_adaptor(name, %{ + icon_square_ext: "png", + icon_square_sha256: sha256 + }) + + write_icon(name, :square, "png", bytes) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => sha8_upper, + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 200 + end + + test "works for rectangle shape", %{conn: conn} do + name = unique_adaptor_name() + bytes = "rect bytes" + sha256 = :crypto.hash(:sha256, bytes) + sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_rectangle_ext: "png", + icon_rectangle_sha256: sha256 + }) + + write_icon(name, :rectangle, "png", bytes) + + params = %{ + "name" => name, + "shape" => "rectangle", + "sha8" => sha8, + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 200 + assert result.resp_body == bytes + end + end + + # --------------------------------------------------------------------------- + # 200 — sha matches, disk cold (strategy fetches) + # --------------------------------------------------------------------------- + + describe "show/2 — match + cold disk" do + test "strategy is called, bytes written to disk, 200 returned", %{conn: conn} do + name = unique_adaptor_name() + bytes = "cold icon bytes" + sha256 = :crypto.hash(:sha256, bytes) + sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_square_ext: "png", + icon_square_sha256: sha256 + }) + + expect( + Lightning.Adaptors.StrategyMock, + :fetch_icon, + 1, + fn ^name, :square -> {:ok, %{data: bytes, ext: "png"}} end + ) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => sha8, + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 200 + + assert get_resp_header(result, "cache-control") == [ + "public, max-age=31536000, immutable" + ] + + assert result.resp_body == bytes + end + end + + # --------------------------------------------------------------------------- + # 302 — stale sha8, current icon exists + # --------------------------------------------------------------------------- + + describe "show/2 — stale sha8" do + test "redirects 302 with no-store and current Location", %{conn: conn} do + name = unique_adaptor_name() + bytes = "current icon bytes" + sha256 = :crypto.hash(:sha256, bytes) + current_sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_square_ext: "png", + icon_square_sha256: sha256 + }) + + stale_sha8 = "00000000" + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => stale_sha8, + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 302 + assert get_resp_header(result, "cache-control") == ["no-store"] + + encoded_name = URI.encode(name, &URI.char_unreserved?/1) + + assert get_resp_header(result, "location") == [ + "/adaptors/icons/#{encoded_name}/square-#{current_sha8}.png" + ] + end + + test "Location URL is lowercase even when sha8 input was uppercase", %{ + conn: conn + } do + name = unique_adaptor_name() + bytes = "case url bytes" + sha256 = :crypto.hash(:sha256, bytes) + current_sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_square_ext: "png", + icon_square_sha256: sha256 + }) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => "FFFFFFFF", + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 302 + [location] = get_resp_header(result, "location") + + # sha8 segment is lowercase; percent-encoded chars use uppercase hex per RFC 3986 + assert location =~ "square-#{current_sha8}.png" + end + + test "Location matches what AdaptorIconURL.build/3 would emit", %{conn: conn} do + name = unique_adaptor_name() + bytes = "channel sync bytes" + sha256 = :crypto.hash(:sha256, bytes) + + insert_adaptor(name, %{ + icon_square_ext: "png", + icon_square_sha256: sha256 + }) + + meta = %{ + icon_square_ext: "png", + icon_square_sha256: sha256, + icon_rectangle_ext: nil, + icon_rectangle_sha256: nil + } + + expected_url = AdaptorIconURL.build(name, meta, :square) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => "00000000", + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 302 + assert get_resp_header(result, "location") == [expected_url] + end + end + + # --------------------------------------------------------------------------- + # 404 cases + # --------------------------------------------------------------------------- + + describe "show/2 — 404" do + test "adaptor not in DB", %{conn: conn} do + params = %{ + "name" => "nonexistent-adaptor-#{System.unique_integer([:positive])}", + "shape" => "square", + "sha8" => "aabbccdd", + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 404 + end + + test "ext mismatch — DB has png, URL says svg", %{conn: conn} do + name = unique_adaptor_name() + sha256 = :crypto.hash(:sha256, "some bytes") + sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_square_ext: "png", + icon_square_sha256: sha256 + }) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => sha8, + "ext" => "svg" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 404 + end + + test "ext mismatch — DB has svg, URL says png", %{conn: conn} do + name = unique_adaptor_name() + sha256 = :crypto.hash(:sha256, "") + sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_square_ext: "svg", + icon_square_sha256: sha256 + }) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => sha8, + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 404 + end + + test "stored ext is nil (no icon for shape)", %{conn: conn} do + name = unique_adaptor_name() + insert_adaptor(name) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => "aabbccdd", + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 404 + end + + test "bad shape value — not square or rectangle", %{conn: conn} do + params = %{ + "name" => unique_adaptor_name(), + "shape" => "circle", + "sha8" => "aabbccdd", + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 404 + end + + test "missing params — fallback clause", %{conn: conn} do + result = AdaptorIconController.show(conn, %{"name" => "something"}) + + assert result.status == 404 + end + + test "stale sha but stored ext is nil (icon removed upstream) — no redirect", + %{ + conn: conn + } do + # Adaptor row exists but has no icon for the square shape. + # Even though there's a stale sha8 in the URL, there's no canonical + # URL to redirect to — 404 instead of 302. + name = unique_adaptor_name() + insert_adaptor(name) + + params = %{ + "name" => name, + "shape" => "square", + "sha8" => "00000000", + "ext" => "png" + } + + result = AdaptorIconController.show(conn, params) + + assert result.status == 404 + end + end + + # --------------------------------------------------------------------------- + # Full router pipeline — proves route + pipe + controller resolves. + # The matrix above already covers the controller's behaviour by direct call. + # --------------------------------------------------------------------------- + + describe "GET /adaptors/icons/... (full router pipeline)" do + test "200 on sha match", %{conn: conn} do + name = unique_adaptor_name() + bytes = "router pipeline bytes" + sha256 = :crypto.hash(:sha256, bytes) + sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + insert_adaptor(name, %{ + icon_square_ext: "png", + icon_square_sha256: sha256 + }) + + write_icon(name, :square, "png", bytes) + + encoded = URI.encode(name, &URI.char_unreserved?/1) + conn = get(conn, "/adaptors/icons/#{encoded}/square-#{sha8}.png") + + assert conn.status == 200 + assert conn.resp_body == bytes + end + + test "404 on unknown adaptor", %{conn: conn} do + conn = get(conn, "/adaptors/icons/nope/square-aabbccdd.png") + + assert conn.status == 404 + end + end + + # --------------------------------------------------------------------------- + # AdaptorIconURL.build/3 + # --------------------------------------------------------------------------- + + describe "AdaptorIconURL.build/3" do + test "returns nil when ext is nil" do + meta = %{ + icon_square_ext: nil, + icon_square_sha256: :crypto.hash(:sha256, "x"), + icon_rectangle_ext: nil, + icon_rectangle_sha256: nil + } + + assert AdaptorIconURL.build("@openfn/language-http", meta, :square) == nil + end + + test "returns nil when sha256 is nil" do + meta = %{ + icon_square_ext: "png", + icon_square_sha256: nil, + icon_rectangle_ext: nil, + icon_rectangle_sha256: nil + } + + assert AdaptorIconURL.build("@openfn/language-http", meta, :square) == nil + end + + test "URL-encodes slashes and @ in adaptor name" do + sha256 = :crypto.hash(:sha256, "bytes") + + meta = %{ + icon_square_ext: "png", + icon_square_sha256: sha256, + icon_rectangle_ext: nil, + icon_rectangle_sha256: nil + } + + url = AdaptorIconURL.build("@openfn/language-http", meta, :square) + + refute is_nil(url) + assert url =~ "%40openfn%2Flanguage-http" + end + + test "sha8 in URL is always 8 lowercase hex chars" do + sha256 = :crypto.hash(:sha256, "bytes") + expected_sha8 = sha256 |> binary_part(0, 4) |> Base.encode16(case: :lower) + + meta = %{ + icon_square_ext: "png", + icon_square_sha256: sha256, + icon_rectangle_ext: nil, + icon_rectangle_sha256: nil + } + + url = AdaptorIconURL.build("@openfn/language-http", meta, :square) + + # sha8 is lowercase; percent-encoded chars use uppercase hex per RFC 3986 + assert url =~ "square-#{expected_sha8}.png" + assert expected_sha8 == String.downcase(expected_sha8) + end + + test "builds distinct URLs for square and rectangle shapes" do + sq_sha = :crypto.hash(:sha256, "square bytes") + rect_sha = :crypto.hash(:sha256, "rectangle bytes") + + meta = %{ + icon_square_ext: "png", + icon_square_sha256: sq_sha, + icon_rectangle_ext: "svg", + icon_rectangle_sha256: rect_sha + } + + sq_url = AdaptorIconURL.build("@openfn/language-http", meta, :square) + rect_url = AdaptorIconURL.build("@openfn/language-http", meta, :rectangle) + + assert sq_url =~ "square-" + assert rect_url =~ "rectangle-" + assert sq_url =~ ".png" + assert rect_url =~ ".svg" + refute sq_url == rect_url + end + + test "sha8 matches what show/2 computes from the same sha256" do + bytes = "verify sha8 computation" + sha256 = :crypto.hash(:sha256, bytes) + + meta = %{ + icon_square_ext: "png", + icon_square_sha256: sha256, + icon_rectangle_ext: nil, + icon_rectangle_sha256: nil + } + + url = AdaptorIconURL.build("name", meta, :square) + + assert sha8_from_bytes(bytes) in String.split(url, ["-", "."]) + end + end +end diff --git a/test/lightning_web/live/credential_live_test.exs b/test/lightning_web/live/credential_live_test.exs index b1151cebcd5..e9110ca7bed 100644 --- a/test/lightning_web/live/credential_live_test.exs +++ b/test/lightning_web/live/credential_live_test.exs @@ -42,6 +42,14 @@ defmodule LightningWeb.CredentialLiveTest do setup :register_and_log_in_user setup :create_project_for_current_user + # Seed credential schemas into Lightning.Adaptors.Repo so + # `Credentials.get_schema/1` (now backed by `Lightning.Adaptors.schema/1`) + # finds them via the DB rather than calling the Strategy mock. + setup do + Lightning.AdaptorTestHelpers.seed_all_credential_schemas() + :ok + end + defp get_decoded_state(url) when is_nil(url) do [ "test", diff --git a/test/lightning_web/live/job_live/adaptor_picker_test.exs b/test/lightning_web/live/job_live/adaptor_picker_test.exs index a293d198054..a20c5fdb9e4 100644 --- a/test/lightning_web/live/job_live/adaptor_picker_test.exs +++ b/test/lightning_web/live/job_live/adaptor_picker_test.exs @@ -1,9 +1,46 @@ defmodule LightningWeb.JobLive.AdaptorPickerTest do - use LightningWeb.ConnCase, async: true + # async: false because we touch the production-named + # `Lightning.Adaptors` supervisor's shared Cachex. + use LightningWeb.ConnCase, async: false + + import Lightning.AdaptorTestHelpers alias LightningWeb.JobLive.AdaptorPicker describe "get_adaptor_version_options/1" do + setup do + {:ok, _} = + Lightning.Adaptors.Repo.upsert_adaptor(%{ + name: "@openfn/language-common", + source: :npm, + latest_version: "1.6.2", + description: nil, + homepage: nil, + repository: nil, + license: nil, + deprecated: false, + schema_data: nil, + schema_sha256: nil, + versions: + Enum.map(["1.0.0", "1.6.2", "2.0.0"], fn v -> + %{ + version: v, + integrity: "sha512-#{v}", + tarball_url: "https://example.com/x-#{v}.tgz", + size_bytes: 1024, + dependencies: %{}, + peer_dependencies: %{}, + published_at: nil, + deprecated: false + } + end) + }) + + clear_global_adaptors_cache() + + :ok + end + test "returns sorted versions with latest option when adaptor exists" do adaptor_name = "@openfn/language-common" diff --git a/test/lightning_web/live/maintenance_live/index_test.exs b/test/lightning_web/live/maintenance_live/index_test.exs new file mode 100644 index 00000000000..68c31bac8a3 --- /dev/null +++ b/test/lightning_web/live/maintenance_live/index_test.exs @@ -0,0 +1,79 @@ +defmodule LightningWeb.MaintenanceLive.IndexTest do + # async: false because the icon-refresh test stubs Lightning.Adaptors.StrategyMock + # globally, which the singleton Scheduler GenServer (not in the test pid's + # caller chain) needs to see. + use LightningWeb.ConnCase, async: false + + import Mox + import Phoenix.LiveViewTest + + setup :set_mox_global + + describe "Index as a regular user" do + setup :register_and_log_in_user + + test "cannot access the maintenance page", %{conn: conn} do + {:ok, _live, html} = + live(conn, ~p"/settings/maintenance", on_error: :raise) + |> follow_redirect(conn, "/projects") + + assert html =~ "Sorry, you don't have access to that." + end + end + + describe "Index as a superuser" do + setup :register_and_log_in_superuser + + test "renders the Refresh Adaptor Registry card", %{conn: conn} do + {:ok, _live, html} = + live(conn, ~p"/settings/maintenance", on_error: :raise) + + assert html =~ "Maintenance" + assert html =~ "Refresh Adaptor Registry" + assert html =~ "Re-fetch the list of available adaptors" + assert html =~ "Run" + end + + test "clicking Run flashes that the refresh was queued", %{conn: conn} do + {:ok, live, _html} = + live(conn, ~p"/settings/maintenance", on_error: :raise) + + live + |> element("#refresh-adaptors-button") + |> render_click() + + assert has_element?( + live, + "p[role=alert][phx-value-key=info]", + "Adaptor refresh queued." + ) + end + + test "renders the Refresh Adaptor Icons card", %{conn: conn} do + {:ok, live, html} = + live(conn, ~p"/settings/maintenance", on_error: :raise) + + assert html =~ "Refresh Adaptor Icons" + assert has_element?(live, "#refresh-icons-button") + end + + test "clicking the icons button reports the refresh result", %{conn: conn} do + stub(Lightning.Adaptors.StrategyMock, :fetch_icons, fn _opts -> + {:ok, %{}} + end) + + {:ok, live, _html} = + live(conn, ~p"/settings/maintenance", on_error: :raise) + + live + |> element("#refresh-icons-button") + |> render_click() + + assert has_element?( + live, + "p[role=alert][phx-value-key=info]", + "Icon refresh complete" + ) + end + end +end diff --git a/test/lightning_web/live/project_live_test.exs b/test/lightning_web/live/project_live_test.exs index fde52664cc0..75516be05f9 100644 --- a/test/lightning_web/live/project_live_test.exs +++ b/test/lightning_web/live/project_live_test.exs @@ -881,6 +881,14 @@ defmodule LightningWeb.ProjectLiveTest do setup :register_and_log_in_user setup :create_project_for_current_user + setup do + # Credential creation flows render the JsonSchemaBodyComponent + # which calls `Credentials.get_schema/1` — now backed by + # `Lightning.Adaptors.schema/1`. + Lightning.AdaptorTestHelpers.seed_credential_schema("http") + :ok + end + test "access project settings page", %{conn: conn, project: project} do {:ok, _view, html} = live(conn, ~p"/projects/#{project}/settings", on_error: :raise) diff --git a/test/lightning_web/live/workflow_live/collaborate_test.exs b/test/lightning_web/live/workflow_live/collaborate_test.exs index a98816facbe..006d1600204 100644 --- a/test/lightning_web/live/workflow_live/collaborate_test.exs +++ b/test/lightning_web/live/workflow_live/collaborate_test.exs @@ -901,6 +901,14 @@ defmodule LightningWeb.WorkflowLive.CollaborateTest do end describe "credential modal interactions" do + # `Credentials.get_schema/1` now reads through + # `Lightning.Adaptors.schema/1`; seed the `http` fixture so the + # JsonSchemaBodyComponent renders without raising. + setup do + Lightning.AdaptorTestHelpers.seed_credential_schema("http") + :ok + end + test "opens credential modal with schema via handle_event", %{conn: conn} do user = insert(:user) diff --git a/test/lightning_web/live/workflow_live/edit_test.exs b/test/lightning_web/live/workflow_live/edit_test.exs index 349f1b5c66f..3466dfc8c92 100644 --- a/test/lightning_web/live/workflow_live/edit_test.exs +++ b/test/lightning_web/live/workflow_live/edit_test.exs @@ -60,6 +60,12 @@ defmodule LightningWeb.WorkflowLive.EditTest do describe "New credential from project context " do setup %{project: project} do + # Seed the credential schemas the job inspector's new-credential + # modal renders. `Credentials.get_schema/1` now reads through + # `Lightning.Adaptors.schema/1`, which falls back to the Strategy + # mock without a seeded row. + Lightning.AdaptorTestHelpers.seed_credential_schema("http") + %{job: job} = workflow_job_fixture(project_id: project.id) workflow = Repo.get(Workflow, job.workflow_id) @@ -131,6 +137,18 @@ defmodule LightningWeb.WorkflowLive.EditTest do end describe "new" do + setup do + # The adaptor picker reads packages/0 and versions/1 through the + # production `Lightning.Adaptors` supervisor's Cachex. Seed the + # adaptors a default job needs, plus the credential schemas + # rendered by `JsonSchemaBodyComponent` inside the new-credential + # modal. + Lightning.AdaptorTestHelpers.seed_common_packages() + Lightning.AdaptorTestHelpers.seed_credential_schema("dhis2") + Lightning.AdaptorTestHelpers.seed_credential_schema("http") + :ok + end + test "builds a new workflow", %{conn: conn, project: project} do {:ok, view, _html} = live(conn, ~p"/projects/#{project.id}/w/new/legacy", on_error: :raise) @@ -1581,8 +1599,20 @@ defmodule LightningWeb.WorkflowLive.EditTest do workflow: workflow, tmp_dir: tmp_dir } do - Mox.stub(Lightning.MockConfig, :adaptor_registry, fn -> - [local_adaptors_repo: tmp_dir] + # The legacy `MockConfig.adaptor_registry` knob is gone — the + # picker now keys "local" mode off `Adaptors.Config.current_source/0`. + # Flip the strategy module to `Lightning.Adaptors.Local` for the + # duration of this test. + prev = Application.get_env(:lightning, Lightning.Adaptors, []) + + Application.put_env( + :lightning, + Lightning.Adaptors, + Keyword.put(prev, :strategy, Lightning.Adaptors.Local) + ) + + on_exit(fn -> + Application.put_env(:lightning, Lightning.Adaptors, prev) end) expected_adaptors = ["foo", "bar", "baz"] @@ -2524,6 +2554,11 @@ defmodule LightningWeb.WorkflowLive.EditTest do project: project, workflow: workflow } do + # The picker (production `Lightning.Adaptors` supervisor) needs + # to see `language-dhis2@3.0.4` as a known version for the + # `change_adaptor_version/2` form submit to validate. + Lightning.AdaptorTestHelpers.seed_common_packages() + project_credential = insert(:project_credential, project: project, diff --git a/test/lightning_web/live/workflow_live/editor_test.exs b/test/lightning_web/live/workflow_live/editor_test.exs index 0fdcea1d47b..19dd82c90f2 100644 --- a/test/lightning_web/live/workflow_live/editor_test.exs +++ b/test/lightning_web/live/workflow_live/editor_test.exs @@ -16,6 +16,19 @@ defmodule LightningWeb.WorkflowLive.EditorTest do setup :create_project_for_current_user setup :create_workflow + # `WorkflowLive.EditorPane` resolves `@latest` via + # `Lightning.Adaptors.PackageName.to_wire/1`, which reads + # `Repo.get_adaptor/2` directly (async-safe). + setup do + insert(:adaptor, + name: "@openfn/language-common", + source: :npm, + latest_version: "1.6.2" + ) + + :ok + end + test "can edit a jobs body", %{ project: project, workflow: workflow, diff --git a/test/mix/tasks/lightning.refresh_adaptors_test.exs b/test/mix/tasks/lightning.refresh_adaptors_test.exs new file mode 100644 index 00000000000..ae46ebc60a2 --- /dev/null +++ b/test/mix/tasks/lightning.refresh_adaptors_test.exs @@ -0,0 +1,85 @@ +defmodule Mix.Tasks.Lightning.RefreshAdaptorsTest do + use ExUnit.Case, async: false + use Mimic + + setup_all do + Mimic.copy(Lightning.Adaptors) + :ok + end + + setup do + Mix.shell(Mix.Shell.Process) + on_exit(fn -> Mix.shell(Mix.Shell.IO) end) + :ok + end + + describe "bare invocation" do + test "calls refresh_now/0 and exits 0 on :ok" do + stub(Lightning.Adaptors, :refresh_now, fn -> :ok end) + Mix.Tasks.Lightning.RefreshAdaptors.run([]) + assert_received {:mix_shell, :info, [_]} + end + + test "exits 2 on other error" do + stub(Lightning.Adaptors, :refresh_now, fn -> {:error, :network_down} end) + + assert catch_exit(Mix.Tasks.Lightning.RefreshAdaptors.run([])) == + {:shutdown, 2} + + assert_received {:mix_shell, :error, [_]} + end + end + + describe "--name flag" do + test "dispatches to refresh_package/1 with the exact package string" do + pkg = "@openfn/language-http" + stub(Lightning.Adaptors, :refresh_package, fn ^pkg -> :ok end) + Mix.Tasks.Lightning.RefreshAdaptors.run(["--name", pkg]) + assert_received {:mix_shell, :info, [_]} + end + + test "exits 1 on {:error, :not_found}" do + stub(Lightning.Adaptors, :refresh_package, fn _pkg -> + {:error, :not_found} + end) + + assert catch_exit( + Mix.Tasks.Lightning.RefreshAdaptors.run([ + "--name", + "@openfn/language-http" + ]) + ) == {:shutdown, 1} + + assert_received {:mix_shell, :error, [_]} + end + + test "exits 2 on other error" do + stub(Lightning.Adaptors, :refresh_package, fn _pkg -> + {:error, :timeout} + end) + + assert catch_exit( + Mix.Tasks.Lightning.RefreshAdaptors.run([ + "--name", + "@openfn/language-http" + ]) + ) == {:shutdown, 2} + + assert_received {:mix_shell, :error, [_]} + end + end + + describe "rejected flags" do + test "raises on unknown --strategy flag" do + assert_raise OptionParser.ParseError, fn -> + Mix.Tasks.Lightning.RefreshAdaptors.run(["--strategy", "local"]) + end + end + + test "raises on unknown --source flag" do + assert_raise OptionParser.ParseError, fn -> + Mix.Tasks.Lightning.RefreshAdaptors.run(["--source", "local"]) + end + end + end +end diff --git a/test/support/adaptor_test_helpers.ex b/test/support/adaptor_test_helpers.ex new file mode 100644 index 00000000000..1e47dd7298d --- /dev/null +++ b/test/support/adaptor_test_helpers.ex @@ -0,0 +1,262 @@ +defmodule Lightning.AdaptorTestHelpers do + @moduledoc """ + Test helpers for seeding the `Lightning.Adaptors.Repo` and clearing the + global `Lightning.Adaptors.Supervisor` Cachex. + + The production `Lightning.Adaptors` supervisor is started by the + application and shared across the test suite — its Cachex persists + across the `Ecto.Adapters.SQL.Sandbox` boundary, so tests that seed + rows via `Lightning.Factories.adaptor/2` must clear the cache to make + those rows visible to facade reads. + + See `test/lightning/adaptors_test.exs` and + `test/lightning/adaptors/store_test.exs` for the canonical patterns + exercised by per-test isolated supervisors. This module covers the + complementary case: tests that touch the production-named supervisor + via `Lightning.Adaptors.{packages, versions, schema, resolve_version}`. + """ + + import Lightning.Factories + + alias Lightning.Adaptors.Supervisor, as: AdaptorsSupervisor + + @doc """ + Clear the production `Lightning.Adaptors` Cachex so subsequent reads + fall back through the DB. + """ + @spec clear_global_adaptors_cache() :: :ok + def clear_global_adaptors_cache do + cache = AdaptorsSupervisor.cache_name(Lightning.Adaptors) + Cachex.clear(cache) + :ok + end + + @doc """ + Insert one `Adaptors.Repo.Adaptor` row using the factory and clear + the global Cachex so it's immediately visible to facade reads. + + `attrs` is forwarded to the `:adaptor` factory verbatim. + """ + @spec seed_adaptor(keyword() | map()) :: Lightning.Adaptors.Repo.Adaptor.t() + def seed_adaptor(attrs \\ []) do + row = insert(:adaptor, attrs) + clear_global_adaptors_cache() + row + end + + @doc """ + Seed a credential schema row keyed by short name (e.g. `"postgresql"`), + reading the JSON body from `test/fixtures/schemas/.json`. + + After `lib/lightning/credentials.ex` was migrated to read schemas via + `Lightning.Adaptors.schema/1`, schema fixtures live in the adaptor + registry rather than on disk; tests that exercise + `Credentials.get_schema/1` must seed them here. + """ + @spec seed_credential_schema(String.t()) :: + Lightning.Adaptors.Repo.Adaptor.t() + def seed_credential_schema(short_name) when is_binary(short_name) do + # Keep the raw JSON binary so field order survives — credential-form + # rendering re-engages `Jason.decode!(_, objects: :ordered_objects)` + # downstream via `Lightning.Credentials.Schema.new/2`. + schema_body = + Path.join(["test", "fixtures", "schemas", "#{short_name}.json"]) + |> File.read!() + + row = + insert(:adaptor, name: short_name, source: :npm, schema_data: schema_body) + + # Cachex's fallback runs in the Courier process — it can't see the + # test-owned sandbox connection. Pre-populate the cache so reads + # never need to fall through to a DB lookup from the Courier. + cache = AdaptorsSupervisor.cache_name(Lightning.Adaptors) + source = AdaptorsSupervisor.source(Lightning.Adaptors) + Cachex.put(cache, {:schema, short_name, source}, {:ok, schema_body}) + + row + end + + @doc """ + Seed every credential schema present in `test/fixtures/schemas/`. + + Use from a `setup` block in tests that exercise multiple credential + types (e.g. `LightningWeb.CredentialLiveTest`). + """ + @spec seed_all_credential_schemas() :: :ok + def seed_all_credential_schemas do + Path.wildcard("test/fixtures/schemas/*.json") + |> Enum.each(fn path -> + # Skip empty fixture files (e.g. `asana.json`, `primero.json` are + # intentional empty placeholders). + if File.stat!(path).size > 0 do + short_name = path |> Path.basename(".json") + seed_credential_schema(short_name) + end + end) + + :ok + end + + @doc """ + Seed an `@openfn/*` adaptor package with a concrete `latest_version` + so `Lightning.Adaptors.PackageName.to_wire/1` resolves `@latest` + correctly. The legacy `AdaptorRegistry` fixture + (`test/fixtures/adaptor_registry_cache.json`) used to provide these + resolutions for free; the migrated facade reads them from Postgres. + """ + @spec seed_adaptor_package(String.t(), String.t() | [String.t()]) :: + Lightning.Adaptors.Repo.Adaptor.t() + def seed_adaptor_package(name, versions) + when is_binary(name) and is_list(versions) do + # Latest is the first entry per legacy registry semantics. + [latest | _] = versions + + {:ok, row} = + Lightning.Adaptors.Repo.upsert_adaptor(%{ + name: name, + source: :npm, + latest_version: latest, + description: nil, + homepage: nil, + repository: nil, + license: nil, + deprecated: false, + schema_data: nil, + schema_sha256: nil, + versions: + Enum.map(versions, fn v -> + %{ + version: v, + integrity: "sha512-#{v}", + tarball_url: "https://example.com/x-#{v}.tgz", + size_bytes: 1024, + dependencies: %{}, + peer_dependencies: %{}, + published_at: nil, + deprecated: false + } + end) + }) + + row + end + + def seed_adaptor_package(name, latest_version) + when is_binary(name) and is_binary(latest_version) do + seed_adaptor_package(name, [latest_version]) + end + + @doc """ + Build a record matching `t:Lightning.Adaptors.Strategy.adaptor_record/0` + for use in `Mox.stub`/`Mox.expect` setups. + """ + @spec build_strategy_adaptor_record(String.t(), String.t()) :: map() + def build_strategy_adaptor_record(name, latest_version) do + %{ + name: name, + source: :npm, + latest_version: latest_version, + description: nil, + homepage: nil, + repository: nil, + license: nil, + deprecated: false, + schema_data: nil, + schema_sha256: nil, + versions: [ + %{ + version: latest_version, + integrity: "sha512-#{latest_version}", + tarball_url: "https://example.com/x-#{latest_version}.tgz", + size_bytes: 1024, + dependencies: %{}, + peer_dependencies: %{}, + published_at: nil, + deprecated: false + } + ] + } + end + + @doc """ + Seed the production `Lightning.Adaptors` supervisor's Cachex with a + pre-built packages map. Useful for tests that exercise + `AdaptorPicker.get_adaptor_version_options/1` under async mode — the + picker calls `Lightning.Adaptors.packages/0` and (when the adaptor is + known) `Lightning.Adaptors.versions/1`, both of which are routed + through Cachex. + + Returns the supervisor source atom for convenience. + """ + @spec warm_packages_cache([map()]) :: :npm | :local + def warm_packages_cache(metas) when is_list(metas) do + cache = AdaptorsSupervisor.cache_name(Lightning.Adaptors) + source = AdaptorsSupervisor.source(Lightning.Adaptors) + + Cachex.put(cache, {:packages, source}, {:ok, metas}) + + source + end + + @doc """ + Bulk-seed the common `@openfn/*` packages used across the test suite + with the versions that the legacy `adaptor_registry_cache.json` + fixture used to publish. + """ + @spec seed_common_packages() :: :ok + def seed_common_packages do + packages = [ + {"@openfn/language-common", ["1.6.2", "1.2.22", "1.1.0"]}, + {"@openfn/language-http", ["3.1.12", "2.0.0", "1.0.0"]}, + {"@openfn/language-postgresql", ["3.2.0", "2.0.0", "1.0.0"]}, + {"@openfn/language-dhis2", ["3.0.4", "2.0.0", "1.0.0"]}, + {"@openfn/language-salesforce", ["3.0.0", "2.0.0", "1.0.0"]}, + {"@openfn/language-godata", ["2.0.0", "1.0.0"]}, + {"@openfn/language-googlesheets", ["2.0.0", "1.0.0"]} + ] + + Enum.each(packages, fn {name, versions} -> + seed_adaptor_package(name, versions) + end) + + # Warm Cachex for the production supervisor so reads from any + # process (including the LiveView's caller chain) see the seeded + # data without falling through to the Cachex Courier's + # sandbox-blind DB query. + cache = AdaptorsSupervisor.cache_name(Lightning.Adaptors) + source = AdaptorsSupervisor.source(Lightning.Adaptors) + + metas = + Enum.map(packages, fn {name, [latest | _]} -> + %{ + name: name, + latest_version: latest, + description: nil, + deprecated: false, + icon_square_ext: nil, + icon_rectangle_ext: nil, + icon_square_sha256: nil, + icon_rectangle_sha256: nil + } + end) + + Cachex.put(cache, {:packages, source}, {:ok, metas}) + + Enum.each(packages, fn {name, versions} -> + version_metas = + Enum.map(versions, fn v -> + %{ + version: v, + integrity: "sha512-#{v}", + size_bytes: 1024, + published_at: nil, + deprecated: false + } + end) + + Cachex.put(cache, {:versions, name, source}, {:ok, version_metas}) + end) + + :ok + end +end diff --git a/test/support/factories.ex b/test/support/factories.ex index 7be93f7d739..925c858863d 100644 --- a/test/support/factories.ex +++ b/test/support/factories.ex @@ -4,6 +4,16 @@ defmodule Lightning.Factories do alias Lightning.Workflows.Snapshot + def adaptor_factory do + %Lightning.Adaptors.Repo.Adaptor{ + name: sequence(:adaptor_name, &"@openfn/language-test-#{&1}"), + source: :npm, + latest_version: "1.0.0", + checked_at: DateTime.utc_now(), + schema_data: nil + } + end + def webhook_auth_method_factory do %Lightning.Workflows.WebhookAuthMethod{ project: build(:project), diff --git a/test/test_helper.exs b/test/test_helper.exs index 7b747fbd37e..2261a9c9f2a 100644 --- a/test/test_helper.exs +++ b/test/test_helper.exs @@ -5,6 +5,8 @@ Mox.defmock(Lightning.AuthProviders.OauthHTTPClient.Mock, for: Tesla.Adapter) Mox.defmock(Lightning.MockSentry, for: Lightning.SentryBehaviour) Mox.defmock(Lightning.Tesla.Mock, for: Tesla.Adapter) +Mox.defmock(Lightning.Adaptors.StrategyMock, for: Lightning.Adaptors.Strategy) + :ok = Application.ensure_started(:ex_machina) Mimic.copy(:hackney) @@ -57,5 +59,31 @@ Application.put_env(:lightning, Lightning.Extensions, external_metrics: Lightning.Extensions.ExternalMetrics ) +# Pin the `Lightning.Adaptors.IconCache` on-disk path to a per-OS-PID +# directory and wipe it at startup so: +# 1. Each `mix test` invocation begins with an empty icon cache — +# `System.unique_integer/1` resets per-VM and recycles, so without +# this, leftover files from a prior run can mask a Mox expectation +# by short-circuiting `IconCache.cached?/4`. +# 2. Concurrent `mix test` invocations (different tmux panes, parallel +# CI shards) use distinct directories and never collide — each BEAM +# has its own OS PID. +icon_dir = + Path.join([ + System.tmp_dir!(), + "lightning_test_icons", + System.pid() + ]) + +File.rm_rf!(icon_dir) +File.mkdir_p!(icon_dir) + +Application.put_env( + :lightning, + Lightning.Adaptors, + Application.get_env(:lightning, Lightning.Adaptors, []) + |> Keyword.put(:icon_path, icon_dir) +) + ExUnit.start() Ecto.Adapters.SQL.Sandbox.mode(Lightning.Repo, :manual)