feat: support multiple local adaptor repos in registry (#4714)

* feat: support multiple local adaptor repos in registry

OPENFN_ADAPTORS_REPO now accepts a colon-separated list of paths so that
a private adaptor repo can be loaded alongside the canonical OpenFn
adaptors monorepo. Order is precedence: when two repos ship a package
with the same dirname, the entry from the earlier path wins and a single
summary warning is logged for the shadowed entries.

The bootstrap layer normalises the env var into the
`local_adaptors_repos` list config key, which is the sole input the
registry consumes. Single-path values still work; they just become a
one-element list. A missing or unreadable repo path is logged as an
error and skipped, so a typo in one entry does not take down the others.

The bundled ws-worker still expects a single path for `@local` adaptor
execution, so colon-separated values only widen the registry view
(picker UI, metadata). RUNNINGLOCAL.md and .env.example document the
limitation.

* fix: use comma separator for OPENFN_ADAPTORS_REPO and run mix format

Colon collides with Windows drive letters (c:/repo); comma matches the
ws-worker parser so the registry view and @Local resolution agree.
Single-path callers are unchanged. Also applies mix format to satisfy
CI and tightens the docs note about missing repos (they are skipped,
not fatal).

* improve shadowed-adaptor warning and trim local-adaptor docs

The shadow warning now names the winning and shadowed repo per package
instead of just listing package names twice, so a local override is
diagnosable from the log alone.

Consolidate the local-adaptors explanation into RUNNINGLOCAL.md: strip
the duplicated prose from .env.example and reduce the bootstrap.ex
comment to the ws-worker separator rationale.

* docs: add changelog entry for multi-root local adaptors (#4714)

---------

Co-authored-by: Stuart Corbishley <corbish@gmail.com>

Author: jeremi

.env.example

@@ -292,11 +292,11 @@
 # For this reason, you have to make sure that the directory exists and it is writable
 # ADAPTORS_REGISTRY_JSON_PATH=/path/to/adaptor_registry_cache.json
 #
-# These 2 envs are used to enable local adaptors mode. OPENFN_ADAPTORS_REPO points
-# to the repo directory which must have a `packages` subdir. LOCAL_ADAPTORS env is
-# the flag used to enable/disable this mode
+# Enable local adaptors mode. OPENFN_ADAPTORS_REPO takes one repo path, or a
+# comma-separated list to merge several. See RUNNINGLOCAL.md for the details.
 # LOCAL_ADAPTORS=true
 # OPENFN_ADAPTORS_REPO=/path/to/repo/
+# OPENFN_ADAPTORS_REPO=/path/to/private,/path/to/canonical
 #
 # Control whether metrics reported by the Workflow editor or Job editor are
 # written to the Lightning logs.

CHANGELOG.md

@@ -17,6 +17,12 @@ and this project adheres to
 
 ### Added
 
+- Support a comma-separated list of paths in `OPENFN_ADAPTORS_REPO`, merging
+  multiple local adaptor repos in precedence order (earlier paths win on name
+  collisions, and shadowed entries are logged). Lets a private repo override or
+  extend the canonical adaptors in local mode.
+  [#4714](https://github.com/OpenFn/lightning/pull/4714)
+
 ### Changed
 
 ### Fixed

RUNNINGLOCAL.md

@@ -159,13 +159,21 @@ To start, set up the following environment variables:
 - `LOCAL_ADAPTORS`: Used to enable or disable the local adaptors mode. Set it to
   `true` to enable.
 - `OPENFN_ADAPTORS_REPO`: This should point to the adaptors monorepo. This is
-  the same variable used when you pass `-m` to the CLI.
+  the same variable used when you pass `-m` to the CLI. It also accepts a
+  comma-separated list of paths to merge multiple repos into the registry; the
+  first path wins on dirname collisions, with a warning logged for shadowed
+  entries. Both the registry view and the bundled `ws-worker` resolve `@local`
+  adaptors against the same list, so a workflow run picks up the same package
+  the picker shows.
 
 Example configuration:
 
 ```sh
 export LOCAL_ADAPTORS=true
 export OPENFN_ADAPTORS_REPO=/path/to/repo/
+
+# Or, merge a private adaptor repo with the canonical one (first wins):
+export OPENFN_ADAPTORS_REPO=/path/to/private,/path/to/canonical
 ```
 
 You can also run the server directly in local mode with:
@@ -174,8 +182,9 @@ You can also run the server directly in local mode with:
 LOCAL_ADAPTORS=true mix phx.server
 ```
 
-Ensure that the `OPENFN_ADAPTORS_REPO` directory is correctly set up with the
-necessary `packages` subdirectory, otherwise the app wont start
+Each path in `OPENFN_ADAPTORS_REPO` must contain a `packages` subdirectory.
+Paths that are missing or unreadable are logged and skipped, so the rest of
+the list still loads.
 
 ### Problems with Apple Silicon
 

lib/lightning/adaptor_registry.ex

@@ -91,8 +91,9 @@ defmodule Lightning.AdaptorRegistry do
   def handle_continue(opts, _state) do
     adaptors =
       case Enum.into(opts, %{}) do
-        %{local_adaptors_repo: repo_path} when is_binary(repo_path) ->
-          read_adaptors_from_local_repo(repo_path)
+        %{local_adaptors_repos: repo_paths}
+        when is_list(repo_paths) and repo_paths != [] ->
+          read_adaptors_from_local_repos(repo_paths)
 
         %{use_cache: use_cache}
         when use_cache === true or is_binary(use_cache) ->
@@ -150,10 +151,21 @@ defmodule Lightning.AdaptorRegistry do
     and uses the cached file for every subsequent start.
     It can either be a boolean, or a string - the latter being a file path
     to set where the cache file is located.
+  - `:local_adaptors_repos` - an ordered list of paths to local adaptor
+    monorepos (each containing a `packages/` subdirectory). When set, the
+    registry skips NPM and lists adaptors from these directories instead.
+    Earlier paths win on dirname collisions; shadowed entries are summarised
+    in a single warning log.
   - `:name` (defaults to AdaptorRegistry) - the name of the process, useful
     for testing and/or running multiple versions of the registry
   """
-  @spec start_link(opts :: [use_cache: boolean() | binary(), name: term()]) ::
+  @spec start_link(
+          opts :: [
+            use_cache: boolean() | binary(),
+            local_adaptors_repos: [binary()],
+            name: term()
+          ]
+        ) ::
           {:error, any} | {:ok, pid}
   def start_link(opts \\ [use_cache: true]) do
     Logger.info("Starting AdaptorRegistry")
@@ -270,20 +282,76 @@ defmodule Lightning.AdaptorRegistry do
     }
   end
 
-  defp read_adaptors_from_local_repo(repo_path) do
-    Logger.debug("Using local adaptors repo at #{repo_path}")
-
-    repo_path
-    |> Path.join("packages")
-    |> File.ls!()
-    |> Enum.map(fn package ->
-      %{
-        name: "@openfn/language-" <> package,
-        repo: "file://" <> Path.join([repo_path, "packages", package]),
-        latest: "local",
-        versions: []
-      }
-    end)
+  defp read_adaptors_from_local_repos(repo_paths) when is_list(repo_paths) do
+    Logger.debug("Using local adaptors repos at #{inspect(repo_paths)}")
+
+    repo_paths
+    |> Enum.flat_map(&adaptors_in_repo/1)
+    |> dedupe_first_wins()
+  end
+
+  defp adaptors_in_repo(repo_path) do
+    packages_path = Path.join(repo_path, "packages")
+
+    case File.ls(packages_path) do
+      {:ok, entries} ->
+        Enum.map(entries, fn package ->
+          %{
+            name: "@openfn/language-" <> package,
+            repo: "file://" <> Path.join([repo_path, "packages", package]),
+            latest: "local",
+            versions: []
+          }
+        end)
+
+      {:error, reason} ->
+        Logger.error(
+          "Skipping local adaptors repo #{inspect(repo_path)}: " <>
+            "cannot list #{inspect(packages_path)} (#{:file.format_error(reason)})"
+        )
+
+        []
+    end
+  end
+
+  # First-occurrence wins: when two roots ship a package with the same
+  # `@openfn/language-<dirname>` name, the entry from the earlier root is kept.
+  # Listing your private repo before the canonical one therefore lets you
+  # override individual adaptors locally without forking the whole canonical
+  # tree. Shadowed entries are summarised in a single warning so the override
+  # case (the intended use of ordering) does not flood logs with one line per
+  # package.
+  defp dedupe_first_wins(adaptors) do
+    {kept_reversed, winners, shadowed} =
+      Enum.reduce(adaptors, {[], %{}, []}, fn adaptor,
+                                              {kept, winners, shadowed} ->
+        if Map.has_key?(winners, adaptor.name) do
+          {kept, winners, [adaptor | shadowed]}
+        else
+          {[adaptor | kept], Map.put(winners, adaptor.name, adaptor), shadowed}
+        end
+      end)
+
+    log_shadowed(Enum.reverse(shadowed), winners)
+    Enum.reverse(kept_reversed)
+  end
+
+  defp log_shadowed([], _winners), do: :ok
+
+  defp log_shadowed(shadowed, winners) do
+    details =
+      shadowed
+      |> Enum.group_by(& &1.name)
+      |> Enum.map_join("; ", fn {name, losers} ->
+        winner = Map.fetch!(winners, name)
+        loser_repos = Enum.map_join(losers, ", ", & &1.repo)
+        "#{name} (using #{winner.repo}, shadowed #{loser_repos})"
+      end)
+
+    Logger.warning(
+      "AdaptorRegistry: #{length(shadowed)} adaptor(s) shadowed by earlier " <>
+        "local-adaptors repo entries: #{details}"
+    )
   end
 
   @doc """
@@ -362,8 +430,9 @@ defmodule Lightning.AdaptorRegistry do
   end
 
   def local_adaptors_enabled? do
-    config = Lightning.Config.adaptor_registry()
-
-    if config[:local_adaptors_repo], do: true, else: false
+    case Lightning.Config.adaptor_registry()[:local_adaptors_repos] do
+      [_ | _] -> true
+      _ -> false
+    end
   end
 end

lib/lightning/config/bootstrap.ex

@@ -203,21 +203,26 @@ defmodule Lightning.Config.Bootstrap do
     config :lightning, :adaptor_service,
       adaptors_path: env!("ADAPTORS_PATH", :string, "./priv/openfn")
 
-    local_adaptors_repo =
-      env!(
-        "OPENFN_ADAPTORS_REPO",
-        :string,
-        Utils.get_env([
-          :lightning,
-          Lightning.AdaptorRegistry,
-          :local_adaptors_repo
-        ])
-      )
+    # Comma-separated to match the ws-worker parser, so the picker view and
+    # @local resolution agree on the same repo list. See RUNNINGLOCAL.md.
+    local_adaptors_repos =
+      env!("OPENFN_ADAPTORS_REPO", :string, nil)
+      |> case do
+        nil ->
+          []
+
+        value when is_binary(value) ->
+          value
+          |> String.split(",", trim: true)
+          |> Enum.map(&String.trim/1)
+          |> Enum.reject(&(&1 == ""))
+          |> Enum.map(&Path.expand/1)
+      end
 
-    use_local_adaptors_repo? =
+    use_local_adaptors_repos? =
       env!("LOCAL_ADAPTORS", &Utils.ensure_boolean/1, false)
       |> tap(fn v ->
-        if v && !is_binary(local_adaptors_repo) do
+        if v && local_adaptors_repos == [] do
           raise """
           LOCAL_ADAPTORS is set to true, but OPENFN_ADAPTORS_REPO is not set.
           """
@@ -231,8 +236,8 @@ defmodule Lightning.Config.Bootstrap do
           :string,
           Utils.get_env([:lightning, Lightning.AdaptorRegistry, :use_cache])
         ),
-      local_adaptors_repo:
-        use_local_adaptors_repo? && Path.expand(local_adaptors_repo)
+      local_adaptors_repos:
+        if(use_local_adaptors_repos?, do: local_adaptors_repos, else: [])
 
     config :lightning,
       schemas_path:

test/lightning/adaptor_registry_test.exs

@@ -131,18 +131,16 @@ defmodule Lightning.AdaptorRegistryTest do
     end
 
     @tag :tmp_dir
-    test "lists directory names of the when local_adaptors_repo is set", %{
-      tmp_dir: tmp_dir,
-      test: test
-    } do
+    test "lists directory names from a single-element local_adaptors_repos list",
+         %{tmp_dir: tmp_dir, test: test} do
       expected_adaptors = ["foo", "bar", "baz"]
 
       Enum.each(expected_adaptors, fn adaptor ->
         [tmp_dir, "packages", adaptor] |> Path.join() |> File.mkdir_p!()
       end)
 
       start_supervised!(
-        {AdaptorRegistry, [name: test, local_adaptors_repo: tmp_dir]}
+        {AdaptorRegistry, [name: test, local_adaptors_repos: [tmp_dir]]}
       )
 
       results = AdaptorRegistry.all(test)
@@ -158,6 +156,109 @@ defmodule Lightning.AdaptorRegistryTest do
         assert expected_result in results
       end
     end
+
+    @tag :tmp_dir
+    test "merges adaptors from multiple local_adaptors_repos", %{
+      tmp_dir: tmp_dir,
+      test: test
+    } do
+      repo_a = Path.join(tmp_dir, "a")
+      repo_b = Path.join(tmp_dir, "b")
+      [repo_a, "packages", "alpha"] |> Path.join() |> File.mkdir_p!()
+      [repo_b, "packages", "beta"] |> Path.join() |> File.mkdir_p!()
+
+      start_supervised!(
+        {AdaptorRegistry, [name: test, local_adaptors_repos: [repo_a, repo_b]]}
+      )
+
+      names = AdaptorRegistry.all(test) |> Enum.map(& &1.name) |> Enum.sort()
+
+      assert names == ["@openfn/language-alpha", "@openfn/language-beta"]
+    end
+
+    @tag :tmp_dir
+    test "first repo wins on collision and emits a warning", %{
+      tmp_dir: tmp_dir,
+      test: test
+    } do
+      repo_a = Path.join(tmp_dir, "a")
+      repo_b = Path.join(tmp_dir, "b")
+      [repo_a, "packages", "http"] |> Path.join() |> File.mkdir_p!()
+      [repo_b, "packages", "http"] |> Path.join() |> File.mkdir_p!()
+
+      log =
+        ExUnit.CaptureLog.capture_log(fn ->
+          start_supervised!(
+            {AdaptorRegistry,
+             [name: test, local_adaptors_repos: [repo_a, repo_b]]}
+          )
+
+          # force the GenServer to finish handle_continue
+          AdaptorRegistry.all(test)
+        end)
+
+      results = AdaptorRegistry.all(test)
+      assert length(results) == 1
+
+      assert hd(results).repo ==
+               "file://" <> Path.join([repo_a, "packages", "http"])
+
+      assert log =~ "@openfn/language-http"
+      assert log =~ "shadowed"
+
+      assert log =~ "using"
+      assert log =~ "file://" <> Path.join([repo_a, "packages", "http"])
+      assert log =~ "file://" <> Path.join([repo_b, "packages", "http"])
+    end
+
+    @tag :tmp_dir
+    test "soft-fails when a repo path is missing or unreadable", %{
+      tmp_dir: tmp_dir,
+      test: test
+    } do
+      good_repo = Path.join(tmp_dir, "good")
+      missing_repo = Path.join(tmp_dir, "does-not-exist")
+      [good_repo, "packages", "alpha"] |> Path.join() |> File.mkdir_p!()
+
+      log =
+        ExUnit.CaptureLog.capture_log(fn ->
+          start_supervised!(
+            {AdaptorRegistry,
+             [name: test, local_adaptors_repos: [missing_repo, good_repo]]}
+          )
+
+          AdaptorRegistry.all(test)
+        end)
+
+      names = AdaptorRegistry.all(test) |> Enum.map(& &1.name)
+      assert names == ["@openfn/language-alpha"]
+      assert log =~ "Skipping local adaptors repo"
+      assert log =~ missing_repo
+    end
+  end
+
+  describe "local_adaptors_enabled?/0" do
+    test "returns true when a non-empty plural list is configured" do
+      Mox.stub(Lightning.MockConfig, :adaptor_registry, fn ->
+        [local_adaptors_repos: ["/some/path"]]
+      end)
+
+      assert AdaptorRegistry.local_adaptors_enabled?()
+    end
+
+    test "returns false when the list is empty" do
+      Mox.stub(Lightning.MockConfig, :adaptor_registry, fn ->
+        [local_adaptors_repos: []]
+      end)
+
+      refute AdaptorRegistry.local_adaptors_enabled?()
+    end
+
+    test "returns false when the key is absent" do
+      Mox.stub(Lightning.MockConfig, :adaptor_registry, fn -> [] end)
+
+      refute AdaptorRegistry.local_adaptors_enabled?()
+    end
   end
 
   describe "resolve_package_name/1" do
@@ -178,10 +279,10 @@ defmodule Lightning.AdaptorRegistryTest do
     end
 
     @tag :tmp_dir
-    test "returns local as the version when local_adaptors_repo config is set",
+    test "returns local as the version when local_adaptors_repos config is set",
          %{tmp_dir: tmp_dir} do
       Mox.stub(Lightning.MockConfig, :adaptor_registry, fn ->
-        [local_adaptors_repo: tmp_dir]
+        [local_adaptors_repos: [tmp_dir]]
       end)
 
       assert AdaptorRegistry.resolve_package_name("@openfn/language-foo@1.2.3") ==