fix(ci): rebuild backends when shared build inputs change (#10975)

The backend matrix path filter only matched files under a backend's own
directory, so a change to shared build infrastructure rebuilt nothing at
all: an empty matrix, every job green, and the change reaching no image.

PR #10946 fixed scripts/build/package-gpu-libs.sh shipping a partial
4-of-8 cuDNN library set, which mixed versions with the venv's pip cuDNN
and produced CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH at inference time.
It merged 1h48m after the weekly full-matrix cron had already run, so no
backend image ever received the fix and nothing signalled that it had
been un-shipped.

Add a SHARED_BUILD_INPUTS table mapping each shared path to the narrowest
set of matrix entries it can honestly invalidate, plus a generic rule for
backend/Dockerfile.<x> (which each entry already names). A full matrix is
417 Linux + 56 Darwin builds, so package-gpu-libs.sh now rebuilds the 176
Python entries rather than everything. Unclassified files under
scripts/build/ fall back to a full rebuild deliberately: over-building is
recoverable, silently shipping nothing is not.

Extract the filtering logic to scripts/lib/backend-filter.mjs so it can be
unit-tested without bun, js-yaml or a GitHub API round-trip, and run those
tests from the existing lint workflow via `make test-ci-scripts`.


Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

Author: localai-bot

.agents/adding-backends.md

@@ -34,7 +34,7 @@ The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside `
 
 **Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
 
-**`scripts/changed-backends.js` registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/changed-backends.js:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
+**Path-filter registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/lib/backend-filter.mjs:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
 
 ```js
 if (item.dockerfile.endsWith("<your-dockerfile-suffix>")) {
@@ -54,7 +54,9 @@ for (const e of m.include.filter(e => e.backend === '<your-backend>')) {
 }"
 ```
 
-A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/changed-backends.js`.
+A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/lib/backend-filter.mjs`.
+
+If your backend consumes a *shared* build input that lives outside its own directory (a new script under `scripts/build/`, a new file copied into every image), add a rule to `SHARED_BUILD_INPUTS` in the same file — the per-backend prefix match cannot see those, and a miss ships your change to no image at all. See `scripts/lib/backend-filter_test.mjs` for the pattern; `make test-ci-scripts` runs it.
 
 **`bump_deps.yaml` registration — REQUIRED for any backend pinning an upstream commit.** If your backend's Makefile has a `*_VERSION?=<sha>` pin to a third-party repo, the daily auto-bump bot at `.github/workflows/bump_deps.yaml` won't notice it unless you register the backend in its matrix. The bot runs `.github/bump_deps.sh` which `grep`s for `^$VAR?=` in the Makefile you list — so the pin MUST live in the Makefile (not in a separate shell script). The bump for ds4 (#9761) had to walk this back because the original landed the pin in `prepare.sh`, which the bot can't see. Pattern (for `antirez/ds4`):
 
@@ -115,7 +117,7 @@ Wiring a backend into `includeDarwin:` is more than the matrix entry:
 
 1. **`includeDarwin:` entry** — `tag-suffix: "-metal-darwin-arm64-<backend>"`, `build-type: "metal"`, `lang: "go"` for go+ggml backends; omit `build-type` for the bespoke C++ ones (llama-cpp / ds4 / privacy-filter). Match an existing entry of the same shape.
 2. **`backend/index.yaml`** — add `metal:` to the backend's `capabilities` map (main and `-development`) and concrete `metal-<backend>` / `metal-<backend>-development` image entries pointing at the `-metal-darwin-arm64-<backend>` images.
-3. **C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/changed-backends.js` returning `backend/cpp/<backend>/` (the generic fallthrough assumes `backend/<lang>/`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/<backend>-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`.
+3. **C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/lib/backend-filter.mjs` returning `backend/cpp/<backend>/` (the generic fallthrough assumes `backend/<lang>/`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/<backend>-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`.
 4. **C++ proto gotcha** — if the backend compiles the generated gRPC/protobuf in a separate CMake target (e.g. `hw_grpc_proto`), that target must link `protobuf::libprotobuf` + `gRPC::grpc++` so the Homebrew include dirs propagate; otherwise macOS fails with `google/protobuf/runtime_version.h not found` (Linux hides this because apt headers sit in `/usr/include`).
 
 The CI path filter only builds a backend on a PR when a file under its directory changes, so a darwin-only YAML edit builds nothing — touch a file under `backend/<lang>/<backend>/` (a one-line comment is enough) in the same PR.
@@ -243,7 +245,7 @@ After adding a new backend, verify:
 
 - [ ] Backend directory structure is complete with all necessary files
 - [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant)
-- [ ] **OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml` `metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default.
+- [ ] **OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml` `metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case (in `scripts/lib/backend-filter.mjs`) for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default.
 - [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
 - [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
 - [ ] Tag suffixes match between workflow file and index.yaml

.agents/ci-caching.md

@@ -114,6 +114,24 @@ Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynami
 - **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff.
 - **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically.
 
+### Shared build inputs
+
+The per-backend prefix match only sees files under a backend's own directory, so a change to shared build infrastructure would rebuild *nothing* — an empty matrix, every job green, and the change reaching no image. That silently un-shipped PR #10946 (a partial-cuDNN packaging fix in `scripts/build/package-gpu-libs.sh`), which merged 1h48m after the weekly cron and so sat unbuilt for a week.
+
+`SHARED_BUILD_INPUTS` in `scripts/lib/backend-filter.mjs` closes that hole. Each rule maps a shared path to the narrowest set of matrix entries it can honestly invalidate, since a full matrix is 417 Linux + 56 Darwin builds:
+
+| Changed path | Rebuilds |
+|---|---|
+| `backend/backend.proto` | everything (all languages compile or copy it) |
+| `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
+| `backend/python/common/` | Python, Linux + Darwin |
+| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
+| `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
+| `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
+| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
+
+Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this.
+
 The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
 
 ## The `DEPS_REFRESH` cache-buster (Python backends)

.github/backend-matrix.yml

@@ -23,7 +23,7 @@
 # checklist in .agents/adding-backends.md (includeDarwin entry, the index.yaml
 # `metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD
 # branch for C/C++ backends, and the inferBackendPathDarwin case in
-# scripts/changed-backends.js so the path filter actually builds it).
+# scripts/lib/backend-filter.mjs so the path filter actually builds it).
 
 # Linux matrix (consumed by backend-jobs).
 include:

.github/workflows/lint.yml

@@ -57,3 +57,12 @@ jobs:
       - uses: actions/checkout@v7
       - name: run packaging script tests
         run: make test-build-scripts
+
+      # The backend matrix path filter fails silently: a miss emits an empty
+      # matrix, every job goes green, and the change reaches no image (#10946).
+      # Its tests need only node, so they ride along with this job.
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+      - name: run CI script tests
+        run: make test-ci-scripts

Makefile

@@ -215,6 +215,13 @@ test-backend-cpp:
 test-build-scripts:
 	@set -e; for t in scripts/build/*_test.sh; do echo "== $$t"; bash "$$t"; done
 
+## Runs the unit tests for the CI helper scripts under scripts/lib/. Currently
+## the backend matrix path filter, whose failure mode is invisible in CI: it
+## emits an empty matrix, every job goes green, and the change ships to no
+## image at all (see PR #10946). Plain `node --test`, no dependencies.
+test-ci-scripts:
+	@set -e; for t in scripts/lib/*_test.mjs; do echo "== $$t"; node --test "$$t"; done
+
 ## Runs the core suite ($(TEST_PATHS)) with statement-coverage instrumentation
 ## and writes a merged profile to $(COVERAGE_PROFILE). Deliberately omits
 ## --fail-fast so a single failure doesn't truncate the coverage number, and

scripts/changed-backends.js

@@ -2,6 +2,11 @@ import fs from "fs";
 import * as yaml from "js-yaml";
 import { Octokit } from "@octokit/core";
 
+import {
+  getAllBackendPaths,
+  filterMatrix,
+} from "./lib/backend-filter.mjs";
+
 // Matrix data lives in a small data-only YAML so both backend.yml (master push)
 // and backend_pr.yml (pull_request) can use a dynamic `matrix: ${{ fromJson(...) }}`
 // for the live job, while this script remains the single source of truth for
@@ -13,125 +18,7 @@ const includesDarwin = matrixYml.includeDarwin;
 const eventPath = process.env.GITHUB_EVENT_PATH;
 const event = JSON.parse(fs.readFileSync(eventPath, "utf8"));
 
-// Infer backend path
-function inferBackendPath(item) {
-  if (item.dockerfile.endsWith("python")) {
-    return `backend/python/${item.backend}/`;
-  }
-  // parakeet-cpp is a Go backend (Dockerfile.golang) wrapping the parakeet.cpp
-  // ggml port via purego. It lives in backend/go/parakeet-cpp/; this explicit
-  // branch (placed before the generic golang one, which would also resolve it
-  // correctly) documents the mapping and guards against a future
-  // dockerfile-suffix change.
-  if (item.backend === "parakeet-cpp") {
-    return `backend/go/parakeet-cpp/`;
-  }
-  // ced is a Go backend (Dockerfile.golang) wrapping the ced.cpp ggml port via
-  // purego, living in backend/go/ced/. Same explicit-branch rationale as
-  // parakeet-cpp above: the generic golang fallthrough would also resolve it,
-  // but this documents the mapping and guards a future dockerfile-suffix change.
-  if (item.backend === "ced") {
-    return `backend/go/ced/`;
-  }
-  // moss-transcribe-cpp is a Go backend (Dockerfile.golang) wrapping the
-  // moss-transcribe.cpp ggml port via purego, living in
-  // backend/go/moss-transcribe-cpp/. Same explicit-branch rationale as
-  // parakeet-cpp / ced: the generic golang fallthrough would also resolve it,
-  // but this documents the mapping and guards a future dockerfile-suffix change.
-  if (item.backend === "moss-transcribe-cpp") {
-    return `backend/go/moss-transcribe-cpp/`;
-  }
-  // moss-tts-cpp is a Go backend (Dockerfile.golang) wrapping the moss-tts.cpp
-  // ggml port via purego, living in backend/go/moss-tts-cpp/. Same
-  // explicit-branch rationale as parakeet-cpp / ced / moss-transcribe-cpp: the
-  // generic golang fallthrough would also resolve it, but this documents the
-  // mapping and guards a future dockerfile-suffix change.
-  if (item.backend === "moss-tts-cpp") {
-    return `backend/go/moss-tts-cpp/`;
-  }
-  if (item.dockerfile.endsWith("golang")) {
-    return `backend/go/${item.backend}/`;
-  }
-  if (item.dockerfile.endsWith("rust")) {
-    return `backend/rust/${item.backend}/`;
-  }
-  if (item.dockerfile.endsWith("ik-llama-cpp")) {
-    return `backend/cpp/ik-llama-cpp/`;
-  }
-  if (item.dockerfile.endsWith("turboquant")) {
-    // turboquant is a llama.cpp fork that reuses backend/cpp/llama-cpp sources
-    // via a thin wrapper Makefile. Changes to either dir should retrigger it.
-    return `backend/cpp/turboquant/`;
-  }
-  if (item.dockerfile.endsWith("bonsai")) {
-    // bonsai is a llama.cpp fork that reuses backend/cpp/llama-cpp sources
-    // via a thin wrapper Makefile. Changes to either dir should retrigger it.
-    return `backend/cpp/bonsai/`;
-  }
-  if (item.dockerfile.endsWith("privacy-filter")) {
-    return `backend/cpp/privacy-filter/`;
-  }
-  if (item.dockerfile.endsWith("ds4")) {
-    return `backend/cpp/ds4/`;
-  }
-  if (item.dockerfile.endsWith("llama-cpp")) {
-    return `backend/cpp/llama-cpp/`;
-  }
-  return null;
-}
-
-function inferBackendPathDarwin(item) {
-  // llama-cpp on Darwin builds from the C++ sources, not a backend/go/llama-cpp
-  // tree (which doesn't exist). The Darwin job is matrix-driven with lang=go
-  // for runner/toolchain selection, but the source path is C++.
-  if (item.backend === "llama-cpp") {
-    return `backend/cpp/llama-cpp/`;
-  }
-  // ds4 is C++ too (built via `make backends/ds4-darwin`); the matrix entry
-  // carries lang=go for runner/toolchain selection, but the source is C++.
-  if (item.backend === "ds4") {
-    return `backend/cpp/ds4/`;
-  }
-  // privacy-filter is C++ too (built via `make backends/privacy-filter-darwin`);
-  // same lang=go-for-runner convention, source under backend/cpp.
-  if (item.backend === "privacy-filter") {
-    return `backend/cpp/privacy-filter/`;
-  }
-  if (!item.lang) {
-    return `backend/python/${item.backend}/`;
-  }
-
-  return `backend/${item.lang}/${item.backend}/`;
-}
-
-// Build a deduplicated map of backend name -> path prefix from all matrix entries
-function getAllBackendPaths() {
-  const paths = new Map();
-  for (const item of includes) {
-    const p = inferBackendPath(item);
-    if (p && !paths.has(item.backend)) {
-      paths.set(item.backend, p);
-    }
-  }
-  for (const item of includesDarwin) {
-    const p = inferBackendPathDarwin(item);
-    if (p && !paths.has(item.backend)) {
-      paths.set(item.backend, p);
-    }
-  }
-  return paths;
-}
-
-const allBackendPaths = getAllBackendPaths();
-
-function backendChanged(backend, pathPrefix, changedFiles) {
-  if (changedFiles.some(file => file.startsWith(pathPrefix))) return true;
-
-  // Fork backends reuse backend/cpp/llama-cpp sources via thin wrappers;
-  // changes to either directory must retrigger their pipelines.
-  return (backend === "turboquant" || backend === "bonsai") &&
-    changedFiles.some(file => file.startsWith("backend/cpp/llama-cpp/"));
-}
+const allBackendPaths = getAllBackendPaths(includes, includesDarwin);
 
 const token = process.env.GITHUB_TOKEN;
 const octokit = new Octokit({ auth: token });
@@ -316,15 +203,10 @@ function emitFullMatrix() {
 function emitFilteredMatrix(changedFiles) {
   console.log("Changed files:", changedFiles);
 
-  const filtered = includes.filter(item => {
-    const backendPath = inferBackendPath(item);
-    if (!backendPath) return false;
-    return backendChanged(item.backend, backendPath, changedFiles);
-  });
-
-  const filteredDarwin = includesDarwin.filter(item => {
-    const backendPath = inferBackendPathDarwin(item);
-    return changedFiles.some(file => file.startsWith(backendPath));
+  const { filtered, filteredDarwin, changedBackends } = filterMatrix({
+    includes,
+    includesDarwin,
+    changedFiles,
   });
 
   console.log("Filtered files:", filtered);
@@ -350,8 +232,8 @@ function emitFilteredMatrix(changedFiles) {
   emitSinglearchShards(singlearch);
 
   // Per-backend boolean outputs
-  for (const [backend, pathPrefix] of allBackendPaths) {
-    const changed = backendChanged(backend, pathPrefix, changedFiles);
+  for (const backend of allBackendPaths.keys()) {
+    const changed = changedBackends.has(backend);
     fs.appendFileSync(process.env.GITHUB_OUTPUT, `${backend}=${changed ? 'true' : 'false'}\n`);
   }
 }