From d329a44dee7849013e07e85d8879a201dafddfd3 Mon Sep 17 00:00:00 2001 From: Snider Date: Wed, 22 Jul 2026 14:47:49 +0100 Subject: [PATCH 1/3] ci: stop GitHub Actions failure spam + land the lem Driver/Native artifact grid MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The GitLab side is green; the failure emails were GitHub Actions. Two verified signatures, each fixed with the documented built-in (config is code — no hand-rolling): 1. "Restore cache failed: Dependencies file is not found ... pattern: go.sum" The module root is go/ (and cli/), not the repo root. Added setup-go's cache-dependency-path input to every failing setup-go (ci.yml -> go/go.sum; build.yml -> go/go.sum + cli/go.sum). deps.yml left alone (it was passing). 2. `task: Command "sysctl -n hw.ncpu" failed` (ubuntu exit 1 / windows 127) go-task evaluates global sh-vars eagerly, so the NCPU var must succeed on every platform. Hardened the fallback chain with a literal floor: getconf ... || nproc ... || sysctl -n hw.ncpu 2>/dev/null || echo 4 Mac path unchanged (getconf _NPROCESSORS_ONLN already returns the core count = 32). build.yml rewritten to an HONEST matrix + the Driver/Native release grid: - Probe finding: lem links DuckDB via cgo; duckdb-go-bindings is //go:build cgo, so CGO_ENABLED=0 fails for every target. There is NO pure-Go lem and NO free cross-compile — each cell is a native cgo build. duckdb supports darwin/{amd64,arm64}, linux/{amd64,arm64}, windows/amd64 (NOT windows/arm64). - GitHub now builds only the cpu cells it can build natively (macos aarch64 + x86_64, linux x86_64 + aarch64, windows x86_64); the dishonest hosted-metal cell is removed. - Metal rides a self-hosted macOS 26 lane, dormant behind `if vars.ENABLE_MACOS_METAL` so it never queues or fails on a push; interim = built manually on the maintainer's Mac. Hosted runners are < macOS 26 and cannot compile the Metal 4 metallibs; a plain build would ship a non-runnable, metallib-less binary. - Each cell is packaged twice from one build: Native {os}-{arch}-lem-{backend}-{ver}.zip (binary `lem`) and Driver {os}-{arch}-lem-driver-{backend}-{ver}.zip (binary `lem-{backend}`); rolling dev prerelease kept, version aligned to dev-. - Dropped GOWORK=off from the binary build: cli pins go/inference v0.14.0 which lags the local tree (missing the dataset pkg), so the workspace must wire cli -> local go/. docs/release-artifacts.md settles the grid, the two packagings, per-cell CI ownership, the metal trade-off, the cgo finding, and the not-yet-wired follow-ups (tagged GitHub release; GitLab grid-named zips — left for on-box validation). .gitlab-ci.yml untouched (green; gates own WHEN). Makefile untouched. cli/tui untouched. Co-Authored-By: Virgil --- .github/workflows/build.yml | 163 ++++++++++++++++++++++++++---------- .github/workflows/ci.yml | 4 + Taskfile.yml | 7 +- docs/release-artifacts.md | 120 ++++++++++++++++++++++++++ 4 files changed, 247 insertions(+), 47 deletions(-) create mode 100644 docs/release-artifacts.md diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 78b3ad41..00352402 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -1,7 +1,16 @@ -# Build the sovereign `lem` binary for every target and ship the zips. -# One binary named `lem` per platform (the GPU backend — metal/rocm/cuda — loads -# its kernel sidecar at runtime); targets are separated by build folder, not by -# binary name. Zips flow to a rolling `dev` prerelease. +# Build the sovereign `lem` binary for every hosted-buildable cell and ship the zips. +# +# lem links DuckDB (go-store/chathistory) via cgo, so there is NO pure-Go build and +# NO cheap cross-compile: each cell is a NATIVE cgo build on a matching runner. +# GitHub owns the portable cpu cells it can build natively; the ROCm/CUDA cells +# (amd, cuda) stay on GitLab's homelab runner (it has the toolchain); the metal +# cell needs a macOS 26 / Metal 4 box to compile the metallibs, so it rides a +# self-hosted Mac lane (dormant until ENABLE_MACOS_METAL is set). +# +# Every build ships in two packagings of the same binary (see docs/release-artifacts.md): +# Native {os}-{arch}-lem-{backend}-{version}.zip -> binary `lem` +# Driver {os}-{arch}-lem-driver-{backend}-{version}.zip -> binary `lem-{backend}` +# Zips flow to a rolling `dev` prerelease; version is `dev-`. name: Build on: @@ -12,77 +21,136 @@ on: permissions: contents: read +env: + GO_VERSION: "1.26" + jobs: - build: - name: lem · ${{ matrix.goos }}/${{ matrix.goarch }} - runs-on: ${{ matrix.os }} + # ---- cpu cells: native cgo, one per hosted os/arch (duckdb ships prebuilt libs + # for darwin/{amd64,arm64}, linux/{amd64,arm64}, windows/amd64 — NOT + # windows/arm64, which is therefore not a cell). The workspace (go.work) + # wires cli -> local go/; GOWORK=off cannot build the binary because cli + # pins a published go/inference tag that lags the local tree. ---------------- + cpu: + name: lem cpu · ${{ matrix.os_name }}/${{ matrix.arch }} + runs-on: ${{ matrix.runner }} strategy: fail-fast: false matrix: include: - - { os: macos-latest, goos: darwin, goarch: arm64, metal: true } - - { os: ubuntu-latest, goos: linux, goarch: amd64 } - - { os: ubuntu-24.04-arm, goos: linux, goarch: arm64 } - - { os: windows-latest, goos: windows, goarch: amd64 } - env: - CGO_ENABLED: "1" - GOWORK: "off" + - { runner: macos-latest, os_name: macos, arch: aarch64, exe: "" } + - { runner: macos-13, os_name: macos, arch: x86_64, exe: "" } + - { runner: ubuntu-latest, os_name: linux, arch: x86_64, exe: "" } + - { runner: ubuntu-24.04-arm, os_name: linux, arch: aarch64, exe: "" } + - { runner: windows-latest, os_name: windows, arch: x86_64, exe: ".exe" } steps: - uses: actions/checkout@v4 + # cpu build needs neither external/mlx (metal) nor the ROCm sources. + + - uses: actions/setup-go@v5 + with: + go-version: ${{ env.GO_VERSION }} + # go.sum lives in the modules, not the repo root; list both so setup-go's + # cache key is honest (build touches cli, cli requires local go/). + cache-dependency-path: | + go/go.sum + cli/go.sum + + - name: Build lem (native cgo; workspace wires cli -> local go/) + shell: bash + env: + CGO_ENABLED: "1" + EXE: ${{ matrix.exe }} + run: | + set -euo pipefail + VERSION="dev-${GITHUB_SHA:0:12}" + mkdir -p bin + go build -trimpath -ldflags "-X main.version=${VERSION}" -o "bin/lem${EXE}" ./cli + test -s "bin/lem${EXE}" + + - uses: actions/upload-artifact@v4 with: - submodules: recursive # metal needs external/mlx + name: lem-cpu-${{ matrix.os_name }}-${{ matrix.arch }} + path: bin/lem${{ matrix.exe }} + if-no-files-found: error + + # ---- metal cell: macos/aarch64, self-contained (metallibs embedded). Hosted + # macOS runners are < macOS 26 and cannot compile the Metal 4 metallibs, so + # a plain build would ship a metallib-less, non-runnable binary. This lane is + # DORMANT on normal pushes (the `if` is false when the variable is unset, so + # it never queues or fails): register a self-hosted macOS 26 arm64 runner and + # set repo variable ENABLE_MACOS_METAL=true to activate it. Interim: the metal + # artifact is built manually on the owner's Mac (task metallib && task build:embed). + metal: + name: lem metal · macos/aarch64 (self-hosted macOS 26) + if: ${{ vars.ENABLE_MACOS_METAL == 'true' }} + runs-on: [self-hosted, macOS, ARM64] + steps: + - uses: actions/checkout@v4 + with: + submodules: recursive # external/mlx for `task metallib` - uses: actions/setup-go@v5 with: - go-version: "1.26" + go-version: ${{ env.GO_VERSION }} + cache-dependency-path: | + go/go.sum + cli/go.sum - name: Install Task run: go install github.com/go-task/task/v3/cmd/task@latest - - name: Build lem + - name: Build self-contained metal lem (metallibs baked in) shell: bash - env: - METAL: ${{ matrix.metal }} run: | - set -e - if [ "$METAL" = "true" ]; then - task metallib && task build:embed # self-contained metal lem - else - task build:native # plain lem (GPU sidecar loads at runtime) - fi - - - name: Assemble build folder (build/dist//lem) - shell: bash - env: - TARGET: ${{ matrix.goos }}-${{ matrix.goarch }} - run: | - set -e - mkdir -p "build/dist/$TARGET" - cp bin/lem* "build/dist/$TARGET/" + set -euo pipefail + VERSION="dev-${GITHUB_SHA:0:12}" + task metallib + task build:embed VERSION="${VERSION}" + test -s bin/lem - uses: actions/upload-artifact@v4 with: - name: lem-${{ matrix.goos }}-${{ matrix.goarch }} - path: build/dist/${{ matrix.goos }}-${{ matrix.goarch }}/ + name: lem-metal-macos-aarch64 + path: bin/lem + if-no-files-found: error + # ---- release: package each built binary under BOTH names (build once, package + # twice) and publish to the rolling `dev` prerelease. Runs only for dev + # pushes; requires cpu to have succeeded and tolerates metal being skipped. ---- release: - name: Zip + rolling dev prerelease - needs: build - if: github.event_name == 'push' && github.ref == 'refs/heads/dev' + name: Zip (Native + Driver) + rolling dev prerelease + needs: [cpu, metal] + if: ${{ !cancelled() && needs.cpu.result == 'success' && github.event_name == 'push' && github.ref == 'refs/heads/dev' }} runs-on: ubuntu-latest permissions: contents: write steps: - uses: actions/download-artifact@v4 with: - path: dist # one subfolder per target (lem--/lem) + path: dist # one dir per cell: dist/lem---/lem[.exe] - - name: Zip each target + - name: Package Native + Driver zips + shell: bash run: | - set -e + set -euo pipefail + VERSION="dev-${GITHUB_SHA:0:12}" + mkdir -p out cd dist - for d in */; do (cd "$d" && zip -r "../${d%/}.zip" .); done - ls -la *.zip + for d in lem-*/; do + name="${d%/}" # e.g. lem-cpu-macos-aarch64 + IFS=- read -r _ backend os arch <<< "$name" # backend/os/arch from the name + bin="$(ls "$d")" # lem or lem.exe + ext=""; case "$bin" in *.exe) ext=".exe";; esac + # Native — the binary a user runs locally, named `lem`. + cp "$d/$bin" "lem${ext}"; chmod 0755 "lem${ext}" + zip -jq "../out/${os}-${arch}-lem-${backend}-${VERSION}.zip" "lem${ext}" + rm -f "lem${ext}" + # Driver — the binary a compute box serves over the API, named `lem-`. + cp "$d/$bin" "lem-${backend}${ext}"; chmod 0755 "lem-${backend}${ext}" + zip -jq "../out/${os}-${arch}-lem-driver-${backend}-${VERSION}.zip" "lem-${backend}${ext}" + rm -f "lem-${backend}${ext}" + done + ls -la ../out - name: Publish rolling dev prerelease uses: softprops/action-gh-release@v2 @@ -90,7 +158,7 @@ jobs: tag_name: dev name: dev (rolling) prerelease: true - files: dist/*.zip + files: out/*.zip env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} @@ -101,7 +169,10 @@ jobs: - uses: actions/checkout@v4 - uses: actions/setup-go@v5 with: - go-version: "1.26" + go-version: ${{ env.GO_VERSION }} + cache-dependency-path: | + go/go.sum + cli/go.sum - uses: actions/setup-node@v4 with: node-version: "20" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index be052d7b..455c5e27 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,6 +26,10 @@ jobs: - uses: actions/setup-go@v5 with: go-version: "1.26" + # The module root is go/ (go.sum lives there, not at the repo root); without + # this, setup-go's cache aborts with "Dependencies file is not found ... + # Supported file pattern: go.sum". This lane only compiles the go/ module. + cache-dependency-path: go/go.sum - name: Portable tests + coverage (default tags — engine/metal excluded) working-directory: go diff --git a/Taskfile.yml b/Taskfile.yml index 93b22f6f..83bc62f3 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -18,8 +18,13 @@ vars: VERSION: dev GO_VERSION_LDFLAGS: '-X main.version={{.VERSION}}' GO_DARWIN_LDFLAGS: '-extldflags=-mmacosx-version-min=26.0' + # Core count for cmake --parallel (metallib:mlx only). go-task evaluates global + # sh-vars eagerly on every invocation, so this must succeed on EVERY platform: + # getconf/nproc cover Linux, sysctl -n hw.ncpu is the macOS path, and the `echo 4` + # floor keeps it green where none of the three exist (Windows, task's internal + # POSIX shell has no such binaries) — a bare `sysctl -n hw.ncpu` was the CI failure. NCPU: - sh: getconf _NPROCESSORS_ONLN 2>/dev/null || nproc 2>/dev/null || sysctl -n hw.ncpu + sh: getconf _NPROCESSORS_ONLN 2>/dev/null || nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4 env: MLX_METALLIB_PATH: '{{.ROOT_DIR}}/build/dist/lib/mlx.metallib' diff --git a/docs/release-artifacts.md b/docs/release-artifacts.md new file mode 100644 index 00000000..fbcc7aee --- /dev/null +++ b/docs/release-artifacts.md @@ -0,0 +1,120 @@ +# Release artifacts — the `lem` Driver/Native grid + +`lem` is one binary. The GPU backend is registered at **build** time, and `lem` also +serves its own HTTP API — so a compute box can run a backend-specific `lem` as a remote +GPU **driver**, and a user's local `lem` consumes it over that API. The release surface +therefore ships every per-`{os,arch,backend}` build in **two packagings of the same binary**: + +| Packaging | Zip name | Contains (one binary, nothing else) | Role | +|-----------|----------|-------------------------------------|------| +| **Native** | `{os}-{arch}-lem-{backend}-{version}.zip` | `lem` (`lem.exe` on Windows) | Run locally on your machine. | +| **Driver** | `{os}-{arch}-lem-driver-{backend}-{version}.zip` | `lem-{backend}` (`.exe` on Windows) | Deploy on a compute box; serves the GPU over the API. | + +Worked example (the maintainer's own setup): + +- On his Mac: `macos-aarch64-lem-metal-v0.15.0.zip` → a `lem` binary for mac + metal. +- On his AMD Linux box: `linux-x86_64-lem-driver-amd-v0.15.0.zip` → a `lem-amd` binary + serving the GPU over the API, which the Mac's `lem` then drives remotely. + +**Build once, package twice.** Each cell is built a single time; the one binary is copied +into the Native zip as `lem` and into the Driver zip as `lem-{backend}`. No double build. + +## Dimensions + +- `os` ∈ {`macos`, `linux`, `windows`} (GOOS `darwin`→`macos`) +- `arch` ∈ {`x86_64`, `aarch64`} (GOARCH `amd64`→`x86_64`, `arm64`→`aarch64`) +- `backend` ∈ {`metal`, `amd`, `cuda`, `cpu`} +- `version` = the git tag on a release build; the rolling `dev` prerelease uses `dev-`. + +## The grid — valid cells only + +| os | arch | backend | Built by | How | +|----|------|---------|----------|-----| +| macos | aarch64 | metal | self-hosted macOS 26 lane (dormant) / manual | `task metallib && task build:embed` | +| macos | aarch64 | cpu | **GitHub** `macos-latest` | native cgo `go build ./cli` | +| macos | x86_64 | cpu | **GitHub** `macos-13` | native cgo | +| linux | x86_64 | amd | **GitLab** homelab | `make lem-amd` | +| linux | x86_64 | cuda | **GitLab** homelab | `make lem-cuda` | +| linux | x86_64 | cpu | **GitHub** `ubuntu-latest` (rolling) · **GitLab** `make lem-cpu-x86` (tag) | native cgo | +| linux | aarch64 | cpu | **GitHub** `ubuntu-24.04-arm` (rolling) · **GitLab** `make lem-cpu-aarch64` (tag) | native cgo | +| windows | x86_64 | cpu | **GitHub** `windows-latest` | native cgo (mingw) | + +### Excluded cells (and why) + +- **cpu · windows/aarch64** — `duckdb-go-bindings` ships no `windows-arm64` prebuilt lib, so + the binary cannot link. Not a cell. +- **metal · anything but macos/aarch64** — Metal is Apple-GPU only. +- **amd, cuda · anything but linux/x86_64** — need the ROCm / CUDA toolchain and its Linux libs. +- **any backend with `CGO_ENABLED=0`** — impossible; see below. + +## Why every cell is a *native* cgo build (no cross-compile) + +`lem` links **DuckDB** (the `go-store` / chathistory driver) through cgo. `duckdb-go-bindings` +guards its prebuilt-lib packages with `//go:build cgo`, so `CGO_ENABLED=0` fails at compile +(`build constraints exclude all Go files ...`). There is **no pure-Go `lem`** and therefore +**no free `GOOS/GOARCH` cross-compile** — cross-building needs a target C toolchain. Each cell +is built natively on a runner of that `os/arch` (GitLab's `lem-cpu-aarch64` is the one +cross-cgo build, via `aarch64-linux-gnu-gcc`). Probed on 2026-07-22 against +`duckdb-go-bindings v0.10504.0`: cgo-off builds fail for every target; native cgo builds pass. + +DuckDB's supported set (the prebuilt libs that exist): `darwin/{amd64,arm64}`, +`linux/{amd64,arm64}`, `windows/amd64`. That set defines the valid `cpu` cells. + +## Which CI owns which cells + +- **GitHub Actions** (`.github/workflows/build.yml`) — the portable **cpu** cells it can build + natively on hosted runners, on every push to `dev`, published to the rolling `dev` + prerelease as `dev-` zips (Native + Driver). Hosted runners have no ROCm/CUDA + toolchain and are older than macOS 26, so they build **only** cpu. +- **GitLab CI** (`.gitlab-ci.yml`, homelab AMD box) — the toolchain-gated **amd** and **cuda** + cells, plus its own **cpu-x86 / cpu-aarch64**, on a git **tag** via the `release` gate and + the `Makefile` targets. This side is unchanged (it is green and has the toolchain). +- **Self-hosted Mac lane** — the **metal** cell (see below). + +The two systems do not fight: GitHub is the continuous `dev` channel; GitLab is the tagged +release for the Linux/GPU cells. + +## The metal cell decision + +Metal's Go side (engine/metal, the objc bridge) is no-cgo and cross-compiles, **but** a usable +metal `lem` embeds the MLX + lthn `.metallib` kernels (`-tags embed_metallib`), and those are +compiled by `task metallib` with **Metal 4 / macOS 26** (`-std=metal4.0`, +`CMAKE_OSX_DEPLOYMENT_TARGET=26.0`) and are gitignored (generated, never committed). + +- A GitHub **hosted** macOS runner is macOS 14/15 → it **cannot** compile the metallibs. A plain + (non-embed) build would resolve them at runtime via `MLX_METALLIB_PATH`, but the user has no + metallib on their machine — so that artifact is **non-runnable**. We do **not** ship a + fabricated/broken metal binary from hosted CI. +- **Decision:** the metal cell ships from a **self-hosted macOS 26 arm64 runner** running the + house build path (`task metallib && task build:embed`), producing the self-contained binary. + The `metal` job in `build.yml` encodes this recipe but is **dormant** — its `if` is false + unless repo variable `ENABLE_MACOS_METAL=true` is set — so it never queues or fails on a + normal push. **Interim** (until a self-hosted runner is registered): the maintainer builds + the metal zip manually on his Mac (the primary dev box already runs `task build:embed` daily). +- **Trade-off:** correctness (a self-contained, runnable metal binary) over convenience (a + hosted job). The cost is one self-hosted runner or a manual step for the metal cell only. + +## Zip layout rule + +Each zip contains **exactly one binary and nothing else** — matching what `build.yml` shipped +before (binary only, no licence file). Self-contained cells honour this directly: + +- **cpu** — native-cgo `lem`, no GPU engine, no sidecar. +- **metal** — `build:embed` bakes the metallibs in. + +**Exception — amd / cuda:** these load a HIP/CUDA kernel **sidecar** (`.hsaco` / `.o`) at +runtime that is not embedded, so those cells ship the binary **plus** its sidecar (the +`Makefile` already tars them together). This is the single documented departure from +binary-only, driven by runtime necessity. + +## Not yet wired (follow-ups) + +- **Tagged GitHub release.** `build.yml` publishes only the rolling `dev` prerelease (dev + pushes). Producing tag-versioned (`v*`) zips for the GitHub cells (macos/windows cpu, metal) + is a small extension: add `push.tags` to `on:` and version from the tag. GitLab already + handles tagged Linux/GPU cells. +- **GitLab grid-named zips.** GitLab currently uploads the raw `lem-{backend}` binaries (already + Driver-named) via `build/bin/`. Emitting the `{os}-{arch}-lem-...-{version}.zip` names (with + the amd/cuda sidecars) is a `Makefile`/`.gitlab-ci.yml` packaging follow-up that must be + validated on the ROCm box — it cannot be exercised from a macOS lane, so it was left untouched + here rather than changed blind. From 51a7afe582d9abb424cc4ea8416eb1ac08b06301 Mon Sep 17 00:00:00 2001 From: Snider Date: Wed, 22 Jul 2026 15:07:05 +0100 Subject: [PATCH 2/3] test: clear the linux test debt the first portable CI run surfaced MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four failures, three platform-blind and one real fixture bug — all reproduced and re-verified on the linux box (full portable suite exit=0 there; agent/ai + driver 335 green on darwin): - engine/metal gemm_steel_test.go had no build constraint — on linux the package collapsed to that one file and failed to build. Tagged darwin && arm64 like every sibling; linux now excludes the package cleanly. - agent/ai TestReadEvents compared time.Time with != after a disk round-trip — passes only when the local zone makes time.Parse hand back time.Local (BST Macs), fails on UTC runners. Now .Equal. - engine/hip: two contract tests carried pre-truth-pass expectations (standalone-native qwen3_6_moe, fitting 16GB plan). The truth-pass source (ArchitectureProfileNotes post-#50) is explicit: no native path replaced the retired composed detour. Realigned to metadata_only / Fits=false with the architecture gap reported honestly; the memory-plan maths assertions stay. - engine/driver: fakeDriverScript ran a bare 'sleep' under the harness's deliberately EMPTIED PATH — exit 127 on spawn, every test driver an instant corpse, the crash-restart test racing its own restart storm (darwin's poll won the race; linux's never did). /bin/sleep by absolute path — the restart loop is now genuinely exercised on both platforms. Co-Authored-By: Virgil --- go/agent/ai/ai_test.go | 2 +- go/engine/driver/fixtures_test.go | 8 ++++-- go/engine/hip/native_contract_test.go | 38 ++++++++++++++++----------- go/engine/metal/gemm_steel_test.go | 2 ++ 4 files changed, 32 insertions(+), 18 deletions(-) diff --git a/go/agent/ai/ai_test.go b/go/agent/ai/ai_test.go index 31360442..ae855d11 100644 --- a/go/agent/ai/ai_test.go +++ b/go/agent/ai/ai_test.go @@ -120,7 +120,7 @@ func TestReadEvents_Good_SkipsMissingDays(t *testing.T) { if len(events) != 2 { t.Fatalf("expected 2 events, got %d", len(events)) } - if events[0].Timestamp != dayOne || events[1].Timestamp != dayThree { + if !events[0].Timestamp.Equal(dayOne) || !events[1].Timestamp.Equal(dayThree) { t.Fatalf("events not returned in chronological order: %+v", events) } } diff --git a/go/engine/driver/fixtures_test.go b/go/engine/driver/fixtures_test.go index 971c672b..2a918f3b 100644 --- a/go/engine/driver/fixtures_test.go +++ b/go/engine/driver/fixtures_test.go @@ -31,8 +31,12 @@ func init() { // ignores its argv (serve --addr ... --model ... — whatever serveArgs built) // and just stays alive until killed: SIGKILL can't be caught, and a plain // `sleep` terminates on SIGTERM by default too, so no trap is needed to -// satisfy spawn/Stop/Status/crash-restart tests. -const fakeDriverScript = "#!/bin/sh\nexec sleep 600\n" +// satisfy spawn/Stop/Status/crash-restart tests. The sleep is invoked by +// absolute path because isolateDriverLookup deliberately empties PATH — a +// bare `sleep` exits 127 instantly, turning every spawn into a silent +// crash-restart storm (the loop raced its own storm and only won on some +// platforms; caught on the linux CI runner). +const fakeDriverScript = "#!/bin/sh\nexec /bin/sleep 600\n" // writeFakeDriver drops an executable fake driver binary named name into dir // and returns its path. diff --git a/go/engine/hip/native_contract_test.go b/go/engine/hip/native_contract_test.go index 49677f5c..0df0a26d 100644 --- a/go/engine/hip/native_contract_test.go +++ b/go/engine/hip/native_contract_test.go @@ -559,30 +559,34 @@ func TestNativeContract_CapabilityReportGenericReactiveRegistryLabels_Good(t *te QuantBits: 4, }, inference.AdapterIdentity{}, true, defaultHIPKernelStatus()) + // qwen3_6_moe has NO native ROCm execution path (the truth-pass source, + // profile.ArchitectureProfileNotes post-#50: nothing replaced the retired + // composed detour) — the registry-derived report must say metadata_only, + // not standalone-native. if report.Labels["engine_feature_architecture"] != "qwen3_6_moe" || report.Labels["engine_feature_family"] != "qwen" || report.Labels["engine_feature_chat_template_id"] != "qwen" || report.Labels["engine_feature_reasoning_parser"] != "qwen" || report.Labels["engine_feature_tool_parser"] != "qwen" || - report.Labels["engine_feature_text_generate"] != "true" || - report.Labels["engine_feature_capabilities"] != "generate,chat.template,reasoning.parse,tool.parse" || - report.Labels["engine_load_status"] != string(ROCmModelLoadStandaloneNative) || - report.Labels["engine_load_target"] != "standalone" || + report.Labels["engine_feature_text_generate"] != "false" || + report.Labels["engine_feature_capabilities"] != "chat.template,reasoning.parse,tool.parse" || + report.Labels["engine_load_status"] != string(ROCmModelLoadMetadataOnly) || + report.Labels["engine_load_target"] != "metadata" || report.Labels["engine_load_staged"] != "false" || - report.Labels["engine_load_native_runtime"] != "true" || - report.Labels["engine_load_standalone"] != "true" || - report.Labels["engine_load_text_generate"] != "true" { + report.Labels["engine_load_native_runtime"] != "false" || + report.Labels["engine_load_standalone"] != "false" || + report.Labels["engine_load_text_generate"] != "false" { t.Fatalf("report labels = %+v, want generic registry-derived Qwen engine feature labels", report.Labels) } modelLoad, ok := report.Capability(inference.CapabilityModelLoad) if !ok || - modelLoad.Labels["engine_load_status"] != string(ROCmModelLoadStandaloneNative) || - modelLoad.Labels["engine_load_target"] != "standalone" || + modelLoad.Labels["engine_load_status"] != string(ROCmModelLoadMetadataOnly) || + modelLoad.Labels["engine_load_target"] != "metadata" || modelLoad.Labels["engine_load_staged"] != "false" || - modelLoad.Labels["engine_load_native_runtime"] != "true" || - modelLoad.Labels["engine_load_standalone"] != "true" || - modelLoad.Labels["engine_load_text_generate"] != "true" { - t.Fatalf("model-load capability = %+v ok=%v, want standalone-native Qwen load-status labels", modelLoad, ok) + modelLoad.Labels["engine_load_native_runtime"] != "false" || + modelLoad.Labels["engine_load_standalone"] != "false" || + modelLoad.Labels["engine_load_text_generate"] != "false" { + t.Fatalf("model-load capability = %+v ok=%v, want metadata-only Qwen load-status labels", modelLoad, ok) } if cap, ok := report.Capability(inference.CapabilityGenerate); !ok || cap.Status != inference.CapabilityStatusPlanned { t.Fatalf("generate capability = %+v ok=%v, registry-routed Qwen remains planned without linked kernels", cap, ok) @@ -2274,8 +2278,12 @@ func TestNativeContract_PlanModelFit_Rocm16GBMoELazyExperts_Good(t *testing.T) { if err != nil { t.Fatalf("PlanModelFit: %v", err) } - if report == nil || !report.Fits || report.MemoryPlan.MachineClass != "rocm-16gb" { - t.Fatalf("fit report = %+v, want fitting ROCm 16GB MoE plan", report) + // The 16GB memory plan computes fully, but qwen3_moe has no native ROCm + // execution path (truth-pass: nothing replaced the retired composed + // detour), so the honest report is ArchitectureOK=false and Fits=false — + // the plan maths is the thing under test, not a native fit claim. + if report == nil || report.Fits || report.ArchitectureOK || report.MemoryPlan.MachineClass != "rocm-16gb" { + t.Fatalf("fit report = %+v, want ROCm 16GB MoE plan with the architecture gap reported honestly", report) } if report.MemoryPlan.CacheMode != "k-q8-v-q4" || report.MemoryPlan.Labels["moe_lazy_experts"] != "true" || report.MemoryPlan.Labels["prefill_chunk_tokens"] != "512" { t.Fatalf("memory plan = %+v, want compact KV, lazy experts, and chunked prefill", report.MemoryPlan) diff --git a/go/engine/metal/gemm_steel_test.go b/go/engine/metal/gemm_steel_test.go index 03b4fc21..c200c54d 100644 --- a/go/engine/metal/gemm_steel_test.go +++ b/go/engine/metal/gemm_steel_test.go @@ -1,5 +1,7 @@ // SPDX-Licence-Identifier: EUPL-1.2 +//go:build darwin && arm64 + package native import ( From 591e1397e8e444442439d342c78ec837b46f383f Mon Sep 17 00:00:00 2001 From: Snider Date: Wed, 22 Jul 2026 16:03:56 +0100 Subject: [PATCH 3/3] ci: drop the two dead cells the live run exposed; fix the sdk generator pin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit macos/x86_64: macos-13 runner label retired — queued forever, no hosted Intel Mac remains. windows/x86_64: lem is not Windows-portable (syscall.Kill + raw-fd reads outside constraints — three live compile errors); the cell returns when a portability campaign lands, not before. sdk: @openapitools/openapi-generator-cli npm wrapper is 2.x (2.40.0) — 7.22.0 conflated it with the Java generator version it manages; ETARGET on every run. Live receipts: CI (portable suite) PASSED on this branch; cpu cells macos/aarch64 + linux/x86_64 + linux/aarch64 built green; metal gate skipped dormant as designed. Co-Authored-By: Virgil --- .github/workflows/build.yml | 10 +++++++--- docs/release-artifacts.md | 8 ++++++-- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 00352402..c00fd842 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -37,11 +37,15 @@ jobs: fail-fast: false matrix: include: + # Dropped cells, proven by live runs 2026-07-22 (docs/release-artifacts.md): + # - macos/x86_64: the macos-13 runner label is retired (queued forever) — + # no hosted Intel Mac remains; revisit only on real demand. + # - windows/x86_64: lem's code is not Windows-portable yet (unix-only + # syscall.Kill + raw-fd reads outside build constraints) — a code + # campaign, not a CI matrix entry; restore the cell when it compiles. - { runner: macos-latest, os_name: macos, arch: aarch64, exe: "" } - - { runner: macos-13, os_name: macos, arch: x86_64, exe: "" } - { runner: ubuntu-latest, os_name: linux, arch: x86_64, exe: "" } - { runner: ubuntu-24.04-arm, os_name: linux, arch: aarch64, exe: "" } - - { runner: windows-latest, os_name: windows, arch: x86_64, exe: ".exe" } steps: - uses: actions/checkout@v4 # cpu build needs neither external/mlx (metal) nor the ROCm sources. @@ -182,7 +186,7 @@ jobs: java-version: "21" - name: Install generators run: | - npm install -g @openapitools/openapi-generator-cli@7.22.0 + npm install -g @openapitools/openapi-generator-cli@2.40.0 # npm wrapper is 2.x; it manages the 7.x Java generator go install github.com/go-task/task/v3/cmd/task@latest - name: Generate SDKs (lem spec -> OpenAPI 3.1 -> typed clients) run: task sdk diff --git a/docs/release-artifacts.md b/docs/release-artifacts.md index fbcc7aee..bb79e287 100644 --- a/docs/release-artifacts.md +++ b/docs/release-artifacts.md @@ -32,15 +32,19 @@ into the Native zip as `lem` and into the Driver zip as `lem-{backend}`. No doub |----|------|---------|----------|-----| | macos | aarch64 | metal | self-hosted macOS 26 lane (dormant) / manual | `task metallib && task build:embed` | | macos | aarch64 | cpu | **GitHub** `macos-latest` | native cgo `go build ./cli` | -| macos | x86_64 | cpu | **GitHub** `macos-13` | native cgo | | linux | x86_64 | amd | **GitLab** homelab | `make lem-amd` | | linux | x86_64 | cuda | **GitLab** homelab | `make lem-cuda` | | linux | x86_64 | cpu | **GitHub** `ubuntu-latest` (rolling) · **GitLab** `make lem-cpu-x86` (tag) | native cgo | | linux | aarch64 | cpu | **GitHub** `ubuntu-24.04-arm` (rolling) · **GitLab** `make lem-cpu-aarch64` (tag) | native cgo | -| windows | x86_64 | cpu | **GitHub** `windows-latest` | native cgo (mingw) | ### Excluded cells (and why) +- **cpu · macos/x86_64** — the `macos-13` runner label is retired (live-proven 2026-07-22: + queued forever); GitHub hosts no Intel Mac any more. Revisit only on real demand. +- **cpu · windows/x86_64** — lem's code is not Windows-portable yet: unix-only + `syscall.Kill` + raw-fd reads sit outside build constraints (live-proven 2026-07-22, + three compile errors). Restoring the cell is a code-portability campaign, not a + matrix entry. - **cpu · windows/aarch64** — `duckdb-go-bindings` ships no `windows-arm64` prebuilt lib, so the binary cannot link. Not a cell. - **metal · anything but macos/aarch64** — Metal is Apple-GPU only.