ci: stop GitHub Actions failure spam + Driver/Native artifact grid

.github/workflows/build.yml

@@ -1,7 +1,16 @@
-# Build the sovereign `lem` binary for every target and ship the zips.
-# One binary named `lem` per platform (the GPU backend — metal/rocm/cuda — loads
-# its kernel sidecar at runtime); targets are separated by build folder, not by
-# binary name. Zips flow to a rolling `dev` prerelease.
+# Build the sovereign `lem` binary for every hosted-buildable cell and ship the zips.
+#
+# lem links DuckDB (go-store/chathistory) via cgo, so there is NO pure-Go build and
+# NO cheap cross-compile: each cell is a NATIVE cgo build on a matching runner.
+# GitHub owns the portable cpu cells it can build natively; the ROCm/CUDA cells
+# (amd, cuda) stay on GitLab's homelab runner (it has the toolchain); the metal
+# cell needs a macOS 26 / Metal 4 box to compile the metallibs, so it rides a
+# self-hosted Mac lane (dormant until ENABLE_MACOS_METAL is set).
+#
+# Every build ships in two packagings of the same binary (see docs/release-artifacts.md):
+#   Native  {os}-{arch}-lem-{backend}-{version}.zip          -> binary `lem`
+#   Driver  {os}-{arch}-lem-driver-{backend}-{version}.zip   -> binary `lem-{backend}`
+# Zips flow to a rolling `dev` prerelease; version is `dev-<short-sha>`.
 name: Build
 
 on:
@@ -12,85 +21,148 @@
 permissions:
   contents: read
 
+env:
+  GO_VERSION: "1.26"
+
 jobs:
-  build:
-    name: lem · ${{ matrix.goos }}/${{ matrix.goarch }}
-    runs-on: ${{ matrix.os }}
+  # ---- cpu cells: native cgo, one per hosted os/arch (duckdb ships prebuilt libs
+  #      for darwin/{amd64,arm64}, linux/{amd64,arm64}, windows/amd64 — NOT
+  #      windows/arm64, which is therefore not a cell). The workspace (go.work)
+  #      wires cli -> local go/; GOWORK=off cannot build the binary because cli
+  #      pins a published go/inference tag that lags the local tree. ----------------
+  cpu:
+    name: lem cpu · ${{ matrix.os_name }}/${{ matrix.arch }}
+    runs-on: ${{ matrix.runner }}
     strategy:
       fail-fast: false
       matrix:
         include:
-          - { os: macos-latest,     goos: darwin,  goarch: arm64, metal: true }
-          - { os: ubuntu-latest,    goos: linux,   goarch: amd64 }
-          - { os: ubuntu-24.04-arm, goos: linux,   goarch: arm64 }
-          - { os: windows-latest,   goos: windows, goarch: amd64 }
-    env:
-      CGO_ENABLED: "1"
-      GOWORK: "off"
+          # Dropped cells, proven by live runs 2026-07-22 (docs/release-artifacts.md):
+          #  - macos/x86_64: the macos-13 runner label is retired (queued forever) —
+          #    no hosted Intel Mac remains; revisit only on real demand.
+          #  - windows/x86_64: lem's code is not Windows-portable yet (unix-only
+          #    syscall.Kill + raw-fd reads outside build constraints) — a code
+          #    campaign, not a CI matrix entry; restore the cell when it compiles.
+          - { runner: macos-latest,     os_name: macos,   arch: aarch64, exe: "" }
+          - { runner: ubuntu-latest,    os_name: linux,   arch: x86_64,  exe: "" }
+          - { runner: ubuntu-24.04-arm, os_name: linux,   arch: aarch64, exe: "" }
     steps:
       - uses: actions/checkout@v4
+        # cpu build needs neither external/mlx (metal) nor the ROCm sources.
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: ${{ env.GO_VERSION }}
+          # go.sum lives in the modules, not the repo root; list both so setup-go's
+          # cache key is honest (build touches cli, cli requires local go/).
+          cache-dependency-path: |
+            go/go.sum
+            cli/go.sum
+
+      - name: Build lem (native cgo; workspace wires cli -> local go/)
+        shell: bash
+        env:
+          CGO_ENABLED: "1"
+          EXE: ${{ matrix.exe }}
+        run: |
+          set -euo pipefail
+          VERSION="dev-${GITHUB_SHA:0:12}"
+          mkdir -p bin
+          go build -trimpath -ldflags "-X main.version=${VERSION}" -o "bin/lem${EXE}" ./cli
+          test -s "bin/lem${EXE}"
+
+      - uses: actions/upload-artifact@v4
         with:
-          submodules: recursive          # metal needs external/mlx
+          name: lem-cpu-${{ matrix.os_name }}-${{ matrix.arch }}
+          path: bin/lem${{ matrix.exe }}
+          if-no-files-found: error
+
+  # ---- metal cell: macos/aarch64, self-contained (metallibs embedded). Hosted
+  #      macOS runners are < macOS 26 and cannot compile the Metal 4 metallibs, so
+  #      a plain build would ship a metallib-less, non-runnable binary. This lane is
+  #      DORMANT on normal pushes (the `if` is false when the variable is unset, so
+  #      it never queues or fails): register a self-hosted macOS 26 arm64 runner and
+  #      set repo variable ENABLE_MACOS_METAL=true to activate it. Interim: the metal
+  #      artifact is built manually on the owner's Mac (task metallib && task build:embed).
+  metal:
+    name: lem metal · macos/aarch64 (self-hosted macOS 26)
+    if: ${{ vars.ENABLE_MACOS_METAL == 'true' }}
+    runs-on: [self-hosted, macOS, ARM64]
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive          # external/mlx for `task metallib`
 
       - uses: actions/setup-go@v5
         with:
-          go-version: "1.26"
+          go-version: ${{ env.GO_VERSION }}
+          cache-dependency-path: |
+            go/go.sum
+            cli/go.sum
 
       - name: Install Task
         run: go install github.com/go-task/task/v3/cmd/task@latest
 
-      - name: Build lem
+      - name: Build self-contained metal lem (metallibs baked in)
         shell: bash
-        env:
-          METAL: ${{ matrix.metal }}
         run: |
-          set -e
-          if [ "$METAL" = "true" ]; then
-            task metallib && task build:embed     # self-contained metal lem
-          else
-            task build:native                     # plain lem (GPU sidecar loads at runtime)
-          fi
-
-      - name: Assemble build folder (build/dist/<target>/lem)
-        shell: bash
-        env:
-          TARGET: ${{ matrix.goos }}-${{ matrix.goarch }}
-        run: |
-          set -e
-          mkdir -p "build/dist/$TARGET"
-          cp bin/lem* "build/dist/$TARGET/"
+          set -euo pipefail
+          VERSION="dev-${GITHUB_SHA:0:12}"
+          task metallib
+          task build:embed VERSION="${VERSION}"
+          test -s bin/lem
 
       - uses: actions/upload-artifact@v4
         with:
-          name: lem-${{ matrix.goos }}-${{ matrix.goarch }}
-          path: build/dist/${{ matrix.goos }}-${{ matrix.goarch }}/
+          name: lem-metal-macos-aarch64
+          path: bin/lem
+          if-no-files-found: error
 
+  # ---- release: package each built binary under BOTH names (build once, package
+  #      twice) and publish to the rolling `dev` prerelease. Runs only for dev
+  #      pushes; requires cpu to have succeeded and tolerates metal being skipped. ----
   release:
-    name: Zip + rolling dev prerelease
-    needs: build
-    if: github.event_name == 'push' && github.ref == 'refs/heads/dev'
+    name: Zip (Native + Driver) + rolling dev prerelease
+    needs: [cpu, metal]
+    if: ${{ !cancelled() && needs.cpu.result == 'success' && github.event_name == 'push' && github.ref == 'refs/heads/dev' }}
     runs-on: ubuntu-latest
     permissions:
       contents: write
     steps:
       - uses: actions/download-artifact@v4
         with:
-          path: dist        # one subfolder per target (lem-<goos>-<goarch>/lem)
+          path: dist        # one dir per cell: dist/lem-<backend>-<os>-<arch>/lem[.exe]
 
-      - name: Zip each target
+      - name: Package Native + Driver zips
+        shell: bash
         run: |
-          set -e
+          set -euo pipefail
+          VERSION="dev-${GITHUB_SHA:0:12}"
+          mkdir -p out
           cd dist
-          for d in */; do (cd "$d" && zip -r "../${d%/}.zip" .); done
-          ls -la *.zip
+          for d in lem-*/; do
+            name="${d%/}"                                 # e.g. lem-cpu-macos-aarch64
+            IFS=- read -r _ backend os arch <<< "$name"   # backend/os/arch from the name
+            bin="$(ls "$d")"                              # lem or lem.exe
+            ext=""; case "$bin" in *.exe) ext=".exe";; esac
+            # Native — the binary a user runs locally, named `lem`.
+            cp "$d/$bin" "lem${ext}"; chmod 0755 "lem${ext}"
+            zip -jq "../out/${os}-${arch}-lem-${backend}-${VERSION}.zip" "lem${ext}"
+            rm -f "lem${ext}"
+            # Driver — the binary a compute box serves over the API, named `lem-<backend>`.
+            cp "$d/$bin" "lem-${backend}${ext}"; chmod 0755 "lem-${backend}${ext}"
+            zip -jq "../out/${os}-${arch}-lem-driver-${backend}-${VERSION}.zip" "lem-${backend}${ext}"
+            rm -f "lem-${backend}${ext}"
+          done
+          ls -la ../out
 
       - name: Publish rolling dev prerelease
         uses: softprops/action-gh-release@v2
         with:
           tag_name: dev
           name: dev (rolling)
           prerelease: true
-          files: dist/*.zip
+          files: out/*.zip
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
@@ -101,7 +173,10 @@
       - uses: actions/checkout@v4
       - uses: actions/setup-go@v5
         with:
-          go-version: "1.26"
+          go-version: ${{ env.GO_VERSION }}
+          cache-dependency-path: |
+            go/go.sum
+            cli/go.sum
       - uses: actions/setup-node@v4
         with:
           node-version: "20"
@@ -111,7 +186,7 @@
           java-version: "21"
       - name: Install generators
         run: |
-          npm install -g @openapitools/openapi-generator-cli@7.22.0
+          npm install -g @openapitools/openapi-generator-cli@2.40.0   # npm wrapper is 2.x; it manages the 7.x Java generator
           go install github.com/go-task/task/v3/cmd/task@latest
       - name: Generate SDKs (lem spec -> OpenAPI 3.1 -> typed clients)
         run: task sdk

.github/workflows/ci.yml

@@ -26,6 +26,10 @@ jobs:
       - uses: actions/setup-go@v5
         with:
           go-version: "1.26"
+          # The module root is go/ (go.sum lives there, not at the repo root); without
+          # this, setup-go's cache aborts with "Dependencies file is not found ...
+          # Supported file pattern: go.sum". This lane only compiles the go/ module.
+          cache-dependency-path: go/go.sum
 
       - name: Portable tests + coverage (default tags — engine/metal excluded)
         working-directory: go

Taskfile.yml

@@ -18,8 +18,13 @@ vars:
   VERSION: dev
   GO_VERSION_LDFLAGS: '-X main.version={{.VERSION}}'
   GO_DARWIN_LDFLAGS: '-extldflags=-mmacosx-version-min=26.0'
+  # Core count for cmake --parallel (metallib:mlx only). go-task evaluates global
+  # sh-vars eagerly on every invocation, so this must succeed on EVERY platform:
+  # getconf/nproc cover Linux, sysctl -n hw.ncpu is the macOS path, and the `echo 4`
+  # floor keeps it green where none of the three exist (Windows, task's internal
+  # POSIX shell has no such binaries) — a bare `sysctl -n hw.ncpu` was the CI failure.
   NCPU:
-    sh: getconf _NPROCESSORS_ONLN 2>/dev/null || nproc 2>/dev/null || sysctl -n hw.ncpu
+    sh: getconf _NPROCESSORS_ONLN 2>/dev/null || nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4
 env:
   MLX_METALLIB_PATH: '{{.ROOT_DIR}}/build/dist/lib/mlx.metallib'
 

docs/release-artifacts.md

@@ -0,0 +1,124 @@
+# Release artifacts — the `lem` Driver/Native grid
+
+`lem` is one binary. The GPU backend is registered at **build** time, and `lem` also
+serves its own HTTP API — so a compute box can run a backend-specific `lem` as a remote
+GPU **driver**, and a user's local `lem` consumes it over that API. The release surface
+therefore ships every per-`{os,arch,backend}` build in **two packagings of the same binary**:
+
+| Packaging | Zip name | Contains (one binary, nothing else) | Role |
+|-----------|----------|-------------------------------------|------|
+| **Native** | `{os}-{arch}-lem-{backend}-{version}.zip` | `lem` (`lem.exe` on Windows) | Run locally on your machine. |
+| **Driver** | `{os}-{arch}-lem-driver-{backend}-{version}.zip` | `lem-{backend}` (`.exe` on Windows) | Deploy on a compute box; serves the GPU over the API. |
+
+Worked example (the maintainer's own setup):
+
+- On his Mac: `macos-aarch64-lem-metal-v0.15.0.zip` → a `lem` binary for mac + metal.
+- On his AMD Linux box: `linux-x86_64-lem-driver-amd-v0.15.0.zip` → a `lem-amd` binary
+  serving the GPU over the API, which the Mac's `lem` then drives remotely.
+
+**Build once, package twice.** Each cell is built a single time; the one binary is copied
+into the Native zip as `lem` and into the Driver zip as `lem-{backend}`. No double build.
+
+## Dimensions
+
+- `os` ∈ {`macos`, `linux`, `windows`}  (GOOS `darwin`→`macos`)
+- `arch` ∈ {`x86_64`, `aarch64`}  (GOARCH `amd64`→`x86_64`, `arm64`→`aarch64`)
+- `backend` ∈ {`metal`, `amd`, `cuda`, `cpu`}
+- `version` = the git tag on a release build; the rolling `dev` prerelease uses `dev-<short-sha>`.
+
+## The grid — valid cells only
+
+| os | arch | backend | Built by | How |
+|----|------|---------|----------|-----|
+| macos | aarch64 | metal | self-hosted macOS 26 lane (dormant) / manual | `task metallib && task build:embed` |
+| macos | aarch64 | cpu | **GitHub** `macos-latest` | native cgo `go build ./cli` |
+| linux | x86_64 | amd | **GitLab** homelab | `make lem-amd` |
+| linux | x86_64 | cuda | **GitLab** homelab | `make lem-cuda` |
+| linux | x86_64 | cpu | **GitHub** `ubuntu-latest` (rolling) · **GitLab** `make lem-cpu-x86` (tag) | native cgo |
+| linux | aarch64 | cpu | **GitHub** `ubuntu-24.04-arm` (rolling) · **GitLab** `make lem-cpu-aarch64` (tag) | native cgo |
+
+### Excluded cells (and why)
+
+- **cpu · macos/x86_64** — the `macos-13` runner label is retired (live-proven 2026-07-22:
+  queued forever); GitHub hosts no Intel Mac any more. Revisit only on real demand.
+- **cpu · windows/x86_64** — lem's code is not Windows-portable yet: unix-only
+  `syscall.Kill` + raw-fd reads sit outside build constraints (live-proven 2026-07-22,
+  three compile errors). Restoring the cell is a code-portability campaign, not a
+  matrix entry.
+- **cpu · windows/aarch64** — `duckdb-go-bindings` ships no `windows-arm64` prebuilt lib, so
+  the binary cannot link. Not a cell.
+- **metal · anything but macos/aarch64** — Metal is Apple-GPU only.
+- **amd, cuda · anything but linux/x86_64** — need the ROCm / CUDA toolchain and its Linux libs.
+- **any backend with `CGO_ENABLED=0`** — impossible; see below.
+
+## Why every cell is a *native* cgo build (no cross-compile)
+
+`lem` links **DuckDB** (the `go-store` / chathistory driver) through cgo. `duckdb-go-bindings`
+guards its prebuilt-lib packages with `//go:build cgo`, so `CGO_ENABLED=0` fails at compile
+(`build constraints exclude all Go files ...`). There is **no pure-Go `lem`** and therefore
+**no free `GOOS/GOARCH` cross-compile** — cross-building needs a target C toolchain. Each cell
+is built natively on a runner of that `os/arch` (GitLab's `lem-cpu-aarch64` is the one
+cross-cgo build, via `aarch64-linux-gnu-gcc`). Probed on 2026-07-22 against
+`duckdb-go-bindings v0.10504.0`: cgo-off builds fail for every target; native cgo builds pass.
+
+DuckDB's supported set (the prebuilt libs that exist): `darwin/{amd64,arm64}`,
+`linux/{amd64,arm64}`, `windows/amd64`. That set defines the valid `cpu` cells.
+
+## Which CI owns which cells
+
+- **GitHub Actions** (`.github/workflows/build.yml`) — the portable **cpu** cells it can build
+  natively on hosted runners, on every push to `dev`, published to the rolling `dev`
+  prerelease as `dev-<short-sha>` zips (Native + Driver). Hosted runners have no ROCm/CUDA
+  toolchain and are older than macOS 26, so they build **only** cpu.
+- **GitLab CI** (`.gitlab-ci.yml`, homelab AMD box) — the toolchain-gated **amd** and **cuda**
+  cells, plus its own **cpu-x86 / cpu-aarch64**, on a git **tag** via the `release` gate and
+  the `Makefile` targets. This side is unchanged (it is green and has the toolchain).
+- **Self-hosted Mac lane** — the **metal** cell (see below).
+
+The two systems do not fight: GitHub is the continuous `dev` channel; GitLab is the tagged
+release for the Linux/GPU cells.
+
+## The metal cell decision
+
+Metal's Go side (engine/metal, the objc bridge) is no-cgo and cross-compiles, **but** a usable
+metal `lem` embeds the MLX + lthn `.metallib` kernels (`-tags embed_metallib`), and those are
+compiled by `task metallib` with **Metal 4 / macOS 26** (`-std=metal4.0`,
+`CMAKE_OSX_DEPLOYMENT_TARGET=26.0`) and are gitignored (generated, never committed).
+
+- A GitHub **hosted** macOS runner is macOS 14/15 → it **cannot** compile the metallibs. A plain
+  (non-embed) build would resolve them at runtime via `MLX_METALLIB_PATH`, but the user has no
+  metallib on their machine — so that artifact is **non-runnable**. We do **not** ship a
+  fabricated/broken metal binary from hosted CI.
+- **Decision:** the metal cell ships from a **self-hosted macOS 26 arm64 runner** running the
+  house build path (`task metallib && task build:embed`), producing the self-contained binary.
+  The `metal` job in `build.yml` encodes this recipe but is **dormant** — its `if` is false
+  unless repo variable `ENABLE_MACOS_METAL=true` is set — so it never queues or fails on a
+  normal push. **Interim** (until a self-hosted runner is registered): the maintainer builds
+  the metal zip manually on his Mac (the primary dev box already runs `task build:embed` daily).
+- **Trade-off:** correctness (a self-contained, runnable metal binary) over convenience (a
+  hosted job). The cost is one self-hosted runner or a manual step for the metal cell only.
+
+## Zip layout rule
+
+Each zip contains **exactly one binary and nothing else** — matching what `build.yml` shipped
+before (binary only, no licence file). Self-contained cells honour this directly:
+
+- **cpu** — native-cgo `lem`, no GPU engine, no sidecar.
+- **metal** — `build:embed` bakes the metallibs in.
+
+**Exception — amd / cuda:** these load a HIP/CUDA kernel **sidecar** (`.hsaco` / `.o`) at
+runtime that is not embedded, so those cells ship the binary **plus** its sidecar (the
+`Makefile` already tars them together). This is the single documented departure from
+binary-only, driven by runtime necessity.
+
+## Not yet wired (follow-ups)
+
+- **Tagged GitHub release.** `build.yml` publishes only the rolling `dev` prerelease (dev
+  pushes). Producing tag-versioned (`v*`) zips for the GitHub cells (macos/windows cpu, metal)
+  is a small extension: add `push.tags` to `on:` and version from the tag. GitLab already
+  handles tagged Linux/GPU cells.
+- **GitLab grid-named zips.** GitLab currently uploads the raw `lem-{backend}` binaries (already
+  Driver-named) via `build/bin/`. Emitting the `{os}-{arch}-lem-...-{version}.zip` names (with
+  the amd/cuda sidecars) is a `Makefile`/`.gitlab-ci.yml` packaging follow-up that must be
+  validated on the ROCm box — it cannot be exercised from a macOS lane, so it was left untouched
+  here rather than changed blind.

go/agent/ai/ai_test.go

@@ -120,7 +120,7 @@ func TestReadEvents_Good_SkipsMissingDays(t *testing.T) {
 	if len(events) != 2 {
 		t.Fatalf("expected 2 events, got %d", len(events))
 	}
-	if events[0].Timestamp != dayOne || events[1].Timestamp != dayThree {
+	if !events[0].Timestamp.Equal(dayOne) || !events[1].Timestamp.Equal(dayThree) {
 		t.Fatalf("events not returned in chronological order: %+v", events)
 	}
 }