|
| 1 | +# Self-hosted CI orchestration |
| 2 | + |
| 3 | +`cppmega_mlx` uses GitHub Actions only as a coordinator. Every job with a |
| 4 | +`runs-on` key targets a repository-owned runner; hosted macOS, Linux, and |
| 5 | +Windows labels are rejected by `tests/test_workflow_runner_policy.py`. |
| 6 | + |
| 7 | +The registration pattern comes from the live `cloud_love` setup: |
| 8 | + |
| 9 | +- workflows target stable custom labels in addition to `self-hosted` and the |
| 10 | + operating-system label; |
| 11 | +- runner registration tokens are temporary GitHub secrets consumed by |
| 12 | + `cloud_love/.github/workflows/bootstrap-cppmega-runners.yml`; |
| 13 | +- no runner token, SSH password, private key, or other credential is stored in |
| 14 | + this repository; |
| 15 | +- all external workflow actions are pinned to full commit SHAs. |
| 16 | + |
| 17 | +## Verified inventory |
| 18 | + |
| 19 | +Snapshot taken 2026-07-14. Online/busy state is live data and must be refreshed |
| 20 | +before an operational claim: |
| 21 | + |
| 22 | +| Inventory ID | Address | Role | GitHub runner and labels | Direct status at verification | |
| 23 | +|---|---:|---|---|---| |
| 24 | +| `mac-studio` | `10.0.0.8` | required macOS MLX | `mac-studio-cppmega-mlx`; `self-hosted,macOS,ARM64,cppmega-mlx-macos` | local and available | |
| 25 | +| `legion-linux` | `10.0.0.16` | required portable Linux | `davidgor-Legion-R9000P-ARX8-cppmega-mlx`; `self-hosted,Linux,X64,cppmega-mlx` | TCP/22 reachable; batch SSH authentication unavailable from the Mac Studio | |
| 26 | +| `windows-10-0-0-11` | `10.0.0.11` | inventory only | no `cppmega_mlx` runner | Windows is not eligible for either lane; batch SSH authentication unavailable | |
| 27 | +| `untrusted-10-0-0-12` | `10.0.0.12` | quarantined inventory only | none | TCP/22 reachable, but the current host key does not match the saved key | |
| 28 | + |
| 29 | +Do not bypass the `10.0.0.12` host-key error with |
| 30 | +`StrictHostKeyChecking=no`. Verify the replacement machine and fingerprint out |
| 31 | +of band, update `known_hosts`, and only then assign it a test lane. |
| 32 | + |
| 33 | +Refresh GitHub's runner view without printing credentials: |
| 34 | + |
| 35 | +```bash |
| 36 | +gh api repos/DatasunriseOU/cppmega_mlx/actions/runners \ |
| 37 | + --jq '.runners[] | {name,os,status,busy,labels:[.labels[].name]}' |
| 38 | +``` |
| 39 | + |
| 40 | +## Shared suites |
| 41 | + |
| 42 | +Both GitHub jobs and the direct orchestrator call |
| 43 | +`scripts/run_self_hosted_ci.py lane`: |
| 44 | + |
| 45 | +- `macos-mlx` runs the focused data, model, inference, evaluation, and |
| 46 | + orchestration policy tests with the established local MLX environment. |
| 47 | +- `linux-portable` creates an isolated temporary venv, installs only the |
| 48 | + already-established portable test dependencies, runs tests with |
| 49 | + `--noconftest`, compiles portable scripts, and never imports Apple's MLX |
| 50 | + runtime. |
| 51 | + |
| 52 | +Each process is started in its own process group. A timeout terminates the |
| 53 | +whole group, returns exit code 124, and records `timed_out` instead of leaving |
| 54 | +children behind. Each step has its own log and the lane always writes |
| 55 | +`receipt.json` with host, source commit, timing, command, status, and exit code. |
| 56 | +The workflows upload these directories with 14-day retention even when a lane |
| 57 | +fails. |
| 58 | + |
| 59 | +## Direct CLI |
| 60 | + |
| 61 | +The direct path does not use GitHub Actions. It probes all selected hosts with |
| 62 | +non-interactive SSH, aborts before dispatch if any required host is |
| 63 | +unavailable, stages the exact requested Git commit, runs the same lane entry |
| 64 | +point, and collects each host's receipt and logs. |
| 65 | + |
| 66 | +Read-only probe and plan: |
| 67 | + |
| 68 | +```bash |
| 69 | +python3 scripts/run_self_hosted_ci.py orchestrate \ |
| 70 | + --dry-run \ |
| 71 | + --receipt-dir /tmp/cppmega-mlx-self-hosted |
| 72 | +``` |
| 73 | + |
| 74 | +Run only the local macOS lane: |
| 75 | + |
| 76 | +```bash |
| 77 | +python3 scripts/run_self_hosted_ci.py orchestrate \ |
| 78 | + --host mac-studio \ |
| 79 | + --ref HEAD \ |
| 80 | + --receipt-dir /tmp/cppmega-mlx-self-hosted |
| 81 | +``` |
| 82 | + |
| 83 | +Run the full matrix after key-based SSH to `10.0.0.16` is available: |
| 84 | + |
| 85 | +```bash |
| 86 | +python3 scripts/run_self_hosted_ci.py orchestrate \ |
| 87 | + --ref HEAD \ |
| 88 | + --receipt-dir /tmp/cppmega-mlx-self-hosted |
| 89 | +``` |
| 90 | + |
| 91 | +The CLI never accepts a password or token argument. SSH uses `BatchMode=yes`, |
| 92 | +disables password and keyboard-interactive authentication, and preserves normal |
| 93 | +host-key verification. Configure an agent-backed SSH key outside the |
| 94 | +repository. The source worktree may be dirty; dispatch still uses the exact |
| 95 | +resolved commit and records `source_dirty` in the orchestration receipt. |
| 96 | + |
| 97 | +Receipts are written under: |
| 98 | + |
| 99 | +```text |
| 100 | +<receipt-dir>/<run-id>/orchestration.json |
| 101 | +<receipt-dir>/<run-id>/<host-id>/probe.json |
| 102 | +<receipt-dir>/<run-id>/<host-id>/receipt.json |
| 103 | +<receipt-dir>/<run-id>/<host-id>/*.log |
| 104 | +``` |
| 105 | + |
| 106 | +Exit code 2 means orchestration/preflight was blocked, exit code 1 means a |
| 107 | +dispatched lane failed, and exit code 0 means every selected required lane |
| 108 | +passed (or a dry-run found every selected required host available). |
| 109 | + |
| 110 | +## Preset shard incident |
| 111 | + |
| 112 | +PR run `29301626406`, job `86986403882`, proved that increasing shard1 to 40 |
| 113 | +minutes was not a fix: setup finished at `03:00:37Z`, the Playwright step ran |
| 114 | +until `03:36:05Z`, and the job hit its 40-minute cap. The matrix is one spec |
| 115 | +file and the base config has `fullyParallel: false`, so Playwright's file-level |
| 116 | +sharding assigned all 912 tests to shard 1 while the other shards had no tests. |
| 117 | + |
| 118 | +The self-hosted workflow now passes `--fully-parallel`, retaining four shards |
| 119 | +that contain test cells rather than files. Playwright has a 12-minute global |
| 120 | +cap and each job has a 20-minute cap, leaving setup and artifact-upload |
| 121 | +headroom while still producing logs on a test timeout. |
0 commit comments