|
| 1 | +# Spec: Registry Test Standardization |
| 2 | + |
| 3 | +Status: proposal · Owner: registry · Last updated: 2026-07-07 |
| 4 | + |
| 5 | +## Problem |
| 6 | + |
| 7 | +Every registry integration test lives under `registry/tests/`, organized by |
| 8 | +**where the WASM binaries are built (`registry/native/`), not by what is being |
| 9 | +tested.** The harness (`registry/tests/helpers.ts`) resolves command binaries at |
| 10 | +`registry/tests/../native/...`, so anything needing a real WASM command was |
| 11 | +parked next to the native build — including tests that have nothing to do with |
| 12 | +registry packages (npm e2e, VFS, sockets, libc conformance). |
| 13 | + |
| 14 | +That layout coupling is already obsolete: |
| 15 | + |
| 16 | +- `packages/runtime-core/scripts/copy-wasm-commands.mjs` vendors the binaries |
| 17 | + into `packages/runtime-core/commands/`, and CI stages them there. |
| 18 | +- The harness already supports an env seam: |
| 19 | + `AGENTOS_WASM_COMMANDS_DIR` / `AGENTOS_C_WASM_COMMANDS_DIR`. |
| 20 | +- `packages/runtime-core/` already has its own `tests/` + `vitest.config.ts`. |
| 21 | + |
| 22 | +Consequences today: the `agentos-registry` test package is **not a pnpm |
| 23 | +workspace member**, so root `pnpm test` (turbo, member-filtered) never runs it; |
| 24 | +`@xterm/headless` doesn't resolve without a manual symlink; and there is no way |
| 25 | +to answer "which packages have no test?" mechanically. |
| 26 | + |
| 27 | +## Goals |
| 28 | + |
| 29 | +1. Organize tests by **subject under test**, not by binary location. |
| 30 | +2. Make "does package X have a test?" answerable by `ls` + a coverage gate. |
| 31 | +3. Re-attach the suites to CI (turbo) so they actually run. |
| 32 | +4. **Report test status; never silently fix or hide failures.** Migration must |
| 33 | + not touch package behavior. A package whose test fails moves as-is and is |
| 34 | + recorded as `failing` — see [Non-goals](#non-goals). |
| 35 | + |
| 36 | +## Non-goals |
| 37 | + |
| 38 | +- **Do not fix failing packages.** Moving a test must not change the command's |
| 39 | + behavior to make a red test green. If a test fails after moving, its status is |
| 40 | + `failing` and it is tracked in the manifest, not patched. |
| 41 | +- No new test coverage is *required* by the migration itself. Writing the |
| 42 | + missing tests (the `no-test` rows) is follow-up work, tracked but separate. |
| 43 | + |
| 44 | +## Target layout |
| 45 | + |
| 46 | +| Home | Contains | Rationale | |
| 47 | +|---|---|---| |
| 48 | +| `registry/software/<pkg>/test/` | Tests exercising **one package's command** | The package owns its behavior | |
| 49 | +| `packages/runtime-core/tests/integration/` | Tests of the **VM/kernel** using commands as generic fixtures | runtime-core already vendors the binaries and has a test setup | |
| 50 | +| `registry/native/tests/` | **libc/sysroot conformance** — validates the C toolchain | Lives next to the artifact it validates | |
| 51 | +| `packages/vm-test-harness/` (new, workspace member) | `helpers.ts` + `terminal-harness.ts` shared harness | One import surface; fixes `@xterm/headless` resolution and turbo wiring | |
| 52 | + |
| 53 | +Binaries are injected via `AGENTOS_WASM_COMMANDS_DIR` pointing at |
| 54 | +`packages/runtime-core/commands` — no relative-path coupling to `registry/native`. |
| 55 | + |
| 56 | +## The rule (enforced) |
| 57 | + |
| 58 | +> Every `*.test.ts` is either **owned by exactly one package** |
| 59 | +> (`software/<pkg>/test/`) **or** lives in a runtime/toolchain home with a |
| 60 | +> one-line reason header. No loose files. No test organized by binary location. |
| 61 | +
|
| 62 | +CI coverage gate (`registry/scripts/check-test-coverage.mjs`): |
| 63 | +- Fail if any `software/*` has neither a `test/` dir nor an entry in the |
| 64 | + **known-gap allowlist**. |
| 65 | +- Allowlist (no direct test expected): meta bundles `common`, |
| 66 | + `build-essential`, `everything`; external wrappers `browserbase`, `vix`. |
| 67 | + |
| 68 | +## Status reporter (`registry/scripts/test-status.mjs`) |
| 69 | + |
| 70 | +A report-only tool. It **does not fix anything** and its default exit code does |
| 71 | +not fail on known-tracked failures. |
| 72 | + |
| 73 | +Behavior: |
| 74 | +1. Enumerate `software/*` + `agent/*`. |
| 75 | +2. For each, resolve its test dir (post-migration) and run it via vitest with |
| 76 | + binaries from `runtime-core/commands`. |
| 77 | +3. Classify each package into one status (below) and print a table + write |
| 78 | + `registry/test-status.json`. |
| 79 | +4. Reconcile against the **expected-status manifest** in this spec: |
| 80 | + - A package whose result matches its recorded status → OK. |
| 81 | + - A `failing`/`not-compiling` package that is still failing → **reported, not |
| 82 | + an error** (exit 0). This is the "don't fix broken packages" contract. |
| 83 | + - A package that **regressed** (recorded `working`, now failing) → exit |
| 84 | + non-zero. Only *new* breakage fails CI. |
| 85 | + |
| 86 | +Status values: |
| 87 | + |
| 88 | +| Status | Meaning | |
| 89 | +|---|---| |
| 90 | +| `working` | Live test, all (or all-but-hardening) pass | |
| 91 | +| `failing` | Live test, real functional failures — tracked, not fixed | |
| 92 | +| `disabled` | Test file exists but is `describe.skip` | |
| 93 | +| `no-test` | Compiles, no test exists | |
| 94 | +| `not-compiling` | Build fails | |
| 95 | +| `not-buildable` | Needs an external artifact absent from this checkout | |
| 96 | +| `meta` | Aggregate bundle — no direct test | |
| 97 | + |
| 98 | +## Migration & coverage manifest |
| 99 | + |
| 100 | +Statuses below are from the 2026-07-07 audit (full run: |
| 101 | +8 failed / 6 passed / 9 skipped files; 34 failed / 84 passed / 59 skipped tests). |
| 102 | +The reporter regenerates them; they are the baseline the reporter reconciles |
| 103 | +against. |
| 104 | + |
| 105 | +### Software packages (29) |
| 106 | + |
| 107 | +| Package | Command(s) | Current test | Action | Target | Status | |
| 108 | +|---|---|---|---|---|---| |
| 109 | +| coreutils | `sh` +80 | `shell-terminal`, `shell-redirect` | move | `software/coreutils/test/` | `working` | |
| 110 | +| fd | `fd` | `fd-find` (shared) | **split** + move | `software/fd/test/` | `working` | |
| 111 | +| findutils | `find`,`xargs` | `fd-find` (shared) | **split** + move | `software/findutils/test/` | `working` | |
| 112 | +| tree | `tree` | `kernel/tree-test` (**misfiled**) | move | `software/tree/test/` | `working` | |
| 113 | +| sqlite3 | `sqlite3` | `sqlite3` | move | `software/sqlite3/test/` | `failing` (1/16: VFS `pwrite`) | |
| 114 | +| zip | `zip` | `zip-unzip` (shared) | **split** + move | `software/zip/test/` | `failing` (3/8 hardening) | |
| 115 | +| unzip | `unzip` | `zip-unzip` (shared) | **split** + move | `software/unzip/test/` | `failing` (3/8 hardening) | |
| 116 | +| curl | `curl` | `curl` | move | `software/curl/test/` | `failing` (24/30; exits 1 incl. `--version`) | |
| 117 | +| git | `git` | `git` (`describe.skip`) | move + unskip via `describeIf` | `software/git/test/` | `disabled` (compiles) | |
| 118 | +| duckdb | `duckdb` | `duckdb` (`describe.skip`) | move + `describeIf` | `software/duckdb/test/` | `disabled` (compiles) | |
| 119 | +| wget | `wget` | `wget` (`describe.skip`) | move + `describeIf` | `software/wget/test/` | `not-compiling` (dup `getpeername`) | |
| 120 | +| codex-cli | `codex`,`codex-exec` | `codex-exec`,`codex-tui` (`describe.skip`) | move + `describeIf` | `software/codex-cli/test/` | `not-buildable` (needs codex-rs fork) | |
| 121 | +| gawk | `awk` | — | **write new** | `software/gawk/test/` | `no-test` | |
| 122 | +| sed | `sed` | — | **write new** | `software/sed/test/` | `no-test` | |
| 123 | +| grep | `grep` | — (only a vehicle in `dynamic-module`) | **write new** | `software/grep/test/` | `no-test` | |
| 124 | +| tar | `tar` | — | **write new** | `software/tar/test/` | `no-test` | |
| 125 | +| gzip | `gzip` | — | **write new** | `software/gzip/test/` | `no-test` | |
| 126 | +| jq | `jq` | — | **write new** | `software/jq/test/` | `no-test` | |
| 127 | +| ripgrep | `rg` | — | **write new** | `software/ripgrep/test/` | `no-test` | |
| 128 | +| yq | `yq` | — | **write new** | `software/yq/test/` | `no-test` | |
| 129 | +| diffutils | `diff` | — | **write new** | `software/diffutils/test/` | `no-test` | |
| 130 | +| file | `file` | — | **write new** | `software/file/test/` | `no-test` | |
| 131 | +| http-get | `http_get` | — | **write new** | `software/http-get/test/` | `no-test` | |
| 132 | +| vim | `vim` | — | **write new** | `software/vim/test/` | `no-test` (compile unverified; heavy `make cmd/vim`) | |
| 133 | +| vix | `vix` | — | allowlist | — | `not-buildable` (external, no source) | |
| 134 | +| browserbase | (`browse`) | — | allowlist | — | `no-test` (external CLI wrapper) | |
| 135 | +| common | (bundle) | — | allowlist | — | `meta` | |
| 136 | +| build-essential | (bundle) | — | allowlist | — | `meta` | |
| 137 | +| everything | (bundle) | — | allowlist | — | `meta` | |
| 138 | + |
| 139 | +### Agent packages (5) |
| 140 | + |
| 141 | +All are JS ACP adapters with **no integration test**. Decision needed: adapter |
| 142 | +smoke tests, or explicitly out of scope (allowlist). |
| 143 | + |
| 144 | +| Package | Action | Status | |
| 145 | +|---|---|---| |
| 146 | +| claude | write new / allowlist | `no-test` | |
| 147 | +| codex | write new / allowlist | `no-test` | |
| 148 | +| opencode | write new / allowlist | `no-test` | |
| 149 | +| pi | write new / allowlist | `no-test` | |
| 150 | +| pi-cli | write new / allowlist | `no-test` | |
| 151 | + |
| 152 | +### Not-a-package tests → relocate out of `registry/tests/` |
| 153 | + |
| 154 | +**→ `packages/runtime-core/tests/integration/`** (VM/kernel): |
| 155 | +- from `wasmvm/`: `net-server`, `net-udp`, `net-unix`, `signal-handler`, |
| 156 | + `wasi-http`, `wasi-spawn`, `dynamic-module-integration` |
| 157 | +- **all of `kernel/`** except `tree-test`: `e2e-npm-*`, `e2e-nextjs-build`, |
| 158 | + `e2e-concurrently`, `e2e-npx-and-pipes`, `e2e-project-matrix`, |
| 159 | + `cross-runtime-*`, `bridge-child-process`, `ctrl-c-shell-behavior`, |
| 160 | + `dispose-behavior`, `error-propagation`, `exec-integration`, |
| 161 | + `fd-inheritance`, `module-resolution`, `node-binary-behavior`, |
| 162 | + `shim-streaming`, `signal-forwarding`, `vfs-consistency` |
| 163 | +- `registry/tests/projects/` (~45 npm fixtures) — move with the kernel e2e |
| 164 | + |
| 165 | +**→ `registry/native/tests/`** (toolchain/libc conformance): |
| 166 | +- `os-test-conformance`, `libc-test-conformance`, `c-parity`, |
| 167 | + `ci-artifact-availability`, `kernel/ci-wasm-artifact-availability` |
| 168 | + |
| 169 | +**Decision:** `envsubst` — command has **no owning package**. Either give it one |
| 170 | +(coreutils?) or move `envsubst.test.ts` to runtime. Currently `working` (6/6). |
| 171 | + |
| 172 | +## Migration phases |
| 173 | + |
| 174 | +1. **Foundation** — extract `packages/vm-test-harness`; make it a workspace |
| 175 | + member; point it at `runtime-core/commands` via env. Wire turbo `test`. |
| 176 | + (Fixes xterm + CI-orphan bugs; moves nothing yet.) |
| 177 | +2. **Relocate runtime + toolchain tests** out of `registry/tests/` per the lists |
| 178 | + above. Pure `git mv`, no behavior change. |
| 179 | +3. **Co-locate package tests** into `software/<pkg>/test/`; split the two shared |
| 180 | + files (`fd-find`, `zip-unzip`); move misfiled `tree-test`; swap |
| 181 | + `describe.skip` → `describeIf(binaryPresent)`. |
| 182 | +4. **Gate** — add `check-test-coverage.mjs` + `test-status.mjs`; commit the |
| 183 | + baseline `test-status.json`; fail CI only on regressions. |
| 184 | +5. **Backfill** — write the `no-test` suites (12 software + 5 agents). Separate |
| 185 | + PRs, tracked by the manifest. |
| 186 | + |
| 187 | +## Open decisions |
| 188 | + |
| 189 | +- Runtime tests home: **`runtime-core/tests/`** (recommended, binaries already |
| 190 | + there) vs. a new `packages/vm-integration-tests`. |
| 191 | +- `envsubst` owner. |
| 192 | +- Agent adapter tests: in scope or allowlist. |
| 193 | +- `describe.skip` → `describeIf`: gate on binary presence, or also on |
| 194 | + network/env (git remote clone, codex `OPENAI_API_KEY`). |
0 commit comments