perf(ci): shard tests for ≤60s per ubuntu job; PR runs ubuntu-only fast path

Cut the monolithic test job (single ~190s vitest run, gated by a 170s cluster
file; 12-29min on CI) into parallel, wall-time-balanced shards so every job on
the PR critical path stays ≤60s — verified on real CI (48 ubuntu shards run
15-59s).

PR critical path = ubuntu only:
- Test sharding (scripts/run-shard.js, driven by the CI matrix): fork-heavy
  packages split via vitest --shard (cluster→3, egg→2, mock→2, schedule→4,
  security→2, plus redis/multipart/development standalone), each capped with
  --maxWorkers so forked egg cluster processes don't oversubscribe the CPU; the
  light remainder greedy bin-packed by per-dir cpu weight into 8 rest-* shards.
- Long single test files split by `describe` block (no semantic change, no
  removed cases) so vitest can parallelize them.
- quality split into a parallel matrix (lint/typecheck/fmtcheck/build/site) so
  the slowest check — not their sum — bounds wall time.
- egg-bin runs without --coverage to stay ≤60s (no threshold; report only).

macOS/Windows whole-suite compatibility (slow: ~13-25min; macOS has only ~5
runner slots) moved to a `test-compat` job that runs at merge time
(merge_group) and on push to next — NOT on pull_request — so it no longer gates
the PR wall while still enforcing cross-platform coverage before code lands.
CI test jobs on a PR drop from 132 to the 48 ubuntu shards.

Latent bugs fixed along the way:
- tegg controller ControllerMetaManager.test.ts triggers a duplicate-proto
  error that polluted the process-global registry under isolate:false, failing
  sibling files; isolated that project (isolate:true).
- clean-dist used `ut run clean --workspaces` which aborts the && chain (root has
  no clean script) → scripts/clean-dist.js.
- typecheck/pretest used `ut run <self> --workspaces`, which includes the root
  and recurses infinitely → scripts/run-workspaces.js (concurrent, root-bin PATH).
- Windows runner images no longer set %HOME%, breaking egg_loader getHomedir;
  export HOME=%USERPROFILE% on the windows compat job.

Windows flakes absorbed by --retry. No test semantics changed or cases removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: killagu

.github/workflows/ci.yml

@@ -18,11 +18,20 @@ permissions:
   actions: write
 
 jobs:
-  typecheck:
+  # Static-analysis / build checks. Each check runs as its own parallel job
+  # (instead of serial steps) so the slowest one — not their sum — bounds the
+  # wall time. Goal: keep each ≤ 60s.
+  quality:
+    strategy:
+      fail-fast: false
+      matrix:
+        check: ['lint', 'typecheck', 'fmtcheck', 'build', 'site']
+
+    name: Quality (${{ matrix.check }})
     runs-on: ubuntu-latest
 
     concurrency:
-      group: typecheck-${{ github.workflow }}-#${{ github.event.pull_request.number || github.head_ref || github.ref }}
+      group: quality-${{ matrix.check }}-${{ github.workflow }}-#${{ github.event.pull_request.number || github.head_ref || github.ref }}
       cancel-in-progress: true
     steps:
       - name: Checkout repository
@@ -41,40 +50,151 @@ jobs:
         run: ut install --from pnpm || (sleep 5 && ut install --from pnpm) || (sleep 10 && ut install --from pnpm)
 
       - name: Run lint
+        if: ${{ matrix.check == 'lint' }}
         run: ut run lint
 
       - name: Run typecheck
+        if: ${{ matrix.check == 'typecheck' }}
         run: ut run typecheck
 
       - name: Run format check
+        if: ${{ matrix.check == 'fmtcheck' }}
         run: ut run fmtcheck
 
       - name: Run build
+        if: ${{ matrix.check == 'build' }}
         run: ut run build
 
       - name: Run site build
+        if: ${{ matrix.check == 'site' }}
         run: ut run site:build
 
   test:
     strategy:
       fail-fast: false
       matrix:
-        os: ['ubuntu-latest', 'macos-latest', 'windows-latest']
+        # ubuntu-latest runs the fine-grained shards: fork-heavy packages are
+        # split into vitest --shard slices and the light remainder into
+        # weight-balanced rest-* shards (see scripts/run-shard.js). Sized so each
+        # shard's `vitest run` step stays ≤ ~60s on ubuntu-latest, where runner
+        # concurrency is high enough to run them all in parallel.
+        os: ['ubuntu-latest']
         node: ['22', '24']
+        shard:
+          - cluster-1
+          - cluster-2
+          - cluster-3
+          - egg-1
+          - egg-2
+          - mock-1
+          - mock-2
+          - schedule-1
+          - schedule-2
+          - schedule-3
+          - schedule-4
+          - development
+          - security-1
+          - security-2
+          - redis
+          - multipart
+          - rest-1
+          - rest-2
+          - rest-3
+          - rest-4
+          - rest-5
+          - rest-6
+          - rest-7
+          - rest-8
+        # macOS/Windows whole-suite compatibility runs live in `test-compat`,
+        # which only runs at merge time (merge_group / push to next) — not on
+        # every PR — because those runners are slow (~13-25min whole-suite) and
+        # macOS has only ~5 concurrent slots, so they would dominate the PR wall.
+        # The PR critical path is this ubuntu fan-out, every shard ≤60s.
+
+    name: Test (${{ matrix.os }}, ${{ matrix.node }}, ${{ matrix.shard }})
+    runs-on: ${{ matrix.os }}
+
+    concurrency:
+      group: test-${{ github.workflow }}-#${{ github.event.pull_request.number || github.head_ref || github.ref }}-(${{ matrix.os }}, ${{ matrix.node }}, ${{ matrix.shard }})
+      cancel-in-progress: true
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6
+
+      - name: Start Redis
+        uses: shogo82148/actions-setup-redis@cff708d63a30aebc0bfaa7276fb709d173f36cb6 # v1
+        with:
+          redis-version: '7'
+          auto-start: 'true'
+
+      - name: Start MySQL
+        uses: shogo82148/actions-setup-mysql@27e74fac04c136a9f4c2dc2ed457df57331b3e0c # v1
+        with:
+          mysql-version: '8'
+          auto-start: 'true'
+      - name: Init DB
+        run: |
+          mysql -uroot -e "CREATE DATABASE IF NOT EXISTS test;"
+
+      - name: Setup utoo
+        uses: utooland/setup-utoo@3a51006d0b66afcc32d1b9177a4b200b74f4a8cb # main
+
+      - name: Set up Node.js
+        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6
+        with:
+          node-version: ${{ matrix.node }}
+
+      - name: Install dependencies
+        run: ut install --from pnpm || (sleep 5 && ut install --from pnpm) || (sleep 10 && ut install --from pnpm)
+
+      - name: Prepare test fixtures (clean dist + db)
+        # Mirrors the original `preci` (pretest) step: clears stale dist so tegg
+        # plugin tests don't double-load src+dist, and runs per-workspace
+        # pretest (e.g. orm DB table init). The shard runner invokes vitest
+        # directly, bypassing the npm `preci` lifecycle, so do it explicitly.
+        run: ut run pretest
 
-    name: Test (${{ matrix.os }}, ${{ matrix.node }})
+      - name: Run tests (shard ${{ matrix.shard }})
+        run: node scripts/run-shard.js ${{ matrix.shard }} -- --coverage
+
+      - name: Run example tests
+        # Run once (on the rest-1 shard) to avoid duplication across shards.
+        if: ${{ matrix.shard == 'rest-1' }}
+        run: |
+          ut run example:test:all
+
+      - name: Code Coverage
+        uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5
+        with:
+          use_oidc: true
+
+  # macOS / Windows whole-suite compatibility runs. These are slow (~13-25min)
+  # and macOS has only ~5 concurrent runner slots, so they would dominate the
+  # PR wall. They run only at merge time (merge_group) and on push to next, so
+  # the PR critical path stays the ubuntu `test` fan-out (each shard ≤60s) while
+  # cross-platform coverage is still enforced before code lands.
+  test-compat:
+    if: ${{ github.event_name != 'pull_request' }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: ['macos-latest', 'windows-latest']
+        node: ['22', '24']
+
+    name: Test compat (${{ matrix.os }}, ${{ matrix.node }})
     runs-on: ${{ matrix.os }}
 
     concurrency:
-      group: test-${{ github.workflow }}-#${{ github.event.pull_request.number || github.head_ref || github.ref }}-(${{ matrix.os }}, ${{ matrix.node }})
+      group: test-compat-${{ github.workflow }}-#${{ github.event.pull_request.number || github.head_ref || github.ref }}-(${{ matrix.os }}, ${{ matrix.node }})
       cancel-in-progress: true
 
     steps:
       - name: Checkout repository
         uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6
 
-      - name: Start Redis (MacOS or Linux)
-        if: ${{ matrix.os == 'macos-latest' || matrix.os == 'ubuntu-latest' }}
+      - name: Start Redis (macOS)
+        if: ${{ matrix.os == 'macos-latest' }}
         uses: shogo82148/actions-setup-redis@cff708d63a30aebc0bfaa7276fb709d173f36cb6 # v1
         with:
           redis-version: '7'
@@ -130,15 +250,14 @@ jobs:
 
           Write-Host "Memurai is ready on 127.0.0.1:6379"
 
-      # install and start MySQL (will automatically start mysqld)
-      - name: Start MySQL (macOS or Linux)
-        if: ${{ matrix.os == 'macos-latest' || matrix.os == 'ubuntu-latest' }}
+      - name: Start MySQL (macOS)
+        if: ${{ matrix.os == 'macos-latest' }}
         uses: shogo82148/actions-setup-mysql@27e74fac04c136a9f4c2dc2ed457df57331b3e0c # v1
         with:
           mysql-version: '8'
           auto-start: 'true'
-      - name: Init DB (macOS or Linux)
-        if: ${{ matrix.os == 'macos-latest' || matrix.os == 'ubuntu-latest' }}
+      - name: Init DB (macOS)
+        if: ${{ matrix.os == 'macos-latest' }}
         run: |
           mysql -uroot -e "CREATE DATABASE IF NOT EXISTS test;"
 
@@ -148,8 +267,6 @@ jobs:
         run: |
           choco install -y mysql
           refreshenv
-          # MySQL default root has no password, set a password and create database/user
-          # & mysqladmin -u root password root
           & mysql -uroot -e "CREATE DATABASE IF NOT EXISTS test;"
 
       - name: Setup utoo
@@ -163,20 +280,16 @@ jobs:
       - name: Install dependencies
         run: ut install --from pnpm || (sleep 5 && ut install --from pnpm) || (sleep 10 && ut install --from pnpm)
 
-      - name: Run tests
-        run: ut run ci
+      - name: Prepare test fixtures (clean dist + db)
+        run: ut run pretest
 
-      - name: Run example tests
-        if: ${{ matrix.os != 'windows-latest' }}
-        run: |
-          ut run example:test:all
+      - name: Set HOME on Windows
+        if: ${{ matrix.os == 'windows-latest' }}
+        shell: pwsh
+        run: echo "HOME=$env:USERPROFILE" >> $env:GITHUB_ENV
 
-      - name: Code Coverage
-        # skip on windows, it will hangup on codecov
-        if: ${{ matrix.os != 'windows-latest' }}
-        uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5
-        with:
-          use_oidc: true
+      - name: Run tests (whole suite)
+        run: node scripts/run-shard.js all -- --coverage
 
   test-egg-bin:
     strategy:
@@ -207,17 +320,19 @@ jobs:
       - name: Install dependencies
         run: ut install --from pnpm || (sleep 5 && ut install --from pnpm) || (sleep 10 && ut install --from pnpm)
 
+      - name: Set HOME on Windows
+        if: ${{ matrix.os == 'windows-latest' }}
+        shell: pwsh
+        run: echo "HOME=$env:USERPROFILE" >> $env:GITHUB_ENV
+
       - name: Run tests
+        # `ci` (vitest --coverage) adds ~50s of instrumentation on egg-bin's
+        # fork-heavy suite, pushing it past 60s. Run without coverage (`test`)
+        # to keep the job ≤60s; egg-bin's fork tests can't be sharded (they
+        # depend on cross-file shared state under isolate:false).
         run: |
           ut run build -- --workspace ./tools/egg-bin
-          ut run ci --workspace @eggjs/bin
-
-      - name: Code Coverage
-        # skip on windows, it will hangup on codecov https://github.com/codecov/codecov-action/issues/1787
-        if: ${{ matrix.os != 'windows-latest' }}
-        uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5
-        with:
-          use_oidc: true
+          ut run test --workspace @eggjs/bin
 
   test-egg-scripts:
     strategy:
@@ -263,9 +378,12 @@ jobs:
     runs-on: ubuntu-latest
     needs:
       - test
+      - test-compat
       - test-egg-bin
       - test-egg-scripts
-      - typecheck
+      - quality
     steps:
       - run: exit 1
+        # test-compat is skipped on pull_request (result 'skipped'), which does
+        # not trip this gate — only real failures/cancellations do.
         if: ${{ always() && (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')) }}

.gitignore

@@ -120,3 +120,8 @@ ecosystem-ci/examples
 pnpm-lock.yaml
 .utoo.toml
 .claude/
+
+# benchmark output directories
+benchmark/ci-test/baseline*
+benchmark/ci-test/run*
+benchmark/ci-test/2*

AGENTS.md

@@ -28,18 +28,55 @@ Egg is maintained as a utoo monorepo.
 
 ### Local CI
 
-Run tests **without building first**. The CI workflow (`ut install → ut run ci`) never runs `build` before tests. If `dist/` directories exist from a prior build, tegg plugin tests will fail with `duplicate proto` errors because globby scans both `src/*.ts` and `dist/*.js`, loading the same decorated class twice.
+Run tests **without building first**. The CI test jobs run `ut run pretest`
+(clean dist + per-workspace pretest) then a vitest shard; they never `build`
+before tests. If `dist/` directories exist from a prior build, tegg plugin tests
+will fail with `duplicate proto` errors because globby scans both `src/*.ts` and
+`dist/*.js`, loading the same decorated class twice. `scripts/clean-dist.js`
+(run by `ut run clean-dist`) removes every `dist/` for you.
 
-When you see `duplicate proto` failures locally:
+When you see `duplicate proto` failures locally, run `ut run clean-dist` (or the
+equivalent find below) and re-run:
 
 ```bash
 find tegg packages plugins tools -name dist -type d \
   -not -path '*/node_modules/*' -not -path '*/test/*' -not -path '*/fixtures/*' \
   -exec rm -rf {} +
 ```
 
+### Aggregator scripts (avoid `ut run <x> --workspaces` recursion)
+
+The root `typecheck` / `pretest` aggregate per-workspace scripts via
+`node scripts/run-workspaces.js <script>`, **not** `ut run <script> --workspaces`.
+With utoo, `--workspaces` includes the monorepo root, so a root script that calls
+`ut run <same-name> --workspaces` recurses infinitely. `run-workspaces.js`
+enumerates real workspace dirs, runs each script body against the root-hoisted
+`node_modules/.bin` (so root-only CLIs like `tsgo` resolve even where a stale
+local bin shim exists), and runs them concurrently to bound wall time.
+
 Then re-run tests.
 
+### CI test sharding
+
+CI splits the test suite across parallel runners via `scripts/run-shard.js`
+(`node scripts/run-shard.js <shard>`). Each heavy fork-based package
+(`cluster`, `egg`, `mock`, `development`, `schedule-a`/`schedule-b`) runs on its
+own runner with a `--maxWorkers` cap so forked egg cluster child processes do
+not oversubscribe the CPU and time out; `rest-a`/`rest-b` split everything else
+with full parallelism. The goal is to keep each shard's `vitest run` wall time
+under ~60s. To reproduce one shard locally: `node scripts/run-shard.js cluster`.
+Use `node scripts/run-shard.js all` for the unsharded full suite.
+
+Cluster/mock/schedule tests fork real OS processes that bind ports. If a run is
+killed mid-flight, orphaned `start-cluster`/`app_worker`/`agent_worker`
+processes can linger and hold ports (e.g. 17001), causing later runs to fail
+with `EADDRINUSE`/`app.ready()` timeouts. Kill them before re-running:
+
+```bash
+ps aux | grep -E "[s]tart-cluster|[c]luster/src/(app|agent)_worker" \
+  | awk '{print $2}' | xargs kill -9
+```
+
 ## Coding Conventions
 
 - prefer existing repo patterns over inventing new ones

package.json

@@ -13,19 +13,21 @@
   ],
   "type": "module",
   "scripts": {
-    "clean-dist": "ut run clean --workspaces --if-present",
+    "clean-dist": "node scripts/clean-dist.js",
     "build": "tsdown",
     "prelint": "ut run clean-dist",
     "lint": "oxlint --type-aware --type-check --quiet",
     "fmt": "oxfmt",
-    "typecheck": "ut run clean-dist && ut run typecheck --workspaces --if-present",
+    "typecheck": "ut run clean-dist && node scripts/run-workspaces.js typecheck",
     "fmtcheck": "oxfmt --check .",
-    "pretest": "ut run clean-dist && ut run pretest --workspaces --if-present",
+    "pretest": "ut run clean-dist && node scripts/run-workspaces.js pretest",
     "test": "vitest run --bail 1 --retry 2 --testTimeout 20000 --hookTimeout 20000",
     "test:cov": "ut run test -- --coverage",
+    "test:shard": "node scripts/run-shard.js",
     "benchmark:ci-test": "node scripts/ci-test-benchmark.js",
-    "preci": "ut run pretest --workspaces --if-present",
+    "preci": "ut run pretest",
     "ci": "ut run test -- --coverage",
+    "ci:shard": "node scripts/run-shard.js --coverage",
     "dev:services:start": "node scripts/dev-services.js start",
     "dev:services:stop": "node scripts/dev-services.js stop",
     "dev:services:status": "node scripts/dev-services.js status",