perf(ci): shard the CDK jest suite across a job matrix (#363 item #2)

[scottschreckengaust]

Summary

Implement item #2 of the build-performance umbrella (#363): shard the CDK Jest suite across a CI job matrix so the now-dominant //cdk:test (~298s after #357/#359) runs in parallel slices and aggregates into the single required build check.

Design is documented in docs/design/CI_BUILD_PERFORMANCE.md (PR #364, §"Item #2 — sharding the CDK suite").

Parent: #363. Predecessor: #357 (done, PR #359 merged).

Approach (fan-out + aggregate gate)

build-shard (matrix: shard ∈ [1..4])   # parallel; NOT individually required
        │
        ▼
build (needs: build-shard)             # single required context on merge_group

• build-shard runs jest --shard=${{ matrix.shard }}/4 for the CDK suite (~119 test files) + uploads per-shard coverage.
• A single build job needs: [build-shard], fails if any shard failed, and remains the one required status context (stable name regardless of shard count).
• Non-test tasks (//cdk:synth:quiet, //cli:*, //docs:build, //agent:quality, drift checks) and the deploy artifact (cdk-agentcore-out) produced exactly once, off the shard critical path.
• Cross-shard coverage merge before threshold enforcement (each shard sees only its slice).
• Self-mutation/drift check runs once (aggregate job), not per-shard.

Acceptance criteria

• CDK Jest runs as a 4-way shard matrix; all 2041 tests execute exactly once across shards (sum of per-shard counts = 2041).
• The required build check remains a single stable context that reports on merge_group (feat(ci): require secrets/deps/SAST on every PR — make the merge gate enforceable (incident #313 class) #327) and fails if any shard fails.
• Coverage thresholds (branches 82 / funcs 94 / lines 91 / stmts 91) enforced once on the merged report; not per-shard.
• Deploy artifact cdk-agentcore-out still produced exactly once; deploy.yml unaffected.
• Self-mutation / "Fail build on mutation" check still runs (once).
• Real 4-core CI before/after posted: target slowest-shard + aggregate wall-clock ≈ ~170s (vs ~298s single), build step toward ~5–6 min.
• No new flakiness; mise run build semantics preserved for local (non-CI) runs.

Implementer notes (from the design doc)

• Overhead dominates past ~4 shards (~95s fixed per-job checkout/install/cache). 4-way is the start; measure before going higher; maximize cache hit-rate first.
• --shard partitions by file count, not runtime — watch for a heavy suite skewing one shard.
• Do not mark individual shard jobs required (context-name fragility / queue deadlock risk).

References

• Parent umbrella: perf(ci): remaining build-performance levers (shard, coverage-gate, runner, path-filter) — follow-ups to #357 #363
• Design: docs/design/CI_BUILD_PERFORMANCE.md (PR docs(design): CI build performance roadmap + implementer notes (#363) #364)
• Required-check-on-merge_group rationale: feat(ci): require secrets/deps/SAST on every PR — make the merge gate enforceable (incident #313 class) #327, comment block atop .github/workflows/build.yml
• Predecessor: perf(ci): replace ts-jest with @swc/jest to cut //cdk:test (~91% of build) — experiment #1 #357 / PR perf(test): skip redundant jest type-check via isolatedModules (#357) #359

---

🤖 Generated with Claude Code

[scottschreckengaust]

Profiling result → pivot: optimize heavy suites first, defer sharding

Before writing any workflow YAML, I proved the shard mechanism locally and profiled per-suite timing. The data says sharding is the wrong first lever here.

Why sharding underperforms on this suite

jest --shard partitions by file, and this suite's cost is pathologically concentrated in a few mega-suites — so no shard split can beat the single heaviest file.

Per-suite timing (8-core local, coverage on), top of 119 suites:

Suite	Time
test/stacks/github-tags.test.ts	135.6s
test/constructs/task-api.test.ts	83.5s
test/stacks/agent.test.ts	50.2s
test/constructs/task-orchestrator.test.ts	49.9s
test/bootstrap/synth-coverage.test.ts	44.4s
(remaining 114 suites)	~74s total

Top 5 = 362s of 437s suite-time (83%).

Measured 4-way shard (cold cache, coverage), wall per shard: 6s / 38s / 36s / 133s. The slowest shard (133s) is gated by github-tags.test.ts, vs ~155s unsharded on the same box — only ~14% faster, for 4× the runner-minutes plus the aggregate-gate / coverage-merge complexity. Sharding cannot break up a 135s single file.

The real win: kill repeated full-stack synth in the heavy suites

github-tags.test.ts calls synthWithTags() — a full new App() + stack synth + Template.fromStack — ~5 times (once per test), each synth ~27s. The same anti-pattern (synth-per-test instead of synth-once-in-beforeAll) is the likely cause in task-api, agent, task-orchestrator, and synth-coverage.

Fixing synth-reuse in the top ~4-5 suites is plausibly a larger cut to //cdk:test than sharding, with zero CI complexity — no matrix, no aggregate gate, no coverage merge, no extra runner-minutes. It also lowers the floor so that IF we shard later, sharding actually pays off.

Decision

• Pivot: optimize the heavy suites' synth-reuse first (new sibling issue under perf(ci): remaining build-performance levers (shard, coverage-gate, runner, path-filter) — follow-ups to #357 #363).
• Defer: sharding (this issue, perf(ci): shard the CDK jest suite across a job matrix (#363 item #2) #365) until the floor is lowered — revisit only if wall-clock is still the constraint afterward.

This issue (#365) stays open but blocked on / superseded-by the synth-reuse work; will re-evaluate post-optimization. Design notes preserved in docs/design/CI_BUILD_PERFORMANCE.md (PR #364).

cc #363 (umbrella).