Skip to content

Commit 2b980ae

Browse files
committed
test(bench-orchestrator): replace golden matrix snapshots with targeted assertions
The golden files pinned every field of all nine profiles, so any intentional matrix change meant regenerating nine JSON snapshots -- churn with the standard snapshot failure mode of blind regeneration. Drop them in favor of targeted assertions on the load-bearing fields (selection and engine/format targets), a structural smoke check that every profile resolves to well-formed entries, and a check on the secondary fields CI depends on (remote_key, scale_factor). The develop/nightly behavior-preservation tests are unchanged. Signed-off-by: Connor Tsui <connor@spiraldb.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQg1cgPLKJg4QVJvE2xa3Q
1 parent 521a932 commit 2b980ae

11 files changed

Lines changed: 35 additions & 1427 deletions

File tree

bench-orchestrator/README.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -235,12 +235,13 @@ jobs:
235235
${{ matrix.scale_factor && format('--opt scale-factor={0}', matrix.scale_factor) || '' }}
236236
```
237237
238-
### Golden and parity tests
238+
### Tests
239239
240-
`tests/test_matrix.py` checks each profile against a checked-in golden file
241-
(`tests/golden/<profile>.json`), so any change to what-runs surfaces as a reviewable diff.
242-
Regenerate them with `UPDATE_GOLDEN=1 pytest`. Two further tests assert that the `develop` and
243-
`nightly` profiles reproduce the current workflow matrices exactly, proving the registry is
240+
`tests/test_matrix.py` covers each profile with targeted assertions on its load-bearing fields --
241+
which benchmarks are selected, their engine/format targets, and the secondary fields CI depends on
242+
(`remote_key`, `scale_factor`, `data_formats`) -- plus a structural smoke check that every profile
243+
resolves to a well-formed matrix. The `develop` and `nightly` profiles additionally assert they
244+
reproduce the target sets the old inline workflow matrices used, proving the registry is
244245
behavior-preserving.
245246

246247
## Example Workflows

bench-orchestrator/tests/golden/compression.json

Lines changed: 0 additions & 9 deletions
This file was deleted.

0 commit comments

Comments
 (0)