@@ -28,11 +28,15 @@ requires a major-version bump whenever a pin changes.
2828What will be changed?
2929---------------------
3030
31- Add a new ` tests/golden/ ` corpus and harness, a new daily integration tier
32- under ` tests/integration/golden_templates/ ` , and a tiny standalone GitHub
33- workflow that enforces the semver rule. The corpus covers both LE and
34- non-LE templates across four axes: SAM resources, packageable CFN resources,
35- LE intrinsics, and cross-cutting features.
31+ Add a new ` tests/golden/ ` corpus and harness (per-PR unit tier), a daily
32+ subprocess-driven tier under ` tests/regression/golden_templates/ ` , and a
33+ tiny standalone GitHub workflow that enforces the semver rule. The corpus
34+ covers both LE and non-LE templates across four axes: SAM resources,
35+ packageable CFN resources, LE intrinsics, and cross-cutting features.
36+
37+ The two tiers share ` tests/golden/templates/ ` as their single corpus
38+ source — the regression tier reads case directories straight from the
39+ unit tier so authoring a new case lands in both places at once.
3640
3741Success criteria for the change
3842-------------------------------
@@ -53,7 +57,7 @@ Success criteria for the change
5357Out-of-Scope
5458------------
5559
56- - No deploy step. The integration tier stops at packaged-template output
60+ - No deploy step. The regression tier stops at packaged-template output
5761 — no CloudFormation, no real Lambdas.
5862- No CFN compatibility check (e.g. ` cfn-lint ` , ` ValidateTemplate ` ) at this
5963 stage. The corpus pins what * SAM-CLI* produces; CFN-side parity is a
@@ -139,14 +143,19 @@ tests/golden/ # NEW
139143 ├── language_extensions/<case>/... # LE axis
140144 └── cross_cutting/<case>/... # nested stacks, AWS::Include, etc.
141145
142- tests/integration /golden_templates/ # NEW
146+ tests/regression /golden_templates/ # NEW
143147├── test_golden_subprocess.py # imports tests/golden/templates as data
144148└── structural_compare.py # looser comparator for real-CLI output
145149```
146150
147151The ` tests/golden/templates/ ` directory is the single source of truth for
148- the corpus. Both the unit tier (` tests/golden/ ` ) and the integration tier
149- (` tests/integration/golden_templates/ ` ) read from it; nothing is duplicated.
152+ the corpus. Both the unit tier (` tests/golden/ ` ) and the regression tier
153+ (` tests/regression/golden_templates/ ` ) read from it; nothing is duplicated.
154+
155+ The regression tier sits alongside the existing ` tests/regression/deploy/ `
156+ and ` tests/regression/package/ ` suites — they're structurally similar:
157+ real ` sam ` subprocess invocation, real AWS credentials via the
158+ integration-tests workflow's OIDC role, daily-only execution.
150159
151160### Per-case directory layout
152161
@@ -197,7 +206,7 @@ no `boto3` stubbing.
197206- Single trailing newline.
198207
199208The harness invokes the same high-level functions the CLI invokes; it does
200- not re-implement the pipeline. The integration tier verifies that contract
209+ not re-implement the pipeline. The regression tier verifies that contract
201210holds.
202211
203212### Pytest collection
@@ -311,10 +320,17 @@ estimated <30s for ~50 cases. The existing Windows `TEMP=D:\Temp` shim
311320already handles temp-dir contention. No new workflow file (the semver gate
312321above is its own workflow, but the corpus tests ride on `build.yml`).
313322
314- # ## Integration tier
323+ # ## Regression tier
324+
325+ Daily run on the existing `integration-tests.yml` workflow. The
326+ workflow's `other-and-e2e` matrix entry already collects
327+ ` tests/regression/` , so adding `tests/regression/golden_templates/` is
328+ picked up automatically — **no new matrix entry, no workflow edit**. The
329+ suite sits alongside the existing `tests/regression/deploy/` and
330+ ` tests/regression/package/` directories, which have the same shape (real
331+ ` sam` subprocess + real AWS credentials, daily-only).
315332
316- Daily run on the existing `integration-tests.yml` matrix (one new entry :
317- `golden-templates`). Per case :
333+ Per case :
318334
3193351. Copy case dir to a temp working dir; copy `src/`.
3203362. `sam build [--language-extensions]` subprocess; assert exit 0; read
@@ -331,7 +347,7 @@ Daily run on the existing `integration-tests.yml` matrix (one new entry:
331347 dict-vs-string differences (the # 9029 bug class).
332348
333349The comparator lives in
334- ` tests/integration /golden_templates/structural_compare.py` and is reused
350+ ` tests/regression /golden_templates/structural_compare.py` and is reused
335351for both build and package outputs.
336352
337353# ### Why structural, not byte-exact
@@ -343,7 +359,7 @@ output) or constant churn from environmental noise. Structural comparison
343359answers "did the user-visible output shape change?" — which is what catches
344360the four bugs we just fixed.
345361
346- If the unit tier passes but the integration tier fails for a case, the
362+ If the unit tier passes but the regression tier fails for a case, the
347363in-process harness has drifted from the CLI surface — investigation, not
348364a re-pin.
349365
@@ -353,7 +369,9 @@ Each PR is independently reviewable; LOC estimates are upper-bound net diff.
353369
354370- **PR 0** (this design doc).
355371- **PR 1** — Harness skeleton, three sentinel cases, semver gate workflow.
356- - **PR 2** — Integration runner.
372+ - **PR 2** — Regression-tier subprocess runner under
373+ ` tests/regression/golden_templates/` . Picked up by the existing
374+ ` other-and-e2e` matrix entry — no workflow edit required.
357375- **PR 3** — SAM resources axis: Function variants (~10 cases).
358376- **PR 4** — SAM resources axis: Api / HttpApi / StateMachine / others
359377 (~10 cases).
@@ -386,7 +404,7 @@ Security Considerations
386404-----------------------
387405
388406The harness intentionally invokes only in-process SAM-CLI code; no
389- network calls, no credentials, no AWS API access. The integration tier
407+ network calls, no credentials, no AWS API access. The regression tier
390408uses the existing integration-tests workflow's IAM role and S3 bucket;
391409no new permissions required.
392410
@@ -405,7 +423,7 @@ Subprocess-only harness (no in-process)
405423Every golden runs the real `sam` CLI as a subprocess. Rejected : process
406424startup × N templates is too slow for a per-PR gate; flakier on Windows;
407425deterministic package output without `moto` is hard. Subprocess invocation
408- is kept, but only at the daily integration tier where speed and
426+ is kept, but only at the daily regression tier where speed and
409427determinism matter less.
410428
411429Snapshot plugin (`syrupy`, `pytest-snapshot`)
0 commit comments