Skip to content

Commit d4a649a

Browse files
committed
docs(designs): place subprocess runner under tests/regression
Move PR 2's subprocess-driven tier from tests/integration/golden_templates/ to tests/regression/golden_templates/. Aligns with the existing tests/regression/deploy/ and tests/regression/package/ suites — same shape (real `sam` subprocess, real AWS credentials, daily-only). Bonus: integration-tests.yml's existing other-and-e2e matrix entry already collects tests/regression/, so no new matrix entry or workflow edit is needed for the regression tier to start running. Renamed "integration tier" → "regression tier" throughout for clarity.
1 parent 3386a5b commit d4a649a

1 file changed

Lines changed: 36 additions & 18 deletions

File tree

designs/golden_templates_and_semver_check.md

Lines changed: 36 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,15 @@ requires a major-version bump whenever a pin changes.
2828
What will be changed?
2929
---------------------
3030

31-
Add a new `tests/golden/` corpus and harness, a new daily integration tier
32-
under `tests/integration/golden_templates/`, and a tiny standalone GitHub
33-
workflow that enforces the semver rule. The corpus covers both LE and
34-
non-LE templates across four axes: SAM resources, packageable CFN resources,
35-
LE intrinsics, and cross-cutting features.
31+
Add a new `tests/golden/` corpus and harness (per-PR unit tier), a daily
32+
subprocess-driven tier under `tests/regression/golden_templates/`, and a
33+
tiny standalone GitHub workflow that enforces the semver rule. The corpus
34+
covers both LE and non-LE templates across four axes: SAM resources,
35+
packageable CFN resources, LE intrinsics, and cross-cutting features.
36+
37+
The two tiers share `tests/golden/templates/` as their single corpus
38+
source — the regression tier reads case directories straight from the
39+
unit tier so authoring a new case lands in both places at once.
3640

3741
Success criteria for the change
3842
-------------------------------
@@ -53,7 +57,7 @@ Success criteria for the change
5357
Out-of-Scope
5458
------------
5559

56-
- No deploy step. The integration tier stops at packaged-template output
60+
- No deploy step. The regression tier stops at packaged-template output
5761
— no CloudFormation, no real Lambdas.
5862
- No CFN compatibility check (e.g. `cfn-lint`, `ValidateTemplate`) at this
5963
stage. The corpus pins what *SAM-CLI* produces; CFN-side parity is a
@@ -139,14 +143,19 @@ tests/golden/ # NEW
139143
├── language_extensions/<case>/... # LE axis
140144
└── cross_cutting/<case>/... # nested stacks, AWS::Include, etc.
141145
142-
tests/integration/golden_templates/ # NEW
146+
tests/regression/golden_templates/ # NEW
143147
├── test_golden_subprocess.py # imports tests/golden/templates as data
144148
└── structural_compare.py # looser comparator for real-CLI output
145149
```
146150

147151
The `tests/golden/templates/` directory is the single source of truth for
148-
the corpus. Both the unit tier (`tests/golden/`) and the integration tier
149-
(`tests/integration/golden_templates/`) read from it; nothing is duplicated.
152+
the corpus. Both the unit tier (`tests/golden/`) and the regression tier
153+
(`tests/regression/golden_templates/`) read from it; nothing is duplicated.
154+
155+
The regression tier sits alongside the existing `tests/regression/deploy/`
156+
and `tests/regression/package/` suites — they're structurally similar:
157+
real `sam` subprocess invocation, real AWS credentials via the
158+
integration-tests workflow's OIDC role, daily-only execution.
150159

151160
### Per-case directory layout
152161

@@ -197,7 +206,7 @@ no `boto3` stubbing.
197206
- Single trailing newline.
198207

199208
The harness invokes the same high-level functions the CLI invokes; it does
200-
not re-implement the pipeline. The integration tier verifies that contract
209+
not re-implement the pipeline. The regression tier verifies that contract
201210
holds.
202211

203212
### Pytest collection
@@ -311,10 +320,17 @@ estimated <30s for ~50 cases. The existing Windows `TEMP=D:\Temp` shim
311320
already handles temp-dir contention. No new workflow file (the semver gate
312321
above is its own workflow, but the corpus tests ride on `build.yml`).
313322

314-
### Integration tier
323+
### Regression tier
324+
325+
Daily run on the existing `integration-tests.yml` workflow. The
326+
workflow's `other-and-e2e` matrix entry already collects
327+
`tests/regression/`, so adding `tests/regression/golden_templates/` is
328+
picked up automatically — **no new matrix entry, no workflow edit**. The
329+
suite sits alongside the existing `tests/regression/deploy/` and
330+
`tests/regression/package/` directories, which have the same shape (real
331+
`sam` subprocess + real AWS credentials, daily-only).
315332

316-
Daily run on the existing `integration-tests.yml` matrix (one new entry:
317-
`golden-templates`). Per case:
333+
Per case:
318334

319335
1. Copy case dir to a temp working dir; copy `src/`.
320336
2. `sam build [--language-extensions]` subprocess; assert exit 0; read
@@ -331,7 +347,7 @@ Daily run on the existing `integration-tests.yml` matrix (one new entry:
331347
dict-vs-string differences (the #9029 bug class).
332348

333349
The comparator lives in
334-
`tests/integration/golden_templates/structural_compare.py` and is reused
350+
`tests/regression/golden_templates/structural_compare.py` and is reused
335351
for both build and package outputs.
336352

337353
#### Why structural, not byte-exact
@@ -343,7 +359,7 @@ output) or constant churn from environmental noise. Structural comparison
343359
answers "did the user-visible output shape change?" — which is what catches
344360
the four bugs we just fixed.
345361

346-
If the unit tier passes but the integration tier fails for a case, the
362+
If the unit tier passes but the regression tier fails for a case, the
347363
in-process harness has drifted from the CLI surface — investigation, not
348364
a re-pin.
349365

@@ -353,7 +369,9 @@ Each PR is independently reviewable; LOC estimates are upper-bound net diff.
353369

354370
- **PR 0** (this design doc).
355371
- **PR 1** — Harness skeleton, three sentinel cases, semver gate workflow.
356-
- **PR 2** — Integration runner.
372+
- **PR 2** — Regression-tier subprocess runner under
373+
`tests/regression/golden_templates/`. Picked up by the existing
374+
`other-and-e2e` matrix entry — no workflow edit required.
357375
- **PR 3** — SAM resources axis: Function variants (~10 cases).
358376
- **PR 4** — SAM resources axis: Api / HttpApi / StateMachine / others
359377
(~10 cases).
@@ -386,7 +404,7 @@ Security Considerations
386404
-----------------------
387405

388406
The harness intentionally invokes only in-process SAM-CLI code; no
389-
network calls, no credentials, no AWS API access. The integration tier
407+
network calls, no credentials, no AWS API access. The regression tier
390408
uses the existing integration-tests workflow's IAM role and S3 bucket;
391409
no new permissions required.
392410

@@ -405,7 +423,7 @@ Subprocess-only harness (no in-process)
405423
Every golden runs the real `sam` CLI as a subprocess. Rejected: process
406424
startup × N templates is too slow for a per-PR gate; flakier on Windows;
407425
deterministic package output without `moto` is hard. Subprocess invocation
408-
is kept, but only at the daily integration tier where speed and
426+
is kept, but only at the daily regression tier where speed and
409427
determinism matter less.
410428

411429
Snapshot plugin (`syrupy`, `pytest-snapshot`)

0 commit comments

Comments
 (0)