You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
harden(ci): require hermetic lifecycle smoke subset on pull-request CI (#2247)
* ci: promote hermetic lifecycle smoke subset to required PR checks
Add a new required "Lifecycle Smoke (Linux)" job to ci.yml, wired into
merge-gate.yml's EXPECTED_CHECKS, running the smallest stable hermetic
subset of the real Consume/Produce/Govern lifecycle contracts at PR
time -- a gap since tests/integration/ previously only ran on
merge_group.
Targets (12 tests, ~13s bare pytest, ~60s projected job wall time,
timeout-minutes: 3):
- test_install_content_hash_roundtrip.py (component, critical_suite.toml)
- test_policy_pinned_constraint_e2e.py (component, critical_suite.toml)
- test_virtual_claude_skill_lock_convergence.py (component, critical_suite.toml)
- test_virtual_package_lifecycle_matrix.py (direct #2240 regression)
- test_architecture_authorities.py::test_ado_lock_coordinates_have_single_owner
(direct #2226 AC14 static guard)
All five are hermetic: IsolatedApmEnvironment denies all AF_INET/
AF_INET6 socket use (incl. loopback), no requires_apm_binary/
requires_github_token/requires_ado_pat markers, no secrets wired into
the new job.
tests/quality/test_ci_topology.py gains a positive guard (job name,
timeout ceiling, exact target tuple derived dynamically from
critical_suite.toml's `component`-marked entries plus the two direct
regression targets, no forbidden credential env, EXPECTED_CHECKS
membership plus four mutation/negative tests proving the guard
actually catches drift (dropped target, swapped AC14 node-id,
injected credential env, removed from EXPECTED_CHECKS).
#2238 is intentionally not duplicated here: its regression coverage is
unit-level and already required via the existing shards.
Updates docs/src/content/docs/contributing/integration-testing.md and
CHANGELOG.md. No src/apm_cli/** changes.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 75891045-fa50-40b7-9354-b4649195f724
EOF
)
* docs(changelog): reference PR #2247 for lifecycle smoke entry
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 75891045-fa50-40b7-9354-b4649195f724
* fold: apm-review-panel + apm-strategy recommendations for #2247
apm-strategy (apm-ceo review): tighten the CHANGELOG entry to a single
contributor-facing line per changelog.instructions.md's one-line-per-PR
convention, and fix a stray non-ASCII em-dash.
apm-review-panel (9-persona fan-out, CEO synthesis: ship_with_followups,
no blocking findings): fold all 5 curated follow-ups.
1. [test-coverage-expert / python-architect / supply-chain-security-expert,
corroborated + empirically proven] Close the hermetic guard's blind
spots in tests/quality/test_ci_topology.py:
- job-level env: was never checked (GitHub Actions inherits it to
every step); add the check + test_lifecycle_smoke_job_level_credential_env_fails.
- step with: blocks could smuggle a credential expression past the
env/run-only scan; add the check + test_lifecycle_smoke_with_block_credential_fails.
- the github.token expression alias resolves to the same automatic
token but doesn't match any FORBIDDEN_CREDENTIAL_ENV substring; add
FORBIDDEN_CREDENTIAL_EXPRESSIONS + test_lifecycle_smoke_github_token_expression_alias_fails.
Full suite: 18/18 passing (was 15/15).
2. [devx-ux-expert / doc-writer] Add the exact copy-paste local-repro
command for the 5-target lifecycle-smoke subset to
integration-testing.md, so a contributor whose PR fails the new
required check has a documented one-liner.
3. [oss-growth-hacker] Update CONTRIBUTING.md's contributor-facing CI-tier
summary (the only place a first-time external contributor learns what
checks to expect) to mention the new Lifecycle Smoke check, framed as
PR-time feedback rather than an unexplained new gate.
4. [doc-writer] Add symmetric "Lifecycle Smoke Verifies" / "Lifecycle
Smoke Failures" entries to integration-testing.md (Tiers 1/2 already
had both), plus a cross-reference to test_ci_topology.py as the
drift guard, and fix the file-list's inconsistent tests/integration/
prefix.
All 9 panelists returned no blocking-severity findings; CEO stance is
ship_with_followups, all curated follow-ups folded in this commit.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 75891045-fa50-40b7-9354-b4649195f724
* refactor(ci): select lifecycle smoke via registered lifecycle_smoke marker
Supersedes explicit file/node-id targeting with a declarative
@pytest.mark.lifecycle_smoke marker so an emptied family fails loudly
(pytest exit code 5, 'no tests collected') instead of a path list
silently shrinking.
- Register lifecycle_smoke in pyproject.toml's pytest markers list.
- Mark the four lifecycle modules at module level
(test_install_content_hash_roundtrip.py,
test_policy_pinned_constraint_e2e.py,
test_virtual_claude_skill_lock_convergence.py,
test_virtual_package_lifecycle_matrix.py) and, at function level,
only test_architecture_authorities.py::test_ado_lock_coordinates_have_single_owner
(the file's other 32 tests remain unmarked).
- ci.yml's lifecycle-smoke job now runs
`pytest --strict-markers -m lifecycle_smoke tests/integration`;
rewrote the job's rationale comment to match.
- tests/quality/test_ci_topology.py: replaced the exact-target-list
assertion and its two target-based mutation tests with semantic
contracts -- marker registered (read from pyproject.toml's existing
markers list, no new manifest constant), command uses
--strict-markers/-m lifecycle_smoke/bounded tests/integration root,
job stays required and hermetic, and the marker family collects
non-empty right now -- backed by six mutation tests proving each
contract is load-bearing (dropped -m expression, wrong marker
string, unbounded root, dropped --strict-markers, unregistered
marker, empty family).
- Docs: integration-testing.md's Tier 3 section and marker-axis table,
CONTRIBUTING.md, and CHANGELOG.md updated to describe marker-based
selection and the measured ~65-70s job wall time (was ~60s; the
full tests/integration/ directory scan for marker deselection adds
~2-8s of collection overhead vs the prior five explicit targets).
Verified: marker-based collection selects the identical 12 tests as
the prior explicit-target list (parity confirmed via --collect-only
diff). Re-ran both failure-injection proofs (pre-#2226 lockfile.py,
pre-#2240 update.py) against the new marker-based command -- both
still fail exactly as before. Confirmed marked tests are unaffected
in ci-integration.yml's merge_group integration job (no -m filter
there). Full lint mirror (ruff check/format, pylint R0801,
lint-auth-signals.sh, lint-architecture-boundaries.sh, actionlint)
clean. tests/quality/test_ci_topology.py + test_test_taxonomy.py +
test_architecture_authorities.py: 67 passed.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 75891045-fa50-40b7-9354-b4649195f724
* feat(ci): promote prune-hook reconciliation [claude] to lifecycle_smoke
Adds pytest.mark.lifecycle_smoke to only the `claude` parametrization
of test_prune_removes_merged_hook_entries_and_sidecar (deliberately
excluding the `cursor` sibling), now that #2249's prune-hook fix has
merged. The existing marker-driven `-m lifecycle_smoke tests/integration`
command in ci.yml auto-picks up the newly-marked test with zero workflow
edit -- confirmed via `git diff` showing no change to the lifecycle-smoke
job section.
Family grows from 12 to 13 tests, all passing (13 passed, ~16.5-17.9s
local across 3 repeated runs -- comparable to the prior 12-test baseline
of ~17.7-18.4s). Failure-injection re-proof: reverting #2249's production
fix (prune.py, hook_integrator.py, uninstall/engine.py, apm_package.py)
to pre-fix content causes exactly `[claude]` to fail under `-m
lifecycle_smoke` selection (12 passed, 1 failed), confirming the gate is
load-bearing for this regression; restored cleanly afterward (working
tree verified clean pre/post).
Also merges origin/main (additive merge commit, no rebase/force-push)
to bring in #2249 plus intervening #2237 (Windows CI fix) and #2246
(lockfile-property tests). Resolved one real conflict in
merge-gate.yml's EXPECTED_CHECKS where main's #2237 follow-on
independently added a new required "Windows Compatibility Gate" check
to the same ternary our branch extended with "Lifecycle Smoke (Linux)"
-- both required-check names now coexist in both event branches.
Docs (integration-testing.md, CHANGELOG.md) updated to enumerate the
6th target and prune's Consume/Produce/Govern coverage. No change
needed to test_ci_topology.py's semantic assertions (count-agnostic
by design) or ci.yml (marker-driven auto-pickup, zero edit).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 75891045-fa50-40b7-9354-b4649195f724
---------
Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
-**Location**: selected declaratively via the registered `lifecycle_smoke` pytest marker, applied to all of `tests/integration/test_install_content_hash_roundtrip.py`, `test_policy_pinned_constraint_e2e.py`, `test_virtual_claude_skill_lock_convergence.py`, `test_virtual_package_lifecycle_matrix.py` (module-level), to only `test_architecture_authorities.py::test_ado_lock_coordinates_have_single_owner` (function-level -- the file's other tests are not part of this family), and to only the `[claude]` parametrization of `test_prune_hook_reconciliation_e2e.py::test_prune_removes_merged_hook_entries_and_sidecar` (the sibling `[cursor]` parametrization is deliberately excluded)
38
+
-**Purpose**: Promote the smallest stable, hermetic slice of the real Consume/Produce/Govern lifecycle contracts onto the PR-time critical path, so a regression in install/lock/audit convergence, ADO lock-coordinate handling, or prune-time hook reconciliation fails the PR instead of only surfacing once a change reaches the merge queue
39
+
-**Scope**: install content-hash roundtrip, policy pinned-constraint enforcement, virtual-skill lock convergence, the virtual/manifestless lifecycle matrix, the ADO lock-coordinate ownership guard, and prune's merged-hook/sidecar reconciliation -- no built binary, no network, no credentials
40
+
-**Duration**: ~65-70s job wall time (hard 3-minute timeout)
41
+
-**Trigger**: every pull request and merge queue run (`ci.yml`'s `lifecycle-smoke` job, required via `merge-gate.yml`)
42
+
-**Selection mechanism**: `pytest --strict-markers -m lifecycle_smoke tests/integration` -- declarative, not an explicit file/node-id list. If every `@pytest.mark.lifecycle_smoke` mark were ever removed, the marker family collects zero tests and pytest exits code 5 ("no tests collected"), failing the job loudly rather than silently shrinking. The marker is registered in `pyproject.toml`'s `[tool.pytest.ini_options]` markers list; `--strict-markers` rejects any typo'd or unregistered mark used as a decorator.
43
+
-**Drift guard**: `tests/quality/test_ci_topology.py` pins the marker's registration, the command's use of `--strict-markers`/`-m lifecycle_smoke`/the bounded `tests/integration` root, hermeticity (no credentials at job- or step-level `env`, no credential expressions in `run:`/`with:`), required-check membership, and that the marker family collects a non-empty set of tests right now -- editing the job without updating that guard fails CI
44
+
-**Run it locally** (the exact command CI runs):
45
+
```bash
46
+
uv run --extra dev pytest -p no:cacheprovider -q --strict-markers \
47
+
-m lifecycle_smoke tests/integration
48
+
```
49
+
36
50
## Running Tests Locally
37
51
38
52
Integration tests live under `tests/integration/` and run via `pytest`
@@ -73,9 +87,14 @@ Pytest markers compose across independent axes:
73
87
| Behavioral | What boundary does the test cross? |`unit`, `component`, `e2e`|
74
88
| Scheduling | When is the test selected? |`integration`, `slow`, `benchmark`, `live`|
75
89
| Prerequisite | What environment must exist? |`requires_*`|
90
+
| CI-selection | Is this test part of a named required CI gate? |`lifecycle_smoke`|
76
91
77
92
`live` is both an opt-in scheduling marker and an external-service
78
93
prerequisite. Behavioral markers do not replace prerequisite markers.
94
+
`lifecycle_smoke` is orthogonal to all three: it does not describe a
95
+
test's boundary, scheduling, or precondition, only that
96
+
`ci.yml`'s required `lifecycle-smoke` job selects it via
97
+
`-m lifecycle_smoke` (see Tier 3 above for the full rationale).
79
98
80
99
The behavioral definitions are:
81
100
@@ -248,7 +267,7 @@ environment end-to-end; for local iteration prefer the direct
248
267
### GitHub Actions Workflow
249
268
250
269
**On PR and merge queue:**
251
-
1. PR-time unit checks run first; merge queue adds Linux smoke, integration, and release-validation gates.
270
+
1. PR-time unit checks and the hermetic Lifecycle Smoke gate run first; merge queue adds Linux smoke, integration, and release-validation gates.
252
271
253
272
**On version tag releases:**
254
273
1. Unit tests + Smoke tests
@@ -325,6 +344,15 @@ Promotion integration tests run on:
- The virtual/manifestless lifecycle matrix: install, lock, frozen-install, update, and audit stay consistent (the direct #2240 regression)
352
+
- The ADO lock-coordinate single-owner guard (the direct #2226 regression)
353
+
- Prune's merged-hook and ownership-sidecar reconciliation for the `claude` target (the direct #2249 regression -- an orphaned package's merged hook entries and sidecar markers must be cleaned up, not left pointing at deleted scripts)
354
+
- No network, no credentials, no built binary required for any of the above
355
+
328
356
## Benefits
329
357
330
358
### **Speed vs Confidence Balance**
@@ -361,6 +389,11 @@ Promotion integration tests run on:
361
389
- Check GitHub Models API availability
362
390
- Review actual vs expected output
363
391
- Test locally with same environment
392
+
393
+
### Lifecycle Smoke Failures
394
+
- These tests are hermetic -- no credentials, no built binary, no network (a real socket attempt raises `OSError`, it does not hang or retry). A failure is a genuine regression, not an environment issue.
395
+
- Run the exact CI command from the "Run it locally" block under Tier 3 above to reproduce.
396
+
- If the failure is about the CI job's shape (marker not registered, wrong `-m`/`--strict-markers` invocation, unbounded root, timeout, empty marker family, or required-check wiring) rather than test logic, check `tests/quality/test_ci_topology.py` -- that guard pins the job's contract and its own failure message will point at what drifted.
364
397
- For hanging issues: Check command transformation in script runner (codex expects prompt content, not file paths)
"requires_inference: requires APM_RUN_INFERENCE_TESTS=1 + model API access",
217
217
"req: binds a test to one or more OpenAPM req-XXX ids (spec-conformance suite)",
218
+
"lifecycle_smoke: promoted to the required PR-time Lifecycle Smoke (Linux) job -- see ci.yml's lifecycle-smoke job and docs/src/content/docs/contributing/integration-testing.md",
0 commit comments