fix(virtual-packages): restore hermetic lifecycle convergence

[danielmeppiel]

fix(virtual-packages): restore and prove the locked lifecycle

TL;DR

This PR composes the virtual-package fixes from #2214, #2217, and #2223 into one hermetic lifecycle matrix: install, lock, frozen replay, update, and audit. The RED contract found one bounded product defect: an unchanged apm update skipped materialization after a failed frozen replay left apm_modules/ empty. The fix reuses the canonical lock-to-install-path owner, restores the cache from unchanged refs without prompting, and preserves the prior lock and deployed bytes on failed resolution.

Important

This lane does not edit critical_suite.toml, TM002, or required-CI promotion. Promotion remains a later serialized change.

Problem (WHY)

• The merged virtual-package fixes had focused contracts, but no single test drove a manifestless Claude Skill and a synthetic virtual file through the complete public lifecycle.
• After cache eviction and a failed frozen content-integrity check, unchanged apm update returned at the no-change plan gate. The lock and deployed bytes survived, but the dependency cache stayed empty and the next audit could not converge.
• Source deletion and invalid Skill frontmatter needed negative twins proving failed resolution cannot commit partial durable state.
• [!] Platform newline behavior could silently change synthetic-manifest bytes and hashes unless the lifecycle asserts canonical LF bytes under a CRLF generation domain.

These failures conflict with P6: "Behavior must be predictable, auditable, and explainable in plain English." The no-change repair also preserves P4's UX floor: "No bug fix, hardening, security patch, or refactor lands if it makes those commands harder, slower, more verbose, or more confusing for a new user."

Approach (WHAT)

• Author one local Git repository containing both canonical virtual shapes: a manifestless claude_skill subdirectory and a synthetic single-file instruction.
• Drive every scenario through real Click commands with remote transport redirected to local Git; do not duplicate package identity, install-path, YAML, or hash logic.
• Assert logical identity, version, exact synthetic-manifest bytes, package/deployment hashes, lock provenance, audit checks, convergence, and last-good durable state.
• Extend only the existing apm update no-change decision so an entirely absent locked cache proceeds through normal materialization without a ref-change prompt.
• Document the observable same-ref cache repair in the CLI reference and packaged APM usage guide.

Implementation (HOW)

File	Intent
src/apm_cli/commands/update.py	Detect a wholly unmaterialized locked tree via LockedDependency.to_dependency_ref().get_install_path(), capture that fact before resolution staging, and allow the unchanged plan to rehydrate without consent. No identity or hash authority is added.
tests/integration/test_virtual_package_lifecycle_matrix.py	Add five positive/negative lifecycle rows and assert the full durable and audit contract through public commands.
tests/unit/commands/test_update_command.py	Trap remote and local-only empty-cache repair without prompts, prove one-pass canonical _local materialization, and cover direct plus hashed transitive local install paths.
CHANGELOG.md	Record the user-visible empty locked-cache repair under Unreleased/Fixed.
docs/src/content/docs/reference/cli/update.md	Explain same-ref cache repair, consent, and dry-run behavior.
docs/src/content/docs/consumer/update-and-refresh.mdx	Scope consent to ref changes and link automatic same-ref cache repair.
packages/apm-guide/.apm/skills/apm-usage/commands.md	Keep the packaged command contract synchronized with the CLI.

Diagrams

Legend: the dashed node is the only new product branch; every other node is the composed lifecycle contract.

flowchart LR
    I[Install] --> L[Lock]
    L --> F[Frozen replay]
    F --> U[Update]
    U --> D{Locked package path exists}
    D -->|yes| N[No-op]
    D -->|no| R[Rehydrate unchanged refs]
    N --> A[Audit]
    R --> A
    F -->|integrity failure| G[Preserve last-good lock and deployed bytes]
    U -->|resolution failure| G
    G --> A
    classDef new stroke-dasharray: 5 5;
    class R new;

Loading

Trade-offs

• Repair only a wholly absent locked cache. Chose the bounded lifecycle defect; rejected partial-cache healing because it needs a separate reconciliation design and broader failure semantics.
• Reuse canonical path selection. Chose LockedDependency.to_dependency_ref().get_install_path(); rejected directory-name heuristics and any second virtual-package identity/path authority.
• No prompt for unchanged refs. Chose automatic materialization because dependency intent does not move; retained the existing consent gate for every ref or revision-pin change.
• Keep promotion serialized. Chose one fresh integration lane; rejected edits to critical_suite.toml, TM002, and CI promotion metadata in this parallel PR.

Benefits

1. Five exact scenario rows cover install -> lock -> frozen replay -> update -> audit.
2. Two valid unchanged paths converge with clean config-consistency, content-integrity, and drift audit checks, including CRLF pressure and cache rehydration.
3. Three drift twins detect missing source, invalid metadata, and same-commit virtual-file content changes while preserving prior lock/deployed bytes.
4. One unchanged update now restores both virtual package shapes from existing refs and reports Restored dependency cache without changing refs.
5. Deterministic owner detection reports zero canonical-owner table touches; all identity, path, YAML, and hash assertions route through existing owners.
6. Focused local-only regressions prove one-pass repair without a consent prompt and preserve canonical direct and hashed transitive _local paths.

Validation

uv run --extra dev pytest -q tests/integration/test_virtual_package_lifecycle_matrix.py:

5 passed in 14.54s

uv run --extra dev pytest -q tests/unit/commands/test_update_command.py tests/integration/test_virtual_package_lifecycle_matrix.py tests/integration/test_virtual_skill_audit_e2e.py tests/integration/test_virtual_claude_skill_lock_convergence.py tests/unit/deps/test_github_downloader_single_file_sha.py tests/unit/integration/test_mcp_config_view.py:

93 passed in 22.76s

Full local gate evidence

uv run --extra dev pytest -q tests/unit tests/test_console.py tests/red_team:

19158 passed, 2 skipped, 21 xfailed, 19 warnings, 86 subtests passed in 146.04s

Quality and authority contracts:

123 passed in 23.21s
[+] assertion-quality ratchet clean: AQ001=5, AQ002=12
[+] exact test duplicate ratchet clean: 1052 files, 8 allowed duplicate group(s)
[+] test contract authority check clean

Spec conformance:

[+] orphan_check OK: 100 requirements aligned across anchors / manifest / Appendix C / pytest markers
140 passed, 2 skipped in 11.46s

uv run apm audit --ci:

[+] Replayed 7 package(s)
[+] No drift detected
[*] All 9 check(s) passed

Canonical lint:

All checks passed!
1534 files already formatted
Your code has been rated at 10.00/10
[+] auth-signal lint clean
[+] architecture boundary lint clean

Final exact head:

036f30eaae2f74da039f4e6308dcbc3d5926ab08
7 files changed, 892 insertions(+), 14 deletions(-)
touched_owners: []
[+] mode_b: no critical-path diff; OK

Mutation evidence

Mutated canonical decision	Expected break
Manifestless Skill version unknown -> 1.0.0	Lifecycle identity assertion failed.
Manifestless audit exemption removed	Clean audit assertion failed.
Atomic synthetic-manifest writer replaced with platform Path.write_text	CRLF byte/hash assertion failed.
Pre-resolution cache-rehydration signal disabled	content-drift-pinned-virtual-file failed because restored module bytes were absent.
Canonical install-path delegation replaced with raw lock-key joining	The local-only regression failed after materialization because _local/local-pkg was still misclassified as absent.

All mutations were restored before commit.

Scenario Evidence

#	Scenario (user promise)	Principle(s)	Test(s) proving it	Type
1	An unchanged virtual Skill and instruction stay byte-identical and audit-clean through the full lifecycle.	Governed by policy, DevX	tests/integration/test_virtual_package_lifecycle_matrix.py::test_virtual_package_lifecycle_matrix[unchanged-on-disk-cache]	integration
2	An unchanged locked project restores an evicted cache under CRLF pressure without changing logical identity, lock bytes, or deployed bytes.	Portability by manifest, Governed by policy, DevX	tests/integration/test_virtual_package_lifecycle_matrix.py::test_virtual_package_lifecycle_matrix[unchanged-rehydrated-cache]	integration
3	Deleting the manifestless Skill source fails update and preserves the last-good lock and deployment.	Secure by default, Governed by policy	tests/integration/test_virtual_package_lifecycle_matrix.py::test_virtual_package_lifecycle_matrix[source-drift-missing-skill] (regression-trap for #2207)	integration
4	Invalid Skill frontmatter fails update and preserves the last-good lock and deployment.	Secure by default, Governed by policy	tests/integration/test_virtual_package_lifecycle_matrix.py::test_virtual_package_lifecycle_matrix[metadata-drift-invalid-frontmatter] (regression-trap for #2206)	integration
5	Same-commit virtual-file tampering fails frozen replay; restoring valid source lets update rebuild the cache and audit cleanly.	Secure by default, Governed by policy	tests/integration/test_virtual_package_lifecycle_matrix.py::test_virtual_package_lifecycle_matrix[content-drift-pinned-virtual-file] (regression-trap for this PR)	integration

How to test

• Run the five-row lifecycle matrix and observe all rows pass.
• Run the content-drift-pinned-virtual-file row alone and observe frozen replay reject the hash mismatch before update restores the cache.
• Run the 93-test focused virtual/update suite and observe no regressions in the canonical metadata, audit, or synthetic-manifest owners.
• Run uv run apm audit --ci from the repository root and observe all 9 checks pass.
• Inspect the diff and confirm critical_suite.toml, TM002, and CI promotion files are untouched.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

This PR fixes a lifecycle convergence gap for virtual/manifestless dependencies by teaching apm update to re-materialize an empty apm_modules/ cache even when the resolved refs are unchanged, and it adds a hermetic end-to-end lifecycle matrix to prove install/lock/frozen/update/audit behavior (including CRLF pressure and failure preservation).

Changes:

• Extend the apm update no-change plan path to proceed when the lock expects dependencies but the module cache is wholly unmaterialized (rehydrate without prompting or ref changes).
• Add unit coverage for the new cache-rehydration predicate and the “no prompt, proceed” behavior.
• Add a new hermetic integration lifecycle matrix test for virtual packages (manifestless Claude skill + synthetic virtual file), and document the observable behavior in CLI docs + packaged guide.

Show a summary per file

File	Description
src/apm_cli/commands/update.py	Adds _module_cache_needs_rehydration and threads a “rehydrate even if plan is unchanged” decision through the plan callback + final reporting.
tests/unit/commands/test_update_command.py	Adds focused unit tests for the new rehydration behavior and predicate edge cases (including ignoring resolution staging).
tests/integration/test_virtual_package_lifecycle_matrix.py	New hermetic integration test matrix covering virtual/manifestless lifecycle, drift twins, and newline-domain determinism.
docs/src/content/docs/reference/cli/update.md	Updates CLI reference to explain same-ref cache repair and consent semantics (including --dry-run remaining read-only).
packages/apm-guide/.apm/skills/apm-usage/commands.md	Keeps the packaged usage guide in sync with the updated apm update behavior/consent model.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Low

[danielmeppiel]

APM Review Panel: ship_with_followups

Hermetic lifecycle convergence restores a missing cache without expanding consent or package identity authority.

cc @sergio-sisternes-epam -- a fresh advisory pass is ready for your review.

All active panelists converge on the same conclusion: this PR introduces no new authority, no consent expansion, and no security regression. The bounded cache-repair path runs only after the plan confirms zero ref movement. The focused unit and integration suites prove the user-facing contract, and four mutation probes confirm the regression traps are load-bearing.

The two doc-writer findings are legitimate truth drift to fold now: the dry-run sentence overpromises no-change plan output, and the consumer guide's blanket consent claim conflicts with automatic same-ref restoration. They are surgical, in-scope corrections. The eager-tuple and integration-message findings are low-cost polish. Quickstart discoverability is later release work, not part of this bounded lane.

Dissent. Python Architect and Performance Expert independently flagged the same eager tuple construction as a nit; there is no severity disagreement.

Aligned with: Portable by manifest: stable virtual package bytes and identity under normal and CRLF generation domains. Secure by default: content, source, and metadata drift fail closed while prior durable state survives. Governed by policy: automatic repair runs only after the update plan confirms zero ref movement. Pragmatic as npm: a missing locked cache restores without a redundant prompt, while dry-run remains read-only.

Growth signal. Release-note angle: apm update self-heals a missing dependency cache from unchanged locked refs.

Panel summary

Persona	B	R	N	Takeaway
Python Architect	0	0	1	Bounded fix routes through canonical owners; no new authority introduced.
CLI Logging Expert	0	0	0	Schema failure after two retries; no validated finding included.
DevX UX Expert	0	0	0	Same-ref repair preserves consent semantics, dry-run behavior, and package-manager UX.
Supply Chain Security Expert	0	0	1	Repair is consent-safe and adds no identity or hash authority.
OSS Growth Hacker	0	1	1	Cache restoration is a retention signal; broader funnel messaging is later scope.
Doc Writer	0	2	0	Two documentation truth corrections should fold now.
Test Coverage Expert	0	0	1	Five integration rows and a unit regression trap prove the behavior.
Performance Expert	0	0	1	The one-time check is bounded; a lazy loop is a small cleanup.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 4 follow-ups

1. [Doc Writer] Correct dry-run wording in the CLI update reference -- the current sentence promises no-change plan output that the CLI does not render.
2. [Doc Writer] Qualify the consumer guide's blanket consent claim -- same-ref cache restoration now proceeds without a prompt, while ref changes still require consent.
3. [Test Coverage Expert] Assert the restoration message in the integration row -- the unit test already proves the string; this binds it to the full lifecycle.
4. [Performance Expert] Use a lazy single-pass cache-path check -- this avoids canonical path construction after the first warm-cache hit.

Architecture

classDiagram
    direction LR
    class LockFile {
      +dependencies dict
    }
    class LockedDependency {
      +to_dependency_ref() DependencyReference
    }
    class DependencyReference {
      +get_install_path(modules_dir) Path
    }
    class UpdateCommand {
      +_module_cache_needs_rehydration(lockfile, modules_dir) bool
      +_plan_callback(plan) bool
    }
    LockFile *-- LockedDependency : contains
    LockedDependency ..> DependencyReference : converts
    UpdateCommand ..> LockFile : reads
    UpdateCommand ..> DependencyReference : canonical install path

Loading

flowchart TD
    A["apm update"] --> B["Read lockfile"]
    B --> C["Check canonical locked paths"]
    C --> D{"Any path materialized"}
    D -->|yes| E["Normal no-change exit"]
    D -->|no| F["Rehydrate unchanged refs"]
    F --> G["Materialize locked tree"]
    G --> H["Audit convergence"]

Loading

Recommendation

Fold the two documentation truth fixes before freezing the branch. The message assertion and lazy-loop cleanup are safe, bounded polish; quickstart expansion should remain separate.

---

Full per-persona findings

Python Architect

• [nit] Eager tuple materialization in cache detection at src/apm_cli/commands/update.py:120
  The tuple builds every canonical install path before any() can short-circuit. The practical cost is negligible for typical lockfiles, but a single-pass loop is clearer.
  Suggested: Return false on the first existing canonical path and otherwise return whether any dependency was seen.

CLI Logging Expert

No validated findings. The final retry returned a legacy schema.

DevX UX Expert

No findings.

Supply Chain Security Expert

• [nit] Consider a verbose pre-materialization cache-repair diagnostic at src/apm_cli/commands/update.py:631
  A verbose-only line could make intentional cache repair easier to distinguish in CI logs. This is not a security gap.

OSS Growth Hacker

• [recommended] Cache-repair behavior is not surfaced in the quickstart funnel at docs/src/content/docs/getting-started/quick-start.md
  This is discoverability work beyond the bounded reference-doc change and should remain a later follow-up.
• [nit] Reference wording uses internal cache terminology at docs/src/content/docs/reference/cli/update.md:104
  "Restores your packages" is easier to scan than "rehydrates the cache."

Auth Expert -- inactive

The diff does not change authentication, tokens, credential resolution, or host fallback.

Doc Writer

• [recommended] Make the dry-run sentence match unchanged-cache output at docs/src/content/docs/reference/cli/update.md:20
  The command emits the generic dry-run notice rather than rendering a no-change plan.
  Suggested: State that --dry-run remains read-only and does not restore files.
• [recommended] Qualify the consumer guide's blanket consent claim at docs/src/content/docs/consumer/update-and-refresh.mdx:27
  Same-ref cache restoration proceeds without consent, so the existing blanket claim conflicts with the updated reference.

Test Coverage Expert

• [nit] The integration row does not assert the cache-restoration success message at tests/integration/test_virtual_package_lifecycle_matrix.py
  The unit test proves the exact string and the integration row proves restored bytes and clean audit. Adding the message assertion tightens the observable contract.
  Proof (passed): tests/unit/commands/test_update_command.py::TestUpdateNoChanges::test_locked_empty_module_cache_rehydrates_without_prompt -- proves: an unchanged locked project restores an evicted cache and reports the repair.

Performance Expert

• [nit] Eager tuple prevents short-circuit on the first existing path at src/apm_cli/commands/update.py:120
  A lazy single-pass loop avoids unnecessary canonical path work after the first warm-cache hit.

_{This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.}