Merge branch 'main' into coreyjadams-python-3.14-support

Author: peterdsharpe

.agents/skills

@@ -0,0 +1 @@
+../skills

.github/CACHE_CONTRACT.md

@@ -37,7 +37,7 @@ from three independent sources: a pinned CUDA container image, a pinned
 | Invalidates when | container image or Python version changes (prefix change → new slot) |
 | Does **not** invalidate on | `uv.lock`, `pyproject.toml`, or kernel source changes (each compiler handles its own source-hash invalidation internally) |
 | Restore semantics | **fail-open**; missing cache only costs compilation time, never correctness |
-| Save semantics | nightly `testmon` job only, via delete-before-save; PR workflows restore but never save |
+| Save semantics | nightly `testmon` job only, via the `replace-cache` action; PR workflows restore but never save |
 
 The JIT compilation cache bundles all JIT compiler artifact directories
 under a single umbrella path.  Each compiler writes to a subdirectory
@@ -57,6 +57,128 @@ old ones.
 To add a new JIT backend: create a subdirectory under `$JIT_CACHE_DIR`,
 set the backend's cache-path env var in the test step, done.
 
+### Testmon database cache (`.testmondata*`)
+
+| Property | Value |
+|---|---|
+| Key | `<TESTMON_CACHE_KEY_PREFIX>-latest` |
+| Prefix encodes | nightly identity (`testmon-nightly`) |
+| Suffix | literal `latest` (mutable slot, refreshed via delete-before-save) |
+| Contents | `.testmondata`, `.testmondata-shm`, `.testmondata-wal` -- testmon's per-test dependency graph and last-run signatures |
+| Invalidates when | prefix is bumped (essentially never, by design) |
+| Does **not** invalidate on | `uv.lock` or `pyproject.toml` changes -- testmon detects changed dependency hashes itself and re-runs only the affected tests |
+| Restore semantics | **fail-open**; a miss only costs full-suite runtime, never correctness, and testmon handles stale DBs gracefully |
+| Save semantics | nightly `testmon` job only, via the `replace-cache` action with `if: always()` so partial DBs from flaky runs still publish |
+
+Historical note: the key was previously suffixed with
+`hashFiles('uv.lock', 'pyproject.toml')`.  Because GitHub Actions
+caches are immutable, two consecutive nightlies with an unchanged
+lockfile (the common case) collided on the same key, and the second
+save logged `Failed to save: Unable to reserve cache` only as a
+*warning*.  The stale DB persisted for days, PRs restored it via the
+prefix fallback, and testmon then invalidated everything because the
+realized environment had drifted away from what the cached DB
+recorded.  Switching to a `-latest` mutable slot via `replace-cache`
+fixes the save bug, and the embedded verify step turns any future
+silent save failure into a hard job failure.
+
+#### Why a separate `ci-requirements.lock`
+
+The cache fix above only addresses *saving* the DB; the DB is still
+worthless to PRs if testmon's environment fingerprint at PR time
+differs from the fingerprint stored at nightly time.  Testmon
+computes that fingerprint from `importlib.metadata.distributions()`
+over the active venv -- i.e. *everything* in
+`.venv/lib/python3.12/site-packages`, not just the lockfile-pinned
+closure.
+
+`setup-uv-env` builds the venv in two layered steps:
+
+1. `uv sync --frozen --group dev --extra <EXTRAS_TAG>` -- deterministic
+   against `uv.lock`.
+2. `uv pip install -r .github/ci-requirements.txt` -- adds CI-only
+   test deps that have no home in pyproject extras (moto,
+   scikit-image, numpy-stl, shapely, multi-storage-client, tensorstore,
+   plus the PyG CUDA wheel swap).
+
+Step 2 is *not* covered by `uv.lock`.  Several of the direct pins in
+`ci-requirements.txt` are absent from `uv.lock` entirely, so their
+transitive closure (`responses`, `xmltodict`, `jsonpath-ng`,
+`lazy-loader`, `tifffile`, `pywavelets`, `imageio`,
+`antlr4-python3-runtime`, ...) gets re-resolved fresh against PyPI on
+every job.  A single transitive minor bump between the nightly that
+publishes the testmon DB and the PR that consumes it changes the
+sorted `name version` string testmon hashes, trips its
+"packages installed have been changed" guard, and re-runs the entire
+suite.
+
+[`.github/ci-requirements.lock`](ci-requirements.lock) is a fully
+pinned closure of `ci-requirements.txt` (direct + transitive), passed
+to the layered install via `--constraint`.  It is generated by
+[`.github/regen-ci-deps-lock.sh`](regen-ci-deps-lock.sh), and must be
+regenerated and committed whenever a `==` pin in
+`ci-requirements.txt` changes.
+
+Two ways to run the regen:
+
+1. **Standalone [`Regen CI-deps Lock`](workflows/regen-ci-deps-lock.yml)
+   workflow** (workflow_dispatch).  Runs the regen on a CPU runner
+   in ~5 min and uploads `.github/ci-requirements.lock` as an
+   artifact for the maintainer to download and commit.  Requires
+   the workflow file to be on the default branch (GitHub refuses
+   workflow_dispatch on files that exist only on feature branches --
+   both the UI dropdown and `gh workflow run --ref` enforce this).
+
+2. **Local docker.**  See the header of
+   [`.github/regen-ci-deps-lock.sh`](regen-ci-deps-lock.sh) for the
+   `docker run …` invocation.  Useful when iterating on the script
+   itself or when the standalone workflow is unavailable (e.g. a
+   feature branch where the workflow file has not yet landed on
+   the default branch).
+
+### Coverage baseline cache (`.coverage*`)
+
+| Property | Value |
+|---|---|
+| Key | `<COVERAGE_CACHE_KEY_PREFIX>-latest` |
+| Prefix encodes | nightly identity (`coverage-nightly`) |
+| Suffix | literal `latest` (mutable slot, refreshed via delete-before-save) |
+| Contents | parallel-mode coverage shards (`.coverage.*`) produced by the nightly's full-suite pytest run, before `coverage combine` |
+| Invalidates when | prefix is bumped |
+| Does **not** invalidate on | `uv.lock` or `pyproject.toml` changes |
+| Restore semantics | **fail-open**; PR coverage merges its own shards on top of the restored baseline |
+| Save semantics | nightly `coverage` job only, via the `replace-cache` action |
+
+Same immutable-key bug class as testmon; migrated to the `-latest`
+slot for parity.
+
+## Reusable building blocks
+
+### `replace-cache` action ([.github/actions/replace-cache/action.yml](actions/replace-cache/action.yml))
+
+All four mutable-slot caches above (uv, JIT, testmon, coverage) share
+the same delete-before-save recipe: GitHub Actions cache slots are
+immutable, so refreshing a `-latest` key requires deleting the
+existing entry, calling `actions/cache/save`, and (because the save
+silently no-ops on key collision) re-querying `gh cache list` to
+confirm the slot now exists.  The `replace-cache` composite action
+encapsulates that recipe:
+
+```yaml
+- name: Replace <some> cache
+  if: <caller-supplied gate>
+  uses: ./.github/actions/replace-cache
+  with:
+    path: <one or more paths>
+    key:  <foo>-latest
+    description: <human-readable label>
+    github-token: ${{ secrets.GITHUB_TOKEN }}
+```
+
+The verify step is on by default (`verify: "true"`).  Disable only
+when verification is genuinely undesirable; the default exists
+because silent save failures are how stale slots persist for days.
+
 ## Why no `.venv` cache
 
 A previous iteration of this pipeline also cached the realized `.venv`
@@ -114,10 +236,12 @@ Guarantees:
 - **Concurrency**: the nightly workflow declares
   `concurrency: nightly-github-uv` with `cancel-in-progress: false` so
   two overlapping runs cannot race on the static `-latest` uv cache key.
-- **Save verification**: after `actions/cache/save@v4` writes the uv
-  download cache slot, the workflow re-queries `gh cache list` to
-  confirm the entry exists. `cache/save` silently no-ops on key
-  collision; without verification a corrupted slot can persist for days.
+- **Save verification**: every mutable-slot save (uv download, JIT,
+  testmon, coverage) goes through the `replace-cache` action, which
+  re-queries `gh cache list` after `actions/cache/save` and fails the
+  job if the slot is not visible.  `cache/save` silently no-ops on
+  key collision and only logs a warning on reservation failure;
+  without verification a corrupted slot can persist for days.
 - **Lockfile-mutation guard**: [.github/actions/setup-uv-env/action.yml](actions/setup-uv-env/action.yml)
   snapshots `sha256(uv.lock)` and `sha256(pyproject.toml)` before any uv
   command runs and compares them again at the end. Any drift (caused by

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -31,7 +31,7 @@ body:
     attributes:
       label: Version
       description: What version of PhysicsNeMo are you running?
-      placeholder: "example: 2.0.0"
+      placeholder: "example: 2.1.0"
     validations:
       required: true
 

.github/actions/bootstrap-cudnn-ci/action.yml

@@ -79,7 +79,10 @@ runs:
         PIP_BREAK_SYSTEM_PACKAGES: "1"
       run: |
         set -euo pipefail
-        pip install --quiet huggingface_hub[hf_xet]
+        # typer >= 0.26 dropped click as a runtime dep, but huggingface_hub's
+        # `hf` CLI still does `import click`. Install click explicitly until
+        # upstream re-adds it (or the CLI stops importing click directly).
+        pip install --quiet huggingface_hub[hf_xet] click
 
     - name: Install uv (pinned)
       shell: bash

.github/actions/download-ci-data/action.yml

@@ -57,7 +57,8 @@ runs:
         fi
 
         if ! command -v hf >/dev/null 2>&1; then
-          uv pip install --system --quiet huggingface_hub[hf_xet]
+          # See bootstrap-cudnn-ci: typer 0.26 dropped click; hf CLI still imports it.
+          uv pip install --system --quiet huggingface_hub[hf_xet] click
         fi
         hf download "${HF_REPO}" \
           --repo-type dataset \

.github/actions/replace-cache/action.yml

@@ -0,0 +1,122 @@
+name: Replace mutable-slot cache
+description: |
+  Save a path to a GitHub Actions cache slot under a mutable key, replacing
+  any existing entry under that key.  The actions/cache/save action is a
+  silent no-op when its key already exists (caches are immutable), so the
+  canonical recipe to "refresh" a `-latest`-style slot is delete-before-save:
+
+    1. Look up the key with `gh cache list` and `gh cache delete` it if
+       present (tolerate missing key for first runs and prefix bumps).
+    2. Save with actions/cache/save@v5.
+    3. Optionally re-query `gh cache list` to confirm the save took effect.
+       GitHub's cache index is eventually consistent, so the verify step
+       retries a few times before failing.  Without this verify step, a
+       collision that prevented the save (e.g. a concurrent writer or a
+       stale slot the delete somehow missed) would surface only as a
+       `Warning: Cache save failed.` line and a corrupted slot would
+       persist indefinitely.
+
+  This action centralises that recipe so callers can express
+  delete-before-save as a single step.  Each invocation site supplies its
+  own `if:` gate (e.g. `if: always()` or `if: <cold-cache>`) at the
+  workflow level; this action does not encode any save policy of its own.
+
+  Failure semantics: the delete tolerates a missing slot, but the save
+  itself and (when enabled) the verify step will fail the job loudly.  A
+  silent save failure on a mutable slot is exactly the bug class this
+  action exists to prevent.
+inputs:
+  path:
+    description: |
+      Cache path(s), forwarded to actions/cache/save.  Multi-line values
+      are supported with the same semantics as actions/cache.
+    required: true
+  key:
+    description: |
+      Mutable-slot cache key (e.g. `foo-latest`).  This is the slot that
+      will be deleted (if present) and then saved.
+    required: true
+  description:
+    description: |
+      Short human-readable label used in log messages, e.g. "uv download
+      cache" or "testmon database".
+    required: false
+    default: "cache"
+  verify:
+    description: |
+      When `"true"` (default), re-query `gh cache list` after the save and
+      fail the job if the slot is not visible within a few retries.  Set
+      to `"false"` only when verification is genuinely undesirable; the
+      default exists because silent save failures are how stale slots
+      persist for days.
+    required: false
+    default: "true"
+  github-token:
+    description: |
+      Token for the `gh` CLI used by the delete and verify steps.  Pass
+      the workflow's `secrets.GITHUB_TOKEN` from the calling workflow.
+    required: true
+runs:
+  using: composite
+  steps:
+    - name: Delete stale ${{ inputs.description }} entry
+      shell: bash
+      env:
+        GH_TOKEN: ${{ inputs.github-token }}
+        CACHE_KEY: ${{ inputs.key }}
+        CACHE_DESC: ${{ inputs.description }}
+        REPO: ${{ github.repository }}
+      run: |
+        set -euo pipefail
+        if ! command -v gh >/dev/null 2>&1; then
+          echo "::error::gh CLI not on PATH; cannot manage ${CACHE_DESC} slot."
+          exit 1
+        fi
+        # Use --json key + --jq for robust matching (no false positives
+        # on prefix overlap from sibling cache keys).
+        existing="$(gh cache list \
+          --repo "$REPO" \
+          --key "$CACHE_KEY" \
+          --json key \
+          --jq '.[].key' \
+          | grep -Fx "$CACHE_KEY" || true)"
+        if [ -n "$existing" ]; then
+          gh cache delete "$CACHE_KEY" --repo "$REPO"
+          echo "deleted stale ${CACHE_DESC}: $CACHE_KEY"
+        else
+          echo "no existing ${CACHE_DESC} to delete: $CACHE_KEY"
+        fi
+
+    - name: Save ${{ inputs.description }}
+      uses: actions/cache/save@v5
+      with:
+        path: ${{ inputs.path }}
+        key: ${{ inputs.key }}
+
+    # actions/cache/save@v5 silently no-ops on key collision; if the
+    # previous delete step somehow left the entry in place (or a
+    # concurrent run repopulated it), we want a hard failure now rather
+    # than a stale cache fed to tomorrow's consumer.
+    - name: Verify ${{ inputs.description }} was saved
+      if: inputs.verify == 'true'
+      shell: bash
+      env:
+        GH_TOKEN: ${{ inputs.github-token }}
+        CACHE_KEY: ${{ inputs.key }}
+        CACHE_DESC: ${{ inputs.description }}
+        REPO: ${{ github.repository }}
+      run: |
+        set -euo pipefail
+        # GitHub's cache index is eventually consistent; use linear back-off
+        # (10 s, 20 s, 30 s, 40 s, 50 s → 150 s total) before failing.
+        for attempt in 1 2 3 4 5; do
+          if gh cache list --repo "$REPO" --key "$CACHE_KEY" --json key --jq '.[].key' \
+              | grep -Fxq "$CACHE_KEY"; then
+            echo "${CACHE_DESC} present: $CACHE_KEY"
+            exit 0
+          fi
+          echo "attempt $attempt: ${CACHE_DESC} not yet visible, sleeping..."
+          sleep $((10 * attempt))
+        done
+        echo "::error::${CACHE_DESC} save did not take effect for key $CACHE_KEY"
+        exit 1