Clean up CI annotations and surface sccache stats from cibuildwheel (#1879)

leofang · claude · web-flow · commit 738dbdc8420d · 2026-04-10T13:14:43.000-04:00
* Suppress pip cache and root-user warnings in Linux container CI jobs Set PIP_CACHE_DIR=/tmp/pip-cache and PIP_ROOT_USER_ACTION=ignore in the container env for test-wheel-linux and coverage-linux jobs. These jobs run as root in Ubuntu containers where /github/home/.cache/pip is not writable, causing ~54 harmless warnings per CI run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Move PIP_ROOT_USER_ACTION from container.env to job-level env The pip root-user warning was still appearing because actions/setup-python runs its internal "pip upgrade" on the host runner, not inside the container. Container-level env vars are invisible to host-side actions. Moving PIP_ROOT_USER_ACTION to the job-level env block makes it available to both host-side actions (setup-python) and container-side run steps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Revert PIP_ROOT_USER_ACTION changes (ineffective for setup-python) The pip root-user warning originates from actions/setup-python's internal ensurepip call, which deliberately strips all PIP_* env vars (CPython design, see python/cpython#139363). Neither container.env nor job-level env can suppress it. See PR comment for full analysis. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Bump actions/cache to v5.0.4 (node24) and disable sccache annotations - Update actions/cache from v4.2.3 (node20) to v5.0.4 (node24) in fetch_ctk to eliminate Node.js 20 deprecation warnings. All runners are on v2.332+ (v5 requires >= 2.327.1). - Set disable_annotations on sccache-action to suppress the cache stats notice annotations and job summaries. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Bump doc_preview actions to node24-compatible versions - JamesIves/github-pages-deploy-action: v4.7.3 (node20) → v4.8.0 (node24) - marocchino/sticky-pull-request-comment: v2.9.2 (node20) → v3.0.3 (node24) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add sccache-summary action and surface container stats in job summary sccache stats were previously lost because cibuildwheel runs compilation inside a manylinux container with its own sccache server instance, while sccache-action on the host sees 0 hits. Fix by dumping sccache stats JSON from inside the container to the host filesystem (via /host/ mount), then reading it in a new composite action that writes a formatted table to GITHUB_STEP_SUMMARY. Inspired by NVIDIA/cccl PR #3621. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix sccache hit rate calculation and add step link hint - Use cache_hits + cache_misses (per language) as the denominator instead of compile_requests, which includes non-compilation calls (linker invocations, etc). This matches sccache's own hit rate. - Add build-step input to reference the cibuildwheel step name in the summary for easier navigation to full stats. - Remove intermediate summary file; only write to GITHUB_STEP_SUMMARY. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Remove build-step input from sccache-summary action GHA has no construct to reference a previous step's name dynamically (steps context only exposes outcome/conclusion/outputs). The label input already identifies which build produced the stats, so the hardcoded build-step name is redundant. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Revert "Remove build-step input from sccache-summary action" This reverts commit f4ed3f3. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
diff --git a/.github/actions/doc_preview/action.yml b/.github/actions/doc_preview/action.yml
@@ -22,7 +22,7 @@ runs:
     # Note: the PR previews will be removed once merged to main or release/* (see below)
     - name: Deploy doc preview
       if: ${{ github.ref_name != 'main' && !startsWith(github.ref_name, 'release/') }}
-      uses: JamesIves/github-pages-deploy-action@6c2d9db40f9296374acc17b90404b6e8864128c8  # v4.7.3
+      uses: JamesIves/github-pages-deploy-action@d92aa235d04922e8f08b40ce78cc5442fcfbfa2f  # v4.8.0
       with:
         git-config-name: cuda-python-bot
         git-config-email: cuda-python-bot@users.noreply.github.com
@@ -32,7 +32,7 @@ runs:
 
     - name: Leave a comment after deployment
       if: ${{ github.ref_name != 'main' && !startsWith(github.ref_name, 'release/') }}
-      uses: marocchino/sticky-pull-request-comment@67d0dec7b07ed060a405f9b2a64b8ab319fdd7db  # v2.9.2
+      uses: marocchino/sticky-pull-request-comment@d4d6b0936434b21bc8345ad45a440c5f7d2c40ff  # v3.0.3
       with:
         header: pr-preview
         number: ${{ inputs.pr-number }}
@@ -49,7 +49,7 @@ runs:
     # The steps below are executed only when building on main or release/*.
     - name: Remove doc preview
       if: ${{ github.ref_name == 'main' || startsWith(github.ref_name, 'release/') }}
-      uses: JamesIves/github-pages-deploy-action@6c2d9db40f9296374acc17b90404b6e8864128c8  # v4.7.3
+      uses: JamesIves/github-pages-deploy-action@d92aa235d04922e8f08b40ce78cc5442fcfbfa2f  # v4.8.0
       with:
         git-config-name: cuda-python-bot
         git-config-email: cuda-python-bot@users.noreply.github.com
@@ -59,7 +59,7 @@ runs:
 
     - name: Leave a comment after removal
       if: ${{ github.ref_name == 'main' || startsWith(github.ref_name, 'release/') }}
-      uses: marocchino/sticky-pull-request-comment@67d0dec7b07ed060a405f9b2a64b8ab319fdd7db  # v2.9.2
+      uses: marocchino/sticky-pull-request-comment@d4d6b0936434b21bc8345ad45a440c5f7d2c40ff  # v3.0.3
       with:
         header: pr-preview
         number: ${{ inputs.pr-number }}
diff --git a/.github/actions/fetch_ctk/action.yml b/.github/actions/fetch_ctk/action.yml
@@ -60,7 +60,7 @@ runs:
 
     - name: Download CTK cache
       id: ctk-get-cache
-      uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684  # v4.2.3
+      uses: actions/cache/restore@668228422ae6a00e4ad889ee87cd7109ec5666a7  # v5.0.4
       continue-on-error: true
       with:
         key: ${{ env.CTK_CACHE_KEY }}
@@ -142,7 +142,7 @@ runs:
     - name: Upload CTK cache
       if: ${{ !cancelled() &&
               steps.ctk-get-cache.outputs.cache-hit != 'true' }}
-      uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684  # v4.2.3
+      uses: actions/cache/save@668228422ae6a00e4ad889ee87cd7109ec5666a7  # v5.0.4
       with:
         key: ${{ env.CTK_CACHE_KEY }}
         path: ./${{ env.CTK_CACHE_FILENAME }}
diff --git a/.github/actions/sccache-summary/action.yml b/.github/actions/sccache-summary/action.yml
@@ -0,0 +1,81 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+name: sccache summary
+description: Parse sccache stats JSON and write a summary table to GITHUB_STEP_SUMMARY
+
+# Inspired by NVIDIA/cccl's prepare-execution-summary.py (PR #3621).
+# Only counts C/C++ and CUDA language hits (excludes PTX/CUBIN which are
+# not included in sccache's compile_requests counter).
+
+inputs:
+  json-file:
+    description: "Path to the sccache stats JSON file (from sccache --show-stats --stats-format=json)"
+    required: true
+  label:
+    description: "Label for the stats row (e.g. cuda.bindings, cuda.core)"
+    required: false
+    default: "sccache"
+  build-step:
+    description: "Name of the cibuildwheel build step (for deep-link in summary)"
+    required: false
+    default: ""
+
+runs:
+  using: composite
+  steps:
+    - name: Report sccache stats
+      shell: bash --noprofile --norc -euo pipefail {0}
+      env:
+        SCCACHE_JSON: ${{ inputs.json-file }}
+        SCCACHE_LABEL: ${{ inputs.label }}
+        SCCACHE_BUILD_STEP: ${{ inputs.build-step }}
+      run: |
+        if [ ! -f "$SCCACHE_JSON" ]; then
+          echo "::warning::sccache stats file not found: $SCCACHE_JSON"
+          exit 0
+        fi
+
+        python3 - <<'PYEOF'
+        import json, os, urllib.parse
+
+        json_file = os.environ["SCCACHE_JSON"]
+        label = os.environ["SCCACHE_LABEL"]
+        build_step = os.environ.get("SCCACHE_BUILD_STEP", "")
+
+        with open(json_file) as f:
+            stats = json.load(f)["stats"]
+
+        # compile_requests includes non-compilation calls (linker, etc).
+        # Use cache_hits + cache_misses as the denominator to match sccache's
+        # own "Cache hits rate" which only counts actual compilation requests.
+        counted_languages = {"C/C++", "CUDA"}
+        hits = sum(
+            v for k, v in stats.get("cache_hits", {}).get("counts", {}).items()
+            if k in counted_languages
+        )
+        misses = sum(
+            v for k, v in stats.get("cache_misses", {}).get("counts", {}).items()
+            if k in counted_languages
+        )
+        total = hits + misses
+        pct = int(100 * hits / total) if total > 0 else 0
+
+        # Build a deep-link to the cibuildwheel step if step name is provided.
+        # GHA step summary links use the format: #step:N:L but we can't know the
+        # step number here. Instead, link to the job page with a search hint.
+        link_note = ""
+        if build_step:
+            link_note = f"\n\n_Full stats in the **{build_step}** step log._\n"
+
+        summary_file = os.environ.get("GITHUB_STEP_SUMMARY", "")
+        if summary_file:
+            with open(summary_file, "a") as sf:
+                sf.write(f"### 📊 {label} — sccache stats\n")
+                sf.write("| Hit Rate | Hits | Misses | Requests |\n")
+                sf.write("|----------|------|--------|----------|\n")
+                sf.write(f"| {pct}% | {hits} | {misses} | {total} |{link_note}\n")
+
+        print(f"{label}: {pct}% hit rate ({hits}/{total})")
+        PYEOF
diff --git a/.github/workflows/build-wheel.yml b/.github/workflows/build-wheel.yml
@@ -48,6 +48,8 @@ jobs:
       # are exposed by this action.
       - name: Enable sccache
         uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad  # 0.0.9
+        with:
+          disable_annotations: 'true'
 
       # xref: https://github.com/orgs/community/discussions/42856#discussioncomment-7678867
       - name: Adding addtional GHA cache-related env vars
@@ -175,13 +177,22 @@ jobs:
             CUDA_PYTHON_PARALLEL_LEVEL=${{ env.CUDA_PYTHON_PARALLEL_LEVEL }}
           # check cache stats before leaving cibuildwheel
           CIBW_BEFORE_TEST_LINUX: >
-            "/host/${{ env.SCCACHE_PATH }}" --show-stats
+            "/host/${{ env.SCCACHE_PATH }}" --show-stats &&
+            "/host/${{ env.SCCACHE_PATH }}" --show-stats --stats-format=json > /host/${{ github.workspace }}/sccache_bindings.json
           # force the test stage to be run (so that before-test is not skipped)
           # TODO: we might want to think twice on adding this, it does a lot of
           # things before reaching this command.
           CIBW_TEST_COMMAND: >
             echo "ok!"
 
+      - name: Report sccache stats (cuda.bindings)
+        if: ${{ inputs.host-platform != 'win-64' }}
+        uses: ./.github/actions/sccache-summary
+        with:
+          json-file: sccache_bindings.json
+          label: "cuda.bindings"
+          build-step: "Build cuda.bindings wheel"
+
       - name: List the cuda.bindings artifacts directory
         run: |
           if [[ "${{ inputs.host-platform }}" == win* ]]; then
@@ -233,13 +244,22 @@ jobs:
             PIP_FIND_LINKS="$(cygpath -w ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }})"
           # check cache stats before leaving cibuildwheel
           CIBW_BEFORE_TEST_LINUX: >
-            "/host${{ env.SCCACHE_PATH }}" --show-stats
+            "/host${{ env.SCCACHE_PATH }}" --show-stats &&
+            "/host${{ env.SCCACHE_PATH }}" --show-stats --stats-format=json > /host/${{ github.workspace }}/sccache_core.json
           # force the test stage to be run (so that before-test is not skipped)
           # TODO: we might want to think twice on adding this, it does a lot of
           # things before reaching this command.
           CIBW_TEST_COMMAND: >
             echo "ok!"
 
+      - name: Report sccache stats (cuda.core)
+        if: ${{ inputs.host-platform != 'win-64' }}
+        uses: ./.github/actions/sccache-summary
+        with:
+          json-file: sccache_core.json
+          label: "cuda.core"
+          build-step: "Build cuda.core wheel"
+
       - name: List the cuda.core artifacts directory and rename
         run: |
           if [[ "${{ inputs.host-platform }}" == win* ]]; then
@@ -412,13 +432,22 @@ jobs:
             PIP_FIND_LINKS="$(cygpath -w ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }})"
           # check cache stats before leaving cibuildwheel
           CIBW_BEFORE_TEST_LINUX: >
-            "/host${{ env.SCCACHE_PATH }}" --show-stats
+            "/host${{ env.SCCACHE_PATH }}" --show-stats &&
+            "/host${{ env.SCCACHE_PATH }}" --show-stats --stats-format=json > /host/${{ github.workspace }}/sccache_core_prev.json
           # force the test stage to be run (so that before-test is not skipped)
           # TODO: we might want to think twice on adding this, it does a lot of
           # things before reaching this command.
           CIBW_TEST_COMMAND: >
             echo "ok!"
 
+      - name: Report sccache stats (cuda.core prev)
+        if: ${{ inputs.host-platform != 'win-64' }}
+        uses: ./.github/actions/sccache-summary
+        with:
+          json-file: sccache_core_prev.json
+          label: "cuda.core (prev CTK)"
+          build-step: "Build cuda.core wheel"
+
       - name: List the cuda.core artifacts directory and rename
         run: |
           if [[ "${{ inputs.host-platform }}" == win* ]]; then
diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml
@@ -50,6 +50,7 @@ jobs:
       image: ubuntu:22.04
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
+        PIP_CACHE_DIR: "/tmp/pip-cache"
     steps:
       - name: Ensure GPU is working
         run: nvidia-smi
diff --git a/.github/workflows/test-wheel-linux.yml b/.github/workflows/test-wheel-linux.yml
@@ -85,6 +85,7 @@ jobs:
       image: ubuntu:22.04
       env:
         NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
+        PIP_CACHE_DIR: "/tmp/pip-cache"
     steps:
       - name: Ensure GPU is working
         run: nvidia-smi
