Skip to content

COMP: Free disk space between build and test in Pixi CI#6011

Merged
hjmjohnson merged 1 commit intoInsightSoftwareConsortium:mainfrom
hjmjohnson:fix-ci-disk-space
Apr 6, 2026
Merged

COMP: Free disk space between build and test in Pixi CI#6011
hjmjohnson merged 1 commit intoInsightSoftwareConsortium:mainfrom
hjmjohnson:fix-ci-disk-space

Conversation

@hjmjohnson
Copy link
Copy Markdown
Member

Problem

The Pixi-Cxx (ubuntu-22.04) job intermittently fails during the Test step with no error message — the runner is killed mid-test due to disk space exhaustion. The runner has 146GB total, and after a ~2 hour ITK build the remaining space is insufficient for CTest to run.

This affects PRs #6007, #6009, #6010 (and likely others). The successful main branch build completed in 2h07m, while failing PR builds ran 2h08m-2h55m before being killed ~3-4 minutes into testing.

Fix

Move the existing .o/.a cleanup to a dedicated "Free disk space after build" step and add additional cleanup:

  • Remove InsightData-*.tar.gz download (~200MB, already extracted)
  • Remove extracted InsightToolkit-* source directory
  • Flush ccache temp files
  • Report df -h after cleanup for diagnostics

Impact

No functional change — the removed files are not needed for testing. The .o files were already being deleted; this adds the tarball and source directory cleanup.

🤖 Generated with Claude Code

@github-actions github-actions bot added type:Compiler Compiler support or related warnings type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots labels Apr 4, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Apr 4, 2026

Greptile Summary

This PR refactors the GitHub Actions CI workflow to free disk space between the build and test phases, addressing intermittent job failures on ubuntu-22.04 caused by disk exhaustion.

  • Moves the existing .o/.a cleanup commands from the Build step into a dedicated Free disk space after build step, improving readability and separation of concerns.
  • Adds removal of the downloaded InsightData-*.tar.gz tarball and the extracted InsightToolkit-* source directory, which are no longer needed after data is copied into .ExternalData/CID.
  • Adds ccache --cleanup to trim the cache and a df -h report for post-cleanup diagnostics.
  • The new step explicitly sets shell: bash, which is correct for the find/rm commands used and is consistent across all three matrix OS targets (Linux, macOS, Windows with Git Bash).
  • Two minor style issues: the comment describing ccache --cleanup is inaccurate (it does not "flush to persistent cache dir"), and the InsightToolkit-* glob is broader than necessary compared to using the pinned ${{ env.ExternalDataVersion }} variable.

Confidence Score: 4/5

Safe to merge — changes are CI-only cleanup steps with no functional impact on build or test logic.

The PR makes purely additive CI maintenance changes. The removed files (tarballs, source dirs, object files) are demonstrably not needed for the test phase. The only issues found are a misleading comment and a slightly broad glob, neither of which causes failures. Score is 4 rather than 5 due to the minor glob safety concern.

Only .github/workflows/pixi.yml is changed; both minor concerns are localized to lines 148-150 of that file.

Important Files Changed

Filename Overview
.github/workflows/pixi.yml Moves existing .o/.a cleanup into a dedicated step and adds tarball/source-dir removal and ccache trim to free disk space before testing; logic is correct but contains a misleading comment on ccache --cleanup and uses a broad glob for source directory removal.

Sequence Diagram

sequenceDiagram
    participant GH as GitHub Actions
    participant Runner as ubuntu-22.04 Runner
    participant FS as Filesystem

    GH->>Runner: Checkout
    GH->>Runner: Install ccache
    GH->>Runner: Restore compiler cache
    GH->>Runner: Free Disk Space (Ubuntu)
    GH->>Runner: Download testing data
    Runner->>FS: curl InsightData-*.tar.gz (save ~200MB)
    Runner->>FS: cmake -E tar xfz (extract InsightToolkit-X.Y.Z/)
    Runner->>FS: cmake -E rename .ExternalData/CID to workspace
    GH->>Runner: Configure (pixi run configure-ci)
    GH->>Runner: Build (pixi run build)
    Runner->>FS: Produces build/ with .o .a binaries
    GH->>Runner: Free disk space after build [NEW STEP]
    Runner->>FS: find build -delete *.o *.a
    Runner->>FS: rm -f InsightData-*.tar.gz
    Runner->>FS: rm -rf InsightToolkit-*
    Runner->>FS: ccache --cleanup (trim to CCACHE_MAXSIZE)
    Runner->>GH: df -h / (post-cleanup report)
    GH->>Runner: Test (pixi run test)
    GH->>Runner: Save compiler cache
Loading

Reviews (1): Last reviewed commit: "COMP: Free disk space between build and ..." | Re-trigger Greptile

hjmjohnson added a commit to hjmjohnson/ITK that referenced this pull request Apr 4, 2026
- Use ExternalDataVersion variable instead of broad InsightToolkit-*
  glob for source directory removal.
- Fix misleading ccache --cleanup comment: it trims the cache to stay
  within CCACHE_MAXSIZE, not "flush to persistent cache dir".

Addresses review comments from @greptile-apps on PR InsightSoftwareConsortium#6011.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The ubuntu-22.04 runner runs out of disk space during the Test step
after a ~2 hour build. The runner has 146GB total and the build tree
plus build artifacts can exceed available space, causing the runner
to be killed mid-test with no error message.

Move the .o/.a cleanup to a dedicated step and add additional cleanup:
- Remove InsightData tarball (~200MB, already extracted)
- Remove extracted source tarball directory
- Flush ccache temp files
- Report disk usage after cleanup for diagnostics

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@hjmjohnson
Copy link
Copy Markdown
Member Author

Addressed both review comments (squashed into the single commit):

  1. ccache comment — Fixed: now reads "Trim ccache to stay within CCACHE_MAXSIZE and remove orphaned entries"
  2. Broad glob — Fixed: now uses the pinned version rm -rf InsightToolkit-${{ env.ExternalDataVersion }} instead of InsightToolkit-*

@hjmjohnson
Copy link
Copy Markdown
Member Author

Analysis: applicability to other CI systems

Reviewed all other CI configurations — the disk space cleanup in this PR is specific to pixi.yml and not applicable to the other systems:

  • arm.yml: Uses a CTest dashboard script (ctest -S dashboard.cmake) that runs build and test in a single step. There is no separate phase between build and test where cleanup can be inserted.
  • pre-commit.yml: Only runs linting/formatting — no C++ build.
  • Azure Pipelines (AzurePipelinesLinux.yml, Windows.yml, MacOS.yml, *Python.yml): All use ctest -S dashboard.cmake with a monolithic build+test step. Azure VMs also have more disk than GitHub Actions runners.

The disk pressure issue is specific to pixi.yml because it is the only workflow with separate pixi run build and pixi run test steps on GitHub Actions ubuntu-22.04 runners (~14 GB free disk).

Copy link
Copy Markdown
Member

@dzenanz dzenanz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this PR been superseded by #6013?

@hjmjohnson
Copy link
Copy Markdown
Member Author

Has this PR been superseded by #6013?

No. Two separate items. (both related disk space optimization). This clears disk space before the tests start running in CI only. #6013 is about clearing huge disk usage in all environments (including the 50GB * Debug/Release * clang/Apple Clang = 200GB) in my mac laptop build environments.

@hjmjohnson hjmjohnson enabled auto-merge April 6, 2026 20:51
@hjmjohnson hjmjohnson merged commit 1c46ed3 into InsightSoftwareConsortium:main Apr 6, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:Compiler Compiler support or related warnings type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants