Skip to content

fix: keep pipeline pytest temp files off /tmp#1797

Open
nightcityblade wants to merge 1 commit intoNVIDIA-NeMo:mainfrom
nightcityblade:fix/issue-1775
Open

fix: keep pipeline pytest temp files off /tmp#1797
nightcityblade wants to merge 1 commit intoNVIDIA-NeMo:mainfrom
nightcityblade:fix/issue-1775

Conversation

@nightcityblade
Copy link
Copy Markdown
Contributor

Description

For the CPU pipelines test shard, send pytest temporary files to $GITHUB_WORKSPACE/pytest-tmp instead of the runner's default /tmp. This keeps the Ray --temp-dir/pytest temp tree off the smaller /tmp partition and matches the workaround described in the issue.

Closes #1775.

Usage

# CI-only change: the pipelines shard now runs roughly like
# pytest tests/pipelines -m "not gpu" --basetemp="$GITHUB_WORKSPACE/pytest-tmp"

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

Signed-off-by: nightcityblade <nightcityblade@gmail.com>
@nightcityblade nightcityblade requested a review from a team as a code owner April 11, 2026 03:12
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented Apr 11, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Apr 11, 2026

Greptile Summary

This PR adds a conditional branch in the cicd-cpu-tests job so that the pipelines matrix shard redirects pytest's temp tree (and the Ray --temp-dir derived from it) to $GITHUB_WORKSPACE/pytest-tmp instead of the default /tmp, preventing the smaller /tmp partition from filling up on runners that execute pipeline tests with Ray.

Confidence Score: 5/5

Safe to merge — minimal, targeted CI-only change with no impact on production code or test logic.

The change is a single conditional block in the CI workflow, correctly scoped to the pipelines shard, uses the well-established --basetemp pytest flag, and the mkdir -p guard prevents any race with pytest trying to create sub-directories. No P0/P1 issues identified.

No files require special attention.

Important Files Changed

Filename Overview
.github/workflows/cicd-main.yml Adds a conditional block for the pipelines shard that creates $GITHUB_WORKSPACE/pytest-tmp and passes --basetemp to pytest; all other shards are unchanged.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[cicd-cpu-tests matrix job\nfolder × python-version] --> B{FOLDER == 'pipelines'?}
    B -- yes --> C[mkdir -p\n$GITHUB_WORKSPACE/pytest-tmp]
    C --> D[coverage run ... pytest\n--basetemp=$GITHUB_WORKSPACE/pytest-tmp]
    D --> E[Temp files → workspace\nlarge partition]
    B -- no --> F[coverage run ... pytest\ndefault basetemp]
    F --> G[Temp files → /tmp\nsmall partition]
Loading

Reviews (1): Last reviewed commit: "fix: keep pipeline pytest temp files off..." | Re-trigger Greptile

@chtruong814 chtruong814 added the needs-follow-up Issue needs follow-up label Apr 13, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed needs-follow-up Issue needs follow-up labels Apr 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: runtime_env tests exhaust /tmp on CPU runners due to Ray venv cloning

3 participants