Skip to content

Add SLURM_MEM_PER_CPU to keep_slurm variables#51

Open
ASLeonard wants to merge 1 commit into
snakemake:mainfrom
ASLeonard:main
Open

Add SLURM_MEM_PER_CPU to keep_slurm variables#51
ASLeonard wants to merge 1 commit into
snakemake:mainfrom
ASLeonard:main

Conversation

@ASLeonard

@ASLeonard ASLeonard commented Jul 6, 2026

Copy link
Copy Markdown

I noticed my slurm jobs started being OOM all of a sudden despite using only a fraction of the allocated memory. Eventually seems to be the changes in d0c515c, where the slurm-jobset (srun) was not inheriting the memory allocation of the slurm-executor (sbatch). I have a memory value in my config default-resources, which was being used instead.

I confirmed this change correctly forwards the memory requested to the actual jobstep via sacct. Our HPC does not allow passing --mem=0, so this may a fragile approach to more generally passing on the mem/mem-per-cpu variable.

Summary by CodeRabbit

  • Bug Fixes
    • Preserved the SLURM memory-per-CPU setting for nested job launches, helping child srun commands inherit the expected memory allocation.
    • Improved consistency for jobs that rely on nested SLURM execution.

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

A single environment variable, SLURM_MEM_PER_CPU, was added to the keep_slurm allowlist in Executor.run_job, so it is now preserved rather than stripped when sanitizing the environment for nested srun invocations.

Changes

Environment allowlist update

Layer / File(s) Summary
Allowlist SLURM_MEM_PER_CPU
snakemake_executor_plugin_slurm_jobstep/__init__.py
Adds SLURM_MEM_PER_CPU to the keep_slurm allowlist so it is retained for nested srun calls instead of being removed.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: preserving SLURM_MEM_PER_CPU in the keep_slurm allowlist.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
snakemake_executor_plugin_slurm_jobstep/__init__.py (1)

208-220: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Consider also keeping SLURM_MEM_PER_NODE.

SLURM_MEM_PER_CPU and SLURM_MEM_PER_NODE correspond to the mutually-exclusive --mem-per-cpu and --mem flags respectively, and only one of them is ever set for a given job. Since this allowlist currently keeps only SLURM_MEM_PER_CPU, jobs submitted with a plain --mem allocation will still have their memory setting stripped before the nested srun, reproducing the same OOM regression this PR fixes for --mem-per-cpu jobs.

Proposed fix
             "SLURM_SUBMIT_HOST",
-            "SLURM_MEM_PER_CPU"
+            "SLURM_MEM_PER_CPU",
+            "SLURM_MEM_PER_NODE",
         }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@snakemake_executor_plugin_slurm_jobstep/__init__.py` around lines 208 - 220,
The allowlist in the Slurm environment preservation block currently keeps only
SLURM_MEM_PER_CPU, so jobs submitted with --mem still lose their memory setting
before nested srun. Update the keep_slurm set in __init__.py to also include
SLURM_MEM_PER_NODE alongside the existing memory variable so both mutually
exclusive Slurm memory settings are preserved.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@snakemake_executor_plugin_slurm_jobstep/__init__.py`:
- Around line 208-220: The allowlist in the Slurm environment preservation block
currently keeps only SLURM_MEM_PER_CPU, so jobs submitted with --mem still lose
their memory setting before nested srun. Update the keep_slurm set in
__init__.py to also include SLURM_MEM_PER_NODE alongside the existing memory
variable so both mutually exclusive Slurm memory settings are preserved.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ccc9a18d-6d0e-44b7-8bf7-250f16375444

📥 Commits

Reviewing files that changed from the base of the PR and between c09349d and 6795442.

📒 Files selected for processing (1)
  • snakemake_executor_plugin_slurm_jobstep/__init__.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant