fix: Circumvent problematic Slurm retry behaviour by selten · Pull Request #470 · snakemake/snakemake-executor-plugin-slurm

selten · 2026-06-11T14:15:54Z

Tested to fix #348 combined with another PR for jobstep plugin. This code was generated with help from AI.

Summary by CodeRabbit

Bug Fixes
- Ensured Snakemake job retry/attempt numbers are properly propagated to submitted SLURM jobs, including array jobs and nested job steps.
- Added reliable cleanup so attempt state is reset after job submission.
Tests
- Added automated coverage for attempt inheritance and reset behavior.
- Verified that attempt propagation does not rely on global environment variables.

coderabbitai · 2026-06-11T14:16:09Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f855d983-e4d6-4720-a8fa-7878e396d637

📥 Commits

Reviewing files that changed from the base of the PR and between 1b08596 and 709afb4.

📒 Files selected for processing (1)

tests/test_attempt_sync.py

🚧 Files skipped from review as they are similar to previous changes (1)

tests/test_attempt_sync.py

Walkthrough

The SLURM executor is enhanced to propagate the Snakemake job attempt number into nested jobstep environments. An internal _inherited_attempt field on ExecutorSettings carries the attempt state, which is set from the job's attempt value when run_array_jobs() or run_job() begins submission, and cleared back to zero in finally blocks. The job execution command includes an export SNAKEMAKE_ATTEMPT prefix to make the value available to the submitted process. Tests verify field semantics, set/clear lifecycle, and confirm that attempt values are not exported as environment variables or included in jobstep arguments.

Changes

Attempt Number Propagation

Layer / File(s)	Summary
ExecutorSettings attempt field `snakemake_executor_plugin_slurm/__init__.py`	Adds internal `_inherited_attempt` integer field on `ExecutorSettings` (non-init, non-repr) to carry attempt state.
Attempt propagation in array and single-job submission `snakemake_executor_plugin_slurm/__init__.py`	Sets `workflow.executor_settings._inherited_attempt` from job attempt at the start of array and single-job submission; refactors first-job selection for array chunks; resets `_inherited_attempt` to zero in `finally` blocks for both paths.
Tests for attempt behavior and env usage `tests/test_attempt_sync.py`	New pytest module verifying presence and semantics of `_inherited_attempt` (init=False, non-repr), set/clear lifecycle for `run_job` and `run_array_jobs`, and assertions that attempt values are not exported globally or included in jobstep arguments.

🎯 3 (Moderate) | ⏱️ ~20 minutes

🐰 I hop and watch attempts take flight,
Exported in commands, nested job light,
Tests check the trail left behind,
Retry counts carried so states align. 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 77.78% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: fixing Slurm's problematic retry behavior by tracking attempt numbers and preventing indefinite retries on OOM errors.
Linked Issues check	✅ Passed	The PR implements mechanism to track and pass attempt numbers via SNAKEMAKE_ATTEMPT environment variable, enabling jobstep executor to detect and handle retry attempts properly to address OOM resubmission problem described in `#348`.
Out of Scope Changes check	✅ Passed	All changes are scoped to attempt number tracking and synchronization (new field, environment variable export, tests) which are directly required to address the OOM handling issue in `#348`.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@snakemake_executor_plugin_slurm/__init__.py`:
- Around line 1125-1126: The array submission path fails to export the per-job
retry attempt into the nested execution command; update the run_array_jobs logic
so that when constructing the nested command (the exec_job string) it
prepends/export the SNAKEMAKE_ATTEMPT environment variable the same way run_job
does (use job.attempt to build "export SNAKEMAKE_ATTEMPT={job.attempt};
{exec_job}"). Locate run_array_jobs and the place where exec_job is assembled
and add the export prefix so each array-task inherits the attempt value.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 62424efe-aca5-4ea5-aa79-380e7c660a28

📥 Commits

Reviewing files that changed from the base of the PR and between cdb97d5 and 8c1cffa.

📒 Files selected for processing (1)

snakemake_executor_plugin_slurm/__init__.py

cmeesters · 2026-06-11T15:08:50Z

Thank you for your contribution!

I will need some time to test, as I am busy these days. But I will try to do a preliminary review understanding, that proper tests will fail until the jobstep executor PR is merged, too.

cmeesters

It would be nice to have a test added to the test suite. But I understand, if this is a bit challenging. Without, however, a release (considering the upcoming holidays) might take a bit longer.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@snakemake_executor_plugin_slurm/__init__.py`:
- Around line 1055-1057: Tests in tests/test_attempt_sync.py expect
_inherited_attempt to be None, but the implementation resets
self.workflow.executor_settings._inherited_attempt to 0; update the tests to
match the implementation by replacing assignments of None with 0 and change
assertions that check "is None" to assert equality/identity with 0 (i.e., assert
_inherited_attempt == 0 or assert _inherited_attempt is 0) so the cleanup
behavior and test expectations are consistent with the _inherited_attempt
default value used in the code.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f760ed6e-0505-4f70-8cae-942e51337eb5

📥 Commits

Reviewing files that changed from the base of the PR and between 8c1cffa and 1b08596.

📒 Files selected for processing (2)

snakemake_executor_plugin_slurm/__init__.py
tests/test_attempt_sync.py

cmeesters · 2026-06-22T10:16:30Z

 class ExecutorSettings(ExecutorSettingsBase):
    """Settings for the SLURM executor plugin."""

+    _inherited_attempt: int = field(


I'm afraid, this will translate --slurm--inherited-attempt. The leading underscore does not hide an option.

I want to have hidden options, only for internal use, myself.

Circumvent problematic Slurm submission: snakemake#348

8c1cffa

selten mentioned this pull request Jun 11, 2026

fix: Circumvent problematic Slurm re-submission snakemake/snakemake-executor-plugin-slurm-jobstep#49

Open

coderabbitai Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread snakemake_executor_plugin_slurm/__init__.py Outdated

Update code after AI review

0b3a453

selten changed the title ~~Circumvent problematic Slurm retry behaviour~~ fix: Circumvent problematic Slurm retry behaviour Jun 11, 2026

cmeesters reviewed Jun 11, 2026

View reviewed changes

Comment thread snakemake_executor_plugin_slurm/__init__.py Outdated

Comment thread snakemake_executor_plugin_slurm/__init__.py Outdated

Peter Selten added 3 commits June 12, 2026 13:38

Alternative attempt that avoids Environment Variables

b20c71a

Add tests

9354da6

Fix formatting and linting

1b08596

coderabbitai Bot reviewed Jun 12, 2026

View reviewed changes

Comment thread snakemake_executor_plugin_slurm/__init__.py

Fix test cases based on coderabbit review

709afb4

cmeesters reviewed Jun 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: Circumvent problematic Slurm retry behaviour#470

fix: Circumvent problematic Slurm retry behaviour#470
selten wants to merge 6 commits into
snakemake:mainfrom
selten:348-jobstep-oom

selten commented Jun 11, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 11, 2026 •

edited

Loading

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

cmeesters commented Jun 11, 2026

Uh oh!

cmeesters left a comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

cmeesters Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

selten commented Jun 11, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cmeesters commented Jun 11, 2026

Uh oh!

cmeesters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cmeesters Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

selten commented Jun 11, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 11, 2026 •

edited

Loading