Skip to content

Fix cached task after abort#6903

Closed
jorgee wants to merge 12 commits into
masterfrom
6884-fix-retries-with-abort
Closed

Fix cached task after abort#6903
jorgee wants to merge 12 commits into
masterfrom
6884-fix-retries-with-abort

Conversation

@jorgee

@jorgee jorgee commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Close #6884
Alternative for #6882
This pull request enhances task retry logic in Nextflow by improving how task failures and aborts are tracked and incorporated into cache key calculation. The changes ensure that both failure and abort counts are considered, making task resumption and retry behavior more robust and predictable.

Task retry and cache logic improvements:

  • The hash used for determining task cache keys now includes both failCount and a new abortedCount to ensure that retries after aborts or failures use a distinct hash.
  • When a cached entry is found with a status of FAILED or ABORTED, the corresponding counters (failCount or abortedCount) on the TaskRun object are incremented, ensuring accurate retry attempts and cache key updates.

Task state tracking enhancements:

  • Added a new abortedCount field to the TaskRun class to track the number of times a task execution has been aborted.
  • Introduced isAborted() and isFailed() helper methods to the TraceRecord class for clearer and more maintainable status checks.

TODO:

  • Check attempts
  • Add test to reproduce failure and check fix

… when task previously aborted or retired

Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@netlify

netlify Bot commented Mar 9, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 9e0f93f
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69e61e983b4ab50008993fd9

Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Comment thread modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy Outdated
@jorgee jorgee marked this pull request as ready for review March 10, 2026 08:54
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@vagkaratzas

Copy link
Copy Markdown

Any update on this one? :'D

@jorgee

jorgee commented Apr 15, 2026

Copy link
Copy Markdown
Contributor Author

This PR is ready, just this minor comment not related to the issue.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorgee does this test need to be updated since I removed the extra logic? it is failing for me locally

@jorgee jorgee Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, now the third execution also retries and execution will take more time. Updating it.

@jorgee jorgee Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems there was an inconsistency with failed counts and failed counts. Removing the code, the number of attempts was not increased but failedCounts yes, and it is also used for retries.

I have renamed the abortedCount to previousFailOrAbortedCount. It stores the aborted and failed cached execution. This is just used for increasing the tries

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I think I understand all the changes. The only problem now is that this e2e test continues to fail locally

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to get the log of your e2e test? I would like to know why is it failing in your case.

jorgee and others added 2 commits April 16, 2026 15:23
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>

@pditommaso pditommaso left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add unit tests to capture this behaviour

Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee

jorgee commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

Please add unit tests to capture this behaviour

There was a validation tests checking the full fuctionality. I have added unit tests in the last commit.

@jorgee jorgee requested a review from pditommaso April 20, 2026 12:42
@bentsherman bentsherman deleted the 6884-fix-retries-with-abort branch June 26, 2026 14:21
@bentsherman

Copy link
Copy Markdown
Member

CORRECTION: Closing in favor of #7194

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache fail when specific conditions are met

4 participants