Skip to content

Handle stuck jobs on Running phase. Fix scheduling cleanup for c2d jobs orphans#989

Merged
mariacarmina merged 10 commits into
mainfrom
bug/c2d-cleanup
Jul 29, 2025
Merged

Handle stuck jobs on Running phase. Fix scheduling cleanup for c2d jobs orphans#989
mariacarmina merged 10 commits into
mainfrom
bug/c2d-cleanup

Conversation

@mariacarmina

@mariacarmina mariacarmina commented Jul 19, 2025

Copy link
Copy Markdown
Contributor

Fixes #988 #980 .

Changes proposed in this PR:

  • fix node crash error when unexisting container was trying to be fetched by c2d monitor loop without deleting sqlite before -> in "RunningAlgorithm" phase, added try-catch and set job status back to ConfiguringVolumes step to re-create the container and resume the job lifecycle
  • fixed: if the algorithm job gets stuck in "PublishingResults" phase -> delete the job from sqlite
  • surround with try-catch scheduler operations of deleting old logs & deleting expired and orphans c2d jobs
  • fixed cleanup orphans jobs: OCEANNODE: Error deleting expired jobs: Cannot read properties of undefined (reading 'id') by passing environment as parameter to getFinishedJobs()
Screenshot 2025-07-19 at 13 56 27

Comment thread src/components/database/C2DDatabase.ts
Comment thread src/components/c2d/compute_engine_docker.ts Outdated

@alexcos20 alexcos20 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@mariacarmina
mariacarmina merged commit fbb04a1 into main Jul 29, 2025
13 checks passed
@mariacarmina
mariacarmina deleted the bug/c2d-cleanup branch July 29, 2025 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scheduler for expired c2d jobs cleanup throws error when invoked

2 participants