Enhance Slurm-GCP resilience to capacity constraints and quota errors#5776
Enhance Slurm-GCP resilience to capacity constraints and quota errors#5776vikramvs-gg wants to merge 5 commits into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request improves the resilience of the Slurm-GCP provisioner by implementing smarter error handling and atomic provisioning. By ensuring that job requests are treated as atomic units and providing a mechanism to gracefully handle GCP capacity constraints, the system now avoids wasteful partial node provisioning and reduces unnecessary API load during outages. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
vikramvs-gg
added
enhancement
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new error_handler.py script to classify GCP API errors (such as quota and capacity exhaustion) and handle them gracefully in Slurm. In resume.py, it implements atomic provisioning for job requests by omitting minCount, fixes a bug with duplicate node provisioning by intersecting job nodes with currently resuming nodes, and adds backoff logic for requeued jobs during failures. The review feedback suggests optimizing the job requeue wait logic by batching the checks to prevent sequential blocking delays, and simplifying the datetime and subprocess imports.
be1982f to
075b91c
Compare
075b91c to
ced8489
Compare
|
|
||
| for job in jobs_to_requeue: | ||
| # To prevent the requeued job from immediately retrying and hammering | ||
| # the GCP API during a stockout, we calculate a randomized backoff (e.g. 3-5 mins) and |
There was a problem hiding this comment.
Does this means that the order of jobs running might be shuffled around between initial attempt and the requeue since Slurm will strictly adhere to the new startTime?
There was a problem hiding this comment.
Yes, some job shuffling is expected, but it's an intentional trade-off to prevent a thundering herd effect. The randomized jitter ensures that if multiple jobs fail simultaneously, they don't wake up at the exact same moment and overwhelm the GCP API.
There was a problem hiding this comment.
I don't really understand why the randomness is necessary, we could do a consistent backoff if we want jobs spaced out and to not overwhelm a given API.
Jobs can often times be queued with the intention that they are dependent on the previous job's success. Yes while a user could combine these into a single job script, I think it probably teeters into unexpected behavior if jobs are run out of order of submission unless the reason can be justified.
There was a problem hiding this comment.
If we use a consistent backoff and 50 jobs fail at the same time, they will all wake up at the exact same second later on. That just creates another massive spike of API requests that will probably hit the quota limit again. The randomness is just there to prevent them from syncing up.
As for job ordering, Slurm doesn't actually guarantee submission order anyway since the backfill scheduler runs jobs out of order when nodes free up. If the jobs actually depend on each other, the user has to use dependency flag. If they do that, the jitter doesn't matter because that keeps the next job blocked until the first one finishes.
There was a problem hiding this comment.
Was thinking more so linear rather than consistent. It is true Slurm doesn't make many guarantees over submission order, however I am concerned about arbitrary queue fragmentation.
Same idea with priority inversion, low priority jobs running before high priority jobs because of an unfavorable number pull.
I understand these are trade-offs you may be willing to make, but if I could suggest a few items:
-
Some sort of guide/notice/documentation update about this behavior somewhere, it appears to not be opt in so it would be good if we had somewhere we could point to regarding why this behavior may occur. We can do some sort of follow up for this part.
-
Could we narrow the 3-5 minute window? Maybe closer to 1 minute or less?
-
Can we potentially preserve user intention around priority weighting? As you mention backfill will attempt to run jobs whenever it can. Maybe we can extract the priority information about the job and use that as part of the backoff calculation?
-
What about heterogenous jobs/job arrays. I think this may cause some issues for such cases. For array jobs, this result in more ResumeProgram runs than initially expected right (due to the job scattering)?
It's good that this is directly connected to cases where we can match the error to quota exhaustion , I think if we can tighten the extremities a bit the drawbacks can be minimized.
There was a problem hiding this comment.
+1 on point 2nd suggestions. Narrowing the backoff window to 60–120s (1–2 minutes) feels like a great middle ground here.
It keeps enough jitter to protect the GCP Compute API from a thundering herd of retries, while addressing the queue delay and priority inversion concerns raised above.
There was a problem hiding this comment.
- Added a note on this behaviour in
docs/slurm-troubleshooting.md. - Yes, narrowed it down to 1-2 minutes.
- If we take priority into calculating jitter, and if there are multiple huge high priority tasks, they will all receive a shorter jitter and may cause starvation for other jobs, and may even cause a thundering herd problem. The scope is to just prevent this by adding random jitter, As we have reduced the delay window, it will not keep jobs waiting in queue for too long.
- Yes it would have caused multiple runs, I have updated the logic to fetch the array/pack id and use it as a seed value for generating random jitter, so that all jobs in an array have same jitter and also added batching to reduce number of queries/updates.
| # the GCP API during a stockout, we calculate a randomized backoff (e.g. 3-5 mins) and | ||
| # explicitly delay the job's next evaluation via the StartTime parameter. | ||
| delay_seconds = random.randint(180, 300) | ||
| backoff_time = (datetime.now() + timedelta(seconds=delay_seconds)).strftime("%Y-%m-%dT%H:%M:%S") |
There was a problem hiding this comment.
Per the design doc, sbatch jobs should requeue with backoff, but interactive srun jobs should fail fast to the user terminal instead of hanging on a delayed StartTime.
Can we check batch_flag == 0 and skip the StartTime backoff for interactive jobs?
There was a problem hiding this comment.
When an srun job fails and error handler requests to requeue, Slurm instantly kills it instead of putting it back in PENDING. As we are only checking jobs in PENDING state, interactive jobs skip the backoff and fail fast.
4 participants
Description
This PR makes the Slurm-GCP provisioner resilient to GCP capacity constraints (stockouts) and quota limitations. It prevents wasted budget from partially provisioned jobs and handles failures gracefully by pushing jobs back into the queue without wedging the partition.
Key Changes
minCountomission):resume.pynow omitsminCountduring thebulkInsertcall for job-triggered requests. This enforces an "All or Nothing" provisioning behavior from GCP for job requests, entirely preventing the "partially provisioned" nodes issue that wastes budget while jobs hang.error_handler.py):ZONE_RESOURCE_POOL_EXHAUSTED,QUOTA_EXCEEDED).AdminCommentfor immediate user visibility.DOWNforever on a capacity error,resume.pyexecutes a "Double Update" (state=downfollowed bystate=power_down) to instantly clear thePOWERING_UPstate, bypassing Slurm's rigidResumeTimeout.PENDING, and itsStartTimeis pushed back with a randomized jitter (3-5 minutes) to prevent aggressive API hammering during a prolonged stockout.