fix(slurm-controller): add HA single-active guard and defer scontrol …

[andybubu]

Summary

Fixes controller race conditions in Slurm High Availability (HA) setups and eliminates slurm_load_ctl_conf error: Socket timed out during reconfiguration.

This PR introduces an HA Single-Active Guard in slurmsync.py, defers scontrol reconfigure calls to a single pass, and polls daemon readiness (scontrol ping) with exponential backoff.

---

Key Problems Solved

1. Dual-Active Execution in Slurm HA Setups

In dual-controller setups (slurmctld-0 Primary and slurmctld-1 Backup):

• slurmsync.py ran concurrently on both controllers every minute.
• Both controllers issued duplicate GCP Compute API calls (instances.start(), delete_instances(), repair.call_rr_api()), leading to API rate-limit errors (429 Quota Exceeded), race conditions in VM lifecycle management, and corrupted repair tracking.

2. Double Restarts & Socket Timeouts

• When both cloud configuration (cloud.conf) and placement topology (topology.conf) changed in a single sync pass, slurmctld was restarted twice sequentially, tearing down mid-registration compute nodes.
• Executing scontrol reconfigure immediately after systemctl restart caused Socket timed out on send/recv operation on large clusters while slurmctld initialized state.

---

Architectural Changes Introduced

1. Active Controller Detection (util.is_active_controller):
   • Non-HA clusters: always returns True for the single controller.
   • HA clusters: queries scontrol ping to determine if the local instance is Primary Active (-0) or Backup Takeover (-1). Returns False on Standby.
2. Single-Active Guard (slurmsync.py):
   • Both controllers synchronize configuration files (cloud.conf / topology.conf).
   • Only the Active controller executes GCP instance management, node repairs, placement group sync, and scontrol reconfigure. Standby controllers log standby status and exit cleanly.
3. Single Deferred Reconfigure & Polled Readiness:
   • Coalesces config and topology changes into a single util.scontrol_reconfigure() call at the end of slurmsync.py.
   • wait_slurmctld_up() polls scontrol ping with exponential backoff before executing scontrol reconfigure.

---

Modified & Added Files

• community/.../scripts/util.py: Added is_active_controller() and wait_slurmctld_up(); updated scontrol_reconfigure().
• community/.../scripts/slurmsync.py: Added is_active_controller guard inside main(); deferred scontrol_reconfigure() execution.
• community/.../scripts/setup.py: Added check_slurmdbd_ready for slurmdbd readiness check.

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses critical stability issues in Slurm High Availability (HA) deployments by implementing an active-controller guard and optimizing the reconfiguration lifecycle. By ensuring only the active controller manages infrastructure state and deferring service reloads until necessary, the changes eliminate race conditions, API quota exhaustion, and intermittent socket timeouts during cluster synchronization.

Highlights

• HA Single-Active Guard: Introduced util.is_active_controller() to prevent concurrent execution of sync tasks on both primary and backup controllers in HA setups, resolving API rate-limiting and race conditions.
• Deferred Reconfiguration: Coalesced configuration and topology updates into a single scontrol reconfigure pass to prevent redundant restarts and associated socket timeout errors.
• Improved Daemon Readiness: Added wait_slurmctld_up() with exponential backoff and check_slurmdbd_ready() to ensure services are fully initialized before attempting configuration updates.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces several robustness improvements to the Slurm controller setup and synchronization scripts. It adds a polling mechanism to wait for slurmdbd to be ready, defers and consolidates scontrol reconfigure calls for configuration and topology updates to prevent socket timeouts, and adds standby-mode detection for HA controllers. However, a critical issue was identified in the is_active_controller function where the string matching for scontrol ping output checks for "controller" instead of "server", which would cause active controllers to falsely identify as standby and skip essential operations. A code suggestion has been provided to resolve this mismatch.