[module_base]: Add 'recovery' transition type to _TRANSITION_TIMEOUT_DEFAULTS#709
Open
vvolam wants to merge 1 commit into
Open
[module_base]: Add 'recovery' transition type to _TRANSITION_TIMEOUT_DEFAULTS#709vvolam wants to merge 1 commit into
vvolam wants to merge 1 commit into
Conversation
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…DEFAULTS
chassisd's DPU power-cycle auto-recovery acquires the module state-transition lock via set_module_state_transition(name, 'recovery'). Since 'recovery' was absent from _TRANSITION_TIMEOUT_DEFAULTS, the transition was rejected ("Cannot acquire state transition lock") and the power-cycle was silently skipped, blocking DPU recovery.
Add "recovery": 600 (10 min) so the recovery lock is accepted and chassisd can power-cycle an unresponsive DPU.
Verified on-device: with the fix, a control-plane-down DPU was power-cycled (reboot-cause "Switch rebooted DPU") and recovered to ready_status=true.
Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add a
"recovery": 600(10 min) entry toModuleBase._TRANSITION_TIMEOUT_DEFAULTS.set_module_state_transition()validates the requestedtransition_typeagainst the keys of_TRANSITION_TIMEOUT_DEFAULTS. chassisd's DPU power-cycle auto-recovery acquires the module state-transition lock viaset_module_state_transition(name, 'recovery'); because'recovery'was missing from the defaults, every recovery acquisition was rejected ("Cannot acquire state transition lock") and the power-cycle was silently skipped — blocking DPU auto-recovery.Motivation and Context
The
'recovery'transition type is used by chassisd (sonic-platform-daemons) to power-cycle an unresponsive DPU. Without a matching entry in_TRANSITION_TIMEOUT_DEFAULTS, the transition is never accepted, so a DPU that loses its control plane is never power-cycled. This unblocks the DPU auto-recovery path in sonic-net/sonic-platform-daemons#829.recoveryhas noplatform.jsonoverride (onlystartup/shutdown/reboot/halt_servicesare overridable in_load_transition_timeouts()), so the 600 s default applies uniformly. The timeout only bounds the power-cycle critical section (pre_shutdown → admin down → admin up → post_startup), which is cleared infinally; 600 s is ample headroom.How Has This Been Tested?
tests/module_base_test.pyto expect the new key.python3 -m pytest tests/module_base_test.py→ 115 passed.ready_status=true.Additional Information (Optional)
Companion to sonic-net/sonic-platform-daemons#829 (chassisd DPU auto-recovery robustness), which issues the
'recovery'transition this default enables.