scaleset: fix reapTimedOutRunners deadlock on creating instances#814
Merged
gabriel-samfira merged 1 commit intoJul 7, 2026
Merged
Conversation
reapTimedOutRunners skipped runners in pending_delete/pending_force_delete/
deleting/deleted status, but not creating/pending_create. When a runner's
scale-set-level bootstrap timeout elapsed while its instance was still
being created in the provider (e.g. CloudStack deploy running long due to
host iowait) and it had never joined github, this tried to transition the
instance straight from creating to pending_delete. That transition isn't
allowed (InstanceCreating only permits -> error or -> running), so
UpdateInstance returned a BadRequestError.
That error was treated as fatal: reapTimedOutRunners immediately unlocked
everything it had processed so far and returned early, and its caller
(consolidateRunnerState) aborted the rest of consolidation for that scale
set on any error. This meant a single runner still legitimately booting
(just slow) could block reaping of every other eligible runner in the same
scale set, repeating every consolidate tick (~5 min) until the poisoned
runner resolved on its own via the provider worker's separate create-
timeout handling. Observed in prod: 'invalid instance status transition
from creating to pending_delete' logged repeatedly for the micro scale set
while CloudStack was slow due to iowait.
Fix:
* Skip InstanceCreating/InstancePendingCreate in reapTimedOutRunners,
mirroring the same skip already done later in consolidateRunnerState.
The provider worker owns create timeouts/failures for these instances.
* Don't let one runner's transition error abort the whole batch: log it,
release just that runner's lock, and keep processing the rest instead
of returning early.
gabriel-samfira
approved these changes
Jul 7, 2026
gabriel-samfira
left a comment
Member
There was a problem hiding this comment.
This is great! Thanks!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
reapTimedOutRunnersskipped runners inpending_delete/pending_force_delete/deleting/deletedstatus, but notcreating/pending_create. When a runner's scale-set-level bootstrap timeout elapsed while its instance was still being created in the provider (e.g. a cloud provider deploy running long due to host iowait) and it had never joined GitHub, this tried to transition the instance straight fromcreatingtopending_delete. That transition isn't allowed (InstanceCreatingonly permits-> erroror-> running), soUpdateInstancereturned aBadRequestError.That error was treated as fatal:
reapTimedOutRunnersimmediately unlocked everything it had processed so far and returned early, and its caller (consolidateRunnerState) aborted the rest of consolidation for that scale set on any error. This meant a single runner still legitimately booting (just slow) could block reaping of every other eligible runner in the same scale set, repeating every consolidate tick (~5 min) until the poisoned runner resolved on its own via the provider worker's separate create-timeout handling.Observed in our prod cluster:
invalid instance status transition from creating to pending_deletelogged repeatedly for a scale set while the underlying cloud provider was slow.Fix:
InstanceCreating/InstancePendingCreateinreapTimedOutRunners, mirroring the same skip already done later inconsolidateRunnerState. The provider worker owns create timeouts/failures for these instances.