feat: expose health_check_type for runner manager ASG#1384
Open
lorenzstorm1 wants to merge 1 commit into
Open
Conversation
…er manager ASG The runner manager ASG currently hardcodes health_check_grace_period=0 and uses the default EC2 health check type. This means OS-level freezes (where userspace hangs but EC2 status checks still pass) are not detected by the ASG. By exposing these fields, users can attach a target group with an application-level health check (e.g. the gitlab-runner Prometheus endpoint on port 9252) and set health_check_type="ELB" to enable automatic replacement of unresponsive manager instances. Defaults remain unchanged (EC2 health check, 0s grace period).
Contributor
|
Hey @lorenzstorm1! 👋 Thank you for your contribution to the project. Please refer to the contribution rules for a quick overview of the process. Make sure that this PR clearly explains:
With submitting this PR you confirm that you hold the rights of the code added and agree that it will published under this LICENSE. The following ChatOps commands are supported:
Simply add a comment with the command in the first line. If you need to pass more information, separate it with a blank line from the command. This message was generated automatically. You are welcome to improve it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
health_check_typeandhealth_check_grace_periodto therunner_instancevariableaws_autoscaling_group.gitlab_runner_instanceresource"EC2"health check type,0grace periodMotivation
The runner manager ASG uses EC2 health checks, which only detect hardware-level failures. OS-level freezes (OOM, kernel panic that doesn't trigger instance stop, EBS I/O stalls) leave the instance appearing healthy from the hypervisor's perspective while all userspace processes are dead.
With this change, users can attach a target group pointing at the gitlab-runner Prometheus endpoint (
:9252/metrics) and sethealth_check_type = "ELB"to get automatic replacement of unresponsive manager instances.Usage
Combined with an ALB/NLB target group attached to the ASG.
Test plan
terraform planshows no changes when using defaults ("EC2",0)health_check_type = "ELB"propagates to the ASG resource🤖 Generated with Claude Code