Skip to content

Commit a3329c9

Browse files
francesco-furlanfrancesco-furlan
authored
fix(shield): prevent stuck allowlist-waiter Job after hook failure (#2619)
1 parent 2301e01 commit a3329c9

8 files changed

Lines changed: 60 additions & 13 deletions

‎charts/shield/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,5 +13,5 @@ maintainers:
1313
- name: mavimo
1414
email: marcovito.moscaritolo@sysdig.com
1515
type: application
16-
version: 1.39.0
16+
version: 1.40.0
1717
appVersion: "1.0.0"

‎charts/shield/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,7 @@ The following table lists the configurable parameters of the `shield` chart and
303303
| gke_autopilot.allowlist_version | The Allowlist version label applied to host-shield workloads. Must match an AllowlistSynchronizer the cluster knows about. (Replaces the deprecated top-level "gke_autopilot_allowlist", which is still honored and takes precedence when set.) | <code>sysdig-agent-v1.1.4</code> |
304304
| gke_autopilot.allowlist_waiter.enabled | Enable the waiter Job | <code>false</code> |
305305
| gke_autopilot.allowlist_waiter.timeout | Maximum time the Job will block on the AllowlistSynchronizer reaching Ready | <code>120s</code> |
306+
| gke_autopilot.allowlist_waiter.active_deadline_seconds | Maximum seconds the waiter Pod is allowed to run before Kubernetes terminates it. Acts as a Job-level guard against the pod hanging before the inner `kubectl wait` timeout fires (image-pull stalls, scheduler delays, admission webhook hangs). Should be greater than `timeout` to leave headroom for pod startup. | <code>300</code> |
306307
| gke_autopilot.allowlist_waiter.service_account_name | Override the name of the waiter ServiceAccount (defaults to <host fullname>-allowlist-waiter) | <code></code> |
307308
| gke_autopilot.allowlist_waiter.rbac.create | Create the RBAC resources for the waiter Job. Acts as the default for the sub-toggles below; set any sub-toggle explicitly to override it. | <code>true</code> |
308309
| gke_autopilot.allowlist_waiter.rbac.service_account.create | Override `rbac.create` for the waiter ServiceAccount only. | <code></code> |

‎charts/shield/templates/host/gke-allowlist-waiter-clusterrole.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ metadata:
99
annotations:
1010
helm.sh/hook: "pre-install,pre-upgrade"
1111
helm.sh/hook-weight: "-5"
12-
helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded"
12+
helm.sh/hook-delete-policy: "before-hook-creation"
1313
{{- with .Values.host.rbac.annotations }}
1414
{{- toYaml . | nindent 4 }}
1515
{{- end }}

‎charts/shield/templates/host/gke-allowlist-waiter-clusterrolebinding.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ metadata:
99
annotations:
1010
helm.sh/hook: "pre-install,pre-upgrade"
1111
helm.sh/hook-weight: "-5"
12-
helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded"
12+
helm.sh/hook-delete-policy: "before-hook-creation"
1313
{{- with .Values.host.rbac.annotations }}
1414
{{- toYaml . | nindent 4 }}
1515
{{- end }}

‎charts/shield/templates/host/gke-allowlist-waiter-job.yaml‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,10 @@ metadata:
99
annotations:
1010
helm.sh/hook: "pre-install,pre-upgrade"
1111
helm.sh/hook-weight: "5"
12-
helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded"
12+
helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded,hook-failed"
1313
spec:
1414
backoffLimit: 3
15+
activeDeadlineSeconds: {{ .Values.gke_autopilot.allowlist_waiter.active_deadline_seconds }}
1516
template:
1617
metadata:
1718
name: {{ include "host.allowlist_waiter.fullname" . }}
@@ -35,12 +36,26 @@ spec:
3536
- /bin/bash
3637
- -c
3738
- |
38-
set -euo pipefail
39-
echo "Waiting for AllowlistSynchronizer/sysdig-agent-allowlist-synchronizer to become Ready..."
39+
set -uo pipefail
40+
NAMESPACE={{ .Release.Namespace }}
41+
SYNC_NAME=sysdig-agent-allowlist-synchronizer
42+
TIMEOUT={{ .Values.gke_autopilot.allowlist_waiter.timeout }}
43+
44+
echo "Waiting for AllowlistSynchronizer/$SYNC_NAME to become Ready (timeout $TIMEOUT)..."
4045
kubectl wait --for=condition=Ready \
41-
allowlistsynchronizer/sysdig-agent-allowlist-synchronizer \
42-
-n {{ .Release.Namespace }} \
43-
--timeout={{ .Values.gke_autopilot.allowlist_waiter.timeout }}
46+
allowlistsynchronizer/"$SYNC_NAME" \
47+
-n "$NAMESPACE" \
48+
--timeout="$TIMEOUT"
49+
ec=$?
50+
51+
if [ "$ec" -ne 0 ]; then
52+
echo "---" >&2
53+
echo "kubectl wait failed (exit $ec). Dumping AllowlistSynchronizer state for diagnostics:" >&2
54+
kubectl describe -n "$NAMESPACE" allowlistsynchronizer/"$SYNC_NAME" >&2 || true
55+
echo "---" >&2
56+
kubectl get -n "$NAMESPACE" allowlistsynchronizer/"$SYNC_NAME" -o yaml >&2 || true
57+
exit "$ec"
58+
fi
4459
echo "AllowlistSynchronizer is Ready."
4560
resources:
4661
{{- toYaml .Values.gke_autopilot.allowlist_waiter.resources | nindent 12 }}

‎charts/shield/templates/host/gke-allowlist-waiter-serviceaccount.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ metadata:
1010
annotations:
1111
helm.sh/hook: "pre-install,pre-upgrade"
1212
helm.sh/hook-weight: "-5"
13-
helm.sh/hook-delete-policy: "before-hook-creation,hook-succeeded"
13+
helm.sh/hook-delete-policy: "before-hook-creation"
1414
{{- with .Values.host.rbac.annotations }}
1515
{{- toYaml . | nindent 4 }}
1616
{{- end }}

‎charts/shield/tests/host/gke-allowlist-synchronizer_test.yaml‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ tests:
176176
value: "-5"
177177
- equal:
178178
path: metadata.annotations["helm.sh/hook-delete-policy"]
179-
value: before-hook-creation,hook-succeeded
179+
value: before-hook-creation
180180
template: templates/host/gke-allowlist-waiter-serviceaccount.yaml
181181

182182
- it: Renders the waiter ClusterRole limited to allowlistsynchronizers
@@ -193,6 +193,9 @@ tests:
193193
- equal:
194194
path: metadata.annotations["helm.sh/hook-weight"]
195195
value: "-5"
196+
- equal:
197+
path: metadata.annotations["helm.sh/hook-delete-policy"]
198+
value: before-hook-creation
196199
- contains:
197200
path: rules
198201
content:
@@ -220,6 +223,9 @@ tests:
220223
- equal:
221224
path: metadata.annotations["helm.sh/hook-weight"]
222225
value: "-5"
226+
- equal:
227+
path: metadata.annotations["helm.sh/hook-delete-policy"]
228+
value: before-hook-creation
223229
- equal:
224230
path: roleRef.kind
225231
value: ClusterRole
@@ -256,7 +262,7 @@ tests:
256262
value: "5"
257263
- equal:
258264
path: metadata.annotations["helm.sh/hook-delete-policy"]
259-
value: before-hook-creation,hook-succeeded
265+
value: before-hook-creation,hook-succeeded,hook-failed
260266
- equal:
261267
path: spec.template.spec.serviceAccountName
262268
value: release-name-shield-host-allowlist-waiter
@@ -268,7 +274,17 @@ tests:
268274
pattern: "^quay\\.io/sysdig/kubectl:[^\\s]+$"
269275
- matchRegex:
270276
path: spec.template.spec.containers[0].command[2]
271-
pattern: "--timeout=120s"
277+
pattern: "TIMEOUT=120s"
278+
# The waiter script must dump AllowlistSynchronizer state on wait failure
279+
# so the next on-caller has actionable diagnostics instead of a bare exit.
280+
- matchRegex:
281+
path: spec.template.spec.containers[0].command[2]
282+
pattern: "kubectl describe .* allowlistsynchronizer"
283+
# Job-level guard against pod hangs before the inner `kubectl wait` timeout fires
284+
# (image-pull stalls, scheduler delays, admission webhook hangs).
285+
- equal:
286+
path: spec.activeDeadlineSeconds
287+
value: 300
272288
# The waiter Pod must NOT carry cloud.google.com/matching-allowlist —
273289
# otherwise GKE Autopilot would block it on the very allowlist it waits to load.
274290
- notExists:
@@ -277,6 +293,16 @@ tests:
277293
path: spec.template.metadata.labels["autopilot.gke.io/no-connect"]
278294
template: templates/host/gke-allowlist-waiter-job.yaml
279295

296+
- it: Honors gke_autopilot.allowlist_waiter.active_deadline_seconds override
297+
set:
298+
gke_autopilot.allowlist_waiter.enabled: true
299+
gke_autopilot.allowlist_waiter.active_deadline_seconds: 600
300+
asserts:
301+
- equal:
302+
path: spec.activeDeadlineSeconds
303+
value: 600
304+
template: templates/host/gke-allowlist-waiter-job.yaml
305+
280306
- it: Renders imagePullSecrets when gke_autopilot.allowlist_waiter.image.pull_secrets is set
281307
set:
282308
gke_autopilot.allowlist_waiter.enabled: true

‎charts/shield/values.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -707,6 +707,11 @@ gke_autopilot:
707707
enabled: false
708708
# Maximum time the Job will block on the AllowlistSynchronizer reaching Ready
709709
timeout: 120s
710+
# Maximum seconds the waiter Pod is allowed to run before Kubernetes terminates it.
711+
# Acts as a Job-level guard against the pod hanging before the inner `kubectl wait`
712+
# timeout fires (image-pull stalls, scheduler delays, admission webhook hangs).
713+
# Should be greater than `timeout` to leave headroom for pod startup.
714+
active_deadline_seconds: 300
710715
# Override the name of the waiter ServiceAccount (defaults to <host fullname>-allowlist-waiter)
711716
service_account_name:
712717
# RBAC controls for the waiter Job. Independent of `host.rbac.*` so the waiter can be

0 commit comments

Comments
 (0)