Skip to content

Commit eab7e0a

Browse files
CoderYellowCoderYellow
committed
feat(helm): add ServiceAccount, RBAC, and external LoadBalancer auto-discovery for Kafka brokers
1 parent 640b0f9 commit eab7e0a

7 files changed

Lines changed: 162 additions & 2 deletions

File tree

‎kafka/README.md‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,31 +51,47 @@ Kafka external access needs one stable address per broker. Enable per-broker Loa
5151
kafka:
5252
externalBrokerServices:
5353
enabled: true
54+
```
55+
56+
By default, the chart creates a ServiceAccount plus namespaced RBAC, then each broker initContainer waits for its own LoadBalancer Service address and uses it as the external advertised listener. This avoids the install-then-upgrade flow for dynamically assigned LoadBalancer IPs.
57+
58+
If you want to disable Kubernetes API discovery and provide static advertised hosts yourself:
59+
60+
```yaml
61+
kafka:
62+
externalBrokerServices:
63+
enabled: true
64+
autoDiscovery:
65+
enabled: false
5466
advertisedHosts:
5567
- 203.0.113.10
5668
- 203.0.113.11
5769
```
5870

59-
For multi-cluster mode, configure the list per cluster:
71+
For multi-cluster mode, configure static lists per cluster only when `autoDiscovery.enabled=false`:
6072

6173
```yaml
6274
kafka:
6375
clusters:
6476
- name: primary
6577
clusterId: MkU3OEVBNTcwNTJENDM2Qk
6678
externalBrokerServices:
79+
autoDiscovery:
80+
enabled: false
6781
advertisedHosts:
6882
- 203.0.113.10
6983
- 203.0.113.11
7084
- name: standby
7185
clusterId: zlFiTJelTOuhnklFwLWixw
7286
externalBrokerServices:
87+
autoDiscovery:
88+
enabled: false
7389
advertisedHosts:
7490
- 203.0.113.12
7591
- 203.0.113.13
7692
```
7793

78-
If your cloud provider assigns IPs dynamically, install once with external services enabled, read the assigned addresses, put those addresses into `advertisedHosts`, then run `helm upgrade`. For production, prefer reserving static IPs and setting `loadBalancerIPs` plus matching `advertisedHosts` up front when your provider supports `loadBalancerIP`.
94+
For production, static LoadBalancer IPs are still preferable when your provider supports them. Set `loadBalancerIPs` to reserve/request the service addresses. You can keep auto discovery enabled so brokers still read the actual assigned addresses at startup.
7995

8096
## User-managed SASL secrets
8197

‎kafka/templates/NOTES.txt‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,15 @@ Internal bootstrap endpoints:
1313
{{- end }}
1414

1515
{{- $globalExternal := .Values.kafka.externalBrokerServices.enabled }}
16+
{{- $globalDiscovery := .Values.kafka.externalBrokerServices.autoDiscovery.enabled }}
1617
{{- $hasExternal := false }}
18+
{{- $hasStaticExternal := false }}
1719
{{- range $cluster := $clusters }}
1820
{{- if (default $globalExternal (dig "externalBrokerServices" "enabled" nil $cluster)) }}
1921
{{- $hasExternal = true }}
22+
{{- if not (default $globalDiscovery (dig "externalBrokerServices" "autoDiscovery" "enabled" nil $cluster)) }}
23+
{{- $hasStaticExternal = true }}
24+
{{- end }}
2025
{{- end }}
2126
{{- end }}
2227

@@ -39,6 +44,10 @@ External broker LoadBalancer services:
3944
Check assigned LoadBalancer IPs/hostnames:
4045
kubectl -n {{ $namespace }} get svc -l app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=broker -o wide
4146

47+
{{- if not $hasStaticExternal }}
48+
Broker external advertised listener discovery is enabled. Each broker waits for its
49+
own LoadBalancer Service address at startup and advertises that address automatically.
50+
{{- else }}
4251
If your LoadBalancer IPs were assigned dynamically, copy this upgrade command after the
4352
external services show an EXTERNAL-IP/hostname. It updates Kafka advertised.listeners
4453
to match the assigned per-broker addresses.
@@ -72,3 +81,4 @@ For production, prefer reserving static LoadBalancer IPs and setting both
7281
externalBrokerServices.loadBalancerIPs and externalBrokerServices.advertisedHosts
7382
before the first install.
7483
{{- end }}
84+
{{- end }}

‎kafka/templates/_helpers.tpl‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,14 @@ app.kubernetes.io/instance: {{ .Release.Name }}
3030
{{- default "cluster.local" .Values.clusterDomain -}}
3131
{{- end -}}
3232

33+
{{- define "kafka.serviceAccountName" -}}
34+
{{- if .Values.serviceAccount.create -}}
35+
{{- default (include "kafka.fullname" .) .Values.serviceAccount.name -}}
36+
{{- else -}}
37+
{{- default "default" .Values.serviceAccount.name -}}
38+
{{- end -}}
39+
{{- end -}}
40+
3341
{{- define "kafka.headlessServiceName" -}}
3442
{{- default (printf "%s-headless" (include "kafka.fullname" .)) .Values.kafka.headlessService.nameOverride -}}
3543
{{- end -}}

‎kafka/templates/kafka-statefulset.yaml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,12 @@
2626
{{- $externalSourceRanges := default $root.Values.kafka.externalBrokerServices.loadBalancerSourceRanges (dig "externalBrokerServices" "loadBalancerSourceRanges" nil $cluster) -}}
2727
{{- $externalLBIPs := default $root.Values.kafka.externalBrokerServices.loadBalancerIPs (dig "externalBrokerServices" "loadBalancerIPs" nil $cluster) -}}
2828
{{- $externalAdvertisedHosts := default $root.Values.kafka.externalBrokerServices.advertisedHosts (dig "externalBrokerServices" "advertisedHosts" nil $cluster) -}}
29+
{{- $externalAutoDiscoveryEnabled := default $root.Values.kafka.externalBrokerServices.autoDiscovery.enabled (dig "externalBrokerServices" "autoDiscovery" "enabled" nil $cluster) -}}
30+
{{- $externalAutoDiscoveryImage := default $root.Values.kafka.externalBrokerServices.autoDiscovery.image.repository (dig "externalBrokerServices" "autoDiscovery" "image" "repository" nil $cluster) -}}
31+
{{- $externalAutoDiscoveryTag := default $root.Values.kafka.externalBrokerServices.autoDiscovery.image.tag (dig "externalBrokerServices" "autoDiscovery" "image" "tag" nil $cluster) -}}
32+
{{- $externalAutoDiscoveryPullPolicy := default $root.Values.kafka.externalBrokerServices.autoDiscovery.image.pullPolicy (dig "externalBrokerServices" "autoDiscovery" "image" "pullPolicy" nil $cluster) -}}
33+
{{- $externalAutoDiscoveryTimeout := int (default $root.Values.kafka.externalBrokerServices.autoDiscovery.timeoutSeconds (dig "externalBrokerServices" "autoDiscovery" "timeoutSeconds" nil $cluster)) -}}
34+
{{- $externalAutoDiscoveryPollInterval := int (default $root.Values.kafka.externalBrokerServices.autoDiscovery.pollIntervalSeconds (dig "externalBrokerServices" "autoDiscovery" "pollIntervalSeconds" nil $cluster)) -}}
2935
{{- $controllerName := default $root.Values.kafka.ports.controller.name (dig "ports" "controller" "name" nil $cluster) -}}
3036
{{- $controllerPort := int (default $root.Values.kafka.ports.controller.containerPort (dig "ports" "controller" "containerPort" nil $cluster)) -}}
3137
{{- $controllerServicePort := int (default $root.Values.kafka.ports.controller.servicePort (dig "ports" "controller" "servicePort" nil $cluster)) -}}
@@ -176,6 +182,39 @@ spec:
176182
securityContext:
177183
{{- toYaml . | nindent 8 }}
178184
{{- end }}
185+
serviceAccountName: {{ include "kafka.serviceAccountName" $root }}
186+
automountServiceAccountToken: {{ $root.Values.serviceAccount.automountServiceAccountToken }}
187+
{{- if and $externalServicesEnabled $externalAutoDiscoveryEnabled }}
188+
initContainers:
189+
- name: external-address-discovery
190+
image: "{{ $externalAutoDiscoveryImage }}:{{ $externalAutoDiscoveryTag }}"
191+
imagePullPolicy: {{ $externalAutoDiscoveryPullPolicy }}
192+
command:
193+
- /bin/sh
194+
- -ec
195+
- |
196+
ORDINAL="${HOSTNAME##*-}"
197+
SERVICE_NAME="{{ $clusterName }}-${ORDINAL}-external"
198+
NAMESPACE="$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)"
199+
ATTEMPTS=$((({{ $externalAutoDiscoveryTimeout }} + {{ $externalAutoDiscoveryPollInterval }} - 1) / {{ $externalAutoDiscoveryPollInterval }}))
200+
i=1
201+
while [ "${i}" -le "${ATTEMPTS}" ]; do
202+
EXTERNAL_HOST="$(kubectl -n "${NAMESPACE}" get svc "${SERVICE_NAME}" -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}' 2>/dev/null || true)"
203+
if [ -n "${EXTERNAL_HOST}" ]; then
204+
printf '%s' "${EXTERNAL_HOST}" > /broker-external/external-host
205+
echo "Discovered ${SERVICE_NAME} external address: ${EXTERNAL_HOST}"
206+
exit 0
207+
fi
208+
echo "Waiting for ${SERVICE_NAME} LoadBalancer external address"
209+
i=$((i + 1))
210+
sleep {{ $externalAutoDiscoveryPollInterval }}
211+
done
212+
echo "Timed out waiting for ${SERVICE_NAME} LoadBalancer external address" >&2
213+
exit 1
214+
volumeMounts:
215+
- name: broker-external
216+
mountPath: /broker-external
217+
{{- end }}
179218
containers:
180219
- name: kafka
181220
image: "{{ $image }}:{{ $tag }}"
@@ -194,13 +233,21 @@ spec:
194233
export KAFKA_CONTROLLER_LISTENER_NAMES="CONTROLLER"
195234
export KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="{{ default $root.Values.kafka.listenerSecurityProtocolMap $cluster.listenerSecurityProtocolMap }}{{- if $externalServicesEnabled }},{{ $externalListenerName }}:SASL_PLAINTEXT{{- end }}"
196235
{{- if $externalServicesEnabled }}
236+
{{- if $externalAutoDiscoveryEnabled }}
237+
EXTERNAL_HOST="$(cat /broker-external/external-host 2>/dev/null || true)"
238+
if [ -z "${EXTERNAL_HOST}" ]; then
239+
echo "Missing discovered external address for ${HOSTNAME}" >&2
240+
exit 1
241+
fi
242+
{{- else }}
197243
EXTERNAL_ADVERTISED_HOSTS="{{ join "," $externalAdvertisedHosts }}"
198244
EXTERNAL_HOST="$(echo "${EXTERNAL_ADVERTISED_HOSTS}" | cut -d, -f$((ORDINAL + 1)))"
199245
if [ -z "${EXTERNAL_HOST}" ]; then
200246
echo "Missing kafka.externalBrokerServices.advertisedHosts entry for ${HOSTNAME}" >&2
201247
exit 1
202248
fi
203249
{{- end }}
250+
{{- end }}
204251
export KAFKA_LISTENERS="SASL_PLAINTEXT://0.0.0.0:{{ $clientPort }}{{- if $externalServicesEnabled }},{{ $externalListenerName }}://0.0.0.0:{{ $externalContainerPort }}{{- end }},CONTROLLER://${POD_FQDN}:{{ $controllerPort }}"
205252
export KAFKA_ADVERTISED_LISTENERS="SASL_PLAINTEXT://${POD_FQDN}:{{ $clientPort }}{{- if $externalServicesEnabled }},{{ $externalListenerName }}://${EXTERNAL_HOST}:{{ $externalServicePort }}{{- end }}"
206253
export KAFKA_INTER_BROKER_LISTENER_NAME="{{ default $root.Values.kafka.interBrokerListenerName $cluster.interBrokerListenerName }}"
@@ -281,6 +328,10 @@ spec:
281328
{{- with (concat (default list $root.Values.kafka.extraVolumeMounts) (default list $cluster.extraVolumeMounts)) }}
282329
{{- toYaml . | nindent 12 }}
283330
{{- end }}
331+
{{- if and $externalServicesEnabled $externalAutoDiscoveryEnabled }}
332+
- name: broker-external
333+
mountPath: /broker-external
334+
{{- end }}
284335
{{- with (default $root.Values.kafka.containerSecurityContext $cluster.containerSecurityContext) }}
285336
securityContext:
286337
{{- toYaml . | nindent 12 }}
@@ -302,6 +353,10 @@ spec:
302353
{{- toYaml . | nindent 8 }}
303354
{{- end }}
304355
volumes:
356+
{{- if and $externalServicesEnabled $externalAutoDiscoveryEnabled }}
357+
- name: broker-external
358+
emptyDir: {}
359+
{{- end }}
305360
{{- if not (default $root.Values.kafka.persistence.enabled (dig "persistence" "enabled" nil $cluster)) }}
306361
- name: data
307362
emptyDir:

‎kafka/templates/rbac.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
{{- $root := . -}}
2+
{{- $clusters := include "kafka.effectiveClusters" . | fromYamlArray -}}
3+
{{- $globalExternal := .Values.kafka.externalBrokerServices.enabled -}}
4+
{{- $globalDiscovery := .Values.kafka.externalBrokerServices.autoDiscovery.enabled -}}
5+
{{- $needsRbac := false -}}
6+
{{- range $cluster := $clusters -}}
7+
{{- $externalEnabled := default $globalExternal (dig "externalBrokerServices" "enabled" nil $cluster) -}}
8+
{{- $discoveryEnabled := default $globalDiscovery (dig "externalBrokerServices" "autoDiscovery" "enabled" nil $cluster) -}}
9+
{{- if and $externalEnabled $discoveryEnabled -}}
10+
{{- $needsRbac = true -}}
11+
{{- end -}}
12+
{{- end -}}
13+
{{- if and .Values.rbac.create $needsRbac }}
14+
apiVersion: rbac.authorization.k8s.io/v1
15+
kind: Role
16+
metadata:
17+
name: {{ include "kafka.fullname" . }}-broker-services
18+
labels:
19+
{{- include "kafka.labels" . | nindent 4 }}
20+
rules:
21+
- apiGroups: [""]
22+
resources: ["services"]
23+
verbs: ["get", "list", "watch"]
24+
---
25+
apiVersion: rbac.authorization.k8s.io/v1
26+
kind: RoleBinding
27+
metadata:
28+
name: {{ include "kafka.fullname" . }}-broker-services
29+
labels:
30+
{{- include "kafka.labels" . | nindent 4 }}
31+
roleRef:
32+
apiGroup: rbac.authorization.k8s.io
33+
kind: Role
34+
name: {{ include "kafka.fullname" . }}-broker-services
35+
subjects:
36+
- kind: ServiceAccount
37+
name: {{ include "kafka.serviceAccountName" . }}
38+
namespace: {{ include "kafka.namespace" . }}
39+
{{- end }}

‎kafka/templates/serviceaccount.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{{- if .Values.serviceAccount.create }}
2+
apiVersion: v1
3+
kind: ServiceAccount
4+
metadata:
5+
name: {{ include "kafka.serviceAccountName" . }}
6+
labels:
7+
{{- include "kafka.labels" . | nindent 4 }}
8+
{{- with .Values.serviceAccount.annotations }}
9+
annotations:
10+
{{- toYaml . | nindent 4 }}
11+
{{- end }}
12+
automountServiceAccountToken: {{ .Values.serviceAccount.automountServiceAccountToken }}
13+
{{- end }}

‎kafka/values.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,15 @@ nodeSelector: {}
1010
tolerations: []
1111
affinity: {}
1212

13+
serviceAccount:
14+
create: true
15+
name: ""
16+
annotations: {}
17+
automountServiceAccountToken: true
18+
19+
rbac:
20+
create: true
21+
1322
kerberos:
1423
enabled: true
1524
existingSecret: ""
@@ -99,6 +108,16 @@ kafka:
99108
containerPort: 19092
100109
protocol: TCP
101110
listenerName: EXTERNAL
111+
autoDiscovery:
112+
# When enabled, each broker initContainer waits for its own LoadBalancer
113+
# Service address and uses that as the external advertised listener.
114+
enabled: true
115+
image:
116+
repository: alpine/k8s
117+
tag: 1.33.11
118+
pullPolicy: IfNotPresent
119+
timeoutSeconds: 600
120+
pollIntervalSeconds: 5
102121
headlessService:
103122
nameOverride: ""
104123
annotations: {}

0 commit comments

Comments
 (0)