Skip to content

Commit 11ece1d

Browse files
razvanrazvan
authored
tests: spark connect/kerberos/iceberg integration test (#704)
* tests: spark connect/kerberos/iceberg integration test * Add comment about spark.addArtifact() * Update docs * Update test and docs to mention that only Spark 3 is affected
1 parent 85715a7 commit 11ece1d

29 files changed

Lines changed: 1351 additions & 1 deletion

‎docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc‎

Lines changed: 92 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,92 @@ spec:
9797
...
9898
```
9999

100+
== Kerberos
101+
102+
NOTE: Kerberos support for Spark Connect is not a first-class feature of the operator. The setup described here is a manual configuration that uses `podOverrides`, `configOverrides`, `envOverrides` and an init container.
103+
104+
A Spark Connect server can authenticate to a Kerberos-secured service, such as a Apache Hive metastore or Apache Hadoop HDFS.
105+
There is, however, an important caveat: the Spark Connect server runs in Spark's `client` deploy mode.
106+
Spark only performs the automatic keytab login (`UserGroupInformation.loginUserFromKeytab`) for YARN, local, Mesos and the Kubernetes _cluster_-mode driver -- *not* for the Kubernetes client mode that the Connect server uses.
107+
As a consequence, setting `spark.kerberos.keytab` and `spark.kerberos.principal` alone does not obtain a Kerberos ticket (TGT), and the SASL/GSSAPI handshake to the metastore fails with `GSS initiate failed` / `Failed to find any Kerberos tgt`.
108+
109+
The workaround is to obtain the ticket yourself with an *init container* that runs `kinit` before the Spark Connect server JVM starts.
110+
The init container reads the keytab and writes a Kerberos credential cache to an `emptyDir` volume that is shared with the `spark` container; the server JVM then picks the ticket up from that cache.
111+
An init container is required because nothing in the client-mode server process performs the login automatically, so the ticket must exist in the credential cache before the JVM opens the metastore connection.
112+
113+
The relevant parts of the `SparkConnectServer` are shown below.
114+
115+
[source,yaml]
116+
----
117+
spec:
118+
server:
119+
envOverrides:
120+
# The shared credential cache populated by the init container. Hadoop's
121+
# UserGroupInformation reads KRB5CCNAME when Kerberos auth is enabled.
122+
KRB5CCNAME: /stackable/krb5/ccache
123+
jvmArgumentOverrides:
124+
add:
125+
# The JVM does NOT read the KRB5_CONFIG environment variable (that is an
126+
# MIT C-library variable). It reads this system property (or /etc/krb5.conf).
127+
- -Djava.security.krb5.conf=/stackable/kerberos/krb5.conf
128+
# The Spark Connect execute thread does not carry the ticket in its
129+
# Subject, so JGSS must fall back to the ambient credential cache.
130+
- -Djavax.security.auth.useSubjectCredsOnly=false
131+
configOverrides:
132+
spark-defaults.conf:
133+
spark.hadoop.hadoop.security.authentication: "kerberos"
134+
spark.hadoop.hive.metastore.uris: "thrift://hive-metastore:9083"
135+
spark.hadoop.hive.metastore.sasl.enabled: "true"
136+
# Use the literal metastore principal, not _HOST (which would resolve to
137+
# the connection host instead of the metastore's service principal).
138+
spark.hadoop.hive.metastore.kerberos.principal: "hive/hive.example.svc.cluster.local@EXAMPLE.COM"
139+
podOverrides:
140+
spec:
141+
initContainers:
142+
- name: kinit
143+
image: oci.stackable.tech/sdp/spark-k8s:4.1.1-stackable0.0.0-dev
144+
command: ["/bin/bash", "-euo", "pipefail", "-c"]
145+
args:
146+
- kinit -kt /stackable/kerberos/keytab spark-connect/spark-connect.example.svc.cluster.local@EXAMPLE.COM
147+
env:
148+
- name: KRB5_CONFIG
149+
value: /stackable/kerberos/krb5.conf
150+
- name: KRB5CCNAME
151+
value: /stackable/krb5/ccache
152+
volumeMounts:
153+
- name: kerberos
154+
mountPath: /stackable/kerberos
155+
- name: krb5-ccache
156+
mountPath: /stackable/krb5
157+
containers:
158+
- name: spark
159+
volumeMounts:
160+
- name: kerberos
161+
mountPath: /stackable/kerberos
162+
- name: krb5-ccache
163+
mountPath: /stackable/krb5
164+
volumes:
165+
- name: krb5-ccache
166+
emptyDir: {}
167+
- name: kerberos
168+
ephemeral:
169+
volumeClaimTemplate:
170+
metadata:
171+
annotations:
172+
secrets.stackable.tech/class: kerberos
173+
secrets.stackable.tech/scope: service=spark-connect
174+
secrets.stackable.tech/kerberos.service.names: spark-connect
175+
spec:
176+
storageClassName: secrets.stackable.tech
177+
accessModes: ["ReadWriteOnce"]
178+
resources:
179+
requests:
180+
storage: "1"
181+
----
182+
183+
The keytab and `krb5.conf` are only required on the server, which is the Spark driver and the only component that talks to the metastore.
184+
Executors that merely read and write data files on (non-kerberized) S3 do not need Kerberos credentials.
185+
100186
== Spark History Server
101187

102188
Unfortunately integration with the Spark History Server is not supported yet.
@@ -112,4 +198,9 @@ The following features are not supported by the Stackable Spark operator yet
112198

113199
== Known Issues
114200

115-
* Dynamically provisioning the iceberg runtime leads to "iceberg.SparkWrite$WriterFactory" ClassNotfoundException when attempting to use it from clients.
201+
* Distributed operations on Apache Iceberg tables fail on the executors *with Spark 3.5.x*.
202+
PySpark calls that trigger a distributed job over an Iceberg table -- for example `DataFrame.collect()` or `DataFrame.count()` -- fail with `java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3`.
203+
Driver-only operations still work: DDL (`CREATE`/`DROP`), `INSERT` of small data, `DataFrame.show()` of a small result, `SHOW TABLES` and `DESCRIBE`.
204+
The cause is upstream in Spark Connect: executor tasks run under a per-session class loader (which fetches session classes from the driver's artifact server) that cannot deserialize the Iceberg scan closures.
205+
It is independent of how the Iceberg runtime is provisioned -- it reproduces with `--packages`, with the jar placed on the system class path (`/stackable/spark/jars`), with `spark.addArtifact()`, and with combinations of these.
206+
*This is fixed in Spark 4* (https://issues.apache.org/jira/browse/SPARK-51537[SPARK-51537] / https://github.com/apache/spark/pull/50475[apache/spark#50475]), where distributed reads and writes of Iceberg tables work; it is not backported to the 3.5.x line (see the still-open https://issues.apache.org/jira/browse/SPARK-46032[SPARK-46032]).

‎tests/templates/kuttl/spark-connect-kerberos/00-assert.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
apiVersion: kuttl.dev/v1beta1
3+
kind: TestAssert
4+
timeout: 900
5+
---
6+
apiVersion: v1
7+
kind: ServiceAccount
8+
metadata:
9+
name: integration-tests-sa

‎tests/templates/kuttl/spark-connect-kerberos/00-patch-ns.yaml.j2‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{% if test_scenario['values']['openshift'] == 'true' %}
2+
# see https://github.com/stackabletech/issues/issues/566
3+
---
4+
apiVersion: kuttl.dev/v1beta1
5+
kind: TestStep
6+
commands:
7+
- script: kubectl patch namespace $NAMESPACE -p '{"metadata":{"labels":{"pod-security.kubernetes.io/enforce":"privileged"}}}'
8+
timeout: 120
9+
{% endif %}

‎tests/templates/kuttl/spark-connect-kerberos/00-serviceaccount.yaml.j2‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
kind: Role
3+
apiVersion: rbac.authorization.k8s.io/v1
4+
metadata:
5+
name: use-integration-tests-scc
6+
rules:
7+
{% if test_scenario['values']['openshift'] == "true" %}
8+
- apiGroups: ["security.openshift.io"]
9+
resources: ["securitycontextconstraints"]
10+
resourceNames: ["privileged"]
11+
verbs: ["use"]
12+
{% endif %}
13+
---
14+
apiVersion: v1
15+
kind: ServiceAccount
16+
metadata:
17+
name: integration-tests-sa
18+
---
19+
kind: RoleBinding
20+
apiVersion: rbac.authorization.k8s.io/v1
21+
metadata:
22+
name: use-integration-tests-scc
23+
subjects:
24+
- kind: ServiceAccount
25+
name: integration-tests-sa
26+
roleRef:
27+
kind: Role
28+
name: use-integration-tests-scc
29+
apiGroup: rbac.authorization.k8s.io

‎tests/templates/kuttl/spark-connect-kerberos/01-assert.yaml.j2‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
apiVersion: kuttl.dev/v1beta1
3+
kind: TestAssert
4+
{% if lookup('env', 'VECTOR_AGGREGATOR') %}
5+
---
6+
apiVersion: v1
7+
kind: ConfigMap
8+
metadata:
9+
name: vector-aggregator-discovery
10+
{% endif %}

‎tests/templates/kuttl/spark-connect-kerberos/01-install-vector-aggregator-discovery-configmap.yaml.j2‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{% if lookup('env', 'VECTOR_AGGREGATOR') %}
2+
---
3+
apiVersion: v1
4+
kind: ConfigMap
5+
metadata:
6+
name: vector-aggregator-discovery
7+
data:
8+
ADDRESS: {{ lookup('env', 'VECTOR_AGGREGATOR') }}
9+
{% endif %}

‎tests/templates/kuttl/spark-connect-kerberos/02-assert.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
apiVersion: kuttl.dev/v1beta1
3+
kind: TestAssert
4+
timeout: 600
5+
---
6+
apiVersion: apps/v1
7+
kind: StatefulSet
8+
metadata:
9+
name: krb5-kdc
10+
status:
11+
readyReplicas: 1
12+
replicas: 1

‎tests/templates/kuttl/spark-connect-kerberos/02-install-krb5-kdc.yaml.j2‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
---
2+
apiVersion: apps/v1
3+
kind: StatefulSet
4+
metadata:
5+
name: krb5-kdc
6+
spec:
7+
selector:
8+
matchLabels:
9+
app: krb5-kdc
10+
template:
11+
metadata:
12+
labels:
13+
app: krb5-kdc
14+
spec:
15+
serviceAccountName: integration-tests-sa
16+
initContainers:
17+
- name: init
18+
image: oci.stackable.tech/sdp/krb5:{{ test_scenario['values']['krb5'] }}-stackable0.0.0-dev
19+
args:
20+
- sh
21+
- -euo
22+
- pipefail
23+
- -c
24+
- |
25+
test -e /var/kerberos/krb5kdc/principal || kdb5_util create -s -P asdf
26+
kadmin.local get_principal -terse root/admin || kadmin.local add_principal -pw asdf root/admin
27+
# stackable-secret-operator principal must match the keytab specified in the SecretClass
28+
kadmin.local get_principal -terse stackable-secret-operator || kadmin.local add_principal -e aes256-cts-hmac-sha384-192:normal -pw asdf stackable-secret-operator
29+
env:
30+
- name: KRB5_CONFIG
31+
value: /stackable/config/krb5.conf
32+
volumeMounts:
33+
- mountPath: /stackable/config
34+
name: config
35+
- mountPath: /var/kerberos/krb5kdc
36+
name: data
37+
containers:
38+
- name: kdc
39+
image: oci.stackable.tech/sdp/krb5:{{ test_scenario['values']['krb5'] }}-stackable0.0.0-dev
40+
args:
41+
- krb5kdc
42+
- -n
43+
env:
44+
- name: KRB5_CONFIG
45+
value: /stackable/config/krb5.conf
46+
volumeMounts:
47+
- mountPath: /stackable/config
48+
name: config
49+
- mountPath: /var/kerberos/krb5kdc
50+
name: data
51+
# Root permissions required on Openshift to access internal ports
52+
{% if test_scenario['values']['openshift'] == "true" %}
53+
securityContext:
54+
runAsUser: 0
55+
{% endif %}
56+
- name: kadmind
57+
image: oci.stackable.tech/sdp/krb5:{{ test_scenario['values']['krb5'] }}-stackable0.0.0-dev
58+
args:
59+
- kadmind
60+
- -nofork
61+
env:
62+
- name: KRB5_CONFIG
63+
value: /stackable/config/krb5.conf
64+
volumeMounts:
65+
- mountPath: /stackable/config
66+
name: config
67+
- mountPath: /var/kerberos/krb5kdc
68+
name: data
69+
# Root permissions required on Openshift to access internal ports
70+
{% if test_scenario['values']['openshift'] == "true" %}
71+
securityContext:
72+
runAsUser: 0
73+
{% endif %}
74+
- name: client
75+
image: oci.stackable.tech/sdp/krb5:{{ test_scenario['values']['krb5'] }}-stackable0.0.0-dev
76+
tty: true
77+
stdin: true
78+
env:
79+
- name: KRB5_CONFIG
80+
value: /stackable/config/krb5.conf
81+
volumeMounts:
82+
- mountPath: /stackable/config
83+
name: config
84+
volumes:
85+
- name: config
86+
configMap:
87+
name: krb5-kdc
88+
volumeClaimTemplates:
89+
- metadata:
90+
name: data
91+
spec:
92+
accessModes:
93+
- ReadWriteOnce
94+
resources:
95+
requests:
96+
storage: 1Gi
97+
---
98+
apiVersion: v1
99+
kind: Service
100+
metadata:
101+
name: krb5-kdc
102+
spec:
103+
selector:
104+
app: krb5-kdc
105+
ports:
106+
- name: kadmin
107+
port: 749
108+
- name: kdc
109+
port: 88
110+
- name: kdc-udp
111+
port: 88
112+
protocol: UDP
113+
---
114+
apiVersion: v1
115+
kind: ConfigMap
116+
metadata:
117+
name: krb5-kdc
118+
data:
119+
krb5.conf: |
120+
[logging]
121+
default = STDERR
122+
kdc = STDERR
123+
admin_server = STDERR
124+
[libdefaults]
125+
dns_lookup_realm = false
126+
ticket_lifetime = 24h
127+
renew_lifetime = 7d
128+
forwardable = true
129+
rdns = false
130+
default_realm = {{ test_scenario['values']['kerberos-realm'] }}
131+
spake_preauth_groups = edwards25519
132+
[realms]
133+
{{ test_scenario['values']['kerberos-realm'] }} = {
134+
acl_file = /stackable/config/kadm5.acl
135+
disable_encrypted_timestamp = false
136+
}
137+
[domain_realm]
138+
.cluster.local = {{ test_scenario['values']['kerberos-realm'] }}
139+
cluster.local = {{ test_scenario['values']['kerberos-realm'] }}
140+
kadm5.acl: |
141+
root/admin *e
142+
stackable-secret-operator *e

‎tests/templates/kuttl/spark-connect-kerberos/03-create-kerberos-secretclass.yaml.j2‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
apiVersion: kuttl.dev/v1beta1
3+
kind: TestStep
4+
commands:
5+
# We need to replace $NAMESPACE (by KUTTL)
6+
- script: |
7+
kubectl apply -n "$NAMESPACE" -f - <<EOF
8+
---
9+
apiVersion: secrets.stackable.tech/v1alpha1
10+
kind: SecretClass
11+
metadata:
12+
name: kerberos-$NAMESPACE
13+
spec:
14+
backend:
15+
kerberosKeytab:
16+
realmName: {{ test_scenario['values']['kerberos-realm'] }}
17+
kdc: krb5-kdc.$NAMESPACE.svc.cluster.local
18+
admin:
19+
mit:
20+
kadminServer: krb5-kdc.$NAMESPACE.svc.cluster.local
21+
adminKeytabSecret:
22+
namespace: $NAMESPACE
23+
name: secret-operator-keytab
24+
adminPrincipal: stackable-secret-operator
25+
EOF
26+
---
27+
apiVersion: v1
28+
kind: Secret
29+
metadata:
30+
name: secret-operator-keytab
31+
data:
32+
# To create keytab. When prompted enter password asdf
33+
# cat | ktutil << 'EOF'
34+
# list
35+
# add_entry -password -p stackable-secret-operator@CLUSTER.LOCAL -k 1 -e aes256-cts-hmac-sha384-192
36+
# wkt /tmp/keytab
37+
# EOF
38+
{% if test_scenario['values']['kerberos-realm'] == 'CLUSTER.LOCAL' %}
39+
keytab: BQIAAABdAAEADUNMVVNURVIuTE9DQUwAGXN0YWNrYWJsZS1zZWNyZXQtb3BlcmF0b3IAAAABZAYWIgEAFAAgm8MCZ8B//XF1tH92GciD6/usWUNAmBTZnZQxLua2TkgAAAAB
40+
{% elif test_scenario['values']['kerberos-realm'] == 'PROD.MYCORP' %}
41+
keytab: BQIAAABbAAEAC1BST0QuTVlDT1JQABlzdGFja2FibGUtc2VjcmV0LW9wZXJhdG9yAAAAAWQZa0EBABQAIC/EnFNejq/K5lX6tX+B3/tkI13TCzkPB7d2ggCIEzE8AAAAAQ==
42+
{% endif %}

0 commit comments

Comments
 (0)