Skip to content

release-25.7.adoc

Latest commit

 

History

History
532 lines (409 loc) · 32.1 KB

release-25.7.adoc

File metadata and controls

532 lines (409 loc) · 32.1 KB
Raw

Release 25.7

25.7.0

Released on 2025-07-23.

Note

The SDP 25.7.0 release is the first release which will only be published on oci.stackable.tech.
Note

Previously, releases of the stackablectl CLI tool were synced with platform releases. The tool itself was never tied to a particular version of SDP. As such, it was decided to version stackablectl independently. Starting with this release, the release notes for stackablectl are located on a separate page.

New platform features

General
Security

  • Users (and services) can now use the new TrustStore resource to request the trust root certificates associated with a SecretClass. See the TrustStore documentation and secret-operator#410.

  • Add support for format-specific annotations to override secret file names. Names can be customized using secret volume annotations which are listed in our documentation. See secret-operator#572.

Automatic cluster domain detection

The default Kubernetes cluster domain name is now fetched from the kubelet API unless explicitly configured. Operators are now required to have the RBAC permission to get nodes/proxy. The helm-charts take care of this.

The CLI argument --kubernetes-node-name or env variable KUBERNETES_NODE_NAME needs to be set. The helm-charts take care of this.

See issues#662.

Observability

  • All operators support sending OpenTelemetry logs and traces via OTLP.

  • All operators use unified CLI arguments and environment variables to configure the exporters.

  • All operators allow exporter configuration via Helm values.

  • All exporters are disabled by default, except for console logs, which remain enabled by default.

Additionally see the official telemetry documentation page and the tracking issue issues#639 for further details.

Authorization

The user-info-fetcher backend now supports fetching user groups from Microsoft Entra (experimental). See opa-operator#712.

Miscellaneous

The operator Helm charts now grant RBAC patch permissions on events.k8s.io/events, so events can be aggregated (e.g. "error happened 10 times over the last 5 minutes"). See issues#748.

Apache Airflow

  • The recent release of Airflow 3 brings with it a raft of changes - with both new features and significant changes to architecture. SDP offers early experimental support for Airflow 3.0.1 as of SDP 25.7. This gives users the chance to use the new UI, to update their DAGs and to test the new OPA authenticator. See docker-images#1074.

  • Set the default airflow.task log level to INFO to not spam the Airflow UI with task logs. See airflow-operator#649.

Apache NiFi

  • Starting with NiFi 2.0, custom Python extensions are supported. See nifi-operator#739 and documentation on custom components.

  • Custom NiFi Archives (NARs) and Python components can be loaded from Git repositories with git-sync. See nifi-operator#739 and documentation on custom components.

  • The Stackable operator for Apache NiFi now supports using Kubernetes for cluster coordination and storage, rather than ZooKeeper. (Requires NiFi 2, migrating existing installs is currently not supported) See NiFi backend documentation and nifi-operator#737.

  • Rolling upgrades are now supported between NiFi 2 versions. This is the default update strategy for NiFi 2 clusters. See the NiFi updating documentation and nifi-operator#736.

  • The Stackable operator for Apache NiFi now supports authorization using the Open Policy Agent (OPA). See the NiFi authorization documentation and issues#47.

Apache Spark

  • This release adds experimental support for Spark Connect. The Spark operator watches for SparkConnectServer custom resources. Preliminary documentation is available and the existing Taxi Data Anomaly Detection demo has been retrofitted to use a JupyterLab client running against a Spark Connect server. Also see the issue tracking support for Spark Connect.

  • This release also ensures that Bouncy Castle libraries are included in the Spark images. This was causing errors in kerberized environments previously. See docker-images#1212 for details.

Stackable listener-operator

The Stackable listener-operator can now use custom LoadBalancer classes and disable NodePort allocation.

Platform improvements

General
Listener integration

Breaking: The Stackable operators for Apache Airflow, Apache Druid, Apache HBase, Apache Hadoop, Apache Hive, Apache NiFi, Apache Spark, Apache ZooKeeper, and Trino are now integrated with the Stackable listener-operator. See the overall tracking issue issues#692 and comment summarizing the breaking changes.

  • Apache Airflow and Apache Superset: Listener classes are now declared at role level: this replaces the previous implementation where this was defined at cluster level and used a specific role service. Metrics are not exposed via the listener service and are - as previously - only available within the cluster. The service naming has been changed to reflect this. See issues#692 (comment).

  • Apache Druid: Listener classes for Brokers, Coordinators and Routers are now defined in spec.brokers|coordinators|routers.roleConfig.listenerClass. Metrics are no longer exposed by the listener Service and are instead now defined on a dedicated ClusterIP service per role group (eg: my-druid-broker-default-metrics). A separate headless Service is now created for each role group for internal communications (e.g. between different Druid services). See issues#692 (comment).

  • Apache HBase:: Coordinator Listener classes are now declared at role level spec.<role>.config.listenerClass instead of spec.clusterConfig.listenerClass. HBase has been patched so that the URLs in the UI screens now use listener endpoints (which are externally reachable as long as the appropriate listenerClass has been selected). See HBase ListenerClass documentation and docker-images#692 (comment).

  • Apache Hive: Introduces a listener service on the metastore role level. Additionally, a -headless service for internal communications as well as a -metrics service for monitoring is introduced at the role group level. See hive-operator#566.

  • Apache NiFi: Listener classes are now defined in spec.nodes.roleConfig.listenerClass. Metrics are no longer exposed by the listener Service and are instead now defined on a dedicated ClusterIP service per role group (eg: my-nifi-node-default-metrics). A separate headless Service is now created for each role group for internal communications (between NiFi nodes). See issues#692 (comment).

  • Apache Spark: The history server UI is now exposed using listeners instead of services created by the operator directly. This implementation also changed the CRD by moving the spec.clusterConfig.listenerClass field to spec.node.config.listenerClass. The new and experimental Spark connect servers in this release are also exposed using listeners in the same fashion as the history server. See issues#692 (comment).

  • Apache ZooKeeper: ListenerClasses are now defined in spec.server.roleConfig.listenerClass. Metrics are no longer exposed by the listener Service and are instead now defined on a dedicated ClusterIP service per role group (eg: my-zk-server-default-metrics). A separate headless Service is now created for each role group for internal communications (eg: access to the leader and leader-election). See issues#692 (comment).

  • Trino: Coordinator Listener classes are now declared at role level spec.coordinators.roleConfig.listenerClass instead of spec.clusterConfig.listenerClass. Both Coordinators and Workers have two services at role group level, a headless service called <cluster>-<role>-<rolegroup>-headless and a service for metrics called <cluster>-<role>-<rolegroup>-metrics. See Trino ListenerClass documentation and issues#692 (comment).

Authorization

The user-info-fetcher AD backend now filters by both UPN and SAN fields when fetching user groups. See the documentation and opa-operator#702.

Miscellaneous

  • All products that are built from source in SDP, which is at the time of this release everything except Airflow and Superset, now have a version suffix to indicate they include custom modifications made by Stackable. The suffix the same as for SDP container images, so ZooKeeper 3.9.3 in SDP 25.7.0 will for example report 3.9.3-stackable25.7.0 as its version. See docker-images#1068.

  • We’ve added source code snapshots directly into our container images to further improve transparency and the debugging experience. Each container image now includes .tar.gz archives containing the exact source code used to build that specific version of the product. They can be found within the /stackable directory of each container image. See the SBOM documentation and docker-images#1069.

  • Our containers include a tool which regularly logs relevant system information to aid in debugging. This was introduced in SDP 25.3.0 and was running every minute. The signal to noise ratio was off which is why we are switching to run this every 30min in this release. See docker-images#1128.

Vulnerabilities

230 CVEs were fixed in the Stackable product images. This includes 3 critical and 97 high-severity CVEs.

Apache Druid

We noticed that a bunch of configuration properties were missing on the middleManager roles. The properties are now available. See druid-operator#740.

Apache Hadoop

  • Various setting defaults have been updated for better performance and reliability. See hdfs-operator#685.

  • The operator now defaults dfs.encrypt.data.transfer.cipher.suite to AES/CTR/NoPadding to improve security and performance. See hdfs-operator#693.

  • Previously Apache Hadoop wouldn’t find and use the native OpenSSL library to speed up crypto operations. This was fixed by adding a missing symlink and backporting HADOOP-18583. See docker-images#1209.

Apache NiFi

  • The operator now emits a warning for deprecated sensitive property algorithms in NiFi version 1 and errors out for version 2 if an unsupported sensitive properties algorithm is used. See the official NiFi documentation and nifi-operator#791.

  • Re-add Iceberg support to NiFi 2.4.0, which was removed upstream starting with version 2.0.0. See NiFi Iceberg documentation and nifi-operator#738.

  • Update patch that allows bypassing the host header validation starting with NiFi 2.4.0. See docker-images#1125.

Apache ZooKeeper

  • The image for Apache ZooKeeper backports various 3.9.4 changes into the supported 3.9.3 version. See docker-images#1143.

  • Prometheus support was added using the built-in Prometheus provider in Apache ZooKeeper. This appends another port (native-metrics). To access the metrics this port in combination with the /metrics endpoint can be used. See zookeeper-operator#955.

Platform deprecations

Apache Hadoop

The built-in Prometheus servlet is now enabled by default and metrics are available under the /prom path of all UI services. The metrics exposed by the JMX exporter are now considered deprecated and will be removed in a future release. See hdfs-operator#695.

Apache Hive

Hive 3 has been deprecated since SDP 25.3.0 and was marked as End of Life on 2024-10-08 by the upstream project. We might remove it in the upcoming release, the final decision will be based on metastore client compatibility and customer feedback.

Apache Spark

This release enables the built in Prometheus servlet for metric scraping. The existing JMX exporter is still enabled but it is deprecated and it will be removed in a future release. See spark-k8s-operator#584.

Platform removals

General

  • Remove CSI registration path migration job (init-container) from secret-operator DaemonSet. See secret-operator#610.

  • This release removes support for previously deprecated product service accounts. Deprecated service accounts named <product_name>-serviceaccount are ignored. Instead, product workloads use service accounts named <resource_name>-serviceaccount where resource_name is the stacklet (or cluster) name as defined in the cluster resource. We advise deleting these orphaned ServiceAccounts manually. See operator-rs#1060.

  • Breaking: The lastUpdatedTime was removed from the ClusterStatus of each operator. Users previously relying on this value should migrate to lastTransitionTime. See operator-rs#1032 and operator-rs#1054.

Container images

Breaking: The oci.stackable.tech/sdp/kcat image has been discontinued starting with SDP 25.7.0. Use oci.stackable.tech/sdp/kafka-testing-tools instead, which includes kcat. Existing oci.stackable.tech/sdp/kcat images for previous SDP releases remain available. See docker-images#1124.

Apache HBase

Starting with this release, support for HBase 2.4 has been removed. This includes metrics exposed via the JMX exporter. All supported HBase versions expose metrics via the built in Prometheus servlet. See hbase-operator#672.

Apache ZooKeeper

Breaking: The -nodeport discovery ConfigMap has been deprecated in 25.3.0 and is removed as of this release. Use the primary discovery ConfigMap instead. See kafka-operator#868.

Trino

Breaking: The PersistentVolumeClaims for Trino coordinator and workers have been removed because they caused problems due to storing its PID in /stackable/data/var/run/launcher.pid. A forceful stop (eg. OOMKilled) could result in a leftover PID in this file. In this case Trino would refuse startup with ERROR: already running as 21. As the PersistentVolumeClaims didn’t store any actual data, they have been removed. See trino-operator#768 and trino-operator#769.

  • Upgrading will result in an error, because Kubernetes currently does not allow changing the volumeClaimTemplates field. Simply delete the mentioned StatefulSet and the operator will re-create it.

  • Orphaned PVCs can be cleaned up. Listing all Trino-related PVCs can be done using the following command: kubectl get pvc -l app.kubernetes.io/name=trino.

  • The .spec.coordinators|workers.config.resources.storage.data field has been removed, as it is no longer needed.

Supported versions

Product versions

As with previous SDP releases, many product images have been updated to their latest versions. Refer to the supported versions documentation for a complete overview including LTS versions or deprecations.

New versions

The following new product versions are now supported:

Deprecated versions

The following product versions are deprecated and will be removed in a later release:

Removed versions

The following product versions are no longer supported. These images for released product versions remain available here. Information on how to browse the registry can be found here.

Note

We removed support for Apache Hadoop 3.3.6 in our operator, but it is still required by HBase, Hive, and Spark in this release.
Kubernetes versions

This release supports the following Kubernetes versions:

  • 1.33

  • 1.32

  • 1.31

  • 1.30

These Kubernetes versions are no longer supported:

  • 1.29

OpenShift versions

This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions:

  • 4.18

  • 4.17

  • 4.16

These OpenShift versions are no longer supported:

  • 4.15

  • 4.14

Upgrade from 25.3

Using stackablectl
Upgrade with a single command

Starting with stackablectl Release 1.0.0 the multiple consecutive commands described below can be shortened to just one command, which executes exactly those steps on its own.

$ stackablectl release upgrade 25.7
Upgrade with multiple consecutive commands

Uninstall the 25.3 release

$ stackablectl release uninstall 25.3

Uninstalled release '25.3'

Use "stackablectl release list" to list available releases.
# ...

Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace.

Important

In this release, two CRDs must be applied/created in addition to being replaced:

  • The Stackable secret-operator CRD introduces the new TrustStore custom resource.

  • The Stackable Operator for Apache Spark CRD introduces a new SparkConnectServer custom resource.

The commands below are adjusted accordingly. This issue is also mentioned in the known issues section below.

 
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/25.7.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/25.7.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/25.7.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/25.7.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/25.7.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/25.7.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/25.7.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/25.7.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/25.7.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/25.7.0/deploy/helm/opa-operator/crds/crds.yaml
# We have a new custom resource: truststores.secrets.stackable.tech
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/25.7.0/deploy/helm/secret-operator/crds/crds.yaml || true
kubectl create -f https://raw.githubusercontent.com/stackabletech/secret-operator/25.7.0/deploy/helm/secret-operator/crds/crds.yaml || true
# We have a new custom resource: sparkconnectservers.spark.stackable.tech
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/25.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml || true
kubectl create -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/25.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml || true
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/25.7.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/25.7.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/25.7.0/deploy/helm/zookeeper-operator/crds/crds.yaml
customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...

Install the 25.7 release

$ stackablectl release install 25.7

Installed release '25.7'

Use "stackablectl operator installed" to list installed operators.
Using Helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all operators that are part of the 25.3 release:

$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
...

Afterward you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace.

Important

In this release, two CRDs must be applied/created in addition to being replaced:

  • The Stackable secret-operator CRD introduces the new TrustStore custom resource.

  • The Stackable Operator for Apache Spark CRD introduces a new SparkConnectServer custom resource.

The commands below are adjusted accordingly. This issue is also mentioned in the known issues section below.

 
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/25.7.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/25.7.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/25.7.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/25.7.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/25.7.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/25.7.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/25.7.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/25.7.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/25.7.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/25.7.0/deploy/helm/opa-operator/crds/crds.yaml
# We have a new custom resource: truststores.secrets.stackable.tech
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/25.7.0/deploy/helm/secret-operator/crds/crds.yaml || true
kubectl create -f https://raw.githubusercontent.com/stackabletech/secret-operator/25.7.0/deploy/helm/secret-operator/crds/crds.yaml || true
# We have a new custom resource: sparkconnectservers.spark.stackable.tech
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/25.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml || true
kubectl create -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/25.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml || true
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/25.7.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/25.7.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/25.7.0/deploy/helm/zookeeper-operator/crds/crds.yaml
customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...

Install the 25.7 release

Note
 helm repo subcommands are not supported for OCI registries. The operators are installed directly, without adding the Helm Chart repository first. 
helm install --wait airflow-operator oci://oci.stackable.tech/sdp-charts/airflow-operator --version 25.7.0
helm install --wait commons-operator oci://oci.stackable.tech/sdp-charts/commons-operator --version 25.7.0
helm install --wait druid-operator oci://oci.stackable.tech/sdp-charts/druid-operator --version 25.7.0
helm install --wait hbase-operator oci://oci.stackable.tech/sdp-charts/hbase-operator --version 25.7.0
helm install --wait hdfs-operator oci://oci.stackable.tech/sdp-charts/hdfs-operator --version 25.7.0
helm install --wait hive-operator oci://oci.stackable.tech/sdp-charts/hive-operator --version 25.7.0
helm install --wait kafka-operator oci://oci.stackable.tech/sdp-charts/kafka-operator --version 25.7.0
helm install --wait listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --version 25.7.0
helm install --wait nifi-operator oci://oci.stackable.tech/sdp-charts/nifi-operator --version 25.7.0
helm install --wait opa-operator oci://oci.stackable.tech/sdp-charts/opa-operator --version 25.7.0
helm install --wait secret-operator oci://oci.stackable.tech/sdp-charts/secret-operator --version 25.7.0
helm install --wait spark-k8s-operator oci://oci.stackable.tech/sdp-charts/spark-k8s-operator --version 25.7.0
helm install --wait superset-operator oci://oci.stackable.tech/sdp-charts/superset-operator --version 25.7.0
helm install --wait trino-operator oci://oci.stackable.tech/sdp-charts/trino-operator --version 25.7.0
helm install --wait zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version 25.7.0

Known issues

  • During the upgrade of the secret-operator CRD, special care must be taken because the CRD introduces the new TrustStore custom resource. It is recommended to apply/create it in addition to replacing it.

  • There are strong indicators that TLS CA handling is broken in products using keytool. As of now, we confirmed that Trino has an issue in properly handling CA rotations due to a limitation with keytool. The report in issues#744 outlines a workaround which requires the following steps:

    1. Identify the secretClass used by the secret-operator to distribute CAs.

    2. Delete this secret causing secret-operator to re-create a new CA secret.

    3. Restart all stackable pods manually.