Skip to content

release-26.3.adoc

Latest commit

 

History

History
602 lines (456 loc) · 31 KB

release-26.3.adoc

File metadata and controls

602 lines (456 loc) · 31 KB
Raw

Release 26.3

26.3.0

Released on 2026-03-16.

Release highlights

  • Generic object overrides are now supported across the SDP.

  • The Stackable operator for OpenSearch adds many new features, such as TLS configuration and service discovery.

  • The User Info Fetcher (UIF) now has experimental support for OpenLDAP, and the Entra backend has been stabilized.

  • The restart-controller is now enabled for nearly all products.
Overview of breaking changes

The following components of the SDP contain breaking changes for this release:

New platform features

General
Miscellaneous

  • This release supports objectOverrides, a list of generic Kubernetes objects, which are merged into the objects created by the operator (e.g. StatefulSets, Listeners or ConfigMaps). The priority of the overrides (in ascending order) is configOverrides → podOverrides → objectOverrides, with the latter overriding the previous. See the concepts page for more details. Tracked in issues#712.

  • A new demo was added that shows Retrieval Augmented Generation (RAG) with OpenSearch as the vector store. Consult the demo documentation page for more details. Added in demos#354.

  • All remaining operators now maintain their own CRDs instead of relying on Helm as a deployment mechanism. Additionally, the conversion webhook is now running alongside the controller for future CRD versioning. Tracked in issues#808.

Apache Airflow

Airflow can now use git-sync to authenticate against a repository using SSH keys as well as basic auth, which was previously the only option. The Airflow operator also now has the ability to convert certain changes between CRD versions: in the case of this change, the operator can automatically convert the field credentialsSecret to credentials: basicAuthSecretName so that this change is non-breaking.

Important

Conversions in the opposite direction - from v1alpha2 to v1alpha1 - are not possible if the v1alpha2 resource definition refers to the new field.

See the DAG mounting usage guide for more details. Implemented in airflow-operator#382.

Apache HBase

The hbase.rest.endpoint field is now added to the restserver rolegroup discovery ConfigMap for advertising the REST servers. Implemented in hbase-operator#708 and hbase-operator#716.

Apache Hive

The Stackable operator for Apache Hive now supports delegating authorization requests to the Open Policy Agent (OPA) using a custom hive-metastore-opa-authorizer plugin. See the usage guide for more information. Implemented in hive-operator#388.

Apache Kafka

  • Previously, Kafka supported OPA in non-TLS mode. Now both TLS and non-TLS modes are supported.

    Important

    It should be noted that this is not yet supported by the Kafka Controller (available in Kafka 4.x).

  • Users of the Stackable operator for Apache Kafka can now migrate existing 3.9.1 clusters from Apache ZooKeeper to the built-in KRaft metadata manager. See the KRaft migration guide which describes the necessary steps to perform the migration. Implemented in kafka-operator#923.

Apache Spark

  • Starting with this release, Spark applications will not be resubmitted automatically in case of failure. Users can restore the previous behavior by setting the new spec.job.retryOnFailureCount property to a non-negative value.
    In addition, small improvements have been made to clean up resources created by Spark applications:

    • Application driver pods are now deleted as soon as they reach a terminal state.

    • Application executor pods are now deleted as soon as possible in case of driver or submit failure.

    Implemented in spark-k8s-operator#647.

  • This release adds first-class support for S3 buckets and connections to Spark Connect servers. Users can give clients access to all S3 buckets or a selected list. For more details, see the Spark Connect usage guide. Implemented in spark-k8s-operator#652.

  • This release introduces Spark Application Templates. These allow application authors to group frequently used application configuration in a central place that is referenced from application objects. For more details on this new operator feature see the template usage guide. Implemented in spark-k8s-operator#660.

Open Policy Agent

  • The User Info Fetcher (UIF) now has experimental support for OpenLDAP as a backend. See the OpenLDAP backend documentation of the UIF for details on how to configure it. Implemented in opa-operator#523.

  • The OPA operator now supports overriding command-line arguments passed to the OPA binary via the cliOverrides property in the CRD. This allows you to customize OPA’s behavior by passing additional or overriding existing command-line flags. See the OPA operator documentation for further details and examples. Implemented in opa-operator#781.

OpenSearch

  • The operator now supports the configuration of TLS for the server (optional) and internal communication (mandatory) using SecretClasses. See the OpenSearch security usage guide for more details. Implemented in opensearch-operator#55.

  • The operator now supports adding secrets to the OpenSearch keystore, e.g. S3 credentials for creating and restoring backups. See the keystore usage guide for more details. Implemented in opensearch-operator#76.

  • A new service discovery mechanism has been introduced, enabling OpenSearch clients to be easily configured. Users can specify whether a role group should be exposed and choose the corresponding ListenerClass. The connection parameters for accessing the OpenSearch cluster are provided in a discovery ConfigMap. Detailed information can be found at the OpenSearch discovery documentation page. Implemented in opensearch-operator#52.

  • The OpenSearch security plugin is now configurable within the OpenSearchCluster specification. There are two modes of management for the configuration files of the security plugin: one uses the provided configuration only to initialize the security index, after which the API manages it; the other allows the operator to manage the configuration directly. See the security documentation page for more details on this.

    Important

    It is essential to remove both podOverrides and configOverrides related to security settings, such as plugins.security.allow_default_init_securityindex.

    Consult the upgrade guide for more detailed instructions.

    Implemented in opensearch-operator#43.

Stackable listener-operator

  • The listener-operator now supports configuring serviceOverrides on ListenerClasses similar to podOverrides on stacklets. This enables users to make arbitrary modifications to the created Services. Implemented in listener-operator#365.

  • The listener-operator now deploys the selected ListenerClass preset on its own instead of relying on Helm. This is done to ensure the ListenerClasses are created after the operator deployed its own CRD. Implemented in listener-operator#369.

Stackable secret-operator

The secret-operator now supports the new secrets.stackable.tech/provision-parts annotation on the secret volume. This annotation allows configuring which parts of the secret material should be provisioned. Using this annotation enables the following use-cases:

  • Use the autoTls backend, but only provision the ca.crt/truststore.p12 for the consumer.

  • Use the kerberosKeytab backend, but only provision the krb5.conf for the consumer.

  • Use the k8sSearch backend to select Secrets which contain public data only and support parsing the partial set of files. Using this annotation disables the strict parsing of files when an explicit format is requested.

See the volume documentation page for more details. Implemented in secret-operator#676.

Platform improvements

General
Vulnerabilities

114 CVEs were fixed in the Stackable product images. This includes 9 critical and 40 high-severity CVEs.

Miscellaneous

  • All operator images have been updated from UBI9 to UBI10 base images. See docker-images#1349.

  • All products (except Apache Hadoop) now have the restart-controller enabled, so that Pods are automatically restarted on config changes (in ConfigMaps and Secrets). See issues#816.

Apache Airflow

  • Previously, only selected providers were included in the Airflow product images. Now we attempt to include all extra packages that are available for Airflow versions 3+, with an overview and explicit exclusions listed in the product image notes. Implemented in docker-images#1336.

  • At times, a race condition can arise between DAG parsing and GitSync fetches. We have added a note to the troubleshooting section of the documentation to suggest a workaround for this problem. Added in airflow-operator#732.

Apache HBase

The hbase-operator-tools have been bumped to 1.3.0 and HBASE-29797 has been backported for all supported HBase images. Implemented in docker-images#1417.

Apache Hadoop

The operator now adds a warning and exit condition to the format-namenodes container script to check for corrupted data after formatting. Implemented in hdfs-operator#751.

Apache NiFi

  • Breaking: The authorization configuration for the operator has been adjusted to more closely match and represent the Apache NiFi interfaces and implementations. The opa, singleUser, and standard authorization options can now be explicitly set. This allows setting an initialAdminUser for file-based authorization.

    spec:
  clusterConfig:
    authorization:
      opa: { .. } # existing
      # OR
      singleUser: {} # new
      # OR
      standard: # new
        accessPolicyProvider:
          fileBased:
            initialAdminUser: CN=admin,OU=admin-group,DC=example,DC=org

    If not provided this defaults to the singleUser.

    Warning

    Users authenticating via LDAP that do not use the opa authorizer (but relied on the operator-generated file-based authorization), now have to explicitly set the standard authorizer and the initialAdminUser. The cluster definition must be extended by an authorization part like shown above.

    This means that the bind user now can differ from the initialAdminUser.

    See the security usage guide for more information. Implemented in nifi-operator#884.

  • There are cases where customers may wish to provide static authorization files (users.xml, authorizer.xml, authorizations.xml) instead of fetching the data dynamically (for instance, when collecting user and group data from AD/Entra).

    Statically provided user/group information and coarse-grained permissions are honoured automatically (as long as the UUIDs match those available in the backend), but for the initial process group this has to be done by a callback mechanism so that the authorizations can be updated after the flow has been registered.

    The NiFi images have been patched for versions 2.6.0 and 2.7.2 to support this. Implemented in nifi-operator#862.

Apache Spark

Spark application jobs can now have pod/node affinities. Previously only the application driver and executors had a dedicated field for this purpose. Consult the pod placement guide for more details. Implemented in spark-k8s-operator#639.

Open Policy Agent

  • Previously, when POST-ing the status following the successful loading of a bundle, a warning was written to the logs because of a superfluous service name. This has now been removed. Implemented in opa-operator#798.

  • The Entra User Info Fetcher (UIF) backend has been stabilized with the introduction of OpaCluster v1alpha2. Conversion between v1alpha1 and v1alpha2 is handled automatically by default. Implemented in opa-operator#801.

Trino

  • Breaking: The operator no longer sets opa.policy.column-masking-uri in access-control.properties but opa.policy.batch-column-masking-uri instead, allowing Trino to fetch multiple column masks in a single request. This now requires a batchColumnMasks rule to be present in the OPA Trino rules, when using the default URI set by the trino-operator.
    The above default can be disabled with the new enableColumnMasking field in the OPA authorization configuration.

    Warning

    The opa field under authorization is now a mandatory enum variant instead of being optional.
    Tip

    It is recommended to use spec.clusterConfig.authorization.opa.enableColumnMasking: false instead of batchColumnMasks = [].

    Implemented in trino-operator#814.

  • The spec.connector.iceberg.metastore field in TrinoCatalog is now optional, as Iceberg also supports other catalogs, such as a REST catalog, which (currently) can only be added using configOverrides. Implemented in trino-operator#841.

Platform fixes

General
Logging

Previously, log entries could be sent multiple times to the Vector aggregator if the Vector sidecar container was restarted. This issue has now been resolved by persisting the Vector state across container restarts. Fixed in issues#821.

Miscellaneous

All operators now correctly and gracefully shut down all concurrent tasks by forwarding the SIGTERM signal. Most operators already correctly handled this before, but this is now done consistently across the board. Tracked in issues#803.

Apache Airflow

  • Previously, Celery workers in Airflow 3 could run into the problem of not executing tasks anymore after a Redis re-connect. Now, this issue is fixed by bumping the celery Python package. Fixed in docker-images#1343.

  • Previously, the operator adopted the default of 4 API workers per webserver. This results in higher resource usage although it is not required for every situation. Now, the default has been set to 1 (it can easily be changed by adding either more webserver replicas or an override). The documentation has been corrected and clarified to reflect this. Fixed in airflow-operator#727.

Apache Druid

Breaking: Starting with Druid 35, the router requires higher CPU resources. CPU request and limit have been bumped from 100m and 400m to 300m and 1200m respectively. Fixed in druid-operator#786.

Apache Hadoop

Previously, some parts of the shell output of the init containers were not logged properly and therefore not aggregated. Now, these log messages are captured correctly. Fixed in hdfs-operator#746.

Apache NiFi

  • The operator now instructs NiFi pods to also listen on the loopback interface so that Kubernetes port-forwards work. Fixed in nifi-operator#870.

  • The operator now utilizes the .spec.clusterConfig.authorization.opa.package field instead of hard-coding the OPA package name to nifi. By default, the NifiCluster name is used as a package name if not explicitly set. Fixed in nifi-operator#881.

Apache Spark

  • Driver pods are now garbage collected when the Spark application is finished. Previously, driver pods created by the submit job would be left hanging even after the job has been deleted. Fixed in spark-k8s-operator#649.

  • Fix application pod specifications when both the history server and the S3 connection reference the same SecretClass object. Previously this would lead to a duplicate volume. With this release that duplication is removed. Fixed in spark-k8s-operator#655.

Apache ZooKeeper

Previously, running zkCli commands in the ZooKeeper image could fail due to an upstream issue ZOOKEEPER-4985. This has now been fixed by setting the environment variable ZOOCFGDIR. Fixed in zookeeper-operator#988.

OpenSearch

Previously, the log file rollover was non-functional. When the log file reached a size of 5 MB, an error message repeatedly appeared in the console, indicating that the log file could not be renamed. With this release, we have resolved the permissions issue, ensuring that the log file rollover works correctly. Fixed in opensearch-operator#106.

Stackable commons-operator

Breaking: Prevent an unnecessary restart of Pod 0 of a product StatefulSet when the STS is initially created. This was caused by commons-operator needing to update the STS immediately after creation, at which point Pod 0 was already created. The problem is now fixed by utilizing an admission webhook, which is able to mutate the StatefulSet during its creation - before Pod 0 is created. Fixed in commons-operator#387.

Warning

The commons-operator now needs the RBAC permission to create and patch mutatingwebhookconfigurations. Operator installations via the Helm Chart automatically add these new permissions.

The webhook can be disabled using --disable-restarter-mutating-webhook or by setting the DISABLE_RESTARTER_MUTATING_WEBHOOK environment variable, which will however revert the fixed behaviour.

Platform removals

Apache Airflow

Breaking: Users of the EXPERIMENTAL_FILE_HEADER and EXPERIMENTAL_FILE_FOOTER fields who rely on AUTH_OID from the flask_appbuilder.const import must switch to AUTH_OAUTH.

If you are using Airflow < 3.1 and still require AUTH_OID, you can manually import it by following the documented instructions. Removed in airflow-operator#737.

Apache Superset

Breaking: Users of the EXPERIMENTAL_FILE_HEADER and EXPERIMENTAL_FILE_FOOTER fields who rely on AUTH_OID from the flask_appbuilder.security.manager import must switch to AUTH_OAUTH.

If you are using Superset < 6.0.0 and still require AUTH_OID, you can manually import it by following the documented instructions. Removed in superset-operator#680 and docker-images#1382.

Stackable secret-operator

Breaking: Removed support for non-persistent ephemeral CSI volumes. This has been deprecated since secret-operator 0.4.0 because it doesn’t support Pod stickiness. Persistent ephemeral volumes that use CSI are still supported and are the recommended way forward.

# Replace this
volumes:
  - csi: # ...

# with the new recommended syntax
volumes:
  - ephemeral:
      volumeClaimTemplate: # ...

Removed in secret-operator#481.

Supported versions

Product versions

As with previous SDP releases, many product images have been updated to newer versions. Refer to the supported versions documentation for a complete overview including LTS versions or deprecations.

New versions

The following new product versions are now supported:

  • Apache Airflow: 3.1.6

  • Apache Druid: 35.0.1

  • Apache HBase: 2.6.4 (LTS)

  • Apache Hive: 4.2.0

  • Apache Kafka: 4.1.1 (experimental)

    • Note that Kafka 3.9.1 is the last version that supports Apache ZooKeeper as the metadata manager.

  • Apache NiFi: 2.7.2

    • Note that this version supports Iceberg, but only with S3 and the Iceberg REST catalog. Hive metastore or HDFS are not supported. Read the official NiFi Iceberg documentation for details.

  • Apache Spark: 3.5.8 (LTS), 4.1.1

    • Note that there are no Apache Iceberg and Delta runtime libraries available for 4.1.1 yet.

  • Apache Superset: 6.0.0

    • OpenStreetView has been promoted as the new default for the Deck.gl visualization which doesn’t require an API key.

    • Consult the official update notes.

  • Open Policy Agent: 1.12.3

  • OpenSearch: 3.4.0

  • Trino: 479

    • The storage-connector is based on Trino 477 because there was no later release.

  • Vector: 0.52.0

Deprecated versions

The following product versions are deprecated and will be removed in a later release:

Removed versions

The following product versions are no longer supported. These images for released product versions remain available in the Stackable OCI Registry. Information on how to browse the registry can be found in the project overview documentation.

Kubernetes versions
Warning

There is one known issue when running on Kubernetes 1.35.2 and earlier. Later versions might have fixed it. See the known issues section below for more details.

This release supports the following Kubernetes versions:

  • 1.35

  • 1.34

  • 1.33

  • 1.32

  • 1.31

These Kubernetes versions are no longer supported:

  • 1.30

  • 1.29

OpenShift versions

This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions:

  • 4.21

  • 4.20

  • 4.19

  • 4.18

These OpenShift versions are no longer supported:

  • 4.17

  • 4.16

Upgrade from 25.11

Tip

Please consult the following upgrade notes when upgrading from 25.11 to 26.3:

  • OpenSearch upgrade guide
Using stackablectl
Upgrade with a single command

Starting with stackablectl Release 1.0.0 the multiple consecutive commands described below can be shortened to just one command, which executes exactly those steps on its own.

$ stackablectl release upgrade 26.3
Upgrade with multiple consecutive commands

Uninstall the 25.11 release

$ stackablectl release uninstall 25.11

Uninstalled release '25.11'

Use "stackablectl release list" to list available releases.
# ...

Install the 26.3 release

$ stackablectl release install 26.3

Installed release '26.3'

Use "stackablectl operator installed" to list installed operators.
Using Helm

Use helm list to list the currently installed operators.

You can use the following command to uninstall all operators that are part of the 25.11 release:

$ helm uninstall airflow-operator <PRODUCT>-operator ...
release "airflow-operator" uninstalled
release "<PRODUCT>-operator" uninstalled
...

Install the 26.3 release

Note

helm repo subcommands are not supported for OCI registries. The operators are installed directly, without adding the Helm Chart repository first.

 
helm install --wait <PRODUCT>-operator oci://oci.stackable.tech/sdp-charts/<PRODUCT>-operator --version 26.3.0

Known issues

  • This release makes zookeeper the default metadata manager for Kafka 3.x clusters. If you are running Kafka 3 with kraft as metadata manager already, you need to patch your existing Kafka custom resource to make this choice explicit after upgrading the SDP version. The Kafka cluster will run in an error state until you patch it. Patch your Kafka resource like this:

    $ kubectl -n <namespace> \
  patch kafkaclusters.kafka.stackable.tech <kafka-resource-name> \
  --type merge -p '{"spec":{"clusterConfig":{"metadataManager":"kraft"}}}'

    Replace <namespace> and <kafka-resource-name> as appropriate.
    See kafka-operator#933.

  • We discovered an issue with the feature gate MaxUnavailableStatefulSet that is enabled by default as a beta feature in Kubernetes 1.35. This issue affects all SDP operators, especially the Trino operator.
    To work around the issue there are two possibilities:

    1. Disable this feature gate in your Kubernetes 1.35 clusters.

    2. Ensure that your product custom resources (e.g. TrinoCluster, SparkApplication, etc.) are applied after any referenced custom resources such as TrinoCatalog, S3Connection, and so on.

  • In Druid 35 and 36, OIDC authentication may fail due to a change in how the authentication method is selected when connecting to an OIDC provider. If your OIDC provider (e.g. Keycloak) advertises private_key_jwt as a supported client authentication method, Druid may attempt to use it, which causes authentication to fail. See druid-operator#800.