[SPARK-57466] Add option for dynamic-config to use mounted-ConfigMap by csviri · Pull Request #707 · apache/spark-kubernetes-operator

csviri · 2026-06-15T08:23:36Z

What changes were proposed in this pull request?

This PR adds a second, mount-based source for dynamic configuration overrides and makes the dynamic-config mechanism pluggable, while retaining the existing ConfigMap-informer behavior for backwards compatibility.

A new config option spark.kubernetes.operator.dynamicConfig.source selects the source:

configMap (default) — the existing behavior: a dedicated Operator runs SparkOperatorConfigMapReconciler, watching a ConfigMap via a Kubernetes informer. Requires RBAC to read ConfigMaps.
file — a new DynamicConfigMonitor periodically reloads a properties file mounted from a ConfigMap volume, requiring no extra RBAC.

Both paths funnel through the same SparkOperatorConfManager.refresh(...) + watched-namespace update logic, so runtime behavior is identical once overrides are applied.

Key changes:

New DynamicConfigMonitor (config/DynamicConfigMonitor.java) — a single-thread scheduled reader of a properties file with start() / stop() / isRunning() lifecycle and change detection against the last loaded
snapshot.
SparkOperator — registerDynamicConfig() dispatches on dynamicConfig.source: the file source returns a DynamicConfigMonitor; the configMap source registers the second informer Operator (restored
registerSparkOperatorConfMonitor() / overrideConfigMonitorConfigs()). The operator still tracks a List<Operator> plus an optional monitor.
SparkOperatorConf — added dynamicConfig.source, dynamicConfig.filePath, dynamicConfig.reloadIntervalSeconds; retained dynamicConfig.selector and dynamicConfig.reconcilerParallelism.
Probes (HealthProbe, ReadinessProbe, ProbeService, ProbeUtil) — health/readiness cover all registered operators (so the informer operator is checked under configMap) and additionally the monitor's running
state under file.
Helm chart:
- dynamicConfig.source added to values.yaml (default configMap).
- The ConfigMap Role/RoleBinding is created only when enable && source == configMap.
- The dynamic-config volume and mount are added only when enable && source == file.
- The dynamic ConfigMap renders a single spark-operator-dynamic.properties key for the file source, or raw per-key data for the configMap source.

Why are the changes needed?

The informer-based source requires the operator to hold cluster/namespace RBAC on ConfigMaps and runs a second JOSDK Operator purely to watch one ConfigMap. The mount-based source removes that RBAC requirement and the extra
informer by letting the kubelet project ConfigMap changes onto a volume that the operator polls. Rather than forcing a migration, this PR offers the mount-based approach as an opt-in alternative while keeping the informer as
the default, so existing deployments are unaffected.

Does this PR introduce any user-facing change?

Yes, additive and backwards-compatible:

New spark.kubernetes.operator.dynamicConfig.source option (default configMap). With the default, behavior is unchanged from the released operator.
New dynamicConfig.filePath and dynamicConfig.reloadIntervalSeconds options, used only by the file source.
New dynamicConfig.source Helm value (default configMap). Opting into file switches the chart to mount the ConfigMap as a volume (rendered under a single spark-operator-dynamic.properties key) and drops the ConfigMap
RBAC Role/RoleBinding.

No change for users who do not set dynamicConfig.source.

How was this patch tested?

Unit tests for both sources:
- configMap — SparkOperatorConfigMapReconcilerTest and the configMap-source case in SparkOperatorTest (asserts a second Operator is registered and no DynamicConfigMonitor).
- file — DynamicConfigMonitorTest (initial load, no-op reload when unchanged, refresh + namespace update on change, isRunning() lifecycle) and the file-source case in SparkOperatorTest.
Updated HealthProbeTest, ReadinessProbeTest, ProbeServiceTest for the List<Operator> + monitor signatures.
e2e watched-namespaces chainsaw test runs against the file source (tests/e2e/helm/dynamic-config-values{,-2}.yaml set source: file); ConfigMaps use the spark-operator-dynamic.properties layout and the propagation
wait was raised to account for kubelet volume-sync plus the reload interval.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

Rework the dynamic-config feature so the operator no longer runs a dedicated Operator instance with a ConfigMap informer. The dynamic-config ConfigMap is now mounted into the operator pod as a volume, and a new DynamicConfigMonitor periodically re-reads the properties file from disk, refreshing SparkOperatorConfManager and the watched-namespaces updater on change. - Add DynamicConfigMonitor with start/stop/isRunning lifecycle - Simplify SparkOperator from List<Operator> to a single Operator plus an optional DynamicConfigMonitor; remove SparkOperatorConfigMapReconciler - Replace dynamicConfig.selector with dynamicConfig.filePath and dynamicConfig.reloadIntervalSeconds - Helm: drop configmaps RBAC Role/RoleBinding, mount the dynamic ConfigMap volume, render a single spark-operator-dynamic.properties key - Probes: switch to a single Operator and include the monitor's running state in /healthz and /readyz Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

csviri · 2026-06-15T14:30:37Z

this is a strange failure, tests are passing:
https://github.com/apache/spark-kubernetes-operator/actions/runs/26576971235

dongjoon-hyun · 2026-06-15T16:30:15Z

Thank you, @csviri .

For the following question, it's just time-out due to the limit of ubuntu-slim GitHub Actions Virtual Environment. Please re-trigger the failed pipeline. We use ubuntu-slim because of the cost-savings.

this is a strange failure, tests are passing: https://github.com/apache/spark-kubernetes-operator/actions/runs/26576971235

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

csviri · 2026-06-16T13:36:58Z

@dongjoon-hyun updated, please take a look if this way with feature flag is how you imagined this. thx!

dongjoon-hyun · 2026-06-16T15:17:37Z

Thank you, @csviri .

cc @aaruna , too.

jiangzho

Love this feature - thanks @csviri !

dongjoon-hyun · 2026-06-16T22:17:39Z

-  }
-
-  @Test
-  void testHealthProbeWithInformerHealthWithMultiOperators() {


May I ask if this is inevitable removal of irrelevant HealthProbe test coverage, @csviri ?

The new testHealthProbeWithDynamicConfigMonitor test coverage contains this?

No, nit properly, good catch, fixed that and ProbeServiceTest. thx!

dongjoon-hyun · 2026-06-16T22:21:01Z

 data:
-  spark.kubernetes.operator.watchedNamespaces: default, spark-1
+  spark-operator-dynamic.properties: |
+    spark.kubernetes.operator.watchedNamespaces=default, spark-1


Do we need to change this?

yes, see below.

dongjoon-hyun · 2026-06-16T22:21:18Z

 data:
-  spark.kubernetes.operator.watchedNamespaces: default
+  spark-operator-dynamic.properties: |
+    spark.kubernetes.operator.watchedNamespaces=default


Do we need to change this?

Same as above: yes, afaik there is no way to mount all ConfigMap data as a single file in K8S. A key is always a new file. So we have to place the whole content under a key in ConfigMap.

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

dongjoon-hyun · 2026-06-17T17:56:25Z

Hi, @csviri . According to your comment, the following sounds irrelevant to this PR's proposal. Could you make a new PR only for that? We can proceed that independent topic before this PR.

yes, afaik there is not way to mount all ConfigMap data in K8S. A key is always a new file.

data:
- spark.kubernetes.operator.watchedNamespaces: default
+   spark-operator-dynamic.properties: |
+        spark.kubernetes.operator.watchedNamespaces=default

csviri · 2026-06-18T07:48:56Z

Hi, @csviri . According to your comment, the following sounds irrelevant to this PR's proposal. Could you make a new PR only for that? We can proceed that independent topic before this PR.

yes, afaik there is not way to mount all ConfigMap data in K8S. A key is always a new file.
data:
- spark.kubernetes.operator.watchedNamespaces: default
+   spark-operator-dynamic.properties: |
+        spark.kubernetes.operator.watchedNamespaces=default

@dongjoon-hyun maybe I was not clear, those are needed/used in the chainsaw test, and needs to be in this format. Or do I miss something?

dongjoon-hyun · 2026-06-18T17:02:35Z

No, you were clear.

maybe I was not clear, those are needed/used in the chainsaw test, and needs to be in this format. Or do I miss something?

Maybe, I wasn't clear enough. My perspective was the following.

This is supposed to introduce a new alternative path which we should not affect the existing functionality and test coverage.
In other words, this PR is supposed to introduce an independent set of test coverage instead of touching the existing test code and test resource.

So, if you need to change some existing test resource for any reasons like the following, it should be done before this PR independently.

A key is always a new file.

csviri changed the title ~~Replace dynamic-config informer with mounted-ConfigMap file watcher~~ Replace dynamic-config controller with mounted-ConfigMap file watcher Jun 15, 2026

csviri changed the title ~~Replace dynamic-config controller with mounted-ConfigMap file watcher~~ [SPARK-57466] Replace dynamic-config controller with mounted-ConfigMap file watcher Jun 15, 2026

csviri marked this pull request as ready for review June 15, 2026 09:18

csviri marked this pull request as draft June 15, 2026 09:18

csviri force-pushed the dynamic-config-map-name-informer-mount branch from fb3cb27 to 949aea7 Compare June 15, 2026 09:24

integration test sleep duration

b1700ed

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

csviri marked this pull request as ready for review June 15, 2026 10:22

csviri marked this pull request as draft June 15, 2026 11:00

fix test

449c1f0

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

csviri marked this pull request as ready for review June 15, 2026 11:07

dongjoon-hyun reviewed Jun 15, 2026

View reviewed changes

Comment thread docs/config_properties.md

csviri added 2 commits June 16, 2026 12:23

add back previous controller based config read

66ace53

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

resurrect test

5e17ffe

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

csviri marked this pull request as draft June 16, 2026 11:46

csviri added 3 commits June 16, 2026 14:05

test fix

3cf9fe6

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

e2e test for config map config

8f53d0a

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

test files

fd85b6d

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

csviri requested a review from dongjoon-hyun June 16, 2026 12:48

csviri marked this pull request as ready for review June 16, 2026 12:58

csviri changed the title ~~[SPARK-57466] Replace dynamic-config controller with mounted-ConfigMap file watcher~~ [SPARK-57466] Add option for dynamic-config to use mounted-ConfigMap Jun 16, 2026

jiangzho approved these changes Jun 16, 2026

View reviewed changes

dongjoon-hyun reviewed Jun 16, 2026

View reviewed changes

revert health probe test

73bc671

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

fix ProbeServiceTest

cd091eb

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Conversation

csviri commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

csviri commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jun 15, 2026

Uh oh!

Uh oh!

csviri commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jun 16, 2026

Uh oh!

jiangzho left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

csviri Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

csviri Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

csviri Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jun 17, 2026

Uh oh!

csviri commented Jun 18, 2026

Uh oh!

dongjoon-hyun commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

csviri commented Jun 15, 2026 •

edited

Loading

csviri commented Jun 15, 2026 •

edited

Loading

csviri commented Jun 16, 2026 •

edited

Loading

csviri Jun 17, 2026 •

edited

Loading

csviri Jun 17, 2026 •

edited

Loading