[Improvement] Separate S3 storage configuration for MLRun and Kubeflow Pipeline by GiladShapira94 · Pull Request #295 · mlrun/ce

GiladShapira94 · 2026-05-12T11:49:08Z

📝 Description

This PR separates the storage credential configuration into two distinct paths: storage.local.* for the bundled in-cluster SeaweedFS (always used by SeaweedFS IAM, the bucket-init job, and KFP Pipelines), and storage.s3.* for external AWS S3 (used only by MLRun and Jupyter when storage.mode: s3).

The default storage.mode is changed from s3 to local, reflecting that the default CE installation uses the bundled SeaweedFS rather than external AWS S3.

New dedicated _helpers.tpl partials (mlrun-ce.seaweedfs.s3.* and mlrun-ce.pipelines.s3.*) ensure Pipelines and SeaweedFS always resolve credentials from storage.local.* regardless of the active storage.mode, eliminating the previous credential cross-contamination when switching modes.

🛠️ Changes Made

charts/mlrun-ce/values.yaml:
- Changed storage.mode default from s3 → local
- Added new storage.local block (accessKey, secretKey, bucket) as the single source of truth for in-cluster SeaweedFS credentials
- Cleared storage.s3.accessKey/secretKey/bucket defaults (now empty strings; only meaningful when mode: s3)
charts/mlrun-ce/templates/_helpers.tpl:
- mlrun-ce.s3.accessKey/secretKey/bucket — now branches on storage.mode (local vs s3)
- Added mlrun-ce.seaweedfs.s3.* helpers — always resolve from storage.local.*
- Added mlrun-ce.pipelines.s3.* helpers — always delegate to mlrun-ce.seaweedfs.s3.*
- mlrun-ce.artifactPath, mlrun-ce.featureStore.dataPrefix, mlrun-ce.model-endpoint.monitoring.* — replaced hardcoded global.infrastructure.aws.bucketName | default "mlrun" with mlrun-ce.s3.bucket
charts/mlrun-ce/templates/config/storage-secret.yaml — AWS_ENDPOINT_URL_S3 now only injected when storage.mode: local; storage.s3 no longer sets a custom endpoint
charts/mlrun-ce/templates/config/storage-validation.yaml — added fail guard for storage.mode: local with missing storage.local.bucket
charts/mlrun-ce/templates/config/mlrun-env-configmap.yaml — updated comment describing per-mode env vars
charts/mlrun-ce/templates/pipelines/** — all pipeline templates now use mlrun-ce.pipelines.s3.* helpers
charts/mlrun-ce/templates/seaweedfs/** — bucket-init job and IAM config now use mlrun-ce.seaweedfs.s3.* helpers
charts/mlrun-ce/templates/NOTES.txt — S3 credentials display updated to reference storage.local.*
charts/mlrun-ce/Chart.yaml — version bumped 0.12.0-rc.1 → 0.12.0-rc.2
charts/mlrun-ce/README.md — version matrix updated to 0.12.0-rc.2

✅ Checklist

🧪 Testing

helm lint charts/mlrun-ce — run locally to catch syntax errors in refactored helpers
helm template mlrun charts/mlrun-ce -f charts/mlrun-ce/values.yaml — render all templates to verify helper resolution
Verify storage.mode: s3 path: render with --set storage.mode=s3,storage.s3.accessKey=foo,storage.s3.secretKey=bar,storage.s3.bucket=mybucket and confirm storage-secret does not contain AWS_ENDPOINT_URL_S3
Verify storage.mode: local path: render with defaults and confirm storage-secret contains AWS_ENDPOINT_URL_S3 pointing at the SeaweedFS service
Confirm pipelines secret mlpipeline-seaweedfs-artifact always uses storage.local.* regardless of storage.mode
End-to-end cluster install (required before merge)

🔗 References

Ticket link: CEML-707
External links:
Design docs links (Optional):

🚨 Breaking Changes?

Yes (explain below)
No

Consumers upgrading from a previous release must:

Rename storage.s3.accessKey/secretKey/bucket → storage.local.accessKey/secretKey/bucket if they were using the default SeaweedFS-backed installation (i.e., the old default mode: s3 pointed at SeaweedFS with seaweed/seaweed123/mlrun).
Set storage.mode: s3 explicitly if they were previously relying on the default mode: s3 to pass external AWS credentials — the new default is local.
Users who supply an external AWS S3 configuration no longer need to clear AWS_ENDPOINT_URL_S3 manually; the secret now omits it when mode: s3.

🔍️ Additional Notes

The three install-mode values files (admin_installation_values.yaml, non_admin_installation_values.yaml, non_admin_cluster_ip_installation_values.yaml) contain no storage.* overrides, so they correctly inherit the new defaults from values.yaml without modification.
KFP Pipelines is intentionally hardwired to SeaweedFS (storage.local.*) in all modes — this is by design and is documented in the updated helper comments.

Warnings

Breaking change — existing storage.s3.* users: Anyone who previously used the default install (which was mode: s3 pointing at SeaweedFS with seaweed/seaweed123) must migrate their overrides to storage.local.*.

Their upgrade path:
```
--set storage.local.accessKey=<old s3.accessKey> \
--set storage.local.secretKey=<old s3.secretKey> \
--set storage.local.bucket=<old s3.bucket>
```

Test CE Workflos

[Fix] Seaweed Change

# Conflicts: # .github/workflows/release.yml # charts/mlrun-ce/Chart.yaml

yaelgen · 2026-05-27T10:30:29Z

@@ -1,6 +1,9 @@
 {{- if and (eq .Values.storage.mode "s3") (not .Values.storage.s3.bucket) }}


storage.s3.{accessKey,secretKey} default to empty strings but only bucket is validated. Switching to s3 mode without creds will silently produce an unusable Secret. Please also fail-fast when accessKey/secretKey are empty (unless global.infrastructure.aws.s3NonAnonymous is true).

…a-kfp # Conflicts: # charts/mlrun-ce/Chart.yaml # charts/mlrun-ce/README.md

…a-kfp

royischoss

Hey few comments

royischoss · 2026-07-07T07:59:51Z

+{{- if eq .Values.storage.mode "local" -}}
+{{- .Values.storage.local.bucket -}}
+{{- else -}}
+{{- coalesce .Values.global.infrastructure.aws.bucketName .Values.storage.s3.bucket "mlrun" -}}


coalesce ordering contradicts the s3 bucket validation guard. storage-validation.yaml forces the user to set storage.s3.bucket in s3 mode (the render fails otherwise), but this line prefers global.infrastructure.aws.bucketName over it.

Either drop the storage.s3.bucket validation, or put storage.s3.bucket first in the coalesce so the required value wins:

Suggested change

{{- coalesce .Values.global.infrastructure.aws.bucketName .Values.storage.s3.bucket "mlrun" -}}

{{- coalesce .Values.storage.s3.bucket .Values.global.infrastructure.aws.bucketName "mlrun" -}}

i did not wanted to change the old flow has it may break use installation

royischoss · 2026-07-07T08:14:38Z

      # - name: alerts

  ui:
+    image:


why do we add image here how it worked before?

As this rc is decoupled with Application version we need to add the image to install the chart with the relevant application version

GiladShapira94 and others added 25 commits April 15, 2026 17:39

first commit

3e26ccb

fix

91aa535

fix run issue

355728a

fix run issue

ba88c03

fix run issue

5ed1f83

remove label

9640e9c

fix after review

cbe71fc

Merge remote-tracking branch 'upstream/development' into ce-worfklows

6264877

fix after review

3413976

Merge remote-tracking branch 'upstream/development' into ce-worfklows

8ec1302

first commit

5364f3c

fix installation issue

61988fb

Merge pull request #1 from GiladShapira94/ce-worfklows

ae6842e

Test CE Workflos

Update release.yml

3ac86b3

change chart version

9618888

Update pr-validation.yml

c95a662

Update pr-validation.yml

59655d2

Merge pull request #3 from GiladShapira94/CEML-696

e945709

[Fix] Seaweed Change

Update release.yml

55691a6

[Fix] testing fix

efa5752

Merge remote-tracking branch 'origin/development' into development

a3dbc06

print the release rc

252af2c

Merge remote-tracking branch 'upstream/development' into development

c6b4d1f

# Conflicts: # .github/workflows/release.yml # charts/mlrun-ce/Chart.yaml

Merge remote-tracking branch 'upstream/development' into development

58fb414

first commit

0f1e167

github-actions Bot added the improvement label May 12, 2026

GiladShapira94 added 4 commits May 19, 2026 13:27

add local mode

f923147

fix small issues

61566c6

fix small issues

c8b21ff

fix small issues

061dcfc

GiladShapira94 added 2 commits May 26, 2026 15:08

revert pipelines changes

53027c7

small fix

889767b

GiladShapira94 requested review from shay79il and yaelgen May 26, 2026 15:14

Merge remote-tracking branch 'upstream/development' into development

8460eb3

yaelgen requested changes May 27, 2026

View reviewed changes

GiladShapira94 added 4 commits June 3, 2026 09:00

Merge remote-tracking branch 'upstream/development' into development

7f0ec90

Merge remote-tracking branch 'upstream/development' into development

414abcd

Merge remote-tracking branch 'upstream/development' into separate-dat…

a167681

…a-kfp # Conflicts: # charts/mlrun-ce/Chart.yaml # charts/mlrun-ce/README.md

fix after reveiw

24515cc

GiladShapira94 requested a review from yaelgen June 3, 2026 13:34

yaelgen approved these changes Jun 4, 2026

View reviewed changes

GiladShapira94 added 4 commits July 6, 2026 13:12

Merge branch 'development' into separate-data-kfp

59b1930

Merge remote-tracking branch 'upstream/development' into separate-dat…

2e2a8e9

…a-kfp

add application version

077f803

update comment

f978a9a

GiladShapira94 requested a review from royischoss July 6, 2026 12:17

royischoss reviewed Jul 7, 2026

View reviewed changes

fix after review

ec12cf5

GiladShapira94 requested a review from royischoss July 7, 2026 09:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Improvement] Separate S3 storage configuration for MLRun and Kubeflow Pipeline#295

[Improvement] Separate S3 storage configuration for MLRun and Kubeflow Pipeline#295
GiladShapira94 wants to merge 41 commits into
mlrun:developmentfrom
GiladShapira94:separate-data-kfp

GiladShapira94 commented May 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yaelgen May 27, 2026

Uh oh!

Uh oh!

Uh oh!

royischoss left a comment

Uh oh!

Uh oh!

Uh oh!

royischoss Jul 7, 2026

Uh oh!

GiladShapira94 Jul 7, 2026

Uh oh!

royischoss Jul 7, 2026

Uh oh!

GiladShapira94 Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -1,6 +1,9 @@
		{{- if and (eq .Values.storage.mode "s3") (not .Values.storage.s3.bucket) }}

	{{- coalesce .Values.global.infrastructure.aws.bucketName .Values.storage.s3.bucket "mlrun" -}}
	{{- coalesce .Values.storage.s3.bucket .Values.global.infrastructure.aws.bucketName "mlrun" -}}

Uh oh!

Conversation

GiladShapira94 commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Description

🛠️ Changes Made

✅ Checklist

🧪 Testing

🔗 References

🚨 Breaking Changes?

🔍️ Additional Notes

Warnings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yaelgen May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

royischoss left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

royischoss Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

GiladShapira94 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

royischoss Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

GiladShapira94 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GiladShapira94 commented May 12, 2026 •

edited

Loading