e2e: add nvmeof driver testing support with k8s-e2e-external-storage by nixpanic · Pull Request #6214 · ceph/ceph-csi

nixpanic · 2026-03-31T11:31:03Z

Add configuration files and scripts to enable NVMeoF driver testing in the
k8s-e2e-external-storage CI job. This includes the driver manifest,
StorageClass template, and VolumeSnapshotClass template for NVMeoF.

Also deploy the NVMe-oF gateway with Rook.

Depends-on: #5672 for NVMe-oF gateway support

nixpanic · 2026-03-31T11:32:02Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-03-31T11:36:37Z

/test ci/centos/k8s-e2e-external-storage/1.35

Job is still running after addressing the MarkDown linter failures.

nixpanic · 2026-04-08T16:40:24Z

@Mergifyio rebase

nixpanic · 2026-04-08T16:41:39Z

/test ci/centos/k8s-e2e-external-storage/1.35

mergify · 2026-04-08T16:41:41Z

Deprecation notice: This pull request comes from a fork and was rebased using bot_account impersonation. This capability will be removed on July 1, 2026. After this date, the rebase action will no longer be able to rebase fork pull requests with this configuration. Please switch to the update action/command to ensure compatibility going forward.

mergify · 2026-04-08T16:41:42Z

rebase

✅ Branch has been successfully rebased

nixpanic · 2026-04-09T08:04:45Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T08:24:32Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T08:50:42Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T09:07:13Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T09:25:09Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T09:51:21Z

job is running need to inspect the result of the nvmeof driver once done.

/cc @gadididi

nixpanic · 2026-04-09T13:03:25Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T13:52:49Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-09T15:00:16Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-10T06:45:17Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-10T08:52:01Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-14T07:52:46Z

/test ci/centos/k8s-e2e-external-storage/1.35

nixpanic · 2026-04-14T07:54:13Z

Disable pvcDataSource (cloning) testing, these do not pass:

Summarizing 3 Failures:
  [FAIL] External Storage [Driver: nvmeof.csi.ceph.com] [Testpattern: Dynamic PV (default fs)] provisioning [It] should provision storage with pvc data source in parallel [Slow] [Slow]
  k8s.io/kubernetes/test/e2e/framework/volume/fixtures.go:574
  [FAIL] External Storage [Driver: nvmeof.csi.ceph.com] [Testpattern: Dynamic PV (block volmode)] provisioning [It] should provision storage with pvc data source in parallel [Slow] [Slow]
  k8s.io/kubernetes/test/e2e/framework/volume/fixtures.go:574
  [TIMEDOUT] External Storage [Driver: nvmeof.csi.ceph.com] [Testpattern: Dynamic PV (block volmode)] provisioning [It] should provision storage with pvc data source
  k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:563

logs

1 hour is too little with the added nvme-of testing. See-also: ceph#6214 Signed-off-by: Niels de Vos <ndevos@ibm.com>

nixpanic · 2026-04-14T11:18:37Z

/test ci/centos/k8s-e2e-external-storage/1.34

nixpanic · 2026-04-14T11:18:41Z

/test ci/centos/k8s-e2e-external-storage/1.33

nixpanic · 2026-04-14T11:20:04Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

nixpanic · 2026-04-14T11:21:19Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

logs

This one is important to check, the nvme-of gw gets deployed by the e2e suite, and now also by scripts/rook.sh.

nixpanic · 2026-04-14T11:51:07Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

logs

This one is important to check, the nvme-of gw gets deployed by the e2e suite, and now also by scripts/rook.sh.

Looks good 🥳

SUCCESS! -- 5 Passed | 0 Failed | 0 Pending | 0 Skipped

iPraveenParihar · 2026-04-14T12:32:51Z

+
+```bash
+export GATEWAY_ADDRESS=10.242.64.32
+export LISTENERS='[{"address": "10.242.64.32"}]'


Need to add mandatory hostname?

indeed, the readme is not in-sync anymore

iPraveenParihar · 2026-04-14T12:38:22Z

+driver: nvmeof.csi.ceph.com
+parameters:
+  clusterID: @@CLUSTER_ID@@
+  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-nvmeof-provisioner


need to use rook-csi-rbd-provisioner ?

right, probably. Maybe this caused the volume-clone failure 🤔

iPraveenParihar · 2026-04-14T12:43:46Z


    cat <<EOF >"${TEMP_DIR}/operatorconfig.yaml"
-apiVersion: csi.ceph.io/v1alpha1
+apiVersion: csi.ceph.io/v1


This commit can be dropped now?

that will happen automatically when this PR is rebased

gadididi · 2026-04-14T12:49:21Z

+   clusterID: @@CLUSTER_ID@@
+   pool: replicapool
+   imageFeatures: layering
+   # FIXME: use nvmeof specific secrets, but they don't exist yet?


I dont think there is right now nvmeof secrets. As far as I know it uses rbd secrets
(maybe we should reconsider in the future to use dedicated secrets)

Yes, it would be good to use nvmeof secrets in the future. The e2e/nvmeof-deploy.go does it for our e2e suite. Probably we rely on Rook to create the secrets for the k8s-e2e-external-storage tests.

Add configuration files and scripts to enable NVMeoF driver testing in the k8s-e2e-external-storage CI job. This includes the driver manifest, StorageClass template, and VolumeSnapshotClass template for NVMeoF. The create-storageclasses.sh script now auto-detects the NVMeoF gateway address from the rook-ceph-nvmeof service, or it can be configured via the GATEWAY_ADDRESS and LISTENERS environment variables. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Niels de Vos <ndevos@ibm.com>

Signed-off-by: Niels de Vos <ndevos@ibm.com>

Deploy the NVMe-oF CSI driver alongside RBD, CephFS, and NFS drivers when using the ceph-csi-operator deployment script. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Niels de Vos <ndevos@ibm.com>

POD_ADDRESS is needed as the nvmeof-gw is deployed in rook-ceph, and ceph-csi in a dedicated testing namespace. Resolving the short-hostname of the gateway is not possible from outside the rook-ceph namespace. The gateway is configured with a host-id as the short-hostname, which needs to match what is passed in the LISTENERS. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Niels de Vos <ndevos@ibm.com>

Signed-off-by: Niels de Vos <ndevos@ibm.com>

nixpanic · 2026-04-15T06:59:50Z

/test ci/centos/k8s-e2e-external-storage/1.35

Rakshith-R · 2026-04-15T10:11:13Z

+    # Support ReadWriteMany access modes (block mode only)
+    RWX: false


this test uses filesystem volumes (works on CephFS), we can not restrict the tests to block-mode only

nixpanic · 2026-04-15T15:06:38Z

/test ci/centos/mini-e2e-helm/k8s-1.34/rbd

nixpanic · 2026-04-15T15:08:00Z

/test ci/centos/mini-e2e-helm/k8s-1.34/rbd

Running RBD (with VaultKMS) test for #6236 (comment)

nixpanic added the ci/skip/multi-arch-build skip building on multiple architectures label Mar 31, 2026

mergify Bot added component/nvme-of Issues and PRs related to NVMe-oF. component/testing Additional test cases or CI work labels Mar 31, 2026

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 44d2d02 to 7f22c0f Compare March 31, 2026 11:36

ceph-csi-bot force-pushed the nvmeof/e2e-external-storage branch from 7f22c0f to 0a51e55 Compare April 8, 2026 16:41

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 0a51e55 to 9d2cfcf Compare April 9, 2026 08:04

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 691d9e4 to ca6dc6e Compare April 9, 2026 08:50

nixpanic force-pushed the nvmeof/e2e-external-storage branch from ca6dc6e to 9e6d963 Compare April 9, 2026 09:07

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 9e6d963 to 5a2e14b Compare April 9, 2026 09:24

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 5a2e14b to 783a7fd Compare April 9, 2026 13:03

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 8cc3df5 to 47c3972 Compare April 9, 2026 14:59

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 47c3972 to 51d1351 Compare April 10, 2026 06:45

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 51d1351 to f182e4c Compare April 10, 2026 08:51

nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Apr 14, 2026

ci: set timeout for k8s-external-storage to 90 minutes

061e96f

1 hour is too little with the added nvme-of testing. See-also: ceph#6214 Signed-off-by: Niels de Vos <ndevos@ibm.com>

This was referenced Apr 14, 2026

ci: set timeout for k8s-external-storage to 90 minutes #6234

Closed

nvmeof: add support for volume cloning #6235

Open

nixpanic requested a review from iPraveenParihar April 14, 2026 11:14

nixpanic marked this pull request as ready for review April 14, 2026 11:15

nixpanic requested review from a team and gadididi April 14, 2026 11:15

iPraveenParihar reviewed Apr 14, 2026

View reviewed changes

gadididi reviewed Apr 14, 2026

View reviewed changes

nixpanic mentioned this pull request Apr 14, 2026

e2e: use the NVMe-oF gateway that is deployed by Rook #6236

Open

nixpanic added 5 commits April 15, 2026 08:58

ci: deploy the NVMe-oF gateway with Rook

627e1e6

Signed-off-by: Niels de Vos <ndevos@ibm.com>

ci: deploy the NVMe-oF driver with the ceph-csi-operator

e27fb36

Deploy the NVMe-oF CSI driver alongside RBD, CephFS, and NFS drivers when using the ceph-csi-operator deployment script. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Niels de Vos <ndevos@ibm.com>

nvmeof: disable pvcDataSource with k8s-storage testing

095dc98

Signed-off-by: Niels de Vos <ndevos@ibm.com>

nixpanic force-pushed the nvmeof/e2e-external-storage branch from 8b5087e to 095dc98 Compare April 15, 2026 06:59

nixpanic requested review from gadididi and iPraveenParihar April 15, 2026 08:05

Rakshith-R reviewed Apr 15, 2026

View reviewed changes

nixpanic requested a review from Rakshith-R April 15, 2026 13:13

		# Support ReadWriteMany access modes (block mode only)
		RWX: false

Conversation

nixpanic commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixpanic commented Mar 31, 2026

Uh oh!

nixpanic commented Mar 31, 2026

Uh oh!

nixpanic commented Apr 8, 2026

Uh oh!

nixpanic commented Apr 8, 2026

Uh oh!

mergify Bot commented Apr 8, 2026

Uh oh!

mergify Bot commented Apr 8, 2026

✅ Branch has been successfully rebased

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 9, 2026

Uh oh!

nixpanic commented Apr 10, 2026

Uh oh!

nixpanic commented Apr 10, 2026

Uh oh!

nixpanic commented Apr 14, 2026

Uh oh!

nixpanic commented Apr 14, 2026

Uh oh!

nixpanic commented Apr 14, 2026

Uh oh!

nixpanic commented Apr 14, 2026

Uh oh!

nixpanic commented Apr 14, 2026

Uh oh!

nixpanic commented Apr 14, 2026

Uh oh!

nixpanic commented Apr 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nixpanic commented Apr 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nixpanic commented Apr 15, 2026

Uh oh!

nixpanic commented Apr 15, 2026

Uh oh!

Reviewers

nixpanic commented Mar 31, 2026 •

edited

Loading