e2e: add nvmeof vac test by gadididi · Pull Request #6276 · ceph/ceph-csi

gadididi · 2026-05-14T06:05:02Z

Add VolumeAttributesClass example and e2e test for NVMe-oF

This PR adds example documentation and end-to-end testing for NVMe-oF VolumeAttributesClass support.

Changes:

Example YAML (examples/nvmeof/volumeattributesclass.yaml): Demonstrates using allowHostNQNs parameter to control host access to volumes, with all QoS parameters documented
E2E test: Validates VAC functionality by creating a PVC with VAC reference and verifying proper volume lifecycle
Test helper: Adds createNVMeOFVolumeAttributesClass() helper for reusable VAC creation in tests

What it tests:

Creating a VAC with allowHostNQNs parameter
Creating a PVC that references the VAC
Volume creation and deletion with VAC applied

This e2e PR is based on that PR: #6251 . Once that will be merged, this PR can be rebased and opened

Depends-on: #6251

mergify · 2026-06-03T10:43:22Z

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 ⛓️ Depends-On Requirements

Requirement based on the presence of Depends-On in the body of the pull request

depends-on = ceph/ceph-csi#6251 [⛓️ nvmeof: Add external client support #6251]

🟢 🚦 Auto-queue

When all merge protections are satisfied, this pull request will be queued automatically.

Copilot

Pull request overview

This PR introduces an NVMe-oF VolumeAttributesClass (VAC) example plus an end-to-end test that provisions an NVMe-oF PVC referencing a VAC (with allowHostNQNs) and then deletes it, aiming to validate VAC support for NVMe-oF.

Changes:

Add an NVMe-oF VAC example manifest documenting allowHostNQNs and NVMe-oF QoS parameters.
Add an NVMe-oF e2e test that creates a VAC, provisions a PVC referencing it, and validates backend object lifecycle.
Add an e2e helper to create NVMe-oF VolumeAttributesClass resources.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
examples/nvmeof/volumeattributesclass.yaml	New VAC example YAML for NVMe-oF (includes `allowHostNQNs` and QoS docs).
e2e/nvmeof.go	New NVMe-oF e2e test covering PVC lifecycle when using a VAC reference.
e2e/nvmeof-deploy.go	New helper to create NVMe-oF VolumeAttributesClass resources in e2e.

Madhu-1 · 2026-06-04T08:25:34Z

+	return wait.PollUntilContextTimeout(context.TODO(), poll, timeout, true, func(ctx context.Context) (bool, error) {
+		_, err = c.StorageV1().VolumeAttributesClasses().Create(ctx, &vac, metav1.CreateOptions{})
+		if err != nil {
+			framework.Logf("error creating VolumeAttributesClass %q: %v", vac.Name, err)


move this line to 279 below already exists error as this is false positive

Madhu-1 · 2026-06-04T08:27:28Z

+			pvc.Spec.VolumeAttributesClassName = &vacName
+
+			ginkgo.By("Creating PVC with VolumeAttributesClass reference")
+			err = createPVCAndvalidatePV(f.ClientSet, pvc, deployTimeout)


dont we need to validate the right allowHostNQNs values are set?

good point,
I think right now there is no nvmeof toolbox for querying the GW, which can return what hosts are in the list. (As we don't verify in create a PVC and delete it the nvmeof namespace is created ..)

@nixpanic , maybe we should reconsider it?

A test that allows attaching and mounting would be nice. Testing for expected failing to mount is not appropriate, it will take too much time and causes CI runs to take much more time (Kubernetes will continue to re-try for a while).

Possibly you could run the nvme command directly in the NVMe-oF node-plugin to verify the inaccessibility, but that may be tricky.

I will check it.
Now I realize I did not use dedicated StorageClass in this test (without subsystemNQN)
I will fix it as well.

Hi, I added in the nvmeof helper test file nvme connect-all with the given external hostNQN. this command runs on the node plugin (simulated as external client) then check by nvme list-subsys if the host is there in the output.

I also override for this test the StorageClass, because for external clients they need dedicated SC (basically without SubsystemNQN field) .

Pull request has been modified.

gadididi · 2026-06-09T09:22:29Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

gadididi · 2026-06-09T09:22:58Z

running at:
https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/blue/organizations/jenkins/mini-e2e_k8s-1.35-nvmeof/detail/mini-e2e_k8s-1.35-nvmeof/68/pipeline

gadididi · 2026-06-09T10:37:11Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

gadididi · 2026-06-09T10:38:17Z

the first has an issue .

  I0609 09:45:16.013444   78661 exec_util.go:206] unexpected error trying to use websockets for pod exec: command terminated with exit code 1
  I0609 09:45:16.013685   78661 pod.go:249] failed to execute command: command terminated with exit code 1
  I0609 09:45:16.013794   78661 nvmeof_helper.go:486] nvme connect-all stdout: No transport specified
  failed to add controller, error Invalid argument

run 2nd on:
https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/blue/organizations/jenkins/mini-e2e_k8s-1.35-nvmeof/detail/mini-e2e_k8s-1.35-nvmeof/69/pipeline

gadididi · 2026-06-09T11:11:17Z

@nixpanic
I ran it again. looks it passed and found the hostNQN from nvme list-subsys:

I0609 11:00:44.408678   79389 nvmeof_helper.go:475] Executing: nvme connect-all --transport=tcp --traddr=10.244.0.20 --hostnqn=nqn.2014-08.org.nvmexpress:test-host-1 2>&1
  I0609 11:00:44.408809   79389 exec_util.go:67] ExecWithOptions {Command:[/bin/sh -c nvme connect-all --transport=tcp --traddr=10.244.0.20 --hostnqn=nqn.2014-08.org.nvmexpress:test-host-1 2>&1] Namespace:cephcsi-e2e-fb28affe PodName:csi-nvmeofplugin-lgm2q ContainerName:csi-nvmeofplugin Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0609 11:00:44.408841   79389 exec_util.go:72] ExecWithOptions: Clientset creation
  I0609 11:00:44.408882   79389 exec_util.go:88] ExecWithOptions: execute(https://192.168.49.2:8443/api/v1/namespaces/cephcsi-e2e-fb28affe/pods/csi-nvmeofplugin-lgm2q/exec?command=%2Fbin%2Fsh&command=-c&command=nvme+connect-all+--transport%3Dtcp+--traddr%3D10.244.0.20+--hostnqn%3Dnqn.2014-08.org.nvmexpress%3Atest-host-1+2%3E%261&container=csi-nvmeofplugin&stderr=true&stdout=true)
  I0609 11:00:44.610326   79389 nvmeof_helper.go:501] Verifying connection with: nvme list-subsys -o json 2>&1
  I0609 11:00:44.610694   79389 exec_util.go:67] ExecWithOptions {Command:[/bin/sh -c nvme list-subsys -o json 2>&1] Namespace:cephcsi-e2e-fb28affe PodName:csi-nvmeofplugin-lgm2q ContainerName:csi-nvmeofplugin Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0609 11:00:44.610856   79389 exec_util.go:72] ExecWithOptions: Clientset creation
  I0609 11:00:44.611000   79389 exec_util.go:88] ExecWithOptions: execute(https://192.168.49.2:8443/api/v1/namespaces/cephcsi-e2e-fb28affe/pods/csi-nvmeofplugin-lgm2q/exec?command=%2Fbin%2Fsh&command=-c&command=nvme+list-subsys+-o+json+2%3E%261&container=csi-nvmeofplugin&stderr=true&stdout=true)
  I0609 11:00:44.674788   79389 nvmeof_helper.go:529] Found host in list-subsys: nqn.2014-08.org.nvmexpress:test-host-1
  I0609 11:00:44.674948   79389 exec_util.go:67] ExecWithOptions {Command:[/bin/sh -c nvme disconnect-all 2>&1 || true] Namespace:cephcsi-e2e-fb28affe PodName:csi-nvmeofplugin-lgm2q ContainerName:csi-nvmeofplugin Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0609 11:00:44.674979   79389 exec_util.go:72] ExecWithOptions: Clientset creation
  I0609 11:00:44.675029   79389 exec_util.go:88] ExecWithOptions: execute(https://192.168.49.2:8443/api/v1/namespaces/cephcsi-e2e-fb28affe/pods/csi-nvmeofplugin-lgm2q/exec?command=%2Fbin%2Fsh&command=-c&command=nvme+disconnect-all+2%3E%261+%7C%7C+true&container=csi-nvmeofplugin&stderr=true&stdout=true)
  I0609 11:00:45.638230   79389 nvmeof_helper.go:552] Disconnected all NVMe connections
  I0609 11:00:45.638412   79389 nvmeof_helper.go:460] SUCCESS: Connected with allowed host NQN nqn.2014-08.org.nvmexpress:test-host-1

nixpanic · 2026-07-06T08:27:03Z

These steps can be done once #6354 passed all CI jobs:

@mergifyio rebase
wait for the rebased changes to be pushed
add the ok-to-test label
wait...

gadididi · 2026-07-06T14:00:03Z

@Mergifyio rebase

mergify · 2026-07-06T14:00:17Z

rebase

🛑 The pull request rule doesn't match anymore

Details

This action has been cancelled.

Add a VolumeAttributesClass example for NVMe-oF volumes demonstrating the mutable parameters supported by the driver: - Host access control via allowHostNQNs parameter (for external hosts) - QoS parameters The example includes a working configuration with host access control enabled, and all optional parameters documented as comments with explanations. Also add an e2e test that validates VAC functionality by: 1. Creating a VAC with allowHostNQNs parameter 2. Creating a PVC that references the VAC 3. Verifying volume creation 4. check the host was added by run nvme connect 5. activate new VAC and recheck the new host 5. Validating cleanup Signed-off-by: gadi-didi <gadi.didi@ibm.com>

gadididi · 2026-07-06T14:03:55Z

@Mergifyio rebase

mergify · 2026-07-06T14:04:28Z

rebase

☑️ Nothing to do, the required conditions are not met

Details

any of:
- #commits-behind > 0 [📌 rebase requirement]
- -linear-history [📌 rebase requirement]
-closed [📌 rebase requirement]
-conflict [📌 rebase requirement]
queue-position = -1 [📌 rebase requirement]

ceph-csi-bot · 2026-07-06T14:05:30Z

/test ci/centos/k8s-e2e-external-storage/1.36

ceph-csi-bot · 2026-07-06T14:05:30Z

/test ci/centos/k8s-e2e-external-storage/1.35

ceph-csi-bot · 2026-07-06T14:05:31Z

/test ci/centos/k8s-e2e-external-storage/1.34

ceph-csi-bot · 2026-07-06T14:05:31Z

/test ci/centos/mini-e2e-helm/k8s-1.36

ceph-csi-bot · 2026-07-06T14:05:31Z

/test ci/centos/mini-e2e-helm/k8s-1.35

ceph-csi-bot · 2026-07-06T14:05:31Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2026-07-06T14:05:32Z

/test ci/centos/mini-e2e/k8s-1.35

ceph-csi-bot · 2026-07-06T14:05:32Z

/test ci/centos/mini-e2e-helm/k8s-1.34

ceph-csi-bot · 2026-07-06T14:05:32Z

/test ci/centos/mini-e2e/k8s-1.36

ceph-csi-bot · 2026-07-06T14:05:32Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2026-07-06T14:05:33Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2026-07-06T19:30:37Z

/test ci/centos/mini-e2e/k8s-1.36

nixpanic · 2026-07-06T19:31:02Z

/test ci/centos/mini-e2e/k8s-1.36

Unrelated failure with NFS. logs

mergify · 2026-07-06T22:28:19Z

Deprecation notice: This pull request comes from a fork and was queued with update_method=rebase and update_bot_account impersonation. This capability will be removed on July 1, 2026. After this date, the merge queue will no longer be able to rebase fork pull requests with this configuration. To avoid disruption, switch to update_method=merge in your queue rule.

mergify · 2026-07-06T22:28:21Z

Merge Queue Status

✅ Entered queue — 2026-07-06 22:28 UTC · Rule: default · triggered by merge protections
✅ Checks skipped · PR is already up-to-date
✅ Merged — 2026-07-06 22:28 UTC · at daedce7f5a3cbb39b16b9ec9e318099e97962824 · rebase

This pull request spent 8 seconds in the queue, including 1 second running CI.

Required conditions to merge

depends-on = ceph/ceph-csi#6251 [⛓️ nvmeof: Add external client support #6251]
- e2e: add nvmeof vac test #6276
github-review-approved [🛡 GitHub branch protection]
- e2e: add nvmeof vac test #6276

gadididi requested a review from nixpanic May 14, 2026 06:05

gadididi self-assigned this May 14, 2026

gadididi added the component/nvme-of Issues and PRs related to NVMe-oF. label May 14, 2026

mergify Bot added the component/testing Additional test cases or CI work label May 14, 2026

gadididi added the keepalive This label can be used to disable stale bot activiity in the repo label May 14, 2026

nixpanic reviewed Jun 3, 2026

View reviewed changes

Comment thread e2e/nvmeof.go Outdated

nixpanic reviewed Jun 3, 2026

View reviewed changes

Comment thread e2e/nvmeof_vac_helper.go Outdated

nixpanic requested changes Jun 3, 2026

View reviewed changes

Comment thread e2e/nvmeof.go Outdated

gadididi marked this pull request as ready for review June 3, 2026 11:38

gadididi force-pushed the e2e/nvmeof_add_vac_test branch 3 times, most recently from ca74032 to 9dab5f7 Compare June 3, 2026 13:43

nixpanic previously approved these changes Jun 4, 2026

View reviewed changes

nixpanic requested a review from a team June 4, 2026 07:50

gadididi requested a review from Copilot June 4, 2026 07:50

Copilot started reviewing on behalf of gadididi June 4, 2026 07:50 View session

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread examples/nvmeof/volumeattributesclass.yaml Outdated

Comment thread e2e/nvmeof.go

Comment thread e2e/nvmeof.go Outdated

Comment thread e2e/nvmeof-deploy.go

Madhu-1 requested changes Jun 4, 2026

View reviewed changes

gadididi force-pushed the e2e/nvmeof_add_vac_test branch from 9dab5f7 to ccacbc3 Compare June 7, 2026 10:03

gadididi requested review from Madhu-1 and nixpanic June 7, 2026 10:05

gadididi force-pushed the e2e/nvmeof_add_vac_test branch from ccacbc3 to e42de3f Compare June 9, 2026 09:21

gadididi force-pushed the e2e/nvmeof_add_vac_test branch from e42de3f to 8b5ab59 Compare June 9, 2026 10:36

Madhu-1 approved these changes Jul 6, 2026

View reviewed changes

nixpanic approved these changes Jul 6, 2026

View reviewed changes

ceph-csi-bot force-pushed the e2e/nvmeof_add_vac_test branch from f674431 to daedce7 Compare July 6, 2026 14:01

gadididi added the ok-to-test Label to trigger E2E tests label Jul 6, 2026

ceph-csi-bot added ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. and removed ok-to-test Label to trigger E2E tests labels Jul 6, 2026

mergify Bot removed the ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. label Jul 6, 2026

mergify Bot merged commit ad2839d into ceph:devel Jul 6, 2026
41 of 43 checks passed

gadididi deleted the e2e/nvmeof_add_vac_test branch July 7, 2026 05:48

Uh oh!

Conversation

gadididi commented May 14, 2026 • edited by nixpanic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add VolumeAttributesClass example and e2e test for NVMe-oF

Changes:

What it tests:

Uh oh!

mergify Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🟢 ⛓️ Depends-On Requirements

🟢 🚦 Auto-queue

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Madhu-1 Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

gadididi Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Madhu-1 Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

gadididi Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

nixpanic Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

gadididi Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

gadididi Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gadididi commented Jun 9, 2026

Uh oh!

gadididi commented Jun 9, 2026

Uh oh!

gadididi commented Jun 9, 2026

Uh oh!

gadididi commented Jun 9, 2026

Uh oh!

gadididi commented Jun 9, 2026

Uh oh!

nixpanic commented Jul 6, 2026

Uh oh!

gadididi commented Jul 6, 2026

Uh oh!

mergify Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 The pull request rule doesn't match anymore

Uh oh!

gadididi commented Jul 6, 2026

Uh oh!

mergify Bot commented Jul 6, 2026

☑️ Nothing to do, the required conditions are not met

Uh oh!

ceph-csi-bot commented Jul 6, 2026

Uh oh!

ceph-csi-bot commented Jul 6, 2026

Uh oh!

ceph-csi-bot commented Jul 6, 2026

Uh oh!

ceph-csi-bot commented Jul 6, 2026

gadididi commented May 14, 2026 •

edited by nixpanic

Loading

mergify Bot commented Jun 3, 2026 •

edited

Loading

mergify Bot commented Jul 6, 2026 •

edited

Loading

mergify Bot commented Jul 6, 2026 •

edited

Loading