NVIDIA: SAUCE: Patch NVMe/NVMeoF driver to support GDS on Linux 7.0 Kernel by sourabgupta3 · Pull Request #373 · NVIDIA/NV-Kernels

sourabgupta3 · 2026-04-16T17:56:52Z

With this change, the NVMe and NVMeoF driver would be enabled to
support GPUDirectStorage(GDS). NVMe driver introduced a way to
use the blk_rq_dma_map API to DMA map requests instead of
scatter gather lists. With these changes, GDS path also adopts
a similar framework where we introduce blk based
APIs(nvfs_blk_rq_dma_map_iter_start and nvfs_blk_rq_dma_map_iter_next)
to map a DMA request. The NVMeoF path remains the same as previous releases.

Signed-off-by: Sourab Gupta sougupta@nvidia.com
Reviewed-by: Kiran Modukuri kmodukuri@nvidia.com

nirmoy · 2026-04-16T18:04:05Z

@sourabgupta3 I would squash these two patches together as these are not upstreamed patches so keeps patch list small.

sourabgupta3 · 2026-04-16T18:07:38Z

@nirmoy For context, on 6.17 these two went as separate commits as we discovered a bug later, that's why I kept it this way to be consistent with 6.17. Do you still want to squash them? I am okay with either option.

jamieNguyenNVIDIA · 2026-04-16T18:10:33Z

@nirmoy For context, on 6.17 these two went as separate commits as we discovered a bug later, that's why I kept it this way to be consistent with 6.17. Do you still want to squash them? I am okay with either option.

I think you should still squash them. It avoids leaving in a known bugged commit and would mean that you'd only need to pick a single patch into the next HWE kernel.

nvmochs · 2026-04-16T18:11:12Z

@nirmoy For context, on 6.17 these two went as separate commits as we discovered a bug later, that's why I kept it this way to be consistent with 6.17. Do you still want to squash them? I am okay with either option.

6.17-HWE will be going away in a few months; this LTS will be supported for 10+ years. We should squash.

sourabgupta3 · 2026-04-16T18:26:38Z

@jamieNguyenNVIDIA @nirmoy Done!

nvmochs · 2026-04-16T23:31:24Z

I confirmed that the squashed patches contain the same content as the version in 6.17-HWE with these differences:

IOD_NVFS_IO = 1U << 31 to avoid enum conflicts
u8 → u32 flags to accommodate the new bit width
Local iod declaration in nvme_pci_prp_iter_next due to changed function scope
blk_rq_dma_map_iter_next API dropped dma_state parameter
nvme_pci_setup_data_prp restructured around nvme_pci_prp_save_mapping

No issues or concerns from me.

Acked-by: Matthew R. Ochs <mochs@nvidia.com>

@sourabgupta3 - Please also submit this change against the 26.04_linux-nvidia-bos branch (which will be the first "HWE" in 26.04). Thanks!

clsotog · 2026-04-16T23:34:11Z

 	struct nvme_request req;
 	struct nvme_command cmd;
-	u8 flags;
+	u32 flags;


At linux-nvidia-6.17 this u8 flags? is this a new fix?

Yes, this is new as there wasn't any space left to accomodate NVFS flags. But I think I should add this also under #ifdef CONFIG_NVFS.

It's needed because Sourab needed to add IOD_NVFS_IO to nvme_iod_flags and it was maxed out on bits that fit within a single byte. So he expanded the field to a large width (an integer) and used a very high bit number (the last bit of an integer) to avoid future conflicts when rebasing. This is the only place that uses these flags and I confirmed that there are no local variables that make a copy of the field (which would have potentially also required a fixup if they too used a single byte for storage).

thanks for the replies.

nvmochs · 2026-04-16T23:44:37Z

I confirmed that the squashed patches contain the same content as the version in 6.17-HWE with these differences:

IOD_NVFS_IO = 1U << 31 to avoid enum conflicts

u8 → u32 flags to accommodate the new bit width

Local iod declaration in nvme_pci_prp_iter_next due to changed function scope

blk_rq_dma_map_iter_next API dropped dma_state parameter

nvme_pci_setup_data_prp restructured around nvme_pci_prp_save_mapping

No issues or concerns from me.

Acked-by: Matthew R. Ochs <mochs@nvidia.com>

@sourabgupta3 - Please also submit this change against the 26.04_linux-nvidia-bos branch (which will be the first "HWE" in 26.04). Thanks!

@sourabgupta3 Actually, there was 1 nit I had that I forgot about...

There are some instances of trailing whitespace added by this PR. Can these be addressed?

sourabgupta3 · 2026-04-16T23:52:16Z

@nvmochs I cleaned up some now. Does this look okay?

nvmochs · 2026-04-16T23:55:10Z

@nvmochs I cleaned up some now. Does this look okay?

It looks great, no more trailing whitespace.

One more tiny nit, in the commit title, can you change "Linux 6.17 Kernel" -> "Linux 7.0 Kernel"?

sourabgupta3 · 2026-04-16T23:57:31Z

Done @nvmochs

sourabgupta3 · 2026-04-17T00:02:25Z

@nvmochs PR against 26.04_linux-nvidia-bos: #376

clsotog

Acked-by: Carol L Soto <csoto@nvidia.com>

nirmoy · 2026-04-17T12:11:37Z

Acked-by: Nirmoy Das <nirmoyd@nvidia.com>

jamieNguyenNVIDIA

General comment.

nvfs-dma.c, nvfs-rdma.c, and nvfs.h all appear to have space-indented code:

nvfs-dma.c lines 31–47: REGISTER_FUNC and UNREGISTER_FUNC bodies use spaces
nvfs-rdma.c lines 32–47: same functions as above
nvfs.h lines 55–68: nvfs_count_ops() and nvfs_get_ops() bodies use spaces
nvfs-dma.h lines 38–47: nvme_nvfs_unmap_sgls inner block uses spaces

The nvfs-dma.h line 174 (else with space indent) is also in nvme_nvfs_map_data.

Can you change these all to tabs?

jamieNguyenNVIDIA · 2026-04-17T20:10:16Z

+			if (!nvfs_ops->nvfs_blk_rq_dma_map_iter_next(req, dma_dev,
+						&iod->dma_state, iter))
+				return false;
+			iod->dma_vecs[iod->nr_dma_vecs].addr = iter->addr;


I can't tell if this matters or not, but if mempool_alloc() fails here, we'll return and not set iod->dma_vecs[0].addr = iter->addr;. Later, when nvme_nvfs_unmap_prps() is called, it will exit early at the if (!iod->dma_vecs) check without looping through iod->nr_dma_vecs (and calling nvfs_dma_unmap_page()). Could we have a leak because of this?

jamieNguyenNVIDIA · 2026-04-17T20:15:18Z

+		iod->dma_vecs = mempool_alloc(nvmeq->dev->dmavec_mempool, GFP_ATOMIC);
+		if (!iod->dma_vecs)
+			return BLK_STS_RESOURCE;
+		iod->dma_vecs[0].addr = iter->addr;


Similar to the previous comment in the PRP path: if dma_pool_alloc() fails here, we return before writing any SGL entries and before iod->cmd.common.dptr.sgl is populated. When nvme_nvfs_unmap_sgls() is called with nr_descriptors == 0, it will call nvfs_dma_unmap_page() using the addr/len from iod->cmd.common.dptr.sgl, which was never set. Could there be a leak because of this?

jamieNguyenNVIDIA · 2026-04-17T20:18:13Z

+		return 0;
+	}
+
+	return ret;


This return looks to be unreachable. Seems like you can drop the else case.

github-actions · 2026-04-17T20:38:35Z

✅ Patchscan: No Missing Fixes

All cherry-picked commits have been checked — no missing upstream fixes found.

nvmochs · 2026-04-22T00:53:07Z

@sourabgupta3 Can you please review/respond to Jamie's comments and questions above? Thanks!

sourabgupta3 · 2026-04-22T00:55:25Z

@nvmochs I already responded to Jamie's comment last week. I am in the middle of testing the fix.

jamieNguyenNVIDIA · 2026-04-22T00:59:51Z

@nvmochs I already responded to Jamie's comment last week. I am in the middle of testing the fix.

FYI: I don't see responses on this PR.

sourabgupta3 · 2026-04-22T01:03:59Z

@jamieNguyenNVIDIA @nvmochs It looks like my comment is stuck in pending state. This is what I have commented:
Both the comments look valid. Nice catch! Unfortunately, I don't have a setup to test these now. I will work on the fixes next week and submit.

Do you know why could it be in pending state?

jamieNguyenNVIDIA · 2026-04-22T03:11:02Z

Hi @sourabgupta3: looks like there are still spaces instead of tabs at:

nvfs-rdma.c:27
nvfs.h:78–127

My other comments appear to have been addressed.

sourabgupta3 · 2026-04-22T04:14:45Z

@jamieNguyenNVIDIA Addressed other comments as well.

jamieNguyenNVIDIA · 2026-04-22T04:24:14Z

 	if (ret)
 		return -ENOMEM;

+#ifdef CONFIG_NVFS


The spacing still looks off here:

The #ifdef CONFIG_NVFS block added to rdma.c (inside nvme_rdma_dma_map_req) uses mostly spaces where the surrounding code uses tabs:

drivers/nvme/host/rdma.c:1490-1500: #ifdef CONFIG_NVFS { ← 8 spaces bool is_nvfs_io = false; ← 8 spaces ret = nvme_rdma_nvfs_map_data(...); ← 8 spaces if (is_nvfs_io) { ← 8 spaces if (ret) ← tab + 8 spaces (!) goto out_free_table; ← tab + 15 spaces (!) return 0; ← 16 spaces } ← tab } ← 8 spaces #endif req->data_sgl.nents = ... ← tab (surrounding code)

The spacing still looks off here:

The #ifdef CONFIG_NVFS block added to rdma.c (inside nvme_rdma_dma_map_req) uses mostly spaces where the surrounding code uses tabs:

drivers/nvme/host/rdma.c:1490-1500: #ifdef CONFIG_NVFS { ← 8 spaces bool is_nvfs_io = false; ← 8 spaces ret = nvme_rdma_nvfs_map_data(...); ← 8 spaces if (is_nvfs_io) { ← 8 spaces if (ret) ← tab + 8 spaces (!) goto out_free_table; ← tab + 15 spaces (!) return 0; ← 16 spaces } ← tab } ← 8 spaces #endif req->data_sgl.nents = ... ← tab (surrounding code)

@sourabgupta3 - Can you review and reply to this comment from Jamie? (I believe this is the final issue before we plan to merge)

…ernel BugLink: https://bugs.launchpad.net/bugs/2134960 With this change, the NVMe and NVMeoF driver would be enabled to support GPUDirectStorage(GDS). NVMe driver introduced a way to use the blk_rq_dma_map API to DMA map requests instead of scatter gather lists. With these changes, GDS path also adopts a similar framework where we introduce blk based APIs(nvfs_blk_rq_dma_map_iter_start and nvfs_blk_rq_dma_map_iter_next) to map a DMA request. The NVMeoF path remains the same as previous releases. Signed-off-by: Sourab Gupta <sougupta@nvidia.com> Reviewed-by: Kiran Modukuri <kmodukuri@nvidia.com>

github-actions · 2026-04-24T22:54:43Z

PR Validation Report

Patchscan ✅ No Missing Fixes

All cherry-picked commits checked — no missing upstream fixes found.

PR Lint ❌ Errors found

Details

Checking 1 commits...

Cherry-pick digest:
┌──────────────┬───────────────────────────────────────────────┬────────────┬─────────┬───────────────────────────┐
│ Local        │ Referenced upstream / Patch subject           │ Patch-ID   │ Subject │ SoB chain                 │
├──────────────┼───────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 28be36885ae9 │ [SAUCE] patch nvme/nvmeof driver to support g │ N/A        │ N/A     │ sougupta                  │
└──────────────┴───────────────────────────────────────────────┴────────────┴─────────┴───────────────────────────┘

Lint: all checks passed.

PR metadata:
W: PR title missing [<branch>] prefix: "NVIDIA: SAUCE: Patch NVMe/NVMeoF driver to support GDS on Linux 7.0 Kernel"
E: PR targets 26.04_linux-nvidia but body has no 'BugLink:' or 'LP:' https://bugs.launchpad.net/... line

nvmochs · 2026-04-24T23:01:35Z

I reviewed the latest commit, no further issues from me.

Acked-by: Matthew R. Ochs <mochs@nvidia.com>

jamieNguyenNVIDIA · 2026-04-24T23:11:37Z

Acked-by: Jamie Nguyen <jamien@nvidia.com>

@sourabgupta3 : Please apply the same changes to pr#376. Thanks!

sourabgupta3 · 2026-04-24T23:15:12Z

@jamieNguyenNVIDIA I did apply the changes to 376 as well, you don't see it there?

jamieNguyenNVIDIA · 2026-04-24T23:31:49Z

@jamieNguyenNVIDIA I did apply the changes to 376 as well, you don't see it there?

I do see them there -- sorry!

jamieNguyenNVIDIA · 2026-04-25T00:53:12Z

Patched applied: https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-nvidia/+git/resolute/commit/?h=main-next&id=6179a7bd45ef08c22f8a384263ed73bf1e7c57f3

sourabgupta3 force-pushed the sg_26.04_linux-nvidia-nvme-fix branch from 0afef66 to fd03681 Compare April 16, 2026 18:25

clsotog reviewed Apr 16, 2026

View reviewed changes

sourabgupta3 force-pushed the sg_26.04_linux-nvidia-nvme-fix branch 2 times, most recently from 21af88d to 93bd3d6 Compare April 16, 2026 23:49

sourabgupta3 force-pushed the sg_26.04_linux-nvidia-nvme-fix branch from 93bd3d6 to 9582112 Compare April 16, 2026 23:56

clsotog self-requested a review April 17, 2026 05:13

clsotog approved these changes Apr 17, 2026

View reviewed changes

jamieNguyenNVIDIA reviewed Apr 17, 2026

View reviewed changes

sourabgupta3 force-pushed the sg_26.04_linux-nvidia-nvme-fix branch from 9582112 to 825c167 Compare April 17, 2026 20:27

sourabgupta3 force-pushed the sg_26.04_linux-nvidia-nvme-fix branch from 825c167 to 09c15e6 Compare April 22, 2026 01:49

sourabgupta3 force-pushed the sg_26.04_linux-nvidia-nvme-fix branch from 09c15e6 to fb14aa9 Compare April 22, 2026 04:13

jamieNguyenNVIDIA reviewed Apr 22, 2026

View reviewed changes

jamieNguyenNVIDIA requested changes Apr 22, 2026

View reviewed changes

sourabgupta3 force-pushed the sg_26.04_linux-nvidia-nvme-fix branch from fb14aa9 to 28be368 Compare April 24, 2026 22:44

jamieNguyenNVIDIA mentioned this pull request Apr 24, 2026

NVIDIA: SAUCE: Patch NVMe/NVMeoF driver to support GDS on Linux 7.0 Kernel #376

Closed

jamieNguyenNVIDIA closed this Apr 25, 2026

Uh oh!

Conversation

sourabgupta3 commented Apr 16, 2026

Uh oh!

nirmoy commented Apr 16, 2026

Uh oh!

sourabgupta3 commented Apr 16, 2026

Uh oh!

jamieNguyenNVIDIA commented Apr 16, 2026

Uh oh!

nvmochs commented Apr 16, 2026

Uh oh!

sourabgupta3 commented Apr 16, 2026

Uh oh!

nvmochs commented Apr 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvmochs commented Apr 16, 2026

Uh oh!

sourabgupta3 commented Apr 16, 2026

Uh oh!

nvmochs commented Apr 16, 2026

Uh oh!

sourabgupta3 commented Apr 16, 2026

Uh oh!

sourabgupta3 commented Apr 17, 2026

Uh oh!

clsotog left a comment

Choose a reason for hiding this comment

Uh oh!

nirmoy commented Apr 17, 2026

Uh oh!

jamieNguyenNVIDIA left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Patchscan: No Missing Fixes

Uh oh!

nvmochs commented Apr 22, 2026

Uh oh!

sourabgupta3 commented Apr 22, 2026

Uh oh!

jamieNguyenNVIDIA commented Apr 22, 2026

Uh oh!

sourabgupta3 commented Apr 22, 2026

Uh oh!

jamieNguyenNVIDIA commented Apr 22, 2026

Uh oh!

sourabgupta3 commented Apr 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 24, 2026

PR Validation Report

Patchscan ✅ No Missing Fixes

PR Lint ❌ Errors found

Uh oh!

nvmochs commented Apr 24, 2026

Uh oh!

jamieNguyenNVIDIA commented Apr 24, 2026

Uh oh!

sourabgupta3 commented Apr 24, 2026

Uh oh!

jamieNguyenNVIDIA commented Apr 24, 2026

github-actions Bot commented Apr 17, 2026 •

edited

Loading