Add optional S3-over-RDMA (NVIDIA cuObject) data plane to remote I/O

[harshavardhana]

Description

Adds an opt-in S3-over-RDMA data plane to KvikIO's remote I/O, backed by NVIDIA cuObject (libcuobjclient). For RDMA-capable S3 endpoints (e.g. MinIO AIStor), RemoteHandle::read()/pread() register the destination buffer (host or device memory) with cuObject, mint an RDMA descriptor, and issue a body-less range GET carrying the SigV4-signed x-amz-rdma-token header; the endpoint RDMA-writes the payload directly into the buffer, bypassing the HTTP body and the CPU copy. The HTTP control plane (range, auth) still flows through the existing libcurl path, so the change is additive.

How it fits KvikIO

• dlopen shim, like cuFile. cuObject's client is a C++ class, so it isn't loaded directly the way libcufile is. A small C-ABI shim (cpp/cuobj_shim/cuobj_shim.cpp) wraps it and is built into the optional libkvikio_cuobj_shim.so only when cuobjclient.h is found. The new cuObjAPI singleton dlopens that shim at runtime (mirroring cuFileAPI), so cuObject stays an optional runtime dependency and KvikIO links neither it nor the shim. Override the path with KVIKIO_CUOBJ_SHIM.
• Opt-in, no behavior change when unset. Activation requires KVIKIO_REMOTE_RDMA=on and a successful cuObject connection (S3Endpoint::supports_rdma()). A 501 reply (x-amz-rdma-reply) fails loudly rather than silently falling back, since RDMA must be explicit. No Python API change.
• Single-shot transfer hooked at read() (a discard write-callback; the payload arrives out of band), so it does not interfere with the libcurl streaming callbacks or the device bounce buffer.

Files

• cpp/cuobj_shim/cuobj_shim.cpp — C-ABI shim over cuObjClient (built optionally).
• cpp/include/kvikio/shim/cuobj.hpp, cpp/src/shim/cuobj.cpp — cuObjAPI dlopen singleton + is_cuobj_available().
• cpp/src/remote_handle.cpp, cpp/include/kvikio/remote_handle.hpp — RemoteEndpoint::supports_rdma()/setopt_rdma(), S3Endpoint overrides, single-shot read_rdma() + RDMA reply check.
• cpp/CMakeLists.txt — cuObject detection + optional shim library.
• docs/source/remote_file.rst — usage.

Draft for design discussion per CONTRIBUTING (new feature). No tracking issue yet — happy to file one and align on the env-var vs explicit-API surface and the dlopen-shim approach before finalizing.

Validated

Built and proven end-to-end on hardware: NVIDIA H200 client → RDMA-capable MinIO AIStor endpoint over 400G RoCE (toolkit cuObject 1.2.0 / cuFile 1.18.0). libkvikio_cuobj_shim.so builds when cuObject is present; RemoteHandle::read() with KVIKIO_REMOTE_RDMA=on returns byte-identical data (64 MiB) for both:

• host (CPU) memory — byte-identical over RDMA, and
• device (GPU) memory — GPUDirect — the device buffer is registered directly with cuObject (no bounce buffer) and filled over RDMA via PCIe P2P / dmabuf; byte-identical on the H200.

supports_rdma() correctly gates on KVIKIO_REMOTE_RDMA + a live cuObject connection, and a missing/501 x-amz-rdma-reply fails loudly. With cuObject absent, the shim is not built and is_cuobj_available() returns false; with KVIKIO_REMOTE_RDMA unset, the standard libcurl data plane is used unchanged.

(Minor: for non-AWS endpoints, construct S3Endpoint(bucket, object) + AWS_ENDPOINT_URL rather than RemoteHandle::open(http_url, S3), since S3Endpoint::is_url_valid only matches amazonaws.com hosts for the explicit S3 type.)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[kingcrimsontianyu]

Thanks for the PR. I'll take a closer look. If possible, could you share any performance numbers for a RDMA-S3 vs regular S3 comparison?