librdmacm: extend rsocket for Redis, iperf3, memcached and more Linux APIs by BatshevaBlack · Pull Request #1717 · linux-rdma/rdma-core

BatshevaBlack · 2026-03-17T08:53:53Z

Extend the rsocket implementation in librdmacm so that applications such as Redis, iperf3, and memcached can use rsocket transparently via LD_PRELOAD (librspreload), and so rsocket aligns with more standard Linux socket and I/O behavior.

Motivation
The rsocket library did not fully support several POSIX/Linux interfaces (epoll, select, accept4, sendfile, fcntl64, and various socket options). Applications that rely on these either failed or fell back to TCP. This change extends the rsocket implementation to implement or fix those interfaces, so the preload can intercept them and route traffic over RDMA.

Changes
fix rpoll timeout handling and select,
fix cm_svc_run
add accept4, dup, fcntl64, sendfile64;
fix rfcntl
extend getsockopt/setsockopt;
fix SOCK_STREAM/SOCK_DGRAM handling and connect service/TCP behavior;
adjust wake-up timeout from rpoll.

shefty

Overall, looks good. A few minor comments, though one asks to revert a change request I made on the prior review. :/ My bad.

BatshevaBlack · 2026-03-17T21:42:32Z

@shefty
I resolved all comments
thanks!

shefty

All my comments have been addressed. Thanks!

BatshevaBlack · 2026-03-19T07:44:21Z

@shefty thanks!
Will you merge it? or what is the next step?

rleon · 2026-03-19T10:12:54Z

@shefty thanks! Will you merge it? or what is the next step?

I will just perform some sanity review and will merge.

Thanks

rleon · 2026-03-19T10:25:53Z

 			  void *optval, socklen_t *optlen);
 	int (*fcntl)(int socket, int cmd, ... /* arg */);
+#if (!defined(_FILE_OFFSET_BITS) || _FILE_OFFSET_BITS != 64) && \
+	(!defined(RDMA_PRELOAD_HAVE_64) || RDMA_PRELOAD_HAVE_64)


why do we need this extra "#if ..."?

We need both: non-LFS vs LFS (_FILE_OFFSET_BITS) and whether libc has fcntl64/sendfile64. I’ll combine those in one CMake flag and use a single #if in preload.c so we don’t duplicate the logic in many places.

rleon · 2026-03-19T10:26:45Z

+			rfcntl(fd, cmd) : real.fcntl64(fd, cmd);
+		break;
+	case F_DUPFD:
+	/*case F_DUPFD_CLOEXEC:*/


This is legacy from the original fcntl commit, there’s no documented reason.

rleon · 2026-03-19T10:28:25Z

 			*optlen = sizeof(int);
 			break;
+		case TCP_INFO:
+			//TODO: support other tcp_info fields.


We implemented TCP_INFO incrementally by demand, not as a full TCP stack mirror—that’s why the TODO calls out other fields.

in case of timeout which causes poll to return, clear all signals that arrived by calling rs_poll_exit. Signed-off-by: Batsheva Black <bblack@nvidia.com>

Without the user_fds mapping, select() could set bits for internal fds instead of the user fds the application passed in, so the wrong (or no) sockets were reported as ready. Keep the list of the fds that are sent to poll in order to know which fd belongs to each rfd when returning the revents to the fds list. Signed-off-by: Batsheva Black <bblack@nvidia.com>

The accept4 implementation extends accept to support the additional atomic flag-setting functionality provided by accept4. Signed-off-by: Batsheva Black <bblack@nvidia.com>

Add preload interception for fcntl64 so rsocket file descriptors support the same flag semantics as the glibc fcntl64 API. Signed-off-by: Batsheva Black <bblack@nvidia.com>

getsockopt: TCP_INFO, TCP_CONGESTION, SO_BROADCAST & IP_TOS. setsockopt: IP_TOS & TCP_CONGESTION. Signed-off-by: Batsheva Black <bblack@nvidia.com>

rfcntl keeps the files flags all in the fd_flags argument. Adding the new field fs_flags to the rs struct allows the fcntl function to keep the file status flags separately from the file descriptor flags. Signed-off-by: Batsheva Black <bblack@nvidia.com>

Add preload interception for sendfile64 so applications using the 64-bit offset sendfile64 API work correctly with rsocket file descriptors. Signed-off-by: Batsheva Black <bblack@nvidia.com>

Add preload interception for dup so that duplicating an rsocket file descriptor produces another rsocket fd that refers to the same connection. Signed-off-by: Batsheva Black <bblack@nvidia.com>

Previously we only added it when rconnect() returned EINPROGRESS. Now also add when connect succeeds so the progress thread can drive state and handle disconnects. Signed-off-by: Batsheva Black <bblack@nvidia.com>

The changes to rpoll to use a signaling fd to wake up blocked threads, combined with suspending polling while rsockets states may be changing _should_ prevent any threads from blocking indefinitely in rpoll() when a desired state change occurs. We periodically wake up any polling thread, so that it can recheck its rsocket states. The sleeping interval was set to an arbitrary value of 5 seconds, this interval is too long for apps that request a connection and are dependent on the thread waking up, so it's changed now to 0.5 seconds, but can be overridden using config files. Signed-off-by: Batsheva Black <bblack@nvidia.com>

Updated type checks to identify socket types even when additional flags are present in the type field. Changed the comparison to use bitwise AND for more accurate detection. Signed-off-by: Batsheva Black <bblack@nvidia.com>

jgunthorpe · 2026-04-10T13:42:54Z

/azp run

azure-pipelines · 2026-04-10T13:43:04Z

Azure Pipelines successfully started running 1 pipeline(s).

The cmake detection test for fcntl64/sendfile64 was clearing CMAKE_REQUIRED_DEFINITIONS without adding -D_GNU_SOURCE, causing the symbols to be invisible in glibc headers and the test to fail. Signed-off-by: Batsheva Black <bblack@nvidia.com>

The check_c_source_compiles test inherited -Wredundant-decls -Werror from CMAKE_C_FLAGS, causing it to wrongly conclude fcntl64/sendfile64 are not in headers. Remove the fragile compile test and infer IN_HEADER directly from HAVE_LFS_WRAPPER_SYMS. Signed-off-by: Batsheva Black <bblack@nvidia.com>

BatshevaBlack mentioned this pull request Mar 17, 2026

librdmacm: extend rsocket for Redis, iperf3, memcached and more Linux APIsRsocket upstream #1702

Closed

shefty reviewed Mar 17, 2026

View reviewed changes

Comment thread librdmacm/preload.c Outdated

Comment thread librdmacm/preload.c Outdated

Comment thread librdmacm/preload.c Outdated

BatshevaBlack force-pushed the rsocket_upstream_no_epoll branch from 77bbfe7 to 3317462 Compare March 17, 2026 21:40

shefty approved these changes Mar 17, 2026

View reviewed changes

rleon reviewed Mar 19, 2026

View reviewed changes

Comment thread librdmacm/CMakeLists.txt Outdated

rleon reviewed Mar 19, 2026

View reviewed changes

Comment thread librdmacm/CMakeLists.txt Outdated

rleon reviewed Mar 19, 2026

View reviewed changes

Comment thread librdmacm/preload.c Outdated

rleon reviewed Mar 19, 2026

View reviewed changes

BatshevaBlack force-pushed the rsocket_upstream_no_epoll branch 7 times, most recently from 3317462 to 1b81b9d Compare March 22, 2026 07:56

BatshevaBlack added 10 commits March 22, 2026 10:55

librdmacm: Fix rpoll in case of timeout

a174774

in case of timeout which causes poll to return, clear all signals that arrived by calling rs_poll_exit. Signed-off-by: Batsheva Black <bblack@nvidia.com>

librdmacm: Add support for accept4 function

add6310

The accept4 implementation extends accept to support the additional atomic flag-setting functionality provided by accept4. Signed-off-by: Batsheva Black <bblack@nvidia.com>

librdmacm: Add support for fcntl64

85f9747

Add preload interception for fcntl64 so rsocket file descriptors support the same flag semantics as the glibc fcntl64 API. Signed-off-by: Batsheva Black <bblack@nvidia.com>

librdmacm: Add support to more optnames in getsockopt, setsockopt

4964485

getsockopt: TCP_INFO, TCP_CONGESTION, SO_BROADCAST & IP_TOS. setsockopt: IP_TOS & TCP_CONGESTION. Signed-off-by: Batsheva Black <bblack@nvidia.com>

librdmacm: Add support for sendfile64

ea89b14

Add preload interception for sendfile64 so applications using the 64-bit offset sendfile64 API work correctly with rsocket file descriptors. Signed-off-by: Batsheva Black <bblack@nvidia.com>

librdmacm: Add support for dup

3c3414e

Add preload interception for dup so that duplicating an rsocket file descriptor produces another rsocket fd that refers to the same connection. Signed-off-by: Batsheva Black <bblack@nvidia.com>

librdmacm: Add rsocket to connect service on success too

6de1340

Previously we only added it when rconnect() returned EINPROGRESS. Now also add when connect succeeds so the progress thread can drive state and handle disconnects. Signed-off-by: Batsheva Black <bblack@nvidia.com>

librdmacm: Fix SOCK_STREAM and SOCK_DGRAM types

87c7aaa

Updated type checks to identify socket types even when additional flags are present in the type field. Changed the comparison to use bitwise AND for more accurate detection. Signed-off-by: Batsheva Black <bblack@nvidia.com>

BatshevaBlack force-pushed the rsocket_upstream_no_epoll branch from 1b81b9d to 945db75 Compare March 30, 2026 06:46

BatshevaBlack force-pushed the rsocket_upstream_no_epoll branch from 945db75 to 87c7aaa Compare April 13, 2026 10:39

BatshevaBlack added 2 commits April 14, 2026 17:50

rleon merged commit 5e7ab6b into linux-rdma:master Apr 15, 2026
14 checks passed

BatshevaBlack deleted the rsocket_upstream_no_epoll branch April 15, 2026 10:21

Conversation

BatshevaBlack commented Mar 17, 2026

Uh oh!

shefty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BatshevaBlack commented Mar 17, 2026

Uh oh!

shefty left a comment

Choose a reason for hiding this comment

Uh oh!

BatshevaBlack commented Mar 19, 2026

Uh oh!

rleon commented Mar 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rleon Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

BatshevaBlack Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

rleon Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

BatshevaBlack Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

rleon Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

BatshevaBlack Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

jgunthorpe commented Apr 10, 2026

Uh oh!

azure-pipelines Bot commented Apr 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants