Skip to content

[#1570] Add iceoryx2-dmabuf crate with SCM_RIGHTS fd sidecar and DMA-BUF wrapper#1572

Open
julienzarka wants to merge 3 commits into
eclipse-iceoryx:mainfrom
julienzarka:iox2-1570-dmabuf-sidecar
Open

[#1570] Add iceoryx2-dmabuf crate with SCM_RIGHTS fd sidecar and DMA-BUF wrapper#1572
julienzarka wants to merge 3 commits into
eclipse-iceoryx:mainfrom
julienzarka:iox2-1570-dmabuf-sidecar

Conversation

@julienzarka

Copy link
Copy Markdown

[#1570] Add iceoryx2-dmabuf - SCM_RIGHTS fd sidecar and typed DMA-BUF wrapper

Closes #1570.

Summary

Adds a new workspace crate iceoryx2-dmabuf that extends iceoryx2 pub/sub
with out-of-band delivery of kernel-owned file descriptors (DMA-BUF frames
from V4L2 ISP, DRM scanout, Vulkan external memory, or generic memfds).

The crate exposes two clearly scoped APIs:

Layer Types Use case
Transport FdSidecarPublisher / FdSidecarSubscriber Any OwnedFd (memfd, eventfd, pipe end, DMA-BUF fd) - no CPU-sync semantics
Payload (feature dma-buf) DmaBufPublisher / DmaBufSubscriber Real DMA-BUF fds; subscriber gets dma_buf::DmaBuf with safe CPU access via MappedDmaBuf::read/write/readwrite

The transport carries fds via SCM_RIGHTS over a per-service Unix-domain
socket, synchronised to the iceoryx2 sample sequence through an
FdSidecarToken user-header. The payload layer is a newtype over the
transport that delegates every DMA_BUF_IOCTL_SYNC to the audited
dma-buf crate (v0.5, by
@mripard) - this PR does not reimplement the
ioctl surface.

Motivation (from #1570)

iceoryx2's typed SHM pool delivers value-type payloads efficiently but
cannot represent kernel-owned DMA-BUF allocations produced by V4L2 ISP,
DRM scanout, or Vulkan external-memory exports. Those frames are
identified by a file descriptor whose numeric value is meaningless
outside the producing process; cross-process transfer requires
SCM_RIGHTS over a Unix-domain socket. iceoryx2-dmabuf adds a thin
sidecar channel that delivers fds in lockstep with the normal iceoryx2
metadata flow, preserving zero-copy semantics end-to-end.

Changes

iceoryx2 (core)

  • Add iceoryx2::port::side_channel::SideChannel trait - a cross-platform
    role marker for out-of-band transports alongside pub/sub. Deliberately
    free of Linux-specific syscall surface so it compiles everywhere.

iceoryx2-dmabuf (new crate)

  • FdSidecarPublisher<S, Meta> / FdSidecarSubscriber<S, Meta> - the
    transport layer. Linux-only at runtime; non-Linux targets compile and
    return UnsupportedPlatform.
  • DmaBufPublisher<S, Meta> / DmaBufSubscriber<S, Meta> - typed
    newtypes over the transport, feature-gated on dma-buf. Accept a
    &dma_buf::DmaBuf on send; yield a dma_buf::DmaBuf on recv.
  • FdSidecarError::DmaBuf(dma_buf::BufferError) - wraps upstream
    BufferError into our non-exhaustive error enum rather than
    re-exporting it.
  • Unix-domain socket accept loop with SO_PEERCRED UID check (feature
    peercred), sequence-number correlation via FdSidecarToken in the
    iceoryx2 sample user-header, and a 50 ms timeout on fd delivery.
  • Injection helpers (feature test-utils) for error-path tests.

benchmarks/dmabuf (new crate)

  • bench_latency - single-publisher/subscriber per-frame latency for
    1080p fd passing (p50/p95/p99).
  • bench_throughput - sustained frames-per-second ceiling.
  • bench_fanout - slowest-consumer p95 across N subscribers.

Cargo features (all off by default except std)

Feature Description Platform
memfd memfd_create-based helpers for tests and examples Linux only
peercred SO_PEERCRED UID verification before fd delivery Linux only
dma-buf DmaBufPublisher / DmaBufSubscriber typed layer Linux only (pulls dma-buf 0.5)
test-utils Injection APIs for error-path integration tests Linux only

Commit structure

[#1570] Add SideChannel trait in iceoryx2::port::side_channel    (+85)
[#1570] Add iceoryx2-dmabuf crate with SCM_RIGHTS fd sidecar ... (+7K)
[#1570] Add iceoryx2-dmabuf benchmarks                           (+0.4K)

Each commit compiles, clippies, and tests in isolation (git bisect green).

Test evidence

macOS aarch64 (Darwin 25.4, cargo 1.88):

  • cargo fmt --check
  • cargo clippy -p iceoryx2-dmabuf --all-features --all-targets -- -D warnings
  • cargo clippy -p iceoryx2-dmabuf --no-default-features --target aarch64-apple-darwin -- -D warnings
  • cargo test -p iceoryx2-dmabuf --features "dma-buf memfd peercred test-utils" ✓ (Linux-gated tests report 0 run)

Linux aarch64 (Ubuntu 25.10, cargo 1.93):

  • cargo fmt --check
  • cargo clippy -p iceoryx2-dmabuf --all-features --all-targets -- -D warnings
  • cargo clippy -p iceoryx2-dmabuf --no-default-features -- -D warnings
  • cargo test -p iceoryx2-dmabuf --features "dma-buf memfd peercred test-utils" ✓ - 28 pass / 0 fail / 2 ignored
  • /dev/dma_heap/system present → real DmaBuf heap round-trip passes with
    DMA_BUF_IOCTL_SYNC firing on both publisher and subscriber

Test suite highlights

  • tests/dmabuf_roundtrip.rs - in-process memfd roundtrip
  • tests/it_fanout.rs - 1 publisher × 3 subscribers × 100 frames
  • tests/it_crash_midsend.rs - producer SIGSTOP between sidecar send
    and iceoryx2 send → subscriber gets NoFdInMessage
  • tests/it_service_gone.rs, tests/it_socket_gone.rs - recovery paths
  • tests/refcount_survival.rs - fd refcount survives N-1 consumer closes
  • tests/it_fd_identity.rs - two-process fstat shows same inode
  • tests/error_paths.rs - TokenMismatch and NoFdInMessage paths
    (test-utils injection)
  • tests/prop_roundtrip.rs - proptest random payload sizes 1 B - 16 MiB
  • tests/it_dmabuf_fd_identity.rs - inode preserved across DmaBuf wrap
  • tests/it_dmabuf_heap_roundtrip.rs - real /dev/dma_heap/system
    allocation, MappedDmaBuf::read with CPU-sync ioctls, byte-pattern
    verification

Examples

  • examples/publish_subscribe_with_fd/{publisher,subscriber}.rs -
    transport layer with memfd payloads.
  • examples/publish_subscribe_dmabuf/{publisher,subscriber}.rs -
    payload layer with dma-heap allocation + MappedDmaBuf::read.

Out of scope for this PR

  • Multi-plane (YUV420/NV12 with separate Y/UV fd) support - single fd per sample only.
  • Async transport - sendmsg/recvmsg are synchronous.
  • sync_file / GPU fence propagation - DMA-BUF only.
  • Windows DXGI_KEYED_MUTEX handles - Linux only.
  • pidfd_getfd(2) alternative transport - future work if benchmarks show benefit.

Checklist

  • Issue exists (#1570)
  • Branch prefix (iox2-1570-dmabuf-sidecar per CONTRIBUTING)
  • Commit prefix [#1570] on every commit
  • Eclipse copyright header on every new .rs file
  • Release notes added in doc/release-notes/iceoryx2-unreleased.md
  • ECA signed
  • CI passes locally on aarch64 Linux and Darwin

Questions for maintainers

  1. Would you prefer one commit (everything bundled) over the current
    three? The trait + crate + benchmarks split keeps the diff reviewable
    in chunks; happy to squash.
  2. Is iceoryx2-dmabuf the right crate name, or should it be e.g.
    iceoryx2-fd-sidecar to reflect that the transport works with
    arbitrary fds and DMA-BUF is one application?
  3. Any preference on whether the dma-buf dependency should be a
    default feature or remain opt-in?

Thanks for the review.

…channel

Introduce a cross-platform role marker trait for out-of-band transports
that travel alongside pub/sub metadata.  The trait is deliberately free
of Linux-specific syscall surface (fd passing, DMA-BUF) so that it
compiles on every target; downstream crates extend it with
platform-specific primitives.

The canonical downstream implementation is `iceoryx2-dmabuf`, which
implements `SideChannel` via a Unix domain socket with `SCM_RIGHTS`
ancillary data for zero-copy DMA-BUF file-descriptor delivery.
…idecar and DMA-BUF wrapper

Introduce a new workspace crate `iceoryx2-dmabuf` that extends iceoryx2
with out-of-band delivery of kernel-owned file descriptors alongside
typed pub/sub metadata samples.

Two clearly scoped APIs coexist in the crate:

* Transport (`FdSidecarPublisher` / `FdSidecarSubscriber`) — passes any
  `OwnedFd` (memfd, eventfd, pipe end, DMA-BUF fd) via `SCM_RIGHTS` over
  a Unix domain socket, synchronised to the iceoryx2 sample sequence
  number through an `FdSidecarToken` user-header.  Linux-only runtime;
  non-Linux targets compile and return `UnsupportedPlatform`.

* Payload (`DmaBufPublisher` / `DmaBufSubscriber`, feature `dma-buf`) —
  newtype wrappers over the transport that accept and yield
  `dma_buf::DmaBuf` values.  CPU-sync ioctls (`DMA_BUF_IOCTL_SYNC`
  start/end) are delegated to the audited upstream `dma-buf` crate
  (v0.5) via `MappedDmaBuf::read/write/readwrite` — `iceoryx2-dmabuf`
  does not reimplement the ioctl surface.

The transport implements `iceoryx2::port::side_channel::SideChannel`
(added in the preceding commit) and also a Linux-specific internal
`FdSideChannel` extension trait that documents the `send_fd` /
`recv_fd_matching` contract.

Cargo features (all off by default except `std`):

* `memfd`      — `memfd_create`-based helpers for tests and examples
* `peercred`   — `SO_PEERCRED` UID verification before fd delivery
* `dma-buf`    — `DmaBufPublisher` / `DmaBufSubscriber` typed layer
                 (Linux only; pulls `dma-buf` 0.5 as optional dep)
* `test-utils` — injection APIs for error-path integration tests

Tested with:

* `cargo test` on Linux aarch64 (OrbStack) and macOS aarch64 — unit,
  integration, and property tests all green (Linux-only tests skip on
  macOS, `/dev/dma_heap/system` heap test skips when unavailable).
* `cargo clippy --all-features --all-targets -- -D warnings` clean on
  both hosts, including `--no-default-features`.
* `cargo fmt --check` clean.
* `cargo doc --no-deps --all-features` emits zero broken-intra-doc-link
  warnings.

Examples:

* `publish_subscribe_with_fd/{publisher,subscriber}.rs` — transport
  layer with memfd payloads.
* `publish_subscribe_dmabuf/{publisher,subscriber}.rs` — payload
  layer with `dma-heap` allocation + `MappedDmaBuf::read`.
Add a `benchmarks-dmabuf` workspace member that measures three
characteristics of the `FdSidecar*` transport from the preceding
`iceoryx2-dmabuf` crate commit:

* `bench_latency`    — single publisher/subscriber end-to-end per-frame
                       latency for 1080p fd passing (p50/p95/p99).
* `bench_throughput` — sustained frames-per-second ceiling with a warm
                       socket buffer.
* `bench_fanout`     — slowest-consumer p95 across 1..N subscribers,
                       exercising the `SCM_RIGHTS` multi-accept path.

The three binaries live under one `benchmarks-dmabuf` crate that picks
the bench to run via `cargo run -p benchmarks-dmabuf --bin <name>`.
A shared `main.rs` parses common flags (frame count, warm-up frames,
subscriber count) so every bench has a uniform CLI.

Linux only at runtime (same constraint as the transport).  Non-Linux
hosts build the crate but the binaries return `UnsupportedPlatform`
immediately.
julienzarka added a commit to julienzarka/iceoryx2 that referenced this pull request May 5, 2026
…ce variant

Captures the design pivot from the standalone sidecar crate (this branch)
to a fully-integrated dmabuf::Service variant per maintainer roadmap
feedback on PR eclipse-iceoryx#1572.

Three deliverables:
- specs/arch-dmabuf-service-variant.adoc — Design C architecture (8
  decisions D1-D8 + Mermaid component diagram + trade-off matrix)
- specs/spec-dmabuf-service-variant.adoc — 49 numbered MUST/SHOULD/MAY
  requirements + commit structure + acceptance criteria
- docs/superpowers/plans/2026-05-05-dmabuf-service-variant.md — 13 TDD
  tasks (0a/0b through 11) with mos-agent dispatch points, including
  Task 5b for back-channel + pool-ack integration in the same PR

Plus archives of the original sidecar PR's spec/plan/PR-message under
iceoryx2-dmabuf/{specs,docs}/ for historical reference.

The new branch (feat/dmabuf-service-variant) will cherry-pick these docs
from this commit then execute Task 0a (architecture validation spike)
before any cal-layer code is written.

Refs eclipse-iceoryx#1570
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add DMA-BUF side channel for zero-copy fd passing via SCM_RIGHTS

1 participant