UCT/IB/MLX5: harden CoCo RC runtime checks#11588
Draft
nbellalou wants to merge 13 commits into
Draft
Conversation
Add configure checks for the rdma-core and kernel APIs used by the CoCo control-object path. The checks cover CC_DMA_BOUNCE detection, parent-domain attributes, CQ parent-domain support, dma-heap headers and DEVX dmabuf UMEM registration so the new code is compiled only when the headers provide the required interfaces.
Detect devices that report IBV_DEVICE_CC_DMA_BOUNCE during MD open and allocate a CoCo parent domain for UCX-owned RDMA control objects. Keep the existing PD as the default path and expose helpers that select the control PD only when CoCo bounce DMA is active.
Create verbs QPs, CQs and SRQs through the CoCo control PD when the device requires bounce DMA. Use the extended CQ creation path with parent-domain attributes in CoCo mode and make the RC verbs capability probe follow the same allocation rules.
Pass CoCo allocation attributes to mlx5 parent domains and allocate DEVX UMEM-backed control buffers from system_cc_shared when CC_DMA_BOUNCE is reported. Route mlx5 tag-matching control QPs through the control PD and reject mlx5 transports or modes that are not made CoCo control-object safe in this milestone.
Keep DEVX doorbell record bookkeeping outside memory exposed to an untrusted CoCo device. Register only the doorbell record page with DEVX, while keeping UCX metadata in private memory.
Contributor
Author
|
🤖 Starting review — findings will be posted here when done. |
Fix CoCo MD error cleanup, quiet expected unsupported probes, share the cc-dma-bounce transport gate, and add behavioral DEVX dbrec coverage. Also apply fork handling to the dmabuf-backed shared control-object mapping path.
Gate CoCo-capable memory domains to the rc_mlx5 profile and keep unsupported IB transports hidden when cc_dma_bounce is active. Fail closed on unsafe rc_mlx5 configuration paths, force safe effective defaults, and mask RMA/atomic capabilities until the later MR and mkey milestones harden them. Document the current CoCo security properties and add gtest coverage for the policy/query/config guards.
Add CoCo-only shared allocation bookkeeping, UMEM authority tracking, and MR/MKey bound validation for mlx5 DEVX memory paths. Keep device-visible allocation metadata private and fail closed for CoCo mkey import/export until bounded metadata is available. Extend test_coco_hardening gtest coverage for shared allocation zeroing/scrubbing, UMEM duplicate and widening checks, MR/MKey bounds, permission narrowing, and the non-CoCo path.
Add CoCo CQ/QP/RMP request records and private object registries for DEVX lifecycle validation. Validate DEVX output length, status, and ID uniqueness before publishing CQN/QPN/RMPN into UCX state. Reject unsupported CoCo object profiles before creation. Add gtest coverage for guarded short outputs, duplicate IDs, invalid object profiles, and wrapper publish-order checks.
Add a private CoCo RC QP registry with poison state and CQ binding validation. Record TX shadow metadata before ringing the doorbell, then validate requester CQEs against the private registry before trusting completion authority. Retire prior unsignaled WQEs when a signaled CQE advances hw_ci. Add gtest coverage for registry validation, poison handling, TX shadow state, and TX CQE replay/rebind rejection.
Track CoCo cyclic SRQ postings in private state and validate RX CQEs against private QP/SRQ ownership before exposing descriptors to upper layers. Validate RX error CQEs against private SRQ shadow state and apply deterministic RX-only poison scope for untrusted completions. Keep TX error CQE mapping covered by the private TX shadow helper. Add gtest coverage for SRQ shadow replay/stale checks, RX CQE validation, error CQE validation, and integration source-shape ordering.
Add public Confidential Computing RDMA hardening documentation for the UCX IB/mlx5 CoCo path. The document summarizes policy scope, the supported RC mlx5 profile, security and non-security properties, private UCX state, DEVX output validation, CQE validation, the taint contract, violation handling, and non-CoCo invariance. Move the production-facing documentation from docs/hardening to docs/confidential-computing and drop node-specific validation notes from the commit. Update the source-shape gtest allowlist to read the new document while keeping checks that raw CoCo CQE and DEVX output reads remain constrained to reviewed validators.
nbellalou
force-pushed
the
coco/devx-runtime-hardening
branch
from
July 1, 2026 09:58
ed7a151 to
3b19029
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Draft PR. This should be merged after PR #11587 and PR #11586.
Why?
The runtime RC path must not trust CQEs or queue metadata supplied by an untrusted NIC. This PR completes the runtime fail-closed checks after the control-object and DEVX validator PRs.
How?