ECKIT-684: Ceph/RADOS backend#304
Open
mcakircali wants to merge 126 commits into
Open
Conversation
… into feature/backend-s3
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a Ceph/RADOS backend expansion in eckit IO, adding higher-level RADOS abstractions (pool/namespace/object), multipart/async handles, a URI manager, and CI/test plumbing to exercise the backend.
Changes:
- Add new RADOS primitives (pool/namespace/object) and higher-level IO (multipart read/write handles, async handle, range/part handle, key/value OMAP wrapper).
- Integrate RADOS URI support via a
URIManager, plus new exception types and updated cluster/context management. - Add/adjust RADOS-focused tests and a dedicated GitHub Actions workflow, and update CMake discovery/build configuration for librados.
Reviewed changes
Copilot reviewed 35 out of 35 changed files in this pull request and generated 19 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/io/test_radoshandle.cc | Updates RADOS handle tests to new object/pool/namespace model and multipart handles |
| tests/io/test_rados_kv.cc | Adds unit test coverage for RADOS OMAP key/value wrapper |
| tests/io/rados-performance.cc | Updates performance test to multipart handles and explicit pool/object usage |
| tests/io/CMakeLists.txt | Updates test conditions/labels and adds KV test target |
| src/eckit/io/rados/RadosURIManager.h | Adds URI manager interface for rados: scheme |
| src/eckit/io/rados/RadosURIManager.cc | Implements URI manager (exists/handles/asString) for rados: scheme |
| src/eckit/io/rados/RadosPool.h | Introduces pool abstraction and lifecycle helpers |
| src/eckit/io/rados/RadosPool.cc | Implements pool parsing and destruction semantics |
| src/eckit/io/rados/RadosNamespace.h | Introduces namespace abstraction and listing helpers |
| src/eckit/io/rados/RadosNamespace.cc | Implements namespace parsing, listing, and destruction |
| src/eckit/io/rados/RadosObject.h | Refactors object identity to pool/namespace/oid and adds handle factories |
| src/eckit/io/rados/RadosObject.cc | Implements object parsing, lifecycle helpers, and handle factory methods |
| src/eckit/io/rados/RadosPartHandle.h | Adds range-read DataHandle implementation for RADOS objects |
| src/eckit/io/rados/RadosPartHandle.cc | Implements RADOS range reads via rados_read |
| src/eckit/io/rados/RadosMultiObjWriteHandle.h | Adds multipart writer handle (optionally async) |
| src/eckit/io/rados/RadosMultiObjWriteHandle.cc | Implements multipart write logic + metadata xattrs |
| src/eckit/io/rados/RadosMultiObjReadHandle.h | Adds multipart reader handle |
| src/eckit/io/rados/RadosMultiObjReadHandle.cc | Implements multipart reads via MultiHandle |
| src/eckit/io/rados/RadosKeyValue.h | Adds OMAP-backed key/value API wrapper |
| src/eckit/io/rados/RadosKeyValue.cc | Implements OMAP CRUD + key listing |
| src/eckit/io/rados/RadosAsyncKeyValue.h | Adds async variant of key/value wrapper (batched AIO) |
| src/eckit/io/rados/RadosAsyncKeyValue.cc | Implements async/batched OMAP operations |
| src/eckit/io/rados/RadosAsyncHandle.h | Adds async write handle built on top of RadosHandle |
| src/eckit/io/rados/RadosAsyncHandle.cc | Implements AIO-based write/flush behavior |
| src/eckit/io/rados/RadosException.h | Adds RADOS-specific exception types |
| src/eckit/io/rados/RadosException.cc | Implements RADOS-specific exception types |
| src/eckit/io/rados/RadosHandle.h | Refactors single-object handle API and adds seek/size semantics |
| src/eckit/io/rados/RadosHandle.cc | Updates read/write path, adds rados_write_full first-write behavior, seek/size |
| src/eckit/io/rados/RadosCluster.h | Refactors cluster ioctx caching to include namespaces + adds OMAP/AIO helpers |
| src/eckit/io/rados/RadosCluster.cc | Implements namespace-aware ioctx cache, OMAP/AIO RAII helpers, pool listing, etc. |
| src/eckit/eckit_config.h.in | Adds eckit_HAVE_RADOS_ADMIN config define |
| src/eckit/CMakeLists.txt | Wires new RADOS source files and uses imported target Ceph::RADOS |
| CMakeLists.txt | Updates RADOS option package name and adds RADOS_ADMIN feature flag |
| cmake/FindRADOS.cmake | Modernizes find-module and adds imported target Ceph::RADOS |
| .github/workflows/ci-rados.yml | Adds dedicated CI workflow spinning up a Ceph demo cluster and running RADOS tests |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
simondsmart
requested changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Contributor Declaration
By opening this pull request, I affirm the following:
🌦️ >> Documentation << 🌦️
https://sites.ecmwf.int/docs/dev-section/eckit/pull-requests/PR-304