feat(io): let FileIO wrap a caller-provided filesystem operator - #618
Conversation
Embedders that manage their own storage backend today have no way to hand it to FileIO: Storage::build is a closed match over registered schemes. FileIOBuilder::with_operator accepts a prebuilt operator (for example a customized local-filesystem service) and short-circuits the scheme dispatch through a new Storage::Custom variant. Paths reach the operator in scheme-relative form, exactly as the built-in local-filesystem backend receives them, so the operator's root decides what they resolve against.
|
cc @JingsongLi — this is the minimal Rust analog of Java Paimon's configurable FileIO ( |
|
|
The hook resolved every path with the local-filesystem rules, which silently mangle scheme'd paths (s3://bucket/a -> 3://bucket/a). Rename it with_fs_operator, document the filesystem contract, and reject scheme'd paths in Storage::CustomFs instead of misresolving them; object-store operators need the bucket/scheme resolution the built-in backends perform, which this hook deliberately does not reimplement. Adds resolution and rejection tests.
|
Thanks @JingsongLi — you're right, the hook resolved every path with the filesystem rules, so a scheme'd path like |
jordepic
changed the title
|
Sorry for the ping here @JingsongLi ! Would you mind taking a look at this one? Thank you! Also, I don't yet have an apache email but I'd love to join the Paimon slack! I'm doing a lot of work here for |
…fest * upstream/main: fix(table): skip unrelated BLOB and vector files in row-range reads (apache#629) feat(datafusion): support database SQL statements (apache#627) fix(datafusion): apply dynamic options to vector search (apache#623) feat(io): let FileIO wrap a caller-provided filesystem operator (apache#618) feat(c): build a catalog-free Table from a resolved schema JSON (apache#595) feat: support batch incremental diff reads (apache#510) fix(spec): accept Java's long memory-size units and reject overflow (apache#621) fix(table): stop answering COUNT(*) from a placeholder row count (apache#624) feat(datafusion): support alter column (apache#625) fix(spec): validate bucket-key against table schema (apache#620) # Conflicts: # crates/paimon/src/table/data_evolution_reader.rs # crates/paimon/src/table/table_scan.rs
2 participants
Purpose
Linked issue: close #616
Storage::buildresolves the backing opendal operator through a closed scheme match, so an embedding engine cannot supply its own operator (custom service, metrics/caching layer, preconfigured client). Java Paimon exposes this axis through theFileIOinterface, theFileIOLoaderSPI, andCatalogContext'spreferIO/fallbackIO; this is the minimal Rust analog.Brief change log
FileIOBuilder::with_operator(op)stores a prebuiltopendal::Operator; when set, it takes precedence over scheme resolution inbuild.Storage::Custom { op }variant carries it through;create()reuses the fs-relative path handling.Tests
cargo test -p paimon --lib io::(85 tests) passes; the builder path is exercised by the existing FileIO suites plus downstream use in StreamFusion, which drives all its state-table IO through an operator injected here.API and Format
Additive API on
FileIOBuilder; no storage-format change.Documentation
Doc comments on the new builder method.