feat: add HF Storage Bucket support via OpenDAL by davanstrien · Pull Request #3 · davanstrien/polars

davanstrien · 2026-04-02T12:02:01Z

Summary

Add hf:// URL support for Polars cloud writes via OpenDAL, enabling:

df.lazy().sink_parquet("hf://buckets/org/name/data.parquet")

Approach

Follows the same pattern as existing cloud backends (S3/GCS/Azure):

crates/polars-io/src/cloud/hf.rs — URL parsing, token resolution, OpenDAL ObjectStore construction
object_store_setup.rs — 6-line CloudType::Hf match arm calling build_hf()
Feature flag hf propagated through workspace Cargo.toml chain

HF URLs flow through the standard FileSink — no custom sink node, no IR special-casing.

Dependencies

Using local path deps for opendal and object_store_opendal during development. Will swap to published crate versions once apache/opendal#7185 ships.

Test plan

cargo check -p polars-stream --features hf compiles
cargo check -p polars-stream no regression without feature
End-to-end test with real HF bucket (pending OpenDAL release)

🤖 Generated with Claude Code

…la-rs#26919)

…rs#26927)

…_cse` (pola-rs#26931)

…_slice_unchecked` (pola-rs#26928)

…ola-rs#26938) Co-authored-by: gabriel <gabriel.g.robin@airbus.com>

…me (pola-rs#26936)

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

pola-rs#26946)

…26860)

Co-authored-by: moktamd <moktamd@users.noreply.github.com> Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

…oes not begin with base path, or contains '..' (pola-rs#26894)

…6764) Co-authored-by: Simon Lin <simonlin.rqmmw@slmail.me>

…tring (pola-rs#26934)

Co-authored-by: Claude <noreply@anthropic.com>

…26958)

Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

pola-rs#27050)

…#27011)

…ction (pola-rs#27113)

…#27104) Co-authored-by: Orson Peters <orsonpeters@gmail.com>

pola-rs#27087) Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

…s#27103)

… to "line" (pola-rs#27122)

…ola-rs#27102)

…a-rs#27123)

Co-authored-by: Dani Pinyol <dani@avatarcognition.com>

…la-rs#27118) Co-authored-by: Orson Peters <orsonpeters@gmail.com>

…7117)

…s#27144)

Add an ObjectStore implementation for `hf://` URLs backed by OpenDAL's HF service, enabling `sink_parquet("hf://buckets/org/name/file.parquet")` to stream directly to Hugging Face Storage Buckets. The implementation follows the same pattern as existing cloud backends (S3/GCS/Azure): a `build_hf()` function in a new `hf.rs` module constructs the ObjectStore, and `object_store_setup.rs` calls it from the `CloudType::Hf` match arm. HF URLs flow through the standard FileSink path with no custom sink node or IR special-casing. New files: - crates/polars-io/src/cloud/hf.rs — URL parsing, token resolution, OpenDAL ObjectStore construction Feature flag: `hf` (opt-in, propagated through the workspace) Dependencies: opendal + object_store_opendal (local path deps for now, will switch to published crate versions once apache/opendal#7185 ships) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Point opendal and object_store_opendal at kszucs/opendal@4c70bd8 (hf-revamp branch) which uses published hf-xet 1.5.0 from crates.io. Once apache/opendal#7185 merges and publishes, these become simple version deps (e.g. opendal = "0.56"). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Manual dispatch workflow that builds Linux x64 and ARM64 wheels with the hf feature enabled via maturin. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

apache/opendal#7185 merged today — HF backend XET support is now on upstream main. Pin to commit 8d3dbcc3ef until a release ships. Next: once opendal publishes a release with services-hf, swap these git deps for a version number (opendal = "0.56" or whatever ships). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

apache/opendal v0.56.0 was published to crates.io on 2026-05-01 with the services-hf feature, so we no longer need to pin to a git rev. Drop the [patch.crates-io] entries and bump the workspace dep declarations from 0.55.0 to 0.56.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

opendal 0.56's HfBackend sits under a 5-deep stack of accessor layers (TypeErase / Correctness / Complete / Simulate / ErrorContext), and the dist-release profile overflows the default 128 query-depth limit when computing async fn body layouts (e.g. HfBackend::create_dir). Gate the bump to feature = "hf" so non-HF builds are unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Kevin-Patyk and others added 30 commits March 13, 2026 21:35

fix(python): Allow DataTypeExpr in pl.lit() (pola-rs#26740)

620e724

fix: Fix panic on lazy concat->filter->slice with CSPE (pola-rs#26907)

3ee3f1b

docs: Query Profiler addition to User Guide (pola-rs#26623)

c5825d6

build(rust): Bump up numpy and pyo3 to 0.28 (pola-rs#26743)

1a6e6b9

fix: Propagate null in min_by / max_by for all-null by groups (po…

2b825ad

…la-rs#26919)

fix: Fix error passing Series of dates to business functions (pola-…

b38f5a7

…rs#26927)

test(python): Set stricter maintain_order in `test_schema_row_index…

2e83690

…_cse` (pola-rs#26931)

fix: Default engine as streaming for collect_batches (pola-rs#26932)

72a083f

fix: Fix the loop bounds in `BitmapBuilder::extend_each_repeated_from…

106bd21

…_slice_unchecked` (pola-rs#26928)

fix(python): Fix search_sorted typing when used with list[float] (p…

a13b8c9

…ola-rs#26938) Co-authored-by: gabriel <gabriel.g.robin@airbus.com>

fix: Fix panic on upsample() with group_by parameter on empty DataFra…

910644f

…me (pola-rs#26936)

build: Bump lz4_flex from 0.12.0 to 0.12.1 (pola-rs#26940)

cbad9d6

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

fix: Fix ColumnNotFound due to projection between filter/cache in CSPE (

3dad3f0

pola-rs#26946)

fix: Follow-up on streaming range-join PR (pola-rs#26944)

79a5333

feat: Support casting Duration to String in ISO 8601 format (pola-rs#…

3f96e7f

…26860)

perf: Optimize .replace() from a single value (pola-rs#26948)

c14c58b

feat: Support Decimal32/64 in scan_parquet (pola-rs#26941)

b448902

fix: Preserve height when unnesting empty struct columns (pola-rs#26947)

1128e2b

Co-authored-by: moktamd <moktamd@users.noreply.github.com> Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

feat: Support Delta deletion vectors in scan_delta (pola-rs#26867)

2f06807

perf: Drop unused filter column above cache (pola-rs#26955)

91a532b

perf: Native streaming forward_fill (pola-rs#26922)

7b2c704

feat: Error if PartitionBy path provider returns absolute path that d…

e264a11

…oes not begin with base path, or contains '..' (pola-rs#26894)

feat: Truncate large binary/utf8 Parquet statistics values (pola-rs#2…

1bd62eb

…6764) Co-authored-by: Simon Lin <simonlin.rqmmw@slmail.me>

feat(python): Raise ChronoFormatWarning when using %f time format s…

5a30568

…tring (pola-rs#26934)

fix: Raise for duplicate columns in over() (pola-rs#26968)

cebe499

Co-authored-by: Claude <noreply@anthropic.com>

perf: Streaming native backward_fill (pola-rs#26967)

6eaca5a

perf: Lower index_of to streaming engine (pola-rs#26923)

4c4c029

chore: Polars version (pola-rs#26971)

7fc9f18

feat: Use UUIDv7 for sink_iceberg directory name generation (pola-rs#…

175208e

…26958)

fix: Informative error for multi-quantile in group_by (pola-rs#26957)

b6ae115

Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

MarcoGorelli and others added 21 commits March 30, 2026 14:48

chore(python): Improve internal typing ahead of using ty / pyrefly (

b535c62

pola-rs#27050)

perf: Take into account size per row in join sampling (pola-rs#27098)

0a5a8a0

fix: Preserve casts for horizontal ops with untyped literals (pola-rs…

89b1646

…#27011)

docs: Put first-time contribution requirements in its own linkable se…

4cbbdfa

…ction (pola-rs#27113)

fix: Ignore ddof parameter in rolling_corr and deprecate (pola-rs…

58a8364

…#27104) Co-authored-by: Orson Peters <orsonpeters@gmail.com>

fix: Apply scalar bound in clip when the Series bound contains nulls (

8bdbabb

pola-rs#27087) Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

refactor(rust): Add memory usage tracking to global allocator (pola-r…

e7b9616

…s#27103)

feat(python): Change default scan/read_lines column name from "lines"…

67d0022

… to "line" (pola-rs#27122)

docs: Make the files used in docs available locally (pola-rs#27121)

bedd896

perf: Use delta stats for mixed hive and non-hive predicate pushdown (p…

264dc83

…ola-rs#27102)

fix: Fix incorrect IO metrics on multi-phase streaming execution (pol…

659815c

…a-rs#27123)

perf: Remove unused expression sorts (pola-rs#27075)

21b24fc

chore(python): Add None & Dataframe to FrameInitTypes (pola-rs#27126)

bdb989d

Co-authored-by: Dani Pinyol <dani@avatarcognition.com>

fix(python): Address a potential overflow in from_epoch scaling (po…

a4d366f

…la-rs#27118) Co-authored-by: Orson Peters <orsonpeters@gmail.com>

chore(python): Make internal typing more precise (part ii) (pola-rs#2…

c05981b

…7117)

refactor: Simplify pyarrow scan and process in batches (pola-rs#26982)

c508c56

fix: Skip extension types for min/max in describe (pola-rs#27120)

27694e4

fix: Output SVG if output_path ends with '.svg' in show_graph (pola-r…

a15b6d9

…s#27144)

chore: Update nightly Rust compiler version (pola-rs#27145)

67422be

feat: Allow group_by() without key exprs (pola-rs#27141)

a06064c

github-actions Bot added enhancement New feature or request python rust title needs formatting labels Apr 2, 2026

davanstrien and others added 5 commits April 7, 2026 07:23

ci: add wheel build workflow with hf feature

ae14881

Manual dispatch workflow that builds Linux x64 and ARM64 wheels with the hf feature enabled via maturin. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add HF Storage Bucket support via OpenDAL#3

feat: add HF Storage Bucket support via OpenDAL#3
davanstrien wants to merge 99 commits into
feature/hf-bucket-sinkfrom
hf-opendal-sink

davanstrien commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

davanstrien commented Apr 2, 2026

Summary

Approach

Dependencies

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants