Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,20 @@ It's possible to build `object_store` for the `wasm32-unknown-unknown` target, h
cargo build -p object_store --target wasm32-unknown-unknown
```

## Disabling `reqwest`

The `aws`, `azure`, `gcp`, and `http` features each bundle a [`reqwest`]-based HTTP transport. To target [`wasm32-wasip1`] (where `reqwest` does not compile) or otherwise supply your own HTTP client, build against the matching `*-base` feature instead:

```
cargo build -p object_store --no-default-features --features aws-base --target wasm32-wasip1
```

See [Disabling `reqwest`] in the crate docs for full details.

[`reqwest`]: https://crates.io/crates/reqwest
[`wasm32-wasip1`]: https://doc.rust-lang.org/rustc/platform-support/wasm32-wasip1.html
[Disabling `reqwest`]: https://docs.rs/object_store/latest/object_store/#disabling-reqwest

## Related Apache Crates

Here are several related crates in different repositories from other Apache projects.
Expand Down
3 changes: 3 additions & 0 deletions src/aws/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@

//! An object store implementation for S3
//!
//! See the [Feature Flags](crate#feature-flags) section for the difference
//! between the `aws` and `aws-base` features.
//!
//! ## Multipart uploads
//!
//! Multipart uploads can be initiated with the [`ObjectStore::put_multipart_opts`] method.
Expand Down
3 changes: 3 additions & 0 deletions src/azure/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@

//! An object store implementation for Azure blob storage
//!
//! See the [Feature Flags](crate#feature-flags) section for the difference
//! between the `azure` and `azure-base` features.
//!
//! ## Streaming uploads
//!
//! [`ObjectStore::put_multipart_opts`] will upload data in blocks and write a blob from those blocks.
Expand Down
3 changes: 3 additions & 0 deletions src/gcp/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@

//! An object store implementation for Google Cloud Storage
//!
//! See the [Feature Flags](crate#feature-flags) section for the difference
//! between the `gcp` and `gcp-base` features.
//!
//! ## Multipart uploads
//!
//! [Multipart uploads](https://cloud.google.com/storage/docs/multipart-uploads)
Expand Down
3 changes: 3 additions & 0 deletions src/http/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@
//!
//! This follows [rfc2518] commonly known as [WebDAV]
//!
//! See the [Feature Flags](crate#feature-flags) section for the difference
//! between the `http` and `http-base` features.
//!
//! Basic get support will work out of the box with most HTTP servers,
//! even those that don't explicitly support [rfc2518]
//!
Expand Down
81 changes: 80 additions & 1 deletion src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,8 @@
doc = "* [`http`]: [HTTP/WebDAV Storage](https://datatracker.ietf.org/doc/html/rfc2518). See [`HttpBuilder`](http::HttpBuilder)"
)]
//!
//! See [Feature Flags](#feature-flags) for the full set of flags.
//!
//! # Why not a Filesystem Interface?
//!
//! The [`ObjectStore`] interface is designed to mirror the APIs
Expand Down Expand Up @@ -517,6 +519,51 @@
//! [Apache Iceberg]: https://iceberg.apache.org/
//! [Delta Lake]: https://delta.io/
//!
//! # Feature Flags
//!
//! The feature set is layered so that you can pick a provider independently
//! of its HTTP transport:
//!
//! * `cloud-base` holds the shared provider implementation (XML/JSON parsing,
//! credentials, retry, etc.) and intentionally does *not* depend on
//! `reqwest`.
//! * `reqwest` enables the built-in [`reqwest`]-based [`HttpConnector`].
//! * `<provider>-base` (`aws-base`, `azure-base`, `gcp-base`, `http-base`)
//! adds the per-provider logic on top of `cloud-base` without pulling in
//! `reqwest`.
//! * `<provider>` (`aws`, `azure`, `gcp`, `http`) is the batteries-included
//! alias for `<provider>-base` + `reqwest` and is the typical choice.
//!
//! ## Provider features
//!
//! | Feature | Enables | Notes |
//! | --- | --- | --- |
//! | `aws` | `aws-base` + `reqwest` | Amazon S3 with the built-in HTTP transport. |
//! | `azure` | `azure-base` + `reqwest` | Azure Blob Storage with the built-in HTTP transport. |
//! | `gcp` | `gcp-base` + `reqwest` | Google Cloud Storage with the built-in HTTP transport. |
//! | `http` | `http-base` + `reqwest` | HTTP/WebDAV with the built-in HTTP transport. |
//! | `aws-base` | provider only | S3 provider without `reqwest`; supply your own [`HttpConnector`]. |
//! | `azure-base` | provider only | Azure provider without `reqwest`; supply your own [`HttpConnector`]. |
//! | `gcp-base` | provider only | GCS provider without `reqwest`; supply your own [`HttpConnector`]. |
//! | `http-base` | provider only | HTTP/WebDAV provider without `reqwest`; supply your own [`HttpConnector`]. |
//!
//! ## Transport and shared features
//!
//! | Feature | Description |
//! | --- | --- |
//! | `reqwest` | Enables the default [`reqwest`]-based [`HttpConnector`]. Pulled in automatically by `aws`, `azure`, `gcp`, and `http`. |
//! | `tls-webpki-roots` | When `reqwest` is enabled, also bundle Mozilla's [`webpki-roots`] CA certificates. See [TLS Certificates](#tls-certificates). |
//! | `cloud-base` | Shared cloud-provider implementation. Pulled in automatically by every `*-base` feature; usually not enabled directly. |
//! | `cloud` | Back-compat alias for `cloud-base` + `reqwest`. Kept for users that previously depended on the `cloud` umbrella feature. |
//!
//! ## Other features
//!
//! | Feature | Description |
//! | --- | --- |
//! | `fs` *(default)* | Local filesystem store via [`LocalFileSystem`](local::LocalFileSystem). |
//! | `tokio` | Enables Tokio-based utilities such as [`BufReader`](buffered::BufReader) and [`BufWriter`](buffered::BufWriter). Pulled in automatically by `fs` and the `*-base` features. |
//! | `integration` | Exposes the [`integration`] module, a reusable test suite for verifying custom [`ObjectStore`] implementations. Not API-stable. |
//!
//! # TLS Certificates
//!
//! Stores that use HTTPS/TLS (this is true for most cloud stores) can choose the source of their [CA]
Expand All @@ -533,9 +580,41 @@
//! Many [`ObjectStore`] implementations permit customization of the HTTP client via
//! the [`HttpConnector`] trait and utilities in the [`client`] module.
//! Examples include injecting custom HTTP headers or using an alternate
//! tokio Runtime I/O requests.
//! tokio Runtime for I/O requests. To replace `reqwest` entirely (rather than
//! tweak the bundled transport) see [Disabling `reqwest`](#disabling-reqwest).
//!
//! [`HttpConnector`]: client::HttpConnector
//!
//! # Disabling `reqwest`
//!
//! The `aws`, `azure`, `gcp`, and `http` features each bundle a
//! [`reqwest`]-based HTTP transport, which is the right choice for most
//! applications. If you would rather supply your own HTTP client — for example
//! to share an existing client, to target a platform where `reqwest` does not
//! compile (such as `wasm32-wasip1`), or to keep `reqwest` out of your
//! dependency tree — use the matching `*-base` feature and provide an
//! [`HttpConnector`](client::HttpConnector) at builder time.
//!
//! Remember to disable the default features so that `fs` (and its transitive
//! dependencies) is not pulled in:
//!
//! ```toml
//! [dependencies]
//! object_store = { version = "0.13", default-features = false, features = ["aws-base"] }
//! ```
//!
//! ```ignore
//! use object_store::aws::AmazonS3Builder;
//!
//! let store = AmazonS3Builder::from_env()
//! // `my_connector` is your own `impl HttpConnector`
//! .with_http_connector(my_connector)
//! .build()?;
//! ```
//!
//! See [Feature Flags](#feature-flags) above for the full set of flags.
//!
//! [`reqwest`]: https://crates.io/crates/reqwest

#[cfg(feature = "aws-base")]
pub mod aws;
Expand Down
Loading