Make reqwest optional

[awesterb]

Which issue does this PR close?

• Closes Make reqwest optional #723 .

Rationale for this change

We're using object_store with a code base that already uses awc as HTTP-client, and would like to avoid having reqwest as additional HTTP client alongside awc. An alternative HttpConnector can already be provided currently, but the reqwest dependency can not yet be avoided.

Also, since reqwest can't be used for wasm32-wasip1 anyways, not having to package it there is nice too.

What changes are included in this PR?

Adds a new reqwest feature that gates dep:reqwest, but is enabled by cloud by default. This means reqwest will still be pulled in by the azure, gcp, aws and http features, as is the case now. For avoiding reqwest, the features cloud-base, azure-base, gcp-base, aws-base and http-base are added.

Some features that depend on reqwest types I gated behind reqwest too, such as client::Certificate. To keep the pull request minimal and non-breaking (see #724 (comment)), I did not attempt to rewrite those to be reqwest-independent. I believe all of the gated features can be avoided when providing your own HttpConnector. (Besides, most of these features were already gated behind !wasm32.)

Tests that use reqwest-gated types were gated behind reqwest too.

Added to CI extra clippy jobs to catch missing gates in the future, and a build job for wasm32-wasip1 without reqwest.

Where reqwest::{Method,StatusCode,header} was used, this was replaced by the http types.

I considered copying the X-no-crypto naming scheme (aws-no-crypto, gcp-no-crypto ...) from GH-707, but decided not to use aws-no-reqwest (etc.) since that would necessitate aws-no-crypto-no-reqwest (and worse) in the future. (Combining !707 could make sense, having X-base mean no crypto and no reqwest.)

Are there any user-facing changes?

The feature reqwest that gates dep:reqwest, and the features cloud-base, azure-base, gcp-base, aws-base and http-base that avoid reqwest.

No breaking changes (according to https://crates.io/crates/cargo-semver-checks.)

[tustvold]

Thank you this seems sensible to me and makes sense, I like the formulation of having a minimal feature flag and a more batteries included one. We should roll this into the crypto work as well.

[kylebarron]

Should we discuss feature naming? Either now or in an issue before the next release? Because now cloud is an alias for reqwest, essentially. Would it read clearer if say azure was defined as

- azure = ["azure-base", "cloud"]
+ azure = ["azure-base", "reqwest"]

because azure-base already depends on cloud-base. We have a diamond dependency graph here.

Should we have some docs about feature flag options?

[kevinjqliu]

+1 we'd want to coordinate on naming since #707 is also renaming these features.

Also see #707 (review)
both -base and -custom makes sense to me

[awesterb]

Renaming -custom to -base is completely fine by me too. (Though it might read a bit odd in the object_store code: custom is not an additional feature.)

I agree the cloud adds nothing to reqwest. It is only there for backwards compatibility. If you want to make this a breaking change it can be removed, of course.

[kevinjqliu]

i actually like -base more 😄 This is my mental model:

cloud-base = shared cloud code
*-base     = provider-specific code without default reqwest
reqwest    = built-in HTTP transport
cloud      = compatibility alias for cloud-base + reqwest
aws/azure/gcp/http = provider base + reqwest

# Shared cloud/provider implementation dependencies.
# This intentionally does NOT include reqwest.
cloud-base = [
  "serde",
  "serde_json",
  "quick-xml",
  "hyper",
  "chrono/serde",
  "base64",
  "rand",
  "ring",
  "http-body-util",
  "form_urlencoded",
  "serde_urlencoded",
  "tokio",
]

# Built-in reqwest-based HTTP transport.
reqwest = [
  "dep:reqwest",
  "reqwest/stream",
]

# Compatibility/convenience alias.
# Historically, cloud meant "common cloud support", including reqwest.
# Keep that behavior for existing users.
cloud = [
  "cloud-base",
  "reqwest",
]

# Provider base features.
# These compile provider logic without forcing reqwest.
azure-base = [
  "cloud-base",
  "httparse",
]

gcp-base = [
  "cloud-base",
  "rustls-pki-types",
]

aws-base = [
  "cloud-base",
  "crc-fast",
  "md-5",
]

http-base = [
  "cloud-base",
]

# Batteries-included provider features.
# These preserve existing behavior: enabling aws/azure/gcp/http gives
# the default reqwest transport.
azure = [
  "azure-base",
  "reqwest",
]

gcp = [
  "gcp-base",
  "reqwest",
]

aws = [
  "aws-base",
  "reqwest",
]

http = [
  "http-base",
  "reqwest",
]

[kevinjqliu]

created an issue to facilitate with discussion
#744

[alamb]

I merged this PR up from main to resolve a conflict.

It seems like @kevinjqliu has gotten consensus on the naming here (base)

• Discuss feature naming for optional transport and crypto support #744

@awesterb are you willing to implement @kylebarron 's suggestion on #724 (comment) ?

[awesterb]

@awesterb are you willing to implement @kylebarron 's suggestion on #724 (comment) ?

Will do!

[awesterb]

@awesterb are you willing to implement @kylebarron 's suggestion on #724 (comment) ?

Done. Strictly speaking, it is a breaking change, according to cargo semver-checks:

Description:
A feature no longer enables another feature in this package's Cargo.toml. This will break downstream crates which rely on the implied feature being enabled.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#cargo-feature-remove-another
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/feature_no_longer_enables_feature.ron

Failed in:
  feature gcp no longer enables cloud
  feature azure no longer enables cloud
  feature aws no longer enables cloud
  feature http no longer enables cloud

(But there are already other breaking changes.)

[alamb]

Done. Strictly speaking, it does cause a breaking change according to cargo semver-checks:

Yes I agree -- this is going to be a breaking change, but that is ok as we are about to put out a non semver compatible release

• Release object store 0.14.0 (breaking) - Target April 2026 #657

[alamb]

Thank you @awesterb -- I think this looks good to me

@kevinjqliu and @tustvold and @kylebarron I would appreciate your reivew as well

I also think we will need to document how these feature flags, which are currently here

• arrow-rs-object-store/src/lib.rs

  Line 76 in 18ed86c

  //! Feature flags are used to enable support for other implementations:

Which renders here:
https://docs.rs/object_store/latest/object_store/#available-objectstore-implementations

[kevinjqliu]

Thanks everyone! Glad we found a way to move forward with this.

I did a brief review, it looks great directionally.

One related question: should we also add a small CI check that asserts the dependency contract directly, i.e. that the *-base features do not pull in reqwest at all? Maybe we could add one lightweight check for the combined base-feature case, something like:

cargo tree \
  --no-default-features \
  --features aws-base,azure-base,gcp-base,http-base \
  --edges normal \
  -i reqwest

and fail CI if reqwest is present.

Not a blocker, but if *-base is meant to be a supported “no reqwest” contract, this seems like a useful guardrail against accidentally reintroducing it in the future.

[kevinjqliu]

LGTM!

One nit: we may want to add a CI check to ensure reqwest is not accidentally added back to the *-base feature dependency graph in the future.

[awesterb]

Thanks for all the movement on this! I've added the comments to Cargo.toml proposed by @kevinjqliu.

Updating the docs.rs documentation, and adding additional checks to the CI are worth doing, but I'd prefer to keep the scope of my own changes limited.

Edits by maintainers are enabled in case anyone wants to contribute to this PR. Feel free!

[kevinjqliu]

thank you @awesterb for all the changes. Im happy to follow up and add the docs changes + CI checks if necessary

[kylebarron]

Is the CI failure pre-existing or new in this PR?

[kevinjqliu]

the CI failure is not related to this PR, fixing the issue in #746

[alamb]

the CI failure is not related to this PR, fixing the issue in #746

I merge that in and will merge up from main for this one too

[alamb]

Updating the docs.rs documentation, and adding additional checks to the CI are worth doing, but I'd prefer to keep the scope of my own changes limited.

Agreed - I will file follow on tickets for these items

[alamb]

I merged up to resolve some conflicts

[alamb]

I have filed follow on tickets

• Add CI check to ensure reqwest is not accidentally added #748
• Document new feature flag system #749

[kevinjqliu]

looks like the ci failed on cargo test --features=aws,azure,gcp,http, could due to the 2 feature = "cloud", declarations:

arrow-rs-object-store/src/integration.rs

Lines 1440 to 1527 in f554e60

	#[cfg(all(
	feature = "cloud",
	not(all(target_arch = "wasm32", target_os = "wasi"))
	))]
	mod marker {
	use super::*;
	use crate::ClientOptions;
	use crate::client::{
	HttpClient, HttpConnector, HttpError, HttpRequest, HttpResponse, HttpService,
	ReqwestConnector,
	};
	use async_trait::async_trait;
	/// A marker inserted into response extensions by [`MarkerHttpConnector`]
	#[derive(Clone, Debug, PartialEq)]
	struct Marker;
	/// [`HttpService`] middleware that tags every response with a [`Marker`]
	#[derive(Debug)]
	struct MarkerService(HttpClient);
	#[async_trait]
	impl HttpService for MarkerService {
	async fn call(&self, req: HttpRequest) -> Result<HttpResponse, HttpError> {
	let mut response = self.0.execute(req).await?;
	response.extensions_mut().insert(Marker);
	Ok(response)
	}
	}
	/// An [`HttpConnector`] that tags the extensions of every HTTP response with a
	/// marker, allowing [`response_extensions`] to verify their propagation
	#[derive(Debug, Default)]
	pub struct MarkerHttpConnector(ReqwestConnector);
	impl HttpConnector for MarkerHttpConnector {
	fn connect(&self, options: &ClientOptions) -> crate::Result<HttpClient> {
	let client = self.0.connect(options)?;
	Ok(HttpClient::new(MarkerService(client)))
	}
	}
	/// Tests that the extensions of HTTP responses are propagated to returned results
	///
	/// The provided store must have been built with [`MarkerHttpConnector`]
	pub async fn response_extensions(storage: &DynObjectStore, test_multipart: bool) {
	delete_fixtures(storage).await;
	let location = Path::from("test_response_extensions/file.txt");
	let data = Bytes::from("arbitrary data");
	let put = storage.put(&location, data.clone().into()).await.unwrap();
	assert!(
	put.extensions.get::<Marker>().is_some(),
	"PutResult should contain the response extensions"
	);
	let get = storage.get(&location).await.unwrap();
	assert!(
	get.extensions.get::<Marker>().is_some(),
	"GetResult should contain the response extensions"
	);
	let list = storage.list_with_delimiter(None).await.unwrap();
	assert!(
	list.extensions.get::<Marker>().is_some(),
	"ListResult should contain the response extensions"
	);
	if test_multipart {
	let mut upload = storage.put_multipart(&location).await.unwrap();
	upload.put_part(data.into()).await.unwrap();
	let complete = upload.complete().await.unwrap();
	assert!(
	complete.extensions.get::<Marker>().is_some(),
	"PutResult of a completed multipart upload should contain the response extensions"
	);
	}
	delete_fixtures(storage).await;
	}
	}
	#[cfg(all(
	feature = "cloud",
	not(all(target_arch = "wasm32", target_os = "wasi"))
	))]
	pub use marker::{MarkerHttpConnector, response_extensions};

we need to use feature = "reqwest", instead

[kylebarron]

Thanks @kevinjqliu; I'll make that change on this branch

[kevinjqliu]

ty! i couldnt verify the wasm32 ci issue, but i suspect its the same issue

[kevinjqliu]

LGTM!

[kevinjqliu]

💯 thanks @awesterb for the PR and everyone else for the reviews

[awesterb]

💯 thanks @awesterb for the PR and everyone else for the reviews

No, thank you all for willing to incorporating this feature! This helps us cut down our dependency tree a lot!