Better testing for different combinations of feature flags, different http clients and crypto libraries

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
As part of the 0.14.0 release, we are revamping how the feature flag system in object_store works, largely triggered by some changes in reqwest, the underlying http client library, way cryptography is handled

Specifically, as described in #744 / designed with ❤️ from @kevinjqliu , we will now support, via feature flags

1. Bring your own http cilent library (as well as supporting reqwest by default)
2. Bring you own crypto library (a well as supporting ring/aws-lc-rs)

This is a good thing as it allows users to customize the library exactly how they want, and minimize unecessary dependencies.

However, this design also results in a large number of feature flags; While adding tests for some combinations, it turns out it was fairly hard to test some of the combinations, as sometimes additional features are needed in other crates.

For example, to test "Run clippy with base features, reqwest and no bundled CryptoProvider" requires activating the rustls-no-provider feature of `reqwest

In this case, since reqwest is a direct dependency of object_store, we can do it

cargo clippy --no-default-features --features aws-base,azure-base,gcp-base,http-base,reqwest,reqwest/rustls-no-provider -- -D warnings

This makes it hard for me to understand what the test is actually testing and if it could be used this way by a downstream crate, especially as I mention in #707 (comment)

I feel one difference is that actual customers will have their own Cargo.toml file where they can directly activate the features in the subcrates, so they won't be compiling just the object_store.

Describe the solution you'd like
I would like to

1. improve the testing of object_store with various feature flag combinations so it is clear what is tested and what is not
2. demonstrate / test that it is possible to use object_store with various flag combinations downstream to the desired effect

Describe alternatives you've considered

One way to test these features would be to create some targeted examples that are not part of the workspace and demonstrate (and verify they compile, etc) how to do the various usecases.

SO that would look like:

1. Remove the targeted clippy and cargo tree checks added in Pluggable Crypto / Update reqwest 0.13 #707 (comment)
2. Add example crates like example/custom_http_client/ and example/custom_http_client_own_crypto projects that are not part of the main workspace and that configures the feature flag in some way
3. Use the example to show: cargo run works, and that they don't bring in unwanted dependencies

Additional context

• Discusion: Discuss feature naming for optional transport and crypto support #744
• New feature flags, make reqwest optional: Make reqwest optional #724
• Reqwest update / feature flag disucssion: Pluggable Crypto / Update reqwest 0.13 #707

[kevinjqliu]

We're doing most of the testing today with proxies.
This would help us get rid of a lot of these proxies:

• arrow-rs-object-store/.github/workflows/ci.yml

  Lines 42 to 90 in dee6094

  	- name: Verify base features do not enable TLS/signing crypto crates
  	# `--prefix none -f '{p}'` prints one `name vX.Y.Z` per line, so the
  	# crate name can be anchored with `^`. Avoid `\b`, which is not
  	# portable (e.g. BSD grep) and would also substring-match crates like
  	# `iri-string`.
  	run: |
  	found=$(cargo tree --no-default-features --features aws-base,azure-base,gcp-base,http-base --edges normal --prefix none -f '{p}' \
  	| grep -E '^(aws-lc-rs|aws-lc-sys|native-tls|openssl|ring|rustls) v' | sort -u || true)
  	if [ -n "$found" ]; then
  	echo "❌ disallowed crate(s) found:"
  	echo "$found"
  	exit 1
  	fi
  	echo "✅ no TLS/signing crypto crates"
  	- name: Verify base features do not enable reqwest
  	run: |
  	found=$(cargo tree --no-default-features --features aws-base,azure-base,gcp-base,http-base --edges normal --prefix none -f '{p}' \
  	| grep -E '^reqwest v' | sort -u || true)
  	if [ -n "$found" ]; then
  	echo "❌ reqwest must not be enabled by *-base features:"
  	echo "$found"
  	exit 1
  	fi
  	echo "✅ *-base features do not enable reqwest"
  	- name: Check ring build does not pull aws-lc-rs
  	# A `ring` build (with a non-aws-lc-rs reqwest TLS backend) must be free
  	# of aws-lc-rs entirely, so users can opt out of it on every target.
  	run: |
  	found=$(cargo tree --no-default-features --features aws-base,azure-base,gcp-base,http-base,reqwest,reqwest/native-tls,ring --edges normal --prefix none -f '{p}' \
  	| grep -E '^(aws-lc-rs|aws-lc-sys) v' | sort -u || true)
  	if [ -n "$found" ]; then
  	echo "❌ aws-lc-rs pulled into a ring-only build:"
  	echo "$found"
  	exit 1
  	fi
  	echo "✅ ring build is free of aws-lc-rs"
  	- name: Check batteries-included features do not pull ring
  	# The batteries-included features use reqwest/rustls, which uses aws-lc-rs
  	# for TLS by default. Keep object_store's default bundled crypto provider
  	# aligned with that choice by ensuring these features don't pull ring.
  	run: |
  	found=$(cargo tree --no-default-features --features aws,azure,gcp,http --edges normal --prefix none -f '{p}' \
  	| grep -E '^ring v' | sort -u || true)
  	if [ -n "$found" ]; then
  	echo "❌ batteries-included features pulled ring:"
  	echo "$found"
  	exit 1
  	fi
  	echo "✅ batteries-included features are free of ring"

• arrow-rs-object-store/.github/workflows/ci.yml

  Lines 117 to 128 in dee6094

  	# Direct `reqwest` needs a TLS backend. Use `rustls-no-provider` so TLS
  	# does not add AWS-LC to ring/custom CryptoProvider checks.
  	- name: Run clippy with base features, reqwest and ring CryptoProvider
  	run: cargo clippy --no-default-features --features aws-base,azure-base,gcp-base,http-base,reqwest,reqwest/rustls-no-provider,ring -- -D warnings
  	- name: Run clippy with base features, reqwest and no bundled CryptoProvider
  	run: cargo clippy --no-default-features --features aws-base,azure-base,gcp-base,http-base,reqwest,reqwest/rustls-no-provider -- -D warnings
  	- name: Run clippy with base features, custom HTTP and aws-lc-rs CryptoProvider
  	run: cargo clippy --no-default-features --features aws-base,azure-base,gcp-base,http-base,aws-lc-rs -- -D warnings
  	- name: Run clippy with base features, custom HTTP and ring CryptoProvider
  	run: cargo clippy --no-default-features --features aws-base,azure-base,gcp-base,http-base,ring -- -D warnings
  	- name: Run clippy with base features, custom HTTP and no bundled CryptoProvider
  	run: cargo clippy --no-default-features --features aws-base,azure-base,gcp-base,http-base -- -D warnings

And would be great to present the tests from the users' perspective, ie

• I want reqwest with custom crypto provider
• I want custom http client with aws-ls-rs
• I want custom http client with custom crypto provider
• etc

[alamb]

And would be great to present the tests from the users' perspective, ie

Yes, 100%