Lightweight DNS cache with shuffle to prevent DNS flooding

[kdn36]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Using object_store from polars with high-concurrency IO in distributed mode on EKS environment using S3 storage may lead to DNS flooding. Subsequently, this results in significant slowdown of the overall query.

Describe the solution you'd like
We would like to see object_store to have a lightweight built-in DNS cache with shuffling, exposed through configuration option(s). Much smaller than a full-blown DNS resolver such as hickory-resolver.

Alternatively, expose a with_dns_resolver client option which accepts a custom resolver (Resolve trait). This would allow us to inject our own resolver. Or expose a public options.client_builder() so we can integrate into the options.client() logic.

Describe alternatives you've considered
(1) Out-of-the-box, object_store give shuffling, but no caching.
(2) One can add reqwest with the hickory-resolver feature, and turn off RandomizeAddresses, but this only gives caching and no shuffling. The cost (importing a full DNS resolver) is high, too high for a dataframe library. The configuration feels hacky.
(3) Once can add a custom client and connector via with_http_connector which takes on the extra DNS responsibilities, but then the logic to handle ClientOptions is technically not viable since the fields nor the options.client() method are not publicly accessible.

For now, option (3) with some extra steps to re-implement ClientOptions may be our best stopgap solution, but it would be cleaner and easier to maintain if all dependent functionality is contained within and configurable as part of the crate (or pass a custom DNS resolver).

We can submit a (draft) PR if that helps.

Additional context
The following setup can trigger DNS flooding:

• default object_store and reqwest as used by polars
• polars distributed with high-concurrency (large TCP pool to servce many concurrent get_range requests)
• polars distributed running on default EKS, e.g., on 32 instances (note, EKS shares DNS, unlike a typical standalone OS)
• (further aggravated by the default ndots:5 and dual-stack IPv4+v6)
  The above results in DNS flooding to the pod that serves CoreDNS.

[tustvold]

FWIW https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/

[kdn36]

Thanks. While that would technically work, it sits in the infra management domain, which we want to avoid for a number of reasons (product boundaries mostly).

We have a working version using with_http_connector with the caching sitting on our side (polars), but I ran into an integration challenge because the ClientOptions fields are private. Let me know if I missed something.

In case there is no appetite to add the actual DNS caching inside object_store, these are the options I see:

1. Add a pub fn client_builder() to impl ClientOptions . This enables us to honor the client_options and add a custom connector. New PR (see also https://github.com/kdn36/arrow-rs-object-store/tree/feat_pub_client_builder for an unfinished working example). Downside: this may expose internals.
2. Extract the fields from ClientOptions individually and re-apply as needed in our fn connect(). This is fragile and adds maintenance overhead.
3. Add a with_dns_resolver acting on a dyn Resolve method to ClientOptions to which we can pass our own DNS resolver (similar to the current ShuffleResolver (details to be worked out). Generalization of RandomizeAddresses. Probably cleanest, but still exposes the reqwest::Resolve trait.

Appreciate your guidance. If it is not too much effort, I should be able to contribute to the chosen direction.

[kdn36]

Update: our preference is to go with the ClientOptions::with_dns_resolver() method, see #728. If we could have the PR reviewed and approved, that would give us a clean path forward. Much appreciated!