Expose underlying object store capabilities (e.g. ordered listing, negative ranges)

[vitoordaz]

Which part is this question about
API

Describe your question
ArrowObjectStore provides a very strong abstraction over object storage and solves many common problems well.

However, there are cases where leveraging features of the underlying storage API could lead to meaningful performance improvements. Currently, these capabilities are not exposed, which forces users into less efficient patterns.

Two concrete examples:

1. Ordered list results
   Today, users of the ArrowObjectStore list API must assume results are unordered, requiring them to fetch all items and sort them in memory.
   However, some backends (e.g., Amazon S3 for non-directory buckets) already return results in lexicographical order. If users could detect this at runtime, they could avoid unnecessary buffering and sorting, and only fall back when ordering is not guaranteed.

2. Negative byte ranges
   Fetching ranges from the end of an object currently requires an additional HEAD request to determine object size before issuing a GET.
   If the underlying store supports negative ranges directly, this extra request could be avoided, reducing latency and request overhead.

Proposal
Would the ArrowObjectStore maintainers be open to introducing a "capabilities" API that exposes which features are supported by the underlying storage implementation?

For example, something along the lines of:

• supports_ordered_listing()
• supports_negative_ranges()

This would allow users to write adaptive logic that takes advantage of backend-specific optimizations while preserving portability.

Curious to hear your thoughts on whether this aligns with the project's design goals.

Additional context

[vitoordaz]

@alamb what do you think about this proposal?

[alamb]

Since users of ObjectStore need to instantiate an ObjectStore instance already, it seems like they already know (or could track) what capabilities their particular ObjectStore instance supports

What is the usecase for a programatic API?

(The proposal sounds reasonable to me, FWIW, though I would want to minimize any change to the ObjectStore trait itself (e.g. maybe it could return a Capablities object where we could add methods like supports_ordered_listing etc

[alamb]

(thank you for this nicely written description)

[vitoordaz]

I think that this would be useful for systems like SlateDB which accept ObjectStore as an argument and therefore cannot control what object storage service is used.

[crepererum]

I wonder if we should design this similar to the attributes system in object_store, i.e. having a string-like dict or set. This would make it easier to extend in the future and may also allow custom capabilities, like this:

enum Capability {
   OrderedListing,
   NegativeRange
   Custom(Cow<'static, str>),
}

impl Capability {
    fn as_str(&self) -> &'str {...}
    fn from_str(s: &str) -> Self {...}
}

struct Capabilities {
    // impl. detail!
    caps: HashSet<Capability>,
}

impl Capabilities {
    fn has(&self, cap: &Capability) -> bool {...}
}

trait ObjectStore {
    fn capabilties(&self) -> &Capabilities;
}

We probably need some work on the internals, ideally Capabilities can be constructed in a const context so that the stores can return a reference w/o creating a singelton on the heap.

@tustvold WDYT?

[tustvold]

My biggest question is whether ObjectStore actually always knows the answer, particularly in the case of S3, there are various S3-compatible stores that don't all behave consistently like S3.

This kind of ties into Andrew's point, that perhaps this is more a question for the config of the application object_store is being integrated into, than for object_store itself?

That being said, not opposed to this, just also not sure we always know

[vitoordaz]

I made a PR for this proposal #732
Let me know what you think folks

[kevinjqliu]

I think enumerating object storage capabilities is interesting and helpful for callers. We just need to make sure the advertised capabilities are up to date with the latest behavior 😄, although object storage behaviors rarely change.

One thing I want to point out is that capabilities might also be tied to the configured store instance, not just the object storage behavior. There are configs that can change behavior, and thus the capabilities.

For example, Azure's list_with_offset behavior changes in emulator mode. For normal Azure, it uses the server-side list_with_offset path. But when is_emulator is enabled, it falls back to list(prefix) plus client-side filtering because Azurite does not support the startFrom query parameter.

[alamb]

I agree this is an interesting potential feature. My concern is that it will be hard to maintain and keep up to date as object store vendors change their support feature sets, etc

I think the ObjectStore API is already quite extensible at runtime (you can add your own ObjectStore wrappers that intercept calls and do what you want)

Maybe we could add a as_any API that would allow people who are interested in specializing the implementation to do so by checking as_any().downcast_ref() 🤔

[kevinjqliu]

My concern is that it will be hard to maintain and keep up to date as object store vendors change their support feature sets, etc

From researching the ordered listing behavior, we already found some edge cases for aws and azure. There's a real concern on maintainability. For example, we can mark GCS with ordered listing capability today. If and when GCS comes out with a product line without this guarantee, we'd have to keep the capabilities up to date.

Maybe we could add a as_any API that would allow people who are interested in specializing the implementation to do so by checking as_any().downcast_ref() 🤔

I had to ask AI to make sense of this 😄
Basically instead of object_store saying:

store.capabilities().has(Capability::OrderedListing)

the caller could say:

if store.as_any().downcast_ref::<AmazonS3>().is_some() {
    // I know this is AmazonS3, so I can specialize
}

[vitoordaz]

as_any approach is going to work if we are dealing with vanilla object storage, like S3, GCP, or Azure. These days, many providers implement S3 APIs, but might have slightly different capabilities. Only one who creates the ObjectStore entity is aware of this.

That is why it's important to let users override default capabilities.

At the moment, only one storage GCP has enabled capabilities (ordered listing) other have none. Which means that we only need to support this for GCP.

However, to address vendor features concerns, we can just default to an empty set of capabilities and delegate to the user this configuration.

[tustvold]

If the user is already configuring these capabilities, and they're not used internally within ObjectStore, what is the advantage of embedding them? If an application wants to make assumptions about the backing object store, that's perhaps part of that application's configuration not object stores?

I'm pretty lukewarm on adding as_any given that people tend to wrap ObjectStore in multiple layers.

[kylebarron]

I think something like as_any would only make sense if we were to implement other, more specialized methods on the specific instance.

Right now AmazonS3 doesn't supply any functionality other than the ObjectStore trait, so downcasting to AmazonS3 doesn't really give you anything. But if we were to implement AmazonS3::ordered_list then downcasting would have utility