Feature Description
@Xuanwo i was working the side to prepare a PR which allows opendal as an external byte source for Polars.
While doing this i came across a few limitation which need Debug to be implemented for Reader and Writer. For now I have implemented these on my end directly, but i was thinking if these can be implemented directly in core since ReadContext is private, maybe some non critical metadata can be exposed?
Problem and Solution
With this I also have the following suggestions,
- When reader is created, maybe a stat call can be triggered eagerly which exposes the
Metadata object as a field in reader. The reader can fail eagerly if the object doesn't exist on this basis.
- as i remember, this eager call is anyways triggered when user passes byte range as
.., so might as well do this beforehand.
- Metadata has a lot of info already but some things like scheme, prefix, and fully qualified path are still missing, but they're available in the
ReaderContext.
What do you think?
Additional Context
Ref: #5972
Are you willing to contribute to the development of this feature?
Feature Description
@Xuanwo i was working the side to prepare a PR which allows opendal as an external byte source for Polars.
While doing this i came across a few limitation which need
Debugto be implemented forReaderandWriter. For now I have implemented these on my end directly, but i was thinking if these can be implemented directly in core sinceReadContextis private, maybe some non critical metadata can be exposed?Problem and Solution
With this I also have the following suggestions,
Metadataobject as a field in reader. The reader can fail eagerly if the object doesn't exist on this basis..., so might as well do this beforehand.ReaderContext.What do you think?
Additional Context
Ref: #5972
Are you willing to contribute to the development of this feature?