Skip to content

Support PutMultipartOptions for MultipartStore::create_multipart for AWS/GCP.#754

Merged
alamb merged 4 commits into
apache:mainfrom
BearMinimum98:main
Jun 17, 2026
Merged

Support PutMultipartOptions for MultipartStore::create_multipart for AWS/GCP.#754
alamb merged 4 commits into
apache:mainfrom
BearMinimum98:main

Conversation

@BearMinimum98

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Adds PutMultipartOptions support to the low-level MultipartStore create flow via a new create_multipart_opts method.

This is implemented for AWS, GCP, in-memory, and wrapper stores. Azure returns NotSupported for non-default tags/attributes because Azure applies these options when finalizing block uploads, not when creating them. Supporting this in the low-level API would require local upload state, which could leak if uploads are never completed or aborted.

Closes #745

Rationale for this change

See issue.

What changes are included in this PR?

  • Adds MultipartStore::create_multipart_opts
  • For AWS/GCP, passes options through to the backend client
  • For PrefixStore/ThrottledStore, delegates options to the wrapped multipart store
  • For in-memory, stores attributes in PartStorage until completion
  • For Azure, returns NotSupported for non-default tags/attributes

Are there any user-facing changes?

Yes. MultipartStore has a new default method, create_multipart_opts, so existing implementations remain source-compatible.

There are no breaking changes to the public API; the new method has a default implementation.

@BearMinimum98

Copy link
Copy Markdown
Contributor Author

@alamb Would greatly appreciate a review today to be able to make it into the release tomorrow, if not possible, disregard. If it doesn't cut the release tomorrow, about when will the next minor version release be?

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @BearMinimum98 -- this change makes sense to me and I found the PR well documented and well tested

🙏

Comment thread src/integration.rs

let result = storage.get(&path).await.unwrap();
assert_eq!(result.meta.size, chunk_size as u64 * 2);
// Some providers add default attributes, e.g. S3 may report a default Content-Type.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Comment thread src/gcp/mod.rs
path: &Path,
opts: PutMultipartOptions,
) -> Result<MultipartId> {
self.client.multipart_initiate(path, opts).await

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me as create_multipart is simply passing in PutMultipartOptions::default() -- adding the options explicitly makes sense

    async fn create_multipart(&self, path: &Path) -> Result<MultipartId> {
        self.client
            .multipart_initiate(path, PutMultipartOptions::default())
            .await
    }

@alamb alamb merged commit 1a61282 into apache:main Jun 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support PutMultipartOptions for MultipartStore::create_multipart

2 participants