HDDS-12542. Support S3 signed multi chunks payload verification

[chungen0126]

What changes were proposed in this pull request?

This is a design doc for supporting s3 multi chunks upload verification.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12542

How was this patch tested?

None.

[Copilot]

Pull request overview

Adds a new design document proposing how Ozone S3 Gateway (S3G) should verify AWS SigV4 streaming chunked payload signatures (currently parsed by SignedChunksInputStream but not cryptographically verified).

Changes:

• Introduces a design doc describing chunk-signature verification approach and performance considerations (buffering + incremental hashing).
• Outlines planned algorithm support (HMAC-SHA256 and mentions ECDSA/SigV4a), plus a testing plan.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

accident

[peterxcli]

I'm thinking if we could reuse the multipart upload? data chunk goes to mpu request, while the final chunk trigger a mpu complete request.

[peterxcli]

we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length.

Could you explain how would this buffer guarantee a ack chunk write success means it's durable?

[chungen0126]

To clarify, the internal buffer in S3G does not acknowledge individual AWS SigV4 chunks. In the S3 protocol, the client does not receive an ACK for each chunk, it only receives a single HTTP 200 OK at the very end of the entire PUT request (or the specific multipart upload part).

The durability is strictly guaranteed by how we handle the downstream stream lifecycle. The S3 Gateway will only return the final HTTP 200 OK to the S3 client after OzoneOutputStream.close() is successfully called. This close() operation ensures that all buffered data is fully flushed, safely committed across DataNodes via the Ratis pipeline, and the metadata is finalized in the Ozone Manager.

If the S3G crashes or restarts while data is still in this 4MB memory buffer, no ACK is ever sent to the client. The connection will be dropped, and the S3 client (like the AWS SDK) will simply treat it as a network failure and automatically retry the upload. Therefore, there is no risk of data loss.

[peterxcli]

The v1, v2 pipeline are already support buffering.

v1:

ozone/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

Lines 287 to 294 in 576fcce

	try (OzoneOutputStream output = openKeyForPut(
	volume.getName(), bucketName, keyPath, length,
	replicationConfig, customMetadata, tags, writeConditions)) {
	long metadataLatencyNs =
	getMetrics().updatePutKeyMetadataStats(startNanos);
	perf.appendMetaLatencyNanos(metadataLatencyNs);
	putLength = IOUtils.copyLarge(multiDigestInputStream, output, 0, length,
	new byte[getIOBufferSize(length)]);

v2:

ozone/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

Lines 276 to 283 in 576fcce

	if (isDatastreamEnabled() && !enableEC && length > getDatastreamMinLength()) {
	perf.appendStreamMode();
	Pair<String, Long> keyWriteResult = ObjectEndpointStreaming
	.put(bucket, keyPath, length, replicationConfig, getChunkSize(),
	customMetadata, tags, multiDigestInputStream, getHeaders(),
	signatureInfo.isSignPayload(), perf, writeConditions);
	md5Hash = keyWriteResult.getKey();
	putLength = keyWriteResult.getValue();

ozone/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpointStreaming.java

Lines 205 to 220 in 576fcce

	private static long writeToStreamOutput(OzoneDataStreamOutput streamOutput,
	InputStream body, int bufferSize,
	long length)
	throws IOException {
	final byte[] buffer = new byte[bufferSize];
	long n = 0;
	while (n < length) {
	final int toRead = Math.toIntExact(Math.min(bufferSize, length - n));
	final int readLength = body.read(buffer, 0, toRead);
	if (readLength == -1) {
	break;
	}
	streamOutput.write(ByteBuffer.wrap(buffer, 0, readLength));
	n += readLength;
	}
	return n;

[chungen0126]

I'm thinking if we could reuse the multipart upload? data chunk goes to mpu request, while the final chunk trigger a mpu complete request.

Yes, MPU can reuse this.

In S3, an MPU UploadPart request behaves exactly like a standard single PUT request. If a client uses chunked encoding for an MPU part, it will go through this exact same SignedChunksInputStream logic. The stream will verify and buffer the chunks for that specific part just like it does for a single PUT.

[peterxcli]

In S3, an MPU UploadPart request behaves exactly like a standard single PUT request. If a client uses chunked encoding for an MPU part, it will go through this exact same SignedChunksInputStream logic. The stream will verify and buffer the chunks for that specific part just like it does for a single PUT.

Actually I’m asking whether this multi‑chunk payload can reuse the existing MPU infrastructure. Sorry, I didn’t phrase it clearly earlier.

[peterxcli]

I think the only thing we need to add is, make the SignedChunksInputStream support materialised the chunked content if it bytes in the inputBody reach the length set in the x-amz-decoded-content-length header, reaming hash (each chunk signature depending on the previous chunk / prior state)

[peterxcli]

The v1, v2 pipeline are already support buffering.

v1:

ozone/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

Lines 287 to 294 in 576fcce

	try (OzoneOutputStream output = openKeyForPut(
	volume.getName(), bucketName, keyPath, length,
	replicationConfig, customMetadata, tags, writeConditions)) {
	long metadataLatencyNs =
	getMetrics().updatePutKeyMetadataStats(startNanos);
	perf.appendMetaLatencyNanos(metadataLatencyNs);
	putLength = IOUtils.copyLarge(multiDigestInputStream, output, 0, length,
	new byte[getIOBufferSize(length)]);

v2:

ozone/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

Lines 276 to 283 in 576fcce

	if (isDatastreamEnabled() && !enableEC && length > getDatastreamMinLength()) {
	perf.appendStreamMode();
	Pair<String, Long> keyWriteResult = ObjectEndpointStreaming
	.put(bucket, keyPath, length, replicationConfig, getChunkSize(),
	customMetadata, tags, multiDigestInputStream, getHeaders(),
	signatureInfo.isSignPayload(), perf, writeConditions);
	md5Hash = keyWriteResult.getKey();
	putLength = keyWriteResult.getValue();

ozone/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpointStreaming.java

Lines 205 to 220 in 576fcce

	private static long writeToStreamOutput(OzoneDataStreamOutput streamOutput,
	InputStream body, int bufferSize,
	long length)
	throws IOException {
	final byte[] buffer = new byte[bufferSize];
	long n = 0;
	while (n < length) {
	final int toRead = Math.toIntExact(Math.min(bufferSize, length - n));
	final int readLength = body.read(buffer, 0, toRead);
	if (readLength == -1) {
	break;
	}
	streamOutput.write(ByteBuffer.wrap(buffer, 0, readLength));
	n += readLength;
	}
	return n;

[chungen0126]

I think the only thing we need to add is, make the SignedChunksInputStream support materialised the chunked content if it bytes in the inputBody reach the length set in the x-amz-decoded-content-length header, reaming hash (each chunk signature depending on the previous chunk / prior state)

Thank you for the catch. You are right. The buffers work well now. I'll update the design doc.

[peterxcli]

Users will bring their s3 secret when they are sending s3 request, so can we just use it to calculate the derived key?

[chungen0126]

Thanks for the review. However, under the AWS Signature V4 protocol, the client never transmits the Secret Key over the network. They only send the Access Key and the calculated Signature.

Since S3G never actually receives the user's Secret Key in the request, it is impossible for S3G to calculate the derived key locally. It must query Ozone Manager (OM), which securely holds the master keys, to perform the derivation.

[ivandika3]

I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature (I might be wrong).

Also querying OM for each write will affect performance so it should not be done. Transmitting the secret key securely from OM to S3G is also another security concern. So is there a way to do some chunk signature validation without knowing the user secret key?

Since this will affect the existing S3 chunked uploads, we need to think about the carefully think about the side-effect. IMO if we need to get the secret key, this feature might be more difficult than expected.

[chungen0126]

Thanks @ivandika3 for the review.

I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature (I might be wrong).

Just to clarify, the verification process require the secret key. The official AWS documentation explains how the signing key is generated. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.
At step 2.

signing key = HMAC-SHA256(HMAC-SHA256(HMAC-SHA256(HMAC-SHA256("AWS4" + "<YourSecretAccessKey>","20130524"),"us-east-1"),"s3"),"aws4_request")

As shown here, the signing key is derived from the secret key.

[chungen0126]

Also querying OM for each write will affect performance so it should not be done. Transmitting the secret key securely from OM to S3G is also another security concern. So is there a way to do some chunk signature validation without knowing the user secret key?

Retrieving the signing key from OM is secure due to the following reasons:

1. One-way Hashing: The key S3G receives is a derived key computed through multiple iterations of HMAC-SHA256. Since HMAC is a one-way cryptographic function, even if this derived key is compromised, it is impossible to get the original secret key
2. Time-bound Scoping: The derived key is strictly bound to a specific date. Because the date is a required input for the HMAC calculation, the key is only valid for that particular day and cannot be reused for requests on any other date.

As mentioned in the design doc:

From a security perspective, this new API will not expose the raw AWS Secret Key to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the Derived Key.

Regarding the performance concern of fetching the signing key from OM.
For a multi-chunk streaming upload, this only adds one additional RPC per request. I believe the impact on overall performance will be small. Or maybe we could piggyback the derived key onto the metadata returned by the createKey call. This would eliminate the extra RPC entirely by providing the key upfront for all subsequent chunk uploads.

[ivandika3]

Just to clarify, the verification process require the secret key. The official AWS documentation explains how the signing key is generated. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.
At step 2.

Yes you're right we do need the secret key. Thanks for the clarification.

One-way Hashing: The key S3G receives is a derived key computed through multiple iterations of HMAC-SHA256. Since HMAC is a one-way cryptographic function, even if this derived key is compromised, it is impossible to get the original secret key

Yes, I think we can use the signing key to mask the secret key (since it's hashed). The main security point is that the response returned by the OM should not include a plain text secret key.

For a multi-chunk streaming upload, this only adds one additional RPC per request. I believe the impact on overall performance will be small. Or maybe we could piggyback the derived key onto the metadata returned by the createKey call. This would eliminate the extra RPC entirely by providing the key upfront for all subsequent chunk uploads.

Hm, to be consistent (handle secret revocation change), S3G need to send an additional RPC per PutObject. However, I think this might have some performance implication since each PutObject will generate around 5+ RPC request (S3 secret fetch, get volume, get bucket, open key, commit key). Additionally, the S3GetSecretRequest is implemented as a Ratis transaction so the latency should be higher than normal read request (without Ratis). One idea is to cache the signing key and refresh periodically, but this has consistency and security implication since the secret key might already be revoked but we might still allow the requests, etc. Unless we have a way to notify S3G about the revocation (looks like Minio implemented this way through when using etcd or it will try to broadcast the changes)

The piggyback logic might work to amortize this secret key (I can see the OMKeyCreateRequest can return the user signingKey if requested). We can think further on this.

Our cluster uses a different strategy so I'm not that well-versed about this. We might need to get people more familiar with this. You can also research other systems like Minio and SeaweedFS.

[peterxcli]

Could you please take a look if the multi digest already had this incremental hashing capability?

[chungen0126]

Although we use MessageDigest here as well, reusing the one from MultiDigestInputStream isn't a good fit. MultiDigestInputStream validates the entire object at once, whereas this implementation requires chunk-by-chunk validation. Trying to combine these two different mechanism would just make the code clunky and less elegant.

[ivandika3]

The current MessageDigest is simplistic since it does not rely on the previous data chunks so might not be appropriate. You might want to implement your own MessageDigest

From below

The next chunk digest depends on the previous signature, so that is the only state you can store.

The implementation can be left up to @chungen0126. The main point (as mentioned in the design doc) is that we should not buffer anything than necessary in S3G (the data needs to be streamed seamlessly).

[ChenSammi]

I'm not sure if we need to implement ECDSA in Sigv4a. ECDSA is slow compared with sha256. AWS introduced Sigv4a for some special endpoints. We don't have this case.

[chungen0126]

Maybe I'll move ECDSA to future work. I agree that there is no use case for it in Ozone.

[ChenSammi]

This "Devrived key" looks good to me. Can it be piggy backed when S3 sends RPC to OM to create the file/object?

[github-actions]

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

[github-actions]

Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.

[chungen0126]

Updated and will start to create some pr for this. @ChenSammi @ivandika3 @peterxcli @henrybear327 PTAL.

[ivandika3]

@chungen0126 LGTM +1. Please go ahead with the subtasks.