HDDS-14660. Implement new table to store part information for multipart uploads by spacemonkd · Pull Request #9886 · apache/ozone

spacemonkd · 2026-03-07T18:42:13Z

What changes were proposed in this pull request?

HDDS-14660. Implement new table to store part information for multipart uploads

Please describe your PR in detail:

This PR adds a new multipartPartsTable to handle storing of parts for Multipart Uploads.
The existing MultipartKeyInfo is updated to handle new fields like owner and ACLs, volume, bucket and key name.
The codec for PartKey and PartInformation has been added.
The new table is added to the list of compacted tables

How is the PartKey encoded?

Say for the below sample input (we are taking 47 as it has 2F in the lower byte):

uploadId = <abcd-1234...>
partNumber = 47 -> taking 47 because it has 2f in the byte representation

Encoding format:
keyBytes = UTF8(uploadId) + / + int32_be(partNumber)

For 47:
int32 big-endian bytes = 00 00 00 2f

So final tail is:

separator / = 2f
part bytes = 00 00 00 2f
Total tail bytes = 2f 00 00 00 2f

Now we run the following checks:

Check full-key first: if byte at len - 5 is /, decode as full key (used to identify full row)
Else, if byte at len - 1 is /, decode as prefix. (this is used for prefix scan for iterating)
Else invalid.

A logical key will look something like the following in the RocksDB entry:

[61 62 63 31 32 33 2d 75 75 69 64 2d 34 35 36 2f 00 00 00 2f]
[--------------uploadId UTF-8---------------][2f][-int32 BE-]

and will be represented as:

{
  uploadID: "abcd123-uuid-456",
  partNumber: 47
} : {
... 
}

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14660

How was this patch tested?

Patch has been tested by running unit tests.

…rt uploads

…ckstyle and findbug

ivandika3

Thanks @spacemonkd for the patch, overall LGTM. Left some comments,

ivandika3

Thanks for the update. LGTM +1.

kerneltime · 2026-03-16T18:21:16Z

What is the plan for layout management? Since on disk structures are changing.

spacemonkd · 2026-03-16T18:27:32Z

@kerneltime We are putting behind a gate check.
schemaVersion: 0 vs schemaVersion: 1 is checked in the actual flow.
For partial upgrades where say one OM is upgraded but finalization is not done, the new field will still be 0 and hence the requests will still refer to the in-memory list.

When the upgrade is finalized then the client will populate schemaVersion: 1 which will redirect new writes to the parts table.
You can refer to below PoC code:

Example gate:

if (!ozoneManager.getVersionManager().isAllowed(OMLayoutFeature.MPU_PARTS_TABLE_SPLIT)
          && multipartKeyInfo.getSchemaVersion() != 0) {
        throw new OMException("MPU parts-table split behavior is not allowed " +
            "before cluster finalization.",
            OMException.ResultCodes.NOT_SUPPORTED_OPERATION_PRIOR_FINALIZATION);
      }

ivandika3 · 2026-04-02T03:52:40Z

@kerneltime Any more comments?

@rakeshadr PTAL

rakeshadr

Thanks @spacemonkd for the patch. Added a few comments, please check it.

rakeshadr

Thanks @spacemonkd for the continuous efforts. LGTM +1
Please reply to the conservations and resolve it, then proceed with merge.

spacemonkd · 2026-04-09T11:35:44Z

Thanks for the reviews @ivandika3 @rakeshadr.
Merging this

szetszwo · 2026-05-22T17:04:33Z

+      final ByteBuffer uploadIdBuffer = input.duplicate();
+      uploadIdBuffer.limit(separatorIndex);
+      uploadIdBuffer.position(start);
+      String uploadId = StandardCharsets.UTF_8.decode(uploadIdBuffer).toString();


@spacemonkd , We should use StringCodec since UTF_8.decode(..) below uses CodingErrorAction.REPLACE. When there is an error, it will silently replace characters and cause unexpected behavior.

When there is an error, the correct behavior is to throw an exception. Filed HDDS-15348.

// java.nio.charset.Charset.java public final CharBuffer decode(ByteBuffer bb) { try { return ThreadLocalCoders.decoderFor(this) .onMalformedInput(CodingErrorAction.REPLACE) .onUnmappableCharacter(CodingErrorAction.REPLACE) .decode(bb); } catch (CharacterCodingException x) { throw new Error(x); // Can't happen } }

HDDS-14660. Implement new table to store part information for multipa…

7a80416

…rt uploads

spacemonkd requested review from adoroszlai, errose28, ivandika3 and rakeshadr March 7, 2026 18:42

spacemonkd added 4 commits March 8, 2026 00:30

Add checks for schemaVersion 1 to not have parts list inline, fix che…

49ed677

…ckstyle and findbug

Fix TestOmMetadataManager

a14cb69

Add multipartPartsTable defs to OmMetadataManager

c627ce9

Fix build issues

45be606

spacemonkd marked this pull request as ready for review March 8, 2026 18:15

spacemonkd self-assigned this Mar 8, 2026

spacemonkd added the s3 S3 Gateway label Mar 8, 2026

spacemonkd requested a review from ChenSammi March 8, 2026 18:16

ChenSammi reviewed Mar 10, 2026

View reviewed changes

Comment thread hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/codec/OMDBDefinition.java Outdated

Fix table format comment

12234c3

spacemonkd requested a review from ChenSammi March 12, 2026 18:29

ivandika3 reviewed Mar 15, 2026

View reviewed changes

Address review comments, add more tests

6d0591b

spacemonkd requested a review from ivandika3 March 15, 2026 15:41

ivandika3 approved these changes Mar 16, 2026

View reviewed changes

rakeshadr reviewed Apr 7, 2026

View reviewed changes

rakeshadr reviewed Apr 8, 2026

View reviewed changes

Comment thread ...op-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmMultipartPartKey.java

Address review comments

ad3fecb

rakeshadr approved these changes Apr 9, 2026

View reviewed changes

Address build issues

eace4f4

spacemonkd merged commit 45ffbf3 into apache:master Apr 9, 2026
45 checks passed

spacemonkd deleted the HDDS-14660 branch April 9, 2026 11:35

szetszwo reviewed May 22, 2026

View reviewed changes

Uh oh!

Conversation

spacemonkd commented Mar 7, 2026

What changes were proposed in this pull request?

How is the PartKey encoded?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

ivandika3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ivandika3 left a comment

Choose a reason for hiding this comment

Uh oh!

kerneltime commented Mar 16, 2026

Uh oh!

spacemonkd commented Mar 16, 2026

Uh oh!

ivandika3 commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rakeshadr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rakeshadr left a comment

Choose a reason for hiding this comment

Uh oh!

spacemonkd commented Apr 9, 2026

Uh oh!

Uh oh!

szetszwo May 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ivandika3 commented Apr 2, 2026 •

edited

Loading