Skip to content

HDDS-14661. Complete MPU parts-table split (1/N): gate hardening + schemaVersion stamping#122

Closed
kerneltime wants to merge 3 commits into
spacemonkd:HDDS-14665from
kerneltime:HDDS-14661-1-foundation
Closed

HDDS-14661. Complete MPU parts-table split (1/N): gate hardening + schemaVersion stamping#122
kerneltime wants to merge 3 commits into
spacemonkd:HDDS-14665from
kerneltime:HDDS-14661-1-foundation

Conversation

@kerneltime

Copy link
Copy Markdown

Summary

Part 1 of a stacked series completing the MPU GC optimization (epic HDDS-10611 — splitting per-part info out of multipartInfoTable into the dormant multipartPartsTable). It is based on the HDDS-14665 branch (apache#10062) so it can be merged into that branch directly; the write-side parts will stack on this branch as their own incremental PRs.

This part is the foundation: two fixes to the upgrade gate, plus the initiate-time schema stamping the write side builds on.

Commits

  1. HDDS-14665. Fix OM crash on commit part after an aborted upload. The gate change moved the multipartKeyInfo == null check ahead of omKeyInfo resolution, so the NO_SUCH_MULTIPART_UPLOAD GC path handed a null part to S3MultipartUploadCommitPartResponse.checkAndUpdateDB, which dereferences it on that status and NPEs when the double buffer flushes the error response → OM terminate. Reachable by a commit-part racing an abort (the exact scenario the GC branch exists for), and independent of schemaVersion (it hits legacy uploads). Restores the null-check to its original position after omKeyInfo is resolved, and adds a flush-based regression test (status-only assertions never exercised the crash).

  2. HDDS-14665. Make the upgrade gate deterministic; fix gate message. Reads the finalization state from the request's preExecute-stamped LayoutVersion (replicated in the Ratis log) instead of a live OMLayoutVersionManager.isAllowed() call inside validateAndUpdateCache. The live read is node-local mutable state in the replicated apply path, which the method's own javadoc cautions against ("do NOT bring external dependencies ... could cause divergence in OM DB states in HA"); the stamped value is identical on every replica and on replay, regardless of how/when finalization advances the metadata layout version. This matches the leader-side @RequestFeatureValidator(CLUSTER_NEEDS_FINALIZATION) pattern already used for ERASURE_CODED_STORAGE_SUPPORT in these same requests. Also corrects the Complete gate message (it read "commit part request") and removes a stray blank line.

  3. HDDS-14661. Stamp MPU schemaVersion at initiate from layout version. Initiate records schemaVersion = 1 only when the stamped layout version has finalized MPU_PARTS_TABLE_SPLIT, otherwise 0. The schema is decided once, at initiate, from replicated state, so an upload's schema never changes mid-flight and any upload initiated before finalization keeps the legacy inline path through commit/complete/abort even if the cluster finalizes during its lifetime. Object-store and FSO paths, with pre- and post-finalization unit tests.

Testing

mvn -pl hadoop-ozone/ozone-manager test for the MPU initiate/commit/complete/abort request and response classes, object-store and FSO: 67 tests, 0 failures, including the new commit-after-abort flush regression and the pre/post-finalization stamping tests.

Coupling note

Commit 3 is safe to merge but must not be finalized in production until the v1 commit/complete/abort handling lands (next PR in this series): a schemaVersion = 1 entry reaching the legacy inline commit path would violate OmMultipartKeyInfo's schemaVersion == 1 invariant. Pre-finalization it is dormant (stamps 0). Commits 1-2 are independently safe and directly harden apache#10062.

What's next (stacked on this branch)

CommitPart v1 write → Complete v1 merge → Abort + expired-abort v1 cleanup → listParts v1 → Recon (HDDS-14666), each as its own small PR so the write side can be reviewed incrementally.

The upgrade-gate change moved the multipartKeyInfo == null check ahead of
the open-key (omKeyInfo) resolution. On the NO_SUCH_MULTIPART_UPLOAD path
the request then handed a null omKeyInfo to S3MultipartUploadCommitPartResponse,
whose checkAndUpdateDB() dereferences it on that status to GC the orphaned
part. When the double buffer flushes the error response it NPEs and the OM
terminates -- reachable by a commit-part racing an abort, and independent of
schemaVersion (affects legacy uploads).

Restore the null-check (and the relocated gate) to their original position,
after omKeyInfo is resolved, so the NO_SUCH response carries a non-null part.
Add a flush-based regression test that drives the response through
checkAndUpdateDB; status-only assertions did not exercise the crash.
Decide the pre-finalization gate from the request's preExecute-stamped
layout version instead of a live OMLayoutVersionManager.isAllowed() call
inside validateAndUpdateCache. The stamped version travels in the Ratis
log, so the gate decision is identical on every replica and on replay,
independent of when/how finalization advances the node-local metadata
layout version. This matches the leader-side @RequestFeatureValidator
pattern already used for ERASURE_CODED_STORAGE_SUPPORT in these requests.

Also fix the Complete gate message (it read "commit part request") and
drop a stray double blank line left by the Abort gate.
Initiate records schemaVersion = 1 only when the request's preExecute-stamped
layout version has finalized MPU_PARTS_TABLE_SPLIT, otherwise 0. The schema is
decided once at initiate from replicated state, so an upload's schema never
changes mid-flight and any upload initiated before finalization keeps the
legacy inline (schemaVersion 0) path through commit/complete/abort even if the
cluster finalizes during its lifetime. Covers both the object-store and FSO
initiate paths, with unit tests for the pre- and post-finalization cases.

Must land together with the v1 commit/complete/abort handling: a schemaVersion 1
entry reaching the legacy inline commit path would violate OmMultipartKeyInfo's
schemaVersion == 1 invariant.
@github-actions

Copy link
Copy Markdown

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@github-actions github-actions Bot added the stale label Jun 25, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.

@github-actions github-actions Bot closed this Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant