HDDS-14661. Complete MPU parts-table split (1/N): gate hardening + schemaVersion stamping#122
Closed
kerneltime wants to merge 3 commits into
Closed
HDDS-14661. Complete MPU parts-table split (1/N): gate hardening + schemaVersion stamping#122kerneltime wants to merge 3 commits into
kerneltime wants to merge 3 commits into
Conversation
The upgrade-gate change moved the multipartKeyInfo == null check ahead of the open-key (omKeyInfo) resolution. On the NO_SUCH_MULTIPART_UPLOAD path the request then handed a null omKeyInfo to S3MultipartUploadCommitPartResponse, whose checkAndUpdateDB() dereferences it on that status to GC the orphaned part. When the double buffer flushes the error response it NPEs and the OM terminates -- reachable by a commit-part racing an abort, and independent of schemaVersion (affects legacy uploads). Restore the null-check (and the relocated gate) to their original position, after omKeyInfo is resolved, so the NO_SUCH response carries a non-null part. Add a flush-based regression test that drives the response through checkAndUpdateDB; status-only assertions did not exercise the crash.
Decide the pre-finalization gate from the request's preExecute-stamped layout version instead of a live OMLayoutVersionManager.isAllowed() call inside validateAndUpdateCache. The stamped version travels in the Ratis log, so the gate decision is identical on every replica and on replay, independent of when/how finalization advances the node-local metadata layout version. This matches the leader-side @RequestFeatureValidator pattern already used for ERASURE_CODED_STORAGE_SUPPORT in these requests. Also fix the Complete gate message (it read "commit part request") and drop a stray double blank line left by the Abort gate.
Initiate records schemaVersion = 1 only when the request's preExecute-stamped layout version has finalized MPU_PARTS_TABLE_SPLIT, otherwise 0. The schema is decided once at initiate from replicated state, so an upload's schema never changes mid-flight and any upload initiated before finalization keeps the legacy inline (schemaVersion 0) path through commit/complete/abort even if the cluster finalizes during its lifetime. Covers both the object-store and FSO initiate paths, with unit tests for the pre- and post-finalization cases. Must land together with the v1 commit/complete/abort handling: a schemaVersion 1 entry reaching the legacy inline commit path would violate OmMultipartKeyInfo's schemaVersion == 1 invariant.
This was referenced Jun 3, 2026
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
|
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 1 of a stacked series completing the MPU GC optimization (epic HDDS-10611 — splitting per-part info out of
multipartInfoTableinto the dormantmultipartPartsTable). It is based on theHDDS-14665branch (apache#10062) so it can be merged into that branch directly; the write-side parts will stack on this branch as their own incremental PRs.This part is the foundation: two fixes to the upgrade gate, plus the initiate-time schema stamping the write side builds on.
Commits
HDDS-14665. Fix OM crash on commit part after an aborted upload. The gate change moved the
multipartKeyInfo == nullcheck ahead ofomKeyInforesolution, so theNO_SUCH_MULTIPART_UPLOADGC path handed a null part toS3MultipartUploadCommitPartResponse.checkAndUpdateDB, which dereferences it on that status and NPEs when the double buffer flushes the error response → OM terminate. Reachable by a commit-part racing an abort (the exact scenario the GC branch exists for), and independent ofschemaVersion(it hits legacy uploads). Restores the null-check to its original position afteromKeyInfois resolved, and adds a flush-based regression test (status-only assertions never exercised the crash).HDDS-14665. Make the upgrade gate deterministic; fix gate message. Reads the finalization state from the request's
preExecute-stampedLayoutVersion(replicated in the Ratis log) instead of a liveOMLayoutVersionManager.isAllowed()call insidevalidateAndUpdateCache. The live read is node-local mutable state in the replicated apply path, which the method's own javadoc cautions against ("do NOT bring external dependencies ... could cause divergence in OM DB states in HA"); the stamped value is identical on every replica and on replay, regardless of how/when finalization advances the metadata layout version. This matches the leader-side@RequestFeatureValidator(CLUSTER_NEEDS_FINALIZATION)pattern already used forERASURE_CODED_STORAGE_SUPPORTin these same requests. Also corrects the Complete gate message (it read "commit part request") and removes a stray blank line.HDDS-14661. Stamp MPU schemaVersion at initiate from layout version. Initiate records
schemaVersion = 1only when the stamped layout version has finalizedMPU_PARTS_TABLE_SPLIT, otherwise 0. The schema is decided once, at initiate, from replicated state, so an upload's schema never changes mid-flight and any upload initiated before finalization keeps the legacy inline path through commit/complete/abort even if the cluster finalizes during its lifetime. Object-store and FSO paths, with pre- and post-finalization unit tests.Testing
mvn -pl hadoop-ozone/ozone-manager testfor the MPU initiate/commit/complete/abort request and response classes, object-store and FSO: 67 tests, 0 failures, including the new commit-after-abort flush regression and the pre/post-finalization stamping tests.Coupling note
Commit 3 is safe to merge but must not be finalized in production until the v1 commit/complete/abort handling lands (next PR in this series): a
schemaVersion = 1entry reaching the legacy inline commit path would violateOmMultipartKeyInfo'sschemaVersion == 1invariant. Pre-finalization it is dormant (stamps 0). Commits 1-2 are independently safe and directly harden apache#10062.What's next (stacked on this branch)
CommitPart v1 write → Complete v1 merge → Abort + expired-abort v1 cleanup → listParts v1 → Recon (HDDS-14666), each as its own small PR so the write side can be reviewed incrementally.