diff --git a/hadoop-hdds/docs/content/design/s3-conditional-requests.md b/hadoop-hdds/docs/content/design/s3-conditional-requests.md index c7e517083817..6e2d1d0eca0e 100644 --- a/hadoop-hdds/docs/content/design/s3-conditional-requests.md +++ b/hadoop-hdds/docs/content/design/s3-conditional-requests.md @@ -3,7 +3,7 @@ title: "S3 Conditional Requests" summary: Design to support S3 conditional requests for atomic operations. date: 2025-11-20 jira: HDDS-13117 -status: draft +status: accepted author: Chu Cheng Li --- >GW: Open key or KEY_ALREADY_EXISTS + opt Open key created + GW->>OM: commitKey() + OM->>OM: Recheck key absence during commit + OM-->>GW: Success or generation mismatch + end + else If-Match: etag + Note over GW,OM: No pre-read for current ETag/updateID on the optimistic path + GW->>OM: createKey(expectedETag = etag) + OM->>OM: Validate ETag, derive updateID, persist expectedDataGeneration + OM-->>GW: Open key with resolved generation or KEY_NOT_FOUND/ETAG_* + opt Open key created + GW->>OM: commitKey() + OM->>OM: Reload open key and reuse atomic rewrite generation check + OM-->>GW: Success or KEY_NOT_FOUND + end + end + GW-->>User: 200 OK or 412 Precondition Failed +``` #### If-None-Match Implementation @@ -120,8 +293,10 @@ public static final long EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS = -1L; ##### S3 Gateway Layer 1. Parse `If-None-Match: *`. -2. Set `existingKeyGeneration = OzoneConsts.EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS`. -3. Call `RpcClient.rewriteKey()`. +2. Use a dedicated client API such as `RpcClient.createKeyIfNotExists()`. +3. The client populates `expectedDataGeneration = + OzoneConsts.EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS` in the + outgoing `KeyArgs`. ##### OM Create Phase @@ -132,50 +307,103 @@ public static final long EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS = -1L; ##### OM Commit Phase (Atomicity) 1. During the commit phase (or strict atomic create), the OM validates that the key still does not exist. -2. If a concurrent client created the key between the Create and Commit phases, the transaction fails with `KET_GENERATION_MISMATCH`. +2. If a concurrent client created the key between the Create and Commit + phases, the transaction fails with a generation-mismatch error. ##### Race Condition Handling -Using `OzoneConsts.EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS = -1` ensures atomicity. If a concurrent write (Client B) commits between Client A's Create and Commit, -Client A's commit fails the `CREATE IF NOT EXISTS` validation check, preserving strict create-if-not-exists semantics. +Using `OzoneConsts.EXPECTED_DATA_GENERATION_CREATE_IF_NOT_EXISTS = -1` +ensures atomicity. If a concurrent write (Client B) commits between +Client A's Create and Commit, Client A's commit fails the +`CREATE IF NOT EXISTS` validation check, preserving strict +create-if-not-exists semantics. -> **Note**: This ability will be added along with [HDDS-13963](https://issues.apache.org/jira/browse/HDDS-13963) (Atomic Create-If-Not-Exists). +> **Note**: This ability will be added along with +> [HDDS-13963](https://issues.apache.org/jira/browse/HDDS-13963) +> (Atomic Create-If-Not-Exists). #### If-Match Implementation -To optimize performance and reduce latency, we avoid a pre-flight check (GetS3KeyDetails) and instead validate the ETag during the OM Write operation. -This requires adding an optional `expectedETag` field to `KeyArgs`. This approach optimizes the "happy path" (successful match) by removing an extra network round trip. -For failing requests, they still incur the cost of a write RPC and Raft log entry, but this is acceptable under optimistic concurrency control assumptions. +`If-Match` should be treated as a compare-and-swap rewrite, not as a +read-then-write sequence in the gateway. The intent is to piggyback on +the existing atomic rewrite machinery without adding a gateway-side +metadata fetch: + +1. The caller sends the ETag it previously observed. +2. The gateway forwards that ETag to OM in the create/open request. +3. OM validates the ETag against the current committed key while + holding the normal bucket/key lock. +4. If the ETag matches, OM extracts the current `updateID` and stores it + as `expectedDataGeneration` in the open key created by `createKey`. +5. The create response returns that open-key state to the client, so the + subsequent `commitKey` request carries the resolved generation rather + than the original ETag predicate. +6. The commit phase then reuses the existing atomic rewrite validation + to detect races before the new object becomes visible. + +This is the optimistic CAS fast path: successful requests avoid an +extra `GetS3KeyDetails` round trip to fetch the current ETag and +`updateID` before issuing the write. Instead, the gateway sends only +the client-supplied ETag, and OM resolves the generation internally as +part of the normal write path. ##### S3 Gateway Layer 1. Parse `If-Match: ""` header. -2. Populate `KeyArgs` with the parsed `expectedETag`. -3. Send the write request (CreateKey) to OM. +2. Call a dedicated client API such as `RpcClient.rewriteKeyIfMatch()` + with the parsed `expectedETag`. +3. The client populates `KeyArgs.expectedETag` and sends the create + request to OM. +4. The gateway does not issue a pre-flight metadata lookup to fetch the + current ETag or `updateID`. +5. Once OM returns the open key with resolved + `expectedDataGeneration`, the normal output-stream commit path carries + that generation on `commitKey`. ##### OM Create Phase -Validation is performed within the `validateAndUpdateCache` method to ensure atomicity within the Ratis state machine application. +Validation is performed within the `validateAndUpdateCache` method to +ensure atomicity within the Ratis state machine application. 1. **Locking**: The OM acquires the write lock for the bucket/key. 2. **Key Lookup**: Retrieve the existing key from `KeyTable`. 3. **Validation**: - **Key Not Found**: If the key does not exist, throw `KEY_NOT_FOUND` (maps to S3 412). - - **No ETag Metadata**: If the existing key (e.g., uploaded via OFS) does not have an ETag property, throw `ETAG_NOT_AVAILABLE` (maps to S3 412). The precondition cannot be evaluated, so we must fail rather than silently proceed. - - **ETag Mismatch**: Compare `existingKey.ETag` with `expectedETag`. If they do not match, throw `ETAG_MISMATCH` (maps to S3 412). -4. **Extract Generation**: If ETag matches, extract `existingKey.updateID`. -5. **Create Open Key**: Create open key entry with `expectedDataGeneration = existingKey.updateID`. + - **No ETag Metadata**: If the existing key (e.g., uploaded via + OFS) does not have an ETag property, throw + `ETAG_NOT_AVAILABLE` (maps to S3 412). The precondition cannot + be evaluated, so we must fail rather than silently proceed. + - **ETag Mismatch**: Compare `existingKey.ETag` with + `expectedETag`. If they do not match, throw `ETAG_MISMATCH` + (maps to S3 412). +4. **Extract Generation**: If ETag matches, extract + `existingKey.updateID`. +5. **Bridge to Atomic Rewrite**: Create the open key entry with + `expectedDataGeneration = existingKey.updateID`, so the remainder of + the flow uses the same atomic rewrite invariant as HDDS-10656. +6. **Return Resolved State**: The create response returns the open-key + metadata containing that generation, so the later `commitKey` request + can carry the same resolved rewrite condition. ##### OM Commit Phase -The commit phase reuses the existing atomic-rewrite validation logic from HDDS-10656: +The commit phase reuses the existing atomic rewrite validation logic +from HDDS-10656: -1. Read open key entry (contains `expectedDataGeneration` set during create phase). +1. Read open key entry (contains `expectedDataGeneration` set during + create phase from the ETag-validated key). 2. Read current committed key from `KeyTable`. 3. Validate `currentKey.updateID == openKey.expectedDataGeneration`. -4. If match, commit succeeds. If mismatch (concurrent modification), throw `KEY_NOT_FOUND` (maps to S3 412). +4. If match, commit succeeds. If mismatch (concurrent modification), + throw `KEY_NOT_FOUND` (maps to S3 412). +5. Clear the conditional fields before persisting the final committed + key so they remain open-key state only. -This approach ensures end-to-end atomicity: even if another client modifies the key between Create and Commit phases, the commit will fail. +This approach ensures end-to-end atomicity: even if another client +modifies the key between Create and Commit phases, the commit will +fail. The gateway never needs to fetch `updateID` itself; OM derives it +from the matched ETag during `createKey`, and the rest of the write then +rides on the standard atomic rewrite path. #### Error Mapping @@ -187,19 +415,361 @@ This approach ensures end-to-end atomicity: even if another client modifies the |`ETAG_NOT_AVAILABLE`|412|PreconditionFailed|If-Match failed (key has no ETag, e.g., created via OFS)| |`ETAG_MISMATCH`|412|PreconditionFailed|If-Match failed (ETag mismatch)| +## AWS S3 Conditional CompleteMultipartUpload Implementation + +Conditional MPU completion should reuse the same conditional write fields +already present in `KeyArgs`, but unlike normal conditional `PUT`, it +does not need an open-key plus later commit bridge. The final key is +assembled and committed inside one OM transaction in +`S3MultipartUploadCompleteRequest.validateAndUpdateCache(...)` while +holding the normal bucket write lock. + +### Request Sequence + +```mermaid +sequenceDiagram + actor User + participant GW as S3 Gateway + participant OM as Ozone Manager + + User->>GW: CompleteMultipartUpload with optional If-None-Match / If-Match + GW->>GW: Parse conditional write headers + GW->>OM: completeMultipartUpload(keyArgs + partsList) + OM->>OM: Acquire bucket lock + OM->>OM: Load current committed key and MPU state + OM->>OM: Validate destination precondition + alt Precondition failed + OM-->>GW: KEY_ALREADY_EXISTS / KEY_NOT_FOUND / ETAG_* + GW-->>User: 412 Precondition Failed + else Preconditions pass + OM->>OM: Validate MPU parts and build final key + OM->>OM: Write key table entry and remove MPU state + OM-->>GW: Success + GW-->>User: 200 OK + end +``` + +### Gateway Flow + +1. Parse `If-None-Match` / `If-Match` using the same shared helper used + by conditional `PUT`. +2. Extend the existing `completeMultipartUpload` client API, or add a + conditional overload, so the gateway can pass the parsed condition to + OM through `OmKeyArgs`. +3. Populate the request as follows: + - `If-None-Match: *` -> + `expectedDataGeneration = OzoneConsts.EXPECTED_GEN_CREATE_IF_NOT_EXISTS` + - `If-Match: ""` -> `expectedETag = ` +4. Ensure the OM protocol translator copies these optional fields from + `OmKeyArgs` into the `CompleteMultiPartUploadRequest` `KeyArgs`. +5. Map conditional validation failures to the same S3 + `PreconditionFailed` response used by the other conditional write + paths. + +### OM Validation + +Validation should occur in +`S3MultipartUploadCompleteRequest.validateAndUpdateCache(...)` after the +request acquires the bucket write lock and before it assembles the final +key from the uploaded parts. + +The proposed flow is: + +1. Load the current committed destination key from `keyTable`. +2. Reuse the same `If-Match` validation helper already used by + conditional `PUT` to convert `expectedETag` into + `expectedDataGeneration`. +3. Reuse the same atomic rewrite validation helper to evaluate: + - create-if-absent semantics for `If-None-Match: *` + - generation/ETag match semantics for `If-Match` +4. If validation passes, continue with the existing MPU complete logic: + validate part order, validate part identity, compute the MPU ETag, + write the final key, and delete the multipart metadata/open-key + state. +5. Clear conditional-only fields before persisting the committed key so + they remain request-scoped metadata rather than part of the final key + state. + +Because the final destination validation and the key-table write happen +under the same OM bucket lock in one request, this path does not need a +separate second-phase commit revalidation step like conditional +`PutObject`. + +### Error Mapping + +| | | | | +|---|---|---|---| +|**OM Error**|**S3 Status**|**S3 Error Code**|**Scenario**| +|`KEY_ALREADY_EXISTS`|412|PreconditionFailed|`If-None-Match` failed because a committed destination object already exists| +|`KEY_NOT_FOUND`|412|PreconditionFailed|`If-Match` failed because the current destination object is missing| +|`ETAG_NOT_AVAILABLE`|412|PreconditionFailed|Destination key has no ETag metadata| +|`ETAG_MISMATCH`|412|PreconditionFailed|Destination ETag mismatch| + +If Ozone later introduces an explicit conflict result for this path, the +gateway should map it to `409 ConditionalRequestConflict`, reusing the +same S3 error mapping as conditional `PutObject`. The initial MPU +complete design does not require a dedicated second-phase conflict +signal because the validation and final key update are already +serialized in one OM transaction. + ## AWS S3 Conditional Read Implementation -TODO +Conditional reads can be implemented fully in the S3 gateway. Unlike conditional writes, no OM write-path changes are +required because the operation is read-only and the gateway already fetches object metadata before streaming the body. + +### Request Sequence + +```mermaid +sequenceDiagram + actor User + participant GW as S3 Gateway + participant OM as Ozone Manager + + User->>GW: GET/HEAD with conditional headers + GW->>OM: getS3KeyDetails() / headS3Object() + OM-->>GW: ETag, modificationTime, key info + GW->>GW: Evaluate ETag/date precedence rules + alt Not modified + GW-->>User: 304 Not Modified + else Precondition failed + GW-->>User: 412 Precondition Failed + else Preconditions pass + opt GET or ranged GET + GW->>OM: getKey() / open data stream + OM-->>GW: Object data + end + GW-->>User: 200 OK / 206 Partial Content + end +``` + +### Gateway Flow + +1. Parse `If-Match`, `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since` from the request. +2. Fetch object metadata using the existing read path: + - `getS3KeyDetails()` for `GetObject` + - `headS3Object()` for `HeadObject` +3. Evaluate the conditional headers against: + - `OzoneConsts.ETAG` for ETag-based checks + - `modificationTime` for date-based checks +4. If evaluation returns `Not Modified`, return `304` immediately without opening the data stream. +5. If evaluation returns `Precondition Failed`, return `412` immediately without opening the data stream. +6. Only when all preconditions pass should the gateway continue with `Range`, `partNumber`, and body streaming. + +### ETag Availability + +Date-based validators work for all keys because `modificationTime` is always available. ETag-based validators need more +care because keys written outside the S3 gateway may not have `OzoneConsts.ETAG` metadata. + +The proposed behavior is: + +- `If-Match` fails with `412` when the key has no ETag metadata, because the gateway cannot prove equality. +- `If-None-Match` is treated as "not matched" when the key has no ETag metadata, so the read may proceed. + +This keeps the implementation conservative for positive matches while still allowing Ozone-native keys to participate +in cache-validation flows based on modification time. + +### Reusable Evaluator + +It is worth introducing one shared helper in the S3 gateway, for example in `EndpointBase`, that evaluates the +combination rules once and returns one of three outcomes: + +- `PROCEED` +- `NOT_MODIFIED` +- `PRECONDITION_FAILED` + +`GetObject` and `HeadObject` can use this helper directly, while `CopyObject` can reuse the same logic for its +source-side validation and remap `NOT_MODIFIED` to `PRECONDITION_FAILED`, matching AWS copy semantics. ## AWS S3 Conditional Copy Implementation -TODO +Conditional copy should reuse both halves of this design: + +- the source-side read validator from conditional `GET` / `HEAD` +- the destination-side atomic write path from conditional `PUT` + +### Request Sequence + +```mermaid +sequenceDiagram + actor User + participant GW as S3 Gateway + participant OM as Ozone Manager + + User->>GW: COPY with source and destination conditions + GW->>OM: get source key details + OM-->>GW: Source metadata + bound content stream + GW->>GW: Evaluate x-amz-copy-source-if-* headers + alt Source precondition failed + GW-->>User: 412 Precondition Failed + else Source passes + GW->>OM: create destination key (normal / If-None-Match / If-Match) + OM->>OM: Validate destination state and open key + OM-->>GW: Open key or 412-mapped error + opt Destination open key created + GW->>OM: commit destination key using source snapshot + OM->>OM: Revalidate generation for conditional writes + OM-->>GW: Success or 412-mapped error + end + GW-->>User: 200 OK or 412 Precondition Failed + end +``` + +### Source Validation + +The current gateway implementation already fetches source metadata before a copy. The proposed change is to make that +metadata lookup authoritative for the entire copy flow: + +1. Resolve the source object into one `OzoneKeyDetails`. +2. Evaluate all `x-amz-copy-source-if-*` headers against that snapshot. +3. If the source preconditions fail, return `412` before any destination key is opened. +4. Reuse `sourceKeyDetails.getContent()` for the copy stream instead of performing a second `getKey()` lookup by name. + +Step 4 is important. `OzoneKeyDetails` already captures the `OmKeyInfo` used for validation and exposes a content +supplier bound to that snapshot. Reusing it avoids a time-of-check / time-of-use gap where the source key could be +re-resolved to a different generation after the precondition checks have already passed. + +### Destination Validation + +Destination conditions should reuse the conditional write APIs introduced for `PutObject`: + +- no destination header: `createKey` +- `If-None-Match: *`: `createKeyIfNotExists` +- `If-Match: ""`: `rewriteKeyIfMatch` + +This gives `CopyObject` the same atomic destination guarantees as conditional `PutObject`: + +1. Validate the current destination state during the open/create phase. +2. Persist the expected destination generation in the open key. +3. Revalidate the generation during commit so concurrent overwrites fail atomically. + +In practice, once conditional `PUT` support exists, most destination-side copy work stays in the gateway and simply +selects the correct client API, exactly as `PutObject` multiplexes between normal create and conditional create/rewrite. + +### Metadata, Tags, and Multipart Copy + +Conditional support should remain orthogonal to the existing metadata/tagging directives: + +- source preconditions are evaluated first +- metadata and tag directives are resolved next +- destination creation mode is selected last + +`UploadPartCopy` can reuse the same source-side evaluator. It does not need the destination `If-Match` / +`If-None-Match` handling because the destination is a multipart upload part rather than a committed object key. + +### Error Mapping + +| | | | | +|---|---|---|---| +|**Failure point**|**OM/Gateway result**|**S3 Status**|**Scenario**| +|Source validator|Gateway precondition failure|412|Source ETag/date condition failed| +|Destination validator|`KEY_ALREADY_EXISTS`|412|`If-None-Match` failed at destination| +|Destination validator|`KEY_NOT_FOUND`|412|`If-Match` failed because destination is missing or changed| +|Destination validator|`ETAG_NOT_AVAILABLE`|412|Destination key has no ETag metadata| +|Destination validator|`ETAG_MISMATCH`|412|Destination ETag mismatch| + +This initial design intentionally reuses the `412` mapping already described for conditional writes. If Ozone later +wants to distinguish in-flight destination races with a dedicated `409 ConditionalRequestConflict`, that can be added +without changing the overall structure above. + +## AWS S3 Conditional Delete Implementation + +Conditional delete is simpler than conditional write or copy because the +delete path is already a single OM transaction. There is no open-key / +commit split, so atomicity comes from validating the precondition and +applying the tombstone while holding the normal delete lock in one +request. + +### Request Sequence + +```mermaid +sequenceDiagram + actor User + participant GW as S3 Gateway + participant OM as Ozone Manager + + User->>GW: DELETE with optional If-Match + alt No If-Match + GW->>OM: deleteKey() + OM->>OM: Apply existing delete semantics + OM-->>GW: Success or KEY_NOT_FOUND + GW-->>User: 204 No Content + else If-Match present + GW->>OM: deleteKey(expectedETag or *) + OM->>OM: Validate current key under lock and write tombstone + OM-->>GW: Success or KEY_NOT_FOUND/ETAG_* + GW-->>User: 204 No Content or 412 Precondition Failed + end +``` + +### Gateway Flow + +1. Parse the optional `If-Match` header. +2. If absent, continue to call the existing delete path. +3. If present, call a conditional delete client API, or extend the + existing delete request to carry the expected match value. +4. Map conditional failures to `412 Precondition Failed` instead of the + unconditional delete behavior of returning `204` on missing keys. + +The gateway should not pre-read the object to validate the ETag. Doing +so would add an avoidable round trip and reintroduce a race. As with +conditional `PUT`, the validation should happen in OM under the bucket +write lock. + +### OM Validation + +The delete request already acquires the bucket write lock and resolves +the target key inside `validateAndUpdateCache`. Conditional delete can +reuse that path with one extra validation step before the tombstone is +written. + +The proposed behavior is: + +1. Look up the current key in `KeyTable`. +2. If `If-Match` is absent, keep the existing behavior. +3. If `If-Match` is `"*"`, require that the key exists. +4. If `If-Match` contains an ETag, require that the key exists and that + `OzoneConsts.ETAG` matches the supplied value. +5. If the key has no ETag metadata, fail the request rather than + deleting an object whose precondition cannot be evaluated. +6. If validation succeeds, continue with the normal delete flow: + update `updateID`, write the tombstone, and account for quota + release. + +Because this all happens in one OM request under lock, there is no need +for an extra generation field or a second-phase revalidation. + +### Reuse of Conditional Write Fields + +If `expectedETag` is added to `KeyArgs` for conditional `PUT`, the same +field can be reused by `DeleteKeyRequest`. The `If-Match: *` case can be +represented either as the literal `*` in `expectedETag` or by a separate +boolean marker. The simpler design is to reuse the literal header value +and interpret it in OM. + +### Error Mapping + +| | | | | +|---|---|---|---| +|**Failure point**|**OM/Gateway result**|**S3 Status**|**Scenario**| +|Conditional delete|`KEY_NOT_FOUND`|412|`If-Match` present but the key does not exist| +|Conditional delete|`ETAG_NOT_AVAILABLE`|412|Key exists but has no ETag metadata| +|Conditional delete|`ETAG_MISMATCH`|412|Supplied ETag does not match current object| + +One gateway nuance is important: today `DeleteObject` maps +`KEY_NOT_FOUND` to `204 No Content` for unconditional deletes. That +mapping must stay conditional on the absence of `If-Match`; otherwise +`If-Match: *` would incorrectly succeed on missing keys. ## References - [AWS S3 Conditional Requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html) +- [AWS S3 Conditional Writes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html) +- [AWS S3 Conditional Deletes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-deletes.html) +- [AWS S3 CompleteMultipartUpload API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) +- [AWS S3 DeleteObject API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) - [RFC 7232 - HTTP Conditional Requests](https://tools.ietf.org/html/rfc7232) - [HDDS-10656 - Atomic Rewrite Key](https://issues.apache.org/jira/browse/HDDS-10656) - [HDDS-13963 - Atomic Create-If-Not-Exists](https://issues.apache.org/jira/browse/HDDS-13963) +- [HDDS-14907 - Conditional Delete (DeleteObject)](https://issues.apache.org/jira/browse/HDDS-14907) - [Leader Election with S3 Conditional Writes](https://www.morling.dev/blog/leader-election-with-s3-conditional-writes/) - [An MVCC-like columnar table on S3 with constant-time deletes](https://simonwillison.net/2025/Oct/11/mvcc-s3/)