Spec: REST KMS credential vending for table encryption#17155
Conversation
Change-Id: Ie0f44cf6c273c4e0a0e7a626c1541d290ef7ae20
| For encrypted table operations, delegated credential sourcing should be consistent. If a catalog | ||
| honors REST access delegation for an encrypted table operation, it should provide all delegated | ||
| credentials required for that operation, including storage credentials and KMS credentials. Clients | ||
| must not be required to combine catalog-vended storage credentials with client-local or static KMS |
There was a problem hiding this comment.
nit: "must not be required to combine ..." I support this approach, but the exact phrase does not feel like it fits within the API "spec". Perhaps rephrasing along the lines of "catalogs should provide all credentials required for storage access...". WDYT?
There was a problem hiding this comment.
Agreed, I rewrote this section to be more clear.
| enum: | ||
| - vended-credentials | ||
| - remote-signing | ||
| - kms-vended-credentials |
There was a problem hiding this comment.
This value is mentioned, but not explicitly described in the doc comment (above).
Does it subsume vended-credentials?.. Or should clients declare both (in case KMS is involved)?
There was a problem hiding this comment.
No it doesn't subsume. A client may support storage vended credentials while not kms vended credentials. As per spec such clients will not be given any credentials for encrypted tables, but for unencrypted ones it's still a valid scenario.
Also, in general - I'd rather avoid including extra meaning to an existing access delegation mode.
| Clients should select the credential config by matching KMS key identifiers referenced by table | ||
| encryption metadata, such as `EncryptedKey.encrypted-by-id`, against `kms-key-ids`. If no | ||
| key-management credential matches a KMS key ID required by table encryption metadata, the client | ||
| must fail the operation with an authorization or missing-credential error. |
There was a problem hiding this comment.
"Must" is too strong here, IMHO. Why do we require the client to fail on something that is essentially a server error? If the client has other means of accessing the necessary file, why not allow it to proceed with the operation? I'd say "can" here. WDYT?
There was a problem hiding this comment.
I think we do want the client to act like this. When vended credentials are provided by the catalog, the client should use them, and use them only.
When vended credentials are provided we don't want clients to silently bypass the catalog's KMS authorization decision by using unrelated local credentials for the same delegated operation. That could also re-introduce a hybrid consumption of credentials which we want to avoid.
I clarified this section that failing is only a requirement when vended credentials are provided by catalog side.
| | `rest-page-size` | null | The page size to use when listing namespaces, tables, or other paginated resources. | | ||
| | `namespace-separator` | `%1F` | The separator character used for namespace levels when communicating with the REST server. | | ||
| | `scan-planning-mode` | `CLIENT` | Controls where scan planning is performed. Supported values: `CLIENT` (client-side planning), `SERVER` (server-side planning). Can be overridden per-table by the server in LoadTableResponse. | | ||
| | `encryption.kms-type` | null | Advertises the catalog-selected KMS provider clients should use for encrypted tables. This is independent of the table's storage provider and may be used with `key-management-credentials` to initialize or configure the KMS client. The value is an open enum with initial values `aws`, `gcp`, and `azure`; catalogs may return additional values when the catalog and client agree on KMS client resolution and credential config. | |
There was a problem hiding this comment.
Looks like there is an omission in the previous (Common properties) section - encryption.kms-impl is listed, but the encryption.kms-type is not (even though it is used in HMS, and described in the encryption.md doc). Maybe you could add a line in that section/table? Or I could send a separate patch.
In any case, we'd need to think if encryption.kms-type should be listed in both "Common properties" and "REST catalog properties" sections; if yes, what's their relationship (differences in purpose; how different values are handled; etc).
There was a problem hiding this comment.
Agreed, we need that too, I have it included in this PR now. Also added some more context how any setting of encryption.kms-type would be overridden by server's /config response when KMS credential vending is turned on.
Change-Id: I4b02a9ef06540232493e5d503fe0a4c44717ec16
3 participants
Summary
Adds REST KMS credential vending to the spec for Iceberg table encryption.
This extends REST access delegation so catalogs can return short-lived, scoped KMS credentials alongside storage credentials. Clients match vended KMS credentials to KMS key IDs referenced by Iceberg encryption metadata and continue to call the KMS directly.
Original design doc: https://docs.google.com/document/d/1VSewbVmjukU5eTiZruCJVmUZcRZvcAsd4JOeC6Lt3fo
Changes
This PR adds:
kms-vended-credentialsas an access-delegation mechanismencryption.kms-typeas a catalog-level KMS provider/type identifierkey-management-credentialsto credential-bearing REST responsesKeyManagementCredential, containing:kms-key-idsconfigkms-key-idsandEncryptedKey.encrypted-by-idCredential sourcing
For encrypted table operations, credential sourcing should be consistent with REST access-delegation rules.
If a catalog honors REST access delegation for an encrypted table operation, it should provide all delegated credentials required for that operation, including storage credentials and KMS credentials.
If REST access delegation is not used or not honored, the client remains responsible for providing the required credentials locally.
Not included
This PR does not add remote KMS wrap/unwrap APIs.
Remote wrap/unwrap may be useful for some environments and can be proposed separately, but it has a different protocol shape and security model because plaintext Iceberg key material would flow through the catalog API.
This PR focuses on KMS credential vending, where the catalog returns short-lived, scoped KMS credentials and clients continue to call the KMS directly.