PXC-5197 [feedback] PXC keyring vault. Information isn't detailed enough. 8.4

patrickbirch · patrickbirch · commit 8c9f2dc34a80 · 2026-04-03T07:51:23.000-05:00
modified:   docs/data-at-rest-encryption.md
diff --git a/docs/data-at-rest-encryption.md b/docs/data-at-rest-encryption.md
@@ -26,6 +26,8 @@ The transit data is defined as data that is transmitted to another node or clien
 
 Percona XtraDB Cluster {{vers}} supports all data at rest generally-available encryption features available from Percona Server for MySQL {{vers}}.
 
+The page is organized around storage backends (`component_keyring_file`, `component_keyring_vault`), [operational maintenance](#operational-maintenance) (including InnoDB master key rotation), [backups and restore for encrypted clusters](#backups-and-restore-for-encrypted-clusters), Vault [authentication lifecycle](#authentication-lifecycle) (tokens, AppRole, Vault Agent or cron), and [disaster recovery](#disaster-recovery) when Vault holds the keys.
+
 ## Use the component_keyring_file
 
 ### Configuration
@@ -90,14 +92,81 @@ Percona XtraDB Cluster does not allow the combination of nodes with encryption a
 
 If a node does not have an encrypted tablespace, the keyring file exists but is empty. Creating an encrypted table on the node populates the keyring file with the necessary encryption keys.
 
-In an operation that is local to the node, you can rotate the key as needed. The `ALTER INSTANCE ROTATE INNODB MASTER KEY` statement is not replicated on cluster.
-
-The JOINER node generates its keyring.
+The JOINER node generates a keyring local to the joiner. InnoDB master key rotation, cluster-wide considerations, and backups are covered in [Operational maintenance](#operational-maintenance) and [Backups and restore for encrypted clusters](#backups-and-restore-for-encrypted-clusters).
 
 ### Compatibility
 
 The Percona XtraDB Cluster SST process with keyring support is backward compatible. A higher-version JOINER can join from a lower-version DONOR, but the reverse is not supported.
 
+## Operational maintenance
+
+### Rotate the InnoDB master key
+
+`ALTER INSTANCE ROTATE INNODB MASTER KEY` replaces the InnoDB master encryption key and re-encrypts existing tablespace keys inside the keyring on the server where the statement runs. Encrypted pages on disk are not rewritten immediately in full; ongoing workload and background operations pick up the new key material according to InnoDB rules described in upstream documentation.
+
+Run rotation on a live member:
+
+```sql
+ALTER INSTANCE ROTATE INNODB MASTER KEY;
+```
+
+Check the error log around the time of rotation for keyring or InnoDB messages. If rotation fails, do not assume later DML on encrypted objects remains safe until the failure is understood and corrected.
+
+#### Replication and cluster behavior
+
+The rotation statement is not replicated through the cluster. Galera does not ship the master key rotation statement in write sets, so each member updates only the keyring state local to that member when you run `ALTER INSTANCE ROTATE INNODB MASTER KEY` on that member.
+
+#### Propagation through the cluster
+
+There is no automatic cluster-wide rotation. To align key age or policy across the cluster, run `ALTER INSTANCE ROTATE INNODB MASTER KEY` on each cluster node in turn during a planned window. Suggested pattern:
+
+1. Pick one member at a time; avoid rotating all members during peak load if the extra I/O is a concern.
+2. On the chosen member, run `ALTER INSTANCE ROTATE INNODB MASTER KEY;` and confirm success in the log.
+3. Repeat for every remaining member. A node that has not been rotated yet keeps the previous master key material for that node until you run the statement on that node.
+4. After the last member completes rotation, verify encrypted workloads on each node (for example `SELECT` from encrypted tables, short maintenance queries).
+
+Plan order and timing so SST and normal replication traffic stay supported. Mixed states (some members rotated, some not) are normal for a short interval; prolonged divergence without a clear reason warrants investigation.
+
+With `component_keyring_file`, each node holds a keyring file local to that node; rotation affects only the node where the command runs. With `component_keyring_vault`, several nodes may share the same Vault mount and secret paths depending on deployment; coordinate rotation with Vault policies and monitoring so every instance still resolves the same logical keys after the operation, and confirm connectivity to Vault before rotating on each node.
+
+GCache encryption uses a separate rotation statement (`ALTER INSTANCE ROTATE GCACHE MASTER KEY`); see [GCache and Write-Set cache encryption](gcache-write-set-cache-encryption.md).
+
+For more background on InnoDB encryption and rotation semantics, see [Percona Server for MySQL: InnoDB data encryption :octicons-link-external-16:](https://docs.percona.com/percona-server/{{vers}}/data-at-rest-encryption.html).
+
+### SST configuration and the `[sst]` section
+
+State Snapshot Transfer (SST) runs XtraBackup on the donor and joiner. If you need additional XtraBackup options so backup or apply stages can access the keyring (non-default paths, plugin or component directories, or similar), declare those options under the `[sst]` group in `my.cnf` using `inno-backup-opts`, `inno-apply-opts`, and `inno-move-opts`. The SST script forwards those options to the corresponding XtraBackup stages; the values must stay consistent with how `mysqld` loads the keyring component on that host.
+
+Other SST tuning (timeouts, transfer format, SSL for the SST channel) also belongs under `[sst]`. For a full list of parameters and examples, see [Percona XtraBackup SST configuration](xtrabackup-sst.md#percona-xtrabackup-sst-configuration).
+
+!!! note
+
+    When using `component_keyring_vault`, SST must use a method that supports Vault (for example XtraBackup-based SST). The section [Configure PXC to use component_keyring_vault component](#configure-pxc-to-use-component_keyring_vault-component) notes that rsync SST is not supported with `component_keyring_vault`.
+
+## Backups and restore for encrypted clusters
+
+Encrypted tablespaces remain opaque without the matching keyring material. Backup design must cover both the data copy and whichever keystore backs the keys.
+
+### With `component_keyring_file`
+
+Include the keyring file pointed to by `component_keyring_file_data` in the same backup policy as the data directory (or copy the keyring from a coordinated snapshot taken at a consistent point in time with the backup vendor’s guidance). Restoring only a data directory without the keyring file that held the tablespace keys prevents decrypting user data.
+
+Percona XtraBackup-based backups read the running server’s keyring configuration; follow [Percona XtraBackup :octicons-link-external-16:](https://docs.percona.com/percona-xtrabackup/{{vers}}/) documentation for privileges, options, and restore procedures specific to your version.
+
+### With `component_keyring_vault`
+
+Physical files from a backup utility are not enough if Vault secrets or access to Vault disappear. A successful restore requires:
+
+* A restorable copy of the database files produced by your backup tool (for example XtraBackup), taken while the backup process could reach Vault using valid credentials.
+* A Vault deployment (or restore of Vault storage) that still contains the keyring secrets at `secret_mount_point`, plus correct KV version and policies.
+* A valid token path for `component_keyring_vault.cnf` on the restored host (often rebuilt via Vault Agent, AppRole, or operator procedures documented in [Authentication lifecycle](#authentication-lifecycle)).
+
+Treat Vault backup, replication, and snapshot procedures as part of the database RPO/RTO. The checklist under [Disaster recovery](#disaster-recovery) complements the backup section.
+
+### Binary logs and other encrypted streams
+
+If binary log encryption or redo encryption is enabled, backup and DR plans must include whatever key material those features require, in line with [Percona Server for MySQL: InnoDB data encryption :octicons-link-external-16:](https://docs.percona.com/percona-server/{{vers}}/data-at-rest-encryption.html).
+
 ## Configure PXC to use component_keyring_vault component
 
 ### component_keyring_vault
@@ -136,10 +205,81 @@ The `secret_mount_point_version` parameter defaults to `AUTO` and controls wheth
 
     Token Security: Avoid embedding long-lived tokens directly in configuration files. Consider using [Vault's AppRole authentication :octicons-link-external-16:](https://developer.hashicorp.com/vault/docs/auth/approle) or dynamic token retrieval mechanisms for enhanced security.
 
-The detailed description of these options can be found in the [Percona Server for MySQL keyring vault component documentation :octicons-link-external-16:](https://docs.percona.com/percona-server/{{vers}}/use-keyring-vault-component.html).
+The detailed description of the keyring vault component options can be found in the [Percona Server for MySQL keyring vault component documentation :octicons-link-external-16:](https://docs.percona.com/percona-server/{{vers}}/use-keyring-vault-component.html).
 
 Vault-server is an external server, so make sure the PXC node can reach the server.
 
+### Authentication lifecycle
+
+The credential in `component_keyring_vault.cnf` is the Vault `token` field (sometimes referred to as `vault_token` in operational runbooks). Vault tokens are not indefinite: each token carries a Time-To-Live (TTL) and expires unless renewed or replaced. When the token expires, the keyring component can no longer read or write keys in Vault, and the server may fail to open encrypted tablespaces or to start.
+
+#### What the server does not do
+
+`component_keyring_vault` reads a bearer token string from JSON configuration and calls Vault’s HTTP API. The keyring component does not implement AppRole login, Kubernetes auth, LDAP, or any other Vault authentication method, and does not renew tokens on a schedule.
+
+The legacy keyring Vault plugin available in older releases behaved the same way at the token layer: the server expected a usable token in configuration rather than performing multi-step Vault login flows.
+
+PXC {{vers}} uses the keyring component model only; the keyring plugin is not supported on 8.4 (see [Upgrade guide](upgrade-guide.md#keyring-plugin-vs-keyring-component)). Regardless of plugin versus component, operators remain responsible for supplying a valid token string through external automation.
+
+#### Minimum external automation
+
+Because the server does not refresh Vault credentials by itself, run at least one of the following on each database host (or deliver an equivalent outcome through your platform):
+
+* [Vault Agent :octicons-link-external-16:](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent) with `auto_auth` (AppRole, AWS, or another method) and a `sink` or `template` that writes a fresh token before the current token’s TTL ends, or
+* A scheduled job (for example `cron`) plus the Vault CLI or API client that logs in, fetches a new token, and updates the JSON the keyring reads—followed by a reload or restart policy your team approves.
+
+Without one of those patterns (or a custom sidecar with the same effect), long-lived static tokens eventually expire and the cluster loses access to keys.
+
+#### Example: AppRole with Vault Agent (illustrative)
+
+The following outlines a common pattern; adjust paths, mount names, and TTLs for production. Full AppRole hardening belongs in [HashiCorp’s AppRole documentation :octicons-link-external-16:](https://developer.hashicorp.com/vault/docs/auth/approle).
+
+1. On the Vault server, enable the AppRole auth method and create a policy that allows the keyring paths your `secret_mount_point` uses (KV read/write as required by [Percona Server keyring vault component documentation :octicons-link-external-16:](https://docs.percona.com/percona-server/{{vers}}/use-keyring-vault-component.html)).
+
+2. Create an AppRole bound to that policy. Distribute `role_id` and `secret_id` to the database host using your secret distribution standard (never commit real values to configuration repositories).
+
+3. Run Vault Agent on the PXC host with `auto_auth` similar to:
+
+```{.text .no-copy}
+pid_file = "/var/run/vault-agent-pid"
+
+vault {
+  address = "https://vault.example.com:8200"
+  ca_cert = "/etc/mysql/vault-ca.pem"
+}
+
+auto_auth {
+  method "approle" {
+    config = {
+      role_id_file_path   = "/etc/mysql/vault/role_id"
+      secret_id_file_path = "/etc/mysql/vault/secret_id"
+    }
+  }
+  sink "file" {
+    config = {
+      path = "/var/lib/mysql-vault/token"
+      mode = 0640
+    }
+  }
+}
+```
+
+4. Point `component_keyring_vault.cnf` at the token the agent maintains. If the component on your build only accepts an inline `token` value, use an agent `template` block to render the full JSON file whenever the token rotates, or run a short wrapper script that copies the sink file into the `token` field and signals a controlled `mysqld` restart. Confirm the exact integration path against the Percona Server version in use.
+
+Equivalent one-off login from an operator shell (useful for testing, not a substitute for renewal automation):
+
+```shell
+vault write -field=token auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"
+```
+
+Paste the returned value into the `token` field only for short-lived tests; production still requires Vault Agent or cron-driven renewal before TTL expiry.
+
+#### Recommended approach summary
+
+Prefer Vault Agent on each PXC host so Vault issues and renews tokens on the local machine. Integrate agent output with `component_keyring_vault.cnf` so the token value stays valid before the credential TTL expires, following your change policy and the [Percona Server keyring vault component documentation :octicons-link-external-16:](https://docs.percona.com/percona-server/{{vers}}/use-keyring-vault-component.html).
+
+Without renewal, expect a cluster-wide outage risk: every node that uses an expired token loses access to encryption keys when those nodes restart or when the keyring next contacts Vault.
+
 !!! warning
 
     SST Limitation: The rsync tool does not support the `component_keyring_vault`. Any rsync-SST on a joiner is aborted if the `component_keyring_vault` is configured.
@@ -195,6 +335,22 @@ CREATE TABLE t1 (c1 INT, PRIMARY KEY pk(c1)) ENCRYPTION='Y';
     ... [ERROR] Component component_keyring_vault reported: 'Could not flush keys to keyring'
     ```
 
+### Disaster recovery
+
+!!! warning
+
+    Vault as a control plane for data access: With `component_keyring_vault`, the Vault service and the network path to Vault become a single point of failure for accessing encrypted data. If Vault is unavailable, nodes may refuse to start or cannot decrypt tablespaces even though data files on disk are intact. High availability, monitoring, and disaster-recovery planning for Vault are as important as planning for the database tier.
+
+Backups and keyring metadata: Day-to-day backup scope for encrypted PXC is described in [Backups and restore for encrypted clusters](#backups-and-restore-for-encrypted-clusters). A physical backup of the datadir alone is not sufficient to recover encrypted data if Vault secrets are lost or if the keyring cannot reach Vault. Your backup and DR procedures must include:
+
+* Vault data and policy: Back up Vault’s own storage (or rely on Vault’s supported replication and snapshot model) so secrets engine data and mounts can be restored. Document mount paths, KV version, and ACL policies used by `component_keyring_vault`.
+
+* Keyring configuration: Securely retain copies of `component_keyring_vault.cnf` (without long-lived plaintext tokens where avoidable), CA material (`vault_ca`), and any automation (for example Vault Agent or AppRole role IDs) needed to obtain a valid token after restore.
+
+* Percona Server / PXC configuration: Include manifest and `my.cnf` fragments that load `component_keyring_vault` so a restored host can load the same component configuration.
+
+Restore drills should verify that a recovered node can authenticate to Vault and that encrypted tables open successfully before relying on the procedure in production.
+
 ## Mix keyring component types
 
 With XtraBackup introducing transition-key logic, you can now mix and match keyring components. For example, node-1 can be configured to use the `component_keyring_file` component while node-2 uses `component_keyring_vault`.
@@ -283,7 +439,7 @@ If the migration fails, the destination keystore is not changed.
 
         For offline migration, no additional key migration options are needed.
 
-* `--keyring-migration-host`: The host where the running server is located. This host is always the local host.
+* `--keyring-migration-host`: The host where the running server is located. The host named by `--keyring-migration-host` is always the local host.
 
 * `--keyring-migration-user`, `--keyring-migration-password`: The username and password for the account used to connect to the running server.
 
@@ -295,7 +451,7 @@ Prerequisite for migration:
 
 Make sure to pass required keyring options and other configuration parameters for the two keyring components. For example, if `component_keyring_file` is one of the components, you must explicitly configure the `component_keyring_file_data` system variable in the my.cnf file.
 
-Other non-keyring options may be required as well. One way to specify these options is by using `--defaults-file` to name an option file that contains the required options.
+Other non-keyring options may be required as well. One way to specify the non-keyring options is by using `--defaults-file` to name an option file that contains the required options.
 
 ```{.text .no-copy}
 [mysqld]
