PS-10921 [DOCS] - [feedback] PS 8.4 keyring vault component

	modified:   docs/use-keyring-vault-component.md
	modified:   snippets/keyring-components-installation.md

Author: patrickbirch

docs/use-keyring-vault-component.md

@@ -1,6 +1,6 @@
 # Use the keyring vault component
 
-The `keyring_vault` component extends the server capabilities and provides an interface for the database with a [HashiCorp Vault :octicons-link-external-16:](https://www.hashicorp.com/products/vault/data-protection) server to store key and secure encryption keys. 
+The `keyring_vault` component extends the server capabilities and provides an interface for the database with a [HashiCorp Vault :octicons-link-external-16:](https://www.hashicorp.com/products/vault/data-protection) server to store key and secure encryption keys.
 
 --8<--- "keyring-components-installation.md"
 
@@ -92,7 +92,7 @@ The `secret_mount_point_version` can be either a `1`, `2`, `AUTO`, or the `secre
 
 If you set the `secret_mount_point_version` to `2` but the path pointed by `secret_mount_point` is based on `KV Secrets Engine - Version 1 (kv)`, an error is reported, and the component fails to initialize.
 
-If you set the `secret_mount_point_version` to `1` but the path pointed by `secret_mount_point` is based on `KV Secrets Engine - Version 2 (kv-v2)`, the component initialization succeeds but any MySQL keyring-related operations fail.
+If you set the `secret_mount_point_version` to `1` but the path pointed by `secret_mount_point` is based on `KV Secrets Engine - Version 2 (kv-v2)`, the component initialization succeeds but any server keyring-related operations fail.
 
 ### Upgrade from Vault Secrets Engine Version 1 to Version 2
 
@@ -108,6 +108,97 @@ Use either of the following methods:
 
     The `keyring_vault` component that works with `kv-v2` secret engines does not use the built-in key versioning capabilities. The keyring key versions are encoded into key names.
 
+## Maintain the Vault connection
+
+Installation is only the first step. In production, the most common cause of `keyring_vault` failures is not a bad configuration but a lost connection to Vault — either because the token the component uses has expired, or because the Vault server itself has sealed. Both leave Percona Server unable to fetch master keys.
+
+The `keyring_vault` component reads the `token` value from the configuration file at startup and uses the stored token for the lifetime of the running server process. The component does not renew the token automatically and does not reread the configuration file while the server is running. Any renewal or rotation must happen outside MySQL, on the Vault side or through a helper process.
+
+### Token expiration and renewal
+
+Every Vault token has a TTL. When the TTL elapses, the token is revoked and any API call made with the revoked token — including the `keyring_vault` component's fetch of a master key — fails with a 403 from Vault.
+
+Symptoms of an expired or revoked token include:
+
+* Startup failure of `mysqld` after a restart, with keyring component errors about initialization or key fetch.
+
+* Failure of `ALTER INSTANCE ROTATE INNODB MASTER KEY`.
+
+* Failure to open an encrypted table after a restart, returning a keyring error.
+
+* A row in `performance_schema.keyring_component_status` showing the component loaded but unable to reach Vault.
+
+Because the component does not renew tokens, pick one of the following patterns so the token the component reads on startup is always valid:
+
+#### Option 1 (recommended): Periodic tokens
+
+A [periodic token](https://developer.hashicorp.com/vault/docs/concepts/tokens#periodic-tokens) has no maximum TTL. As long as the token is renewed within each period, the token continues to work indefinitely. This renewal model matches how a long-running database server uses Vault.
+
+Create a role that issues periodic tokens and use the role to mint the token placed in `component_keyring_vault.cnf`:
+
+```bash
+vault write auth/token/roles/percona-keyring \
+    allowed_policies="percona-keyring-policy" \
+    period="24h" \
+    renewable=true
+
+vault token create -role=percona-keyring -format=json
+```
+
+Store the resulting token in the `token` field of your `keyring_vault` configuration file. Pair the periodic token with a renewer (see Option 2) so the token is renewed at least once per `period`.
+
+#### Option 2: Sidecar renewer
+
+Regardless of whether the token is periodic or has a fixed max TTL, a sidecar process must call `vault token renew` before each expiry. Two common implementations:
+
+* `vault agent` configured with an `auto_auth` block. The agent authenticates, writes the resulting token to a sink, and renews the token automatically. Point `component_keyring_vault.cnf` at a token minted from the same auth method, or generate the config with agent templating.
+
+* A systemd timer or cron job that runs `vault token renew` on a schedule shorter than the token TTL (for example, every 1 hour for a 24-hour token). Run the job as a dedicated service account, not as root, and log renewal failures to your alerting system.
+
+Whichever you pick, alert on renewal failure. A silently failing renewer is indistinguishable from no renewer at all until the next MySQL restart — at which point the server will not come back up.
+
+#### Operational checklist
+
+* Treat the `token` value as a secret on par with the Vault unseal keys. Restrict filesystem permissions on `component_keyring_vault.cnf` to the `mysql` user.
+
+* Monitor token TTL with `vault token lookup` and alert when remaining TTL drops below a safe threshold.
+
+* Rotate the token only during a maintenance window: update the configuration file, then restart `mysqld`. The component reads the file only at startup.
+
+* Share one token with only one server. Pair one-token-per-server with the existing `secret_mount_point` warning earlier on this page.
+
+### Vault seal status
+
+On startup, and every time the component performs a key operation, `keyring_vault` reaches the Vault HTTP API. If Vault is [sealed](https://developer.hashicorp.com/vault/docs/concepts/seal), that API returns a 503 for any request against the secrets engine, and the component cannot read or write keys — even with a perfectly valid token.
+
+Vault becomes sealed in several ordinary situations:
+
+* A host reboot or package upgrade restarts the Vault process.
+
+* An operator manually runs `vault operator seal`.
+
+* Vault seals itself in response to a detected integrity issue.
+
+While Vault is sealed, Percona Server behavior depends on timing:
+
+* If `mysqld` is already running and keys for currently-open tables are cached in memory, reads and writes against those tables continue to work.
+
+* If an operation needs a new key fetch — opening an encrypted table that was not already open, rotating the master key, or creating an encrypted tablespace — the operation fails.
+
+* If `mysqld` is restarted while Vault is sealed, InnoDB cannot unwrap tablespace keys, and startup fails or encrypted tables remain inaccessible until Vault is unsealed.
+
+#### Recommended practice
+
+* Unseal Vault before starting or restarting Percona Server. Confirm with `vault status` that `Sealed` is `false`.
+
+* Automate unsealing with auto-unseal. Use [auto-unseal](https://developer.hashicorp.com/vault/docs/concepts/seal#auto-unseal) backed by a cloud KMS, HSM, or Transit secret engine so that a Vault restart does not require manual intervention. Without auto-unseal, a Vault reboot that happens outside business hours can block every Percona Server that depends on the sealed Vault.
+
+* Order your startup dependencies. If Vault and MySQL run on the same host or are managed by the same orchestrator, make `mysqld` depend on Vault being unsealed, not merely on Vault being reachable.
+
+* Monitor `sys/health`. Vault's `/v1/sys/health` endpoint reports seal status. Alert on `sealed=true` independently of whether any database has noticed yet, so operators can act before the next MySQL restart.
+
+* Keep unseal keys and recovery keys offline and distributed. This key custody practice is a Vault concern rather than a Percona one, but a sealed Vault whose unseal keys are lost is equivalent to permanent data loss for every Percona Server that depended on that Vault.
+
 !!! admonition "See also"
 
     [Hashicorp Documentation: Installing Vault :octicons-link-external-16:](https://www.vaultproject.io/docs/install/index.html)

snippets/keyring-components-installation.md

@@ -1,16 +1,36 @@
 
-The component must be installed with a manifest. A keyring component is not loaded with the `--early-plugin-load` option on the server. The server uses a manifest and the component consults its configuration file during initialization. You should only load a keyring component with a manifest file. Do not use the `INSTALL_COMPONENT` statement, which loads the keyring components too late in the startup sequence of the server. For example, `InnoDB` requires the component, but because the components are registered in the `mysql.component` table, this table is loaded after `InnoDB` initialization.
+Percona Server relies on a keyring to safeguard the master keys that encrypt data at rest. A manifest file loads the keyring component, which reads a separate configuration file during initialization. Avoid `--early-plugin-load` and `INSTALL COMPONENT`: neither mechanism can load a keyring early enough in startup.
 
-You should create a global manifest file named `mysqld.my` in the installation directory and, optionally, create a local manifest file, also named `mysqld.my` in a data directory.
+### Why the manifest is the only supported load path
 
-To install a keyring component, do the following:
+The keyring must be live before `InnoDB` opens an encrypted page, which rules out any mechanism that depends on a running SQL layer. A typical startup proceeds in this order:
 
-1. Write a manifest in a valid JSON format
+1. `mysqld` parses startup configuration and reads the manifest file next to the binary.
 
-2. Write a configuration file
+2. The server loads components named in the manifest.
 
-A manifest file indicates which component to load. If the manifest file does not exist, the server does not load the component associated with that file. During startup, the server reads the global manifest file from the installation directory. The global manifest file can contain the required information or point to a local manifest file located in the data directory. If you have multiple server instances that use different keyring components use a local manifest file in each data directory to load the correct keyring component for that instance.
+3. `InnoDB` initializes, replays the redo log, and opens tablespaces.
+
+4. The SQL layer accepts connections.
+
+The keyring must be ready between steps 1 and 3. Both alternative mechanisms miss that window:
+
+* `INSTALL COMPONENT` runs as SQL, so the statement cannot execute until step 4. The registration lives in `mysql.component`, an `InnoDB` table the server reads only after `InnoDB` initializes — a circular dependency when the system tablespace is encrypted. Crash recovery also runs before SQL, so an encrypted redo log must be readable without any SQL layer.
+
+* `--early-plugin-load` applies to legacy keyring plugins, not components. Plugins and components load through separate subsystems; the flag cannot locate component entry points. The manifest is the only early-load channel for components.
+
+One practical consequence: a component registered through `INSTALL COMPONENT` on a running server disappears on the next restart, so `InnoDB` fails to unwrap tablespace keys without a manifest file on disk. A missing or malformed `mysqld.my` is therefore a startup failure for any instance with encrypted tablespaces.
+
+Place a global manifest named `mysqld.my` in the server installation directory. For per-instance overrides, add a local manifest — also named `mysqld.my` — in the data directory.
+
+To install a keyring component:
+
+1. Write the manifest file in valid JSON.
+
+2. Write the component's configuration file in valid JSON.
+
+The manifest names the component to load. Without a matching manifest file, the server quietly skips the component. On startup, the server reads the global manifest from the installation directory; the global manifest either holds the component entries directly or delegates to a local manifest in the data directory. When instances on the same host require different keyring components, place a local manifest in each data directory so every instance loads the correct component.
 
 !!! warning
 
-    Enable only one keyring plugin or one keyring component at a time for each server instance. Enabling multiple keyring plugins or keyring components or mixing keyring plugins or keyring components is not supported and may result in data loss.
+    Run exactly one keyring per server instance. Percona Server does not support multiple keyring plugins, multiple keyring components, or any mix of plugin and component — such configurations risk data loss.