K8SPG-911: Add pg_tde support

This commit adds native pg_tde extension support into operator.

**This commit only adds Vault KMS support for pg_tde. KMIP support will
be added in future releases.**

When pg_tde is enabled and Vault configuration is provided, the operator:
- appends pg_tde into shared_preload_libraries,
- mounts Vault token and CA secrets into database containers,
- runs CREATE EXTENSION in all databases,
- creates Vault provider by running pg_tde_add_global_key_provider_vault_v2,
- create a global key by running pg_tde_create_key_using_global_key_provider,
- sets the default key by running pg_tde_set_default_key_using_global_key_provider.

-> Example configuration

pg_tde:
  enabled: true
  vault:
    host: https://vault-service.vault-service.svc:8200
    mountPath: tde
    tokenSecret:
      name: vault-secret
      key: token
    caSecret:
      name: vault-secret
      key: ca.crt

Note that:
- Mount path needs to be a KV v2 storage engine.
- caSecret is optional and can be omitted if you want to use http. But
  in my testing I couldn't manage the make vault work without TLS. It
  responds with HTTP 405 if I disable TLS in vault.
- tokenSecret and caSecret can be the same secret or different. Operator
  doesn't assume anything about the contents of the secrets since you'll
  need to set secret keys in cr.yaml yourself.
- Using a non-root token requires more configuration. Check out pg_tde
  docs for that. But don't forget to add these in the Vault policy:

```
path "sys/internal/ui/mounts/*" {
      capabilities = ["read"]
}

path "sys/mounts/*" {
      capabilities = ["read"]
}
```

-> API changes

pg_tde requires more configuration options than other extensions
operator supports. This required us make some changes in the extensions
API. With these changes, 'spec.extensions.builtin' section is deprecated
and all builtin extensions are moved to 'spec.extensions.<extension>'
(i.e. 'spec.extensions.pg_stat_monitor'). Right now extensions can be
enabled/disabled with the old and the new method. If two methods are
used at the same time, 'spec.extensions.builtin' takes precedence.

-> Status changes

A hash will be calculated using pg_tde configuration provided by user.
Operator uses this hash to understand if config is changed and it should
reconfigure pg_tde. The hash can be found in status.pgTDERevision field
of **PostgresCluster** object. This hash will be removed when pg_tde is
disabled.

Operator also communicates the status of pg_tde with conditions. The
condition with type=PGTDEEnabled can be found in both PerconaPGCluster
and PostgresCluster statuses.

-> Disabling pg_tde

Disabling pg_tde is more complex than other extensions:
- First of all any encrypted objects must be dropped before disabling.
  Otherwise DROP EXTENSION will fail with a descriptive error message.
  **Operator won't drop anything, user needs to do this manually.**
- The extension needs to be disabled in two steps:
  1. First set pg_tde.enabled=false without removing the vault section.
     Operator will drop the extension and restart the pods.
  2. Then you can remove pg_tde.vault. Database pods will be restarted
     again to remove secret mounts from containers.
- It's recommended to run CHECKPOINT before removing pg_tde.vault. Even
  though extension is dropped, Postgres might still try to use encrypted
  objects during recovery after restart and it might try to access token
  secret. CHECKPOINT helps you prevent this failure case.

-> Deleting and recreating clusters

If cluster with pg_tde enabled is deleted but PVCs are retained, on
recreation you'll see some errors about pg_tde in operator logs. They
happen because the vault provider and/or global key already exists.
Operator will handle these errors gracefully and configure pg_tde. Same
thing applies when pg_tde is disabled and re-enabled. Since both vault
provider and global key already exists, operator will handle "already
exists" errors and configure pg_tde.

The global key name is determined by cluster's .metadata.uid. For
example 'global-master-key-ad19534a-d778-460e-ac87-ca38ef5e6755'. This
means the key will be changed if cluster is deleted and recreated. As
long as the old key and the new key is accessible to pg_tde, this won't
cause any issues. pg_tde will handle it as it handles key rotation.

-> Validations

- You can't set pg_tde.enabled=true without setting pg_tde.vault.
- If you already had pg_tde.enabled, you can't remove pg_tde section
  completely.
- If you already had pg_tde.enabled, you can't remove pg_tde.vault
  section completely.

---------

K8SPG-911: pg_tde improvements/fixes

- add pg version validation
- explicitly disable wal encryption
- enable pg_tde in restore job
- [e2e] read from all pods after restore
- use pg_tde binaries in patroni
- fix vault provider change

All items except the last is straightforward. Fixing the vault provider
change, required a lot of changes.

The problem with changing the Vault token in pg_tde was that pg_tde
requires both the new and the old token at the same time to perform the
change. This is not trivial to achieve on K8s, since operator needs to
mount the new secret to the pods and somehow needs the keep the old
secret mounted.

To achieve this, operator performs provider change in two phases:

1. In the first phase, operator keeps the old secret mounted in the pod
   and prevents restart. Then it fetches the new secret contents and
   stores them in temporary files in `/pgdata` directory. Then, operator
   runs pg_tde_change_global_key_provider_vault_v2.
2. In the second phase, operator mounts the new secret and restarts the
   pods. Then it runs pg_tde_change_global_key_provider_vault_v2 with
   standard credential paths. At the end of this phase, temporary files
   are cleaned up.

Author: egegunes

build/crd/crunchy/generated/postgres-operator.crunchydata.com_postgresclusters.yaml

@@ -13594,6 +13594,53 @@ spec:
                 type: boolean
               extensions:
                 properties:
+                  pg_tde:
+                    properties:
+                      enabled:
+                        type: boolean
+                      vault:
+                        properties:
+                          caSecret:
+                            description: Name of the secret that contains the CA certificate
+                              for SSL verification.
+                            properties:
+                              key:
+                                type: string
+                              name:
+                                type: string
+                            required:
+                            - key
+                            - name
+                            type: object
+                          host:
+                            description: Host of Vault server.
+                            type: string
+                          mountPath:
+                            default: secret/data
+                            description: The mount point on the Vault server where
+                              the key provider should store the keys.
+                            type: string
+                          tokenSecret:
+                            description: Name of the secret that contains the access
+                              token with read and write access to the mount path.
+                            properties:
+                              key:
+                                type: string
+                              name:
+                                type: string
+                            required:
+                            - key
+                            - name
+                            type: object
+                        required:
+                        - host
+                        - tokenSecret
+                        type: object
+                    type: object
+                    x-kubernetes-validations:
+                    - message: vault is required for enabling pg_tde
+                      rule: '!has(self.enabled) || (has(self.enabled) && self.enabled
+                        == false) || has(self.vault)'
                   pgAudit:
                     type: boolean
                   pgRepack:
@@ -13605,6 +13652,11 @@ spec:
                   pgvector:
                     type: boolean
                 type: object
+                x-kubernetes-validations:
+                - message: to disable pg_tde first set enabled=false without removing
+                    vault and wait for pod restarts
+                  rule: '!has(oldSelf.pg_tde) || !has(oldSelf.pg_tde.vault) || !has(oldSelf.pg_tde.enabled)
+                    || !oldSelf.pg_tde.enabled || has(self.pg_tde.vault)'
               image:
                 description: |-
                   The image name to use for PostgreSQL containers. When omitted, the value
@@ -30971,6 +31023,10 @@ spec:
                     description: The PostgreSQL system identifier reported by Patroni.
                     type: string
                 type: object
+              pgTDERevision:
+                description: Identifies the pg_tde configuration that have been installed
+                  into PostgreSQL.
+                type: string
               pgbackrest:
                 description: Status information for pgBackRest
                 properties:

build/crd/percona/generated/pgv2.percona.com_perconapgclusters.yaml

@@ -13693,6 +13693,8 @@ spec:
                 description: The specification of extensions.
                 properties:
                   builtin:
+                    description: 'Deprecated: Use extensions.<extension> instead.
+                      This field will be removed after 2.11.0.'
                     properties:
                       pg_audit:
                         type: boolean
@@ -13722,6 +13724,78 @@ spec:
                     description: PullPolicy describes a policy for if/when to pull
                       a container image
                     type: string
+                  pg_audit:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
+                  pg_repack:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
+                  pg_stat_monitor:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
+                  pg_stat_statements:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
+                  pg_tde:
+                    properties:
+                      enabled:
+                        type: boolean
+                      vault:
+                        properties:
+                          caSecret:
+                            description: Name of the secret that contains the CA certificate
+                              for SSL verification.
+                            properties:
+                              key:
+                                type: string
+                              name:
+                                type: string
+                            required:
+                            - key
+                            - name
+                            type: object
+                          host:
+                            description: Host of Vault server.
+                            type: string
+                          mountPath:
+                            default: secret/data
+                            description: The mount point on the Vault server where
+                              the key provider should store the keys.
+                            type: string
+                          tokenSecret:
+                            description: Name of the secret that contains the access
+                              token with read and write access to the mount path.
+                            properties:
+                              key:
+                                type: string
+                              name:
+                                type: string
+                            required:
+                            - key
+                            - name
+                            type: object
+                        required:
+                        - host
+                        - tokenSecret
+                        type: object
+                    type: object
+                    x-kubernetes-validations:
+                    - message: vault is required for enabling pg_tde
+                      rule: '!has(self.enabled) || (has(self.enabled) && self.enabled
+                        == false) || has(self.vault)'
+                  pgvector:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
                   storage:
                     properties:
                       bucket:
@@ -13804,6 +13878,11 @@ spec:
                         type: string
                     type: object
                 type: object
+                x-kubernetes-validations:
+                - message: to disable pg_tde first set enabled=false without removing
+                    vault and wait for pod restarts
+                  rule: '!has(oldSelf.pg_tde) || !has(oldSelf.pg_tde.vault) || !has(oldSelf.pg_tde.enabled)
+                    || !oldSelf.pg_tde.enabled || has(self.pg_tde.vault)'
               image:
                 description: The image name to use for PostgreSQL containers.
                 type: string
@@ -28778,6 +28857,9 @@ spec:
             - postgresVersion
             type: object
             x-kubernetes-validations:
+            - message: pg_tde is only supported for PG17 and above
+              rule: '!has(self.extensions) || !has(self.extensions.pg_tde) || !has(self.extensions.pg_tde.enabled)
+                || !self.extensions.pg_tde.enabled || self.postgresVersion >= 17'
             - message: PostgresVersion must be >= 15 if grantPublicSchemaAccess exists
                 and is true
               rule: '!has(self.users) || self.postgresVersion >= 15 || self.users.all(u,

config/crd/bases/pgv2.percona.com_perconapgclusters.yaml

@@ -14332,6 +14332,8 @@ spec:
                 description: The specification of extensions.
                 properties:
                   builtin:
+                    description: 'Deprecated: Use extensions.<extension> instead.
+                      This field will be removed after 2.11.0.'
                     properties:
                       pg_audit:
                         type: boolean
@@ -14361,6 +14363,78 @@ spec:
                     description: PullPolicy describes a policy for if/when to pull
                       a container image
                     type: string
+                  pg_audit:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
+                  pg_repack:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
+                  pg_stat_monitor:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
+                  pg_stat_statements:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
+                  pg_tde:
+                    properties:
+                      enabled:
+                        type: boolean
+                      vault:
+                        properties:
+                          caSecret:
+                            description: Name of the secret that contains the CA certificate
+                              for SSL verification.
+                            properties:
+                              key:
+                                type: string
+                              name:
+                                type: string
+                            required:
+                            - key
+                            - name
+                            type: object
+                          host:
+                            description: Host of Vault server.
+                            type: string
+                          mountPath:
+                            default: secret/data
+                            description: The mount point on the Vault server where
+                              the key provider should store the keys.
+                            type: string
+                          tokenSecret:
+                            description: Name of the secret that contains the access
+                              token with read and write access to the mount path.
+                            properties:
+                              key:
+                                type: string
+                              name:
+                                type: string
+                            required:
+                            - key
+                            - name
+                            type: object
+                        required:
+                        - host
+                        - tokenSecret
+                        type: object
+                    type: object
+                    x-kubernetes-validations:
+                    - message: vault is required for enabling pg_tde
+                      rule: '!has(self.enabled) || (has(self.enabled) && self.enabled
+                        == false) || has(self.vault)'
+                  pgvector:
+                    properties:
+                      enabled:
+                        type: boolean
+                    type: object
                   storage:
                     properties:
                       bucket:
@@ -14443,6 +14517,11 @@ spec:
                         type: string
                     type: object
                 type: object
+                x-kubernetes-validations:
+                - message: to disable pg_tde first set enabled=false without removing
+                    vault and wait for pod restarts
+                  rule: '!has(oldSelf.pg_tde) || !has(oldSelf.pg_tde.vault) || !has(oldSelf.pg_tde.enabled)
+                    || !oldSelf.pg_tde.enabled || has(self.pg_tde.vault)'
               image:
                 description: The image name to use for PostgreSQL containers.
                 type: string
@@ -29417,6 +29496,9 @@ spec:
             - postgresVersion
             type: object
             x-kubernetes-validations:
+            - message: pg_tde is only supported for PG17 and above
+              rule: '!has(self.extensions) || !has(self.extensions.pg_tde) || !has(self.extensions.pg_tde.enabled)
+                || !self.extensions.pg_tde.enabled || self.postgresVersion >= 17'
             - message: PostgresVersion must be >= 15 if grantPublicSchemaAccess exists
                 and is true
               rule: '!has(self.users) || self.postgresVersion >= 15 || self.users.all(u,

config/crd/bases/postgres-operator.crunchydata.com_postgresclusters.yaml

@@ -13554,6 +13554,53 @@ spec:
                 type: boolean
               extensions:
                 properties:
+                  pg_tde:
+                    properties:
+                      enabled:
+                        type: boolean
+                      vault:
+                        properties:
+                          caSecret:
+                            description: Name of the secret that contains the CA certificate
+                              for SSL verification.
+                            properties:
+                              key:
+                                type: string
+                              name:
+                                type: string
+                            required:
+                            - key
+                            - name
+                            type: object
+                          host:
+                            description: Host of Vault server.
+                            type: string
+                          mountPath:
+                            default: secret/data
+                            description: The mount point on the Vault server where
+                              the key provider should store the keys.
+                            type: string
+                          tokenSecret:
+                            description: Name of the secret that contains the access
+                              token with read and write access to the mount path.
+                            properties:
+                              key:
+                                type: string
+                              name:
+                                type: string
+                            required:
+                            - key
+                            - name
+                            type: object
+                        required:
+                        - host
+                        - tokenSecret
+                        type: object
+                    type: object
+                    x-kubernetes-validations:
+                    - message: vault is required for enabling pg_tde
+                      rule: '!has(self.enabled) || (has(self.enabled) && self.enabled
+                        == false) || has(self.vault)'
                   pgAudit:
                     type: boolean
                   pgRepack:
@@ -13565,6 +13612,11 @@ spec:
                   pgvector:
                     type: boolean
                 type: object
+                x-kubernetes-validations:
+                - message: to disable pg_tde first set enabled=false without removing
+                    vault and wait for pod restarts
+                  rule: '!has(oldSelf.pg_tde) || !has(oldSelf.pg_tde.vault) || !has(oldSelf.pg_tde.enabled)
+                    || !oldSelf.pg_tde.enabled || has(self.pg_tde.vault)'
               image:
                 description: |-
                   The image name to use for PostgreSQL containers. When omitted, the value
@@ -30869,6 +30921,10 @@ spec:
                     description: The PostgreSQL system identifier reported by Patroni.
                     type: string
                 type: object
+              pgTDERevision:
+                description: Identifies the pg_tde configuration that have been installed
+                  into PostgreSQL.
+                type: string
               pgbackrest:
                 description: Status information for pgBackRest
                 properties: