From 6e5a950e572b24a5c9e1dac303599dfd30fce5a9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 03:57:19 +1200 Subject: [PATCH 01/11] feat(crd): canopy_source alt to kopia_secret_ref + canopy_desired_snapshot_id status MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 1 of docs/plans/canopy-cr-materialisation.md. Purely additive: existing kopiaSecretRef replicas keep working. - Spec: kopiaSecretRef becomes Option; new canopySource { group, type } alternative. Reconciler validates exactly one is set — surfaces SecretRefAndCanopySource / NoCredentialSource / CanopySourced via the KopiaSecretValid condition. - Status: new canopyDesiredSnapshotId field (populated by the canopy worklist syncer in a follow-up commit; consumed by the reconciler in step 4 to drive Restore CR creation without a snapshot-list Job). - Job builders (build_snapshot_list_job, build_restore_job) still unwrap kopia_secret_ref via .expect() — the credential-source validation guarantees it's set before these are called. Step 3 wires the canopy proxy-sidecar branch through a KopiaSource enum. - crds.yaml regenerated; README spec+status tables updated. --- README.md | 4 +- crds.yaml | 1104 +++++++++++++++++++ docs/plans/canopy-cr-materialisation.md | 211 ++++ src/controllers/replica.rs | 108 +- src/controllers/replica/resources.rs | 11 +- src/controllers/replica/scheduling.rs | 5 +- src/controllers/replica/schema_migration.rs | 3 +- src/controllers/replica/tests.rs | 10 +- src/controllers/restore/builders.rs | 10 +- src/controllers/restore/tests.rs | 10 +- src/types/replica.rs | 47 +- tests/helpers.rs | 5 +- 12 files changed, 1479 insertions(+), 49 deletions(-) create mode 100644 crds.yaml create mode 100644 docs/plans/canopy-cr-materialisation.md diff --git a/README.md b/README.md index 3d6c6be..4626e95 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,8 @@ Defines a continuously-refreshed replica of a PostgreSQL database restored from | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| -| `kopiaSecretRef` | `SecretReference` | Yes | — | Reference to a Secret containing Kopia repository credentials (`bucket`, `region`, `repositoryPassword`, `accessKeyId`, `secretAccessKey`). | +| `kopiaSecretRef` | `SecretReference` | One of | — | Reference to a Secret containing Kopia repository credentials (`bucket`, `region`, `repositoryPassword`, `accessKeyId`, `secretAccessKey`). Mutually exclusive with `canopySource`. | +| `canopySource` | `CanopySource` | One of | — | Route kopia through the canopy-mediated proxy sidecar instead of a static Secret. `{ group, type }`. Managed by the canopy worklist syncer; humans usually don't hand-author these. Mutually exclusive with `kopiaSecretRef`. | | `snapshotFilter` | `SnapshotFilter` | No | — | Filter criteria to select which Kopia snapshot to restore. | | `schedule` | `string` | Yes | — | Cron expression controlling how often new restores are triggered. | | `scheduleJitter` | `string` | No | `"10m"` | Random jitter added to scheduled restores (friendly duration, e.g. `"5m"`, `"1h"`). | @@ -175,6 +176,7 @@ Additional fields for `target: graphQL`: | `lastRestoreCompletedAt` | `Time` | When the last restore completed. | | `nextScheduledRestore` | `Time` | When the next scheduled restore will occur. | | `latestAvailableSnapshot` | `string` | Snapshot ID of the latest available snapshot matching the filter. | +| `canopyDesiredSnapshotId` | `string` | For canopy-sourced replicas: the snapshot the canopy worklist syncer wants restored. The reconciler triggers a new restore when this differs from the current one. | | `connectionInfo` | `ConnectionInfo` | Connection details (host, port, database, username, password secret). | | `queuePosition` | `uint32` | Position in the global restore queue. | | `notifications` | `[]NotificationStatus` | Status of each configured notification target. | diff --git a/crds.yaml b/crds.yaml new file mode 100644 index 0000000..1527aa6 --- /dev/null +++ b/crds.yaml @@ -0,0 +1,1104 @@ +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: postgresphysicalreplicas.pgro.bes.au +spec: + group: pgro.bes.au + names: + categories: + - all + kind: PostgresPhysicalReplica + plural: postgresphysicalreplicas + shortNames: + - ppr + singular: postgresphysicalreplica + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .status.phase + name: Phase + type: string + - jsonPath: .status.serviceName + name: Service + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + - jsonPath: .status.nextScheduledRestore + name: Next restore + type: date + name: v1alpha1 + schema: + openAPIV3Schema: + description: Auto-generated derived type for PostgresPhysicalReplicaSpec via `CustomResource` + properties: + spec: + properties: + affinity: + description: Affinity is a group of affinity scheduling rules. + nullable: true + properties: + nodeAffinity: + description: Describes node affinity scheduling rules for the pod. + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node matches the corresponding matchExpressions; the node(s) with the highest sum are the most preferred. + items: + description: An empty preferred scheduling term matches all objects with implicit weight 0 (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op). + properties: + preference: + description: A node selector term, associated with the corresponding weight. + properties: + matchExpressions: + description: A list of node selector requirements by node's labels. + items: + description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: The label key that the selector applies to. + type: string + operator: + description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements by node's fields. + items: + description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: The label key that the selector applies to. + type: string + operator: + description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + weight: + description: Weight associated with matching the corresponding nodeSelectorTerm, in the range 1-100. + format: int32 + type: integer + required: + - preference + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to an update), the system may or may not try to eventually evict the pod from its node. + properties: + nodeSelectorTerms: + description: Required. A list of node selector terms. The terms are ORed. + items: + description: A null or empty node selector term matches no objects. The requirements of them are ANDed. The TopologySelectorTerm type implements a subset of the NodeSelectorTerm. + properties: + matchExpressions: + description: A list of node selector requirements by node's labels. + items: + description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: The label key that the selector applies to. + type: string + operator: + description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchFields: + description: A list of node selector requirements by node's fields. + items: + description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: The label key that the selector applies to. + type: string + operator: + description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + type: object + type: array + required: + - nodeSelectorTerms + type: object + type: object + podAffinity: + description: Describes pod affinity scheduling rules (e.g. co-locate this pod in the same node, zone, etc. as some other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + matchLabelKeys: + description: MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + mismatchLabelKeys: + description: MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + namespaceSelector: + description: A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + namespaces: + description: namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied. + items: + description: Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key matches that of any node on which a pod of the set of pods is running + properties: + labelSelector: + description: A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + matchLabelKeys: + description: MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + mismatchLabelKeys: + description: MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + namespaceSelector: + description: A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + namespaces: + description: namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + podAntiAffinity: + description: Describes pod anti-affinity scheduling rules (e.g. avoid putting this pod in the same node, zone, etc. as some other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: The scheduler will prefer to schedule pods to nodes that satisfy the anti-affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling anti-affinity expressions, etc.), compute a sum by iterating through the elements of this field and subtracting "weight" from the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated with the corresponding weight. + properties: + labelSelector: + description: A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + matchLabelKeys: + description: MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + mismatchLabelKeys: + description: MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + namespaceSelector: + description: A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + namespaces: + description: namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: weight associated with matching the corresponding podAffinityTerm, in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + requiredDuringSchedulingIgnoredDuringExecution: + description: If the anti-affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the anti-affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied. + items: + description: Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key matches that of any node on which a pod of the set of pods is running + properties: + labelSelector: + description: A label query over a set of resources, in this case pods. If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + matchLabelKeys: + description: MatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both matchLabelKeys and labelSelector. Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + mismatchLabelKeys: + description: MismatchLabelKeys is a set of pod label keys to select which pods will be taken into consideration. The keys are used to lookup values from the incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` to select the group of existing pods which pods will be taken into consideration for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming pod labels will be ignored. The default value is empty. The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + namespaceSelector: + description: A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + namespaces: + description: namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + topologyKey: + description: This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + type: object + type: object + analyticsUsername: + default: analytics + description: Username for analytics connections + type: string + canopySource: + description: |- + Route kopia through the canopy-mediated proxy sidecar instead of + a static credentials Secret. Mutually exclusive with + `kopiaSecretRef`. + + When set: + - the restore + snapshot-list Jobs get the pgro-canopy-proxy + sidecar and dummy AWS keys (kopia talks to `[::1]:`); + - the reconciler skips the snapshot-list step and instead reads + the snapshot to restore from `status.canopyDesiredSnapshotId` + (populated by the canopy worklist syncer); + - the replica is treated as pgro-internal state — the canopy + syncer materialises + tears down these CRs based on canopy's + worklist. Manual edits to a canopy-managed CR are re-asserted + on the next tick. + nullable: true + properties: + group: + description: Canopy server-group id (UUID) whose backups this replica restores. + type: string + type: + description: |- + Canopy backup type (e.g. `tamanu-postgres`). The + `(consumer, group, type)` external-restore grant on canopy's side + gates access. + type: string + required: + - group + - type + type: object + kopiaSecretRef: + description: |- + Reference to a Secret containing kopia repository credentials. + Mutually exclusive with `canopySource`. Exactly one of the two + must be set; the reconciler surfaces `KopiaSecretValid=False` + with reason `SecretRefAndCanopySource` if both (or neither) are + present. + nullable: true + properties: + name: + description: name is unique within a namespace to reference a secret resource. + type: string + namespace: + description: namespace defines the space within which the secret name must be unique. + type: string + type: object + minimumTtl: + description: Don't restore a new snapshot within this duration of the last restore completing + format: duration + nullable: true + type: string + notifications: + items: + properties: + headers: + additionalProperties: + anyOf: + - {} + - required: + - secretKeyRef + properties: + secretKeyRef: + properties: + key: + type: string + name: + type: string + required: + - key + - name + type: object + x-kubernetes-preserve-unknown-fields: true + nullable: true + type: object + includePassword: + type: boolean + method: + type: string + mutation: + type: string + target: + enum: + - webhook + - graphQL + type: string + url: + type: string + variablesTemplate: + type: string + required: + - target + - url + type: object + type: array + persistentSchemas: + description: List of schema names to migrate from the previous restore to the new restore on each switchover. + items: + type: string + nullable: true + type: array + podAnnotations: + additionalProperties: + type: string + nullable: true + type: object + postgresExtraConfig: + description: Extra lines appended to postgresql.conf (e.g. shared_preload_libraries) + nullable: true + type: string + readOnly: + default: true + description: Set database to read-only mode + type: boolean + resources: + description: ResourceRequirements describes the compute resource requirements. + nullable: true + properties: + claims: + description: |- + Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container. + + This field depends on the DynamicResourceAllocation feature gate. + + This field is immutable. It can only be set for containers. + items: + description: ResourceClaim references one entry in PodSpec.ResourceClaims. + properties: + name: + description: Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. + type: string + request: + description: Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. + type: string + required: + - name + type: object + type: array + limits: + additionalProperties: + description: "Quantity is a fixed-point representation of a number. It provides convenient marshaling/unmarshaling in JSON and YAML, in addition to String() and AsInt64() accessors.\n\nThe serialization format is:\n\n``` ::= \n\n\t(Note that may be empty, from the \"\" case in .)\n\n ::= 0 | 1 | ... | 9 ::= | ::= | . | . | . ::= \"+\" | \"-\" ::= | ::= | | ::= Ki | Mi | Gi | Ti | Pi | Ei\n\n\t(International System of units; See: http://physics.nist.gov/cuu/Units/binary.html)\n\n ::= m | \"\" | k | M | G | T | P | E\n\n\t(Note that 1024 = 1Ki but 1000 = 1k; I didn't choose the capitalization.)\n\n ::= \"e\" | \"E\" ```\n\nNo matter which of the three exponent forms is used, no quantity may represent a number greater than 2^63-1 in magnitude, nor may it have more than 3 decimal places. Numbers larger or more precise will be capped or rounded up. (E.g.: 0.1m will rounded up to 1m.) This may be extended in the future if we require larger or smaller quantities.\n\nWhen a Quantity is parsed from a string, it will remember the type of suffix it had, and will use the same type again when it is serialized.\n\nBefore serializing, Quantity will be put in \"canonical form\". This means that Exponent/suffix will be adjusted up or down (with a corresponding increase or decrease in Mantissa) such that:\n\n- No precision is lost - No fractional digits will be emitted - The exponent (or suffix) is as large as possible.\n\nThe sign will be omitted unless the number is negative.\n\nExamples:\n\n- 1.5 will be serialized as \"1500m\" - 1.5Gi will be serialized as \"1536Mi\"\n\nNote that the quantity will NEVER be internally represented by a floating point number. That is the whole point of this exercise.\n\nNon-canonical values will still parse as long as they are well formed, but will be re-emitted in their canonical form. (So always use canonical form, or don't diff.)\n\nThis format is intended to make it difficult to use these numbers without writing some sort of special handling code in the hopes that that will cause implementors to also use a fixed point implementation." + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + description: "Quantity is a fixed-point representation of a number. It provides convenient marshaling/unmarshaling in JSON and YAML, in addition to String() and AsInt64() accessors.\n\nThe serialization format is:\n\n``` ::= \n\n\t(Note that may be empty, from the \"\" case in .)\n\n ::= 0 | 1 | ... | 9 ::= | ::= | . | . | . ::= \"+\" | \"-\" ::= | ::= | | ::= Ki | Mi | Gi | Ti | Pi | Ei\n\n\t(International System of units; See: http://physics.nist.gov/cuu/Units/binary.html)\n\n ::= m | \"\" | k | M | G | T | P | E\n\n\t(Note that 1024 = 1Ki but 1000 = 1k; I didn't choose the capitalization.)\n\n ::= \"e\" | \"E\" ```\n\nNo matter which of the three exponent forms is used, no quantity may represent a number greater than 2^63-1 in magnitude, nor may it have more than 3 decimal places. Numbers larger or more precise will be capped or rounded up. (E.g.: 0.1m will rounded up to 1m.) This may be extended in the future if we require larger or smaller quantities.\n\nWhen a Quantity is parsed from a string, it will remember the type of suffix it had, and will use the same type again when it is serialized.\n\nBefore serializing, Quantity will be put in \"canonical form\". This means that Exponent/suffix will be adjusted up or down (with a corresponding increase or decrease in Mantissa) such that:\n\n- No precision is lost - No fractional digits will be emitted - The exponent (or suffix) is as large as possible.\n\nThe sign will be omitted unless the number is negative.\n\nExamples:\n\n- 1.5 will be serialized as \"1500m\" - 1.5Gi will be serialized as \"1536Mi\"\n\nNote that the quantity will NEVER be internally represented by a floating point number. That is the whole point of this exercise.\n\nNon-canonical values will still parse as long as they are well formed, but will be re-emitted in their canonical form. (So always use canonical form, or don't diff.)\n\nThis format is intended to make it difficult to use these numbers without writing some sort of special handling code in the hopes that that will cause implementors to also use a fixed point implementation." + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + schedule: + description: Cron expression for scheduled restores + type: string + scheduleJitter: + default: 10m + description: Random jitter added to scheduled restores + format: duration + type: string + serviceAnnotations: + additionalProperties: + type: string + nullable: true + type: object + snapshotFilter: + nullable: true + properties: + descriptionPattern: + description: Glob pattern for filtering snapshot descriptions + nullable: true + type: string + hostPattern: + description: Glob pattern for filtering snapshot hosts + nullable: true + type: string + pathPattern: + description: |- + Glob pattern for filtering snapshot source paths. + Windows paths are normalised to Unix style (e.g. `D:\Full` → `/D/Full`). + nullable: true + type: string + tags: + additionalProperties: + type: string + nullable: true + type: object + type: object + storageClass: + nullable: true + type: string + storageSizeMaximum: + default: 2Ti + description: |- + Maximum allowed size for the restore PVC. The restore will fail if the + computed size exceeds this limit. Defaults to 2Ti. + nullable: true + x-kubernetes-int-or-string: true + storageSizeOverride: + description: Override dynamic sizing with a fixed PVC size + nullable: true + x-kubernetes-int-or-string: true + switchoverGracePeriod: + default: 5m + description: Wait before deleting old restore after switchover + format: duration + type: string + tolerations: + items: + description: The pod this Toleration is attached to tolerates any taint that matches the triple using the matching operator . + properties: + effect: + description: Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys. + type: string + operator: + description: Operator represents a key's relationship to the value. Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category. Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). + type: string + tolerationSeconds: + description: TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system. + format: int64 + type: integer + value: + description: Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string. + type: string + type: object + type: array + required: + - schedule + type: object + status: + nullable: true + properties: + canopyDesiredSnapshotId: + description: |- + Snapshot id the canopy worklist syncer wants restored. Populated + each syncer tick for canopy-sourced replicas (`spec.canopySource` + is set); unused otherwise. The reconciler triggers a new restore + when this differs from the current one. + nullable: true + type: string + conditions: + items: + description: Condition contains details for one aspect of the current state of this API Resource. + properties: + lastTransitionTime: + description: lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: message is a human readable message indicating details about the transition. This may be an empty string. + type: string + observedGeneration: + description: observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. + format: int64 + type: integer + reason: + description: reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty. + type: string + status: + description: status of the condition, one of True, False, Unknown. + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + connectionInfo: + nullable: true + properties: + database: + type: string + host: + type: string + passwordSecret: + type: string + port: + format: uint16 + maximum: 65535.0 + minimum: 0.0 + type: integer + username: + type: string + required: + - database + - host + - passwordSecret + - port + - username + type: object + consecutiveRestoreFailures: + description: |- + Number of consecutive restore failures for this replica. + Reset to 0 on a successful restore. After 3 consecutive failures + the operator stops scheduling new restores until the condition is + cleared (e.g. by a spec change or manual intervention). + format: uint32 + minimum: 0.0 + nullable: true + type: integer + currentRestore: + nullable: true + type: string + lastRestoreCompletedAt: + description: Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers. + format: date-time + nullable: true + type: string + latestAvailableSnapshot: + nullable: true + type: string + nextScheduledRestore: + description: Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers. + format: date-time + nullable: true + type: string + notifications: + items: + properties: + lastError: + nullable: true + type: string + lastSentAt: + description: Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers. + format: date-time + nullable: true + type: string + name: + type: string + success: + default: false + type: boolean + required: + - name + type: object + type: array + persistentSchemaDataSize: + description: |- + Measured size of persistent schema data from the last successful migration (bytes). + Used to size the next restore PVC. + nullable: true + x-kubernetes-int-or-string: true + phase: + enum: + - Pending + - Restoring + - Ready + - Failed + - null + nullable: true + type: string + previousRestore: + nullable: true + type: string + queuePosition: + format: uint32 + minimum: 0.0 + nullable: true + type: integer + scheduleInputHash: + description: |- + Hash of the schedule inputs used to compute `nextScheduledRestore`, + so we only recompute when the inputs actually change. + nullable: true + type: string + schemaMigrationJob: + description: Name of the Job performing schema migration (persistent_schemas only) + nullable: true + type: string + schemaMigrationPhase: + description: Phase of schema migration. See [`SchemaMigrationPhase`]. + nullable: true + type: string + serviceName: + nullable: true + type: string + type: object + required: + - spec + title: PostgresPhysicalReplica + type: object + served: true + storage: true + subresources: + status: {} +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: postgresphysicalrestores.pgro.bes.au +spec: + group: pgro.bes.au + names: + categories: + - all + kind: PostgresPhysicalRestore + plural: postgresphysicalrestores + shortNames: + - pprestore + singular: postgresphysicalrestore + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .status.phase + name: Phase + type: string + - jsonPath: .spec.replica.name + name: Replica + type: string + - jsonPath: .spec.snapshotSize + name: Snapshot size + type: string + - jsonPath: .spec.storageSize + name: Storage size + type: string + - jsonPath: .status.postgresVersion + name: Postgres version + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha1 + schema: + openAPIV3Schema: + description: Auto-generated derived type for PostgresPhysicalRestoreSpec via `CustomResource` + properties: + spec: + properties: + replica: + description: Reference to parent PostgresPhysicalReplica + properties: + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + required: + - name + type: object + snapshot: + description: Kopia snapshot ID to restore + type: string + snapshotSize: + description: Size of the snapshot from kopia metadata + x-kubernetes-int-or-string: true + snapshotTime: + description: Kopia snapshot start time (ISO 8601) + nullable: true + type: string + storageSize: + description: Calculated PVC size (snapshot_size * 1.1) + x-kubernetes-int-or-string: true + required: + - replica + - snapshot + - snapshotSize + - storageSize + type: object + status: + nullable: true + properties: + activatedAt: + description: When service switched to this restore + format: date-time + nullable: true + type: string + conditions: + items: + description: Condition contains details for one aspect of the current state of this API Resource. + properties: + lastTransitionTime: + description: lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: message is a human readable message indicating details about the transition. This may be an empty string. + type: string + observedGeneration: + description: observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. + format: int64 + type: integer + reason: + description: reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty. + type: string + status: + description: status of the condition, one of True, False, Unknown. + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + createdAt: + description: Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers. + format: date-time + nullable: true + type: string + credentialsSecret: + description: Shared credentials secret (owned by parent replica) + nullable: true + type: string + deployment: + nullable: true + type: string + phase: + enum: + - Pending + - Restoring + - Ready + - Switching + - Active + - Failed + - null + nullable: true + type: string + postgresVersion: + nullable: true + type: string + pvc: + nullable: true + type: string + restoreJob: + nullable: true + properties: + completedAt: + description: Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers. + format: date-time + nullable: true + type: string + name: + type: string + phase: + type: string + required: + - name + - phase + type: object + restoredAt: + description: When restore job completed + format: date-time + nullable: true + type: string + type: object + required: + - spec + title: PostgresPhysicalRestore + type: object + served: true + storage: true + subresources: + status: {} diff --git a/docs/plans/canopy-cr-materialisation.md b/docs/plans/canopy-cr-materialisation.md new file mode 100644 index 0000000..37d7e5b --- /dev/null +++ b/docs/plans/canopy-cr-materialisation.md @@ -0,0 +1,211 @@ +# Plan: canopy path via CR materialisation + +## Problem + +PR #73 shipped a parallel canopy path that duplicates most of what the +CRD path already does — a separate syncer, a separate reporter, a +separate postgres Deployment builder, a separate Service builder, a +separate restore-Job builder, a separate state machine (Namespace +labels + annotations). Result: `persistentSchemas`, `switchoverGracePeriod`, +blue/green refresh, credential-reset, analytics-user provisioning, +schema migration — all of which the CRD path implements — need to be +re-implemented on the canopy path from scratch. That's what caused +the "in-place PVC overwrite" (unacceptable), the disabled +`persistent_schemas`, and the current warn-and-hope situation. + +The CR path already has blue/green, phase machine, switchover with +grace, schema migration, analytics-user, resources, tolerations, +affinity, `serviceAnnotations`, `minimumTtl`, deletion cascade via +ownerRefs. Everything the canopy path wants. + +## Approach + +Make the canopy syncer materialise `PostgresPhysicalReplica` CRs from +the worklist. The CR is pgro-internal state — canopy remains the +operator interface. Two things about the pipeline change; everything +else is untouched: + +1. **Credential source** in the Job builders: proxy-sidecar variant + (dummy keys + `[::1]:` endpoint) when a new `canopy_source` + field is set on the CR, instead of env-from-Secret. +2. **Snapshot selection**: skip snapshot-list when `canopy_source` is + set; take snapshot id from a new `Status.canopy_desired_snapshot_id` + field written by the syncer each tick. + +Everything else — reconciler, Restore CR machinery, switchover, schema +migration, notifications — works unchanged. + +## Concrete changes + +### 1. Extend `PostgresPhysicalReplicaSpec` + +- Add `canopy_source: Option` where + `CanopySource { group: Uuid, type: String }`. +- Make `kopia_secret_ref` optional; exactly one of the two must be set. +- Validation: at admission time (webhook or reconcile-time check — + reconcile-time is simpler, matches existing `Error::InvalidKopiaSecret` + pattern), reject a spec that sets both or neither. +- Add `Status.canopy_desired_snapshot_id: Option` — the + syncer writes it, the reconciler consumes it. +- CRD regeneration + README table update per AGENTS.md. + +### 2. Job builders: proxy-sidecar branch + +- Introduce a `KopiaSource` enum (or similar) consumed by + `build_restore_job` and `build_snapshot_list_job` (though + snapshot-list gets skipped for canopy, keep the option for + symmetry). +- `KopiaSource::Secret { kopia_secret_ref, creds }` — current + behaviour, env-from-Secret. +- `KopiaSource::CanopyProxy { broker_url, region, group, type, + repo_password_source, server_id }` — dummy keys, `[::1]:` + endpoint via a pgro-canopy-proxy sidecar container, port-file + handshake, kopia connects via loopback. +- Pull the shell wrapper + sidecar container spec from + `src/controllers/canopy/builders.rs` into the CRD-path Job builder, + then delete the canopy-path builder. + +### 3. Replica reconciler: skip snapshot-list when canopy-sourced + +- If `spec.canopy_source.is_some()`: skip the snapshot-list Job + entirely. Take the snapshot from `status.canopy_desired_snapshot_id`. +- When `status.canopy_desired_snapshot_id` changes (or on schedule + fire), create a new `PostgresPhysicalRestore` CR pointing at the new + snapshot — same code path as today, just a different snapshot source. +- Everything else in the reconciler (Restore phase machine, switchover, + grace, schema migration, credential reset, analytics user) stays + untouched. + +### 4. Canopy syncer becomes tiny + +Replace ~1000 lines of parallel machinery with: + +```rust +async fn tick(&self) -> Result<()> { + let entries = self.ctx.canopy.worklist().await?; + let existing = self.list_canopy_managed_replicas().await?; + + for entry in &entries { + self.ensure_replica_cr(entry).await?; + } + for cr in &existing { + if !worklist_covers(cr, &entries) { + self.delete_replica_cr(cr).await?; + } + } + Ok(()) +} +``` + +Where `ensure_replica_cr`: +- Namespace: `pgro-r--<8hex>` (as today). +- Spec: from `IntentConfig(entry.intent)` — resources, service + annotations, persistent_schemas, min_ttl, switchover_grace, + read_only, analytics_username. Plus `canopy_source` = { group, type }. +- Status patch: `canopy_desired_snapshot_id = entry.snapshot_id`. +- Label: `pgro.bes.au/managed-by=pgro-canopy` (discovery key). + +The syncer no longer touches PVCs, Jobs, Deployments, Services, +Secrets — the existing CR reconciler owns all of that. + +### 5. Guardrail: reject user edits to canopy-managed CRs + +Simplest: in the replica reconciler, if the CR has label +`pgro.bes.au/managed-by=pgro-canopy` and the spec differs from what +the intent config would produce, re-apply the intent config. The +canopy syncer's next tick re-asserts anyway. + +Not building an admission webhook yet — the reconcile-time re-assert +is enough to converge, and users can't hurt themselves for long. + +### 6. Restore-verification reporter + +Two options: +- **(a)** New notification target `RestoreCanopyReport` in + `notifications.rs` alongside webhook / graphQL. Reuses the existing + retry / status-tracking pipeline. +- **(b)** Small dedicated controller watching Restore CRs, POSTs on + terminal phase transitions. + +(a) is simpler; use it. The report body is built from Replica + +Restore CR fields (already have everything: replica_id from label, +group/server from spec.canopy_source + status, snapshot_id from +restore.spec, postgres_version from restore.status, observed_at +from restore.status.activatedAt). + +## Retire + +- `src/controllers/canopy/builders.rs` — delete. Job builder branch + moves to CRD-path builder; postgres Deployment / Service / PVC + builders die (CRD path has them). +- `src/controllers/canopy/reporter.rs` — delete. Replaced by the + notification target. +- Most of `src/controllers/canopy.rs` — the diff/action/dispatch + machinery + provision/refresh/teardown functions all die. What + remains: the tick loop + `ensure_replica_cr` / `delete_replica_cr`. +- `Context.canopy_broker_base_url` / `canopy_proxy_image` / + `canopy_pgdata_pvc_size` / `canopy_stats` — most stay, but the + PVC-size default moves to being a CR spec field the syncer sets from + IntentConfig. + +## Keep + +- `src/controllers/canopy/intent.rs` — still drives per-intent CR + spec generation. `IntentConfig` gets a helper method: + `fn to_replica_spec_patch(&self, entry: &WorklistEntry) -> serde_json::Value` + (or similar). +- `src/bin/canopy_proxy.rs` — the sidecar binary. +- Broker route + `Context.canopy_stats` — the sidecar callback. +- Tailscale sidecar in `operator.yaml`. +- `src/canopy.rs` client wrapper. + +## Sequence + +Order that minimises "broken states" between commits: + +1. **Add `canopy_source` + status field to the CRD.** New optional + fields; existing behaviour unchanged. CRD regen + README update. +2. **Extend the Job builders with `KopiaSource` enum.** Both paths + route through it; legacy path uses `KopiaSource::Secret`. No + behaviour change yet. +3. **Job builder gains `CanopyProxy` variant.** Emits the sidecar + spec + dummy keys. Not called yet. +4. **Reconciler: skip snapshot-list when `canopy_source` is set.** + Take snapshot from `Status.canopy_desired_snapshot_id`. Guarded so + legacy path is unaffected. +5. **`IntentConfig::to_replica_spec_patch`** — converts an intent + a + worklist entry into a Replica-spec JSON patch. +6. **Rewrite canopy syncer** to materialise CRs. All old logic + deleted; only ensure_replica_cr / delete_replica_cr remain. +7. **Retire** `canopy/builders.rs` + `canopy/reporter.rs`. +8. **Add canopy notification target** in `notifications.rs`; hook it + at the Restore terminal-transition callsites. +9. **CI + README updates.** + +Each step's `cargo check` + `cargo test --lib` must pass. Fmt + clippy +before each commit. + +## Verification + +- Unit tests survive at every commit (176+). +- Legacy `kopiaSecretRef` integration tests continue to pass — steps + 1-4 are additive to that path. +- Canopy path integration test in a follow-up (still gated on the + stub-canopy server). +- Manual: apply a canopy-labelled Replica CR by hand pointing at a + test worklist; watch the reconciler run through Restoring → + Ready → Switching → Active exactly as with a legacy Replica. + +## What NOT to build in this PR + +- Admission webhook. Reconcile-time re-assert is the guardrail. +- Blue-green refactor of the canopy path — the CR path already does + blue/green. +- Anything about `disaster-recovery` intent — separately gone in #79. + +## Follow-up (out of scope for this plan) + +- Stub-canopy server + wire integration test back into CI. +- Publish a snippet in the ops handoff about how canopy declarations + map onto pgro's internal CRs, and how to poke at them via kubectl + for debugging. diff --git a/src/controllers/replica.rs b/src/controllers/replica.rs index 2512647..9a684be 100644 --- a/src/controllers/replica.rs +++ b/src/controllers/replica.rs @@ -94,63 +94,109 @@ pub async fn reconcile(replica: Arc, ctx: Arc) let client = &ctx.client; - // Validate kopia Secret - let secret_name = replica - .spec - .kopia_secret_ref - .name - .as_deref() - .unwrap_or_default(); - let secrets: Api = Api::namespaced(client.clone(), &namespace); - let secret = match secrets.get(secret_name).await { - Ok(s) => s, - Err(e) => { + // Validate the credential source: exactly one of kopiaSecretRef or + // canopySource must be set. Legacy replicas take the Secret path; + // canopy-managed ones take the proxy-sidecar path and get their + // creds via the operator's in-cluster broker at Job time. + match (&replica.spec.kopia_secret_ref, &replica.spec.canopy_source) { + (Some(_), Some(_)) => { warn!( replica = name, - secret = ?replica.spec.kopia_secret_ref, - error = %e, - "kopia secret not found" + "kopiaSecretRef and canopySource are mutually exclusive" ); replica .update_condition( client, "KopiaSecretValid", "False", - "SecretNotFound", - &format!("Secret {secret_name} not found: {e}"), + "SecretRefAndCanopySource", + "kopiaSecretRef and canopySource are mutually exclusive; set exactly one", ) .await?; return Ok(Action::requeue(Duration::from_secs(60))); } - }; - - let _creds = match kopia::validate_kopia_secret(&secret) { - Ok(c) => { + (None, None) => { + warn!( + replica = name, + "neither kopiaSecretRef nor canopySource set" + ); replica .update_condition( client, "KopiaSecretValid", - "True", - "SecretValid", - "All required keys present", + "False", + "NoCredentialSource", + "one of kopiaSecretRef or canopySource must be set", ) .await?; - c + return Ok(Action::requeue(Duration::from_secs(60))); + } + (Some(secret_ref), None) => { + let secret_name = secret_ref.name.as_deref().unwrap_or_default(); + let secrets: Api = Api::namespaced(client.clone(), &namespace); + let secret = match secrets.get(secret_name).await { + Ok(s) => s, + Err(e) => { + warn!( + replica = name, + secret = ?secret_ref, + error = %e, + "kopia secret not found" + ); + replica + .update_condition( + client, + "KopiaSecretValid", + "False", + "SecretNotFound", + &format!("Secret {secret_name} not found: {e}"), + ) + .await?; + return Ok(Action::requeue(Duration::from_secs(60))); + } + }; + match kopia::validate_kopia_secret(&secret) { + Ok(_) => { + replica + .update_condition( + client, + "KopiaSecretValid", + "True", + "SecretValid", + "All required keys present", + ) + .await?; + } + Err(e) => { + warn!(replica = name, error = %e, "kopia secret invalid"); + replica + .update_condition( + client, + "KopiaSecretValid", + "False", + "SecretInvalid", + &e.to_string(), + ) + .await?; + return Ok(Action::requeue(Duration::from_secs(60))); + } + } } - Err(e) => { - warn!(replica = name, error = %e, "kopia secret invalid"); + (None, Some(_canopy_source)) => { + // Nothing to pre-validate at this stage — the proxy sidecar + // fetches short-lived creds at Job time via the operator's + // broker; any auth failure surfaces there. replica .update_condition( client, "KopiaSecretValid", - "False", - "SecretInvalid", - &e.to_string(), + "True", + "CanopySourced", + "canopy-managed replica; credentials issued by canopy at Job time", ) .await?; - return Ok(Action::requeue(Duration::from_secs(60))); } - }; + } replica.ensure_credentials_secret(client).await?; diff --git a/src/controllers/replica/resources.rs b/src/controllers/replica/resources.rs index a3ca54b..aba73e4 100644 --- a/src/controllers/replica/resources.rs +++ b/src/controllers/replica/resources.rs @@ -55,7 +55,16 @@ pub fn build_snapshot_list_job( kopia_image: &str, callback_url: &str, ) -> Result { - let kopia_secret = &replica.spec.kopia_secret_ref; + // Legacy-path Job builder — `kopia_secret_ref` is guaranteed set by + // the reconciler's credential-source validation before we reach + // here. The canopy path uses a different Job shape (proxy sidecar) + // wired in a follow-up commit; snapshot-list is skipped entirely + // for canopy-sourced replicas. + let kopia_secret = replica + .spec + .kopia_secret_ref + .as_ref() + .expect("build_snapshot_list_job called on legacy path; kopia_secret_ref must be set"); let replica_name = replica.name_any(); let mut env_vars = vec![ diff --git a/src/controllers/replica/scheduling.rs b/src/controllers/replica/scheduling.rs index 9a74fe5..eb6a350 100644 --- a/src/controllers/replica/scheduling.rs +++ b/src/controllers/replica/scheduling.rs @@ -231,10 +231,11 @@ mod tests { ..Default::default() }, spec: PostgresPhysicalReplicaSpec { - kopia_secret_ref: SecretReference { + kopia_secret_ref: Some(SecretReference { name: Some("test-secret".into()), namespace: None, - }, + }), + canopy_source: None, snapshot_filter: None, schedule: schedule.into(), schedule_jitter: TimeSpan(Span::new().seconds(0)), diff --git a/src/controllers/replica/schema_migration.rs b/src/controllers/replica/schema_migration.rs index 21c7f23..e7a99df 100644 --- a/src/controllers/replica/schema_migration.rs +++ b/src/controllers/replica/schema_migration.rs @@ -285,7 +285,8 @@ mod tests { ..Default::default() }, spec: crate::types::PostgresPhysicalReplicaSpec { - kopia_secret_ref: Default::default(), + kopia_secret_ref: Some(Default::default()), + canopy_source: None, snapshot_filter: None, schedule: "0 * * * *".into(), schedule_jitter: crate::util::TimeSpan(jiff::Span::new()), diff --git a/src/controllers/replica/tests.rs b/src/controllers/replica/tests.rs index 2f575d8..83debf7 100644 --- a/src/controllers/replica/tests.rs +++ b/src/controllers/replica/tests.rs @@ -18,10 +18,11 @@ fn make_replica( ..Default::default() }, spec: PostgresPhysicalReplicaSpec { - kopia_secret_ref: SecretReference { + kopia_secret_ref: Some(SecretReference { name: Some("creds".into()), namespace: None, - }, + }), + canopy_source: None, snapshot_filter: None, schedule: "0 * * * *".into(), schedule_jitter: TimeSpan(jiff::Span::new()), @@ -176,10 +177,11 @@ fn snapshot_list_job_rotates_kopia_logs() { ..Default::default() }, spec: PostgresPhysicalReplicaSpec { - kopia_secret_ref: SecretReference { + kopia_secret_ref: Some(SecretReference { name: Some("creds".into()), namespace: None, - }, + }), + canopy_source: None, snapshot_filter: None, schedule: "0 * * * *".into(), schedule_jitter: TimeSpan(jiff::Span::new()), diff --git a/src/controllers/restore/builders.rs b/src/controllers/restore/builders.rs index dad78d5..0e0fddc 100644 --- a/src/controllers/restore/builders.rs +++ b/src/controllers/restore/builders.rs @@ -513,7 +513,15 @@ pub fn build_restore_job( kopia_image: &str, cache_pressure_callback_url: &str, ) -> Result { - let kopia_secret = &replica.spec.kopia_secret_ref; + // Legacy-path restore Job builder. The canopy path uses a different + // Job shape (proxy sidecar) that will be wired in a follow-up commit + // via a `KopiaSource` enum; for now the reconciler's credential-source + // validation guarantees `kopia_secret_ref` is set before we reach here. + let kopia_secret = replica + .spec + .kopia_secret_ref + .as_ref() + .expect("build_restore_job called on legacy path; kopia_secret_ref must be set"); let pvc_name = format!("{}-data", restore.name_any()); let cache_pvc_name = kopia_cache_pvc_name(&restore.spec.replica.name); diff --git a/src/controllers/restore/tests.rs b/src/controllers/restore/tests.rs index 5589e7f..4b0dccb 100644 --- a/src/controllers/restore/tests.rs +++ b/src/controllers/restore/tests.rs @@ -17,10 +17,11 @@ fn deployment_uses_affinity_not_node_selector() { let mut replica = PostgresPhysicalReplica::new( "test-replica", PostgresPhysicalReplicaSpec { - kopia_secret_ref: SecretReference { + kopia_secret_ref: Some(SecretReference { name: Some("kopia-secret".to_string()), namespace: None, - }, + }), + canopy_source: None, snapshot_filter: None, schedule: "0 */6 * * *".into(), schedule_jitter: TimeSpan(Span::new().minutes(10)), @@ -101,10 +102,11 @@ fn test_restore_and_replica() -> (PostgresPhysicalRestore, PostgresPhysicalRepli let replica = PostgresPhysicalReplica::new( "test-replica", PostgresPhysicalReplicaSpec { - kopia_secret_ref: SecretReference { + kopia_secret_ref: Some(SecretReference { name: Some("kopia-secret".to_string()), namespace: None, - }, + }), + canopy_source: None, snapshot_filter: None, schedule: "0 */6 * * *".into(), schedule_jitter: TimeSpan(Span::new().minutes(10)), diff --git a/src/types/replica.rs b/src/types/replica.rs index 1d9c4f1..0d26e52 100644 --- a/src/types/replica.rs +++ b/src/types/replica.rs @@ -35,8 +35,30 @@ use super::HeaderValue; )] #[serde(rename_all = "camelCase")] pub struct PostgresPhysicalReplicaSpec { - /// Reference to a Secret containing kopia repository credentials - pub kopia_secret_ref: SecretReference, + /// Reference to a Secret containing kopia repository credentials. + /// Mutually exclusive with `canopySource`. Exactly one of the two + /// must be set; the reconciler surfaces `KopiaSecretValid=False` + /// with reason `SecretRefAndCanopySource` if both (or neither) are + /// present. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub kopia_secret_ref: Option, + + /// Route kopia through the canopy-mediated proxy sidecar instead of + /// a static credentials Secret. Mutually exclusive with + /// `kopiaSecretRef`. + /// + /// When set: + /// - the restore + snapshot-list Jobs get the pgro-canopy-proxy + /// sidecar and dummy AWS keys (kopia talks to `[::1]:`); + /// - the reconciler skips the snapshot-list step and instead reads + /// the snapshot to restore from `status.canopyDesiredSnapshotId` + /// (populated by the canopy worklist syncer); + /// - the replica is treated as pgro-internal state — the canopy + /// syncer materialises + tears down these CRs based on canopy's + /// worklist. Manual edits to a canopy-managed CR are re-asserted + /// on the next tick. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub canopy_source: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub snapshot_filter: Option, @@ -120,6 +142,20 @@ fn default_analytics_username() -> String { "analytics".to_string() } +/// Points a replica at a canopy-declared restore-replica instead of a +/// static kopia Secret. Set via the canopy worklist syncer; humans +/// shouldn't hand-author these — they're managed for you. +#[derive(Debug, Clone, Deserialize, Serialize, JsonSchema)] +#[serde(rename_all = "camelCase")] +pub struct CanopySource { + /// Canopy server-group id (UUID) whose backups this replica restores. + pub group: String, + /// Canopy backup type (e.g. `tamanu-postgres`). The + /// `(consumer, group, type)` external-restore grant on canopy's side + /// gates access. + pub r#type: String, +} + #[derive(Debug, Clone, Deserialize, Serialize, JsonSchema)] #[serde(rename_all = "camelCase")] pub struct SnapshotFilter { @@ -260,6 +296,13 @@ pub struct PostgresPhysicalReplicaStatus { #[serde(default, skip_serializing_if = "Option::is_none")] pub latest_available_snapshot: Option, + /// Snapshot id the canopy worklist syncer wants restored. Populated + /// each syncer tick for canopy-sourced replicas (`spec.canopySource` + /// is set); unused otherwise. The reconciler triggers a new restore + /// when this differs from the current one. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub canopy_desired_snapshot_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] pub connection_info: Option, diff --git a/tests/helpers.rs b/tests/helpers.rs index 2c919cb..0254c19 100644 --- a/tests/helpers.rs +++ b/tests/helpers.rs @@ -138,10 +138,11 @@ pub fn build_replica(name: &str, secret_ref: &str, opts: ReplicaOpts) -> Postgre PostgresPhysicalReplica::new( name, PostgresPhysicalReplicaSpec { - kopia_secret_ref: SecretReference { + kopia_secret_ref: Some(SecretReference { name: Some(secret_ref.into()), namespace: None, - }, + }), + canopy_source: None, snapshot_filter: None, schedule: opts.schedule, schedule_jitter: opts.schedule_jitter.unwrap_or_default(), From 5f16a45a7b93ee2ad4516dfbcc3510f3988e3b7b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 04:08:31 +1200 Subject: [PATCH 02/11] feat(canopy): KopiaSource enum + build_restore_job proxy-sidecar branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce a KopiaSource enum that captures whether kopia auth goes through a static Secret (legacy) or through a canopy-proxy sidecar. build_restore_job now branches on that: the canopy path prepends a shell prelude that waits for the sidecar's proxy-port file, exports KOPIA_ENDPOINT/KOPIA_DISABLE_TLS, and adds the canopy-proxy container + shared emptyDir Volume + proxy-sidecar Pod label so the operator broker NetworkPolicy admits it. The env-from-Secret block is unchanged in shape; canopy's materialised Secret carries dummy AWS creds because the proxy re-signs upstream. Callers pass an Option<&CanopyProxyArgs> — required whenever replica.spec.canopy_source is set (enforced by expect()). --- src/controllers/restore.rs | 11 + src/controllers/restore/builders.rs | 408 ++++++++++++++++++---------- src/controllers/restore/tests.rs | 3 + src/kopia.rs | 44 +++ src/types/replica.rs | 30 ++ 5 files changed, 354 insertions(+), 142 deletions(-) diff --git a/src/controllers/restore.rs b/src/controllers/restore.rs index dd37826..425fe28 100644 --- a/src/controllers/restore.rs +++ b/src/controllers/restore.rs @@ -446,6 +446,16 @@ async fn reconcile_restoring( let replica = replicas.get(replica_name).await?; let cache_pressure_url = ctx.cache_pressure_callback_url(namespace, name); + let stats_callback_url = ctx.canopy_stats_callback_url(namespace, &job_name); + let canopy_proxy = if replica.spec.canopy_source.is_some() { + Some(builders::CanopyProxyArgs { + image: &ctx.canopy_proxy_image, + broker_base_url: &ctx.canopy_broker_base_url, + stats_callback_url: &stats_callback_url, + }) + } else { + None + }; let job = build_restore_job( restore, &job_name, @@ -453,6 +463,7 @@ async fn reconcile_restoring( &replica, &ctx.kopia_image(), &cache_pressure_url, + canopy_proxy.as_ref(), )?; jobs.create(&PostParams::default(), &job).await? } diff --git a/src/controllers/restore/builders.rs b/src/controllers/restore/builders.rs index 0e0fddc..63ed9b2 100644 --- a/src/controllers/restore/builders.rs +++ b/src/controllers/restore/builders.rs @@ -5,9 +5,9 @@ use k8s_openapi::{ apps::v1::{Deployment, DeploymentSpec}, batch::v1::{Job, JobSpec}, core::v1::{ - Container, ContainerPort, EnvVar, ExecAction, PersistentVolumeClaim, - PersistentVolumeClaimSpec, PodSpec, PodTemplateSpec, Probe, ResourceRequirements, - SecretReference, Volume, VolumeMount, VolumeResourceRequirements, + Container, ContainerPort, EmptyDirVolumeSource, EnvVar, ExecAction, + PersistentVolumeClaim, PersistentVolumeClaimSpec, PodSpec, PodTemplateSpec, Probe, + ResourceRequirements, SecretReference, Volume, VolumeMount, VolumeResourceRequirements, }, }, apimachinery::pkg::{api::resource::Quantity, apis::meta::v1::LabelSelector}, @@ -20,10 +20,27 @@ use super::restore_owner_reference; use crate::{ controllers::{env_from_secret, env_from_secret_optional, kopia_writable_env}, error::{Error, Result}, + kopia::KopiaSource, quantity::compute_shm_and_shared_buffers, types::*, }; +/// Standard label used on Pods that carry a canopy-proxy sidecar so the +/// operator's broker NetworkPolicy can admit their ingress. +pub const PROXY_SIDECAR_POD_LABEL: (&str, &str) = ("pgro.bes.au/proxy-sidecar", "true"); + +/// Extra inputs the canopy path needs on top of the legacy args. +pub struct CanopyProxyArgs<'a> { + /// Image of the canopy-proxy sidecar (same image as the operator; the + /// container runs the `canopy-proxy` binary instead of `operator`). + pub image: &'a str, + /// Base URL the sidecar hits for STS creds + /// (e.g. `http://postgres-restore-operator.pgro-system.svc:9091`). + pub broker_base_url: &'a str, + /// Callback URL the sidecar POSTs its final TrafficStats to on shutdown. + pub stats_callback_url: &'a str, +} + /// Name of the credential-reset Job for a given restore. pub fn credential_reset_job_name(restore_name: &str) -> String { format!("{restore_name}-cred-reset") @@ -512,20 +529,39 @@ pub fn build_restore_job( replica: &PostgresPhysicalReplica, kopia_image: &str, cache_pressure_callback_url: &str, + canopy_proxy: Option<&CanopyProxyArgs<'_>>, ) -> Result { - // Legacy-path restore Job builder. The canopy path uses a different - // Job shape (proxy sidecar) that will be wired in a follow-up commit - // via a `KopiaSource` enum; for now the reconciler's credential-source - // validation guarantees `kopia_secret_ref` is set before we reach here. - let kopia_secret = replica - .spec - .kopia_secret_ref - .as_ref() - .expect("build_restore_job called on legacy path; kopia_secret_ref must be set"); + let source = replica.kopia_source(); + let kopia_secret = SecretReference { + name: Some(source.secret_name().to_string()), + namespace: None, + }; let pvc_name = format!("{}-data", restore.name_any()); let cache_pvc_name = kopia_cache_pvc_name(&restore.spec.replica.name); - let restore_script = r#"set -e + // Canopy prelude waits for the sidecar to bind, then exports the + // loopback endpoint/disable-tls so the main script's connect step + // picks them up via ENDPOINT_ARGS. No-op on the legacy path. + let canopy_prelude = if source.is_canopy_proxy() { + r#"PORT_FILE="/var/run/pgro/proxy-port" +for _ in $(seq 1 30); do + [ -f "$PORT_FILE" ] && break + sleep 1 +done +if [ ! -f "$PORT_FILE" ]; then + echo "ERROR: canopy-proxy sidecar did not write port file within 30s" >&2 + exit 1 +fi +export KOPIA_ENDPOINT="[::1]:$(cat "$PORT_FILE")" +export KOPIA_DISABLE_TLS=true +echo "kopia connecting via canopy proxy at ${KOPIA_ENDPOINT}" + +"# + } else { + "" + }; + + let restore_script_body = r#"set -e mkdir -p /tmp/kopia/config /tmp/kopia/logs /tmp/kopia/cache @@ -622,33 +658,232 @@ echo "$VERSION" > /pgdata/.postgres-version echo -n "$VERSION" > /dev/termination-log "#; + let restore_script = format!("{canopy_prelude}{restore_script_body}"); + + // Base env: shared between legacy and canopy. env_from_secret uses the + // same key names in both Secrets (bucket, region, accessKeyId, + // secretAccessKey, repositoryPassword); the canopy path just fills + // AWS creds with dummy values because the proxy re-signs upstream. + let mut env: Vec = [ + vec![ + EnvVar { + name: "SNAPSHOT_ID".to_string(), + value: Some(restore.spec.snapshot.clone()), + ..Default::default() + }, + EnvVar { + name: "KOPIA_CONTENT_CACHE_MB".to_string(), + value: Some(kopia_content_cache_mb(&restore.spec.snapshot_size).to_string()), + ..Default::default() + }, + EnvVar { + name: "KOPIA_METADATA_CACHE_MB".to_string(), + value: Some(KOPIA_METADATA_CACHE_MB.to_string()), + ..Default::default() + }, + EnvVar { + name: "CACHE_PRESSURE_CALLBACK_URL".to_string(), + value: Some(cache_pressure_callback_url.to_string()), + ..Default::default() + }, + ], + kopia_writable_env(), + vec![ + env_from_secret("KOPIA_BUCKET", &kopia_secret, "bucket"), + env_from_secret("KOPIA_REGION", &kopia_secret, "region"), + env_from_secret("AWS_ACCESS_KEY_ID", &kopia_secret, "accessKeyId"), + env_from_secret("AWS_SECRET_ACCESS_KEY", &kopia_secret, "secretAccessKey"), + env_from_secret("KOPIA_PASSWORD", &kopia_secret, "repositoryPassword"), + ], + ] + .concat(); + + // Legacy Secrets may carry KOPIA_ENDPOINT / KOPIA_DISABLE_TLS as an + // escape hatch (e.g. for MinIO); canopy sets them via the shell prelude + // after the proxy binds, so we skip these optional keys on the canopy + // path to avoid a needless lookup. + if !source.is_canopy_proxy() { + env.push(env_from_secret_optional( + "KOPIA_ENDPOINT", + &kopia_secret, + "endpoint", + )); + env.push(env_from_secret_optional( + "KOPIA_DISABLE_TLS", + &kopia_secret, + "disableTls", + )); + } + + let volume_mounts = vec![ + VolumeMount { + name: "pgdata".to_string(), + mount_path: "/pgdata".to_string(), + ..Default::default() + }, + VolumeMount { + name: "kopia-cache".to_string(), + mount_path: "/tmp/kopia".to_string(), + ..Default::default() + }, + ]; + let mut volumes = vec![ + Volume { + name: "pgdata".to_string(), + persistent_volume_claim: Some( + k8s_openapi::api::core::v1::PersistentVolumeClaimVolumeSource { + claim_name: pvc_name, + read_only: Some(false), + }, + ), + ..Default::default() + }, + Volume { + name: "kopia-cache".to_string(), + persistent_volume_claim: Some( + k8s_openapi::api::core::v1::PersistentVolumeClaimVolumeSource { + claim_name: cache_pvc_name, + read_only: Some(false), + }, + ), + ..Default::default() + }, + ]; + + let mut pod_labels = BTreeMap::from([ + ( + "pgro.bes.au/replica".to_string(), + restore.spec.replica.name.clone(), + ), + ("pgro.bes.au/restore".to_string(), restore.name_any()), + ]); + + let mut containers = Vec::with_capacity(2); + containers.push(Container { + name: "restore".to_string(), + image: Some(kopia_image.to_string()), + command: Some(vec!["/bin/sh".to_string(), "-c".to_string()]), + args: Some(vec![restore_script]), + env: Some(env), + volume_mounts: Some(volume_mounts), + resources: Some(ResourceRequirements { + requests: Some(BTreeMap::from([ + ("cpu".to_string(), Quantity("500m".to_string())), + ("memory".to_string(), Quantity("1Gi".to_string())), + ])), + limits: Some(BTreeMap::from([ + ("cpu".to_string(), Quantity("2".to_string())), + ("memory".to_string(), Quantity("4Gi".to_string())), + ])), + ..Default::default() + }), + ..Default::default() + }); + + if let KopiaSource::CanopyProxy { + group, backup_type, .. + } = &source + { + let proxy = canopy_proxy.expect( + "build_restore_job called with canopy_source but no CanopyProxyArgs; caller must \ + thread proxy config from Context when replica.spec.canopy_source is set", + ); + + containers[0] + .volume_mounts + .as_mut() + .unwrap() + .push(VolumeMount { + name: "proxy-shared".to_string(), + mount_path: "/var/run/pgro".to_string(), + ..Default::default() + }); + + containers.push(Container { + name: "canopy-proxy".to_string(), + image: Some(proxy.image.to_string()), + // Same image as the operator; run the `canopy-proxy` binary + // instead of the default `operator` entrypoint. + command: Some(vec!["canopy-proxy".to_string()]), + env: Some(vec![ + EnvVar { + name: "PGRO_BROKER_URL".to_string(), + value: Some(proxy.broker_base_url.to_string()), + ..Default::default() + }, + EnvVar { + name: "PGRO_GROUP".to_string(), + value: Some(group.clone()), + ..Default::default() + }, + EnvVar { + name: "PGRO_TYPE".to_string(), + value: Some(backup_type.clone()), + ..Default::default() + }, + EnvVar { + name: "PGRO_STATS_CALLBACK_URL".to_string(), + value: Some(proxy.stats_callback_url.to_string()), + ..Default::default() + }, + ]), + volume_mounts: Some(vec![VolumeMount { + name: "proxy-shared".to_string(), + mount_path: "/var/run/pgro".to_string(), + ..Default::default() + }]), + resources: Some(ResourceRequirements { + requests: Some(BTreeMap::from([ + ("cpu".to_string(), Quantity("50m".to_string())), + ("memory".to_string(), Quantity("64Mi".to_string())), + ])), + limits: Some(BTreeMap::from([ + ("cpu".to_string(), Quantity("500m".to_string())), + ("memory".to_string(), Quantity("256Mi".to_string())), + ])), + ..Default::default() + }), + ..Default::default() + }); + + volumes.push(Volume { + name: "proxy-shared".to_string(), + empty_dir: Some(EmptyDirVolumeSource { + medium: Some("Memory".to_string()), + ..Default::default() + }), + ..Default::default() + }); + + pod_labels.insert( + PROXY_SIDECAR_POD_LABEL.0.to_string(), + PROXY_SIDECAR_POD_LABEL.1.to_string(), + ); + } + + // Canopy's proxy refreshes creds so long restores aren't + // credential-bounded, only reachability-bounded — bump to 4 h. + let active_deadline_seconds = if source.is_canopy_proxy() { + 14400 + } else { + 7200 + }; + Ok(Job { metadata: ObjectMeta { name: Some(job_name.to_string()), namespace: Some(namespace.to_string()), - labels: Some(BTreeMap::from([ - ( - "pgro.bes.au/replica".to_string(), - restore.spec.replica.name.clone(), - ), - ("pgro.bes.au/restore".to_string(), restore.name_any()), - ])), + labels: Some(pod_labels.clone()), owner_references: Some(vec![restore_owner_reference(restore)]), ..Default::default() }, spec: Some(JobSpec { backoff_limit: Some(3), - active_deadline_seconds: Some(7200), // 2 hours - ttl_seconds_after_finished: Some(120), // safety net if operator misses deletion + active_deadline_seconds: Some(active_deadline_seconds), + ttl_seconds_after_finished: Some(120), template: PodTemplateSpec { metadata: Some(ObjectMeta { - labels: Some(BTreeMap::from([ - ( - "pgro.bes.au/replica".to_string(), - restore.spec.replica.name.clone(), - ), - ("pgro.bes.au/restore".to_string(), restore.name_any()), - ])), + labels: Some(pod_labels), ..Default::default() }), spec: Some(PodSpec { @@ -659,119 +894,8 @@ echo -n "$VERSION" > /dev/termination-log fs_group: Some(999), ..Default::default() }), - - containers: vec![Container { - name: "restore".to_string(), - image: Some(kopia_image.to_string()), - command: Some(vec!["/bin/sh".to_string(), "-c".to_string()]), - args: Some(vec![restore_script.to_string()]), - env: Some( - [ - vec![ - EnvVar { - name: "SNAPSHOT_ID".to_string(), - value: Some(restore.spec.snapshot.clone()), - ..Default::default() - }, - EnvVar { - name: "KOPIA_CONTENT_CACHE_MB".to_string(), - value: Some( - kopia_content_cache_mb(&restore.spec.snapshot_size) - .to_string(), - ), - ..Default::default() - }, - EnvVar { - name: "KOPIA_METADATA_CACHE_MB".to_string(), - value: Some(KOPIA_METADATA_CACHE_MB.to_string()), - ..Default::default() - }, - EnvVar { - name: "CACHE_PRESSURE_CALLBACK_URL".to_string(), - value: Some(cache_pressure_callback_url.to_string()), - ..Default::default() - }, - ], - kopia_writable_env(), - vec![ - env_from_secret("KOPIA_BUCKET", kopia_secret, "bucket"), - env_from_secret("KOPIA_REGION", kopia_secret, "region"), - env_from_secret( - "AWS_ACCESS_KEY_ID", - kopia_secret, - "accessKeyId", - ), - env_from_secret( - "AWS_SECRET_ACCESS_KEY", - kopia_secret, - "secretAccessKey", - ), - env_from_secret( - "KOPIA_PASSWORD", - kopia_secret, - "repositoryPassword", - ), - env_from_secret_optional( - "KOPIA_ENDPOINT", - kopia_secret, - "endpoint", - ), - env_from_secret_optional( - "KOPIA_DISABLE_TLS", - kopia_secret, - "disableTls", - ), - ], - ] - .concat(), - ), - volume_mounts: Some(vec![ - VolumeMount { - name: "pgdata".to_string(), - mount_path: "/pgdata".to_string(), - ..Default::default() - }, - k8s_openapi::api::core::v1::VolumeMount { - name: "kopia-cache".to_string(), - mount_path: "/tmp/kopia".to_string(), - ..Default::default() - }, - ]), - resources: Some(ResourceRequirements { - requests: Some(BTreeMap::from([ - ("cpu".to_string(), Quantity("500m".to_string())), - ("memory".to_string(), Quantity("1Gi".to_string())), - ])), - limits: Some(BTreeMap::from([ - ("cpu".to_string(), Quantity("2".to_string())), - ("memory".to_string(), Quantity("4Gi".to_string())), - ])), - ..Default::default() - }), - ..Default::default() - }], - volumes: Some(vec![ - Volume { - name: "pgdata".to_string(), - persistent_volume_claim: Some( - k8s_openapi::api::core::v1::PersistentVolumeClaimVolumeSource { - claim_name: pvc_name, - read_only: Some(false), - }, - ), - ..Default::default() - }, - Volume { - name: "kopia-cache".to_string(), - persistent_volume_claim: Some( - k8s_openapi::api::core::v1::PersistentVolumeClaimVolumeSource { - claim_name: cache_pvc_name, - read_only: Some(false), - }, - ), - ..Default::default() - }, - ]), + containers, + volumes: Some(volumes), ..Default::default() }), }, diff --git a/src/controllers/restore/tests.rs b/src/controllers/restore/tests.rs index 4b0dccb..9c6ec92 100644 --- a/src/controllers/restore/tests.rs +++ b/src/controllers/restore/tests.rs @@ -156,6 +156,7 @@ fn restore_job_has_ttl_seconds_after_finished() { &replica, "kopia:latest", "http://operator/api/v1/cache-pressure/default/test-restore", + None, ) .unwrap(); let ttl = job @@ -182,6 +183,7 @@ fn restore_job_mounts_persistent_kopia_cache() { &replica, "kopia:latest", "http://operator/api/v1/cache-pressure/default/test-restore", + None, ) .unwrap(); let pod_spec = job.spec.unwrap().template.spec.unwrap(); @@ -328,6 +330,7 @@ fn restore_job_passes_cache_caps_and_log_rotation() { &replica, "kopia:latest", "http://operator/api/v1/cache-pressure/default/test-restore", + None, ) .unwrap(); let pod_spec = job.spec.unwrap().template.spec.unwrap(); diff --git a/src/kopia.rs b/src/kopia.rs index 990fcf5..32664d6 100644 --- a/src/kopia.rs +++ b/src/kopia.rs @@ -230,6 +230,50 @@ pub fn kopia_connect_args(creds: &KopiaCredentials) -> Vec { args } +/// The credential source a kopia Job runs against — determines whether +/// the Job uses a user-authored `kopiaSecretRef` Secret with real AWS +/// keys, or the canopy proxy-sidecar path with dummy keys and STS creds +/// refreshed by the sidecar. Derived from the parent +/// `PostgresPhysicalReplica`'s spec. +#[derive(Debug, Clone)] +pub enum KopiaSource { + /// Legacy path: env vars sourced from a Secret containing real AWS + /// credentials + bucket/region/repo-password. + Secret { + /// Namespace-local Secret name (from `spec.kopiaSecretRef.name`). + secret_name: String, + }, + /// Canopy path: proxy sidecar handles credential refresh; kopia sees + /// dummy keys pointing at a loopback endpoint. The named Secret holds + /// dummy AWS keys + canopy-provided bucket/region/prefix + repo + /// password; the canopy syncer materialises it before the Job runs. + CanopyProxy { + /// Namespace-local Secret name (materialised by the canopy syncer + /// as `-canopy-creds`). + secret_name: String, + /// Canopy group id, passed to the sidecar as `PGRO_GROUP`. + group: String, + /// Canopy backup type, passed to the sidecar as `PGRO_TYPE`. + backup_type: String, + }, +} + +impl KopiaSource { + /// Name of the namespace-local Secret that carries kopia's env vars + /// (bucket, region, prefix, repo password, and — for legacy — real + /// AWS keys; for canopy — dummy keys). + pub fn secret_name(&self) -> &str { + match self { + Self::Secret { secret_name } => secret_name, + Self::CanopyProxy { secret_name, .. } => secret_name, + } + } + + pub fn is_canopy_proxy(&self) -> bool { + matches!(self, Self::CanopyProxy { .. }) + } +} + /// Parameters for the kopia-via-canopy-proxy connect convention. /// /// kopia talks to a loopback bestool S3P proxy with dummy keys; the proxy holds diff --git a/src/types/replica.rs b/src/types/replica.rs index 0d26e52..ca0941e 100644 --- a/src/types/replica.rs +++ b/src/types/replica.rs @@ -491,6 +491,36 @@ impl PostgresPhysicalReplica { pub fn creds_secret_name(&self) -> String { format!("{name}-creds", name = self.name_any()) } + + /// Name of the operator-materialised Secret that holds the canopy path's + /// dummy AWS keys + canopy-provided bucket/region/prefix/repo-password. + /// Only meaningful when `spec.canopy_source` is set — the canopy syncer + /// creates this Secret before the reconciler spawns a restore Job. + pub fn canopy_creds_secret_name(&self) -> String { + format!("{name}-canopy-creds", name = self.name_any()) + } + + /// Derive the credential source for kopia Jobs — the reconciler has + /// already validated exactly one of `kopia_secret_ref` / `canopy_source` + /// is set before we reach any callsite that needs this, so the + /// `.expect` never fires in practice. + pub fn kopia_source(&self) -> crate::kopia::KopiaSource { + if let Some(canopy) = &self.spec.canopy_source { + crate::kopia::KopiaSource::CanopyProxy { + secret_name: self.canopy_creds_secret_name(), + group: canopy.group.clone(), + backup_type: canopy.r#type.clone(), + } + } else { + let secret_name = self + .spec + .kopia_secret_ref + .as_ref() + .and_then(|r| r.name.clone()) + .expect("kopia_source called with neither kopia_secret_ref nor canopy_source set"); + crate::kopia::KopiaSource::Secret { secret_name } + } + } } #[cfg(test)] From e252a14f2f68a03364525e34d0d822594ed4bb0c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 04:11:17 +1200 Subject: [PATCH 03/11] feat(canopy): reconciler skips snapshot-list on canopy path When `replica.spec.canopy_source` is set, the reconciler no longer creates or processes a snapshot-list Job. Instead it reads `status.canopyDesiredSnapshotId` (written by the canopy worklist syncer in a later step) and calls `create_restore_for_snapshot` directly when the desired snapshot differs from the currently active one. Canopy replicas rely on intent-driven `storage_size_override` for PVC sizing since the snapshot size isn't known ahead of the restore; a condition `SnapshotAvailable=False (CanopyPending)` surfaces the wait state when the syncer hasn't set the desired snapshot yet. --- src/controllers/replica.rs | 86 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 82 insertions(+), 4 deletions(-) diff --git a/src/controllers/replica.rs b/src/controllers/replica.rs index 9a684be..7267cfa 100644 --- a/src/controllers/replica.rs +++ b/src/controllers/replica.rs @@ -546,12 +546,18 @@ pub async fn reconcile(replica: Arc, ctx: Arc) } } - // Process any existing snapshot-list job regardless of scheduling state. - // This runs before the in-progress / should_restore gates so that - // completed jobs are always cleaned up promptly. + // Canopy-sourced replicas don't run snapshot-list Jobs — the desired + // snapshot ID is written into `status.canopyDesiredSnapshotId` by the + // canopy worklist syncer. We still allocate the job-name here so the + // downstream `if snapshot_job.is_none()` gate reads uniformly. + let is_canopy = replica.spec.canopy_source.is_some(); let snapshot_job_name = format!("{name}-snapshot-list"); let jobs: Api = Api::namespaced(client.clone(), &namespace); - let snapshot_job = jobs.get_opt(&snapshot_job_name).await?; + let snapshot_job = if is_canopy { + None + } else { + jobs.get_opt(&snapshot_job_name).await? + }; if let Some(ref job) = snapshot_job { match classify_job(job) { @@ -888,6 +894,78 @@ pub async fn reconcile(replica: Arc, ctx: Arc) } drop(queue); + if is_canopy { + // Canopy path: the worklist syncer has already picked a + // snapshot for us. Use `status.canopyDesiredSnapshotId` + // directly and skip the snapshot-list Job entirely. + let desired = replica + .status + .as_ref() + .and_then(|s| s.canopy_desired_snapshot_id.as_ref()); + let Some(desired) = desired else { + debug!( + replica = name, + "canopy-sourced replica has no canopyDesiredSnapshotId yet, waiting for syncer" + ); + replica + .update_condition( + client, + "SnapshotAvailable", + "False", + "CanopyPending", + "Waiting for canopy worklist syncer to set canopyDesiredSnapshotId", + ) + .await?; + return Ok(Action::requeue(Duration::from_secs(30))); + }; + + let current_snapshot_id = active_restore.map(|r| r.spec.snapshot.as_str()); + if current_snapshot_id == Some(desired.as_str()) { + debug!( + replica = name, + snapshot = desired, + "canopy desired snapshot already active, skipping" + ); + } else { + info!( + replica = name, + snapshot = desired, + "canopy desired snapshot changed, creating restore" + ); + // Canopy replicas rely on intent-driven + // `storage_size_override` for sizing; snapshot size is + // not known upfront so we pass 0. The override branch + // in create_restore_for_snapshot then picks the fixed + // PVC size. + let info = SnapshotInfo { + id: desired.clone(), + size: 0, + start_time: String::new(), + }; + let created = replica.create_restore_for_snapshot(client, &info).await?; + if created { + ctx.metrics.restores_started_total.inc(); + if let Err(e) = ctx + .recorder + .publish( + &Event { + type_: EventType::Normal, + reason: "RestoreStarted".into(), + note: Some(format!("Started restore from snapshot {desired}")), + action: "Restore".into(), + secondary: None, + }, + &replica.object_ref(&()), + ) + .await + { + warn!(replica = name, error = %e, "failed to publish RestoreStarted event"); + } + } + } + return Ok(Action::requeue(Duration::from_secs(30))); + } + info!(replica = name, "creating snapshot list job"); let callback_url = ctx.snapshot_callback_url(&namespace, &name); let job = build_snapshot_list_job( From a23945986a31ad563d37a286b56cbd93f289864b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 04:16:41 +1200 Subject: [PATCH 04/11] feat(canopy): intent config module + canopy-desired-snapshot trigger MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce `src/controllers/canopy/intent.rs` with the three supported intents — verify, analytics-dev, analytics-dbt — and their fixed configuration (resources, read_only, minimum_ttl, persistent_schemas, service_annotations, switchover_grace_period, storage_size_override). IntentConfig::to_replica_spec materialises a PostgresPhysicalReplicaSpec from a WorklistEntry + intent config; {name} in service annotations is substituted with the entry's name at materialisation time. The operator's SUPPORTED_INTENTS constant now re-exports the intent module's SUPPORTED slice, so adding an intent only touches one place. Adjust the reconciler so a change to status.canopyDesiredSnapshotId becomes a restore trigger on the canopy path (in addition to schedule, never-restored, active-restore-deleted). Extract PostgresPhysicalReplica:: within_minimum_ttl so the canopy trigger honours minimum_ttl even when the schedule cron isn't the driver — analytics-dbt's 2h TTL still gates canopy-provided newer snapshots. --- src/bin/operator.rs | 8 +- src/controllers/canopy.rs | 1 + src/controllers/canopy/intent.rs | 272 ++++++++++++++++++++++++++ src/controllers/replica.rs | 21 ++ src/controllers/replica/scheduling.rs | 22 +++ 5 files changed, 317 insertions(+), 7 deletions(-) create mode 100644 src/controllers/canopy/intent.rs diff --git a/src/bin/operator.rs b/src/bin/operator.rs index acf5d2d..f6a69f3 100644 --- a/src/bin/operator.rs +++ b/src/bin/operator.rs @@ -25,7 +25,7 @@ use postgres_restore_operator::{ Context, DEFAULT_CANOPY_PGDATA_PVC_SIZE, DEFAULT_CANOPY_PROXY_IMAGE, DEFAULT_DEPLOYMENT_READY_TIMEOUT_SECS, DEFAULT_KOPIA_IMAGE, }, - controllers, + controllers::{self, canopy::intent::SUPPORTED as PGRO_SUPPORTED_INTENTS}, types::{PostgresPhysicalReplica, PostgresPhysicalRestore}, }; @@ -44,12 +44,6 @@ const DEFAULT_BROKER_ADDR: &str = "[::]:9091"; const DEFAULT_CANOPY_RECONCILE_INTERVAL_SECS: u64 = 30; const CONFIGMAP_NAME: &str = "postgres-restore-operator-config"; -/// Intent set pgro registers with canopy on startup; only worklist entries -/// with a matching intent will be dispatched. `disaster-recovery` is not -/// yet supported — the code has no rehearsal lifecycle beyond "make it -/// writable", which is not what DR actually needs. -const PGRO_SUPPORTED_INTENTS: &[&str] = &["verify", "analytics"]; - /// Annotate the operator's own pod with the running version. async fn annotate_own_pod(client: &Client, namespace: &str) { let pod_name = match std::env::var("HOSTNAME") { diff --git a/src/controllers/canopy.rs b/src/controllers/canopy.rs index db15fc1..46e3a0e 100644 --- a/src/controllers/canopy.rs +++ b/src/controllers/canopy.rs @@ -37,6 +37,7 @@ use uuid::Uuid; use crate::{context::Context, error::Result}; mod builders; +pub mod intent; mod reporter; pub use builders::{ CanopyRestoreJobConfig, KOPIA_JOB_NAME, PGDATA_PVC_NAME, POSTGRES_DEPLOYMENT_NAME, diff --git a/src/controllers/canopy/intent.rs b/src/controllers/canopy/intent.rs new file mode 100644 index 0000000..85ccf94 --- /dev/null +++ b/src/controllers/canopy/intent.rs @@ -0,0 +1,272 @@ +//! Intent-driven configuration for canopy-backed replicas. +//! +//! Each canopy `WorklistEntry` carries an `intent` string. pgro registers +//! its supported intents at startup ([`SUPPORTED`]) so canopy only +//! dispatches entries it can handle. On each tick, the syncer looks up the +//! entry's intent in [`config_for`] and calls +//! [`IntentConfig::to_replica_spec`] to materialise a +//! `PostgresPhysicalReplicaSpec` — the CR path takes over from there. +//! +//! Adding an intent is a two-step edit: +//! 1. Add a new [`IntentConfig`] entry to [`config_for`]. +//! 2. Extend [`SUPPORTED`] with the new intent name. + +use std::collections::BTreeMap; + +use bestool_canopy::WorklistEntry; +use jiff::Span; +use k8s_openapi::{ + api::core::v1::{ResourceRequirements, SecretReference}, + apimachinery::pkg::api::resource::Quantity, +}; + +use crate::{ + types::{CanopySource, PostgresPhysicalReplicaSpec}, + util::TimeSpan, +}; + +/// pgro-supported intent names, registered with canopy at operator startup. +/// Canopy only dispatches worklist entries whose intent appears here. +pub const SUPPORTED: &[&str] = &["verify", "analytics-dev", "analytics-dbt"]; + +/// Intent-derived spec fragments merged onto the base replica spec. +#[derive(Debug, Clone)] +pub struct IntentConfig { + pub resources: Option, + pub read_only: bool, + pub minimum_ttl: Option, + pub persistent_schemas: Option>, + /// Service annotations. `{name}` in a value is substituted with the + /// worklist entry's `name` at materialisation time. + pub service_annotations: Option>, + pub switchover_grace_period: TimeSpan, + pub storage_size_override: Quantity, +} + +fn resources(cpu_req: &str, mem_req: &str, cpu_lim: &str, mem_lim: &str) -> ResourceRequirements { + ResourceRequirements { + requests: Some(BTreeMap::from([ + ("cpu".to_string(), Quantity(cpu_req.to_string())), + ("memory".to_string(), Quantity(mem_req.to_string())), + ])), + limits: Some(BTreeMap::from([ + ("cpu".to_string(), Quantity(cpu_lim.to_string())), + ("memory".to_string(), Quantity(mem_lim.to_string())), + ])), + ..Default::default() + } +} + +/// Look up the fixed configuration for a supported intent name. Returns +/// `None` for unsupported intents — canopy shouldn't dispatch these +/// because they aren't in [`SUPPORTED`], but callers must still handle +/// the possibility (e.g. a worklist entry sneaking through during an +/// operator downgrade). +pub fn config_for(intent: &str) -> Option { + match intent { + "verify" => Some(IntentConfig { + resources: Some(resources("250m", "512Mi", "2", "2Gi")), + read_only: true, + minimum_ttl: None, + persistent_schemas: None, + service_annotations: None, + switchover_grace_period: TimeSpan(Span::new().minutes(5)), + storage_size_override: Quantity("20Gi".to_string()), + }), + "analytics-dev" => Some(IntentConfig { + resources: Some(resources("500m", "2Gi", "4", "8Gi")), + read_only: true, + minimum_ttl: None, + persistent_schemas: None, + service_annotations: None, + switchover_grace_period: TimeSpan(Span::new().minutes(5)), + storage_size_override: Quantity("50Gi".to_string()), + }), + "analytics-dbt" => Some(IntentConfig { + resources: Some(resources("500m", "2Gi", "4", "8Gi")), + read_only: true, + minimum_ttl: Some(TimeSpan(Span::new().hours(2))), + persistent_schemas: Some(vec!["dbt".to_string()]), + service_annotations: Some(BTreeMap::from([ + ("tailscale.com/expose".to_string(), "true".to_string()), + ( + "tailscale.com/hostname".to_string(), + "infra-replica-{name}".to_string(), + ), + ])), + switchover_grace_period: TimeSpan(Span::new().minutes(2)), + storage_size_override: Quantity("50Gi".to_string()), + }), + _ => None, + } +} + +/// Substitute `{name}` in each value with `entry_name`. Other braces are +/// left alone; there's only one placeholder in current use. +fn substitute_annotations( + base: BTreeMap, + entry_name: &str, +) -> BTreeMap { + base.into_iter() + .map(|(k, v)| (k, v.replace("{name}", entry_name))) + .collect() +} + +impl IntentConfig { + /// Materialise a `PostgresPhysicalReplicaSpec` for a canopy-managed + /// replica. The syncer patches the CR with this spec on Provision / + /// re-asserts it on subsequent ticks so drift from manual edits is + /// self-healing. + pub fn to_replica_spec( + &self, + entry: &WorklistEntry, + notifications: Vec, + ) -> PostgresPhysicalReplicaSpec { + PostgresPhysicalReplicaSpec { + kopia_secret_ref: None, + canopy_source: Some(CanopySource { + group: entry.group_id.to_string(), + r#type: entry.r#type.to_string(), + }), + snapshot_filter: None, + // Long cadence — the actual restore trigger on the canopy + // path is a change to `status.canopyDesiredSnapshotId` + // written by the worklist syncer. The cron is a + // belt-and-braces fallback (e.g. missed status watch). + schedule: "H * * * *".to_string(), + schedule_jitter: TimeSpan(Span::new().minutes(10)), + minimum_ttl: self.minimum_ttl, + switchover_grace_period: self.switchover_grace_period, + analytics_username: "analytics".to_string(), + storage_class: None, + storage_size_override: Some(self.storage_size_override.clone()), + resources: self.resources.clone(), + service_annotations: self + .service_annotations + .clone() + .map(|a| substitute_annotations(a, &entry.name)), + pod_annotations: None, + affinity: None, + tolerations: Vec::new(), + read_only: self.read_only, + postgres_extra_config: None, + notifications, + persistent_schemas: self.persistent_schemas.clone(), + storage_size_maximum: Quantity("2Ti".to_string()), + } + } + + /// Name of the namespace-local Secret the canopy syncer materialises + /// with the worklist entry's bucket / region / prefix / repo password + /// + dummy AWS keys. `build_restore_job` mounts it via env_from_secret. + pub fn canopy_creds_secret_name(replica_name: &str) -> String { + format!("{replica_name}-canopy-creds") + } + + /// Convenience: build a `SecretReference` pointing at the canopy creds + /// Secret for the given replica. + pub fn canopy_creds_secret_ref(replica_name: &str) -> SecretReference { + SecretReference { + name: Some(Self::canopy_creds_secret_name(replica_name)), + namespace: None, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use uuid::Uuid; + + fn entry(intent: &str, name: &str) -> WorklistEntry { + serde_json::from_value(serde_json::json!({ + "replica_id": Uuid::new_v4().to_string(), + "group_id": Uuid::new_v4().to_string(), + "server_id": Uuid::new_v4().to_string(), + "type": "tamanu-postgres", + "intent": intent, + "name": name, + "snapshot_id": "abc123", + "snapshot_at": "2026-07-01T00:00:00Z", + "storage": "s3", + "bucket": "canopy-test", + "prefix": "", + "region": "ap-southeast-2", + })) + .unwrap() + } + + #[test] + fn config_for_verify() { + let cfg = config_for("verify").expect("verify is supported"); + assert!(cfg.read_only); + assert!(cfg.minimum_ttl.is_none()); + assert!(cfg.persistent_schemas.is_none()); + } + + #[test] + fn config_for_analytics_dbt_has_all_extras() { + let cfg = config_for("analytics-dbt").expect("analytics-dbt is supported"); + assert!(cfg.minimum_ttl.is_some()); + assert_eq!( + cfg.persistent_schemas.as_deref(), + Some(&["dbt".to_string()][..]) + ); + assert!(cfg.service_annotations.is_some()); + } + + #[test] + fn config_for_unknown_intent() { + assert!(config_for("disaster-recovery").is_none()); + assert!(config_for("").is_none()); + } + + #[test] + fn supported_names_all_resolve() { + for name in SUPPORTED { + assert!( + config_for(name).is_some(), + "SUPPORTED lists {name} but config_for returned None" + ); + } + } + + #[test] + fn to_replica_spec_substitutes_name_in_service_annotations() { + let cfg = config_for("analytics-dbt").unwrap(); + let e = entry("analytics-dbt", "example-site"); + let spec = cfg.to_replica_spec(&e, vec![]); + let annos = spec.service_annotations.expect("dbt has annotations"); + assert_eq!( + annos.get("tailscale.com/hostname").map(String::as_str), + Some("infra-replica-example-site") + ); + assert_eq!( + annos.get("tailscale.com/expose").map(String::as_str), + Some("true") + ); + } + + #[test] + fn to_replica_spec_sets_canopy_source_from_entry() { + let cfg = config_for("verify").unwrap(); + let e = entry("verify", "test"); + let spec = cfg.to_replica_spec(&e, vec![]); + assert!(spec.kopia_secret_ref.is_none()); + let cs = spec.canopy_source.expect("canopy_source must be set"); + assert_eq!(cs.r#type, "tamanu-postgres"); + assert_eq!(cs.group, e.group_id.to_string()); + } + + #[test] + fn to_replica_spec_dbt_carries_migration_settings() { + let cfg = config_for("analytics-dbt").unwrap(); + let e = entry("analytics-dbt", "test"); + let spec = cfg.to_replica_spec(&e, vec![]); + assert!(spec.minimum_ttl.is_some()); + assert_eq!( + spec.persistent_schemas.as_deref(), + Some(&["dbt".to_string()][..]) + ); + } +} diff --git a/src/controllers/replica.rs b/src/controllers/replica.rs index 7267cfa..fe55386 100644 --- a/src/controllers/replica.rs +++ b/src/controllers/replica.rs @@ -861,8 +861,29 @@ pub async fn reconcile(replica: Arc, ctx: Arc) let schedule_decision = replica.check_schedule(); + // On the canopy path, the worklist syncer updates + // `status.canopyDesiredSnapshotId` when canopy offers a newer snapshot. + // Trigger a restore whenever the desired snapshot differs from what the + // active restore already carries, in addition to the usual schedule / + // never-restored / active-deleted triggers. minimum_ttl still gates: + // the intent may declare an explicit lower bound on restore frequency + // even in the face of a newer canopy snapshot. + let canopy_desired_changed = is_canopy + && match ( + replica + .status + .as_ref() + .and_then(|s| s.canopy_desired_snapshot_id.as_ref()), + active_restore.map(|r| r.spec.snapshot.as_str()), + ) { + (Some(desired), Some(current)) => desired != current, + (Some(_), None) => true, + _ => false, + } && !replica.within_minimum_ttl(now); + let should_restore = never_restored || active_restore_deleted + || canopy_desired_changed || matches!(schedule_decision, ScheduleDecision::Trigger); if should_restore && snapshot_job.is_none() { diff --git a/src/controllers/replica/scheduling.rs b/src/controllers/replica/scheduling.rs index eb6a350..89b73f9 100644 --- a/src/controllers/replica/scheduling.rs +++ b/src/controllers/replica/scheduling.rs @@ -151,6 +151,28 @@ impl PostgresPhysicalReplica { Some(next.into()) } + /// True when the replica's `minimum_ttl` is configured and the last + /// restore completed within that window relative to `now`. Used by both + /// the cron path (via [`Self::check_schedule`]) and the canopy path + /// where TTL gates the desired-snapshot trigger. + pub fn within_minimum_ttl(&self, now: Timestamp) -> bool { + let Some(ref minimum_ttl) = self.spec.minimum_ttl else { + return false; + }; + let Some(last_completed) = self + .status + .as_ref() + .and_then(|s| s.last_restore_completed_at.as_ref()) + else { + return false; + }; + let not_before = last_completed + .0 + .to_zoned(TimeZone::UTC) + .saturating_add(minimum_ttl.0); + now.to_zoned(TimeZone::UTC) < not_before + } + pub fn check_schedule(&self) -> ScheduleDecision { let name = self.name_any(); let schedule = &self.spec.schedule; From ba22290b5b3a1deb385a7094bbb40deb2649e545 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 04:21:56 +1200 Subject: [PATCH 05/11] feat(canopy): rewrite syncer to materialise PostgresPhysicalReplica CRs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Retire the parallel canopy Job/PVC/Service builders and the reporter alongside them. In their place the syncer now: - lists Namespaces labelled `pgro.bes.au/managed-by=pgro-canopy`; - for each worklist entry, ensures a labelled Namespace, a `canopy-replica` PostgresPhysicalReplica CR (spec built via IntentConfig::to_replica_spec), and a namespace-local canopy-creds Secret carrying the entry's bucket/region/prefix + repo password + dummy AWS keys; - patches `status.canopyDesiredSnapshotId` on the CR whenever canopy offers a snapshot newer than what the CR has seen; - tears down Namespaces (cascades everything inside) that no longer have a matching worklist entry. Restore Job creation, Deployment, verification, and switchover now go through the same CR machinery legacy replicas use. RestoreVerification POSTs to canopy will land as a notification target in a follow-up commit — the /api/v1/canopy-stats callback route the proxy sidecar posts to is unchanged. --- src/bin/operator.rs | 10 +- src/context.rs | 9 +- src/controllers/canopy.rs | 652 ++++++++++----------------- src/controllers/canopy/builders.rs | 700 ----------------------------- src/controllers/canopy/reporter.rs | 608 ------------------------- 5 files changed, 232 insertions(+), 1747 deletions(-) delete mode 100644 src/controllers/canopy/builders.rs delete mode 100644 src/controllers/canopy/reporter.rs diff --git a/src/bin/operator.rs b/src/bin/operator.rs index f6a69f3..80c1685 100644 --- a/src/bin/operator.rs +++ b/src/bin/operator.rs @@ -22,8 +22,8 @@ use tracing::{debug, info, warn}; use postgres_restore_operator::{ canopy::{self, DEFAULT_SOCKS5_PROXY}, context::{ - Context, DEFAULT_CANOPY_PGDATA_PVC_SIZE, DEFAULT_CANOPY_PROXY_IMAGE, - DEFAULT_DEPLOYMENT_READY_TIMEOUT_SECS, DEFAULT_KOPIA_IMAGE, + Context, DEFAULT_CANOPY_PROXY_IMAGE, DEFAULT_DEPLOYMENT_READY_TIMEOUT_SECS, + DEFAULT_KOPIA_IMAGE, }, controllers::{self, canopy::intent::SUPPORTED as PGRO_SUPPORTED_INTENTS}, types::{PostgresPhysicalReplica, PostgresPhysicalRestore}, @@ -214,8 +214,6 @@ async fn main() -> anyhow::Result<()> { }); ctx.canopy_proxy_image = std::env::var("CANOPY_PROXY_IMAGE") .unwrap_or_else(|_| DEFAULT_CANOPY_PROXY_IMAGE.to_string()); - ctx.canopy_pgdata_pvc_size = std::env::var("CANOPY_PGDATA_PVC_SIZE") - .unwrap_or_else(|_| DEFAULT_CANOPY_PGDATA_PVC_SIZE.to_string()); ctx.canopy_broker_base_url = if let Ok(url) = std::env::var("CANOPY_BROKER_BASE_URL") { url } else if let Ok(svc) = std::env::var("OPERATOR_SERVICE_NAME") { @@ -558,8 +556,8 @@ async fn post_cache_pressure( } /// Accept the canopy-proxy sidecar's final TrafficStats POST on shutdown. -/// The body is opaque JSON — the reporter deserializes it when building -/// the RestoreVerification. +/// The body is opaque JSON — the canopy notification target deserializes +/// it when building the RestoreVerification. async fn post_canopy_stats( State(state): State, Path((namespace, job)): Path<(String, String)>, diff --git a/src/context.rs b/src/context.rs index 241bcda..8568f14 100644 --- a/src/context.rs +++ b/src/context.rs @@ -17,7 +17,6 @@ pub const DEFAULT_DEPLOYMENT_READY_TIMEOUT_SECS: u64 = 30 * 60; /// sidecar to a different tag from the operator itself. pub const DEFAULT_CANOPY_PROXY_IMAGE: &str = "ghcr.io/beyondessential/postgres-restore-operator:latest"; -pub const DEFAULT_CANOPY_PGDATA_PVC_SIZE: &str = "20Gi"; pub struct Context { pub client: Client, @@ -39,17 +38,14 @@ pub struct Context { /// Image reference for the pgro-canopy-proxy sidecar container. Set /// by the operator startup from `CANOPY_PROXY_IMAGE`. pub canopy_proxy_image: String, - /// Default pgdata PVC size used when the operator provisions a - /// canopy-backed replica (per intent). - pub canopy_pgdata_pvc_size: String, /// In-memory store for snapshot-list results POSTed by jobs. pub snapshot_results: Arc, /// In-memory store for schema migration results POSTed by jobs. pub schema_migration_results: Arc, /// In-memory store for canopy-proxy sidecar TrafficStats keyed by /// `{namespace}/{job}`. Written on sidecar exit via the operator's - /// `/api/v1/canopy-stats/...` callback; read by the reporter when - /// building `RestoreVerification.s3_*_bytes`. + /// `/api/v1/canopy-stats/...` callback; read by the canopy + /// notification target when building `RestoreVerification.s3_*_bytes`. pub canopy_stats: Arc, /// Base URL the operator is reachable at from within the cluster, /// e.g. `http://postgres-restore-operator.pgro-system.svc:8080`. @@ -89,7 +85,6 @@ impl Context { canopy: None, canopy_broker_base_url: String::new(), canopy_proxy_image: DEFAULT_CANOPY_PROXY_IMAGE.to_string(), - canopy_pgdata_pvc_size: DEFAULT_CANOPY_PGDATA_PVC_SIZE.to_string(), snapshot_results: Arc::new(CallbackStore::default()), schema_migration_results: Arc::new(CallbackStore::default()), canopy_stats: Arc::new(CallbackStore::default()), diff --git a/src/controllers/canopy.rs b/src/controllers/canopy.rs index 46e3a0e..bd932c2 100644 --- a/src/controllers/canopy.rs +++ b/src/controllers/canopy.rs @@ -1,20 +1,18 @@ //! Canopy worklist syncer — pgro's third top-level controller. //! //! Ticks periodically (default 30s, jittered ±20%), fetches -//! `GET /restore-worklist`, discovers the pgro-managed Namespaces already -//! in the cluster, and reconciles the diff by provisioning / refreshing / -//! tearing down per-replica Namespaces. Unlike the `replica` and `restore` -//! controllers this one is **not** CRD-watched: cluster state (labelled -//! Namespaces + annotations) is the runtime model, canopy's worklist is -//! the spec, and there is no intermediate CR. +//! `GET /restore-worklist`, and reconciles the diff into pgro's own +//! `PostgresPhysicalReplica` CRs. Each entry lives in its own labelled +//! Namespace and carries a canopy-creds Secret + a +//! `PostgresPhysicalReplica` CR with `spec.canopySource` set. The replica +//! and restore controllers pick up from there. //! -//! Actual Job creation for the restore itself is delegated to the Job -//! builder (see step 7 in the integration spec); this module only writes -//! Namespaces and their labels/annotations, leaving `restore-state` -//! transitions for the follow-up commits. +//! This module owns the "canopy is desired-state, pgro reconciles" edge +//! only. Everything downstream — Job creation, Deployment, verification — +//! goes through the same CR machinery as legacy replicas. use std::{ - collections::{HashMap, HashSet}, + collections::{BTreeMap, HashSet}, sync::Arc, time::Duration, }; @@ -22,29 +20,27 @@ use std::{ use bestool_canopy::WorklistEntry; use futures::stream::{self, StreamExt}; use k8s_openapi::{ - api::core::v1::Namespace, + ByteString, + api::core::v1::{Namespace, Secret}, apimachinery::pkg::apis::meta::v1::{ObjectMeta, Time}, }; use kube::{ Api, ResourceExt, - api::{DeleteParams, ListParams, PostParams}, + api::{DeleteParams, ListParams, Patch, PatchParams, PostParams}, }; use rand::RngExt; use sha2::{Digest, Sha256}; use tracing::{debug, error, info, warn}; use uuid::Uuid; -use crate::{context::Context, error::Result}; +use crate::{ + context::Context, + controllers::canopy::intent::{IntentConfig, config_for}, + error::Result, + types::{PostgresPhysicalReplica, PostgresPhysicalReplicaSpec}, +}; -mod builders; pub mod intent; -mod reporter; -pub use builders::{ - CanopyRestoreJobConfig, KOPIA_JOB_NAME, PGDATA_PVC_NAME, POSTGRES_DEPLOYMENT_NAME, - POSTGRES_SERVICE_NAME, PROXY_SIDECAR_POD_LABEL, PostgresDeploymentConfig, - build_canopy_postgres_deployment, build_canopy_postgres_service, build_canopy_restore_job, - build_pgdata_pvc, -}; /// How many per-entry reconciliations run concurrently within one tick. /// Keeps the k8s apiserver from being hit by a stampede when the worklist @@ -54,13 +50,7 @@ const RECONCILE_CONCURRENCY: usize = 8; /// Jitter multiplier applied to the reconcile interval each tick (±20%). const JITTER_RATIO: f64 = 0.2; -/// Labels applied to canopy-managed Namespaces. -/// -/// Labels are the discovery key: `LIST Namespaces -/// label=pgro.bes.au/managed-by=pgro-canopy` returns every canopy-backed -/// replica in one call. They also carry the immutable identity of the -/// replica (declaration id, group, server, type, intent) — mutable -/// runtime state lives in [`annotations`]. +/// Labels applied to canopy-managed Namespaces and CRs. pub mod labels { pub const MANAGED_BY: &str = "pgro.bes.au/managed-by"; pub const MANAGED_BY_VALUE: &str = "pgro-canopy"; @@ -71,65 +61,9 @@ pub mod labels { pub const INTENT: &str = "pgro.bes.au/intent"; } -/// Annotations on canopy-managed Namespaces — mutable per-replica state. -pub mod annotations { - /// The snapshot canopy wants restored. Populated on every successful - /// worklist sync; compared against `LAST_RESTORED_SNAPSHOT_ID` to - /// detect the "newer snapshot available" refresh trigger. - pub const DESIRED_SNAPSHOT_ID: &str = "pgro.bes.au/desired-snapshot-id"; - pub const DESIRED_SNAPSHOT_AT: &str = "pgro.bes.au/desired-snapshot-at"; - pub const LAST_RESTORED_SNAPSHOT_ID: &str = "pgro.bes.au/last-restored-snapshot-id"; - pub const LAST_RESTORED_AT: &str = "pgro.bes.au/last-restored-at"; - pub const RESTORE_STATE: &str = "pgro.bes.au/restore-state"; - pub const LAST_VERIFICATION_REPORTED_AT: &str = "pgro.bes.au/last-verification-reported-at"; - pub const LAST_VERIFICATION_ERROR: &str = "pgro.bes.au/last-verification-error"; - /// Operator escape hatch: setting this annotation to any value triggers - /// a refresh on the next tick and is cleared once the refresh Job is - /// spawned. - pub const FORCE_REFRESH: &str = "pgro.bes.au/force-refresh"; -} - -/// Values for the `RESTORE_STATE` annotation. -pub mod restore_state { - pub const PENDING: &str = "pending"; - pub const RESTORING: &str = "restoring"; - pub const ACTIVE: &str = "active"; - pub const FAILED: &str = "failed"; - pub const TERMINATING: &str = "terminating"; -} - -/// The reconciliation decision for one (worklist entry, current namespace) pair. -#[derive(Debug, PartialEq, Eq)] -pub enum Action { - /// Worklist has an entry, no matching namespace exists — create one. - Provision, - /// Both present, refresh needed for the given reason. - Refresh(RefreshReason), - /// Namespace exists but worklist has no entry — tear down. - Teardown, - /// Both present, in sync, healthy — nothing to do this tick. - NoOp, -} - -#[derive(Debug, PartialEq, Eq)] -pub enum RefreshReason { - /// Canopy is offering a snapshot newer than what we last restored. - NewerSnapshot, - /// `last-restored-at` exceeded the declaration's `freshness_seconds`. - FreshnessExpired, - /// Operator set the `force-refresh` annotation. - Forced, -} - /// Compute the k8s Namespace name for a worklist entry. /// /// Format: `-<8-hex(SHA-256(replica_id || server_id))>`. -/// -/// The slug is derived from the operator-set declaration name in canopy so -/// namespaces are human-recognisable; the 8-hex disambiguator covers the -/// case where a group-wide declaration (`server_id=NULL` in canopy) -/// expands to one worklist entry per live server in the group — all -/// carrying the same `replica_id` and `name`, only `server_id` differs. pub fn namespace_name_for(entry: &WorklistEntry) -> String { format!( "{}-{}", @@ -179,8 +113,7 @@ fn slug(s: &str) -> String { } /// Apply ±20% jitter to a Duration. Scopes the (non-Send) thread rng so the -/// caller can `.await` after receiving the result — `run_forever` needs -/// this since ThreadRng can't be held across an `await`. +/// caller can `.await` after receiving the result. fn jittered(base: Duration) -> Duration { let mut rng = rand::rng(); let jitter = rng.random_range(-JITTER_RATIO..JITTER_RATIO); @@ -188,10 +121,6 @@ fn jittered(base: Duration) -> Duration { } /// 8-hex-char disambiguator from SHA-256 of `replica_id || server_id`. -/// -/// Not cryptographic — just a stable, DNS-safe way to keep namespaces -/// for different (replica, server) pairs from colliding when they share -/// a `name` (group-wide declaration expanded across servers). fn short_hash(replica_id: Uuid, server_id: Uuid) -> String { let mut hasher = Sha256::new(); hasher.update(replica_id.as_bytes()); @@ -204,92 +133,9 @@ fn short_hash(replica_id: Uuid, server_id: Uuid) -> String { out } -/// Diff the worklist against the discovered Namespaces and produce one -/// [`Action`] per (namespace, entry) pair. Entries and namespaces are -/// matched by `pgro.bes.au/declaration-id` label / `WorklistEntry.replica_id`; -/// unmatched entries produce a `Provision`, unmatched namespaces produce a -/// `Teardown`. Refresh vs NoOp decisions live in `evaluate_existing`. -/// -/// Pure function; testable without a cluster. -pub fn diff(entries: &[WorklistEntry], namespaces: &[Namespace]) -> Vec<(String, Action)> { - let mut by_replica: HashMap> = HashMap::new(); - for ns in namespaces { - if let Some(replica) = ns - .labels() - .get(labels::DECLARATION_ID) - .and_then(|s| Uuid::parse_str(s).ok()) - { - by_replica.entry(replica).or_default().push(ns.clone()); - } - } - - let mut seen_replicas: HashSet = HashSet::new(); - let mut actions = Vec::new(); - - for entry in entries { - let target_name = namespace_name_for(entry); - let matched = by_replica - .get(&entry.replica_id) - .and_then(|nss| nss.iter().find(|ns| ns.name_any() == target_name).cloned()); - match matched { - Some(existing) => { - let action = evaluate_existing(entry, &existing); - actions.push((target_name, action)); - } - None => actions.push((target_name, Action::Provision)), - } - seen_replicas.insert(entry.replica_id); - } - - // Namespaces with no matching worklist entry → teardown. Match by the - // full derived name so a namespace lingering after a declaration was - // deleted and re-created with the same id but a different server won't - // be reused mid-flight. - for ns in namespaces { - let name = ns.name_any(); - let already = actions.iter().any(|(n, _)| n == &name); - if !already { - actions.push((name, Action::Teardown)); - } - } - - actions -} - -/// Refresh decision for an existing namespace vs its worklist entry. Pure. -fn evaluate_existing(entry: &WorklistEntry, ns: &Namespace) -> Action { - let annos = ns.annotations(); - if annos.get(annotations::FORCE_REFRESH).is_some() { - return Action::Refresh(RefreshReason::Forced); - } - let last_restored = annos.get(annotations::LAST_RESTORED_SNAPSHOT_ID); - if let (Some(desired), Some(last)) = (entry.snapshot_id.as_ref(), last_restored) - && desired != last - { - return Action::Refresh(RefreshReason::NewerSnapshot); - } - if entry.snapshot_id.is_some() && last_restored.is_none() { - // Namespace exists but never completed a restore — treat as still - // in the initial provisioning attempt. The provision path will - // re-observe the Job on the next tick. - return Action::NoOp; - } - if let (Some(fresh_secs), Some(last_at_str)) = ( - entry.freshness_seconds, - annos.get(annotations::LAST_RESTORED_AT), - ) && fresh_secs > 0 - && let Ok(last_at) = last_at_str.parse::() - { - let elapsed_secs = jiff::Timestamp::now() - .duration_since(last_at) - .as_secs() - .max(0); - if (elapsed_secs as u64) > (fresh_secs as u64) { - return Action::Refresh(RefreshReason::FreshnessExpired); - } - } - Action::NoOp -} +/// Name of the canopy-owned CR inside each per-replica Namespace. Fixed — +/// each namespace holds exactly one canopy-managed CR. +pub const CR_NAME: &str = "canopy-replica"; /// The syncer controller. Holds a shared `Context`; runs a periodic tick /// via [`Self::run_forever`], or one-shot via [`Self::tick`] for tests. @@ -318,8 +164,9 @@ impl CanopyController { } } - /// One reconciliation pass. Fetches the worklist, lists namespaces, - /// dispatches per-entry actions concurrently. + /// One reconciliation pass. Fetches the worklist, lists managed + /// Namespaces, provisions CRs where missing / patches desired-snapshot + /// where changed / tears down orphans. pub async fn tick(&self) -> Result<()> { let Some(canopy) = self.ctx.canopy.as_ref() else { return Ok(()); @@ -335,86 +182,95 @@ impl CanopyController { )); let namespaces = ns_api.list(¶ms).await?.items; - let actions = diff(&entries, &namespaces); + let entry_ns_names: HashSet = entries.iter().map(namespace_name_for).collect(); + info!( worklist_entries = entries.len(), existing_namespaces = namespaces.len(), - actions = actions.len(), "canopy worklist syncer tick" ); - let entries_by_replica: HashMap = - entries.iter().map(|e| (e.replica_id, e)).collect(); - let ns_by_name: HashMap = - namespaces.iter().map(|n| (n.name_any(), n)).collect(); - + // Provision / refresh each entry concurrently. let ctx = self.ctx.clone(); - stream::iter(actions) - .for_each_concurrent(RECONCILE_CONCURRENCY, |(ns_name, action)| { + stream::iter(entries) + .for_each_concurrent(RECONCILE_CONCURRENCY, |entry| { let ctx = ctx.clone(); - let entry = entries_by_replica - .iter() - .find_map(|(_, e)| { - if namespace_name_for(e) == ns_name { - Some(*e) - } else { - None - } - }) - .cloned(); - let existing = ns_by_name.get(&ns_name).map(|n| (*n).clone()); async move { - if let Err(err) = dispatch(&ctx, &ns_name, action, entry, existing).await { + let ns_name = namespace_name_for(&entry); + if let Err(err) = reconcile_entry(&ctx, &ns_name, &entry).await { warn!( namespace = %ns_name, + replica_id = %entry.replica_id, error = %err, - "canopy per-namespace reconciliation failed" + "canopy reconcile_entry failed" ); } } }) .await; - // Re-list namespaces so the reporter observes any state we just - // wrote. Cheap — one apiserver call for the pgro-canopy label. - let namespaces = ns_api.list(¶ms).await?.items; - reporter::observe_and_report(&self.ctx, &namespaces).await; + // Teardown namespaces with no matching worklist entry. + let orphans: Vec = namespaces + .into_iter() + .filter(|ns| !entry_ns_names.contains(&ns.name_any())) + .collect(); + let ctx = self.ctx.clone(); + stream::iter(orphans) + .for_each_concurrent(RECONCILE_CONCURRENCY, |ns| { + let ctx = ctx.clone(); + async move { + let ns_name = ns.name_any(); + if let Err(err) = teardown(&ctx, &ns_name).await { + warn!( + namespace = %ns_name, + error = %err, + "canopy teardown failed" + ); + } + } + }) + .await; Ok(()) } } -async fn dispatch( - ctx: &Context, - ns_name: &str, - action: Action, - entry: Option, - existing: Option, -) -> Result<()> { - match action { - Action::Provision => { - let Some(entry) = entry else { - return Ok(()); // shouldn't happen; diff produces Provision only with an entry - }; - provision(ctx, ns_name, &entry).await - } - Action::Refresh(reason) => { - let Some(entry) = entry else { return Ok(()) }; - refresh(ctx, ns_name, &entry, reason, existing).await - } - Action::Teardown => teardown(ctx, ns_name, existing).await, - Action::NoOp => Ok(()), - } -} +/// Ensure the namespace, canopy-creds Secret, and PostgresPhysicalReplica +/// CR exist and reflect the worklist entry. Patches +/// `status.canopyDesiredSnapshotId` when canopy is offering a snapshot the +/// CR hasn't seen yet. +async fn reconcile_entry(ctx: &Context, ns_name: &str, entry: &WorklistEntry) -> Result<()> { + let Some(intent) = config_for(&entry.intent) else { + warn!( + namespace = %ns_name, + intent = %entry.intent, + "worklist entry has unsupported intent — skipping" + ); + return Ok(()); + }; + + ensure_namespace(ctx, ns_name, entry).await?; + + // Fetch the repo password (canopy's restore-credentials endpoint carries + // it alongside chained STS creds; the sidecar refreshes the STS half + // out-of-band). Bucket/region/prefix come from the worklist entry + // directly. + let Some(canopy) = ctx.canopy.as_ref() else { + return Ok(()); + }; + let creds = canopy + .restore_credentials(&entry.r#type.to_string(), entry.group_id) + .await?; -/// Create the Namespace for a new worklist entry with immutable labels + the -/// initial mutable annotations. Follow-up commits wire up the PVC / Deployment / -/// restore Job (step 7 in the integration spec); this tick just records the -/// intent by creating the Namespace and marking `restore-state=pending`. -async fn provision(ctx: &Context, ns_name: &str, entry: &WorklistEntry) -> Result<()> { - info!(namespace = %ns_name, replica_id = %entry.replica_id, "canopy: provisioning replica namespace"); + ensure_canopy_creds_secret(ctx, ns_name, entry, &creds.repo_password.0).await?; + ensure_replica_cr(ctx, ns_name, entry, &intent).await?; + set_desired_snapshot(ctx, ns_name, entry).await?; - let mut labels_map = std::collections::BTreeMap::new(); + Ok(()) +} + +async fn ensure_namespace(ctx: &Context, ns_name: &str, entry: &WorklistEntry) -> Result<()> { + let mut labels_map = BTreeMap::new(); labels_map.insert(labels::MANAGED_BY.into(), labels::MANAGED_BY_VALUE.into()); labels_map.insert(labels::DECLARATION_ID.into(), entry.replica_id.to_string()); labels_map.insert(labels::GROUP.into(), entry.group_id.to_string()); @@ -422,23 +278,10 @@ async fn provision(ctx: &Context, ns_name: &str, entry: &WorklistEntry) -> Resul labels_map.insert(labels::TYPE.into(), entry.r#type.to_string()); labels_map.insert(labels::INTENT.into(), entry.intent.to_string()); - let mut annos = std::collections::BTreeMap::new(); - annos.insert( - annotations::RESTORE_STATE.into(), - restore_state::PENDING.into(), - ); - if let Some(sid) = &entry.snapshot_id { - annos.insert(annotations::DESIRED_SNAPSHOT_ID.into(), sid.clone()); - } - if let Some(sat) = &entry.snapshot_at { - annos.insert(annotations::DESIRED_SNAPSHOT_AT.into(), sat.clone()); - } - let ns = Namespace { metadata: ObjectMeta { name: Some(ns_name.to_string()), labels: Some(labels_map), - annotations: Some(annos), ..Default::default() }, ..Default::default() @@ -452,110 +295,163 @@ async fn provision(ctx: &Context, ns_name: &str, entry: &WorklistEntry) -> Resul } Err(err) => return Err(err.into()), } - - // If canopy hasn't yet issued a snapshot for this (server, type) the - // entry has no snapshot_id — nothing to restore this tick. The next - // worklist tick will re-check. - let Some(snapshot_id) = entry.snapshot_id.as_deref() else { - info!(namespace = %ns_name, "canopy: entry has no snapshot yet, waiting"); - return Ok(()); - }; - - // The repo password comes from canopy's restore-credentials response - // (bundled with the STS creds). Fetch it once here; the proxy sidecar - // fetches STS creds itself on each refresh via the broker. - let Some(canopy) = ctx.canopy.as_ref() else { - return Ok(()); - }; - let creds = canopy - .restore_credentials(&entry.r#type.to_string(), entry.group_id) - .await?; - - spawn_pvc_and_job(ctx, ns_name, entry, snapshot_id, &creds.repo_password.0).await?; Ok(()) } -async fn spawn_pvc_and_job( +/// Materialise the canopy-creds Secret with the entry's bucket / region / +/// prefix / repo password + dummy AWS keys. Kopia reads these via +/// `env_from_secret` in the restore Job; the proxy sidecar overrides the +/// AWS auth by re-signing upstream with real (refreshed) STS creds. +async fn ensure_canopy_creds_secret( ctx: &Context, ns_name: &str, entry: &WorklistEntry, - snapshot_id: &str, repo_password: &str, ) -> Result<()> { - let pvc = builders::build_pgdata_pvc(ns_name, &ctx.canopy_pgdata_pvc_size); - let pvc_api: Api = - Api::namespaced(ctx.client.clone(), ns_name); - match pvc_api.create(&PostParams::default(), &pvc).await { - Ok(_) => {} - Err(kube::Error::Api(err)) if err.code == 409 => {} - Err(err) => return Err(err.into()), - } + let secret_name = IntentConfig::canopy_creds_secret_name(CR_NAME); - let job_name = format!("restore-{}", short_id(snapshot_id)); - let stats_callback_url = ctx.canopy_stats_callback_url(ns_name, &job_name); - let cfg = builders::CanopyRestoreJobConfig { - entry, - namespace: ns_name, - job_name: &job_name, - kopia_image: &ctx.kopia_image(), - canopy_proxy_image: &ctx.canopy_proxy_image, - broker_base_url: &ctx.canopy_broker_base_url, - stats_callback_url: &stats_callback_url, - snapshot_id, - repo_password, - pgdata_pvc_size: &ctx.canopy_pgdata_pvc_size, + let mut data: BTreeMap = BTreeMap::new(); + data.insert( + "bucket".into(), + ByteString(entry.bucket.clone().into_bytes()), + ); + data.insert( + "region".into(), + ByteString(entry.region.clone().into_bytes()), + ); + data.insert( + "prefix".into(), + ByteString(entry.prefix.clone().into_bytes()), + ); + data.insert( + "repositoryPassword".into(), + ByteString(repo_password.as_bytes().to_vec()), + ); + // Dummy AWS keys — kopia carries these to satisfy its s3 backend arg + // validation, but every request is re-signed by the proxy sidecar + // before it hits real S3. + data.insert( + "accessKeyId".into(), + ByteString(b"PROXY_DUMMY_ACCESS_KEY".to_vec()), + ); + data.insert( + "secretAccessKey".into(), + ByteString(b"PROXY_DUMMY_SECRET_KEY".to_vec()), + ); + + let secret = Secret { + metadata: ObjectMeta { + name: Some(secret_name.clone()), + namespace: Some(ns_name.to_string()), + labels: Some(BTreeMap::from([( + labels::MANAGED_BY.into(), + labels::MANAGED_BY_VALUE.into(), + )])), + ..Default::default() + }, + type_: Some("Opaque".into()), + data: Some(data), + ..Default::default() }; - let job = builders::build_canopy_restore_job(&cfg); - let job_api: Api = - Api::namespaced(ctx.client.clone(), ns_name); - match job_api.create(&PostParams::default(), &job).await { - Ok(_) => { - info!(namespace = %ns_name, job = %job_name, "canopy: created restore Job"); - } - Err(kube::Error::Api(err)) if err.code == 409 => { - debug!(namespace = %ns_name, job = %job_name, "canopy: restore Job already exists"); - } - Err(err) => return Err(err.into()), - } - Ok(()) -} -/// First 8 chars of a snapshot id, DNS-safe. Used as a Job name suffix so -/// the Job for the current desired snapshot is stably named. -fn short_id(s: &str) -> String { - s.chars() - .filter(|c| c.is_ascii_alphanumeric()) - .take(8) - .collect::() - .to_ascii_lowercase() + let api: Api = Api::namespaced(ctx.client.clone(), ns_name); + let mut patch_value = serde_json::to_value(&secret)?; + patch_value["apiVersion"] = serde_json::json!("v1"); + patch_value["kind"] = serde_json::json!("Secret"); + api.patch( + &secret_name, + &PatchParams::apply("postgres-restore-operator").force(), + &Patch::Apply(&patch_value), + ) + .await?; + Ok(()) } -/// Placeholder — refresh flow lands in the follow-up commit alongside the -/// Job builder. For now, just log the reason and update the desired-snapshot -/// annotation so the next tick can pick up if the entry keeps changing. -async fn refresh( +/// Create the PostgresPhysicalReplica CR if it doesn't exist, or re-apply +/// its spec to self-heal drift (canopy-managed CRs aren't user-editable). +async fn ensure_replica_cr( ctx: &Context, ns_name: &str, entry: &WorklistEntry, - reason: RefreshReason, - _existing: Option, + intent: &IntentConfig, ) -> Result<()> { - info!(namespace = %ns_name, replica_id = %entry.replica_id, ?reason, "canopy: refresh needed (Job spawn not yet implemented)"); - let _ = (ctx, ns_name, entry); + let spec = intent.to_replica_spec(entry, Vec::new()); + + let mut labels_map = BTreeMap::new(); + labels_map.insert(labels::MANAGED_BY.into(), labels::MANAGED_BY_VALUE.into()); + labels_map.insert(labels::DECLARATION_ID.into(), entry.replica_id.to_string()); + labels_map.insert(labels::GROUP.into(), entry.group_id.to_string()); + labels_map.insert(labels::SERVER.into(), entry.server_id.to_string()); + labels_map.insert(labels::TYPE.into(), entry.r#type.to_string()); + labels_map.insert(labels::INTENT.into(), entry.intent.to_string()); + + let cr = replica_cr(CR_NAME, ns_name, labels_map, spec); + let api: Api = Api::namespaced(ctx.client.clone(), ns_name); + let mut patch_value = serde_json::to_value(&cr)?; + patch_value["apiVersion"] = serde_json::json!("pgro.bes.au/v1alpha1"); + patch_value["kind"] = serde_json::json!("PostgresPhysicalReplica"); + api.patch( + CR_NAME, + &PatchParams::apply("postgres-restore-operator").force(), + &Patch::Apply(&patch_value), + ) + .await?; Ok(()) } -/// Placeholder — teardown flow (drain, delete children, delete namespace) -/// lands in a follow-up commit. For now, mark the namespace terminating so -/// operators can see it via `kubectl` and delete the namespace directly. -async fn teardown(ctx: &Context, ns_name: &str, existing: Option) -> Result<()> { - info!(namespace = %ns_name, "canopy: worklist no longer covers this namespace; tearing down"); - if existing.is_none() { +fn replica_cr( + name: &str, + namespace: &str, + labels: BTreeMap, + spec: PostgresPhysicalReplicaSpec, +) -> PostgresPhysicalReplica { + PostgresPhysicalReplica { + metadata: ObjectMeta { + name: Some(name.to_string()), + namespace: Some(namespace.to_string()), + labels: Some(labels), + ..Default::default() + }, + spec, + status: None, + } +} + +async fn set_desired_snapshot(ctx: &Context, ns_name: &str, entry: &WorklistEntry) -> Result<()> { + let Some(desired) = entry.snapshot_id.as_deref() else { + return Ok(()); + }; + + let api: Api = Api::namespaced(ctx.client.clone(), ns_name); + let cr = api.get(CR_NAME).await?; + let current = cr + .status + .as_ref() + .and_then(|s| s.canopy_desired_snapshot_id.as_deref()); + if current == Some(desired) { return Ok(()); } + + let patch = serde_json::json!({ + "status": { "canopyDesiredSnapshotId": desired } + }); + api.patch_status( + CR_NAME, + &PatchParams::apply("postgres-restore-operator"), + &Patch::Merge(&patch), + ) + .await?; + info!( + namespace = %ns_name, + snapshot = desired, + "canopy: updated canopyDesiredSnapshotId" + ); + Ok(()) +} + +async fn teardown(ctx: &Context, ns_name: &str) -> Result<()> { + info!(namespace = %ns_name, "canopy: worklist no longer covers this namespace; tearing down"); let api: Api = Api::all(ctx.client.clone()); - // Namespace deletion cascades to everything inside it; k8s handles the - // child cleanup. match api.delete(ns_name, &DeleteParams::default()).await { Ok(_) => Ok(()), Err(kube::Error::Api(err)) if err.code == 404 => Ok(()), @@ -563,8 +459,8 @@ async fn teardown(ctx: &Context, ns_name: &str, existing: Option) -> } } -/// Now-timestamp helper for annotation values. Keeps RFC3339 stringification -/// centralised so tests can rely on the format. +/// Now-timestamp helper for annotation values. Kept as a public helper for +/// integration tests that assert on RFC3339 format. pub fn now_rfc3339() -> String { Time(jiff::Timestamp::now()).0.to_string() } @@ -650,100 +546,4 @@ mod tests { let name = namespace_name_for(&e); assert!(name.len() <= 63, "got {} chars: {name}", name.len()); } - - #[test] - fn diff_missing_namespace_is_provision() { - let e = entry("nauru", Uuid::new_v4(), Uuid::new_v4()); - let actions = diff(std::slice::from_ref(&e), &[]); - assert_eq!(actions.len(), 1); - assert_eq!(actions[0].1, Action::Provision); - } - - #[test] - fn diff_orphan_namespace_is_teardown() { - let ns = Namespace { - metadata: ObjectMeta { - name: Some("orphan".into()), - labels: Some(std::collections::BTreeMap::from([( - labels::MANAGED_BY.into(), - labels::MANAGED_BY_VALUE.into(), - )])), - ..Default::default() - }, - ..Default::default() - }; - let actions = diff(&[], &[ns]); - assert_eq!(actions.len(), 1); - assert_eq!(actions[0].1, Action::Teardown); - } - - #[test] - fn diff_matched_pair_is_noop_when_never_restored_yet() { - let replica = Uuid::new_v4(); - let server = Uuid::new_v4(); - let e = entry("nauru", replica, server); - let ns = Namespace { - metadata: ObjectMeta { - name: Some(namespace_name_for(&e)), - labels: Some(std::collections::BTreeMap::from([ - (labels::MANAGED_BY.into(), labels::MANAGED_BY_VALUE.into()), - (labels::DECLARATION_ID.into(), replica.to_string()), - ])), - ..Default::default() - }, - ..Default::default() - }; - let actions = diff(&[e], &[ns]); - assert_eq!(actions.len(), 1); - assert_eq!(actions[0].1, Action::NoOp); - } - - #[test] - fn diff_newer_snapshot_triggers_refresh() { - let replica = Uuid::new_v4(); - let server = Uuid::new_v4(); - let mut e = entry("nauru", replica, server); - e.snapshot_id = Some("new-snap".into()); - let ns = Namespace { - metadata: ObjectMeta { - name: Some(namespace_name_for(&e)), - labels: Some(std::collections::BTreeMap::from([ - (labels::MANAGED_BY.into(), labels::MANAGED_BY_VALUE.into()), - (labels::DECLARATION_ID.into(), replica.to_string()), - ])), - annotations: Some(std::collections::BTreeMap::from([( - annotations::LAST_RESTORED_SNAPSHOT_ID.into(), - "old-snap".into(), - )])), - ..Default::default() - }, - ..Default::default() - }; - let actions = diff(&[e], &[ns]); - assert_eq!(actions[0].1, Action::Refresh(RefreshReason::NewerSnapshot)); - } - - #[test] - fn diff_forced_refresh_annotation_wins() { - let replica = Uuid::new_v4(); - let server = Uuid::new_v4(); - let e = entry("nauru", replica, server); - let ns = Namespace { - metadata: ObjectMeta { - name: Some(namespace_name_for(&e)), - labels: Some(std::collections::BTreeMap::from([ - (labels::MANAGED_BY.into(), labels::MANAGED_BY_VALUE.into()), - (labels::DECLARATION_ID.into(), replica.to_string()), - ])), - annotations: Some(std::collections::BTreeMap::from([( - annotations::FORCE_REFRESH.into(), - "now".into(), - )])), - ..Default::default() - }, - ..Default::default() - }; - let actions = diff(&[e], &[ns]); - assert_eq!(actions[0].1, Action::Refresh(RefreshReason::Forced)); - } } diff --git a/src/controllers/canopy/builders.rs b/src/controllers/canopy/builders.rs deleted file mode 100644 index 298affc..0000000 --- a/src/controllers/canopy/builders.rs +++ /dev/null @@ -1,700 +0,0 @@ -//! Canopy-path Job + PVC builders. -//! -//! The CRD path (`src/controllers/restore/builders.rs`) has its own -//! `build_restore_job` deeply tied to `PostgresPhysicalReplica` / -//! `PostgresPhysicalRestore` spec fields. The canopy path has no CRDs — it -//! works from a `WorklistEntry` + labelled Namespace. This module holds the -//! canopy-side equivalents, small enough to live independently rather than -//! forcing a shared abstraction across two different data models. - -use std::collections::BTreeMap; - -use bestool_canopy::WorklistEntry; -use k8s_openapi::{ - api::{ - apps::v1::Deployment, - batch::v1::{Job, JobSpec}, - core::v1::{ - Container, EmptyDirVolumeSource, EnvVar, PersistentVolumeClaim, - PersistentVolumeClaimSpec, PodSpec, PodTemplateSpec, ResourceRequirements, Service, - ServicePort, ServiceSpec, Volume, VolumeMount, VolumeResourceRequirements, - }, - }, - apimachinery::pkg::{api::resource::Quantity, util::intstr::IntOrString}, -}; -use kube::api::ObjectMeta; - -use crate::{ - controllers::{canopy::labels, kopia_writable_env}, - kopia::{ProxyConnect, kopia_connect_args_proxy}, -}; - -/// Default kopia image used by the canopy path's restore Job when the -/// operator's dynamic `kopia_image` isn't accessible from the caller. -/// Kept in sync with `context::DEFAULT_KOPIA_IMAGE`; callers should always -/// pass the value from `ctx.kopia_image()`. -pub const KOPIA_JOB_NAME: &str = "restore"; - -/// Name of the canopy-path restore PVC. Namespaces are per-replica so a -/// stable short name is fine. -pub const PGDATA_PVC_NAME: &str = "pgdata"; - -/// Standard label used on the proxy-sidecar-carrying Pod so the operator's -/// broker NetworkPolicy can admit its ingress (see spec §4.3). -pub const PROXY_SIDECAR_POD_LABEL: (&str, &str) = ("pgro.bes.au/proxy-sidecar", "true"); - -/// Config the caller (worklist syncer) supplies to build a canopy restore Job. -pub struct CanopyRestoreJobConfig<'a> { - pub entry: &'a WorklistEntry, - pub namespace: &'a str, - pub job_name: &'a str, - pub kopia_image: &'a str, - pub canopy_proxy_image: &'a str, - /// Base URL the sidecar hits for creds, e.g. - /// `http://postgres-restore-operator.pgro-system.svc:9091`. From - /// `Context::canopy_broker_base_url`. - pub broker_base_url: &'a str, - /// Callback URL the sidecar POSTs its final TrafficStats to on - /// shutdown, from `Context::canopy_stats_callback_url`. - pub stats_callback_url: &'a str, - /// Snapshot the operator wants restored — comes from the worklist - /// entry when Provision or Refresh, but callers pass it explicitly - /// because the syncer may know a more recent value. - pub snapshot_id: &'a str, - pub repo_password: &'a str, - pub pgdata_pvc_size: &'a str, -} - -/// Build the pgdata PVC for a canopy-backed replica. Namespace-scoped, one -/// PVC per replica namespace; sized from `pgdata_pvc_size` (caller decides -/// per intent). -pub fn build_pgdata_pvc(namespace: &str, size: &str) -> PersistentVolumeClaim { - PersistentVolumeClaim { - metadata: ObjectMeta { - name: Some(PGDATA_PVC_NAME.into()), - namespace: Some(namespace.into()), - ..Default::default() - }, - spec: Some(PersistentVolumeClaimSpec { - access_modes: Some(vec!["ReadWriteOnce".into()]), - resources: Some(VolumeResourceRequirements { - requests: Some(BTreeMap::from([("storage".into(), Quantity(size.into()))])), - ..Default::default() - }), - ..Default::default() - }), - ..Default::default() - } -} - -/// Shell wrapper the kopia container runs on the canopy path. Waits for the -/// proxy sidecar to publish its ephemeral port to `/var/run/pgro/proxy-port` -/// (30s timeout — the sidecar writes it as soon as the proxy binds, which -/// is essentially instant), reads the port, invokes kopia against the -/// loopback endpoint, then discovers PGDATA + writes `.postgres-version` -/// so the postgres Deployment can pick the right image. -fn kopia_wrapper_script() -> &'static str { - r#"set -e - -mkdir -p /tmp/kopia/config /tmp/kopia/logs /tmp/kopia/cache - -PORT_FILE="/var/run/pgro/proxy-port" -for _ in $(seq 1 30); do - [ -f "$PORT_FILE" ] && break - sleep 1 -done -if [ ! -f "$PORT_FILE" ]; then - echo "ERROR: canopy-proxy sidecar did not write port file within 30s" >&2 - exit 1 -fi -PROXY_PORT=$(cat "$PORT_FILE") -CANOPY_ENDPOINT="[::1]:${PROXY_PORT}" -echo "kopia connecting via canopy proxy at ${CANOPY_ENDPOINT}" - -# Connect via proxy: kopia talks to [::1] with dummy keys, the proxy holds -# the live STS creds. -CONNECT_ARGS_FILE="/tmp/kopia-connect-args" -cat > "$CONNECT_ARGS_FILE" </dev/null | while read -r f; do - dir=$(dirname "$f") - [ -d "$dir/global" ] && echo "$dir" - done | sort -t/ -k4 -rn | head -1) -fi -if [ -z "$PGDATA_DIR" ]; then - echo "ERROR: no PG_VERSION found in restored data" >&2 - exit 1 -fi -echo "Found PGDATA at: $PGDATA_DIR" -ln -sfn "$PGDATA_DIR" /pgdata/pgdata -rm -f "$PGDATA_DIR/postmaster.pid" -VERSION=$(cat /pgdata/pgdata/PG_VERSION) -echo "Detected postgres version: $VERSION" -echo -n "$VERSION" > /pgdata/.postgres-version -# Mirror the version to the container termination message so the operator -# can read it back after the Pod terminates (via -# read_job_termination_message) and mirror it onto the namespace -# annotation the postgres Deployment reads. -echo -n "$VERSION" > /dev/termination-log -"# -} - -/// Build the canopy-path restore Job. Two containers in one Pod: the kopia -/// container (talks to `[::1]:` via the proxy) and the pgro-published -/// canopy-proxy sidecar. Both share an emptyDir volume for the port-file -/// coordination handshake. -pub fn build_canopy_restore_job(cfg: &CanopyRestoreJobConfig<'_>) -> Job { - // Serialize the kopia connect args once; the wrapper script injects the - // [::1]: endpoint at runtime because the port isn't known until - // the sidecar binds. Placeholder marker: @ENDPOINT@ (replaced by the - // wrapper's sed). - let connect_args = kopia_connect_args_proxy(&ProxyConnect { - endpoint: "@ENDPOINT@", - bucket: &cfg.entry.bucket, - region: &cfg.entry.region, - prefix: &cfg.entry.prefix, - repository_password: cfg.repo_password, - server_id: &cfg.entry.server_id.to_string(), - }); - let connect_args_serialized = connect_args - .into_iter() - .map(|a| shlex_quote(&a)) - .collect::>() - .join(" \\\n "); - - let mut pod_labels = BTreeMap::new(); - pod_labels.insert( - PROXY_SIDECAR_POD_LABEL.0.into(), - PROXY_SIDECAR_POD_LABEL.1.into(), - ); - pod_labels.insert( - labels::DECLARATION_ID.into(), - cfg.entry.replica_id.to_string(), - ); - pod_labels.insert(labels::SERVER.into(), cfg.entry.server_id.to_string()); - pod_labels.insert("pgro.bes.au/job-kind".into(), "canopy-restore".into()); - - let script = kopia_wrapper_script().replace("${CONNECT_ARGS}", &connect_args_serialized); - - let kopia_container = Container { - name: KOPIA_JOB_NAME.into(), - image: Some(cfg.kopia_image.into()), - command: Some(vec!["/bin/sh".into(), "-c".into()]), - args: Some(vec![script]), - env: Some( - [ - vec![ - EnvVar { - name: "SNAPSHOT_ID".into(), - value: Some(cfg.snapshot_id.into()), - ..Default::default() - }, - // `snapshot restore` runs as a separate kopia process from - // `repository connect`, so it must re-open the repository and - // needs the password. Headless containers have no keyring for - // kopia to retrieve the persisted credential from, so without - // this env var it falls back to an interactive prompt and dies - // with "inappropriate ioctl for device". kopia reads - // KOPIA_PASSWORD on every invocation. - EnvVar { - name: "KOPIA_PASSWORD".into(), - value: Some(cfg.repo_password.into()), - ..Default::default() - }, - ], - kopia_writable_env(), - ] - .concat(), - ), - volume_mounts: Some(vec![ - VolumeMount { - name: "pgdata".into(), - mount_path: "/pgdata".into(), - ..Default::default() - }, - VolumeMount { - name: "kopia-cache".into(), - mount_path: "/tmp/kopia".into(), - ..Default::default() - }, - VolumeMount { - name: "proxy-shared".into(), - mount_path: "/var/run/pgro".into(), - ..Default::default() - }, - ]), - resources: Some(ResourceRequirements { - requests: Some(BTreeMap::from([ - ("cpu".into(), Quantity("500m".into())), - ("memory".into(), Quantity("1Gi".into())), - ])), - limits: Some(BTreeMap::from([ - ("cpu".into(), Quantity("2".into())), - ("memory".into(), Quantity("4Gi".into())), - ])), - ..Default::default() - }), - ..Default::default() - }; - - let sidecar_container = Container { - name: "canopy-proxy".into(), - image: Some(cfg.canopy_proxy_image.into()), - // Override the image's default ENTRYPOINT (`operator`) — same image - // ships both binaries; the sidecar container runs `canopy-proxy`. - command: Some(vec!["canopy-proxy".into()]), - env: Some(vec![ - EnvVar { - name: "PGRO_BROKER_URL".into(), - value: Some(cfg.broker_base_url.into()), - ..Default::default() - }, - EnvVar { - name: "PGRO_GROUP".into(), - value: Some(cfg.entry.group_id.to_string()), - ..Default::default() - }, - EnvVar { - name: "PGRO_TYPE".into(), - value: Some(cfg.entry.r#type.to_string()), - ..Default::default() - }, - EnvVar { - name: "PGRO_REGION".into(), - value: Some(cfg.entry.region.clone()), - ..Default::default() - }, - EnvVar { - name: "PGRO_STATS_CALLBACK_URL".into(), - value: Some(cfg.stats_callback_url.into()), - ..Default::default() - }, - ]), - volume_mounts: Some(vec![VolumeMount { - name: "proxy-shared".into(), - mount_path: "/var/run/pgro".into(), - ..Default::default() - }]), - resources: Some(ResourceRequirements { - requests: Some(BTreeMap::from([ - ("cpu".into(), Quantity("50m".into())), - ("memory".into(), Quantity("64Mi".into())), - ])), - limits: Some(BTreeMap::from([ - ("cpu".into(), Quantity("500m".into())), - ("memory".into(), Quantity("256Mi".into())), - ])), - ..Default::default() - }), - ..Default::default() - }; - - Job { - metadata: ObjectMeta { - name: Some(cfg.job_name.into()), - namespace: Some(cfg.namespace.into()), - labels: Some(BTreeMap::from([ - ("pgro.bes.au/job-kind".into(), "canopy-restore".into()), - ( - labels::DECLARATION_ID.into(), - cfg.entry.replica_id.to_string(), - ), - ])), - ..Default::default() - }, - spec: Some(JobSpec { - backoff_limit: Some(3), - // Longer than the CRD path's 2h — the proxy refreshes creds so - // long restores aren't credential-bounded, only reachability-bounded. - active_deadline_seconds: Some(14400), // 4 hours - ttl_seconds_after_finished: Some(600), - template: PodTemplateSpec { - metadata: Some(ObjectMeta { - labels: Some(pod_labels), - ..Default::default() - }), - spec: Some(PodSpec { - restart_policy: Some("Never".into()), - // Termination grace: kopia may need a beat to flush kopia - // caches; the sidecar exits within a second of SIGTERM. - termination_grace_period_seconds: Some(30), - containers: vec![kopia_container, sidecar_container], - volumes: Some(vec![ - Volume { - name: "pgdata".into(), - persistent_volume_claim: Some( - k8s_openapi::api::core::v1::PersistentVolumeClaimVolumeSource { - claim_name: PGDATA_PVC_NAME.into(), - ..Default::default() - }, - ), - ..Default::default() - }, - Volume { - name: "kopia-cache".into(), - empty_dir: Some(EmptyDirVolumeSource::default()), - ..Default::default() - }, - Volume { - name: "proxy-shared".into(), - empty_dir: Some(EmptyDirVolumeSource { - medium: Some("Memory".into()), - ..Default::default() - }), - ..Default::default() - }, - ]), - ..Default::default() - }), - }, - // The syncer polls Job status via labels; no need for a - // Selector-based ownership contract with the operator. - ..Default::default() - }), - ..Default::default() - } -} - -/// Name of the Postgres Deployment + Service the canopy path creates in -/// each replica namespace on restore success. -pub const POSTGRES_DEPLOYMENT_NAME: &str = "postgres"; -pub const POSTGRES_SERVICE_NAME: &str = "postgres"; -pub const POSTGRES_PORT: i32 = 5432; -/// Default analytics role name for canopy-backed replicas. Overridable -/// per-namespace via annotation later; for now, one name for every -/// canopy-backed replica. -pub const CANOPY_ANALYTICS_USERNAME: &str = "analytics"; - -/// Everything the syncer's `ensure_postgres` needs to build the postgres -/// Deployment via the shared [`crate::controllers::restore::builders:: -/// build_postgres_deployment_with`]. Fills the intent-driven defaults -/// (read_only, resources, etc.) and threads the CRD-independent bits. -pub struct PostgresDeploymentConfig<'a> { - pub namespace: &'a str, - pub intent: &'a str, - pub replica_id: &'a str, - pub server_id: &'a str, - /// Detected postgres major version, e.g. `"16"`. Read from the - /// restore Job's termination message (via - /// `read_job_termination_message`). - pub postgres_major_version: &'a str, - pub snapshot_id: &'a str, - pub snapshot_time: &'a str, - /// Analytics credentials Secret reference. The reporter creates the - /// Secret ahead of the Deployment. - pub analytics_secret: &'a k8s_openapi::api::core::v1::SecretReference, - pub analytics_secret_key: &'a str, -} - -/// Build the postgres Deployment for a canopy-backed replica by -/// forwarding to the shared CRD/canopy builder. Both currently-supported -/// intents (`verify`, `analytics`) run the replica read-only; `analytics` -/// gets heavier default resources since it serves queries. Unknown -/// intents fall through to the verify-shaped defaults; canopy won't -/// dispatch them because pgro doesn't register them via -/// `PGRO_SUPPORTED_INTENTS`. -/// -/// All the heavy lifting (pg_resetwal fallback, locale fixing, -/// analytics-user creation, REINDEX-on-startup) comes from the shared -/// builder — see `src/controllers/restore/builders.rs:: -/// build_postgres_deployment_with`. -pub fn build_canopy_postgres_deployment( - cfg: &PostgresDeploymentConfig<'_>, -) -> Result { - use crate::controllers::restore::builders::{ - PostgresDeploymentInputs, build_postgres_deployment_with, - }; - - let read_only = true; - - let (cpu_req, mem_req, cpu_lim, mem_lim, shm) = match cfg.intent { - "analytics" => ("500m", "2Gi", "4", "8Gi", "2Gi"), - _ => ("250m", "512Mi", "2", "2Gi", "512Mi"), - }; - let (shm_size, shared_buffers_mb) = - crate::quantity::compute_shm_and_shared_buffers(&Some(ResourceRequirements { - requests: Some(BTreeMap::from([ - ("cpu".into(), Quantity(cpu_req.into())), - ("memory".into(), Quantity(mem_req.into())), - ])), - limits: Some(BTreeMap::from([ - ("cpu".into(), Quantity(cpu_lim.into())), - ("memory".into(), Quantity(mem_lim.into())), - ])), - ..Default::default() - })); - // The compute helper may pick a shm size smaller than what we want per - // intent; take max of the two. - let shm_size_final = if quantity_ge(&shm_size, &Quantity(shm.into())) { - shm_size - } else { - Quantity(shm.into()) - }; - - let mut labels: BTreeMap = BTreeMap::new(); - labels.insert(labels::MANAGED_BY.into(), labels::MANAGED_BY_VALUE.into()); - labels.insert(labels::DECLARATION_ID.into(), cfg.replica_id.into()); - labels.insert(labels::SERVER.into(), cfg.server_id.into()); - labels.insert(labels::INTENT.into(), cfg.intent.into()); - labels.insert("app.kubernetes.io/name".into(), "postgres".into()); - labels.insert("pgro.bes.au/canopy-replica".into(), "true".into()); - let match_labels: BTreeMap<_, _> = labels - .iter() - .filter(|(k, _)| { - k.as_str() == "app.kubernetes.io/name" || k.as_str() == "pgro.bes.au/canopy-replica" - }) - .map(|(k, v)| (k.clone(), v.clone())) - .collect(); - - let inputs = PostgresDeploymentInputs { - name: POSTGRES_DEPLOYMENT_NAME, - namespace: cfg.namespace, - pvc_name: PGDATA_PVC_NAME, - pg_version: cfg.postgres_major_version, - shm_size: shm_size_final, - shared_buffers_mb, - read_only, - postgres_extra_config: None, - analytics_username: CANOPY_ANALYTICS_USERNAME, - analytics_password_secret: cfg.analytics_secret, - analytics_password_key: cfg.analytics_secret_key, - snapshot_id: cfg.snapshot_id, - snapshot_time: cfg.snapshot_time, - labels, - match_labels, - pod_annotations: None, - // Canopy path uses namespace-cascade deletion for teardown; no - // per-Deployment owner reference. - owner_references: None, - postgres_resources: Some(ResourceRequirements { - requests: Some(BTreeMap::from([ - ("cpu".into(), Quantity(cpu_req.into())), - ("memory".into(), Quantity(mem_req.into())), - ])), - limits: Some(BTreeMap::from([ - ("cpu".into(), Quantity(cpu_lim.into())), - ("memory".into(), Quantity(mem_lim.into())), - ])), - ..Default::default() - }), - affinity: None, - tolerations: Vec::new(), - }; - - build_postgres_deployment_with(&inputs) -} - -/// Rough Quantity comparison for the shm-size floor. Both sides are -/// short mebibyte-suffix strings we generated; if parsing fails we treat -/// LHS as smaller so the intent floor wins (safer default). -fn quantity_ge(a: &Quantity, b: &Quantity) -> bool { - let parse = |q: &Quantity| -> Option { - let s = q.0.trim_end_matches("Mi").trim_end_matches("Gi"); - s.parse::() - .ok() - .map(|n| if q.0.ends_with("Gi") { n * 1024 } else { n }) - }; - match (parse(a), parse(b)) { - (Some(av), Some(bv)) => av >= bv, - _ => false, - } -} - -/// ClusterIP Service exposing the postgres Deployment inside the -/// replica namespace. Selects by the same labels the shared Deployment -/// builder puts on the pod template. -pub fn build_canopy_postgres_service(namespace: &str) -> Service { - let mut selector = BTreeMap::new(); - selector.insert("app.kubernetes.io/name".into(), "postgres".into()); - selector.insert("pgro.bes.au/canopy-replica".into(), "true".into()); - Service { - metadata: ObjectMeta { - name: Some(POSTGRES_SERVICE_NAME.into()), - namespace: Some(namespace.into()), - ..Default::default() - }, - spec: Some(ServiceSpec { - selector: Some(selector), - ports: Some(vec![ServicePort { - name: Some("postgres".into()), - port: POSTGRES_PORT, - target_port: Some(IntOrString::String("postgres".into())), - protocol: Some("TCP".into()), - ..Default::default() - }]), - ..Default::default() - }), - ..Default::default() - } -} - -/// Very small shell quoter for the connect args. kopia args have no `'` or -/// null bytes; the strings are canopy-supplied bucket/region/etc. + our -/// hardcoded dummy keys, so this is defensive rather than needed for -/// correctness. -fn shlex_quote(s: &str) -> String { - if s.is_empty() - || s.chars() - .any(|c| !(c.is_ascii_alphanumeric() || "-_.=/:[]@".contains(c))) - { - // Wrap in single quotes; there is no way for a `'` to appear in the - // kopia arg space we build, so a naive single-quote wrap is safe. - format!("'{s}'") - } else { - s.to_string() - } -} - -#[cfg(test)] -mod tests { - use super::*; - use uuid::Uuid; - - fn worklist_entry() -> WorklistEntry { - serde_json::from_value(serde_json::json!({ - "replica_id": Uuid::new_v4().to_string(), - "group_id": Uuid::new_v4().to_string(), - "server_id": Uuid::new_v4().to_string(), - "type": "tamanu-postgres", - "intent": "verify", - "name": "test", - "snapshot_id": "abc123", - "snapshot_at": "2026-07-01T00:00:00Z", - "storage": "s3", - "bucket": "canopy-test", - "prefix": "", - "region": "ap-southeast-2", - })) - .unwrap() - } - - #[test] - fn build_canopy_restore_job_shape() { - let entry = worklist_entry(); - let cfg = CanopyRestoreJobConfig { - entry: &entry, - namespace: "pgro-r-abc", - job_name: "restore-1", - kopia_image: "kopia/kopia:0.22.3", - canopy_proxy_image: "ghcr.io/beyondessential/postgres-restore-operator:latest", - broker_base_url: "http://postgres-restore-operator.pgro-system.svc:9091", - stats_callback_url: "http://postgres-restore-operator.pgro-system.svc:8080/api/v1/canopy-stats/pgro-r-abc/restore-1", - snapshot_id: "abc123", - repo_password: "supersecret", - pgdata_pvc_size: "10Gi", - }; - let job = build_canopy_restore_job(&cfg); - - let spec = job.spec.as_ref().unwrap(); - let pod_spec = spec.template.spec.as_ref().unwrap(); - assert_eq!(pod_spec.containers.len(), 2); - assert_eq!(pod_spec.containers[0].name, "restore"); - assert_eq!(pod_spec.containers[1].name, "canopy-proxy"); - - // Sidecar gets the broker URL. - let sidecar_env = pod_spec.containers[1].env.as_ref().unwrap(); - assert!(sidecar_env.iter().any(|e| e.name == "PGRO_BROKER_URL" - && e.value.as_deref() - == Some("http://postgres-restore-operator.pgro-system.svc:9091"))); - assert!( - sidecar_env - .iter() - .any(|e| e.name == "PGRO_TYPE" && e.value.as_deref() == Some("tamanu-postgres")) - ); - - // Pod is labeled for the broker NetworkPolicy. - let pod_labels = job - .spec - .as_ref() - .unwrap() - .template - .metadata - .as_ref() - .unwrap() - .labels - .as_ref() - .unwrap(); - assert_eq!( - pod_labels - .get(PROXY_SIDECAR_POD_LABEL.0) - .map(String::as_str), - Some(PROXY_SIDECAR_POD_LABEL.1), - ); - - // Kopia container has the SNAPSHOT_ID env and mounts the shared proxy-port emptyDir. - let kopia_env = pod_spec.containers[0].env.as_ref().unwrap(); - assert!( - kopia_env - .iter() - .any(|e| e.name == "SNAPSHOT_ID" && e.value.as_deref() == Some("abc123")) - ); - // KOPIA_PASSWORD must be on the container env so `snapshot restore` - // (a separate process from `repository connect`) can re-open the - // repository without an interactive password prompt. - assert!( - kopia_env - .iter() - .any(|e| e.name == "KOPIA_PASSWORD" && e.value.as_deref() == Some("supersecret")) - ); - let kopia_mounts = pod_spec.containers[0].volume_mounts.as_ref().unwrap(); - assert!( - kopia_mounts - .iter() - .any(|m| m.name == "proxy-shared" && m.mount_path == "/var/run/pgro") - ); - - // Shell script wraps the connect args and reads the port file. - let script = pod_spec.containers[0].args.as_ref().unwrap()[0].clone(); - assert!(script.contains("/var/run/pgro/proxy-port")); - assert!(script.contains("kopia snapshot restore")); - assert!(script.contains("--disable-tls")); - } - - #[test] - fn shlex_quote_leaves_simple_strings_bare() { - assert_eq!(shlex_quote("abc123-_"), "abc123-_"); - assert_eq!( - shlex_quote("--endpoint=[::1]:1234"), - "--endpoint=[::1]:1234" - ); - } - - #[test] - fn shlex_quote_wraps_specials() { - assert_eq!(shlex_quote("hello world"), "'hello world'"); - assert_eq!(shlex_quote(""), "''"); - } - - #[test] - fn build_pgdata_pvc_has_size_and_rwo() { - let pvc = build_pgdata_pvc("ns", "20Gi"); - let spec = pvc.spec.unwrap(); - assert_eq!(spec.access_modes, Some(vec!["ReadWriteOnce".into()])); - let req = spec.resources.unwrap().requests.unwrap(); - assert_eq!(req.get("storage"), Some(&Quantity("20Gi".into()))); - assert_eq!(pvc.metadata.name, Some(PGDATA_PVC_NAME.into())); - assert_eq!(pvc.metadata.namespace, Some("ns".into())); - } -} diff --git a/src/controllers/canopy/reporter.rs b/src/controllers/canopy/reporter.rs deleted file mode 100644 index 1382ae7..0000000 --- a/src/controllers/canopy/reporter.rs +++ /dev/null @@ -1,608 +0,0 @@ -//! Restore-verification reporter (signal 3). -//! -//! On each syncer tick, observes each managed Namespace's restore Job. When -//! the Job reaches a terminal state we transition the Namespace's -//! `restore-state` annotation and — if it hasn't been done yet — post a -//! `RestoreVerification` to canopy. At-most-once-per-terminal-state gated -//! by the `pgro.bes.au/last-verification-reported-at` annotation; failure -//! to report is recorded in `pgro.bes.au/last-verification-error` and -//! retried on the next tick. -//! -//! On restore-Job success the reporter also ensures the postgres -//! Deployment + Service exist for the namespace, so the RestoreVerification -//! reflects postgres coming up (not just kopia exiting 0). The Job's -//! termination message carries the detected postgres major version; -//! we mirror it to the namespace's `pgro.bes.au/postgres-version` -//! annotation before creating the Deployment. -//! -//! S3 traffic tallies come from the canopy-proxy sidecar's callback POST -//! to `/api/v1/canopy-stats/{ns}/{job}` (see `Context::canopy_stats`) -//! and are included in the RestoreVerification on the next tick. - -use std::collections::BTreeMap; - -use bestool_canopy::{Outcome, RestoreVerification, WorklistEntry}; -use k8s_openapi::{ - ByteString, - api::{ - apps::v1::Deployment, - batch::v1::Job, - core::v1::{Namespace, Secret, Service}, - }, - apimachinery::pkg::apis::meta::v1::ObjectMeta, -}; -use kube::{ - Api, ResourceExt, - api::{ListParams, Patch, PatchParams, PostParams}, -}; -use tracing::{debug, info, warn}; -use uuid::Uuid; - -use crate::{ - context::Context, - controllers::canopy::{annotations, labels, restore_state}, - error::Result, -}; - -/// Kick the reporter for every managed namespace: transition state on -/// terminal Jobs, then emit verification reports where owed. Called from -/// [`super::CanopyController::tick`] after the provision/refresh/teardown -/// dispatch. -pub async fn observe_and_report(ctx: &Context, namespaces: &[Namespace]) { - for ns in namespaces { - if let Err(err) = observe_one(ctx, ns).await { - warn!( - namespace = %ns.name_any(), - error = %err, - "canopy reporter: failed to observe namespace" - ); - } - } -} - -async fn observe_one(ctx: &Context, ns: &Namespace) -> Result<()> { - let ns_name = ns.name_any(); - let annos = ns.annotations(); - let current_state = annos - .get(annotations::RESTORE_STATE) - .map(String::as_str) - .unwrap_or(restore_state::PENDING); - - // If we're in a terminal state and already reported, nothing to do. - let terminal = matches!(current_state, restore_state::ACTIVE | restore_state::FAILED); - let reported = annos.contains_key(annotations::LAST_VERIFICATION_REPORTED_AT); - if terminal && reported { - return Ok(()); - } - - // Otherwise look at the Jobs to figure out where we are. - let job_api: Api = Api::namespaced(ctx.client.clone(), &ns_name); - let jobs = job_api - .list(&ListParams::default().labels("pgro.bes.au/job-kind=canopy-restore")) - .await?; - - let Some(job) = latest_job(&jobs.items) else { - // No Job yet — provision is still pending. Nothing to report. - return Ok(()); - }; - - let (new_state, outcome, error_msg) = classify_job(job); - - // Transition the namespace's state annotation if it changed. - if new_state != current_state { - set_annotations( - ctx, - &ns_name, - &[ - (annotations::RESTORE_STATE, Some(new_state.into())), - match outcome { - Some(Outcome::Success) => ( - annotations::LAST_RESTORED_SNAPSHOT_ID, - annos - .get(annotations::DESIRED_SNAPSHOT_ID) - .cloned() - .or_else(|| snapshot_id_from_job(job).map(String::from)), - ), - _ => (annotations::LAST_RESTORED_SNAPSHOT_ID, None), - }, - ( - annotations::LAST_RESTORED_AT, - if outcome == Some(Outcome::Success) { - Some(super::now_rfc3339()) - } else { - None - }, - ), - ], - ) - .await?; - } - - // On restore-Job success, ensure the postgres Deployment + Service - // exist. This is what actually brings the restored data up as a - // running database — the Job just materialized bytes onto the PVC. - // We do this BEFORE emitting the verification report so replica_healthy - // reflects postgres coming up, not just kopia exiting 0. - if outcome == Some(Outcome::Success) { - // Mirror the restore Job's termination message (the detected - // postgres major version) onto the namespace annotation so the - // Deployment builder can pick the right postgres image. - if annos.get("pgro.bes.au/postgres-version").is_none() - && let Some(v) = crate::controllers::read_job_termination_message( - &ctx.client, - &ns_name, - &job.name_any(), - super::KOPIA_JOB_NAME, - ) - .await - { - let _ = - set_annotations(ctx, &ns_name, &[("pgro.bes.au/postgres-version", Some(v))]).await; - // Re-fetch the namespace so ensure_postgres sees the annotation. - } - let refreshed_ns = Api::::all(ctx.client.clone()) - .get(&ns_name) - .await - .unwrap_or_else(|_| ns.clone()); - if let Err(err) = ensure_postgres(ctx, &refreshed_ns).await { - warn!( - namespace = %ns_name, - error = %err, - "canopy reporter: failed to ensure postgres Deployment/Service" - ); - } - } - - // Only report from a terminal state. - let Some(outcome) = outcome else { - return Ok(()); - }; - - // If we already reported for this terminal round, nothing more to do. - // (The reported-at gate is cleared when the next refresh spawns a new Job; - // for now the annotation is a per-namespace at-most-once — a follow-up can - // key it by snapshot_id if that turns out to be too coarse.) - if annos.contains_key(annotations::LAST_VERIFICATION_REPORTED_AT) { - return Ok(()); - } - - if let Err(err) = send_verification(ctx, ns, outcome, error_msg.as_deref()).await { - warn!( - namespace = %ns_name, - error = %err, - "canopy reporter: restore-verification POST failed; will retry next tick" - ); - set_annotations( - ctx, - &ns_name, - &[(annotations::LAST_VERIFICATION_ERROR, Some(err.to_string()))], - ) - .await?; - } else { - info!(namespace = %ns_name, ?outcome, "canopy reporter: verification reported"); - set_annotations( - ctx, - &ns_name, - &[ - ( - annotations::LAST_VERIFICATION_REPORTED_AT, - Some(super::now_rfc3339()), - ), - (annotations::LAST_VERIFICATION_ERROR, None), - ], - ) - .await?; - } - Ok(()) -} - -/// Ensure the postgres Deployment + Service (and their prereq Secret) -/// exist for a namespace whose restore Job succeeded. Idempotent — 409s -/// on creation are ignored so re-runs don't clobber state. -/// -/// Delegates the actual Deployment shape to the shared builder in -/// `restore/builders.rs` so canopy-backed replicas get the same init -/// treatment as the legacy CRD path (locale rewriting, pg_resetwal -/// fallback, analytics-user provisioning, REINDEX-on-startup, restore -/// metadata in `_pgro.restore_info`). -async fn ensure_postgres(ctx: &Context, ns: &Namespace) -> Result<()> { - let ns_name = ns.name_any(); - - let version = annotations_version(ns).unwrap_or_else(|| "16".to_string()); - let ns_labels = ns.labels(); - let intent = ns_labels - .get("pgro.bes.au/intent") - .map(String::as_str) - .unwrap_or("verify"); - let replica_id = ns_labels - .get("pgro.bes.au/declaration-id") - .map(String::as_str) - .unwrap_or(""); - let server_id = ns_labels - .get("pgro.bes.au/server") - .map(String::as_str) - .unwrap_or(""); - let annos = ns.annotations(); - let snapshot_id = annos - .get(annotations::LAST_RESTORED_SNAPSHOT_ID) - .or_else(|| annos.get(annotations::DESIRED_SNAPSHOT_ID)) - .map(String::as_str) - .unwrap_or(""); - let snapshot_time = annos - .get(annotations::DESIRED_SNAPSHOT_AT) - .map(String::as_str) - .unwrap_or(""); - - // Ensure the analytics-user Secret. The password is randomly - // generated once per namespace; the setup-auth initContainer picks - // it up via env. - let secret_name = "analytics-credentials"; - let secret_key = "password"; - ensure_analytics_secret(ctx, &ns_name, secret_name, secret_key).await?; - - // Ensure the Deployment. - let secret_ref = k8s_openapi::api::core::v1::SecretReference { - name: Some(secret_name.into()), - namespace: Some(ns_name.clone()), - }; - let dep = super::build_canopy_postgres_deployment(&super::PostgresDeploymentConfig { - namespace: &ns_name, - intent, - replica_id, - server_id, - postgres_major_version: &version, - snapshot_id, - snapshot_time, - analytics_secret: &secret_ref, - analytics_secret_key: secret_key, - })?; - let dep_api: Api = Api::namespaced(ctx.client.clone(), &ns_name); - match dep_api.create(&PostParams::default(), &dep).await { - Ok(_) => info!(namespace = %ns_name, "canopy: created postgres Deployment"), - Err(kube::Error::Api(err)) if err.code == 409 => { - debug!(namespace = %ns_name, "canopy: postgres Deployment already exists"); - } - Err(err) => return Err(err.into()), - } - - // Ensure the Service. - let svc = super::build_canopy_postgres_service(&ns_name); - let svc_api: Api = Api::namespaced(ctx.client.clone(), &ns_name); - match svc_api.create(&PostParams::default(), &svc).await { - Ok(_) => info!(namespace = %ns_name, "canopy: created postgres Service"), - Err(kube::Error::Api(err)) if err.code == 409 => { - debug!(namespace = %ns_name, "canopy: postgres Service already exists"); - } - Err(err) => return Err(err.into()), - } - Ok(()) -} - -/// Read the postgres major version stashed on the namespace annotation. -/// The restore Job's script writes /pgdata/.postgres-version onto the -/// PVC; the reporter mirrors it to the namespace annotation the first -/// time it sees it (see `annotate_postgres_version_from_job`). -fn annotations_version(ns: &Namespace) -> Option { - ns.annotations() - .get("pgro.bes.au/postgres-version") - .cloned() -} - -async fn ensure_analytics_secret( - ctx: &Context, - namespace: &str, - name: &str, - key: &str, -) -> Result<()> { - let api: Api = Api::namespaced(ctx.client.clone(), namespace); - if api.get_opt(name).await?.is_some() { - return Ok(()); - } - let password = generate_password(); - let secret = Secret { - metadata: ObjectMeta { - name: Some(name.into()), - namespace: Some(namespace.into()), - ..Default::default() - }, - string_data: None, - data: Some(std::collections::BTreeMap::from([( - key.into(), - ByteString(password.into_bytes()), - )])), - ..Default::default() - }; - match api.create(&PostParams::default(), &secret).await { - Ok(_) => Ok(()), - Err(kube::Error::Api(err)) if err.code == 409 => Ok(()), - Err(err) => Err(err.into()), - } -} - -/// Random alphanumeric password for the postgres superuser. 32 chars is -/// well above brute-force practicality for a network-isolated Service. -fn generate_password() -> String { - use rand::seq::IndexedRandom; - let chars: &[u8] = b"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; - let mut rng = rand::rng(); - (0..32) - .map(|_| *chars.choose(&mut rng).unwrap() as char) - .collect() -} - -/// Pick the most recent restore Job by creation timestamp. Usually there -/// is only one, but a refresh may leave the old one lingering until TTL. -fn latest_job(jobs: &[Job]) -> Option<&Job> { - jobs.iter().max_by_key(|j| { - j.metadata - .creation_timestamp - .as_ref() - .map(|t| t.0) - .unwrap_or(jiff::Timestamp::MIN) - }) -} - -/// Read the snapshot id from the running Job's env — mirrors what the -/// syncer set at Job creation time. -fn snapshot_id_from_job(job: &Job) -> Option<&str> { - let spec = job.spec.as_ref()?; - let containers = spec.template.spec.as_ref()?.containers.iter(); - for c in containers { - if c.name == super::KOPIA_JOB_NAME - && let Some(env) = c.env.as_ref() - { - for e in env { - if e.name == "SNAPSHOT_ID" - && let Some(v) = &e.value - { - return Some(v); - } - } - } - } - None -} - -/// Classify a Job's terminal state into (new_state, outcome, error). -/// `outcome` is `None` while the Job is still running. -fn classify_job(job: &Job) -> (&'static str, Option, Option) { - let status = match job.status.as_ref() { - Some(s) => s, - None => return (restore_state::PENDING, None, None), - }; - if status.succeeded.unwrap_or(0) > 0 { - return (restore_state::ACTIVE, Some(Outcome::Success), None); - } - if status.failed.unwrap_or(0) > 0 { - // Grab the most informative failure condition message if present. - let msg = status - .conditions - .as_ref() - .and_then(|cs| { - cs.iter().find_map(|c| { - if c.type_ == "Failed" && c.status == "True" { - c.message.clone().or_else(|| c.reason.clone()) - } else { - None - } - }) - }) - .unwrap_or_else(|| "restore Job failed".to_string()); - return (restore_state::FAILED, Some(Outcome::Failure), Some(msg)); - } - // Job is running (active > 0) — transition to `restoring`. - if status.active.unwrap_or(0) > 0 { - return (restore_state::RESTORING, None, None); - } - (restore_state::PENDING, None, None) -} - -async fn send_verification( - ctx: &Context, - ns: &Namespace, - outcome: Outcome, - error_msg: Option<&str>, -) -> Result<()> { - let Some(canopy) = ctx.canopy.as_ref() else { - return Err(crate::error::Error::Canopy( - "canopy client not configured".into(), - )); - }; - let ns_labels = ns.labels(); - let annos = ns.annotations(); - - let replica_id = ns_labels - .get(labels::DECLARATION_ID) - .and_then(|s| Uuid::parse_str(s).ok()); - let Some(group) = ns_labels - .get(labels::GROUP) - .and_then(|s| Uuid::parse_str(s).ok()) - else { - return Err(crate::error::Error::Canopy(format!( - "namespace {} missing group label", - ns.name_any() - ))); - }; - let Some(server_id) = ns_labels - .get(labels::SERVER) - .and_then(|s| Uuid::parse_str(s).ok()) - else { - return Err(crate::error::Error::Canopy(format!( - "namespace {} missing server label", - ns.name_any() - ))); - }; - let backup_type = ns_labels - .get(labels::TYPE) - .map(String::as_str) - .unwrap_or(""); - let intent = ns_labels - .get(labels::INTENT) - .map(String::as_str) - .unwrap_or(""); - let snapshot_id = annos - .get(annotations::LAST_RESTORED_SNAPSHOT_ID) - .or_else(|| annos.get(annotations::DESIRED_SNAPSHOT_ID)) - .map(String::as_str); - let replica_healthy = matches!(outcome, Outcome::Success); - - // Stats are POSTed by the sidecar on shutdown to /api/v1/canopy-stats/... - // and land in ctx.canopy_stats keyed by `{namespace}/{job}`. We look - // them up by the terminal Job's name. Missing stats are non-fatal — - // the report goes out without them. - let ns_name = ns.name_any(); - let stats = latest_stats_for_namespace(ctx, ns).await; - - let report = RestoreVerification { - replica_id, - group, - server_id, - r#type: backup_type, - intent, - snapshot_id, - outcome, - error: error_msg, - replica_healthy, - postgres_version: None, - observed_at: jiff::Timestamp::now(), - s3_sent_raw_bytes: stats.as_ref().map(|s| s.sent_raw_bytes as i64), - s3_sent_payload_bytes: stats.as_ref().map(|s| s.sent_payload_bytes as i64), - s3_received_raw_bytes: stats.as_ref().map(|s| s.received_raw_bytes as i64), - s3_received_payload_bytes: stats.as_ref().map(|s| s.received_payload_bytes as i64), - }; - - let result = canopy.restore_verification(&report).await; - drop(ns_name); // silence unused-let warning if the field isn't consumed - result -} - -/// Sidecar-reported stats. Mirrors the shape the sidecar POSTs on -/// shutdown (see `src/bin/canopy_proxy.rs::StatsFile`). -#[derive(Debug, serde::Deserialize)] -struct SidecarStats { - sent_raw_bytes: u64, - sent_payload_bytes: u64, - received_raw_bytes: u64, - received_payload_bytes: u64, -} - -/// Pull the latest sidecar stats for this namespace's most-recent restore -/// Job out of `ctx.canopy_stats`. Non-destructive read via `take` — a -/// second reporter pass wouldn't find them, but we've already stamped -/// `last-verification-reported-at` at that point so this doesn't matter. -async fn latest_stats_for_namespace(ctx: &Context, ns: &Namespace) -> Option { - let job_api: Api = Api::namespaced(ctx.client.clone(), &ns.name_any()); - let jobs = job_api - .list(&ListParams::default().labels("pgro.bes.au/job-kind=canopy-restore")) - .await - .ok()?; - let job = latest_job(&jobs.items)?; - let job_name = job.name_any(); - let raw = ctx.canopy_stats.take(&ns.name_any(), &job_name)?; - match serde_json::from_str::(&raw) { - Ok(s) => Some(s), - Err(err) => { - warn!( - namespace = %ns.name_any(), - job = %job_name, - error = %err, - "canopy reporter: sidecar stats callback body did not parse" - ); - None - } - } -} - -/// Patch the given annotations on a Namespace. `None` values delete the -/// annotation; `Some(v)` sets it. Uses a strategic-merge patch so we don't -/// clobber annotations we don't own. -async fn set_annotations( - ctx: &Context, - ns_name: &str, - updates: &[(&str, Option)], -) -> Result<()> { - let mut annos = serde_json::Map::new(); - for (key, val) in updates { - annos.insert( - (*key).to_string(), - match val { - Some(v) => serde_json::Value::String(v.clone()), - None => serde_json::Value::Null, - }, - ); - } - let patch = serde_json::json!({ - "metadata": { "annotations": annos }, - }); - let api: Api = Api::all(ctx.client.clone()); - api.patch( - ns_name, - &PatchParams::apply("postgres-restore-operator").force(), - &Patch::Merge(&patch), - ) - .await?; - debug!(namespace = %ns_name, ?updates, "canopy reporter: patched annotations"); - Ok(()) -} - -// Silence unused-warning for helpers we plan to use in follow-ups. -#[allow(dead_code)] -fn _ensure_types_used(entry: &WorklistEntry) { - let _ = entry.replica_id; - let _ = BTreeMap::::new(); -} - -#[cfg(test)] -mod tests { - use super::*; - use k8s_openapi::api::batch::v1::JobStatus; - - fn job_with_status(status: JobStatus) -> Job { - Job { - status: Some(status), - ..Default::default() - } - } - - #[test] - fn classify_running_job_is_restoring() { - let job = job_with_status(JobStatus { - active: Some(1), - ..Default::default() - }); - let (state, outcome, _) = classify_job(&job); - assert_eq!(state, restore_state::RESTORING); - assert!(outcome.is_none()); - } - - #[test] - fn classify_succeeded_job_is_active_success() { - let job = job_with_status(JobStatus { - succeeded: Some(1), - ..Default::default() - }); - let (state, outcome, _) = classify_job(&job); - assert_eq!(state, restore_state::ACTIVE); - assert_eq!(outcome, Some(Outcome::Success)); - } - - #[test] - fn classify_failed_job_is_failed_failure() { - let job = job_with_status(JobStatus { - failed: Some(3), - ..Default::default() - }); - let (state, outcome, err) = classify_job(&job); - assert_eq!(state, restore_state::FAILED); - assert_eq!(outcome, Some(Outcome::Failure)); - assert!(err.is_some()); - } - - #[test] - fn classify_pending_job_is_pending() { - let job = job_with_status(JobStatus::default()); - let (state, outcome, _) = classify_job(&job); - assert_eq!(state, restore_state::PENDING); - assert!(outcome.is_none()); - } -} From 5f1de442fcc433138c03f24210a7cf501d34c734 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 04:27:49 +1200 Subject: [PATCH 06/11] feat(canopy): report RestoreVerification to canopy on switchover/failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add `controllers::canopy::verification` — one `report` fn that builds a `bestool_canopy::RestoreVerification` from a PostgresPhysicalReplica + PostgresPhysicalRestore and POSTs it via `ctx.canopy.restore_verification`. - No-op when replica.spec.canopy_source is None or the canopy client isn't configured. - Reads group/server/type/intent/declaration-id from CR labels (set by the canopy syncer at Provision time). - Pulls S3 traffic stats out of `ctx.canopy_stats` (populated by the proxy sidecar's shutdown callback). - Success path fires from the switchover-complete branch in replica.rs; failure path fires from `fail_restore` in restore.rs. --- src/controllers/canopy.rs | 1 + src/controllers/canopy/verification.rs | 142 +++++++++++++++++++++++++ src/controllers/replica.rs | 11 ++ src/controllers/restore.rs | 18 ++++ 4 files changed, 172 insertions(+) create mode 100644 src/controllers/canopy/verification.rs diff --git a/src/controllers/canopy.rs b/src/controllers/canopy.rs index bd932c2..cc1cb0c 100644 --- a/src/controllers/canopy.rs +++ b/src/controllers/canopy.rs @@ -41,6 +41,7 @@ use crate::{ }; pub mod intent; +pub mod verification; /// How many per-entry reconciliations run concurrently within one tick. /// Keeps the k8s apiserver from being hit by a stampede when the worklist diff --git a/src/controllers/canopy/verification.rs b/src/controllers/canopy/verification.rs new file mode 100644 index 0000000..29caf3d --- /dev/null +++ b/src/controllers/canopy/verification.rs @@ -0,0 +1,142 @@ +//! POST `RestoreVerification` back to canopy on restore success / failure. +//! +//! Canopy sees the whole restore cycle as three signals: it dispatches an +//! entry (signal 1), pgro reports the outcome (signal 3), canopy closes the +//! loop. This module owns signal 3 — one function called at each terminal +//! transition (switchover success, restore failure). + +use bestool_canopy::{Outcome, RestoreVerification}; +use jiff::Timestamp; +use kube::ResourceExt; +use serde::Deserialize; +use tracing::{info, warn}; +use uuid::Uuid; + +use crate::{ + context::Context, + controllers::canopy::labels, + types::{PostgresPhysicalReplica, PostgresPhysicalRestore}, +}; + +/// The sidecar posts this JSON shape to `/api/v1/canopy-stats/{ns}/{job}` +/// on exit; kept in sync with `src/bin/canopy_proxy.rs::StatsFile`. +#[derive(Debug, Clone, Copy, Default, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CanopyStats { + pub sent_raw_bytes: u64, + pub sent_payload_bytes: u64, + pub received_raw_bytes: u64, + pub received_payload_bytes: u64, +} + +/// Report the outcome of a canopy-managed restore to canopy. No-op for +/// non-canopy replicas (missing `spec.canopy_source`) and when the canopy +/// client isn't configured. +pub async fn report( + ctx: &Context, + replica: &PostgresPhysicalReplica, + restore: &PostgresPhysicalRestore, + outcome: Outcome, + error: Option<&str>, +) { + if replica.spec.canopy_source.is_none() { + return; + } + let Some(canopy) = ctx.canopy.as_ref() else { + return; + }; + + let labels = replica.labels(); + let Some(group) = labels + .get(labels::GROUP) + .and_then(|s| Uuid::parse_str(s).ok()) + else { + warn!( + replica = %replica.name_any(), + "canopy verification: replica CR missing {} label, skipping report", + labels::GROUP, + ); + return; + }; + let Some(server_id) = labels + .get(labels::SERVER) + .and_then(|s| Uuid::parse_str(s).ok()) + else { + warn!( + replica = %replica.name_any(), + "canopy verification: replica CR missing {} label, skipping report", + labels::SERVER, + ); + return; + }; + let replica_id = labels + .get(labels::DECLARATION_ID) + .and_then(|s| Uuid::parse_str(s).ok()); + let backup_type = labels + .get(labels::TYPE) + .map(String::as_str) + .unwrap_or_default() + .to_string(); + let intent = labels + .get(labels::INTENT) + .map(String::as_str) + .unwrap_or_default() + .to_string(); + + let restore_job = format!("{}-restore", restore.name_any()); + let stats = ctx + .canopy_stats + .take(&replica.namespace().unwrap_or_default(), &restore_job) + .and_then(|raw| match serde_json::from_str::(&raw) { + Ok(s) => Some(s), + Err(err) => { + warn!( + restore = %restore.name_any(), + error = %err, + "canopy verification: failed to parse sidecar stats JSON" + ); + None + } + }) + .unwrap_or_default(); + + let postgres_version = restore + .status + .as_ref() + .and_then(|s| s.postgres_version.as_deref()); + + let replica_healthy = matches!(outcome, Outcome::Success); + + let report = RestoreVerification { + replica_id, + group, + server_id, + r#type: &backup_type, + intent: &intent, + snapshot_id: Some(restore.spec.snapshot.as_str()), + outcome, + error, + replica_healthy, + postgres_version, + observed_at: Timestamp::now(), + s3_sent_raw_bytes: Some(stats.sent_raw_bytes as i64), + s3_sent_payload_bytes: Some(stats.sent_payload_bytes as i64), + s3_received_raw_bytes: Some(stats.received_raw_bytes as i64), + s3_received_payload_bytes: Some(stats.received_payload_bytes as i64), + }; + + match canopy.restore_verification(&report).await { + Ok(()) => info!( + replica = %replica.name_any(), + restore = %restore.name_any(), + ?outcome, + "canopy verification reported" + ), + Err(err) => warn!( + replica = %replica.name_any(), + restore = %restore.name_any(), + error = %err, + "canopy verification report failed" + ), + } +} diff --git a/src/controllers/replica.rs b/src/controllers/replica.rs index fe55386..05b6ded 100644 --- a/src/controllers/replica.rs +++ b/src/controllers/replica.rs @@ -343,6 +343,17 @@ pub async fn reconcile(replica: Arc, ctx: Arc) .send_notifications(client, &ctx.http_client, switching, &ctx.metrics) .await; + // Canopy verification (signal 3) — no-op unless the replica has + // spec.canopy_source. + crate::controllers::canopy::verification::report( + &ctx, + &replica, + switching, + bestool_canopy::Outcome::Success, + None, + ) + .await; + return Ok(Action::requeue(Duration::from_secs(10))); } diff --git a/src/controllers/restore.rs b/src/controllers/restore.rs index 425fe28..6aa6f42 100644 --- a/src/controllers/restore.rs +++ b/src/controllers/restore.rs @@ -296,6 +296,24 @@ async fn fail_restore( warn!(replica = replica_name, error = %e, "failed to publish RestoreFailed event"); } + // Canopy verification (signal 3, failure) — no-op unless the replica + // has spec.canopy_source. The status_patch usually carries an error + // message on `.status.conditions[?type=='RestoreFailed'].message`, but + // we don't have it in a stable spot here; keep the report to + // outcome+snapshot for now. + let restores: Api = Api::namespaced(ctx.client.clone(), namespace); + if let (Ok(restore), Ok(replica)) = (restores.get(name).await, replicas.get(replica_name).await) + { + crate::controllers::canopy::verification::report( + ctx, + &replica, + &restore, + bestool_canopy::Outcome::Failure, + None, + ) + .await; + } + Ok(Action::requeue(Duration::from_secs(300))) } From a4298cd315a92063bc938eed0c249fe696b582f9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 04:30:24 +1200 Subject: [PATCH 07/11] chore(canopy): drop CANOPY_PGDATA_PVC_SIZE from operator.yaml env The parallel canopy path used a fixed pgdata PVC size; the CR path uses per-intent `storage_size_override` instead (see intent module), so `CANOPY_PGDATA_PVC_SIZE` is no longer read anywhere. --- operator.yaml | 3 --- 1 file changed, 3 deletions(-) diff --git a/operator.yaml b/operator.yaml index 82367aa..702acda 100644 --- a/operator.yaml +++ b/operator.yaml @@ -222,9 +222,6 @@ spec: # operators can pin to a specific tag. # - name: CANOPY_PROXY_IMAGE # value: ghcr.io/beyondessential/pgro-canopy-proxy:latest - # Default PVC size for canopy-provisioned pgdata. - # - name: CANOPY_PGDATA_PVC_SIZE - # value: "20Gi" ports: - name: http containerPort: 8080 From 8381961168a233029149b64ce61fd5941e58a7f8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 04:44:47 +1200 Subject: [PATCH 08/11] feat(canopy): run snapshot-list on canopy path to get real sizing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The step-4 shortcut (skip snapshot-list, use canopyDesiredSnapshotId directly with size=0) meant every canopy replica ran with the intent's static storage_size_override regardless of the actual snapshot size, and kopia_content_cache_mb floored out. Instead, run the snapshot-list Job on both paths (canopy adds the proxy-sidecar + shared emptyDir volume, same shape as build_restore_job). On the result side: - legacy: apply snapshot_filter + latest_snapshot as before; - canopy: pick the snapshot whose id matches canopyDesiredSnapshotId, producing a real total_size_bytes for PVC + cache sizing. Surface a canopy-specific condition (SnapshotAvailable=False, CanopyDesiredSnapshotMissing) when the desired snapshot isn't in the kopia listing yet — usually a fresh backup canopy is offering ahead of its indexer catching up. --- src/controllers/replica.rs | 147 ++++++++---------- src/controllers/replica/resources.rs | 213 ++++++++++++++++++++------- src/controllers/replica/tests.rs | 11 +- 3 files changed, 231 insertions(+), 140 deletions(-) diff --git a/src/controllers/replica.rs b/src/controllers/replica.rs index 05b6ded..4d9afec 100644 --- a/src/controllers/replica.rs +++ b/src/controllers/replica.rs @@ -557,18 +557,13 @@ pub async fn reconcile(replica: Arc, ctx: Arc) } } - // Canopy-sourced replicas don't run snapshot-list Jobs — the desired - // snapshot ID is written into `status.canopyDesiredSnapshotId` by the - // canopy worklist syncer. We still allocate the job-name here so the - // downstream `if snapshot_job.is_none()` gate reads uniformly. + // Snapshot-list runs on both paths. Canopy-sourced replicas fetch the + // same snapshot metadata but the result handler filters by + // `status.canopyDesiredSnapshotId` instead of the CR's snapshot_filter. let is_canopy = replica.spec.canopy_source.is_some(); let snapshot_job_name = format!("{name}-snapshot-list"); let jobs: Api = Api::namespaced(client.clone(), &namespace); - let snapshot_job = if is_canopy { - None - } else { - jobs.get_opt(&snapshot_job_name).await? - }; + let snapshot_job = jobs.get_opt(&snapshot_job_name).await?; if let Some(ref job) = snapshot_job { match classify_job(job) { @@ -606,13 +601,28 @@ pub async fn reconcile(replica: Arc, ctx: Arc) match kopia::parse_snapshot_list_output(raw) { Ok(all_snapshots) => { - let filtered = kopia::filter_snapshots( - &all_snapshots, - replica.spec.snapshot_filter.as_ref(), - ); - let latest = kopia::latest_snapshot(&filtered); + // Canopy path: canopy already picked a specific + // snapshot; select the metadata for that ID + // instead of applying snapshot_filter + "latest". + // Legacy path: filter by snapshot_filter, pick + // latest. + let picked = if is_canopy { + let desired = replica + .status + .as_ref() + .and_then(|s| s.canopy_desired_snapshot_id.as_deref()); + desired + .and_then(|id| all_snapshots.iter().find(|s| s.id == id)) + .cloned() + } else { + let filtered = kopia::filter_snapshots( + &all_snapshots, + replica.spec.snapshot_filter.as_ref(), + ); + kopia::latest_snapshot(&filtered).cloned() + }; - if let Some(snap) = latest { + if let Some(ref snap) = picked { let size = snap.total_size_bytes(); replica .update_status_field(client, "latestAvailableSnapshot", &snap.id) @@ -693,6 +703,30 @@ pub async fn reconcile(replica: Arc, ctx: Arc) warn!(replica = name, error = %e, "failed to publish RestoreCreationBlocked event"); } } + } else if is_canopy { + let desired = replica + .status + .as_ref() + .and_then(|s| s.canopy_desired_snapshot_id.as_deref()) + .unwrap_or(""); + warn!( + replica = name, + desired = desired, + "canopy: canopyDesiredSnapshotId not present in kopia snapshot list" + ); + replica + .update_condition( + client, + "SnapshotAvailable", + "False", + "CanopyDesiredSnapshotMissing", + &format!( + "canopyDesiredSnapshotId {desired} was not found in the \ + kopia snapshot list — canopy may be pointing at a \ + snapshot that hasn't been indexed yet" + ), + ) + .await?; } else { warn!( replica = name, @@ -926,86 +960,25 @@ pub async fn reconcile(replica: Arc, ctx: Arc) } drop(queue); - if is_canopy { - // Canopy path: the worklist syncer has already picked a - // snapshot for us. Use `status.canopyDesiredSnapshotId` - // directly and skip the snapshot-list Job entirely. - let desired = replica - .status - .as_ref() - .and_then(|s| s.canopy_desired_snapshot_id.as_ref()); - let Some(desired) = desired else { - debug!( - replica = name, - "canopy-sourced replica has no canopyDesiredSnapshotId yet, waiting for syncer" - ); - replica - .update_condition( - client, - "SnapshotAvailable", - "False", - "CanopyPending", - "Waiting for canopy worklist syncer to set canopyDesiredSnapshotId", - ) - .await?; - return Ok(Action::requeue(Duration::from_secs(30))); - }; - - let current_snapshot_id = active_restore.map(|r| r.spec.snapshot.as_str()); - if current_snapshot_id == Some(desired.as_str()) { - debug!( - replica = name, - snapshot = desired, - "canopy desired snapshot already active, skipping" - ); - } else { - info!( - replica = name, - snapshot = desired, - "canopy desired snapshot changed, creating restore" - ); - // Canopy replicas rely on intent-driven - // `storage_size_override` for sizing; snapshot size is - // not known upfront so we pass 0. The override branch - // in create_restore_for_snapshot then picks the fixed - // PVC size. - let info = SnapshotInfo { - id: desired.clone(), - size: 0, - start_time: String::new(), - }; - let created = replica.create_restore_for_snapshot(client, &info).await?; - if created { - ctx.metrics.restores_started_total.inc(); - if let Err(e) = ctx - .recorder - .publish( - &Event { - type_: EventType::Normal, - reason: "RestoreStarted".into(), - note: Some(format!("Started restore from snapshot {desired}")), - action: "Restore".into(), - secondary: None, - }, - &replica.object_ref(&()), - ) - .await - { - warn!(replica = name, error = %e, "failed to publish RestoreStarted event"); - } - } - } - return Ok(Action::requeue(Duration::from_secs(30))); - } - info!(replica = name, "creating snapshot list job"); let callback_url = ctx.snapshot_callback_url(&namespace, &name); + let stats_callback_url = ctx.canopy_stats_callback_url(&namespace, &snapshot_job_name); + let canopy_proxy = if is_canopy { + Some(crate::controllers::restore::builders::CanopyProxyArgs { + image: &ctx.canopy_proxy_image, + broker_base_url: &ctx.canopy_broker_base_url, + stats_callback_url: &stats_callback_url, + }) + } else { + None + }; let job = build_snapshot_list_job( &replica, &snapshot_job_name, &namespace, &ctx.kopia_image(), &callback_url, + canopy_proxy.as_ref(), )?; jobs.create(&PostParams::default(), &job).await?; return Ok(Action::requeue(Duration::from_secs(10))); diff --git a/src/controllers/replica/resources.rs b/src/controllers/replica/resources.rs index aba73e4..5bd3b55 100644 --- a/src/controllers/replica/resources.rs +++ b/src/controllers/replica/resources.rs @@ -6,8 +6,9 @@ use k8s_openapi::{ api::{ batch::v1::{Job, JobSpec}, core::v1::{ - Container, EnvVar, LocalObjectReference, Pod, PodSpec, PodTemplateSpec, - ResourceRequirements, Secret, Service, ServicePort, ServiceSpec, + Container, EmptyDirVolumeSource, EnvVar, LocalObjectReference, Pod, PodSpec, + PodTemplateSpec, ResourceRequirements, Secret, SecretReference, Service, ServicePort, + ServiceSpec, Volume, VolumeMount, }, }, apimachinery::pkg::{api::resource::Quantity, util::intstr::IntOrString}, @@ -22,8 +23,12 @@ use tracing::{info, warn}; use super::generate_password; use crate::{ - controllers::{env_from_secret, env_from_secret_optional}, + controllers::{ + env_from_secret, env_from_secret_optional, + restore::builders::{CanopyProxyArgs, PROXY_SIDECAR_POD_LABEL}, + }, error::Result, + kopia::KopiaSource, types::*, }; @@ -54,36 +59,60 @@ pub fn build_snapshot_list_job( namespace: &str, kopia_image: &str, callback_url: &str, + canopy_proxy: Option<&CanopyProxyArgs<'_>>, ) -> Result { - // Legacy-path Job builder — `kopia_secret_ref` is guaranteed set by - // the reconciler's credential-source validation before we reach - // here. The canopy path uses a different Job shape (proxy sidecar) - // wired in a follow-up commit; snapshot-list is skipped entirely - // for canopy-sourced replicas. - let kopia_secret = replica - .spec - .kopia_secret_ref - .as_ref() - .expect("build_snapshot_list_job called on legacy path; kopia_secret_ref must be set"); + let source = replica.kopia_source(); + let kopia_secret = SecretReference { + name: Some(source.secret_name().to_string()), + namespace: None, + }; let replica_name = replica.name_any(); let mut env_vars = vec![ - env_from_secret("KOPIA_BUCKET", kopia_secret, "bucket"), - env_from_secret("KOPIA_REGION", kopia_secret, "region"), - env_from_secret("AWS_ACCESS_KEY_ID", kopia_secret, "accessKeyId"), - env_from_secret("AWS_SECRET_ACCESS_KEY", kopia_secret, "secretAccessKey"), - env_from_secret("KOPIA_PASSWORD", kopia_secret, "repositoryPassword"), - env_from_secret_optional("KOPIA_ENDPOINT", kopia_secret, "endpoint"), - env_from_secret_optional("KOPIA_DISABLE_TLS", kopia_secret, "disableTls"), + env_from_secret("KOPIA_BUCKET", &kopia_secret, "bucket"), + env_from_secret("KOPIA_REGION", &kopia_secret, "region"), + env_from_secret("AWS_ACCESS_KEY_ID", &kopia_secret, "accessKeyId"), + env_from_secret("AWS_SECRET_ACCESS_KEY", &kopia_secret, "secretAccessKey"), + env_from_secret("KOPIA_PASSWORD", &kopia_secret, "repositoryPassword"), ]; - + if !source.is_canopy_proxy() { + env_vars.push(env_from_secret_optional( + "KOPIA_ENDPOINT", + &kopia_secret, + "endpoint", + )); + env_vars.push(env_from_secret_optional( + "KOPIA_DISABLE_TLS", + &kopia_secret, + "disableTls", + )); + } env_vars.push(EnvVar { name: "SNAPSHOT_CALLBACK_URL".to_string(), value: Some(callback_url.to_string()), ..Default::default() }); - let script = r#"set -e + let canopy_prelude = if source.is_canopy_proxy() { + r#"PORT_FILE="/var/run/pgro/proxy-port" +for _ in $(seq 1 30); do + [ -f "$PORT_FILE" ] && break + sleep 1 +done +if [ ! -f "$PORT_FILE" ]; then + echo "ERROR: canopy-proxy sidecar did not write port file within 30s" >&2 + exit 1 +fi +export KOPIA_ENDPOINT="[::1]:$(cat "$PORT_FILE")" +export KOPIA_DISABLE_TLS=true +echo "kopia snapshot-list via canopy proxy at ${KOPIA_ENDPOINT}" >&2 + +"# + } else { + "" + }; + + let script_body = r#"set -e ENDPOINT_ARGS="" if [ -n "$KOPIA_ENDPOINT" ]; then @@ -120,18 +149,115 @@ if [ -n "$SNAPSHOT_CALLBACK_URL" ]; then echo "Callback response: HTTP $HTTP_CODE" >&2 fi "#; + let script = format!("{canopy_prelude}{script_body}"); + + let mut kopia_volume_mounts: Vec = Vec::new(); + let mut volumes: Vec = Vec::new(); + let mut pod_labels = BTreeMap::from([ + ("pgro.bes.au/replica".to_string(), replica_name.clone()), + ( + "pgro.bes.au/job-type".to_string(), + "snapshot-list".to_string(), + ), + ]); + let mut containers = vec![Container { + name: "snapshot-list".to_string(), + image: Some(kopia_image.to_string()), + command: Some(vec!["/bin/sh".to_string(), "-c".to_string()]), + args: Some(vec![script]), + env: Some(env_vars), + volume_mounts: Some(kopia_volume_mounts.clone()), + resources: Some(ResourceRequirements { + requests: Some(BTreeMap::from([ + ("cpu".to_string(), Quantity("50m".to_string())), + ("memory".to_string(), Quantity("64Mi".to_string())), + ])), + ..Default::default() + }), + ..Default::default() + }]; + + if let KopiaSource::CanopyProxy { + group, backup_type, .. + } = &source + { + let proxy = canopy_proxy.expect( + "build_snapshot_list_job called with canopy_source but no CanopyProxyArgs; caller \ + must thread proxy config from Context when replica.spec.canopy_source is set", + ); + + kopia_volume_mounts.push(VolumeMount { + name: "proxy-shared".to_string(), + mount_path: "/var/run/pgro".to_string(), + ..Default::default() + }); + containers[0].volume_mounts = Some(kopia_volume_mounts); + + containers.push(Container { + name: "canopy-proxy".to_string(), + image: Some(proxy.image.to_string()), + command: Some(vec!["canopy-proxy".to_string()]), + env: Some(vec![ + EnvVar { + name: "PGRO_BROKER_URL".to_string(), + value: Some(proxy.broker_base_url.to_string()), + ..Default::default() + }, + EnvVar { + name: "PGRO_GROUP".to_string(), + value: Some(group.clone()), + ..Default::default() + }, + EnvVar { + name: "PGRO_TYPE".to_string(), + value: Some(backup_type.clone()), + ..Default::default() + }, + EnvVar { + name: "PGRO_STATS_CALLBACK_URL".to_string(), + value: Some(proxy.stats_callback_url.to_string()), + ..Default::default() + }, + ]), + volume_mounts: Some(vec![VolumeMount { + name: "proxy-shared".to_string(), + mount_path: "/var/run/pgro".to_string(), + ..Default::default() + }]), + resources: Some(ResourceRequirements { + requests: Some(BTreeMap::from([ + ("cpu".to_string(), Quantity("50m".to_string())), + ("memory".to_string(), Quantity("64Mi".to_string())), + ])), + limits: Some(BTreeMap::from([ + ("cpu".to_string(), Quantity("500m".to_string())), + ("memory".to_string(), Quantity("256Mi".to_string())), + ])), + ..Default::default() + }), + ..Default::default() + }); + + volumes.push(Volume { + name: "proxy-shared".to_string(), + empty_dir: Some(EmptyDirVolumeSource { + medium: Some("Memory".to_string()), + ..Default::default() + }), + ..Default::default() + }); + + pod_labels.insert( + PROXY_SIDECAR_POD_LABEL.0.to_string(), + PROXY_SIDECAR_POD_LABEL.1.to_string(), + ); + } Ok(Job { metadata: ObjectMeta { name: Some(job_name.to_string()), namespace: Some(namespace.to_string()), - labels: Some(BTreeMap::from([ - ("pgro.bes.au/replica".to_string(), replica_name.clone()), - ( - "pgro.bes.au/job-type".to_string(), - "snapshot-list".to_string(), - ), - ])), + labels: Some(pod_labels.clone()), owner_references: Some(vec![replica.owner_reference()]), ..Default::default() }, @@ -141,32 +267,17 @@ fi ttl_seconds_after_finished: Some(120), template: PodTemplateSpec { metadata: Some(ObjectMeta { - labels: Some(BTreeMap::from([ - ("pgro.bes.au/replica".to_string(), replica_name), - ( - "pgro.bes.au/job-type".to_string(), - "snapshot-list".to_string(), - ), - ])), + labels: Some(pod_labels), ..Default::default() }), spec: Some(PodSpec { restart_policy: Some("Never".to_string()), - containers: vec![Container { - name: "snapshot-list".to_string(), - image: Some(kopia_image.to_string()), - command: Some(vec!["/bin/sh".to_string(), "-c".to_string()]), - args: Some(vec![script.to_string()]), - env: Some(env_vars), - resources: Some(ResourceRequirements { - requests: Some(BTreeMap::from([ - ("cpu".to_string(), Quantity("50m".to_string())), - ("memory".to_string(), Quantity("64Mi".to_string())), - ])), - ..Default::default() - }), - ..Default::default() - }], + containers, + volumes: if volumes.is_empty() { + None + } else { + Some(volumes) + }, ..Default::default() }), }, diff --git a/src/controllers/replica/tests.rs b/src/controllers/replica/tests.rs index 83debf7..e45bf7e 100644 --- a/src/controllers/replica/tests.rs +++ b/src/controllers/replica/tests.rs @@ -206,8 +206,15 @@ fn snapshot_list_job_rotates_kopia_logs() { status: None, }; - let job = build_snapshot_list_job(&replica, "test-snap", "default", "kopia:latest", "http://x") - .expect("job builds"); + let job = build_snapshot_list_job( + &replica, + "test-snap", + "default", + "kopia:latest", + "http://x", + None, + ) + .expect("job builds"); let script = job.spec.unwrap().template.spec.unwrap().containers[0] .args .as_ref() From f72003f555f4e41c988e48422123607eba90edd5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 04:47:39 +1200 Subject: [PATCH 09/11] feat(canopy): forward fail_restore reason to RestoreVerification.error fail_restore now takes a short reason string. Each call site supplies what actually went wrong (kopia Job failed after backoff, version detection Job failed, deployment not Ready within timeout, etc.), and the canopy verification target passes it to canopy in the RestoreVerification's `error` field. Operators see the reason on canopy's UI without having to trawl the operator's k8s events. --- src/controllers/restore.rs | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/src/controllers/restore.rs b/src/controllers/restore.rs index 6aa6f42..7ce2daf 100644 --- a/src/controllers/restore.rs +++ b/src/controllers/restore.rs @@ -224,6 +224,7 @@ async fn fail_restore( name: &str, replica_name: &str, status_patch: serde_json::Value, + error: &str, ) -> Result { update_restore_status(&ctx.client, namespace, name, status_patch).await?; @@ -297,10 +298,8 @@ async fn fail_restore( } // Canopy verification (signal 3, failure) — no-op unless the replica - // has spec.canopy_source. The status_patch usually carries an error - // message on `.status.conditions[?type=='RestoreFailed'].message`, but - // we don't have it in a stable spot here; keep the report to - // outcome+snapshot for now. + // has spec.canopy_source. The caller passes a short reason so + // operators see it on canopy's UI without having to trawl k8s events. let restores: Api = Api::namespaced(ctx.client.clone(), namespace); if let (Ok(restore), Ok(replica)) = (restores.get(name).await, replicas.get(replica_name).await) { @@ -309,7 +308,7 @@ async fn fail_restore( &replica, &restore, bestool_canopy::Outcome::Failure, - None, + Some(error), ) .await; } @@ -592,6 +591,7 @@ async fn reconcile_restoring( "phase": "Failed", }, }), + "kopia restore Job failed after backoff exhausted", ) .await; } @@ -819,6 +819,7 @@ async fn reconcile_ready( name, replica_name, serde_json::json!({ "phase": "Failed" }), + "version detection Job succeeded but did not report a postgres version", ) .await; } @@ -838,6 +839,7 @@ async fn reconcile_ready( name, replica_name, serde_json::json!({ "phase": "Failed" }), + "version detection Job failed after backoff exhausted", ) .await; } @@ -906,6 +908,10 @@ async fn reconcile_ready( name, replica_name, serde_json::json!({ "phase": "Failed" }), + &format!( + "postgres Deployment did not become Ready within {timeout_secs}s of \ + restore completion" + ), ) .await; } From 809aaa46ebb89e4cb3a92ee8bd4bb30df8cdfc47 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 05:25:03 +1200 Subject: [PATCH 10/11] chore: exclude generated crds.yaml from typos The file is regenerated from k8s-openapi schemas whose descriptions use 'ANDed' / 'ORed' style jargon that typos flags as misspellings. Exclude the whole file rather than allowlisting each upstream term. --- _typos.toml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/_typos.toml b/_typos.toml index b67b8b7..bbc3459 100644 --- a/_typos.toml +++ b/_typos.toml @@ -1,3 +1,8 @@ [default.extend-words] # BRIN: postgres "Block Range Index" type. BRIN = "BRIN" + +[files] +# Generated from k8s-openapi schemas; upstream docstrings use +# "ANDed" / "ORed" style jargon typos flags as misspellings. +extend-exclude = ["crds.yaml"] From 696fc6c491da8783750b573df9f748afadcb8ac1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix=20Saparelli?= Date: Thu, 2 Jul 2026 05:21:47 +1200 Subject: [PATCH 11/11] unplan: canopy-cr-materialisation done Ships across three PRs (#82 + #83 + #84). Two conscious deviations from the plan: - Step 4: snapshot-list runs on both paths (not skipped on canopy) so canopy replicas also get real PVC + kopia-cache sizing from the snapshot metadata. Result-picking is by canopyDesiredSnapshotId on the canopy path. - Step 8: canopy RestoreVerification uses a direct hook (controllers::canopy::verification::report) rather than a NotificationConfig variant. No retry pipeline. Rationale: canopy reporting is coupled to spec.canopy_source, not user-configurable, and doesn't semantically belong alongside webhook / graphQL. Deferred follow-ups (out of scope per plan): - stub-canopy server + wire canopy_integration test into CI - ops handoff snippet: canopy declarations -> pgro CRs mapping --- docs/plans/canopy-cr-materialisation.md | 211 ------------------------ 1 file changed, 211 deletions(-) delete mode 100644 docs/plans/canopy-cr-materialisation.md diff --git a/docs/plans/canopy-cr-materialisation.md b/docs/plans/canopy-cr-materialisation.md deleted file mode 100644 index 37d7e5b..0000000 --- a/docs/plans/canopy-cr-materialisation.md +++ /dev/null @@ -1,211 +0,0 @@ -# Plan: canopy path via CR materialisation - -## Problem - -PR #73 shipped a parallel canopy path that duplicates most of what the -CRD path already does — a separate syncer, a separate reporter, a -separate postgres Deployment builder, a separate Service builder, a -separate restore-Job builder, a separate state machine (Namespace -labels + annotations). Result: `persistentSchemas`, `switchoverGracePeriod`, -blue/green refresh, credential-reset, analytics-user provisioning, -schema migration — all of which the CRD path implements — need to be -re-implemented on the canopy path from scratch. That's what caused -the "in-place PVC overwrite" (unacceptable), the disabled -`persistent_schemas`, and the current warn-and-hope situation. - -The CR path already has blue/green, phase machine, switchover with -grace, schema migration, analytics-user, resources, tolerations, -affinity, `serviceAnnotations`, `minimumTtl`, deletion cascade via -ownerRefs. Everything the canopy path wants. - -## Approach - -Make the canopy syncer materialise `PostgresPhysicalReplica` CRs from -the worklist. The CR is pgro-internal state — canopy remains the -operator interface. Two things about the pipeline change; everything -else is untouched: - -1. **Credential source** in the Job builders: proxy-sidecar variant - (dummy keys + `[::1]:` endpoint) when a new `canopy_source` - field is set on the CR, instead of env-from-Secret. -2. **Snapshot selection**: skip snapshot-list when `canopy_source` is - set; take snapshot id from a new `Status.canopy_desired_snapshot_id` - field written by the syncer each tick. - -Everything else — reconciler, Restore CR machinery, switchover, schema -migration, notifications — works unchanged. - -## Concrete changes - -### 1. Extend `PostgresPhysicalReplicaSpec` - -- Add `canopy_source: Option` where - `CanopySource { group: Uuid, type: String }`. -- Make `kopia_secret_ref` optional; exactly one of the two must be set. -- Validation: at admission time (webhook or reconcile-time check — - reconcile-time is simpler, matches existing `Error::InvalidKopiaSecret` - pattern), reject a spec that sets both or neither. -- Add `Status.canopy_desired_snapshot_id: Option` — the - syncer writes it, the reconciler consumes it. -- CRD regeneration + README table update per AGENTS.md. - -### 2. Job builders: proxy-sidecar branch - -- Introduce a `KopiaSource` enum (or similar) consumed by - `build_restore_job` and `build_snapshot_list_job` (though - snapshot-list gets skipped for canopy, keep the option for - symmetry). -- `KopiaSource::Secret { kopia_secret_ref, creds }` — current - behaviour, env-from-Secret. -- `KopiaSource::CanopyProxy { broker_url, region, group, type, - repo_password_source, server_id }` — dummy keys, `[::1]:` - endpoint via a pgro-canopy-proxy sidecar container, port-file - handshake, kopia connects via loopback. -- Pull the shell wrapper + sidecar container spec from - `src/controllers/canopy/builders.rs` into the CRD-path Job builder, - then delete the canopy-path builder. - -### 3. Replica reconciler: skip snapshot-list when canopy-sourced - -- If `spec.canopy_source.is_some()`: skip the snapshot-list Job - entirely. Take the snapshot from `status.canopy_desired_snapshot_id`. -- When `status.canopy_desired_snapshot_id` changes (or on schedule - fire), create a new `PostgresPhysicalRestore` CR pointing at the new - snapshot — same code path as today, just a different snapshot source. -- Everything else in the reconciler (Restore phase machine, switchover, - grace, schema migration, credential reset, analytics user) stays - untouched. - -### 4. Canopy syncer becomes tiny - -Replace ~1000 lines of parallel machinery with: - -```rust -async fn tick(&self) -> Result<()> { - let entries = self.ctx.canopy.worklist().await?; - let existing = self.list_canopy_managed_replicas().await?; - - for entry in &entries { - self.ensure_replica_cr(entry).await?; - } - for cr in &existing { - if !worklist_covers(cr, &entries) { - self.delete_replica_cr(cr).await?; - } - } - Ok(()) -} -``` - -Where `ensure_replica_cr`: -- Namespace: `pgro-r--<8hex>` (as today). -- Spec: from `IntentConfig(entry.intent)` — resources, service - annotations, persistent_schemas, min_ttl, switchover_grace, - read_only, analytics_username. Plus `canopy_source` = { group, type }. -- Status patch: `canopy_desired_snapshot_id = entry.snapshot_id`. -- Label: `pgro.bes.au/managed-by=pgro-canopy` (discovery key). - -The syncer no longer touches PVCs, Jobs, Deployments, Services, -Secrets — the existing CR reconciler owns all of that. - -### 5. Guardrail: reject user edits to canopy-managed CRs - -Simplest: in the replica reconciler, if the CR has label -`pgro.bes.au/managed-by=pgro-canopy` and the spec differs from what -the intent config would produce, re-apply the intent config. The -canopy syncer's next tick re-asserts anyway. - -Not building an admission webhook yet — the reconcile-time re-assert -is enough to converge, and users can't hurt themselves for long. - -### 6. Restore-verification reporter - -Two options: -- **(a)** New notification target `RestoreCanopyReport` in - `notifications.rs` alongside webhook / graphQL. Reuses the existing - retry / status-tracking pipeline. -- **(b)** Small dedicated controller watching Restore CRs, POSTs on - terminal phase transitions. - -(a) is simpler; use it. The report body is built from Replica + -Restore CR fields (already have everything: replica_id from label, -group/server from spec.canopy_source + status, snapshot_id from -restore.spec, postgres_version from restore.status, observed_at -from restore.status.activatedAt). - -## Retire - -- `src/controllers/canopy/builders.rs` — delete. Job builder branch - moves to CRD-path builder; postgres Deployment / Service / PVC - builders die (CRD path has them). -- `src/controllers/canopy/reporter.rs` — delete. Replaced by the - notification target. -- Most of `src/controllers/canopy.rs` — the diff/action/dispatch - machinery + provision/refresh/teardown functions all die. What - remains: the tick loop + `ensure_replica_cr` / `delete_replica_cr`. -- `Context.canopy_broker_base_url` / `canopy_proxy_image` / - `canopy_pgdata_pvc_size` / `canopy_stats` — most stay, but the - PVC-size default moves to being a CR spec field the syncer sets from - IntentConfig. - -## Keep - -- `src/controllers/canopy/intent.rs` — still drives per-intent CR - spec generation. `IntentConfig` gets a helper method: - `fn to_replica_spec_patch(&self, entry: &WorklistEntry) -> serde_json::Value` - (or similar). -- `src/bin/canopy_proxy.rs` — the sidecar binary. -- Broker route + `Context.canopy_stats` — the sidecar callback. -- Tailscale sidecar in `operator.yaml`. -- `src/canopy.rs` client wrapper. - -## Sequence - -Order that minimises "broken states" between commits: - -1. **Add `canopy_source` + status field to the CRD.** New optional - fields; existing behaviour unchanged. CRD regen + README update. -2. **Extend the Job builders with `KopiaSource` enum.** Both paths - route through it; legacy path uses `KopiaSource::Secret`. No - behaviour change yet. -3. **Job builder gains `CanopyProxy` variant.** Emits the sidecar - spec + dummy keys. Not called yet. -4. **Reconciler: skip snapshot-list when `canopy_source` is set.** - Take snapshot from `Status.canopy_desired_snapshot_id`. Guarded so - legacy path is unaffected. -5. **`IntentConfig::to_replica_spec_patch`** — converts an intent + a - worklist entry into a Replica-spec JSON patch. -6. **Rewrite canopy syncer** to materialise CRs. All old logic - deleted; only ensure_replica_cr / delete_replica_cr remain. -7. **Retire** `canopy/builders.rs` + `canopy/reporter.rs`. -8. **Add canopy notification target** in `notifications.rs`; hook it - at the Restore terminal-transition callsites. -9. **CI + README updates.** - -Each step's `cargo check` + `cargo test --lib` must pass. Fmt + clippy -before each commit. - -## Verification - -- Unit tests survive at every commit (176+). -- Legacy `kopiaSecretRef` integration tests continue to pass — steps - 1-4 are additive to that path. -- Canopy path integration test in a follow-up (still gated on the - stub-canopy server). -- Manual: apply a canopy-labelled Replica CR by hand pointing at a - test worklist; watch the reconciler run through Restoring → - Ready → Switching → Active exactly as with a legacy Replica. - -## What NOT to build in this PR - -- Admission webhook. Reconcile-time re-assert is the guardrail. -- Blue-green refactor of the canopy path — the CR path already does - blue/green. -- Anything about `disaster-recovery` intent — separately gone in #79. - -## Follow-up (out of scope for this plan) - -- Stub-canopy server + wire integration test back into CI. -- Publish a snippet in the ops handoff about how canopy declarations - map onto pgro's internal CRs, and how to poke at them via kubectl - for debugging.