feat(backup-policy): KRO RGD for AWS Backup with gold/silver/bronze tiers + DR-aware StorageClasses#661
Draft
allamand wants to merge 9 commits into
Conversation
Port from PR aws-samples#633 (feat/peeks-backup-policy-rgd) to the new structure introduced by PR aws-samples#642 (feature/platform-cluster-kro-ack). The RGD provisions BackupVault + BackupPlan + BackupSelection per tier via ACK backup-controller. Tier semantics live on the BackupPolicy CR, not on the StorageClass — backup selection happens at cluster/namespace level via tag matching (peeks.io/backup-tier + peeks.io/spoke). - gold: RPO 1h, retention 30d, cold 90d, cross-region copy - silver: RPO 24h, retention 14d, cross-region copy - bronze: RPO 24h, retention 7d, same-region only
Port the storageclass-resources chart from PR aws-samples#633 into the new platform-manifests-bootstrap chart from PR aws-samples#642 — single bootstrap chart for early cluster resources (NodePools, ClusterSecretStore, StorageClasses). - New template peeks-storageclass.yaml renders ebs/efs SCs from values - New values.yaml with defaults (gpu/customNodepools/storageClasses) - peeks-gp3: single PeEKS default SC, tier semantics live on BackupPolicy - Volume tags propagated via tagSpecification_N parameters for AWS Backup selection (peeks.io/spoke, peeks.io/managed-by) - README-storageclasses.md preserves design notes from the standalone chart
…K capability role Add inline policy 'ManagePodIdentityAssociations' to the ACK capability role so that ACK controllers managed by the EKS ACK capability can create EKS Pod Identity associations and pass scoped workload roles. Required for any new ACK controller (e.g. ack-backup) wired through kro pod-identity instances. - iam:PassRole scoped to role/<name>-* with iam:PassedToService = pods.eks.amazonaws.com - eks:*PodIdentityAssociation* + tagging actions scoped to cluster/<name> and podidentityassociation/<name>/* - accountId and name come from schema.spec for cross-account / multi-cluster reuse
The eks-capabilities-kro ClusterRoleBinding only bound the synthetic User "capabilities.eks.amazonaws.com" which EKS uses for some internal calls. When the KRO capability assumes its IAM role to act on K8s resources (e.g. while reconciling the podidentity RGD which declares iam.services.k8s.aws/Policy and Role children), the API server sees the STS username 'arn:aws:sts::<acct>:assumed-role/<prefix>-<cluster>-kro-capability-role/KRO' instead. Without this second subject, KRO fails with: policies.iam.services.k8s.aws is forbidden: User "arn:aws:sts::...:assumed-role/peeks-hub-kro-capability-role/KRO" cannot get resource "policies" in API group "iam.services.k8s.aws" This was not exposed before because the podidentity RGD had never been instantiated (the kro+ACK pod-identity path enableACKKRO=true is new in this branch, used by ack-backup-controller). RBAC subjects do not support wildcards, so the templated literal ARN is rendered from chart values (accountId/resourcePrefix/clusterName), which are now plumbed via the multi-acct entry in addons/registry/platform.yaml.
Wire a new 'enableACKKRO' flag on per-addon chart configs in the appset-chart. When set, the addon ApplicationSet gets a second source pointing to gitops/addons/charts/kro/instances/pod-identity, which materializes ACK iam.Role + iam.Policy + eks.PodIdentityAssociation via the podidentity.kro.run RGD instead of the legacy Crossplane path. - 'enableACKKRO' and 'enableAckPodIdentity' are mutually exclusive; the template fails fast if both are set on the same addon. - All existing addons keep their behavior (no flag flip). - Required for ACK controllers not yet bundled in the EKS ACK capability (e.g. ack-backup, currently in preview at v0.1.1). Adapted from upstream PR aws-samples#631 to fit the appset-chart structure on the feature/platform-cluster-kro-ack base (PR aws-samples#642). Credit to the upstream author.
Register the ack-backup-controller (chart oci://public.ecr.aws/aws-controllers-k8s/backup-chart:0.1.1) as a regular addon in gitops/addons/registry/platform.yaml. Why an addon and not the EKS ACK capability? The backup ACK controller is in PREVIEW (v0.1.1, May 2026) and is NOT yet bundled in the managed 'ACK' capability. Until AWS ships it as part of the capability, it is deployed via Helm. How does it get its IAM? Through the new enableACKKRO=true path on appset-chart (previous commit). That path renders gitops/addons/charts/kro/instances/pod-identity, which expands via the podidentity.kro.run RGD to ACK iam.Role + iam.Policy + eks.PodIdentityAssociation for SA ack-backup-controller in ack-system. Policy follows least-privilege: - backup:* + backup-storage:* on all resources - iam:PassRole scoped to role/* (controller passes backup service roles) - kms:Decrypt/Encrypt/GenerateDataKey/CreateGrant for encrypted vaults Enabled only on the hub cluster via 'enable_ack_backup: true' in hub-config.yaml; spokes keep it false. Required prerequisite for the BackupPolicy RGD (PR aws-samples#644) to materialize its referenced backup.services.k8s.aws CRDs (BackupVault, BackupPlan, BackupSelection). Adapted from upstream PR aws-samples#631.
The 'kro-manifests-hub' ApplicationSet recursively applies every YAML under gitops/addons/charts/kro/resource-groups/manifests/. Three sample BackupPolicy instances under .../backup-policy/examples/ were therefore applied by Argo CD on the hub before the BackupPolicy CRD existed, which broke the whole sync run with: resource mapping not found for kind "BackupPolicy" in version "kro.run/v1alpha1" That fail-fast aborted the apply of the *RGDs themselves* (backup-policy, pod-identity, eks, etc.), so no RGD ever became Active on the hub — chicken-and-egg. Move the three sample instances to gitops/addons/charts/kro/resource-groups/examples/backup-policy/, which sits outside the apply path. The RGD ships alone in manifests/, like the other RGDs (eks, pod-identity, ray-service, huggingface). Add a short README in the new directory documenting the manual-apply workflow once the RGD is Active and ACK Backup CRDs exist.
The KRO ResourceGraphDefinition 'backuppolicy.kro.run' instantiates BackupVault / BackupPlan / BackupSelection (apiGroup backup.services.k8s.aws). Without this rule, KRO reconciles fail with: backupvaults.backup.services.k8s.aws is forbidden: User "...:assumed-role/<prefix>-<cluster>-kro-capability-role/KRO" cannot get resource "backupvaults" Mirrors the existing rules for ec2/eks/iam/ecr ACK groups.
The BackupPolicy RGD's BackupSelection matches on 'aws:ResourceTag/peeks.io/backup-tier' but no StorageClass currently emits this tag, so volumes never enter a BackupPlan. Add three tier-labeled StorageClasses — peeks-gold-gp3, peeks-silver-gp3, peeks-bronze-gp3 — with backup-tier tags propagated to EBS volumes via the existing 'tags:' helm value. Workloads opt into a tier by picking the matching SC in their PVC manifest, so backup tier is workload-driven (no central ConfigMap, no platform ticket). The legacy ebs-sc-gp3 and the default peeks-gp3 are preserved unchanged. Bronze keeps reclaimPolicy=Delete (cheap/ephemeral); gold and silver keep Retain. Tier policies (gold/silver/bronze RPO + retention + cross-region copy) are defined in the BackupPolicy RGD — this commit only wires the emitter side. Ported from feat/peeks-backup-policy-rgd commit 6862a00 onto the rebased platform-manifests-bootstrap chart layout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the
BackupPolicyKRO ResourceGraphDefinition (RGD) and the supporting plumbing so that AWS Backup for EKS can run end-to-end on a hub provisioned byfeature/platform-cluster-kro-ack. Workloads opt into a backup tier (gold / silver / bronze) by picking the matching StorageClass; the BackupSelection in the RGD matches volumes by tag and includes them in the right BackupPlan.Why
PR #642 lands the kind+ACK+KRO bootstrap and the
EksClusterWithVpcRGD. There is no equivalent for AWS Backup yet: today, anyone running the platform either writes raw ACK Backup CRs or wires AWS Backup out-of-band. This PR closes that gap with the same KRO/ACK pattern as the rest of the resource-groups directory:BackupPolicy) per spoke + tier.peeks.io/backup-tiertag emitted by the StorageClass at PVC creation time — workload-driven, no central ConfigMap, no platform ticket.Scope of this PR
Included — BackupPolicy + the strict set of prereqs that make it functional:
7f301532BackupPolicyRGD with gold / silver / bronze tiersb37a1167peeks-gp3StorageClass intoplatform-manifests-bootstrap6d3e5005ManagePodIdentityAssociationsinline policy on ACK capability roleaabebc141ef9b694appset-chartenableACKKRO path for kro+ACK pod identityefa981e0ack-backup-controllervia kro+ACK pod identityae02465c6e4742edbackup.services.k8s.aws6fcb00e5peeks-gold-gp3/peeks-silver-gp3/peeks-bronze-gp3) — closes the missing piece (RGD's BackupSelection matchespeeks.io/backup-tier, but no SC currently emits this tag)Excluded (kept on a separate working branch, not part of this PR) — unrelated commits also living on the upstream-fork: ingress modes (
cloudfront | tls), kind-kro-ack bootstrap fixes (AWS_PROFILE,wait-for-eks, BYO IdC),cert-managermemory bump,rg-eks-vpcReady condition. Those are independent scopes and will be proposed separately if useful upstream.Architecture
Tier policies (in the RGD)
Cross-region copy targets are passed in by the claim (
copyDestinationVaultARN) — the destination vault is expected to exist in the DR region (could be nested in a future RGD revision; flagged as a known limit in the README).Tag matching
Volumes inherit the
peeks.io/backup-tierandpeeks.io/spoketags from the StorageClass via the EBS CSI driver'stagSpecification_<N>parameter, which the consolidatedpeeks-storageclass.yamltemplate renders from atags:map per StorageClass:spokeNameis a top-level value merged into every EBS SC aspeeks.io/spoke=<spokeName>.Resource graph
KRO reconciles the chain in order;
BackupSelectionwaits for theBackupPlanACK.ResourceSyncedcondition before applying.Validation
Files changed
15 files changed, +821 / -0.
Compatibility
ebs-sc-gp3and the previous defaultpeeks-gp3are preserved unchanged. Workloads that don't opt into a tier keep their current SC.reclaimPolicy=Delete(cheap / ephemeral); gold and silver useRetain.Known limits (v1)
copyDestinationVaultARNmust be pre-provisioned in the DR region. A future iteration could nest that vault in the same RGD.RestoreSelectionRGD — not in this PR.Checklist
feature/platform-cluster-kro-ack(extends PR feat: kind-kro-ack cluster provider (bootstrap hub via ACK+KRO) #642 family).helm templaterenders cleanly.kubectl apply --dry-run=server.Related
AppExposureRGD)feature/platform-cluster-kro-ack)