Skip to content

feat(backup-policy): KRO RGD for AWS Backup with gold/silver/bronze tiers + DR-aware StorageClasses#661

Draft
allamand wants to merge 9 commits into
aws-samples:feature/platform-cluster-kro-ackfrom
allamand:feat/backup-policy-clean
Draft

feat(backup-policy): KRO RGD for AWS Backup with gold/silver/bronze tiers + DR-aware StorageClasses#661
allamand wants to merge 9 commits into
aws-samples:feature/platform-cluster-kro-ackfrom
allamand:feat/backup-policy-clean

Conversation

@allamand

Copy link
Copy Markdown
Contributor

What

Adds the BackupPolicy KRO ResourceGraphDefinition (RGD) and the supporting plumbing so that AWS Backup for EKS can run end-to-end on a hub provisioned by feature/platform-cluster-kro-ack. Workloads opt into a backup tier (gold / silver / bronze) by picking the matching StorageClass; the BackupSelection in the RGD matches volumes by tag and includes them in the right BackupPlan.

Why

PR #642 lands the kind+ACK+KRO bootstrap and the EksClusterWithVpc RGD. There is no equivalent for AWS Backup yet: today, anyone running the platform either writes raw ACK Backup CRs or wires AWS Backup out-of-band. This PR closes that gap with the same KRO/ACK pattern as the rest of the resource-groups directory:

  • One CRD (BackupPolicy) per spoke + tier.
  • Tier semantics declared in the RGD (RPO, retention, cross-region copy).
  • BackupSelection matches on the peeks.io/backup-tier tag emitted by the StorageClass at PVC creation time — workload-driven, no central ConfigMap, no platform ticket.

Scope of this PR

Included — BackupPolicy + the strict set of prereqs that make it functional:

Commit Scope
7f301532 BackupPolicy RGD with gold / silver / bronze tiers
b37a1167 Merge peeks-gp3 StorageClass into platform-manifests-bootstrap
6d3e5005 Scope ManagePodIdentityAssociations inline policy on ACK capability role
aabebc14 Bind KRO capability STS username for ACK IAM access (multi-acct)
1ef9b694 appset-chart enableACKKRO path for kro+ACK pod identity
efa981e0 Wire ack-backup-controller via kro+ACK pod identity
ae02465c Move example instances out of GitOps apply path
6e4742ed Grant kro capability access to backup.services.k8s.aws
6fcb00e5 DR-aware tier StorageClasses (peeks-gold-gp3 / peeks-silver-gp3 / peeks-bronze-gp3) — closes the missing piece (RGD's BackupSelection matches peeks.io/backup-tier, but no SC currently emits this tag)

Excluded (kept on a separate working branch, not part of this PR) — unrelated commits also living on the upstream-fork: ingress modes (cloudfront | tls), kind-kro-ack bootstrap fixes (AWS_PROFILE, wait-for-eks, BYO IdC), cert-manager memory bump, rg-eks-vpc Ready condition. Those are independent scopes and will be proposed separately if useful upstream.

Architecture

Tier policies (in the RGD)

Tier RPO Retention Cold storage Cross-region copy
gold 1h 30d after 90d yes
silver 24h 14d yes
bronze 24h 7d no

Cross-region copy targets are passed in by the claim (copyDestinationVaultARN) — the destination vault is expected to exist in the DR region (could be nested in a future RGD revision; flagged as a known limit in the README).

Tag matching

# BackupSelection conditions in rgd-backup-policy.yaml
conditions:
  stringEquals:
    - conditionKey: "aws:ResourceTag/peeks.io/backup-tier"
      conditionValue: <tier>
    - conditionKey: "aws:ResourceTag/peeks.io/spoke"
      conditionValue: <spokeName>

Volumes inherit the peeks.io/backup-tier and peeks.io/spoke tags from the StorageClass via the EBS CSI driver's tagSpecification_<N> parameter, which the consolidated peeks-storageclass.yaml template renders from a tags: map per StorageClass:

peeks-gold-gp3:
  reclaimPolicy: Retain
  volumeType: gp3
  iops: 3000
  throughput: 125
  tags:
    peeks.io/managed-by: peeks-platform
    peeks.io/backup-tier: gold

spokeName is a top-level value merged into every EBS SC as peeks.io/spoke=<spokeName>.

Resource graph

BackupPolicy CR (claim)
   │
   ▼
┌─ BackupVault (ACK)            ── always
│  name: peeks-<spoke>-<tier>
└────
   │
   ▼
┌─ BackupPlan (ACK)             ── one variant per tier (includeWhen)
│  rules: schedule + lifecycle
│  copyActions for gold/silver
└────
   │
   ▼
┌─ BackupSelection (ACK)        ── one variant per tier
│  iamRoleARN: <from claim>
│  resources:
│    - arn:aws:ec2:*:*:volume/*
│    - arn:aws:eks:*:*:cluster/*
│  conditions: peeks.io/backup-tier + peeks.io/spoke
└────

KRO reconciles the chain in order; BackupSelection waits for the BackupPlan ACK.ResourceSynced condition before applying.

Validation

# RGD syntax
yq eval '.' gitops/addons/charts/kro/resource-groups/manifests/backup-policy/rgd-backup-policy.yaml

# server-side dry-run
kubectl --context hub apply --dry-run=server \
  -f gitops/addons/charts/kro/resource-groups/manifests/backup-policy/rgd-backup-policy.yaml

# Helm rendering — 5 StorageClasses (legacy, default, 3 tiers)
helm template t gitops/addons/charts/platform-manifests-bootstrap/ \
  --set spokeName=spoke-test | grep -c "^kind: StorageClass"
# → 5

# end-to-end — instance expands into BackupVault + BackupPlan + BackupSelection
kubectl --context hub apply -f gitops/addons/charts/kro/resource-groups/examples/backup-policy/instance-bronze.yaml
kubectl --context hub get backupvault,backupplan,backupselection -A

Files changed

 gitops/addons/charts/kro/resource-groups/examples/backup-policy/{README,instance-{bronze,gold,silver}}.yaml      | new (4 files, examples kept out of apply path)
 gitops/addons/charts/kro/resource-groups/manifests/backup-policy/{README.md,rgd-backup-policy.yaml}                 | new (cœur)
 gitops/addons/charts/kro/resource-groups/manifests/eks/rg-eks.yaml                                                  | scope ManagePodIdentityAssociations
 gitops/addons/charts/kro/resource-groups/templates/eks-capabilities-rbac.yaml                                       | KRO STS username binding
 gitops/addons/charts/platform-manifests-bootstrap/{README-storageclasses.md, templates/peeks-storageclass.yaml, values.yaml} | SC consolidation + tiers
 gitops/addons/registry/platform.yaml                                                                                 | register backup-policy & ack-backup
 platform-charts/appset-chart/templates/{_pod_identity_kro.tpl, application-set.yaml}                                  | enableACKKRO pod-identity wiring
 platform/infra/terraform/hub-config.yaml                                                                              | grant capability access to backup.services.k8s.aws

15 files changed, +821 / -0.

Compatibility

  • Legacy ebs-sc-gp3 and the previous default peeks-gp3 are preserved unchanged. Workloads that don't opt into a tier keep their current SC.
  • No deletions in this PR (additive only).
  • Bronze SC uses reclaimPolicy=Delete (cheap / ephemeral); gold and silver use Retain.

Known limits (v1)

  • No KMS key rotation / custom encryption — uses the default vault key.
  • Heavy stateful workloads (Postgres, Kafka) need an application-level snapshot workflow on top of AWS Backup; AWS Backup gives a crash-consistent FS snapshot, not a logical DB backup.
  • copyDestinationVaultARN must be pre-provisioned in the DR region. A future iteration could nest that vault in the same RGD.
  • Restore side (how PVs land in a target cluster) is the responsibility of a companion RestoreSelection RGD — not in this PR.

Checklist

  • Targets feature/platform-cluster-kro-ack (extends PR feat: kind-kro-ack cluster provider (bootstrap hub via ACK+KRO) #642 family).
  • Additive — no existing files renamed or deleted in functional paths (only example instances moved out of the GitOps apply path, on purpose).
  • helm template renders cleanly.
  • RGD passes kubectl apply --dry-run=server.
  • Examples isolated from the GitOps apply path so they don't accidentally reconcile in every spoke.
  • Scope kept narrow — unrelated commits (ingress modes, kind bootstrap fixes) excluded for separate PRs if needed.

Related

allamand added 9 commits May 18, 2026 10:15
Port from PR aws-samples#633 (feat/peeks-backup-policy-rgd) to the new structure
introduced by PR aws-samples#642 (feature/platform-cluster-kro-ack).

The RGD provisions BackupVault + BackupPlan + BackupSelection per tier
via ACK backup-controller. Tier semantics live on the BackupPolicy CR,
not on the StorageClass — backup selection happens at cluster/namespace
level via tag matching (peeks.io/backup-tier + peeks.io/spoke).

- gold: RPO 1h, retention 30d, cold 90d, cross-region copy
- silver: RPO 24h, retention 14d, cross-region copy
- bronze: RPO 24h, retention 7d, same-region only
Port the storageclass-resources chart from PR aws-samples#633 into the new
platform-manifests-bootstrap chart from PR aws-samples#642 — single bootstrap
chart for early cluster resources (NodePools, ClusterSecretStore,
StorageClasses).

- New template peeks-storageclass.yaml renders ebs/efs SCs from values
- New values.yaml with defaults (gpu/customNodepools/storageClasses)
- peeks-gp3: single PeEKS default SC, tier semantics live on BackupPolicy
- Volume tags propagated via tagSpecification_N parameters for AWS Backup
  selection (peeks.io/spoke, peeks.io/managed-by)
- README-storageclasses.md preserves design notes from the standalone chart
…K capability role

Add inline policy 'ManagePodIdentityAssociations' to the ACK capability role
so that ACK controllers managed by the EKS ACK capability can create EKS Pod
Identity associations and pass scoped workload roles. Required for any new
ACK controller (e.g. ack-backup) wired through kro pod-identity instances.

- iam:PassRole scoped to role/<name>-* with iam:PassedToService = pods.eks.amazonaws.com
- eks:*PodIdentityAssociation* + tagging actions scoped to cluster/<name>
  and podidentityassociation/<name>/*
- accountId and name come from schema.spec for cross-account / multi-cluster reuse
The eks-capabilities-kro ClusterRoleBinding only bound the synthetic
User "capabilities.eks.amazonaws.com" which EKS uses for some internal
calls. When the KRO capability assumes its IAM role to act on K8s
resources (e.g. while reconciling the podidentity RGD which declares
iam.services.k8s.aws/Policy and Role children), the API server sees the
STS username 'arn:aws:sts::<acct>:assumed-role/<prefix>-<cluster>-kro-capability-role/KRO'
instead.

Without this second subject, KRO fails with:
  policies.iam.services.k8s.aws is forbidden: User
  "arn:aws:sts::...:assumed-role/peeks-hub-kro-capability-role/KRO"
  cannot get resource "policies" in API group "iam.services.k8s.aws"

This was not exposed before because the podidentity RGD had never been
instantiated (the kro+ACK pod-identity path enableACKKRO=true is new in
this branch, used by ack-backup-controller). RBAC subjects do not
support wildcards, so the templated literal ARN is rendered from
chart values (accountId/resourcePrefix/clusterName), which are now
plumbed via the multi-acct entry in addons/registry/platform.yaml.
Wire a new 'enableACKKRO' flag on per-addon chart configs in the
appset-chart. When set, the addon ApplicationSet gets a second source
pointing to gitops/addons/charts/kro/instances/pod-identity, which
materializes ACK iam.Role + iam.Policy + eks.PodIdentityAssociation via
the podidentity.kro.run RGD instead of the legacy Crossplane path.

- 'enableACKKRO' and 'enableAckPodIdentity' are mutually exclusive;
  the template fails fast if both are set on the same addon.
- All existing addons keep their behavior (no flag flip).
- Required for ACK controllers not yet bundled in the EKS ACK
  capability (e.g. ack-backup, currently in preview at v0.1.1).

Adapted from upstream PR aws-samples#631 to fit the
appset-chart structure on the feature/platform-cluster-kro-ack base
(PR aws-samples#642). Credit to the upstream author.
Register the ack-backup-controller (chart oci://public.ecr.aws/aws-controllers-k8s/backup-chart:0.1.1)
as a regular addon in gitops/addons/registry/platform.yaml.

Why an addon and not the EKS ACK capability?
The backup ACK controller is in PREVIEW (v0.1.1, May 2026) and is NOT
yet bundled in the managed 'ACK' capability. Until AWS ships it as part
of the capability, it is deployed via Helm.

How does it get its IAM?
Through the new enableACKKRO=true path on appset-chart (previous commit).
That path renders gitops/addons/charts/kro/instances/pod-identity, which
expands via the podidentity.kro.run RGD to ACK iam.Role + iam.Policy +
eks.PodIdentityAssociation for SA ack-backup-controller in ack-system.

Policy follows least-privilege:
- backup:* + backup-storage:* on all resources
- iam:PassRole scoped to role/* (controller passes backup service roles)
- kms:Decrypt/Encrypt/GenerateDataKey/CreateGrant for encrypted vaults

Enabled only on the hub cluster via 'enable_ack_backup: true' in
hub-config.yaml; spokes keep it false.

Required prerequisite for the BackupPolicy RGD (PR aws-samples#644) to materialize
its referenced backup.services.k8s.aws CRDs (BackupVault, BackupPlan,
BackupSelection).

Adapted from upstream PR aws-samples#631.
The 'kro-manifests-hub' ApplicationSet recursively applies every YAML
under gitops/addons/charts/kro/resource-groups/manifests/. Three sample
BackupPolicy instances under .../backup-policy/examples/ were therefore
applied by Argo CD on the hub before the BackupPolicy CRD existed,
which broke the whole sync run with:

  resource mapping not found for kind "BackupPolicy" in version "kro.run/v1alpha1"

That fail-fast aborted the apply of the *RGDs themselves*
(backup-policy, pod-identity, eks, etc.), so no RGD ever became Active
on the hub — chicken-and-egg.

Move the three sample instances to
gitops/addons/charts/kro/resource-groups/examples/backup-policy/, which
sits outside the apply path. The RGD ships alone in manifests/, like
the other RGDs (eks, pod-identity, ray-service, huggingface).

Add a short README in the new directory documenting the manual-apply
workflow once the RGD is Active and ACK Backup CRDs exist.
The KRO ResourceGraphDefinition 'backuppolicy.kro.run' instantiates
BackupVault / BackupPlan / BackupSelection (apiGroup
backup.services.k8s.aws). Without this rule, KRO reconciles fail with:

  backupvaults.backup.services.k8s.aws is forbidden:
  User "...:assumed-role/<prefix>-<cluster>-kro-capability-role/KRO"
  cannot get resource "backupvaults"

Mirrors the existing rules for ec2/eks/iam/ecr ACK groups.
The BackupPolicy RGD's BackupSelection matches on
'aws:ResourceTag/peeks.io/backup-tier' but no StorageClass currently
emits this tag, so volumes never enter a BackupPlan.

Add three tier-labeled StorageClasses — peeks-gold-gp3, peeks-silver-gp3,
peeks-bronze-gp3 — with backup-tier tags propagated to EBS volumes via
the existing 'tags:' helm value. Workloads opt into a tier by picking
the matching SC in their PVC manifest, so backup tier is workload-driven
(no central ConfigMap, no platform ticket).

The legacy ebs-sc-gp3 and the default peeks-gp3 are preserved unchanged.
Bronze keeps reclaimPolicy=Delete (cheap/ephemeral); gold and silver
keep Retain.

Tier policies (gold/silver/bronze RPO + retention + cross-region copy)
are defined in the BackupPolicy RGD — this commit only wires the
emitter side. Ported from feat/peeks-backup-policy-rgd commit 6862a00
onto the rebased platform-manifests-bootstrap chart layout.
@allamand allamand marked this pull request as draft May 22, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant