fix: EKS cluster template and kro RGD issues by allamand · Pull Request #744 · aws-samples/appmod-blueprints

allamand · 2026-06-09T18:55:03Z

Summary

Fixes multiple issues in the EKS cluster Backstage template and kro ResourceGraphDefinition.

Closes #743

Changes

Severity	File	Fix
High	`platform/backstage/templates/eks-cluster-template/template.yaml`	Add missing `enableAwsEfsCsiDriver` parameter definition
High	`gitops/addons/charts/kro/resource-groups/manifests/eks/rg-eks-vpc.yaml`	Fix `readyWhen`: `state` → `clusterState` to match EksCluster RGD status schema
Medium	`rg-eks-vpc.yaml`	Fix duplicate YAML key `argocd.argoproj.io/sync-options` by combining values
Medium	`template.yaml`	Fix component link: add `gitops/` prefix, use dynamic `${{ parameters.tenant }}`
Low	`template.yaml`	Fix `spec.owner`: `guests` → `user:guest`

Testing

Validate kro readyWhen condition resolves correctly with an active EKS cluster
Verify Backstage template renders enableAwsEfsCsiDriver parameter correctly
Confirm ArgoCD sync-options are both applied after the annotation fix

- Add missing enableAwsEfsCsiDriver parameter to Backstage template - Fix readyWhen condition: use clusterState instead of state in rg-eks-vpc.yaml - Fix duplicate argocd.argoproj.io/sync-options annotation (YAML key collision) - Fix Backstage component link: add gitops/ prefix and use dynamic tenant param - Fix spec.owner: use user:guest instead of bare guests

The ACK secretsmanager controller running as an EKS Capability needs ClusterRole permissions to manage secretsmanager.services.k8s.aws CRDs. Without this, the controller cannot resolve secretString references to k8s secrets, causing clusterConfigSecret to never sync.

…s to true

…conciliation - Add critical section in eks-best-practices.md to prevent deleting K8s resources to force reconciliation - Add troubleshooting best practices in aws-env-best-practices.md emphasizing patience and non-destructive debugging - These rules prevent breaking status propagation chains in Kro resource graphs and ACK controllers - Recommend 15-30 minute wait time for controller reconciliation after config changes

allamand · 2026-06-11T09:13:46Z

Missing: `secretsmanager` IAMRoleSelector in `iam-role-selectors.yaml`

The PR adds RBAC for secretsmanager.services.k8s.aws in eks-capabilities-rbac.yaml (good), but the rg-eks.yaml creates Secrets Manager resources (clusterConfigSecret, clusterSecretsSecret) in spoke namespaces. Without an IAMRoleSelector for secretsmanager, ACK uses the default capability role which has no secretsmanager:* permissions → the secrets fail with AccessDeniedException.

Fix needed

Add to gitops/addons/charts/multi-acct/templates/iam-role-selectors.yaml (inside the range loop):

---
apiVersion: services.k8s.aws/v1alpha1
kind: IAMRoleSelector
metadata:
  name: secretsmanager-{{ $key }}
spec:
  arn: "arn:aws:iam::{{ $value }}:role/{{ $.Values.global.resourcePrefix }}-cluster-mgmt-secretsmanager"
  namespaceSelector:
    names:
      - {{ $key }}
  resourceTypeSelector:
    - group: secretsmanager.services.k8s.aws
      kind: ""
      version: ""

Reference

This was fixed on feature/cloudfront-on-agent-platform in commit 7f123cde — see the full template there.

Also: apply the PR #744 changes to `feature/cloudfront-on-agent-platform`

The following changes from this PR should be cherry-picked onto our branch:

Merge duplicate argocd.argoproj.io/sync-options annotations into one line
Change observability defaults to true (enable_cni_metrics_helper, enable_prometheus_node_exporter, enable_kube_state_metrics)
Fix readyWhen to use eks.status.clusterState instead of eks.status.state
Add secretsmanager RBAC in eks-capabilities-rbac.yaml

- Merge duplicate sync-options annotations - Enable observability by default (cni_metrics, prometheus_node_exporter, kube_state_metrics) - Fix readyWhen: use eks.status.clusterState instead of eks.status.state - Add secretsmanager RBAC in eks-capabilities-rbac

…RoleName

- Merge duplicate sync-options annotations - Enable observability by default (cni_metrics, prometheus_node_exporter, kube_state_metrics) - Fix readyWhen: use eks.status.clusterState instead of eks.status.state - Add secretsmanager RBAC in eks-capabilities-rbac

allamand · 2026-06-13T10:37:05Z

Tested in https://studio.us-east-1.prod.workshops.aws/events/5c8791b6-1d88-4cdb-8b36-cbc47e8b1b1e#TEAMS

shapirov103

Very cool, looks great, please see my comment.

shapirov103 · 2026-06-18T14:04:13Z

  # Get admin role name from current AWS context
-  ADMIN_ROLE_NAME=$(aws sts get-caller-identity --query 'Arn' --output text | sed 's|.*assumed-role/||' | sed 's|/.*||')
+  # Use WSParticipantRole (Workshop Studio standard role). Fallback to caller identity for self-paced deployments.
+  ADMIN_ROLE_NAME=$(aws iam list-roles --query 'Roles[?contains(RoleName,`WSParticipantRole`)].RoleName' --output text 2>/dev/null)


WSParticipant role is specific to the workshop. Should go into the gitlab repo. What is the plan to start moving this out of the repo? Otherwise the refactoring effort for agent-platform will break the workshop and will be hard to merge back to main.

We don't use utils.sh in the agentic branch (pr-709) but will need to double check there that we don't reference it out of the workshop pattern

if we don't use this in the agentic branch, but the agentic branch is going to target main soon - what will happen with this change? Does it mean someone will have to replicate in the workshop? if so, why not do it now, save on the merge, I am afraid it maybe lost otherwise after the merge.

yes if the plan is to merge agentic work into main quickly then we can save effort. but for now, main is broken with the optional module to create eks staging cluster not working. so everything depend on the timeline we are agree to merge agentic work to main, and I think there is still some work to do before that

Sounds good, we can apply the fix now, the sh change is one of the things in the PR, other changes seem to be applicable regardless of the workshop

…fix branch name

allamand · 2026-06-24T13:48:12Z

All 5 fixes from this PR are already present in feature/cloudfront-on-agent-platform (PR #709) — no separate port needed. They will land on main when PR #709 merges.

shapirov103

LGTM (utils.sh change to move out of the repo, deferred)

hmuthusamy

LGTM

Workshop User added 4 commits June 9, 2026 17:10

fix: default cni-metrics, prometheus-node-exporter, kube-state-metric…

ca00341

…s to true

fix: zsh-compatible backstage-auth.sh and WSParticipantRole for admin…

e4d97e0

…RoleName

allamand marked this pull request as ready for review June 13, 2026 10:36

shapirov103 reviewed Jun 18, 2026

View reviewed changes

fix(backstage): remove stale ingressDomainName param + fix double-suf…

c628b11

…fix branch name

allamand added the ready-to-merge this PR is ready to be merged label Jun 24, 2026

shapirov103 approved these changes Jun 24, 2026

View reviewed changes

hmuthusamy approved these changes Jun 25, 2026

View reviewed changes

shapirov103 merged commit fcc7536 into main Jul 2, 2026
6 checks passed

shapirov103 deleted the fix/eks-cluster-template-issues branch July 2, 2026 13:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: EKS cluster template and kro RGD issues#744

fix: EKS cluster template and kro RGD issues#744
shapirov103 merged 6 commits into
mainfrom
fix/eks-cluster-template-issues

allamand commented Jun 9, 2026

Uh oh!

allamand commented Jun 11, 2026

Uh oh!

allamand commented Jun 13, 2026

Uh oh!

shapirov103 left a comment

Uh oh!

shapirov103 Jun 18, 2026

Uh oh!

allamand Jun 18, 2026

Uh oh!

shapirov103 Jun 24, 2026

Uh oh!

allamand Jun 24, 2026

Uh oh!

shapirov103 Jun 24, 2026

Uh oh!

allamand commented Jun 24, 2026

Uh oh!

shapirov103 left a comment

Uh oh!

hmuthusamy left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

allamand commented Jun 9, 2026

Summary

Changes

Testing

Uh oh!

allamand commented Jun 11, 2026

Missing: secretsmanager IAMRoleSelector in iam-role-selectors.yaml

Fix needed

Reference

Also: apply the PR #744 changes to feature/cloudfront-on-agent-platform

Uh oh!

allamand commented Jun 13, 2026

Uh oh!

shapirov103 left a comment

Choose a reason for hiding this comment

Uh oh!

shapirov103 Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

allamand Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

shapirov103 Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

allamand Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

shapirov103 Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

allamand commented Jun 24, 2026

Uh oh!

shapirov103 left a comment

Choose a reason for hiding this comment

Uh oh!

hmuthusamy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Missing: `secretsmanager` IAMRoleSelector in `iam-role-selectors.yaml`

Also: apply the PR #744 changes to `feature/cloudfront-on-agent-platform`