Migrate DORA RDS Cluster to KRO and ACK

[zjaco13]

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[allamand]

Thanks @zjaco13 i'm starting a test on this

[allamand]

Review — fix/dora-to-kro (tested on Workshop Studio event)

I deployed this branch to a fresh Workshop Studio environment (account 609897678650, region us-west-2) and validated the full DORA-to-KRO flow end-to-end.

The KRO RGD, ACK resources, and IAMRoleSelectors are all structurally correct and the RDS Aurora cluster provisions successfully. However two blockers prevent DevLake from becoming operational:

---

🔴 Bug 1 — enable_ack_rds label never added to hub cluster secret

The ack-rds ArgoCD Application is never created because the hub cluster Argo secret is missing the enable_ack_rds: "true" label. The ApplicationSet selector requires it:

selector:
  matchExpressions:
    - key: enable_ack_rds
      operator: In
      values: ['true']

In this environment the RDS ACK CRDs happened to already be present from another mechanism, so DBCluster/DBInstance resources were still created — but this will silently break on a clean deploy where the CRDs aren't pre-installed.

Fix: Add enable_ack_rds = "true" to the hub cluster secret labels in Terraform, alongside enable_devlake = "true" (same file that populates the other enable_* annotations).

---

🔴 Bug 2 — Password mismatch after RDS managed secret rotation

mysql-setup-workflow fails with:

ERROR 1045 (28000): Access denied for user 'root'@'10.0.35.131' (using password: YES)

Root cause: The workflow runs once and captures the RDS master password into devlake-mysql-auth.MYSQL_ROOT_PASSWORD at that moment. The devlake-mysql-credentials ExternalSecret has refreshInterval: 1h and syncs the latest RDS-managed secret. After one rotation cycle the two diverge:

• devlake-mysql-credentials.password → current RDS password ✅
• devlake-mysql-auth.MYSQL_ROOT_PASSWORD → password at workflow run time (stale) ❌

Fix: Replace the one-shot kubectl create secret in the create-secrets workflow step with an ExternalSecret that sources MYSQL_ROOT_PASSWORD directly from devlake-mysql-credentials. This keeps the auth secret permanently in sync with RDS rotations.

---

✅ Everything else looks correct

• RDS Aurora cluster provisions and reaches available state ✅
• KRO RelationalDatabase → correctly orchestrates DBSubnetGroup, SecurityGroup, DBCluster, DBInstance via ACK ✅
• devlake-mysql-service ExternalName service points to correct RDS endpoint ✅
• {{ .username }} / {{ .password }} Go template syntax in the KRO RGD ExternalSecret works fine with ESO engineVersion: v2 ✅
• IAMRoleSelector ARN peeks-cluster-mgmt-rds matches the actual IAM role ✅
• devlake-mysql-credentials secret populated with both username and password keys ✅

---

Minor observations (non-blocking)

• create-secrets step mounts mysql-passwords as a volume but reads it via kubectl get secret instead — the volumeMounts entry is dead code
• AWSKeyManagementServicePowerUser on the RDS ACK role is broader than needed; only kms:Decrypt + kms:GenerateDataKey are required
• PushSecret still uses external-secrets.io/v1alpha1 — acceptable for now but worth tracking