diff --git a/.gitignore b/.gitignore index ab1b0dfcb..349fb1a75 100644 --- a/.gitignore +++ b/.gitignore @@ -175,3 +175,9 @@ private/ # Local platform config (user-specific, not committed) config.local.yaml + +# Local scratch / chat session artifacts (never commit) +.local/ + +# Pinned platform artifact clone (OAP-style consumption) +.platform/ diff --git a/.kiro/steering/troubleshooting.md b/.kiro/steering/troubleshooting.md index 659c40256..e57410161 100644 --- a/.kiro/steering/troubleshooting.md +++ b/.kiro/steering/troubleshooting.md @@ -266,6 +266,86 @@ annotations: For Kargo specifically this lives in `gitops/overlays/environments/control-plane/kargo/values.yaml` under `api.ingress.annotations`. +### CloudFront platform URLs hang (curl 000): VPC origin points to a stale/recreated ALB + +**Symptom**: In cloudfront exposure mode, every platform URL (`https://$CF/keycloak/...`, +`/backstage`, `/grafana`, `/argo-workflows`, ...) times out with curl **`000`** — not `404`, +not `5xx`. `idc:configure` loops forever on "Keycloak not ready" (the SAML descriptor never +returns 200). A **direct in-cluster** curl to the internal ALB DNS returns `200/302`, proving +the ALB, targets and LBC listener rules are healthy — only the **CloudFront → ALB** hop fails. + +**Root cause**: The platform ALB (`-platform`) was **deleted and recreated** (new +ARN/DNS), but the CloudFront **VPC Origin is still bound to the OLD (deleted) ALB ARN**. +`hub-distribution` only acts when no `-platform` distribution exists yet, so on a re-run it +**skips** and never re-points the VPC origin → CloudFront keeps sending traffic to a dead ALB → +the connection hangs → `000`. + +Who recreates the ALB: the **AWS Load Balancer Controller**. When the pre-created ALB's +*immutable* attributes don't match the `platform` IngressClassParams, the LBC can't modify them +in place, so it **deletes and recreates** the ALB. The classic trigger is a **scheme mismatch**: +`create-alb` builds the ALB `internal` (required for a CloudFront VPC-Origin backend), but if +`IngressClassParams.spec.scheme` is `internet-facing`, scheme is immutable → LBC delete+recreate +loop. Every recreation orphans the VPC origin. (CloudTrail shows `DeleteLoadBalancer` by +`-LBCPodIdentityRole` with userAgent `elbv2.k8s.aws/...`.) + +**Diagnosis**: +```bash +HUB=peeks-hub # adjust to your resource prefix + +# 1. Direct in-cluster hit to the internal ALB (bypasses CloudFront). 200/302 => ALB is fine, +# so the break is the CloudFront->ALB hop (stale VPC origin), NOT the ALB/targets/rules. +ALB_DNS=$(aws elbv2 describe-load-balancers --names "$HUB-platform" --query 'LoadBalancers[0].DNSName' --output text) +kubectl --context "$HUB" -n keycloak run curltest --rm -i --restart=Never --image=curlimages/curl --command -- \ + curl -s -o /dev/null -w '%{http_code}\n' --max-time 10 "http://$ALB_DNS/keycloak/realms/master" + +# 2. THE check: does the VPC origin's ARN match the CURRENT ALB ARN? (mismatch = stale = the bug) +ALB_ARN=$(aws elbv2 describe-load-balancers --names "$HUB-platform" --query 'LoadBalancers[0].LoadBalancerArn' --output text) +DIST=$(aws cloudfront list-distributions --query "DistributionList.Items[?Comment=='$HUB-platform'].Id" --output text) +VO=$(aws cloudfront get-distribution --id "$DIST" --query 'Distribution.DistributionConfig.Origins.Items[0].VpcOriginConfig.VpcOriginId' --output text) +echo "VPC origin ARN : $(aws cloudfront get-vpc-origin --id "$VO" --query 'VpcOrigin.VpcOriginEndpointConfig.Arn' --output text)" +echo "current ALB ARN: $ALB_ARN" # if these differ, CloudFront points at a dead ALB + +# 3. Confirm the LBC is the one deleting/recreating the ALB (and when) +aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteLoadBalancer \ + --max-results 5 --query 'Events[].{Time:EventTime,User:Username}' --output table +``` + +**Fix**: +- **Stop the churn at the source**: ensure the `platform` IngressClassParams `scheme` is + `internal` in cloudfront mode so the LBC **adopts** the pre-created internal ALB instead of + delete-recreating it (`gitops/addons/registry/core.yaml` `ingress-class-alb.valuesObject.scheme` + + `gitops/addons/charts/ingress-class-alb/templates/ingressclass.yaml`). After this, CloudTrail + should show **no recurring** `DeleteLoadBalancer` events and the ALB ARN stays stable. +- **If a stale VPC origin already exists**, re-point CloudFront at the current ALB: create a NEW + VPC origin for the current ALB ARN → update the distribution's origin to the new `VpcOriginId` + and current ALB DNS → wait for `Deployed` → delete the old VPC origin. Note a VPC origin + **cannot be updated in place while attached to a distribution** (`CannotUpdateEntityWhileInUse`), + so you must create-new / swap / delete-old rather than edit its ARN. + +**Prevention (root cause) — implemented**: +- `IngressClassParams.spec.scheme = internal` in cloudfront mode so the LBC adopts the internal + ALB in place instead of delete-recreating it (scheme is immutable). +- `create-alb` selects PRIVATE subnets (by `kubernetes.io/role/internal-elb` tag, then by + `MapPublicIpOnLaunch==false`) and **tags them `kubernetes.io/role/internal-elb=1`**, so the LBC + discovers the same subnet set and does not `SetSubnets`/churn the ALB. See + `cluster-providers/common/Taskfile.cloudfront.yaml` (`create-alb`). + +**Future hardening (NOT yet implemented) — `cloudfront:sync-vpc-origin` reconcile**: +If an ALB is ever recreated despite the above (re-runs, other immutable-attr drift), a reconcile +task would self-heal the stale origin. Proposed design (idempotent; no-op when the ARN already +matches, so cheap on healthy installs): +1. Guard on `EXPOSURE_MODE == cloudfront`; resolve current `-platform` ALB ARN + DNS; resolve + the distribution (`Comment == -platform`) and its origin `VpcOriginId` → that VPC origin's ARN. +2. If `VPC-origin ARN == current ALB ARN` → **no-op** (done). +3. Else drift → create-new / swap / delete-old (a VPC origin can't be edited in place while + attached): `create-vpc-origin` for the current ALB → poll `Deployed` → `update-distribution` + origin to the new `VpcOriginId` + current ALB DNS → poll `Deployed` → `delete-vpc-origin` (old) + → re-authorize `CloudFront-VPCOrigins-Service-SG → :80`. +Wire into `install:phase1-cloudfront` (both providers) after `sync-domain`. Cost is one or two +CloudFront deploys (~5–15 min each) ONLY on drift; a no-op otherwise. Deliberately left +unimplemented for now — the prevention above should keep the ALB stable; add this only if churn +recurs. + ## ACK (AWS Controllers for Kubernetes) Issues ### ACK "scheduled for deletion" loop @@ -316,6 +396,46 @@ The `services.k8s.aws/force-reconcile` annotation does NOT always work (especial **Fix**: Delete the CRD manually (`kubectl delete crd .kro.run`), then sync to let KRO recreate it. **Warning**: This deletes all instances of that CRD — may trigger resource deletion in AWS. Only do this when safe. +### Hub kro/ack capability RBAC-denied on ACK resources (kro-provisioned spokes) +**Symptoms**: A spoke provisioned via the **KRO path** never gets its EKS cluster. The +`EksclusterWithVpc` instance on the hub shows a reconcile error like: +``` +vpcs.ec2.services.k8s.aws "-vpc" is forbidden: User +"arn:aws:sts:::assumed-role/-kro-capability-role/KRO" cannot get resource +"vpcs" in API group "ec2.services.k8s.aws" in the namespace "" +``` + +**Root cause**: A kro-provisioned spoke's `EksclusterWithVpc` claim is reconciled by the +**hub's** kro capability (`-kro-capability-role/KRO`), which renders the ACK +VPC/subnet/EKS CRs. The hub's kro/ack capability roles are NOT cluster-admin (they get +`AmazonEKSClusterPolicy` / inline ACK policies, mirroring the kro-ack RGD). The +`eks-capabilities-rbac` ClusterRole that grants the ACK API groups only targeted +`enable_kro_manifests` (spokes); the hub uses `enable_kro_manifests_hub`, so the hub never +got the RBAC. The chart comment "redundant on the hub because the capability role has +ClusterAdminPolicy" is wrong — the hub kro/ack cap is not cluster-admin in either the +crossplane or kro-ack flow. This is a **shared latent gap**; it surfaces on whichever flow +first exercises a KRO-provisioned spoke (crossplane: `spoke-prod`). + +**Fix (GitOps)**: `eks-capabilities-rbac-hub` registry entry (`gitops/addons/registry/platform.yaml`), +gated on `enable_kro_manifests_hub`, deploys the same `eks-capabilities-kro` ClusterRole+Binding +on the hub, bound to `-kro-capability-role/KRO` and `-ack-capability-role/ACK`. + +**Applying to an already-running hub**: the hub's `addonsRepoRevision` tracks the branch, so a +hard-refresh of the `cluster-addons` ApplicationSet regenerates it and syncs +`eks-capabilities-rbac-hub-` automatically: +```bash +kubectl -n argocd annotate applicationset cluster-addons argocd.argoproj.io/refresh=hard --overwrite +``` +(Or apply the rendered ClusterRole/Binding directly for an immediate unblock.) + +**Manual recovery — KRO does NOT self-heal the stuck ACK object**: after the RBAC is fixed, +KRO will not reconcile *over* the ACK resource that was left half-created during the denied +window. Delete the stuck ACK object so KRO recreates it cleanly: +```bash +kubectl --context delete vpcs.ec2.services.k8s.aws -vpc -n +# KRO recreates it (now permitted) and the spoke provisioning resumes. +``` + ## Crossplane Issues ### NAT Gateway reference resolution race condition diff --git a/ITERATION_PLAN.md b/ITERATION_PLAN.md index 669dd4244..5d8cc9633 100644 --- a/ITERATION_PLAN.md +++ b/ITERATION_PLAN.md @@ -162,17 +162,19 @@ The root Taskfile delegates to providers but each provider implements tasks diff --- -### 10. Remove ArgoCD capability create/delete Job when Crossplane supports it +### 10. Remove ArgoCD capability create/delete Job when Crossplane supports it ✅ DONE **Priority:** Low **Labels:** enhancement, tech-debt **Ref:** https://github.com/crossplane-contrib/provider-upjet-aws/pull/2015 -The `create-capability.yaml` and `delete-capability.yaml` Jobs use AWS CLI to manage the EKS ArgoCD Capability because Crossplane's EKS provider doesn't support it yet. Once `provider-upjet-aws` adds `EKSCapability` support: -- Replace Jobs with Crossplane managed resources -- Remove `manifests/argocd/create-capability.yaml` and `delete-capability.yaml` -- Remove `argocd:capability` and `argocd:delete-capability` tasks -- Add capability to the PlatformCluster composition or as a separate claim +**Done** (provider-aws-eks v2.6.1 adds the `Capability` MR). The EKS Capabilities +(KRO/ACK/ArgoCD) are now created declaratively: +- Crossplane path: native `Capability` MRs in the `platform-cluster` Composition, + gated by `spec.capabilities..enabled` (hub + spokes). +- The imperative `create-capability.yaml` / `delete-capability.yaml` Jobs and the + `argocd:capability` / `argocd:delete-capability` / `spokes:create-capabilities` + tasks were removed; `hub:seed` waits for the Capability MRs to be Ready. --- diff --git a/README.md b/README.md index 3db0e1876..bf13b8b9c 100644 --- a/README.md +++ b/README.md @@ -132,13 +132,13 @@ helper: ```bash # Idempotent: skips if a valid config.local.yaml already exists -./create-config.sh +workshop/create-config.sh # Force overwrite -FORCE=true ./create-config.sh +FORCE=true workshop/create-config.sh # Override the branch (e.g. when WORKSHOP_GIT_BRANCH points to a release tag) -REPO_REVISION=feature/cloudfront-on-agent-platform ./create-config.sh +REPO_REVISION=... workshop/create-config.sh ``` This is a standalone script (not a Taskfile task) because every task reads diff --git a/Taskfile.yaml b/Taskfile.yaml index 13a7de9ff..bea1889bc 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -25,6 +25,12 @@ includes: vars: CONFIG_FILE: "{{.CONFIG_FILE}}" optional: true + kind-kro-ack: + taskfile: ./cluster-providers/kind-kro-ack/Taskfile.yaml + dir: ./cluster-providers/kind-kro-ack + vars: + CONFIG_FILE: "{{.CONFIG_FILE}}" + optional: true tasks: default: diff --git a/applications/java/deployment/templates/kro/application.yaml b/applications/java/deployment/templates/kro/application.yaml index b22eea565..a5c3b8ff2 100644 --- a/applications/java/deployment/templates/kro/application.yaml +++ b/applications/java/deployment/templates/kro/application.yaml @@ -21,7 +21,6 @@ spec: enabled: true path: /java-app rewrite: true - healthcheckPath: /java-app/ functionalGate: enabled: true image: "httpd:alpine" diff --git a/applications/java/kargo/deploy-kargo.sh b/applications/java/kargo/deploy-kargo.sh index 02025b841..a0d4e8bb4 100755 --- a/applications/java/kargo/deploy-kargo.sh +++ b/applications/java/kargo/deploy-kargo.sh @@ -33,9 +33,8 @@ if ! aws iam get-role --role-name "$KARGO_ROLE" --region "$AWS_REGION" >/dev/nul echo " ✓ IAM role $KARGO_ROLE created" fi KARGO_ROLE_ARN=$(aws iam get-role --role-name "$KARGO_ROLE" --query 'Role.Arn' --output text --region "$AWS_REGION") - if ! aws eks list-pod-identity-associations --cluster-name "$HUB_CLUSTER" --region "$AWS_REGION" \ - --query "associations[?namespace=='kargo'&&serviceAccount=='kargo-controller']" --output text 2>/dev/null | grep -q .; then + --query "associations[?namespace=='kargo'&&serviceAccount=='kargo-controller']" --output text 2>/dev/null | grep -q .; then aws eks create-pod-identity-association \ --cluster-name "$HUB_CLUSTER" --namespace kargo --service-account kargo-controller \ --role-arn "$KARGO_ROLE_ARN" --region "$AWS_REGION" >/dev/null @@ -45,7 +44,6 @@ if ! aws eks list-pod-identity-associations --cluster-name "$HUB_CLUSTER" --regi kubectl rollout status deployment -n kargo --timeout=60s 2>/dev/null || true fi - # Deploy the project (creates namespace) echo "Creating Kargo project..." envsubst < project.yaml | kubectl apply -f - diff --git a/applications/java/kargo/project.yaml b/applications/java/kargo/project.yaml index ec4a5a80f..fec1eb182 100644 --- a/applications/java/kargo/project.yaml +++ b/applications/java/kargo/project.yaml @@ -5,6 +5,13 @@ metadata: annotations: argocd.argoproj.io/sync-wave: "-1" --- +# Auto-promotion enabled for both stages. This is required because Kargo v1.7.5 +# has a webhook bug that blocks manually-created Promotion resources when stages +# use inline `uses:` steps (the webhook incorrectly reports "Stage defines no +# promotion steps"). The controller bypasses the webhook for auto-promotions, +# so enabling auto-promotion is the reliable path. +# +# Freight flows: Warehouse → dev (auto) → prod (auto, after dev verifies). apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: @@ -14,4 +21,7 @@ spec: promotionPolicies: - stageSelector: name: dev - autoPromotionEnabled: true # Enable auto-promotion for dev \ No newline at end of file + autoPromotionEnabled: true + - stageSelector: + name: prod + autoPromotionEnabled: true diff --git a/applications/java/kargo/promotiontask.yaml b/applications/java/kargo/promotiontask.yaml index 6c7f51613..4b91bbfd4 100644 --- a/applications/java/kargo/promotiontask.yaml +++ b/applications/java/kargo/promotiontask.yaml @@ -1,3 +1,8 @@ +# NOTE: This PromotionTask is kept for reference only. It is NOT referenced by +# the Stage resources (stages.yaml uses inline steps instead). Kargo v1.7.5's +# admission webhook incorrectly rejects Stages that only reference a task, and +# the yaml-update step cannot address annotation keys with dots. See stages.yaml +# for the working inline implementation. apiVersion: kargo.akuity.io/v1alpha1 kind: PromotionTask metadata: @@ -32,7 +37,7 @@ spec: updates: - key: spec.components.0.properties.image value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} - - key: metadata.annotations.app\.oam\.dev/publishVersion + - key: metadata.annotations["app.oam.dev/publishVersion"] value: ${{ imageFrom(vars.image).Tag }} - uses: git-commit config: diff --git a/applications/java/kargo/stages.yaml b/applications/java/kargo/stages.yaml index 5beb5df5e..c84aafe7f 100644 --- a/applications/java/kargo/stages.yaml +++ b/applications/java/kargo/stages.yaml @@ -1,3 +1,16 @@ +# Kargo Stages for the Java app. +# +# Promotion steps are inlined (not referenced via PromotionTask) because +# Kargo v1.7.5 admission webhook rejects Stages that only carry a task +# reference ("Stage defines no promotion steps"). Inline steps work reliably +# with both auto-promotion (controller-initiated) and manual promotion. +# +# The publishVersion annotation update is intentionally omitted — Kargo v1.7.5 +# yaml-update does not support keys containing dots (neither bracket notation +# nor dot-escaping works). The image field update alone is sufficient to trigger +# reconciliation for both kro (spec.image) and kubevela (spec.components.0.properties.image) +# paths. Both steps use continueOnError so the promotion succeeds regardless of +# which manifest format is present. apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: @@ -5,7 +18,6 @@ metadata: namespace: java-app-kargo annotations: kargo.akuity.io/color: green - kargo.akuity.io/auto-promotion: "false" spec: requestedFreight: - origin: @@ -15,9 +27,50 @@ spec: direct: true promotionTemplate: spec: + vars: + - name: gitURL + value: ${GITLAB_URL}/${GIT_USERNAME}/java.git + - name: image + value: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/peeks/java steps: - - task: - name: update-image + - uses: git-clone + config: + repoURL: ${{ vars.gitURL }} + checkout: + - branch: main + path: ./src + - uses: yaml-update + as: update-image-kro + continueOnError: true + config: + path: ./src/deployment/${{ ctx.stage }}/application.yaml + updates: + - key: spec.image + value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} + - uses: yaml-update + as: update-image-kubevela + continueOnError: true + config: + path: ./src/deployment/${{ ctx.stage }}/application.yaml + updates: + - key: spec.components.0.properties.image + value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} + - uses: git-commit + config: + path: ./src + message: "Promote ${{ ctx.stage }} to ${{ imageFrom(vars.image).Tag }}" + author: + name: Kargo + email: kargo@akuity.io + - uses: git-push + config: + path: ./src + - uses: argocd-update + config: + apps: + - name: java-${{ ctx.stage }}-cd + sources: + - repoURL: ${{ vars.gitURL }} --- apiVersion: kargo.akuity.io/v1alpha1 kind: Stage @@ -36,6 +89,47 @@ spec: - dev promotionTemplate: spec: + vars: + - name: gitURL + value: ${GITLAB_URL}/${GIT_USERNAME}/java.git + - name: image + value: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/peeks/java steps: - - task: - name: update-image \ No newline at end of file + - uses: git-clone + config: + repoURL: ${{ vars.gitURL }} + checkout: + - branch: main + path: ./src + - uses: yaml-update + as: update-image-kro + continueOnError: true + config: + path: ./src/deployment/${{ ctx.stage }}/application.yaml + updates: + - key: spec.image + value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} + - uses: yaml-update + as: update-image-kubevela + continueOnError: true + config: + path: ./src/deployment/${{ ctx.stage }}/application.yaml + updates: + - key: spec.components.0.properties.image + value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} + - uses: git-commit + config: + path: ./src + message: "Promote ${{ ctx.stage }} to ${{ imageFrom(vars.image).Tag }}" + author: + name: Kargo + email: kargo@akuity.io + - uses: git-push + config: + path: ./src + - uses: argocd-update + config: + apps: + - name: java-${{ ctx.stage }}-cd + sources: + - repoURL: ${{ vars.gitURL }} diff --git a/applications/next-js/deployment/dev/application.yaml b/applications/next-js/deployment/dev/application.yaml index 1d3fbcb56..189c869f7 100644 --- a/applications/next-js/deployment/dev/application.yaml +++ b/applications/next-js/deployment/dev/application.yaml @@ -22,7 +22,6 @@ spec: enabled: true path: /unicorn rewrite: false - healthcheckPath: /unicorn --- apiVersion: v1 kind: Service diff --git a/applications/rust/deployment/templates/kro/application.yaml b/applications/rust/deployment/templates/kro/application.yaml index c20d8bdd3..64cab1f63 100644 --- a/applications/rust/deployment/templates/kro/application.yaml +++ b/applications/rust/deployment/templates/kro/application.yaml @@ -29,7 +29,7 @@ spec: enabled: true path: /rust-app rewrite: true - healthcheckPath: /rust-app/ + healthcheckPath: /collection/FRONT_PAGE ## Uncomment the sections below to enable metrics-driven progressive delivery (Module 30) # functionalGate: # enabled: true diff --git a/applications/rust/deployment/templates/kubevela/application.yaml b/applications/rust/deployment/templates/kubevela/application.yaml index 68573d247..7155930cc 100644 --- a/applications/rust/deployment/templates/kubevela/application.yaml +++ b/applications/rust/deployment/templates/kubevela/application.yaml @@ -43,6 +43,10 @@ spec: properties: domain: "" rewritePath: true + # The Rust app only serves 2xx under /collection/*, so the ALB target + # health check must target a served path (default "/" returns 404 → + # unhealthy targets on the OSS LBC). Uses the healthcheckPath trait param. + healthcheckPath: /collection/FRONT_PAGE http: /rust-app: 8080 diff --git a/cluster-providers/common/Taskfile.ray.yaml b/cluster-providers/common/Taskfile.ray.yaml new file mode 100644 index 000000000..cf0077737 --- /dev/null +++ b/cluster-providers/common/Taskfile.ray.yaml @@ -0,0 +1,126 @@ +version: "3" +set: [errexit, nounset, pipefail] +silent: true + +# Shared Ray ML infrastructure tasks. +# Provider-agnostic — used by both kind-crossplane and kind-kro-ack. +# Required vars: HUB_CLUSTER_NAME, RESOURCE_PREFIX, AWS_ACCOUNT_ID, AWS_REGION, ROOT_DIR +tasks: + setup: + desc: "Set up Ray ML infrastructure: S3 bucket, ECR repo, vLLM image, S3 CSI driver, IAM + Pod Identity" + vars: + RAY_BUCKET: "{{.RESOURCE_PREFIX}}-ray-models-{{.AWS_ACCOUNT_ID}}" + RAY_ECR_REPO: "{{.RESOURCE_PREFIX}}-ray-vllm-custom" + RAY_ROLE_PRESTAGE: "{{.RESOURCE_PREFIX}}-model-prestage-role" + RAY_ROLE_WORKER: "{{.RESOURCE_PREFIX}}-ray-worker-role" + DOCKERFILE_PATH: "{{.ROOT_DIR}}/platform/infra/terraform/common/Dockerfile.ray-vllm" + cmds: + - echo "▸ Setting up Ray ML infrastructure..." + # 1. S3 bucket + - cmd: | + if aws s3api head-bucket --bucket "{{.RAY_BUCKET}}" --region {{.AWS_REGION}} 2>/dev/null; then + echo " ✓ S3 bucket {{.RAY_BUCKET}} exists" + else + aws s3api create-bucket --bucket "{{.RAY_BUCKET}}" --region {{.AWS_REGION}} \ + --create-bucket-configuration LocationConstraint={{.AWS_REGION}} --output text >/dev/null + aws s3api put-public-access-block --bucket "{{.RAY_BUCKET}}" \ + --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true + echo " ✓ S3 bucket {{.RAY_BUCKET}} created" + fi + # 2. ECR repo + - cmd: | + if aws ecr describe-repositories --repository-names "{{.RAY_ECR_REPO}}" --region {{.AWS_REGION}} >/dev/null 2>&1; then + echo " ✓ ECR repo {{.RAY_ECR_REPO}} exists" + else + aws ecr create-repository --repository-name "{{.RAY_ECR_REPO}}" --region {{.AWS_REGION}} \ + --image-tag-mutability MUTABLE --output text >/dev/null + echo " ✓ ECR repo {{.RAY_ECR_REPO}} created" + fi + # 3. Build and push vLLM image (background - non-blocking) + - cmd: | + IMAGE_URI="{{.AWS_ACCOUNT_ID}}.dkr.ecr.{{.AWS_REGION}}.amazonaws.com/{{.RAY_ECR_REPO}}:latest" + if aws ecr describe-images --repository-name "{{.RAY_ECR_REPO}}" --image-ids imageTag=latest --region {{.AWS_REGION}} >/dev/null 2>&1; then + echo " ✓ vLLM image already exists" + else + echo " ▸ Building vLLM image in background (5-10 min)..." + aws ecr get-login-password --region {{.AWS_REGION}} | docker login --username AWS --password-stdin {{.AWS_ACCOUNT_ID}}.dkr.ecr.{{.AWS_REGION}}.amazonaws.com >/dev/null 2>&1 + nohup bash -c "docker build -t '$IMAGE_URI' -f '{{.DOCKERFILE_PATH}}' '$(dirname {{.DOCKERFILE_PATH}})' && docker push '$IMAGE_URI' && echo '[ray] ✓ vLLM image pushed' >> /tmp/ray-build.log" > /tmp/ray-build.log 2>&1 & + echo " ✓ Build started (PID $!, log: /tmp/ray-build.log)" + echo " Run 'task ray:wait-image' to wait for completion." + fi + # 4. S3 CSI driver addon + - cmd: | + if aws eks describe-addon --cluster-name {{.HUB_CLUSTER_NAME}} --addon-name aws-mountpoint-s3-csi-driver --region {{.AWS_REGION}} >/dev/null 2>&1; then + echo " ✓ S3 CSI driver addon installed" + else + aws eks create-addon --cluster-name {{.HUB_CLUSTER_NAME}} --addon-name aws-mountpoint-s3-csi-driver --region {{.AWS_REGION}} --output text >/dev/null + aws eks wait addon-active --cluster-name {{.HUB_CLUSTER_NAME}} --addon-name aws-mountpoint-s3-csi-driver --region {{.AWS_REGION}} 2>/dev/null || true + echo " ✓ S3 CSI driver addon installed" + fi + # 5. IAM roles + - cmd: | + TRUST='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"pods.eks.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]}' + S3_RW='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject","s3:ListBucket"],"Resource":["arn:aws:s3:::{{.RAY_BUCKET}}","arn:aws:s3:::{{.RAY_BUCKET}}/*"]}]}' + S3_RO='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject","s3:ListBucket"],"Resource":["arn:aws:s3:::{{.RAY_BUCKET}}","arn:aws:s3:::{{.RAY_BUCKET}}/*"]}]}' + for ROLE_INFO in "{{.RAY_ROLE_PRESTAGE}}|$S3_RW" "{{.RAY_ROLE_WORKER}}|$S3_RO"; do + ROLE="${ROLE_INFO%%|*}"; POLICY="${ROLE_INFO#*|}" + if aws iam get-role --role-name "$ROLE" >/dev/null 2>&1; then + echo " ✓ IAM role $ROLE exists" + else + aws iam create-role --role-name "$ROLE" --assume-role-policy-document "$TRUST" --output text >/dev/null + aws iam put-role-policy --role-name "$ROLE" --policy-name s3-access --policy-document "$POLICY" + echo " ✓ IAM role $ROLE created" + fi + done + # 6. Pod Identity associations + - cmd: | + PRESTAGE_ARN="arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/{{.RAY_ROLE_PRESTAGE}}" + WORKER_ARN="arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/{{.RAY_ROLE_WORKER}}" + for SA_INFO in "ray-system|model-prestage-sa|$PRESTAGE_ARN" "ray-system|ray-worker-sa|$WORKER_ARN" "kube-system|s3-csi-driver-sa|$WORKER_ARN"; do + NS="${SA_INFO%%|*}"; REST="${SA_INFO#*|}"; SA="${REST%%|*}"; ARN="${REST#*|}" + if aws eks list-pod-identity-associations --cluster-name {{.HUB_CLUSTER_NAME}} --namespace "$NS" --service-account "$SA" --region {{.AWS_REGION}} --query 'associations[0].associationId' --output text 2>/dev/null | grep -q "^a-"; then + echo " ✓ PIA $SA in $NS exists" + else + aws eks create-pod-identity-association --cluster-name {{.HUB_CLUSTER_NAME}} --namespace "$NS" --service-account "$SA" --role-arn "$ARN" --region {{.AWS_REGION}} --output text >/dev/null 2>&1 + echo " ✓ PIA $SA in $NS created" + fi + done + - echo "✓ Ray ML infrastructure ready (bucket={{.RAY_BUCKET}}, image={{.RAY_ECR_REPO}}:latest)" + + prestage-models: + desc: "Download an ML model to the Ray S3 bucket (MODEL_REPO_ID + MODEL_NAME override the default)" + vars: + RAY_BUCKET: "{{.RESOURCE_PREFIX}}-ray-models-{{.AWS_ACCOUNT_ID}}" + # Generic default; consumers override (e.g. workshop passes TinyLlama). + MODEL_REPO_ID: '{{.MODEL_REPO_ID | default "TinyLlama/TinyLlama-1.1B-Chat-v1.0"}}' + MODEL_NAME: '{{.MODEL_NAME | default "tinyllama"}}' + cmds: + - cmd: | + pip3 install -q huggingface_hub 2>/dev/null + echo "Downloading {{.MODEL_REPO_ID}}..." + python3 -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='{{.MODEL_REPO_ID}}', local_dir='/tmp/{{.MODEL_NAME}}', local_dir_use_symlinks=False)" + echo "Uploading to s3://{{.RAY_BUCKET}}/models/{{.MODEL_NAME}}/..." + aws s3 sync /tmp/{{.MODEL_NAME}}/ s3://{{.RAY_BUCKET}}/models/{{.MODEL_NAME}}/ --region {{.AWS_REGION}} --no-progress + rm -rf /tmp/{{.MODEL_NAME}} + echo "✓ {{.MODEL_REPO_ID}} uploaded to models/{{.MODEL_NAME}}/" + + + wait-image: + desc: "Wait for the background vLLM image build to complete" + vars: + RAY_ECR_REPO: "{{.RESOURCE_PREFIX}}-ray-vllm-custom" + cmds: + - cmd: | + echo "Waiting for vLLM image build..." + for i in $(seq 1 60); do + if aws ecr describe-images --repository-name "{{.RAY_ECR_REPO}}" --image-ids imageTag=latest --region {{.AWS_REGION}} >/dev/null 2>&1; then + echo "✓ vLLM image ready in ECR" + exit 0 + fi + if [ -f /tmp/ray-build.log ]; then + tail -1 /tmp/ray-build.log + fi + sleep 10 + done + echo "✗ Timeout waiting for image (check /tmp/ray-build.log)" + exit 1 diff --git a/cluster-providers/kind-crossplane/README.md b/cluster-providers/kind-crossplane/README.md index d296148a5..4012a5794 100644 --- a/cluster-providers/kind-crossplane/README.md +++ b/cluster-providers/kind-crossplane/README.md @@ -36,10 +36,10 @@ task install a. crossplane:helm Install Crossplane (version from addons/registry/platform.yaml) b. crossplane:providers Render crossplane-base chart (providers + functions only, no ProviderConfig) c. crossplane:provider-config Apply bootstrap ProviderConfig (aws-credentials secret) - d. crossplane:claims Apply XRD/Composition, PlatformCluster claim, pod identities (see below) + d. crossplane:claims Apply XRD/Composition (incl. KRO/ACK/ArgoCD Capability MRs), PlatformCluster claim, pod identities (see below) 6. hub:seed a. Wait for EKS cluster, IAM roles, and pod identities to become Ready - b. argocd:capability Create EKS ArgoCD Capability via Job + b. (wait capabilities) Wait for the KRO/ACK/ArgoCD Capability MRs (created by the Composition) to be Ready c. secrets-manager:seed Write hub config to AWS Secrets Manager d. secrets-manager:seed-keycloak Seed keycloak passwords into Secrets Manager e. hub:install-eso Helm install External Secrets on the hub @@ -102,7 +102,6 @@ Supporting manifests in `manifests/`: | Path | Purpose | |------|---------| -| `argocd/create-capability.yaml` | Job that calls the EKS API to create the ArgoCD Capability | | `argocd/appproject.yaml` | ArgoCD AppProject definition | | `crossplane/provider-config-bootstrap.yaml` | ProviderConfig using the `aws-credentials` secret (Kind-only, not used on hub) | | `external-secrets/cluster-secret-store.yaml` | ClusterSecretStore for AWS Secrets Manager | @@ -126,63 +125,25 @@ When using `oss` mode, the provider must also: 4. Create the `aws-load-balancer-controller-sa` service account in `kube-system` on the hub 5. Add `alb.ingress.kubernetes.io/target-type: ip` to ingresses using ClusterIP services -## Exposure Modes & Load Balancer Pre-Creation +## Exposure (domain + TLS) -The platform supports two exposure modes, selected by the `domain` field in -`config.local.yaml` (resolved into `EXPOSURE_MODE` in the Taskfile): +The platform never provisions a domain or a TLS edge — the **consumer** owns both. +`config.local.yaml` provides: -| Mode | When | How traffic reaches the platform | -|------|------|----------------------------------| -| `domain` | a real domain is configured | Ingress/Service LBs are reached directly (Route53/own DNS) | -| `cloudfront` (default) | no domain configured | CloudFront distributions front the platform ALB and the GitLab NLB | +| Field | Meaning | +|-------|---------| +| `domain` | Ingress hostname (Route53 / CloudFront / anything the consumer provisions). Used for ingress host rules + printed URLs. The platform never creates it. | +| `insecure` | `false` (default) → the platform ALB serves **HTTPS** with an ACM cert (auto-discovered by host, or `certificateArn`). `true` → the ALB serves plain **HTTP** and the consumer terminates TLS upstream (e.g. CloudFront in front of the ALB). | +| `certificateArn` | Optional ACM ARN for the HTTPS listener when `insecure: false`. | -### Why load balancers are pre-created (cloudfront mode) +The ingress templates render `HTTP:80` (insecure) or `HTTPS:443 + ssl-redirect + host` +(secure) from `.Values.global.insecure`. The AWS Load Balancer Controller creates and owns +the shared platform ALB normally (addon Ingresses join the `platform` group via +`alb.ingress.kubernetes.io/group.name: platform`); the platform does **not** pre-create load +balancers or CloudFront. -A CloudFront distribution needs a stable **origin DNS name** at creation time, but the -addon Ingresses/Services that would normally create those load balancers don't sync until -much later in `task install`. To break this chicken-and-egg, `cloudfront:setup-exposure` -(in `Taskfile.cloudfront.yaml`, run early in the install flow) **pre-creates** the load -balancers and the CloudFront distributions that point at them: - -- `cloudfront:create-alb` → ALB `-platform` (+ placeholder HTTP:80 404 listener) -- `cloudfront:hub-distribution` → CloudFront for the ALB; DNS saved to `private/cloudfront-domain` -- `cloudfront:gitlab-nlb` → NLB `-gitlab` -- `cloudfront:gitlab-distribution` → CloudFront for the NLB; DNS saved to `private/gitlab-cloudfront-domain` - -`cloudfront:sync-origins` is an idempotent safety net that re-points each distribution if the -LB DNS ever changes. - -### How the addons adopt the pre-created LBs (tag-based, NOT an annotation) - -The pre-created LBs are **adopted** by the AWS Load Balancer Controller when the addon -Ingresses/Services later reconcile — there is **no** `elbv2.k8s.aws/resource-id` annotation. -Adoption works because each LB is created with the exact resource **tags** the LBC uses to -identify resources it owns: - -**Platform ALB** (shared by Keycloak, Backstage, Grafana, Argo Workflows, JupyterHub, …): -``` -elbv2.k8s.aws/cluster = -ingress.k8s.aws/stack = platform # the ALB "group" key -ingress.k8s.aws/resource = LoadBalancer -``` -All those addon Ingresses join the same ALB group via -`alb.ingress.kubernetes.io/group.name: platform`. The LBC keys an ALB to a group by the -`ingress.k8s.aws/stack` tag, finds the pre-created ALB, and adopts it instead of creating a -new one. The placeholder 80 listener keeps the ALB valid until the first Ingress appears. - -**GitLab NLB** (adopted by the GitLab `LoadBalancer` Service): -``` -elbv2.k8s.aws/cluster = -service.k8s.aws/stack = gitlab/gitlab # / -service.k8s.aws/resource = LoadBalancer -``` -The GitLab Service is annotated `service.beta.kubernetes.io/aws-load-balancer-name: -gitlab`; -the LBC matches by name + these `service.k8s.aws/*` tags and adopts the existing NLB. - -> **Important (NLB security groups):** the GitLab NLB is pre-created **with the cluster -> security group**, not with zero SGs. An NLB created with zero SGs can never have SGs added -> later, which would block the LBC from attaching its managed frontend SG when it adopts the -> NLB. The LBC replaces the SG set via `SetSecurityGroups` during reconcile. +Consumers that front the ALB with CloudFront (e.g. the workshop) set `insecure: true` and +provision the CloudFront distribution + VPC origin themselves — see `workshop/`. ## Spoke Cluster Provisioning diff --git a/cluster-providers/kind-crossplane/Taskfile.cloudfront.yaml b/cluster-providers/kind-crossplane/Taskfile.cloudfront.yaml deleted file mode 100644 index 60d06df68..000000000 --- a/cluster-providers/kind-crossplane/Taskfile.cloudfront.yaml +++ /dev/null @@ -1,163 +0,0 @@ -version: "3" -set: [errexit, nounset, pipefail] -silent: true - -# CloudFront exposure-mode tasks (extracted from the main Taskfile). -# Only run when EXPOSURE_MODE=cloudfront (guarded per-task via status:). -# Included by the main Taskfile under the "cloudfront:" namespace. -tasks: - sync-origins: - desc: Ensure CloudFront distributions point to the current ALB/NLB DNS (idempotent safety net) - cmds: - - cmd: | - if [ "{{.EXPOSURE_MODE}}" != "cloudfront" ]; then exit 0; fi - printf '{{.C_STEP}}▸ Syncing CloudFront origins with current LB DNS...{{.C_RESET}}\n' - # Platform ALB - ALB_DNS=$(aws elbv2 describe-load-balancers --names "{{.HUB_CLUSTER_NAME}}-platform" --region {{.AWS_REGION}} --query 'LoadBalancers[0].DNSName' --output text 2>/dev/null) - DIST_ID=$(aws cloudfront list-distributions --query "DistributionList.Items[?Comment=='{{.HUB_CLUSTER_NAME}}-platform'].Id" --output text 2>/dev/null) - if [ -n "$DIST_ID" ] && [ "$DIST_ID" != "None" ] && [ -n "$ALB_DNS" ] && [ "$ALB_DNS" != "None" ]; then - CURRENT_ORIGIN=$(aws cloudfront get-distribution-config --id "$DIST_ID" --query 'DistributionConfig.Origins.Items[0].DomainName' --output text) - if [ "$CURRENT_ORIGIN" != "$ALB_DNS" ]; then - echo "Updating platform CloudFront origin: $CURRENT_ORIGIN → $ALB_DNS" - ETAG=$(aws cloudfront get-distribution-config --id "$DIST_ID" --query 'ETag' --output text) - aws cloudfront get-distribution-config --id "$DIST_ID" --query 'DistributionConfig' --output json | \ - jq --arg dns "$ALB_DNS" '.Origins.Items[0].DomainName = $dns' > /tmp/cf-platform-config.json - aws cloudfront update-distribution --id "$DIST_ID" --if-match "$ETAG" --distribution-config file:///tmp/cf-platform-config.json > /dev/null - echo "Platform CloudFront updated" - else - echo "Platform CloudFront origin OK" - fi - fi - # GitLab NLB - NLB_DNS=$(aws elbv2 describe-load-balancers --names "{{.HUB_CLUSTER_NAME}}-gitlab" --region {{.AWS_REGION}} --query 'LoadBalancers[0].DNSName' --output text 2>/dev/null) - DIST_ID=$(aws cloudfront list-distributions --query "DistributionList.Items[?Comment=='{{.HUB_CLUSTER_NAME}}-gitlab'].Id" --output text 2>/dev/null) - if [ -n "$DIST_ID" ] && [ "$DIST_ID" != "None" ] && [ -n "$NLB_DNS" ] && [ "$NLB_DNS" != "None" ]; then - CURRENT_ORIGIN=$(aws cloudfront get-distribution-config --id "$DIST_ID" --query 'DistributionConfig.Origins.Items[0].DomainName' --output text) - if [ "$CURRENT_ORIGIN" != "$NLB_DNS" ]; then - echo "Updating GitLab CloudFront origin: $CURRENT_ORIGIN → $NLB_DNS" - ETAG=$(aws cloudfront get-distribution-config --id "$DIST_ID" --query 'ETag' --output text) - aws cloudfront get-distribution-config --id "$DIST_ID" --query 'DistributionConfig' --output json | \ - jq --arg dns "$NLB_DNS" '.Origins.Items[0].DomainName = $dns' > /tmp/cf-gitlab-config.json - aws cloudfront update-distribution --id "$DIST_ID" --if-match "$ETAG" --distribution-config file:///tmp/cf-gitlab-config.json > /dev/null - echo "GitLab CloudFront updated" - else - echo "GitLab CloudFront origin OK" - fi - fi - - setup-exposure: - desc: Pre-create ALB and CloudFront distributions (cloudfront mode only) - status: - - '[ "{{.EXPOSURE_MODE}}" != "cloudfront" ]' - cmds: - - task: create-alb - - task: hub-distribution - - task: gitlab-nlb - - task: gitlab-distribution - create-alb: - desc: Pre-create the platform ALB for CloudFront (cloudfront mode only) - status: - - aws elbv2 describe-load-balancers --names {{.HUB_CLUSTER_NAME}}-platform --region {{.AWS_REGION}} 2>/dev/null - cmds: - - printf '{{.C_STEP}}▸ Creating platform ALB...{{.C_RESET}}\n' - - | - set -e - VPC_ID=$(aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.vpcId' --output text) - PUBLIC_SUBNETS=$(aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.subnetIds' --output text | tr '\t' '\n') - PUBLIC_SUBNETS=$(aws ec2 describe-subnets --subnet-ids $PUBLIC_SUBNETS --region {{.AWS_REGION}} --query 'Subnets[?MapPublicIpOnLaunch==`true`].SubnetId' --output text | tr '\t' ' ') - CLUSTER_SG=$(aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text) - SG_NAME="{{.HUB_CLUSTER_NAME}}-platform-alb-sg" - SG_ID=$(aws ec2 describe-security-groups --filters "Name=group-name,Values=$SG_NAME" "Name=vpc-id,Values=$VPC_ID" --region {{.AWS_REGION}} --query 'SecurityGroups[0].GroupId' --output text 2>/dev/null) - if [ "$SG_ID" = "None" ] || [ -z "$SG_ID" ]; then - SG_ID=$(aws ec2 create-security-group --group-name "$SG_NAME" --description "Platform ALB" --vpc-id "$VPC_ID" --region {{.AWS_REGION}} --query 'GroupId' --output text) - aws ec2 authorize-security-group-ingress --group-id "$SG_ID" --protocol tcp --port 80 --cidr 0.0.0.0/0 --region {{.AWS_REGION}} >/dev/null - aws ec2 authorize-security-group-ingress --group-id "$SG_ID" --protocol tcp --port 443 --cidr 0.0.0.0/0 --region {{.AWS_REGION}} >/dev/null - aws ec2 authorize-security-group-ingress --group-id "$CLUSTER_SG" --protocol -1 --source-group "$SG_ID" --region {{.AWS_REGION}} 2>/dev/null || true - fi - ALB_ARN=$(aws elbv2 create-load-balancer \ - --name "{{.HUB_CLUSTER_NAME}}-platform" \ - --subnets $PUBLIC_SUBNETS \ - --security-groups "$SG_ID" \ - --scheme internet-facing \ - --type application \ - --tags Key=elbv2.k8s.aws/cluster,Value={{.HUB_CLUSTER_NAME}} Key=ingress.k8s.aws/stack,Value=platform Key=ingress.k8s.aws/resource,Value=LoadBalancer \ - --region {{.AWS_REGION}} \ - --query 'LoadBalancers[0].LoadBalancerArn' --output text) - aws elbv2 create-listener \ - --load-balancer-arn "$ALB_ARN" \ - --protocol HTTP --port 80 \ - --default-actions "Type=fixed-response,FixedResponseConfig={MessageBody=Not Found,StatusCode=404,ContentType=text/plain}" \ - --region {{.AWS_REGION}} > /dev/null - ALB_DNS=$(aws elbv2 describe-load-balancers --load-balancer-arns "$ALB_ARN" --region {{.AWS_REGION}} --query 'LoadBalancers[0].DNSName' --output text) - echo "ALB created: $ALB_DNS" - - printf '{{.C_OK}}✓ Platform ALB ready.{{.C_RESET}}\n' - - hub-distribution: - desc: Create CloudFront distribution for the platform ALB (idempotent) - status: - - aws cloudfront list-distributions --query "DistributionList.Items[?Comment=='{{.HUB_CLUSTER_NAME}}-platform'].Id" --output text | grep -q . - cmds: - - printf '{{.C_STEP}}▸ Creating platform CloudFront distribution...{{.C_RESET}}\n' - - | - set -e - ALB_DNS=$(aws elbv2 describe-load-balancers --names "{{.HUB_CLUSTER_NAME}}-platform" --region {{.AWS_REGION}} --query 'LoadBalancers[0].DNSName' --output text) - CF_CONFIG=$(cat <<'CFEOF' - {"CallerReference":"{{.HUB_CLUSTER_NAME}}-TIMESTAMP","Comment":"{{.HUB_CLUSTER_NAME}}-platform","Enabled":true,"Origins":{"Quantity":1,"Items":[{"Id":"alb-origin","DomainName":"ALB_DNS_PLACEHOLDER","CustomOriginConfig":{"HTTPPort":80,"HTTPSPort":443,"OriginProtocolPolicy":"http-only","OriginSslProtocols":{"Quantity":1,"Items":["TLSv1.2"]}}}]},"DefaultCacheBehavior":{"TargetOriginId":"alb-origin","ViewerProtocolPolicy":"redirect-to-https","AllowedMethods":{"Quantity":7,"Items":["GET","HEAD","OPTIONS","PUT","POST","PATCH","DELETE"],"CachedMethods":{"Quantity":2,"Items":["GET","HEAD"]}},"CachePolicyId":"4135ea2d-6df8-44a3-9df3-4b5a84be39ad","OriginRequestPolicyId":"216adef6-5c7f-47e4-b989-5492eafa07d3","Compress":true},"ViewerCertificate":{"CloudFrontDefaultCertificate":true},"PriceClass":"PriceClass_100"} - CFEOF - ) - CF_CONFIG=$(echo "$CF_CONFIG" | sed "s|ALB_DNS_PLACEHOLDER|$ALB_DNS|g; s|TIMESTAMP|$(date +%s)|g") - CF_DOMAIN=$(aws cloudfront create-distribution --distribution-config "$CF_CONFIG" --query "Distribution.DomainName" --output text) - mkdir -p {{.ROOT_DIR}}/private - echo -n "$CF_DOMAIN" > {{.ROOT_DIR}}/private/cloudfront-domain - echo "CloudFront: $CF_DOMAIN" - - printf '{{.C_OK}}✓ Platform CloudFront ready.{{.C_RESET}}\n' - - gitlab-nlb: - desc: Pre-create GitLab NLB for CloudFront (cloudfront mode only) - status: - - aws elbv2 describe-load-balancers --names {{.HUB_CLUSTER_NAME}}-gitlab --region {{.AWS_REGION}} 2>/dev/null - cmds: - - printf '{{.C_STEP}}▸ Creating GitLab NLB...{{.C_RESET}}\n' - - | - set -e - VPC_ID=$(aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.vpcId' --output text) - PUBLIC_SUBNETS=$(aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.subnetIds' --output text | tr '\t' '\n') - PUBLIC_SUBNETS=$(aws ec2 describe-subnets --subnet-ids $PUBLIC_SUBNETS --region {{.AWS_REGION}} --query 'Subnets[?MapPublicIpOnLaunch==`true`].SubnetId' --output text | tr '\t' ' ') - # Create WITH a security group: an NLB created with zero SGs can never have - # SGs added later, which blocks the AWS LBC from attaching its managed - # frontend SG when it adopts this NLB. Seed with the cluster SG; the LBC - # replaces the set via SetSecurityGroups during reconcile. - CLUSTER_SG=$(aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text) - NLB_ARN=$(aws elbv2 create-load-balancer \ - --name "{{.HUB_CLUSTER_NAME}}-gitlab" \ - --subnets $PUBLIC_SUBNETS \ - --security-groups "$CLUSTER_SG" \ - --scheme internet-facing \ - --type network \ - --tags Key=elbv2.k8s.aws/cluster,Value={{.HUB_CLUSTER_NAME}} Key=service.k8s.aws/stack,Value=gitlab/gitlab Key=service.k8s.aws/resource,Value=LoadBalancer \ - --region {{.AWS_REGION}} \ - --query 'LoadBalancers[0].LoadBalancerArn' --output text) - NLB_DNS=$(aws elbv2 describe-load-balancers --load-balancer-arns "$NLB_ARN" --region {{.AWS_REGION}} --query 'LoadBalancers[0].DNSName' --output text) - echo "GitLab NLB: $NLB_DNS" - - printf '{{.C_OK}}✓ GitLab NLB ready.{{.C_RESET}}\n' - - gitlab-distribution: - desc: Create CloudFront distribution for GitLab NLB (idempotent) - status: - - aws cloudfront list-distributions --query "DistributionList.Items[?Comment=='{{.HUB_CLUSTER_NAME}}-gitlab'].Id" --output text | grep -q . - cmds: - - printf '{{.C_STEP}}▸ Creating GitLab CloudFront distribution...{{.C_RESET}}\n' - - | - set -e - NLB_DNS=$(aws elbv2 describe-load-balancers --names "{{.HUB_CLUSTER_NAME}}-gitlab" --region {{.AWS_REGION}} --query 'LoadBalancers[0].DNSName' --output text) - CF_CONFIG=$(cat <<'CFEOF' - {"CallerReference":"{{.HUB_CLUSTER_NAME}}-gitlab-TIMESTAMP","Comment":"{{.HUB_CLUSTER_NAME}}-gitlab","Enabled":true,"Origins":{"Quantity":1,"Items":[{"Id":"gitlab-nlb","DomainName":"NLB_DNS_PLACEHOLDER","CustomOriginConfig":{"HTTPPort":80,"HTTPSPort":443,"OriginProtocolPolicy":"http-only","OriginSslProtocols":{"Quantity":1,"Items":["TLSv1.2"]}}}]},"DefaultCacheBehavior":{"TargetOriginId":"gitlab-nlb","ViewerProtocolPolicy":"redirect-to-https","AllowedMethods":{"Quantity":7,"Items":["GET","HEAD","OPTIONS","PUT","POST","PATCH","DELETE"],"CachedMethods":{"Quantity":2,"Items":["GET","HEAD"]}},"CachePolicyId":"4135ea2d-6df8-44a3-9df3-4b5a84be39ad","OriginRequestPolicyId":"216adef6-5c7f-47e4-b989-5492eafa07d3","Compress":true},"ViewerCertificate":{"CloudFrontDefaultCertificate":true},"PriceClass":"PriceClass_100"} - CFEOF - ) - CF_CONFIG=$(echo "$CF_CONFIG" | sed "s|NLB_DNS_PLACEHOLDER|$NLB_DNS|g; s|TIMESTAMP|$(date +%s)|g") - CF_DOMAIN=$(aws cloudfront create-distribution --distribution-config "$CF_CONFIG" --query "Distribution.DomainName" --output text) - mkdir -p {{.ROOT_DIR}}/private - echo -n "$CF_DOMAIN" > {{.ROOT_DIR}}/private/gitlab-cloudfront-domain - echo "GitLab CloudFront: $CF_DOMAIN" - - printf '{{.C_OK}}✓ GitLab CloudFront ready.{{.C_RESET}}\n' - diff --git a/cluster-providers/kind-crossplane/Taskfile.workshop.yaml b/cluster-providers/kind-crossplane/Taskfile.workshop.yaml deleted file mode 100644 index 1263a439a..000000000 --- a/cluster-providers/kind-crossplane/Taskfile.workshop.yaml +++ /dev/null @@ -1,28 +0,0 @@ -version: "3" -set: [errexit, nounset, pipefail] -silent: true - -# Workshop-specific setup tasks (AWS Workshop Studio / kind-crossplane variant). -# Included by the main Taskfile under the "workshop:" namespace. -# These tasks create AWS resources that are required by the workshop content -# but are not part of the platform itself (e.g. model staging buckets). -tasks: - setup: - desc: Create workshop-specific AWS resources (idempotent) - cmds: - - task: create-ray-models-bucket - - create-ray-models-bucket: - desc: Create the Ray model staging S3 bucket for AI/ML modules - status: - - aws s3api head-bucket --bucket {{.HUB_CLUSTER_NAME}}-ray-models-{{.AWS_ACCOUNT_ID}} --region {{.AWS_REGION}} 2>/dev/null - cmds: - - printf '{{.C_STEP}}▸ Creating Ray model staging bucket...{{.C_RESET}}\n' - - | - BUCKET="{{.HUB_CLUSTER_NAME}}-ray-models-{{.AWS_ACCOUNT_ID}}" - aws s3api create-bucket --bucket "$BUCKET" --region {{.AWS_REGION}} \ - {{if ne .AWS_REGION "us-east-1"}}--create-bucket-configuration LocationConstraint={{.AWS_REGION}}{{end}} \ - --no-cli-pager >/dev/null - aws s3api put-public-access-block --bucket "$BUCKET" --region {{.AWS_REGION}} \ - --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" >/dev/null - - printf '{{.C_OK}}✓ s3://{{.HUB_CLUSTER_NAME}}-ray-models-{{.AWS_ACCOUNT_ID}} ready.{{.C_RESET}}\n' diff --git a/cluster-providers/kind-crossplane/Taskfile.yaml b/cluster-providers/kind-crossplane/Taskfile.yaml index 31bd3a249..dac666028 100644 --- a/cluster-providers/kind-crossplane/Taskfile.yaml +++ b/cluster-providers/kind-crossplane/Taskfile.yaml @@ -3,28 +3,14 @@ set: [errexit, nounset, pipefail] silent: true includes: - cloudfront: - taskfile: ./Taskfile.cloudfront.yaml - vars: - AWS_REGION: '{{.AWS_REGION}}' - HUB_CLUSTER_NAME: '{{.HUB_CLUSTER_NAME}}' - ROOT_DIR: '{{.ROOT_DIR}}' - EXPOSURE_MODE: - sh: 'if [ -z "$(yq ''.domain // ""'' {{.ROOT_DIR}}/config.local.yaml)" ]; then echo "cloudfront"; else echo "domain"; fi' - C_STEP: '{{.C_STEP}}' - C_OK: '{{.C_OK}}' - C_RESET: '{{.C_RESET}}' - C_INFO: '{{.C_INFO}}' - workshop: - taskfile: ./Taskfile.workshop.yaml + ray: + taskfile: ../common/Taskfile.ray.yaml vars: AWS_REGION: '{{.AWS_REGION}}' AWS_ACCOUNT_ID: '{{.AWS_ACCOUNT_ID}}' HUB_CLUSTER_NAME: '{{.HUB_CLUSTER_NAME}}' RESOURCE_PREFIX: '{{.RESOURCE_PREFIX}}' - C_STEP: '{{.C_STEP}}' - C_OK: '{{.C_OK}}' - C_RESET: '{{.C_RESET}}' + ROOT_DIR: '{{.ROOT_DIR}}' env: @@ -65,11 +51,20 @@ vars: RESOURCE_PREFIX: sh: yq '.resourcePrefix // ""' {{.CONFIG_FILE}} INGRESS_NAME: - sh: yq '.ingressName // ""' {{.CONFIG_FILE}} + sh: yq '.ingressName // ""' {{.CONFIG_FILE}} | grep -v '^$' || echo "{{.HUB_CLUSTER_NAME}}-platform" INGRESS_SECURITY_GROUPS: sh: yq '.ingressSecurityGroups // ""' {{.CONFIG_FILE}} + # Exposure: the platform does NOT provision a domain or CloudFront. The consumer + # provides `domain` and terminates TLS however they like. `insecure: true` -> the + # ALB serves plain HTTP (consumer fronts it with CloudFront/etc.); false (default) + # -> ALB serves HTTPS with an ACM cert. exposure_mode is derived from insecure for + # the cluster-secret annotation chain the addon charts already consume. + INSECURE: + sh: yq '.insecure // false' {{.CONFIG_FILE}} EXPOSURE_MODE: - sh: 'if [ -z "$(yq ''.domain // ""'' {{.CONFIG_FILE}})" ]; then echo "cloudfront"; else echo "domain"; fi' + sh: 'if [ "$(yq ''.insecure // false'' {{.CONFIG_FILE}})" = "true" ]; then echo "cloudfront"; else echo "domain"; fi' + CERTIFICATE_ARN: + sh: yq '.certificateArn // ""' {{.CONFIG_FILE}} CAPABILITY_NAME: sh: yq '.argocdCapability.name // "argocd"' {{.CONFIG_FILE}} IDC_INSTANCE_ARN: @@ -130,45 +125,44 @@ tasks: - 'printf "{{.C_INFO}} [%ds] kind:create done{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' - task: credentials:setup - 'printf "{{.C_INFO}} [%ds] credentials:setup done{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' - - task: argocd:install - - 'printf "{{.C_INFO}} [%ds] argocd:install done{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' - task: crossplane:setup - 'printf "{{.C_INFO}} [%ds] crossplane:setup done{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' + # The platform does not provision a domain/CloudFront. hub:seed reads the + # consumer-provided `domain` from config into ingress_domain_name. - task: hub:seed - 'printf "{{.C_INFO}} [%ds] hub:seed done{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' + # Force ESO to immediately refresh the cluster secret from Secrets Manager + # so all annotations (exposure_mode, ingress_domain_name, etc.) are populated + # before ArgoCD generates ApplicationSet child apps at wave 5+ + - task: hub:force-eso-refresh - task: hub:wait-for-sync - 'printf "{{.C_INFO}} [%ds] hub:wait-for-sync done{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' # Phase 1 (parallel): CloudFront+identity restart || observability seeding - - task: install:phase1-parallel - - 'printf "{{.C_INFO}} [%ds] phase1-parallel done (cloudfront+restart || observability){{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' + - task: install:post-sync + - 'printf "{{.C_INFO}} [%ds] post-sync done (identity restart || observability seed){{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' - task: hub:wait-for-full-sync - 'printf "{{.C_INFO}} [%ds] hub:wait-for-full-sync done{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' - # Phase 1b (parallel): langfuse restart || urls || gitlab clone — all independent - - task: install:phase1b-parallel - - task: hub:set-overlay-repo - - 'printf "{{.C_INFO}} [%ds] hub:set-overlay-repo done{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' - # Phase 2 (parallel): spoke-dev (crossplane) || spoke-prod (kro) || idc:configure - - task: install:phase2-parallel - - 'printf "{{.C_INFO}} [%ds] phase2-parallel done (spokes || idc){{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' - - | - # Pull local fleet-config so it reflects the spoke overlay changes just pushed - LOCAL_FLEET="$HOME/environment/fleet-config" - if [ -d "$LOCAL_FLEET/.git" ]; then - git -C "$LOCAL_FLEET" pull --ff-only -q 2>/dev/null || true - fi - - 'printf "{{.C_OK}}✓ Bootstrap complete in %ds. Run task status.{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' - - install:phase1b-parallel: + # Wait for ALL Crossplane providers (including ec2) to be Healthy before spoke provisioning + - task: hub:wait-for-providers + # Phase 1b (parallel): langfuse restart || urls — all independent + - task: install:finalize + # NOTE: spoke enablement and fleet-overlay wiring are CONSUMER concerns (the + # consumer owns the gitops/fleet repo). This generic provider provisions the + # hub only and exposes the PlatformCluster abstraction; consumers (e.g. the + # workshop, or OAP) declare spokes against their own repo / via PlatformCluster + # claims. See workshop/ for the reference consumer. + - 'printf "{{.C_OK}}✓ Platform hub bootstrap complete in %ds. Run task status.{{.C_RESET}}\n" "$(($(date +%s) - {{.INSTALL_START}}))"' + + install:finalize: internal: true deps: - hub:restart-langfuse - urls - - gitlab:clone-repos - install:phase1-parallel: + install:post-sync: internal: true deps: - - install:phase1-cloudfront + - install:phase1-identity - install:phase1-playwright-prefetch - install:phase1-observability-bg @@ -210,33 +204,13 @@ tasks: printf '{{.C_OK}}✓ Playwright chromium pre-fetched.{{.C_RESET}}\n' fi - install:phase1-cloudfront: + install:phase1-identity: internal: true cmds: - - task: cloudfront:sync-origins + # Domain is consumer-provided (seeded from config at hub:seed). Nothing to + # sync here anymore — just restart identity-dependent pods. - task: hub:restart-identity-pods - install:phase2-parallel: - internal: true - deps: - - install:phase2-spoke-dev - - install:phase2-spoke-prod - - idc:configure - - install:phase2-spoke-dev: - internal: true - cmds: - - task: spokes:enable-crossplane - vars: { CLI_ARGS: "{{.RESOURCE_PREFIX}}-spoke-dev" } - - task: spokes:create-capabilities - vars: { CLI_ARGS: "{{.RESOURCE_PREFIX}}-spoke-dev" } - - install:phase2-spoke-prod: - internal: true - cmds: - - task: spokes:enable-kro - vars: { CLI_ARGS: "{{.RESOURCE_PREFIX}}-spoke-prod" } - credentials:setup: desc: Create AWS credentials secret (auto-detects EC2 vs local, skips if exists) status: @@ -346,6 +320,8 @@ tasks: status: - kubectl get providers.pkg.crossplane.io provider-aws-iam -o jsonpath='{.status.conditions[?(@.type=="Healthy")].status}' 2>/dev/null | grep -q True - kubectl get providerconfig.kubernetes.crossplane.io default 2>/dev/null + # Also verify pod identity associations exist in AWS (not just provider Healthy) + - aws eks list-pod-identity-associations --cluster-name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --namespace crossplane-system --query 'length(associations)' --output text 2>/dev/null | grep -qv '^0$' vars: K8S_PROVIDER_VERSION: sh: yq '.crossplane-base.valuesObject.providers.kubernetes.version // "v1.2.1"' {{.GITOPS_ROOT}}/addons/registry/platform.yaml @@ -362,12 +338,12 @@ tasks: --set functions.patchAndTransform.version=v0.10.0 --set functions.environmentConfigs.version=v0.3.0 --set functions.celFilter.version=v0.2.0 - --set providers.family.version=v2.5.3 - --set providers.iam.version=v2.5.3 + --set providers.family.version=v2.6.1 + --set providers.iam.version=v2.6.1 --set providers.iam.createIdentity=false - --set providers.eks.version=v2.5.3 + --set providers.eks.version=v2.6.1 --set providers.eks.createIdentity=false - --set providers.ec2.version=v2.5.3 + --set providers.ec2.version=v2.6.1 --set providers.ec2.createIdentity=false --set providers.kubernetes.version={{.K8S_PROVIDER_VERSION}} --set providers.kubernetes.serviceAccount=provider-kubernetes @@ -434,6 +410,13 @@ tasks: --set clusters.hub.kubernetesVersion={{.HUB_K8S_VERSION}} --set clusters.hub.vpcCidr={{.HUB_VPC_CIDR}} --set clusters.hub.resourcePrefix={{.RESOURCE_PREFIX}} + --set-string clusters.hub.accountId={{.AWS_ACCOUNT_ID}} + --set clusters.hub.capabilities.kro.enabled=true + --set clusters.hub.capabilities.ack.enabled=true + --set clusters.hub.capabilities.argocd.enabled=true + --set-string clusters.hub.capabilities.argocd.idcInstanceArn={{.IDC_INSTANCE_ARN}} + --set-string clusters.hub.capabilities.argocd.idcRegion={{.IDC_REGION}} + --set-string clusters.hub.capabilities.argocd.adminGroupId={{.IDC_ADMIN_GROUP_ID}} {{.MNG_SETS}} | kubectl apply -f - - sed "s|CLUSTER_NAME|{{.HUB_CLUSTER_NAME}}|g" claims/argocd-capability-role.yaml | kubectl apply -f - @@ -453,8 +436,8 @@ tasks: --set aws.region={{.AWS_REGION}} --set aws.clusterName={{.HUB_CLUSTER_NAME}} --set aws.accountId={{.AWS_ACCOUNT_ID}} - --set providers.iam.version=v2.5.3 - --set providers.eks.version=v2.5.3 + --set providers.iam.version=v2.6.1 + --set providers.eks.version=v2.6.1 | kubectl apply -f - - helm template crossplane-pod-identity {{.GITOPS_ROOT}}/addons/charts/crossplane-pod-identity --set aws.region={{.AWS_REGION}} @@ -472,7 +455,8 @@ tasks: --query 'cluster.status' --output text 2>/dev/null | grep -q ACTIVE; then printf '{{.C_OK}}✓ Hub cluster already ACTIVE — skipping Crossplane XR wait.{{.C_RESET}}\n' else - kubectl wait --for=condition=Ready --timeout=3500s platformclusters.platform.gitops.io/{{.HUB_CLUSTER_NAME}} -n crossplane-system + kubectl wait --for=condition=Ready --timeout=5400s platformclusters.platform.gitops.io/{{.HUB_CLUSTER_NAME}} -n crossplane-system || \ + printf '{{.C_ERR}}⚠ XR wait timed out — continuing via ACTIVE check below.{{.C_RESET}}\\n' fi - printf '{{.C_STEP}}▸ Verifying hub EKS cluster is ACTIVE...{{.C_RESET}}\n' - | @@ -511,9 +495,14 @@ tasks: - printf '{{.C_STEP}}▸ Waiting for pod identity resources...{{.C_RESET}}\n' - kubectl -n crossplane-system wait --for=condition=Ready --timeout=300s roles.iam.aws.upbound.io --all - kubectl -n crossplane-system wait --for=condition=Ready --timeout=600s podidentityassociations.eks.aws.upbound.io --all - - task: argocd:capability + # EKS Capabilities (argocd/kro/ack) are now created declaratively by the + # platform-cluster Composition (spec.capabilities.* on the hub claim), reconciled + # by Crossplane once the hub EKS cluster is ACTIVE. Wait for them to be Ready here + # so the ArgoCD capability CRDs are served before hub:apply-root-appset — replaces + # the imperative argocd:capability Job (create-capability.yaml). + - printf '{{.C_STEP}}▸ Waiting for EKS Capabilities (argocd/kro/ack) to be Ready...{{.C_RESET}}\n' + - kubectl wait --for=condition=Ready --timeout=900s capabilities.eks.aws.upbound.io --all - task: hub:create-mgmt-roles - - task: cloudfront:setup-exposure - task: secrets-manager:seed - task: secrets-manager:seed-keycloak - task: secrets-manager:seed-secrets @@ -539,7 +528,8 @@ tasks: "Action": ["sts:AssumeRole", "sts:TagSession"], "Condition": {"ArnLike": {"aws:PrincipalArn": [ "arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/{{.ACK_CAPABILITY_ROLE}}", - "arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/{{.RESOURCE_PREFIX}}-*-ack-capability-role" + "arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/{{.RESOURCE_PREFIX}}-ack-*", + "arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/{{.RESOURCE_PREFIX}}-spoke-*-ack-capability-role" ]}} }] } @@ -574,9 +564,7 @@ tasks: "Version": "2012-10-17", "Statement": [ {"Effect": "Allow", "Action": ["eks:*"], "Resource": "*"}, - {"Effect": "Allow", "Action": ["iam:GetRole"], "Resource": "arn:aws:iam::*:role/*"}, - {"Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::*:role/*", - "Condition": {"StringEquals": {"iam:PassedToService": "pods.eks.amazonaws.com"}}} + {"Effect": "Allow", "Action": ["iam:PassRole","iam:GetRole","iam:ListRoleTags"], "Resource": ["arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/{{.RESOURCE_PREFIX}}-*","arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/*-microservice-*"]} ] }' # Allow ACK capability role to assume cluster-mgmt roles @@ -589,7 +577,6 @@ tasks: }] }' - printf '{{.C_OK}}✓ cluster-mgmt roles ready.{{.C_RESET}}\n' - - task: workshop:setup hub:install-eso: desc: Install External Secrets Operator on the hub cluster @@ -669,11 +656,81 @@ tasks: desc: Apply root-appset to the hub cluster cmds: - task: hub:kubeconfig + # Gate: the ArgoCD EKS Capability status can report ACTIVE before its CRDs are served in the + # cluster. root-appset.yaml contains an ApplicationSet, so we must wait until + # applicationsets.argoproj.io is registered and Established before applying, otherwise kubectl + # fails with "no matches for kind ApplicationSet in version argoproj.io/v1alpha1". + - printf '{{.C_STEP}}▸ Waiting for ArgoCD capability CRDs (ApplicationSet/Application) to be served...{{.C_RESET}}\n' + - | + KC={{.ROOT_DIR}}/private/hub-kubeconfig + for i in $(seq 1 60); do + if KUBECONFIG=$KC kubectl get crd applicationsets.argoproj.io applications.argoproj.io >/dev/null 2>&1; then + break + fi + printf '{{.C_INFO}} [%s/60] ArgoCD CRDs not ready yet...{{.C_RESET}}\n' "$i" + sleep 10 + done + if ! KUBECONFIG=$KC kubectl get crd applicationsets.argoproj.io >/dev/null 2>&1; then + printf '{{.C_ERR}}ERROR: ArgoCD ApplicationSet CRD never became available — the argocd EKS Capability has not reconciled.{{.C_RESET}}\n' + exit 1 + fi + KUBECONFIG=$KC kubectl wait --for=condition=Established \ + crd/applicationsets.argoproj.io crd/applications.argoproj.io --timeout=120s - printf '{{.C_STEP}}▸ Applying root-appset to hub...{{.C_RESET}}\n' - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl apply -f {{.GITOPS_ROOT}}/bootstrap/root-appset.yaml - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig - printf '{{.C_OK}}✓ Root appset applied. Hub is now self-managing.{{.C_RESET}}\n' + hub:force-eso-refresh: + desc: Force ESO to immediately refresh cluster secret so annotations are populated before ArgoCD wave 5 apps render + internal: true + cmds: + - task: hub:kubeconfig + - | + KC={{.ROOT_DIR}}/private/hub-kubeconfig + printf '{{.C_STEP}}▸ Forcing ESO refresh of cluster secret annotations...{{.C_RESET}}\n' + # Force-refresh the fleet-secret ExternalSecret so ESO pulls latest metadata from Secrets Manager + ES=$(KUBECONFIG=$KC kubectl -n argocd get externalsecret -o name 2>/dev/null | grep -i fleet-secret | grep {{.HUB_CLUSTER_NAME}} | head -1 || true) + if [ -n "$ES" ]; then + KUBECONFIG=$KC kubectl -n argocd annotate "$ES" \ + force-sync="$(date +%s)" --overwrite >/dev/null 2>&1 || true + # Wait up to 60s for annotations to be populated + for i in $(seq 1 12); do + VAL=$(KUBECONFIG=$KC kubectl get secret {{.HUB_CLUSTER_NAME}} -n argocd \ + -o jsonpath='{.metadata.annotations.exposure_mode}' 2>/dev/null) + [ -n "$VAL" ] && printf '{{.C_OK}}✓ Cluster secret annotations populated (exposure_mode=%s){{.C_RESET}}\n' "$VAL" && break + sleep 5 + done + else + printf '{{.C_INFO}} fleet-secret ESO not found yet — waiting 30s for initial sync{{.C_RESET}}\n' + sleep 30 + fi + - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig + + hub:wait-for-providers: + desc: Wait for all Crossplane providers to be Healthy (ensures ec2/rds/etc. CRDs available before spoke provisioning) + ignore_error: true + cmds: + - task: hub:kubeconfig + - | + KC={{.ROOT_DIR}}/private/hub-kubeconfig + [ ! -f "$KC" ] && KC={{.ROOT_DIR}}/private/kubeconfig + printf '{{.C_STEP}}▸ Waiting for all Crossplane providers to be Healthy...{{.C_RESET}}\n' + for i in $(seq 1 60); do + UNHEALTHY=$(KUBECONFIG=$KC kubectl get providers.pkg.crossplane.io -o jsonpath='{.items[?(@.status.conditions[?(@.type=="Healthy")].status!="True")].metadata.name}' 2>/dev/null | wc -w | tr -d ' ') + TOTAL=$(KUBECONFIG=$KC kubectl get providers.pkg.crossplane.io --no-headers 2>/dev/null | wc -l | tr -d ' ') + [ "$UNHEALTHY" = "0" ] && [ "$TOTAL" -gt 0 ] && printf '{{.C_OK}}✓ All %s Crossplane providers Healthy.{{.C_RESET}}\n' "$TOTAL" && exit 0 + [ $((i % 4)) -eq 0 ] && printf '{{.C_INFO}} [%ds] %s/%s providers Healthy...{{.C_RESET}}\n' "$((i*15))" "$((TOTAL-UNHEALTHY))" "$TOTAL" + # After 5min, force-sync crossplane-base to unblock stuck ArgoCD retry loops + if [ "$i" -eq 20 ]; then + printf '{{.C_WARN}} Providers not all Healthy after 5min — force-syncing crossplane-base...{{.C_RESET}}\n' + KUBECONFIG=$KC kubectl patch application "crossplane-base-{{.HUB_CLUSTER_NAME}}" -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"revision":"HEAD"}}}' 2>/dev/null || true + fi + sleep 15 + done + printf '{{.C_WARN}} Providers not all Healthy after 15min — continuing anyway.{{.C_RESET}}\n' + - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig + hub:wait-for-sync: desc: Wait for hub ArgoCD apps to reach Synced state vars: @@ -728,18 +785,13 @@ tasks: - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig urls: - desc: Export platform URLs to ~/.bashrc.d/platform.sh + desc: Print platform URLs (alias for print-urls) + aliases: [print-urls] vars: - INGRESS_DOMAIN: - sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/cloudfront-domain ]; then - cat {{.ROOT_DIR}}/private/cloudfront-domain - else - echo "{{.DOMAIN}}" - fi + INGRESS_DOMAIN: '{{.DOMAIN}}' GITLAB_DOMAIN: sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then + if [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain else echo "{{.DOMAIN}}" @@ -752,929 +804,40 @@ tasks: sh: aws eks describe-capability --cluster-name {{.HUB_CLUSTER_NAME}} --capability-name argocd --query 'capability.configuration.argoCd.serverUrl' --output text --region {{.AWS_REGION}} 2>/dev/null || echo "" cmds: - | - PLATFORM_FILE="$HOME/.bashrc.d/platform.sh" - update_var() { - local name="$1" value="$2" - if grep -q "^export ${name}=" "$PLATFORM_FILE" 2>/dev/null; then - sed -i "s|^export ${name}=.*|export ${name}=\"${value}\"|" "$PLATFORM_FILE" - else - echo "export ${name}=\"${value}\"" >> "$PLATFORM_FILE" - fi - } - update_var "INGRESS_DOMAIN" "{{.INGRESS_DOMAIN}}" - update_var "BACKSTAGE_URL" "https://{{.INGRESS_DOMAIN}}/backstage" - update_var "KEYCLOAK_ADMIN_URL" "https://{{.INGRESS_DOMAIN}}/keycloak/admin" - update_var "WORKFLOWS_URL" "https://{{.INGRESS_DOMAIN}}/argo-workflows" - # Kargo runs at the platform root (default ingress rule; it has no /kargo prefix). - update_var "KARGO_URL" "https://{{.INGRESS_DOMAIN}}" - update_var "GRAFANA_URL" "https://{{.INGRESS_DOMAIN}}/grafana" - update_var "GITLAB_URL" "https://{{.GITLAB_DOMAIN}}" - update_var "GITLAB_DOMAIN" "{{.GITLAB_DOMAIN}}" - update_var "MODEL_S3_BUCKET" "{{.HUB_CLUSTER_NAME}}-ray-models-{{.AWS_ACCOUNT_ID}}" - update_var "ARGOCD_URL" "{{.ARGOCD_URL}}" - update_var "ARGOCD_DOMAIN" "$(echo '{{.ARGOCD_URL}}' | sed 's|https://||')" - # ARGOCD_SERVER is needed by the ArgoCD CLI (argocd app list etc.). - # Without it every argocd command fails with "Argo CD server address unspecified". - ARGOCD_SERVER_HOST="$(echo '{{.ARGOCD_URL}}' | sed 's|https://||;s|/.*||')" - update_var "ARGOCD_SERVER" "$ARGOCD_SERVER_HOST" - update_var "ARGOCD_OPTS" "--grpc-web" - # Retrieve an initial ArgoCD token via Playwright SSO so the CLI works - # immediately after install without requiring a manual argocd-refresh-token. - # Best-effort: never fail the install on a token retrieval error. - if [ -n "{{.ARGOCD_URL}}" ] && [ "{{.ARGOCD_URL}}" != "None" ]; then - SCRIPT_DIR="{{.ROOT_DIR}}/platform/infra/terraform/scripts" - ARGOCD_TOKEN=$(python3 "$SCRIPT_DIR/argocd_token_automation.py" \ - --url "{{.ARGOCD_URL}}" \ - --username "user1" \ - --password "{{.USER_PASSWORD}}" \ - --output token 2>/tmp/argocd-token-seed.log || echo "") - if [ -n "$ARGOCD_TOKEN" ] && [ "$ARGOCD_TOKEN" != "Failed to retrieve token" ]; then - update_var "ARGOCD_AUTH_TOKEN" "$ARGOCD_TOKEN" - printf ' ArgoCD CLI: token seeded (argocd app list ready)\n' - else - printf ' ArgoCD CLI: token seeding failed — run argocd-refresh-token manually\n' - fi - fi - update_var "KEYCLOAK_ADMIN_PASSWORD" "{{.KC_ADMIN_PASSWORD}}" - update_var "USER_PASSWORD" "{{.USER_PASSWORD}}" - # USER1_PASSWORD: alias of USER_PASSWORD. Workshop content and helper - # scripts (e.g. backstage-auth.sh) reference $USER1_PASSWORD; keep both in - # sync so either name works. - update_var "USER1_PASSWORD" "{{.USER_PASSWORD}}" - update_var "HUB_CLUSTER_NAME" "{{.HUB_CLUSTER_NAME}}" - update_var "GIT_TOKEN" "{{.USER_PASSWORD}}" - update_var "GIT_USERNAME" "user1" - # Spoke DNS: use ALB hostname from spoke clusters (fallback to empty) - DNS_DEV=$(aws elbv2 describe-load-balancers --region {{.AWS_REGION}} --query "LoadBalancers[?contains(LoadBalancerName,'spoke-dev')].DNSName | [0]" --output text 2>/dev/null || echo "") - DNS_PROD=$(aws elbv2 describe-load-balancers --region {{.AWS_REGION}} --query "LoadBalancers[?contains(LoadBalancerName,'spoke-prod')].DNSName | [0]" --output text 2>/dev/null || echo "") - [ "$DNS_DEV" = "None" ] && DNS_DEV="" - [ "$DNS_PROD" = "None" ] && DNS_PROD="" - # Only persist DNS_DEV/DNS_PROD when we actually discovered an ALB. At install - # time no spoke app ingress exists yet, so these are usually empty — and - # writing empty values here would CLOBBER the live per-shell discovery that - # ~/.bashrc.d/platform.sh performs on every interactive shell (it re-reads the - # spoke ingress hostnames). Skipping the empty write lets the dynamic block win. - [ -n "$DNS_DEV" ] && update_var "DNS_DEV" "$DNS_DEV" - [ -n "$DNS_PROD" ] && update_var "DNS_PROD" "$DNS_PROD" - # Argo Workflows URL: content references both WORKFLOWS_URL and ARGOWF_URL; keep - # them in sync. JUPYTERHUB_URL is referenced by the Module 4 AI/ML pages. - update_var "ARGOWF_URL" "https://{{.INGRESS_DOMAIN}}/argo-workflows" - update_var "JUPYTERHUB_URL" "https://{{.INGRESS_DOMAIN}}/jupyterhub" - printf '{{.C_OK}}✓ Platform URLs exported to %s{{.C_RESET}}\n' "$PLATFORM_FILE" + printf '{{.C_OK}}Platform URLs{{.C_RESET}}\n' printf ' Backstage: https://{{.INGRESS_DOMAIN}}/backstage\n' printf ' Keycloak: https://{{.INGRESS_DOMAIN}}/keycloak\n' printf ' Argo WF: https://{{.INGRESS_DOMAIN}}/argo-workflows\n' - printf ' Kargo: https://{{.INGRESS_DOMAIN}}/ (root)\n' + printf ' Kargo: https://{{.INGRESS_DOMAIN}}/\n' printf ' Grafana: https://{{.INGRESS_DOMAIN}}/grafana\n' - printf ' GitLab: https://{{.GITLAB_DOMAIN}}\n' printf ' ArgoCD: {{.ARGOCD_URL}}\n' + printf '\n' + printf '{{.C_INFO}}Credentials{{.C_RESET}}\n' + printf ' User: user1 / {{.USER_PASSWORD}}\n' + printf ' Keycloak: admin / {{.KC_ADMIN_PASSWORD}}\n' - gitlab:clone-repos: - desc: Clone all GitLab repos locally into ~/environment - vars: - GITLAB_DOMAIN_INT: - sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then - cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain - else - echo "{{.DOMAIN}}" - fi - GIT_USERNAME: - sh: yq '.gitUsername // "user1"' {{.CONFIG_FILE}} - GIT_PASSWORD: - sh: yq '.hub.gitlabRootPassword // ""' {{.CONFIG_FILE}} | grep -v '^$' || (aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text 2>/dev/null | jq -r '.user_password // "{{.HUB_CLUSTER_NAME}}"') - REPOS: - sh: echo "fleet-config java rust golang dotnet next-js" - cmds: - - printf '{{.C_STEP}}▸ Cloning GitLab repos to ~/environment...{{.C_RESET}}\n' - - | - BASE_URL="https://{{.GIT_USERNAME}}:{{.GIT_PASSWORD}}@{{.GITLAB_DOMAIN_INT}}/{{.GIT_USERNAME}}" - FAILED="" - # Wait for the gitlab-init Job to complete before attempting any clones. - # The Job creates and seeds all repos; cloning before it finishes captures - # README-only stubs or fails entirely (repos don't exist yet). - printf ' Waiting for gitlab-init Job to complete (up to 20min)...\n' - if ! kubectl --context peeks-hub wait --for=condition=Complete \ - job/gitlab-init -n gitlab --timeout=1200s 2>/dev/null; then - printf ' ⚠ gitlab-init did not complete in 20min — will try clones anyway.\n' - fi - for REPO in {{.REPOS}}; do - if [ "$REPO" = "fleet-config" ]; then - DEST="$HOME/environment/$REPO" - else - DEST="$HOME/environment/applications/$REPO" - fi - REPO_URL="$BASE_URL/$REPO.git" - if [ -d "$DEST/.git" ]; then - # Already cloned. The gitlab-init Job creates each repo with only a - # README first, then pushes the real source/overlay content later in - # the same Job. An earlier clone-repos run (or this one racing ahead) - # may therefore have captured the README-only state. Pull to pick up - # any content pushed after the initial clone — fast-forward only so we - # never clobber participant commits. - echo " $REPO already cloned — pulling latest" - git -C "$DEST" pull --ff-only -q 2>/dev/null || echo " ($REPO: pull skipped — local changes or no upstream)" - continue - fi - # GitLab repos are created asynchronously by the gitlab-init Job (Helm - # post-install hook, which waits for GitLab readiness). Without waiting, - # this task ran before the repos existed and every clone failed, leaving - # ~/environment/applications empty and no local fleet-config checkout. - # Wait (up to ~5min) for each repo to be reachable AND seeded before - # cloning. Reachable alone is not enough: the repo exists with only a - # README the instant it is created, but the application source is pushed - # a few seconds later. Cloning on first-reachable captured the stub - # (README/.git only), leaving e.g. ~/environment/applications/next-js - # without deployable code. Wait until the remote has a non-README file - # (fleet-config is exempt — it is content-seeded differently). - REACHABLE=false - for i in $(seq 1 18); do - if git ls-remote "$REPO_URL" >/dev/null 2>&1; then - if [ "$REPO" = "fleet-config" ]; then REACHABLE=true; break; fi - # Seeded check: shallow-clone to a temp dir and require a non-README - # tracked file, so we don't clone the README-only initial commit. - TMP=$(mktemp -d) - if git clone --depth 1 "$REPO_URL" "$TMP" >/dev/null 2>&1; then - if [ -n "$(cd "$TMP" && git ls-files | grep -v -i '^readme' | head -1)" ]; then - rm -rf "$TMP"; REACHABLE=true; break - fi - fi - rm -rf "$TMP" - fi - sleep 10 - done - if [ "$REACHABLE" != true ]; then - echo " ⚠ $REPO not reachable/seeded after 5min — skipping (create later)." - FAILED="$FAILED $REPO" - continue - fi - mkdir -p "$(dirname "$DEST")" - if git clone "$REPO_URL" "$DEST" 2>/dev/null; then - echo " ✓ $REPO cloned" - else - echo " ⚠ $REPO clone failed" - FAILED="$FAILED$REPO " - fi - done - if [ -n "$FAILED" ]; then - printf '{{.C_ERR}}⚠ Some GitLab repos were not cloned:%s{{.C_RESET}}\n' "$FAILED" - printf '{{.C_INFO}} Re-run later: task kind-crossplane:gitlab:clone-repos{{.C_RESET}}\n' - else - printf '{{.C_OK}}✓ GitLab repos cloned.{{.C_RESET}}\n' - fi - - hub:set-overlay-repo: - desc: >- - Wire overlay_repo_url -> GitLab fleet-config on the hub cluster config so - every overlay-aware Argo app (incl. multi-acct) reads $overlay from - fleet-config instead of falling back to the GitHub addons repo. Runs LATE - (after GitLab + fleet-config exist) because $overlay is a real Argo source: - pointing it at a non-existent/unauthenticated repo breaks ALL overlay-aware - apps. Self-protecting: verifies reachability, registers an ArgoCD repo - credential, flips /config (ESO reconciles the cluster secret), then - verifies the flip resolved and auto-reverts if it did not. Idempotent. - vars: - GITLAB_DOMAIN_INT: - sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then - cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain - else - echo "{{.DOMAIN}}" - fi - GIT_USERNAME: - sh: yq '.gitUsername // "user1"' {{.CONFIG_FILE}} - GIT_PASSWORD: - sh: yq '.hub.gitlabRootPassword // ""' {{.CONFIG_FILE}} | grep -v '^$' || (aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text 2>/dev/null | jq -r '.user_password') - # Root of the fleet-config overlay structure (configs/, overlays/) per the - # GitLab init-job seed. Keep "" unless the overlay repo nests under a prefix. - OVERLAY_REPO_BASEPATH: "" - cmds: - - | - OVERLAY_URL="https://{{.GITLAB_DOMAIN_INT}}/{{.GIT_USERNAME}}/fleet-config.git" - SECRET_KEY="{{.HUB_CLUSTER_NAME}}/config" - KC="{{.ROOT_DIR}}/private/hub-kubeconfig" - - # 1. Reachability guard. $overlay is a REAL Argo source — never point it at - # an unreachable repo. This is one shell block, so exit 0 here correctly - # skips ALL wiring below (the earlier multi-block version did not). - # fleet-config is created asynchronously by the gitlab-init Job, so it - # can be a minute or two behind this step. Previously we skipped on the - # first miss — but then the overlay was never wired to fleet-config and - # the spoke entries that enable-crossplane/enable-kro push later were - # IGNORED by the appset (hub kept reading the GitHub addons fallback), - # so NO spoke EKS clusters were ever provisioned. Wait (up to ~5min) - # for it to become reachable before deciding. - FLEET_LS="https://{{.GIT_USERNAME}}:{{.GIT_PASSWORD}}@{{.GITLAB_DOMAIN_INT}}/{{.GIT_USERNAME}}/fleet-config.git" - printf '{{.C_STEP}}▸ Verifying fleet-config overlay repo is reachable...{{.C_RESET}}\n' - FLEET_OK=false - for i in $(seq 1 30); do - if git ls-remote "$FLEET_LS" >/dev/null 2>&1; then FLEET_OK=true; break; fi - sleep 10 - done - if [ "$FLEET_OK" != true ]; then - printf '{{.C_ERR}}⚠ fleet-config not reachable after 5min — skipping overlay wiring (no change made).{{.C_RESET}}\n' - printf '{{.C_INFO}} Re-run later: task kind-crossplane:hub:set-overlay-repo{{.C_RESET}}\n' - exit 0 - fi - printf '{{.C_OK}}✓ fleet-config reachable.{{.C_RESET}}\n' - - # 2. Hub kubeconfig (inline so the guard governs everything) - rm -f "$KC" - aws eks update-kubeconfig --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --alias {{.HUB_CLUSTER_NAME}} --kubeconfig "$KC" >/dev/null - - # 3. ArgoCD repo credential for the private GitLab fleet-config repo - printf '{{.C_STEP}}▸ Registering ArgoCD repository credential for fleet-config...{{.C_RESET}}\n' - cat </{{.GIT_USERNAME}}/.git) that ArgoCD must pull; without a - # matching credential template those CD Applications fail with ComparisonError. - # A repo-creds secret applies to every repository whose URL starts with `url`. - printf '{{.C_STEP}}▸ Registering ArgoCD repo-creds for {{.GIT_USERNAME}} GitLab namespace...{{.C_RESET}}\n' - cat < leave it - # untouched. Never re-flip/verify/revert an already-working overlay, since a - # transient verify hiccup must never unset a good overlay. - if [ "$PREV_URL" = "$OVERLAY_URL" ]; then - printf '{{.C_OK}}✓ Overlay already wired to fleet-config (%s) — leaving untouched (idempotent).{{.C_RESET}}\n' "$OVERLAY_URL" - rm -f "$KC" - exit 0 - fi - - # 4. Flip overlay_* in /config (ESO reconciles the cluster secret annotations) - printf '{{.C_STEP}}▸ Wiring overlay_repo_url into %s...{{.C_RESET}}\n' "$SECRET_KEY" - EXISTING=$(aws secretsmanager get-secret-value --secret-id "$SECRET_KEY" --region {{.AWS_REGION}} --query SecretString --output text) - UPDATED=$(echo "$EXISTING" | jq --arg url "$OVERLAY_URL" --arg rev "main" --arg bp "{{.OVERLAY_REPO_BASEPATH}}" \ - '.metadata = (.metadata | fromjson | . + {overlay_repo_url: $url, overlay_repo_revision: $rev, overlay_repo_basepath: $bp} | tojson)') - aws secretsmanager put-secret-value --secret-id "$SECRET_KEY" --secret-string "$UPDATED" --region {{.AWS_REGION}} >/dev/null - echo "Set overlay_repo_url=$OVERLAY_URL (basepath='{{.OVERLAY_REPO_BASEPATH}}')" - - # 5. Force ESO re-sync; wait until the new overlay actually PROPAGATES to the - # cluster-addons app source (avoids the false-positive the old verify had). - ES=$(KUBECONFIG="$KC" kubectl -n argocd get externalsecret -o name 2>/dev/null | grep -i fleet-secret | head -1 || true) - [ -n "$ES" ] && KUBECONFIG="$KC" kubectl -n argocd annotate "$ES" force-sync="$(date +%s)" --overwrite >/dev/null 2>&1 || true - printf '{{.C_STEP}}▸ Waiting for cluster-addons to pick up the new overlay source...{{.C_RESET}}\n' - PROPAGATED=false - for i in $(seq 1 30); do - SRC=$(KUBECONFIG="$KC" kubectl -n argocd get application cluster-addons -o jsonpath='{.spec.sources[?(@.ref=="overlay")].repoURL}' 2>/dev/null || echo "") - [ "$SRC" = "$OVERLAY_URL" ] && { PROPAGATED=true; break; } - sleep 10 - done - - # 6. Verify it resolves cleanly; auto-revert on any ComparisonError/InvalidSpec. - OK=false - if [ "$PROPAGATED" = true ]; then - printf '{{.C_STEP}}▸ Overlay propagated; verifying clean resolution...{{.C_RESET}}\n' - for i in $(seq 1 24); do - SYNC=$(KUBECONFIG="$KC" kubectl -n argocd get application cluster-addons -o jsonpath='{.status.sync.status}' 2>/dev/null || echo "") - ERR=$(KUBECONFIG="$KC" kubectl -n argocd get application cluster-addons -o jsonpath='{range .status.conditions[*]}{.type}{"\n"}{end}' 2>/dev/null | grep -iE "error|invalid" || true) - if [ -z "$ERR" ] && { [ "$SYNC" = "Synced" ] || [ "$SYNC" = "OutOfSync" ]; }; then OK=true; break; fi - printf '{{.C_INFO}} waiting for clean resolution (sync=%s) (%d/24)...{{.C_RESET}}\n' "$SYNC" "$i" - sleep 10 - done - else - printf '{{.C_ERR}}⚠ Overlay did not propagate to cluster-addons in time.{{.C_RESET}}\n' - fi - - if [ "$OK" != true ]; then - EXISTING=$(aws secretsmanager get-secret-value --secret-id "$SECRET_KEY" --region {{.AWS_REGION}} --query SecretString --output text) - if [ -n "$PREV_URL" ]; then - # A previous overlay was already wired — RESTORE it rather than dropping to - # the GitHub base. Falling back to base renders clusters:{} and would - # cascade-delete live spoke clusters; restoring keeps them intact. - printf '{{.C_ERR}}⚠ New overlay not resolving cleanly — restoring previous overlay (%s).{{.C_RESET}}\n' "$PREV_URL" - REVERTED=$(echo "$EXISTING" | jq --arg url "$PREV_URL" --arg rev "$PREV_REV" --arg bp "$PREV_BP" \ - '.metadata = (.metadata | fromjson | . + {overlay_repo_url: $url, overlay_repo_revision: $rev, overlay_repo_basepath: $bp} | tojson)') - else - # No overlay was set before this run — clean rollback to the GitHub base. - printf '{{.C_ERR}}⚠ Overlay not resolving cleanly and none was set before — rolling back to GitHub base.{{.C_RESET}}\n' - REVERTED=$(echo "$EXISTING" | jq '.metadata = (.metadata | fromjson | del(.overlay_repo_url, .overlay_repo_revision, .overlay_repo_basepath) | tojson)') - fi - aws secretsmanager put-secret-value --secret-id "$SECRET_KEY" --secret-string "$REVERTED" --region {{.AWS_REGION}} >/dev/null - [ -n "$ES" ] && KUBECONFIG="$KC" kubectl -n argocd annotate "$ES" force-sync="$(date +%s)" --overwrite >/dev/null 2>&1 || true - # Only strip the cluster-secret annotations on a true rollback-to-base (no prev); - # for a restore, let ESO reconcile the previous values back from SecretsManager. - if [ -z "$PREV_URL" ]; then - KUBECONFIG="$KC" kubectl -n argocd annotate secret {{.HUB_CLUSTER_NAME}} overlay_repo_url- overlay_repo_revision- overlay_repo_basepath- >/dev/null 2>&1 || true - fi - printf '{{.C_ERR}} Fix GitLab reachability/credential, then re-run.{{.C_RESET}}\n' - rm -f "$KC" - exit 0 - fi - printf '{{.C_OK}}✓ Overlay wired to fleet-config and resolving fleet-wide (hub + spokes).{{.C_RESET}}\n' - rm -f "$KC" - - spokes:enable-crossplane: - desc: "Enable a spoke cluster via Crossplane (usage: task spokes:enable-crossplane -- )" - vars: - CLUSTER: '{{.CLI_ARGS}}' - GITLAB_DOMAIN_INT: - sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then - cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain - else - echo "{{.DOMAIN}}" - fi - GIT_PASSWORD: - sh: yq '.hub.gitlabRootPassword // ""' {{.CONFIG_FILE}} | grep -v '^$' || (aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text 2>/dev/null | jq -r '.user_password') - cmds: - - | - if [ -z "{{.CLUSTER}}" ]; then echo "Usage: task spokes:enable-crossplane -- "; exit 1; fi - FLEET_URL="https://root:root-{{.GIT_PASSWORD}}@{{.GITLAB_DOMAIN_INT}}/user1/fleet-config.git" - # fleet-config is created asynchronously by the gitlab-init Job (Helm - # post-install hook, which waits for GitLab readiness), so this task can - # race ahead of it. Cloning an absent repo left $REPO_DIR without a .git - # dir and the later `git add -A` failed with exit 128, taking down - # `task install` (install:phase2-spoke-dev). Wait (up to ~5min) for the - # repo to be reachable, then verify the clone produced a git repo. - printf '{{.C_STEP}}▸ Waiting for fleet-config repo to be reachable...{{.C_RESET}}\n' - for i in $(seq 1 30); do - git ls-remote "$FLEET_URL" >/dev/null 2>&1 && break - if [ "$i" -eq 30 ]; then - printf '{{.C_ERR}}⚠ fleet-config not reachable after 5min — cannot enable spoke {{.CLUSTER}} via Crossplane.{{.C_RESET}}\n' - printf '{{.C_INFO}} GitLab/gitlab-init may still be initializing. Re-run later: task kind-crossplane:spokes:enable-crossplane -- {{.CLUSTER}}{{.C_RESET}}\n' - exit 1 - fi - sleep 10 - done - printf '{{.C_OK}}✓ fleet-config reachable.{{.C_RESET}}\n' - REPO_DIR=$(mktemp -d) - git clone "$FLEET_URL" "$REPO_DIR" - if [ ! -d "$REPO_DIR/.git" ]; then - printf '{{.C_ERR}}⚠ fleet-config clone failed — aborting enable-crossplane for {{.CLUSTER}}.{{.C_RESET}}\n' - rm -rf "$REPO_DIR" - exit 1 - fi - mkdir -p "$REPO_DIR/gitops/fleet/spoke-values/tenants/workshop/crossplane-clusters" - XP_VALUES="$REPO_DIR/gitops/fleet/spoke-values/tenants/workshop/crossplane-clusters/values.yaml" - # Idempotency: keep the existing vpcCidr if this spoke is already enabled so - # re-running install does not generate a new random CIDR and churn the VPC. - if [ -f "$XP_VALUES" ] && yq -e '.clusters."{{.CLUSTER}}".vpcCidr' "$XP_VALUES" >/dev/null 2>&1; then - echo " Spoke {{.CLUSTER}} already has a vpcCidr — keeping existing values (idempotent)." - else - # adminRoleArn grants the console/participant admin role cluster-admin on - # the spoke. adminInstanceRoleArn grants the IDE/instance *SharedRole* a - # SECOND cluster-admin access entry — required so `kubectl` run from the - # IDE (which assumes the instance role, NOT adminRoleName) works against - # the spoke. create-config.sh deliberately excludes the SharedRole from - # adminRoleName, so we detect it here. Both are idempotent / optional. - ADMIN_ROLE=$(yq '.adminRoleName // ""' {{.CONFIG_FILE}}) - ADMIN_ARN_LINE="" - if [ -n "$ADMIN_ROLE" ]; then - ADMIN_ARN_LINE=" adminRoleArn: arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/$ADMIN_ROLE" - fi - INSTANCE_ROLE=$(aws iam list-roles --query "Roles[?contains(RoleName,'SharedRole')].RoleName | [0]" --output text 2>/dev/null || echo "") - ADMIN_INSTANCE_ARN_LINE="" - if [ -n "$INSTANCE_ROLE" ] && [ "$INSTANCE_ROLE" != "None" ] && [ "$INSTANCE_ROLE" != "null" ]; then - ADMIN_INSTANCE_ARN_LINE=" adminInstanceRoleArn: arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/$INSTANCE_ROLE" - fi - cat > "$XP_VALUES" < "$REPO_DIR/gitops/fleet/members/{{.CLUSTER}}/values.yaml" </dev/null && break - echo " Push attempt $i failed (conflict), rebasing..." - git pull --rebase 2>/dev/null && sleep 2 - done - fi - rm -rf "$REPO_DIR" - # Refresh the local fleet-config checkout so the committed overlay change is - # visible in ~/environment/fleet-config (the task pushes from a temp clone). - # Best-effort and non-fatal: skipped if absent, warns if not fast-forwardable. - LOCAL_FLEET="$HOME/environment/fleet-config" - if [ -d "$LOCAL_FLEET/.git" ]; then - (git -C "$LOCAL_FLEET" fetch origin -q && git -C "$LOCAL_FLEET" pull --ff-only -q) \ - && printf '{{.C_INFO}} Refreshed local fleet-config checkout.{{.C_RESET}}\n' \ - || printf '{{.C_INFO}} Local fleet-config not auto-updated — run: git -C %s pull{{.C_RESET}}\n' "$LOCAL_FLEET" - fi - printf '{{.C_OK}}✓ Spoke {{.CLUSTER}} enabled via Crossplane in fleet-config overlay.{{.C_RESET}}\n' - - spokes:create-capabilities: - desc: "Create KRO + ACK EKS Capabilities on a Crossplane-provisioned spoke (usage: task spokes:create-capabilities -- )" - # KRO-path spokes get their kro/ACK capabilities declaratively from the EksclusterWithVpc - # RGD. Crossplane-path spokes do NOT, so create them here the same way the hub does - # (aws eks create-capability), with per-spoke capability IAM roles. Idempotent. - vars: - CLUSTER: '{{.CLI_ARGS}}' - cmds: - - | - set -e - if [ -z "{{.CLUSTER}}" ]; then echo "Usage: task spokes:create-capabilities -- "; exit 1; fi - CLUSTER="{{.CLUSTER}}" - REGION="{{.AWS_REGION}}" - ACCOUNT="{{.AWS_ACCOUNT_ID}}" - PREFIX="{{.RESOURCE_PREFIX}}" - # CLUSTER already carries the resource prefix (e.g. peeks-spoke-staging), so the - # capability role name is just --capability-role — prepending PREFIX - # again would double it (peeks-peeks-...). PREFIX is still used below for the - # cluster-mgmt-* assume-role resource ARNs. - ACK_ROLE_NAME="${CLUSTER}-ack-capability-role" - KRO_ROLE_NAME="${CLUSTER}-kro-capability-role" - - # Wait for the Crossplane-provisioned spoke EKS cluster to be ACTIVE (up to ~30 min). - printf '{{.C_STEP}}▸ [%s] Waiting for spoke EKS cluster ACTIVE...{{.C_RESET}}\n' "$CLUSTER" - i=0 - until aws eks describe-cluster --name "$CLUSTER" --region "$REGION" --query 'cluster.status' --output text 2>/dev/null | grep -q ACTIVE; do - i=$((i+1)); [ "$i" -ge 120 ] && { printf '{{.C_ERR}}⚠ %s not ACTIVE after timeout — skipping capability creation.{{.C_RESET}}\n' "$CLUSTER"; exit 0; } - sleep 15 - done - - ASSUME_DOC='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"capabilities.eks.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]}' - - # --- ACK capability role (AssumeWorkloadRoles + ManageIRSARoles) --- - printf '{{.C_STEP}}▸ [%s] Ensuring ACK capability role %s...{{.C_RESET}}\n' "$CLUSTER" "$ACK_ROLE_NAME" - aws iam get-role --role-name "$ACK_ROLE_NAME" >/dev/null 2>&1 || \ - aws iam create-role --role-name "$ACK_ROLE_NAME" --assume-role-policy-document "$ASSUME_DOC" >/dev/null - aws iam put-role-policy --role-name "$ACK_ROLE_NAME" --policy-name AssumeWorkloadRoles --policy-document \ - "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"sts:AssumeRole\",\"sts:TagSession\"],\"Resource\":[\"arn:aws:iam::${ACCOUNT}:role/${PREFIX}-cluster-mgmt-*\"]}]}" >/dev/null - aws iam put-role-policy --role-name "$ACK_ROLE_NAME" --policy-name ManageIRSARoles --policy-document \ - "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"iam:GetRole\",\"iam:CreateRole\",\"iam:DeleteRole\",\"iam:TagRole\",\"iam:UntagRole\",\"iam:UpdateRole\",\"iam:UpdateAssumeRolePolicy\",\"iam:AttachRolePolicy\",\"iam:DetachRolePolicy\",\"iam:ListAttachedRolePolicies\",\"iam:ListRolePolicies\",\"iam:ListRoleTags\",\"iam:ListInstanceProfilesForRole\",\"iam:PutRolePolicy\",\"iam:DeleteRolePolicy\",\"iam:GetRolePolicy\"],\"Resource\":[\"arn:aws:iam::${ACCOUNT}:role/${CLUSTER}-*\"]}]}" >/dev/null - ACK_ROLE_ARN="arn:aws:iam::${ACCOUNT}:role/${ACK_ROLE_NAME}" - - # --- KRO capability role (AmazonEKSClusterPolicy) --- - printf '{{.C_STEP}}▸ [%s] Ensuring KRO capability role %s...{{.C_RESET}}\n' "$CLUSTER" "$KRO_ROLE_NAME" - aws iam get-role --role-name "$KRO_ROLE_NAME" >/dev/null 2>&1 || \ - aws iam create-role --role-name "$KRO_ROLE_NAME" --assume-role-policy-document "$ASSUME_DOC" >/dev/null - aws iam attach-role-policy --role-name "$KRO_ROLE_NAME" --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy >/dev/null 2>&1 || true - KRO_ROLE_ARN="arn:aws:iam::${ACCOUNT}:role/${KRO_ROLE_NAME}" - - # --- Create the capabilities (idempotent) --- - for cap in "kro:KRO:$KRO_ROLE_ARN" "ack:ACK:$ACK_ROLE_ARN"; do - NAME="${cap%%:*}"; rest="${cap#*:}"; TYPE="${rest%%:*}"; ROLE_ARN="${rest#*:}" - STATUS=$(aws eks describe-capability --cluster-name "$CLUSTER" --capability-name "$NAME" --region "$REGION" --query 'capability.status' --output text 2>/dev/null || echo "NOT_FOUND") - if [ "$STATUS" = "NOT_FOUND" ]; then - printf '{{.C_STEP}}▸ [%s] Creating %s capability...{{.C_RESET}}\n' "$CLUSTER" "$TYPE" - # The capability role was just created/updated above; EKS may reject - # CreateCapability with "trust policy is invalid" until IAM has - # propagated the role's trust policy. Retry on that transient error. - j=0 - until aws eks create-capability --region "$REGION" --cluster-name "$CLUSTER" --capability-name "$NAME" --type "$TYPE" --role-arn "$ROLE_ARN" --delete-propagation-policy RETAIN >/dev/null 2>/tmp/create-cap-err; do - j=$((j+1)) - if [ "$j" -ge 12 ]; then - printf '{{.C_ERR}}⚠ [%s] %s capability creation failed after retries:{{.C_RESET}}\n' "$CLUSTER" "$TYPE"; cat /tmp/create-cap-err; exit 1 - fi - if grep -q "trust policy" /tmp/create-cap-err; then - printf '{{.C_INFO}} [%s] Waiting for IAM trust policy propagation (attempt %s)...{{.C_RESET}}\n' "$CLUSTER" "$j" - sleep 10 - else - cat /tmp/create-cap-err; exit 1 - fi - done - else - printf '{{.C_INFO}} [%s] %s capability already exists (%s).{{.C_RESET}}\n' "$CLUSTER" "$TYPE" "$STATUS" - fi - done - - # --- Wait for both ACTIVE --- - for NAME in kro ack; do - i=0 - until aws eks describe-capability --cluster-name "$CLUSTER" --capability-name "$NAME" --region "$REGION" --query 'capability.status' --output text 2>/dev/null | grep -q ACTIVE; do - i=$((i+1)); [ "$i" -ge 40 ] && { printf '{{.C_ERR}}⚠ [%s] %s capability not ACTIVE after timeout.{{.C_RESET}}\n' "$CLUSTER" "$NAME"; break; } - sleep 15 - done - done - printf '{{.C_OK}}✓ [%s] KRO + ACK capabilities created.{{.C_RESET}}\n' "$CLUSTER" - - spokes:enable-kro: - desc: "Enable a spoke cluster via KRO (usage: task spokes:enable-kro -- )" - vars: - CLUSTER: '{{.CLI_ARGS}}' - GITLAB_DOMAIN_INT: - sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then - cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain - else - echo "{{.DOMAIN}}" - fi - GIT_PASSWORD: - sh: yq '.hub.gitlabRootPassword // ""' {{.CONFIG_FILE}} | grep -v '^$' || (aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text 2>/dev/null | jq -r '.user_password') - cmds: - - | - if [ -z "{{.CLUSTER}}" ]; then echo "Usage: task spokes:enable-kro -- "; exit 1; fi - FLEET_URL="https://root:root-{{.GIT_PASSWORD}}@{{.GITLAB_DOMAIN_INT}}/user1/fleet-config.git" - # fleet-config is created asynchronously by the gitlab-init Job (Helm - # post-install hook), which itself waits for GitLab to become ready. When - # enable-kro runs in phase2 it can race ahead of that job. Cloning an - # absent/empty repo previously left $REPO_DIR without a .git dir, so the - # later `git add -A` failed with exit 128 and took down the whole - # `task install`. Wait (up to ~5min) for the repo to be reachable, then - # verify the clone actually produced a git repo before proceeding. - printf '{{.C_STEP}}▸ Waiting for fleet-config repo to be reachable...{{.C_RESET}}\n' - for i in $(seq 1 30); do - git ls-remote "$FLEET_URL" >/dev/null 2>&1 && break - if [ "$i" -eq 30 ]; then - printf '{{.C_ERR}}⚠ fleet-config not reachable after 5min — cannot enable spoke {{.CLUSTER}} via KRO.{{.C_RESET}}\n' - printf '{{.C_INFO}} GitLab/gitlab-init may still be initializing. Re-run later: task kind-crossplane:spokes:enable-kro -- {{.CLUSTER}}{{.C_RESET}}\n' - exit 1 - fi - sleep 10 - done - printf '{{.C_OK}}✓ fleet-config reachable.{{.C_RESET}}\n' - REPO_DIR=$(mktemp -d) - git clone "$FLEET_URL" "$REPO_DIR" - if [ ! -d "$REPO_DIR/.git" ]; then - printf '{{.C_ERR}}⚠ fleet-config clone failed — aborting enable-kro for {{.CLUSTER}}.{{.C_RESET}}\n' - rm -rf "$REPO_DIR" - exit 1 - fi - mkdir -p "$REPO_DIR/gitops/fleet/spoke-values/tenants/workshop/kro-clusters/clusters" - cat > "$REPO_DIR/gitops/fleet/spoke-values/tenants/workshop/kro-clusters/clusters/{{.CLUSTER}}.json" < InvalidSubnet.Range). - # Idempotency: only generate a (new random) CIDR when this spoke is not - # already enabled. Re-running install must NOT churn the CIDR — an EC2 VPC - # cidrBlock is immutable, so a new octet would force VPC recreation/orphans. - KRO_VALUES="$REPO_DIR/gitops/fleet/spoke-values/tenants/workshop/kro-clusters/values.yaml" - if [ -f "$KRO_VALUES" ] && yq -e '.clusters."{{.CLUSTER}}".cidr.vpcCidr' "$KRO_VALUES" >/dev/null 2>&1; then - echo " Spoke {{.CLUSTER}} already has a CIDR — keeping existing values (idempotent)." - else - OCTET=$(shuf -i 2-254 -n 1) - cat > "$KRO_VALUES" </dev/null || echo "") - yq -i 'del(.clusters."{{.CLUSTER}}".adminRoles)' "$KRO_VALUES" 2>/dev/null || true - if [ -n "$CONSOLE_ROLE" ] && [ "$CONSOLE_ROLE" != "null" ]; then - CONSOLE_ROLE="$CONSOLE_ROLE" yq -i '.clusters."{{.CLUSTER}}".adminRoleName = env(CONSOLE_ROLE)' "$KRO_VALUES" - fi - if [ -n "$INSTANCE_ROLE" ] && [ "$INSTANCE_ROLE" != "None" ] && [ "$INSTANCE_ROLE" != "null" ]; then - INSTANCE_ROLE="$INSTANCE_ROLE" yq -i '.clusters."{{.CLUSTER}}".adminInstanceRoleName = env(INSTANCE_ROLE)' "$KRO_VALUES" - fi - # multi-acct (ACK CARM): map this spoke's namespace -> AWS account so the - # hub's ACK controllers render an IAMRoleSelector per service and assume - # the {{.RESOURCE_PREFIX}}-cluster-mgmt- roles when provisioning the - # spoke. The KRO EksclusterWithVpc lands in namespace == clusterName, which - # is the key the multi-acct chart selects on. Only KRO spokes need this — - # ACK does the provisioning; Crossplane spokes do not use CARM. - # Path = the appset-chart external per-addon overlay tier: - # $overlay/configs/multi-acct/values.yaml - # The fleet-config repo (seeded by the GitLab init-job) keeps configs/ and - # overlays/ at the REPO ROOT (overlay_repo_basepath=""), so this lives at - # configs/multi-acct/values.yaml — NOT under gitops/. (Spoke cluster values - # use gitops/fleet/... because that appset falls back to fleetRepoBasepath.) - # yq merge keeps it idempotent and accumulates entries across spokes. - mkdir -p "$REPO_DIR/configs/multi-acct" - MA_FILE="$REPO_DIR/configs/multi-acct/values.yaml" - [ -f "$MA_FILE" ] || printf 'clusters: {}\n' > "$MA_FILE" - yq -i '.clusters."{{.CLUSTER}}" = "{{.AWS_ACCOUNT_ID}}"' "$MA_FILE" - # platform-manifests: disable HuggingFaceModel CRs on KRO spokes. The - # HuggingFaceModel CRD (kro.run) is not installed on spokes, so the chart's - # CRs fail to apply ("no matches for kind HuggingFaceModel"); these models - # target GPU/Trainium clusters, not spokes. An empty list makes the chart's - # huggingfaceModels guard render nothing. Written to the per-cluster overlay - # tier the platform-manifests appset reads last: - # $overlay/overlays/clusters//platform-manifests/values.yaml - mkdir -p "$REPO_DIR/overlays/clusters/{{.CLUSTER}}/platform-manifests" - printf 'huggingfaceModels: []\n' > "$REPO_DIR/overlays/clusters/{{.CLUSTER}}/platform-manifests/values.yaml" - cd "$REPO_DIR" - git add -A - if git diff --cached --quiet; then - printf '{{.C_INFO}} No fleet-config changes for {{.CLUSTER}} — already enabled (idempotent).{{.C_RESET}}\n' - else - git commit -m "feat: enable spoke {{.CLUSTER}} via KRO (+ multi-acct CARM mapping)" - for i in 1 2 3; do - git push 2>/dev/null && break - echo " Push attempt $i failed (conflict), rebasing..." - git pull --rebase 2>/dev/null && sleep 2 - done - fi - rm -rf "$REPO_DIR" - # Refresh the local fleet-config checkout so the committed overlay change is - # visible in ~/environment/fleet-config (the task pushes from a temp clone). - # Best-effort and non-fatal: skipped if absent, warns if not fast-forwardable. - LOCAL_FLEET="$HOME/environment/fleet-config" - if [ -d "$LOCAL_FLEET/.git" ]; then - (git -C "$LOCAL_FLEET" fetch origin -q && git -C "$LOCAL_FLEET" pull --ff-only -q) \ - && printf '{{.C_INFO}} Refreshed local fleet-config checkout.{{.C_RESET}}\n' \ - || printf '{{.C_INFO}} Local fleet-config not auto-updated — run: git -C %s pull{{.C_RESET}}\n' "$LOCAL_FLEET" - fi - printf '{{.C_OK}}✓ Spoke {{.CLUSTER}} enabled via KRO + multi-acct CARM mapping in fleet-config overlay.{{.C_RESET}}\n' - - spokes:disable-crossplane: - desc: "DELETE a Crossplane spoke on purpose: remove from overlay, then prune (usage: task spokes:disable-crossplane -- )" - vars: - CLUSTER: '{{.CLI_ARGS}}' - GITLAB_DOMAIN_INT: - sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then - cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain - else - echo "{{.DOMAIN}}" - fi - GIT_PASSWORD: - sh: yq '.hub.gitlabRootPassword // ""' {{.CONFIG_FILE}} | grep -v '^$' || (aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text 2>/dev/null | jq -r '.user_password') - cmds: - - | - if [ -z "{{.CLUSTER}}" ]; then echo "Usage: task spokes:disable-crossplane -- "; exit 1; fi - # Crossplane clusters are grouped under one per-tenant app (clusters-). - APP="clusters-workshop" - - # 1. Declarative removal: drop the cluster entry from the overlay (Git). - FLEET_URL="https://root:root-{{.GIT_PASSWORD}}@{{.GITLAB_DOMAIN_INT}}/user1/fleet-config.git" - # Guard: a silently-failed clone would leave an empty $REPO_DIR, making us - # wrongly conclude the cluster is absent from the overlay and skip the Git - # removal (stale overlay left behind). Wait for fleet-config then verify. - printf '{{.C_STEP}}▸ Waiting for fleet-config repo to be reachable...{{.C_RESET}}\n' - for i in $(seq 1 30); do - git ls-remote "$FLEET_URL" >/dev/null 2>&1 && break - if [ "$i" -eq 30 ]; then - printf '{{.C_ERR}}⚠ fleet-config not reachable after 5min — cannot disable spoke {{.CLUSTER}} via Crossplane.{{.C_RESET}}\n' - exit 1 - fi - sleep 10 - done - REPO_DIR=$(mktemp -d) - git clone "$FLEET_URL" "$REPO_DIR" - if [ ! -d "$REPO_DIR/.git" ]; then - printf '{{.C_ERR}}⚠ fleet-config clone failed — aborting disable-crossplane for {{.CLUSTER}}.{{.C_RESET}}\n' - rm -rf "$REPO_DIR" - exit 1 - fi - XP_VALUES="$REPO_DIR/gitops/fleet/spoke-values/tenants/workshop/crossplane-clusters/values.yaml" - if [ -f "$XP_VALUES" ] && yq -e '.clusters."{{.CLUSTER}}"' "$XP_VALUES" >/dev/null 2>&1; then - yq -i 'del(.clusters."{{.CLUSTER}}")' "$XP_VALUES" - ( cd "$REPO_DIR" && git add -A && git commit -m "feat: disable spoke {{.CLUSTER}} (remove from Crossplane overlay)" && git push 2>/dev/null ) - printf '{{.C_OK}}✓ Removed {{.CLUSTER}} from the Crossplane overlay (Git).{{.C_RESET}}\n' - else - printf '{{.C_INFO}} {{.CLUSTER}} not present in Crossplane overlay — proceeding to prune any leftover resources.{{.C_RESET}}\n' - fi - rm -rf "$REPO_DIR" - - # 2. Deliberate prune. With prune:false on the appset this manual --prune is - # the explicit destructive step. Refresh + show the diff first so the - # operator sees exactly what will be deleted. - printf '{{.C_ERR}}▸ Pruning app %s — this DELETES the live cluster {{.CLUSTER}} + its VPC.{{.C_RESET}}\n' "$APP" - if command -v argocd >/dev/null 2>&1; then - argocd app get "$APP" --refresh >/dev/null 2>&1 || true - argocd app diff "$APP" || true - if ! argocd app sync "$APP" --prune --timeout 600; then - printf '{{.C_ERR}}⚠ argocd prune failed (auth?). Run: argocd-refresh-token && source ~/.bashrc.d/platform.sh, then:{{.C_RESET}}\n' - printf ' argocd app sync %s --prune\n' "$APP" - fi - else - printf '{{.C_INFO}} argocd CLI not found. Finish the deletion manually:{{.C_RESET}}\n' - printf ' argocd app sync %s --prune\n' "$APP" - fi - - spokes:disable-kro: - desc: "DELETE a KRO spoke on purpose: remove from overlay, then prune (usage: task spokes:disable-kro -- )" - vars: - CLUSTER: '{{.CLI_ARGS}}' - GITLAB_DOMAIN_INT: - sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then - cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain - else - echo "{{.DOMAIN}}" - fi - GIT_PASSWORD: - sh: yq '.hub.gitlabRootPassword // ""' {{.CONFIG_FILE}} | grep -v '^$' || (aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text 2>/dev/null | jq -r '.user_password') - cmds: - - | - if [ -z "{{.CLUSTER}}" ]; then echo "Usage: task spokes:disable-kro -- "; exit 1; fi - # KRO generates one app per cluster: clusters-kro-. - APP="clusters-kro-{{.CLUSTER}}" - FLEET_URL="https://root:root-{{.GIT_PASSWORD}}@{{.GITLAB_DOMAIN_INT}}/user1/fleet-config.git" - # Guard: a silently-failed clone would leave an empty $REPO_DIR, making the - # overlay cleanup below a no-op (stale kro overlay + marker left behind). - # Wait for fleet-config then verify the clone produced a git repo. - printf '{{.C_STEP}}▸ Waiting for fleet-config repo to be reachable...{{.C_RESET}}\n' - for i in $(seq 1 30); do - git ls-remote "$FLEET_URL" >/dev/null 2>&1 && break - if [ "$i" -eq 30 ]; then - printf '{{.C_ERR}}⚠ fleet-config not reachable after 5min — cannot disable spoke {{.CLUSTER}} via KRO.{{.C_RESET}}\n' - exit 1 - fi - sleep 10 - done - REPO_DIR=$(mktemp -d) - git clone "$FLEET_URL" "$REPO_DIR" - if [ ! -d "$REPO_DIR/.git" ]; then - printf '{{.C_ERR}}⚠ fleet-config clone failed — aborting disable-kro for {{.CLUSTER}}.{{.C_RESET}}\n' - rm -rf "$REPO_DIR" - exit 1 - fi - BASE="$REPO_DIR/gitops/fleet/spoke-values/tenants/workshop/kro-clusters" - KRO_VALUES="$BASE/values.yaml" - MARKER="$BASE/clusters/{{.CLUSTER}}.json" - MA_FILE="$REPO_DIR/configs/multi-acct/values.yaml" - - # PHASE 1: remove ONLY the cluster spec from values.yaml. Keep the clusters/*.json - # marker for now so the appset still generates clusters-kro- and we can prune - # through it. (Deleting the marker first would make the appset remove the app with - # preserveResourcesOnDeletion=true -> the EksclusterWithVpc would be ORPHANED, not - # deleted.) - if [ -f "$KRO_VALUES" ] && yq -e '.clusters."{{.CLUSTER}}"' "$KRO_VALUES" >/dev/null 2>&1; then - yq -i 'del(.clusters."{{.CLUSTER}}")' "$KRO_VALUES" - ( cd "$REPO_DIR" && git add -A && git commit -m "feat: disable spoke {{.CLUSTER}} (remove KRO cluster spec)" && git push 2>/dev/null ) - printf '{{.C_OK}}✓ Removed {{.CLUSTER}} spec from the KRO overlay (Git).{{.C_RESET}}\n' - else - printf '{{.C_INFO}} {{.CLUSTER}} spec not present in KRO overlay — proceeding to prune any leftovers.{{.C_RESET}}\n' - fi - - # PHASE 2: deliberate prune through the still-existing app. - printf '{{.C_ERR}}▸ Pruning app %s — this DELETES the live cluster {{.CLUSTER}} + its VPC.{{.C_RESET}}\n' "$APP" - PRUNED=false - if command -v argocd >/dev/null 2>&1; then - argocd app get "$APP" --refresh >/dev/null 2>&1 || true - argocd app diff "$APP" || true - if argocd app sync "$APP" --prune --timeout 600; then PRUNED=true; else - printf '{{.C_ERR}}⚠ argocd prune failed (auth?). Run: argocd-refresh-token && source ~/.bashrc.d/platform.sh, then: argocd app sync %s --prune{{.C_RESET}}\n' "$APP" - fi - else - printf '{{.C_INFO}} argocd CLI not found. Run manually then re-run this task to clean up: argocd app sync %s --prune{{.C_RESET}}\n' "$APP" - fi - - # PHASE 3: cleanup the now-empty app + CARM mapping (only after a successful prune, - # so we never orphan a live resource). The appset removes clusters-kro- once - # the marker is gone; its resources are already pruned. - if [ "$PRUNED" = true ]; then - CHANGED=0 - [ -f "$MARKER" ] && { rm -f "$MARKER"; CHANGED=1; } - [ -f "$MA_FILE" ] && yq -e '.clusters."{{.CLUSTER}}"' "$MA_FILE" >/dev/null 2>&1 && { yq -i 'del(.clusters."{{.CLUSTER}}")' "$MA_FILE"; CHANGED=1; } - if [ "$CHANGED" = "1" ]; then - ( cd "$REPO_DIR" && git add -A && git commit -m "chore: remove {{.CLUSTER}} KRO marker + multi-acct mapping (post-prune cleanup)" && git push 2>/dev/null ) - printf '{{.C_OK}}✓ Cleaned up {{.CLUSTER}} marker + multi-acct mapping; app will be removed by the appset.{{.C_RESET}}\n' - fi - else - printf '{{.C_INFO}} Skipped marker/CARM cleanup (prune not confirmed). Re-run this task after a successful prune.{{.C_RESET}}\n' - fi - rm -rf "$REPO_DIR" - - spokes:disable-all: - desc: >- - DELETE every spoke cluster (KRO + Crossplane) declared in the fleet-config - overlay, then wait for them to disappear from AWS. Used as Phase 0 of - `destroy` so spokes are removed while the hub controllers (KRO/ACK/Crossplane) - are still alive to reconcile their deletion. Idempotent. - vars: - GITLAB_DOMAIN_INT: - sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then - cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain - else - echo "{{.DOMAIN}}" - fi - GIT_PASSWORD: - sh: yq '.hub.gitlabRootPassword // ""' {{.CONFIG_FILE}} | grep -v '^$' || (aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text 2>/dev/null | jq -r '.user_password') - cmds: - - | - set -u - # 1. Discover spokes from the fleet-config overlay (the single source of truth - # for cluster specs). If the overlay is unreachable we must NOT silently - # proceed to hub teardown — surviving spokes would be orphaned. - REPO_DIR=$(mktemp -d) - if ! git clone "https://root:root-{{.GIT_PASSWORD}}@{{.GITLAB_DOMAIN_INT}}/user1/fleet-config.git" "$REPO_DIR" 2>/dev/null; then - printf '{{.C_ERR}}⚠ fleet-config overlay unreachable — cannot discover spokes automatically.{{.C_RESET}}\n' - printf '{{.C_INFO}} If any spoke clusters exist, delete them first with:{{.C_RESET}}\n' - printf ' task spokes:disable-kro -- / task spokes:disable-crossplane -- \n' - rm -rf "$REPO_DIR"; exit 0 - fi - KRO_VALUES="$REPO_DIR/gitops/fleet/spoke-values/tenants/workshop/kro-clusters/values.yaml" - XP_VALUES="$REPO_DIR/gitops/fleet/spoke-values/tenants/workshop/crossplane-clusters/values.yaml" - KRO_SPOKES=""; XP_SPOKES="" - [ -f "$KRO_VALUES" ] && KRO_SPOKES=$(yq '.clusters // {} | keys | .[]' "$KRO_VALUES" 2>/dev/null) - [ -f "$XP_VALUES" ] && XP_SPOKES=$(yq '.clusters // {} | keys | .[]' "$XP_VALUES" 2>/dev/null) - rm -rf "$REPO_DIR" - - if [ -z "$(echo $KRO_SPOKES $XP_SPOKES | tr -d ' ')" ]; then - printf '{{.C_OK}}✓ No spoke clusters declared in the overlay — nothing to tear down.{{.C_RESET}}\n' - exit 0 - fi - printf '{{.C_STEP}}▸ Spokes to delete — KRO: [%s] Crossplane: [%s]{{.C_RESET}}\n' "$(echo $KRO_SPOKES | tr '\n' ' ')" "$(echo $XP_SPOKES | tr '\n' ' ')" - - # 2. Deliberate deletion via the existing per-cluster disable tasks (they remove - # the cluster from the overlay in Git, then run a scoped `argocd app sync --prune`). - for c in $KRO_SPOKES; do - printf '{{.C_STEP}}▸ Disabling KRO spoke %s...{{.C_RESET}}\n' "$c" - task spokes:disable-kro -- "$c" || printf '{{.C_ERR}}⚠ disable-kro failed for %s (continuing).{{.C_RESET}}\n' "$c" - done - for c in $XP_SPOKES; do - printf '{{.C_STEP}}▸ Disabling Crossplane spoke %s...{{.C_RESET}}\n' "$c" - task spokes:disable-crossplane -- "$c" || printf '{{.C_ERR}}⚠ disable-crossplane failed for %s (continuing).{{.C_RESET}}\n' "$c" - done - - # 3. Wait for the spoke EKS clusters to actually disappear from AWS BEFORE the - # hub teardown removes the controllers that reconcile their deletion. Without - # this, killing the hub mid-deletion leaves orphaned EKS clusters + VPCs. - ALL_SPOKES="$KRO_SPOKES $XP_SPOKES" - printf '{{.C_STEP}}▸ Waiting for spoke EKS clusters to delete in AWS (up to 30m)...{{.C_RESET}}\n' - for i in $(seq 1 90); do - LIVE="" - for c in $ALL_SPOKES; do - aws eks describe-cluster --name "$c" --region {{.AWS_REGION}} >/dev/null 2>&1 && LIVE="$LIVE $c" - done - if [ -z "$(echo $LIVE | tr -d ' ')" ]; then - printf '{{.C_OK}}✓ All spoke clusters deleted from AWS.{{.C_RESET}}\n'; break - fi - printf '{{.C_INFO}} Still deleting:%s ... (%s/90){{.C_RESET}}\n' "$LIVE" "$i" - sleep 20 - done - # Final check — warn loudly (but do not hard-fail) if a spoke survived, so the - # operator can intervene before/after the hub goes away. - STILL_LIVE="" - for c in $ALL_SPOKES; do - aws eks describe-cluster --name "$c" --region {{.AWS_REGION}} >/dev/null 2>&1 && STILL_LIVE="$STILL_LIVE $c" - done - if [ -n "$(echo $STILL_LIVE | tr -d ' ')" ]; then - printf '{{.C_ERR}}⚠ Spoke clusters still present after wait:%s{{.C_RESET}}\n' "$STILL_LIVE" - printf '{{.C_ERR}} Proceeding with hub teardown will ORPHAN them. See troubleshooting.md →{{.C_RESET}}\n' - printf '{{.C_ERR}} "Spoke Cluster Stuck Deletion / Recovery". Recommend aborting and fixing first.{{.C_RESET}}\n' - fi hub:restart-identity-pods: desc: Restart pods that depend on Pod Identity (external-dns, LBC, Crossplane providers) after identities are ready + status: + - | + KC={{.ROOT_DIR}}/private/hub-kubeconfig + # Skip if all providers are Healthy and oldest provider pod is >5min old + UNHEALTHY=$(KUBECONFIG=$KC kubectl get providers.pkg.crossplane.io \ + -o jsonpath='{.items[?(@.status.conditions[?(@.type=="Healthy")].status!="True")].metadata.name}' \ + 2>/dev/null | wc -w) + [ "$UNHEALTHY" != "0" ] && exit 1 + # Use kubectl to get pod start time as epoch via jsonpath, then compare + OLDEST_EPOCH=$(KUBECONFIG=$KC kubectl get pods -n crossplane-system \ + -l pkg.crossplane.io/revision \ + --sort-by=.metadata.creationTimestamp \ + -o jsonpath='{.items[0].metadata.creationTimestamp}' 2>/dev/null) + [ -z "$OLDEST_EPOCH" ] && exit 1 + # Convert ISO8601 to epoch (GNU date) + POD_START=$(date -d "$OLDEST_EPOCH" +%s 2>/dev/null || echo 0) + NOW=$(date +%s) + AGE=$(( NOW - POD_START )) + [ "$AGE" -gt 300 ] cmds: - task: hub:kubeconfig - printf '{{.C_STEP}}▸ Terminating any stuck ArgoCD sync operations on pod-identities apps...{{.C_RESET}}\n' @@ -1715,6 +878,11 @@ tasks: - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout status deploy -n kagent litellm --timeout=120s 2>/dev/null || true - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig - printf '{{.C_OK}}✓ Identity-dependent pods restarted.{{.C_RESET}}\n' + # Wait 60s for EKS Pod Identity agent to inject credentials into new pods + # before ArgoCD syncs crossplane-base and tries to create IAM roles + - printf '{{.C_INFO}} Waiting 90s for Pod Identity credentials to be injected...{{.C_RESET}}\n' + - sleep 90 + - printf '{{.C_OK}}✓ Pod Identity credentials ready.{{.C_RESET}}\n' hub:restart-langfuse: desc: Restart Langfuse to pick up Keycloak client secret (created by config job after initial deploy) @@ -1775,19 +943,15 @@ tasks: aws ec2 describe-subnets --subnet-ids $SUBNETS --region {{.AWS_REGION}} --query 'Subnets[?MapPublicIpOnLaunch==`false`].SubnetId' --output text | tr '\t' ',' CLUSTER_SG: sh: aws eks describe-cluster --name $(yq '.hub.clusterName' {{.CONFIG_FILE}}) --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text - INGRESS_DOMAIN: - sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/cloudfront-domain ]; then - cat {{.ROOT_DIR}}/private/cloudfront-domain - else - echo "{{.DOMAIN}}" - fi + INGRESS_DOMAIN: '{{.DOMAIN}}' ADMIN_ROLE_NAME: sh: yq '.adminRoleName // ""' {{.CONFIG_FILE}} GITLAB_DOMAIN: sh: | - if [ "{{.EXPOSURE_MODE}}" = "cloudfront" ] && [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then + if [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain + else + echo "{{.DOMAIN}}" fi cmds: - | @@ -1811,7 +975,7 @@ tasks: --arg cluster_security_group_id "{{.CLUSTER_SG}}" \ '{id: $id, subnet_ids: $subnet_ids, cluster_security_group_id: $cluster_security_group_id}' | jq -c .) # Base metadata (string), then re-merge preserved overlay_* keys (only when set). - METADATA='{"addonsRepoURL":"{{.REPO_URL}}","addonsRepoRevision":"{{.REPO_REVISION}}","addonsRepoBasepath":"{{.REPO_BASEPATH}}","fleetRepoURL":"{{.FLEET_REPO_URL}}","fleetRepoRevision":"{{.FLEET_REPO_REVISION}}","fleetRepoBasepath":"{{.FLEET_REPO_BASEPATH}}","aws_region":"{{.AWS_REGION}}","aws_account_id":"{{.AWS_ACCOUNT_ID}}","aws_cluster_name":"{{.HUB_CLUSTER_NAME}}","aws_cluster_arn":"{{.CLUSTER_ARN}}","aws_vpc_id":"{{.VPC_ID}}","aws_subnet_ids":"{{.SUBNET_IDS}}","aws_private_subnet_ids":"{{.PRIVATE_SUBNET_IDS}}","aws_cluster_security_group_id":"{{.CLUSTER_SG}}","alb_controller_mode":"oss","ingress_domain_name":"{{.INGRESS_DOMAIN}}","ingress_name":"{{.INGRESS_NAME}}","ingress_security_groups":"{{.INGRESS_SECURITY_GROUPS}}","exposure_mode":"{{.EXPOSURE_MODE}}","resource_prefix":"{{.RESOURCE_PREFIX}}","admin_role_name":"{{.ADMIN_ROLE_NAME}}","gitlab_domain_name":"{{.GITLAB_DOMAIN}}","git_token":"{{.USER_PASSWORD}}","aws_argocd_url":"{{.ARGOCD_URL}}"}' + METADATA='{"addonsRepoURL":"{{.REPO_URL}}","addonsRepoRevision":"{{.REPO_REVISION}}","addonsRepoBasepath":"{{.REPO_BASEPATH}}","fleetRepoURL":"{{.FLEET_REPO_URL}}","fleetRepoRevision":"{{.FLEET_REPO_REVISION}}","fleetRepoBasepath":"{{.FLEET_REPO_BASEPATH}}","aws_region":"{{.AWS_REGION}}","aws_account_id":"{{.AWS_ACCOUNT_ID}}","aws_cluster_name":"{{.HUB_CLUSTER_NAME}}","aws_cluster_arn":"{{.CLUSTER_ARN}}","aws_vpc_id":"{{.VPC_ID}}","aws_subnet_ids":"{{.SUBNET_IDS}}","aws_private_subnet_ids":"{{.PRIVATE_SUBNET_IDS}}","aws_cluster_security_group_id":"{{.CLUSTER_SG}}","alb_controller_mode":"oss","ingress_domain_name":"{{.INGRESS_DOMAIN}}","ingress_name":"{{.INGRESS_NAME}}","ingress_security_groups":"{{.INGRESS_SECURITY_GROUPS}}","exposure_mode":"{{.EXPOSURE_MODE}}","insecure":"{{.INSECURE}}","certificate_arn":"{{.CERTIFICATE_ARN}}","resource_prefix":"{{.RESOURCE_PREFIX}}","admin_role_name":"{{.ADMIN_ROLE_NAME}}","gitlab_domain_name":"{{.GITLAB_DOMAIN}}","git_token":"{{.USER_PASSWORD}}","aws_argocd_url":"{{.ARGOCD_URL}}"}' if [ -n "$OVL_URL" ]; then METADATA=$(echo "$METADATA" | jq -c --arg u "$OVL_URL" --arg r "$OVL_REV" --arg b "$OVL_BP" '. + {overlay_repo_url:$u, overlay_repo_revision:$r, overlay_repo_basepath:$b}') echo "Preserving existing overlay wiring: overlay_repo_url=$OVL_URL" @@ -1919,7 +1083,7 @@ tasks: BS_PG_PASS=$(openssl rand -base64 18 | tr -d '/+=' | head -c 24) UPDATED=$(echo "$UPDATED" | jq --arg p "$BS_PG_PASS" '. + {backstage_postgres_password: $p}') fi - UPDATED=$(echo "$UPDATED" | jq --arg t "$USER_PASS" '. + {git_token: $t}') + # git_token is seeded by CDK at deploy time (root-) — no runtime seeding needed. aws secretsmanager create-secret \ --name "$SECRET_KEY" \ --secret-string "$UPDATED" \ @@ -2055,11 +1219,10 @@ tasks: --arg grafana_api_key "$AMG_API_KEY" \ --arg grafana_url "$AMG_ENDPOINT" \ --arg user_password "$USER_PASS" \ - --arg git_token "$USER_PASS" \ --arg backstage_postgres_password "$BS_PG_PASS" \ --arg user_password_hash "$KARGO_HASH" \ --arg user_password_key "$KARGO_KEY" \ - '. + {amp_endpoint_url: $amp_query_url, amp_region: $amp_region, grafana_api_key: $grafana_api_key, grafana_url: $grafana_url, user_password: $user_password, git_token: $git_token, backstage_postgres_password: $backstage_postgres_password, user_password_hash: $user_password_hash, user_password_key: $user_password_key}') + '. + {amp_endpoint_url: $amp_query_url, amp_region: $amp_region, grafana_api_key: $grafana_api_key, grafana_url: $grafana_url, user_password: $user_password, backstage_postgres_password: $backstage_postgres_password, user_password_hash: $user_password_hash, user_password_key: $user_password_key}') # Generate the grafana_mysql_password the grafana-dashboards # ExternalSecret expects, but only if it isn't already there # (so re-runs don't rotate it out from under live consumers). @@ -2225,40 +1388,6 @@ tasks: kubectl rollout restart deploy/grafana-operator -n grafana-operator 2>/dev/null || true echo "Grafana ExternalSecrets resynced and grafana-operator restarted to pick up seeded credentials." - argocd:capability: - desc: Create EKS ArgoCD Capability (skips if done) - status: - - kubectl -n crossplane-system get job create-argocd-capability -o jsonpath='{.status.succeeded}' 2>/dev/null | grep -q 1 - vars: - CAPABILITY_ROLE_ARN: - sh: kubectl -n crossplane-system get roles.iam.aws.upbound.io argocd-capability-role -o jsonpath='{.status.atProvider.arn}' 2>/dev/null || echo "" - cmds: - - kubectl -n crossplane-system create configmap argocd-capability-config - --from-literal=CLUSTER_NAME={{.HUB_CLUSTER_NAME}} - --from-literal=AWS_REGION={{.AWS_REGION}} - --from-literal=AWS_ACCOUNT_ID={{.AWS_ACCOUNT_ID}} - --from-literal=CAPABILITY_NAME={{.CAPABILITY_NAME}} - --from-literal=CAPABILITY_ROLE_ARN={{.CAPABILITY_ROLE_ARN}} - --from-literal=IDC_INSTANCE_ARN={{.IDC_INSTANCE_ARN}} - --from-literal=IDC_REGION={{.IDC_REGION}} - --from-literal=IDC_ADMIN_GROUP_ID={{.IDC_ADMIN_GROUP_ID}} - --dry-run=client -o yaml | kubectl apply -f - - - kubectl delete job create-argocd-capability -n crossplane-system 2>/dev/null || true - - kubectl apply -f manifests/argocd/create-capability.yaml - - kubectl -n crossplane-system wait --for=condition=Complete --timeout=2400s job/create-argocd-capability - - argocd:delete-capability: - desc: Delete EKS ArgoCD Capability via Job - cmds: - - kubectl -n crossplane-system create configmap argocd-capability-config - --from-literal=CLUSTER_NAME={{.HUB_CLUSTER_NAME}} - --from-literal=AWS_REGION={{.AWS_REGION}} - --from-literal=CAPABILITY_NAME={{.CAPABILITY_NAME}} - --dry-run=client -o yaml | kubectl apply -f - - - kubectl delete job delete-argocd-capability -n crossplane-system 2>/dev/null || true - - kubectl apply -f manifests/argocd/delete-capability.yaml - - kubectl -n crossplane-system wait --for=condition=Complete --timeout=2400s job/delete-argocd-capability - status: desc: Check bootstrap status cmds: @@ -2387,11 +1516,12 @@ tasks: - task: credentials:refresh - kubectl -n crossplane-system rollout restart deploy -l pkg.crossplane.io/revision 2>/dev/null || true - kubectl -n crossplane-system rollout status deploy -l pkg.crossplane.io/revision --timeout=120s 2>/dev/null || true - # Phase 0: Tear down spoke clusters FIRST, while the hub controllers - # (KRO/ACK/Crossplane) are still alive to reconcile their deletion. Doing this - # after hub:destroy-addons would remove those controllers and orphan the spokes. - - cmd: task spokes:disable-all - ignore_error: true + # NOTE: spoke teardown is a CONSUMER concern (the consumer owns spoke + # declaration — see workshop/). Spokes MUST be torn down by the consumer + # BEFORE this platform destroy runs, while the hub controllers + # (KRO/ACK/Crossplane) are still alive to reconcile their deletion — + # otherwise the spokes are orphaned. The generic platform no longer + # enumerates or deletes spokes. # Phase 1: Clean hub addons (continue on failure so remaining phases still run) - cmd: task hub:destroy-addons ignore_error: true @@ -2410,11 +1540,9 @@ tasks: kubectl -n crossplane-system wait --for=delete policies.iam.aws.upbound.io -l platform.gitops.io/role --timeout=120s 2>/dev/null || true kubectl -n crossplane-system wait --for=delete roles.iam.aws.upbound.io -l platform.gitops.io/role --timeout=120s 2>/dev/null || true ignore_error: true - # Phase 3: Delete ArgoCD capability via Job, then its IAM role (needs IAM provider alive) - # Use `task:` instead of `cmd: task ...` so the call doesn't go through gosh, - # which mis-parses some yq-driven dynamic vars in credentials:setup. - - task: argocd:delete-capability - ignore_error: true + # Phase 3: Delete the capability IAM role. The EKS Capabilities are managed as + # Crossplane Capability MRs (deletePropagationPolicy RETAIN); the AWS-API fallback + # below force-deletes them so the cluster delete in Phase 4 isn't blocked. - cmd: | sed "s|CLUSTER_NAME|{{.HUB_CLUSTER_NAME}}|g" claims/argocd-capability-role.yaml | kubectl delete --ignore-not-found --wait=false -f - 2>/dev/null || true kubectl -n crossplane-system wait --for=delete roles.iam.aws.upbound.io/argocd-capability-role --timeout=120s 2>/dev/null || true @@ -2469,10 +1597,10 @@ tasks: --set aws.region={{.AWS_REGION}} \ --set aws.clusterName={{.HUB_CLUSTER_NAME}} \ --set aws.accountId={{.AWS_ACCOUNT_ID}} \ - --set providers.family.version=v2.5.3 \ - --set providers.iam.version=v2.5.3 \ - --set providers.eks.version=v2.5.3 \ - --set providers.ec2.version=v2.5.3 \ + --set providers.family.version=v2.6.1 \ + --set providers.iam.version=v2.6.1 \ + --set providers.eks.version=v2.6.1 \ + --set providers.ec2.version=v2.6.1 \ | kubectl delete --ignore-not-found --wait=false -f - 2>/dev/null || true kubectl -n crossplane-system wait --for=delete podidentityassociations.eks.aws.upbound.io --all --timeout=120s 2>/dev/null || true kubectl -n crossplane-system wait --for=delete roles.iam.aws.upbound.io -l platform.gitops.io/role --timeout=120s 2>/dev/null || true @@ -2663,6 +1791,13 @@ tasks: --set clusters.hub.kubernetesVersion={{.HUB_K8S_VERSION}} --set clusters.hub.vpcCidr={{.HUB_VPC_CIDR}} --set clusters.hub.resourcePrefix={{.RESOURCE_PREFIX}} + --set-string clusters.hub.accountId={{.AWS_ACCOUNT_ID}} + --set clusters.hub.capabilities.kro.enabled=true + --set clusters.hub.capabilities.ack.enabled=true + --set clusters.hub.capabilities.argocd.enabled=true + --set-string clusters.hub.capabilities.argocd.idcInstanceArn={{.IDC_INSTANCE_ARN}} + --set-string clusters.hub.capabilities.argocd.idcRegion={{.IDC_REGION}} + --set-string clusters.hub.capabilities.argocd.adminGroupId={{.IDC_ADMIN_GROUP_ID}} {{.MNG_SETS}} | kubectl apply -f - - kubectl apply -f claims/ @@ -2726,6 +1861,7 @@ tasks: done printf '{{.C_OK}}✓ Spoke provider identity seeding complete.{{.C_RESET}}\n' + hub:destroy-addons: desc: Remove all addons from the hub cluster (no git changes needed) vars: diff --git a/cluster-providers/kind-kro-ack/.gitignore b/cluster-providers/kind-kro-ack/.gitignore new file mode 100644 index 000000000..1f6786d81 --- /dev/null +++ b/cluster-providers/kind-kro-ack/.gitignore @@ -0,0 +1 @@ +private/ diff --git a/cluster-providers/kind-kro-ack/README.md b/cluster-providers/kind-kro-ack/README.md new file mode 100644 index 000000000..d1b6ea81d --- /dev/null +++ b/cluster-providers/kind-kro-ack/README.md @@ -0,0 +1,113 @@ +# Kind + KRO + ACK Provider + +Zero-Terraform bootstrap for the hub cluster using KRO ResourceGraphDefinitions and ACK controllers instead of Crossplane. Spins up an ephemeral Kind cluster, installs ACK (IAM, EKS, EC2) + KRO, provisions all AWS infrastructure (VPC, EKS, IAM), seeds ArgoCD on the hub, and then the Kind cluster can be deleted. + +## Prerequisites + +| Requirement | Notes | +|-------------|-------| +| kind | Local Kubernetes cluster | +| kubectl | Cluster access | +| helm 3.x | Chart installation | +| yq | YAML processing | +| aws CLI | Configured with credentials | +| `config.local.yaml` | See below | + +### config.local.yaml + +```yaml +hub: + clusterName: hub + kubernetesVersion: "1.32" + vpcCidr: "10.0.0.0/16" + adminRoleName: Admin # IAM role for cluster admin access entry +aws: + accountId: "123456789012" + region: us-west-2 + profile: default +repo: + url: https://github.com/aws-samples/appmod-blueprints + revision: main + basepath: gitops +domain: example.com +resourcePrefix: peeks +ingressName: hub-ingress +ingressSecurityGroups: "" +argocdCapability: + enabled: "true" # Set to "false" to use Helm-installed ArgoCD instead +identityCenter: + instanceArn: arn:aws:sso:::instance/ssoins-XXXXXXXX # Must exist + region: us-east-1 # Region where IDC is deployed + identityStoreId: d-XXXXXXXXXX # Optional, auto-detected if omitted + adminGroupId: "" # Optional, created by idc:setup if empty +``` + +**IDC prerequisite**: An Identity Center instance must exist in your account/organization. The `idc:setup` task creates groups and users automatically. + +## Task Commands + +| Command | Description | +|---------|-------------| +| `task install` | Full bootstrap: Kind → ACK+KRO → AWS infra → ArgoCD → self-managing hub | +| `task status` | Show state of Kind, Helm releases, KRO RGDs, instances | +| `task destroy-kind` | Delete Kind cluster only (hub persists) | +| `task destroy` | Full teardown: delete KRO instances, wait for ACK cleanup, delete Kind | +| `task credentials:refresh` | Force-refresh AWS credentials secret | + +## Bootstrap Flow + +``` +task install + 1. kind:create Create Kind cluster + 2. credentials:setup Create aws-credentials secret for ACK controllers + 3. argocd:install Helm install ArgoCD + 4. ack:install Helm install ACK controllers (IAM v1.3.16, EKS v1.6.0, EC2 v1.3.4) + 5. kro:install Helm install KRO (v0.6.1) + 6. kro:apply-rgds Apply ResourceGraphDefinitions (rg-vpc, rg-eks, rg-eks-vpc) + 7. hub:claim Apply EksclusterWithVpc instance (creates VPC + EKS + IAM + pod identities) + 8. hub:seed Wait for ACTIVE, create access entry, ArgoCD capability, seed secrets, ESO, root-appset + 9. hub:wait-for-sync Wait for hub ArgoCD apps to converge +``` + +## Architecture: Crossplane vs KRO+ACK + +| Aspect | kind-crossplane | kind-kro-ack | +|--------|----------------|--------------| +| Controllers | Crossplane + AWS providers | ACK IAM + EKS + EC2 + KRO | +| Resource model | XRD → Composition → Managed Resources | ResourceGraphDefinition → ACK CRs | +| Credentials | ProviderConfig + Secret | `aws.credentials.secretName` in Helm values | +| ArgoCD Capability | Job (EKS API) | Job (same — ACK Capability CRD used on hub only) | +| Dependency ordering | Composition pipeline functions | KRO `readyWhen` expressions | +| Spoke provisioning | Hub Crossplane | Hub ACK+KRO (via EKS Capabilities) | + +## Key Differences + +1. **No Crossplane dependency** — ACK controllers are first-party AWS, no Upbound provider DRC issues +2. **Simpler resource model** — KRO ResourceGraphDefinitions are plain YAML with CEL expressions +3. **Same RGDs for bootstrap and runtime** — the hub uses identical RGDs to provision spokes +4. **Pod Identity native** — ACK uses Pod Identity on the hub; on Kind bootstrap, credentials Secret is mounted + +## Credentials on Kind + +ACK controllers on Kind cannot use Pod Identity (no IMDS). The Taskfile creates an `aws-credentials` Secret and configures each ACK controller Helm chart with: + +```yaml +aws: + credentials: + secretName: aws-credentials + secretKey: credentials +``` + +On the hub (after bootstrap), ACK runs as an EKS Capability with its own IAM role — no credentials Secret needed. + +### Keeping the Kind cluster alive + +If you keep the Kind cluster running (e.g. to manage hub infrastructure updates), AWS session tokens will expire (typically 12h). When this happens, ACK controllers stop reconciling and resources show `SYNCED: Unknown`. + +Fix: +```bash +task credentials:refresh +kubectl rollout restart deploy -n ack-system +``` + +For long-lived Kind clusters, consider using an IAM user with static credentials or running on an EC2 instance with an instance profile (credentials auto-rotate via IMDS). diff --git a/cluster-providers/kind-kro-ack/Taskfile.yaml b/cluster-providers/kind-kro-ack/Taskfile.yaml new file mode 100644 index 000000000..1447b9dea --- /dev/null +++ b/cluster-providers/kind-kro-ack/Taskfile.yaml @@ -0,0 +1,1667 @@ +version: "3" +set: [errexit, nounset, pipefail] +silent: true + +includes: + ray: + taskfile: ../common/Taskfile.ray.yaml + vars: + AWS_REGION: '{{.AWS_REGION}}' + AWS_ACCOUNT_ID: '{{.AWS_ACCOUNT_ID}}' + HUB_CLUSTER_NAME: '{{.HUB_CLUSTER_NAME}}' + RESOURCE_PREFIX: '{{.RESOURCE_PREFIX}}' + ROOT_DIR: '{{.ROOT_DIR}}' + +env: + KUBECONFIG: "{{.ROOT_DIR}}/private/kubeconfig" + AWS_PAGER: "" + +vars: + GITOPS_ROOT: "{{.ROOT_DIR}}/gitops" + REQUIRED_CLIS: [kind, kubectl, helm, yq, aws] + HUB_CLUSTER_NAME: + sh: yq '.hub.clusterName' {{.CONFIG_FILE}} + HUB_K8S_VERSION: + sh: yq '.hub.kubernetesVersion // "1.35"' {{.CONFIG_FILE}} + HUB_VPC_CIDR: + sh: yq '.hub.vpcCidr // "10.0.0.0/16"' {{.CONFIG_FILE}} + HUB_VPC_PREFIX: + sh: yq '.hub.vpcCidr // "10.0.0.0/16"' {{.CONFIG_FILE}} | cut -d. -f1-2 + AWS_ACCOUNT_ID: + sh: yq '.aws.accountId' {{.CONFIG_FILE}} + AWS_REGION: + sh: yq '.aws.region' {{.CONFIG_FILE}} + REPO_URL: + sh: yq '.repo.url' {{.CONFIG_FILE}} + REPO_REVISION: + sh: yq '.repo.revision' {{.CONFIG_FILE}} + REPO_BASEPATH: + sh: yq '.repo.basepath' {{.CONFIG_FILE}} + DOMAIN: + sh: yq '.domain' {{.CONFIG_FILE}} + RESOURCE_PREFIX: + sh: yq '.resourcePrefix // ""' {{.CONFIG_FILE}} + INGRESS_NAME: + sh: yq '.ingressName // ""' {{.CONFIG_FILE}} + INGRESS_SECURITY_GROUPS: + sh: yq '.ingressSecurityGroups // ""' {{.CONFIG_FILE}} + # Exposure: platform does not provision a domain/CloudFront. insecure -> HTTP-only + # ALB (consumer terminates TLS upstream); false -> HTTPS with ACM. exposure_mode is + # derived for the cluster-secret annotation chain the addon charts consume. + INSECURE: + sh: yq '.insecure // false' {{.CONFIG_FILE}} + EXPOSURE_MODE: + sh: 'if [ "$(yq ''.insecure // false'' {{.CONFIG_FILE}})" = "true" ]; then echo "cloudfront"; else echo "domain"; fi' + INGRESS_CERT_ARN: + sh: yq '.certificateArn // ""' {{.CONFIG_FILE}} + ADMIN_ROLE_NAME: + sh: yq '.hub.adminRoleName // "Admin"' {{.CONFIG_FILE}} + CAPABILITY_NAME: + sh: yq '.argocdCapability.name // "argocd"' {{.CONFIG_FILE}} + IDC_INSTANCE_ARN: + sh: yq '.identityCenter.instanceArn' {{.CONFIG_FILE}} + IDC_REGION: + sh: yq '.identityCenter.region' {{.CONFIG_FILE}} + IDC_IDENTITY_STORE_ID: + sh: yq '.identityCenter.identityStoreId // ""' {{.CONFIG_FILE}} + IDC_ADMIN_GROUP_ID: + sh: yq '.identityCenter.adminGroupId // ""' {{.CONFIG_FILE}} + KIND_CLUSTER_NAME: "{{.HUB_CLUSTER_NAME}}-bootstrap" + CALLER_ROLE_ARN: + sh: | + ARN=$(aws sts get-caller-identity --query "Arn" --output text 2>/dev/null || true) + if [ -z "$ARN" ]; then + echo "" + elif echo "$ARN" | grep -q ":assumed-role/"; then + echo "$ARN" | sed 's|arn:aws:sts::|arn:aws:iam::|;s|:assumed-role/|:role/|;s|/[^/]*$||' + else + echo "$ARN" + fi + ACK_NAMESPACE: "ack-system" + KRO_NAMESPACE: "kro-system" + ACK_IAM_VERSION: "1.6.4" + ACK_EKS_VERSION: "1.13.0" + ACK_EC2_VERSION: "1.12.0" + ACK_SECRETSMANAGER_VERSION: "1.3.1" + KRO_VERSION: "0.9.2" + ARGOCD_VERSION: + sh: yq '.argocd.defaultVersion' {{.ROOT_DIR}}/gitops/addons/registry/core.yaml + C_INFO: + sh: printf '\033[1;36m' + C_OK: + sh: printf '\033[1;32m' + C_STEP: + sh: printf '\033[1;33m' + C_ERR: + sh: printf '\033[1;31m' + C_RESET: + sh: printf '\033[0m' + +tasks: + validate: + desc: Pre-flight checks + cmds: + - cmd: command -v {{.ITEM}} >/dev/null 2>&1 || { echo "Missing {{.ITEM}}"; exit 1; } + for: + var: REQUIRED_CLIS + - test -f {{.CONFIG_FILE}} || { echo "config.local.yaml not found"; exit 1; } + - aws sts get-caller-identity >/dev/null || { echo "AWS credentials not configured"; exit 1; } + - printf '{{.C_OK}}✓ Pre-flight checks passed.{{.C_RESET}}\n' + + install: + desc: Bootstrap the platform (idempotent) + vars: + INSTALL_START: + sh: date +%s + cmds: + - task: kind:create + - task: kind:kubeconfig + - task: credentials:setup + - task: ack:install + - task: kro:install + - task: kro:install-eso-bootstrap + - task: kro:install-cluster-secret-store-bootstrap + - task: kro:apply-rgds + - task: hub:claim + - task: hub:wait-for-eks + - task: hub:seed + - task: hub:force-eso-refresh + - task: hub:wait-for-sync + vars: { MAX_OUTOFSYNC: "20" } + # Bootstrap the hub Crossplane providers' credentials (create provider-aws-iam/eks pod + # identities + restart) so crossplane-base's provider roles/PIAs (amp/rds/grafana/devlake) + # can reconcile. Done imperatively here (like the crossplane/terraform flows) rather than in + # the RGD: the hub is provisioned once by the transient kind cluster, so an RGD-based hub + # bootstrap can't self-heal an already-running hub. crossplane-base only declares the + # downstream provider PIAs (createIdentity:false for iam/eks) — nothing else seeds the first + # credential, so without this the whole crossplane chain stays credential-less. + - task: hub:bootstrap-crossplane-identity + # Enable spokes EARLY (kro-ack-specific optimization): KRO-provisioned spokes are + # created by the hub's ACK EKS capability, NOT the Crossplane providers, so spoke + # NOTE: spoke enablement and fleet-overlay wiring are CONSUMER concerns (the + # consumer owns the gitops/fleet repo). This generic provider provisions the + # hub only and exposes the PlatformCluster abstraction; consumers (e.g. the + # workshop, or OAP) declare spokes against their own repo / via PlatformCluster + # claims. See workshop/ for the reference consumer. + # Crossplane provider bring-up for the hub's OWN observability (AMP/RDS/Grafana). + - task: install:post-sync + - task: hub:wait-for-full-sync + # Wait for ALL Crossplane providers (including ec2) to be Healthy. + - task: hub:wait-for-providers + - task: install:finalize + - printf '{{.C_OK}}✓ Platform hub bootstrap complete in %ds. Run task status.{{.C_RESET}}\n' "$(($(date +%s) - {{.INSTALL_START}}))" + + install:post-sync: + internal: true + deps: + - hub:restart-identity-pods + - install:phase1-observability-bg + + install:phase1-observability-bg: + internal: true + desc: Seed observability in background — non-blocking so slow AMP provisioning doesn't delay install + cmds: + - | + REPO_ROOT="${WORKSPACE_PATH:-/home/ec2-user/environment}/${WORKING_REPO:-platform-on-eks-workshop}" + ( cd "$REPO_ROOT" && task kind-kro-ack:secrets-manager:seed-observability >> /tmp/observability-seed.log 2>&1 ) & + printf '{{.C_INFO}} Observability seeding running in background (log: /tmp/observability-seed.log){{.C_RESET}}\n' + + install:finalize: + internal: true + deps: + - hub:restart-langfuse + - urls + + kind:create: + desc: Create Kind cluster (skips if exists) + status: + - kind get clusters 2>/dev/null | grep -q "^{{.KIND_CLUSTER_NAME}}$" + cmds: + - kind create cluster --name {{.KIND_CLUSTER_NAME}} --config kind.yaml --kubeconfig {{.KUBECONFIG}} + + kind:kubeconfig: + desc: Ensure kind cluster kubeconfig is present (idempotent) + cmds: + - mkdir -p {{.ROOT_DIR}}/private + - kind export kubeconfig --name {{.KIND_CLUSTER_NAME}} --kubeconfig {{.KUBECONFIG}} 2>/dev/null || true + - kubectl config use-context kind-{{.KIND_CLUSTER_NAME}} --kubeconfig {{.KUBECONFIG}} 2>/dev/null || true + + credentials:setup: + desc: Create AWS credentials secret (skips if exists) + status: + - kubectl -n {{.ACK_NAMESPACE}} get secret aws-credentials 2>/dev/null + cmds: + - task: credentials:refresh + + credentials:refresh: + desc: Force-refresh AWS credentials secret + cmds: + - kubectl create namespace {{.ACK_NAMESPACE}} --dry-run=client -o yaml | kubectl apply -f - + - | + set -e + # Prefer explicit aws.profile from config — IMDS only used as fallback when no profile + # is set (or set to "default"). Without this, EC2-hosted runners (Cloud9, Workshop Studio, + # any EC2-based bootstrap host) silently override the user's intended account by reading + # the instance role from IMDS, causing ACK controllers to provision in the wrong account. + AWS_PROFILE_NAME=$(yq '.aws.profile // ""' {{.CONFIG_FILE}}) + AKI=""; SAK=""; ST="" + if [ -n "$AWS_PROFILE_NAME" ] && [ "$AWS_PROFILE_NAME" != "default" ]; then + printf ' Using AWS profile from config: %s\n' "$AWS_PROFILE_NAME" + EXPORTED=$(aws configure export-credentials --profile "$AWS_PROFILE_NAME" --format env-no-export) + AKI=$(echo "$EXPORTED" | sed -n 's/^AWS_ACCESS_KEY_ID=//p') + SAK=$(echo "$EXPORTED" | sed -n 's/^AWS_SECRET_ACCESS_KEY=//p') + ST=$(echo "$EXPORTED" | sed -n 's/^AWS_SESSION_TOKEN=//p') + else + TOKEN=$(curl -s -m 2 -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" 2>/dev/null || true) + if [ -n "$TOKEN" ]; then + printf ' Using EC2 instance role credentials (IMDS)\n' + ROLE=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/) + CREDS=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE) + AKI=$(echo "$CREDS" | jq -r .AccessKeyId) + SAK=$(echo "$CREDS" | jq -r .SecretAccessKey) + ST=$(echo "$CREDS" | jq -r .Token) + else + printf ' Using AWS default profile\n' + EXPORTED=$(aws configure export-credentials --profile default --format env-no-export) + AKI=$(echo "$EXPORTED" | sed -n 's/^AWS_ACCESS_KEY_ID=//p') + SAK=$(echo "$EXPORTED" | sed -n 's/^AWS_SECRET_ACCESS_KEY=//p') + ST=$(echo "$EXPORTED" | sed -n 's/^AWS_SESSION_TOKEN=//p') + fi + fi + [ -z "$AKI" ] || [ -z "$SAK" ] && echo "ERROR: empty credentials" && exit 1 + rm -f {{.ROOT_DIR}}/private/aws-creds.ini + install -m 600 /dev/null {{.ROOT_DIR}}/private/aws-creds.ini + { + echo "[default]" + echo "aws_access_key_id = $AKI" + echo "aws_secret_access_key = $SAK" + [ -n "$ST" ] && echo "aws_session_token = $ST" + } > {{.ROOT_DIR}}/private/aws-creds.ini + - kubectl -n {{.ACK_NAMESPACE}} create secret generic aws-credentials + --from-file=credentials={{.ROOT_DIR}}/private/aws-creds.ini + --dry-run=client -o yaml | kubectl apply -f - + - | + # Also refresh eso-aws-credentials (ESO needs individual keys, not INI format) + CREDS=$(kubectl -n {{.ACK_NAMESPACE}} get secret aws-credentials -o jsonpath='{.data.credentials}' | base64 -d) + kubectl -n {{.ACK_NAMESPACE}} create secret generic eso-aws-credentials \ + --from-literal=access-key-id="$(echo "$CREDS" | grep aws_access_key_id | awk '{print $3}')" \ + --from-literal=secret-access-key="$(echo "$CREDS" | grep aws_secret_access_key | awk '{print $3}')" \ + --from-literal=session-token="$(echo "$CREDS" | grep aws_session_token | awk '{print $3}')" \ + --dry-run=client -o yaml | kubectl apply -f - + - rm -f {{.ROOT_DIR}}/private/aws-creds.ini + + argocd:install: + desc: Install ArgoCD (skips if deployed) + status: + - helm status argocd -n argocd 2>/dev/null | grep -q deployed + cmds: + - helm repo add argo https://argoproj.github.io/argo-helm --force-update + - helm upgrade --install argocd argo/argo-cd + --namespace argocd --create-namespace + --version {{.ARGOCD_VERSION}} + --wait --timeout 5m + + ack:install: + desc: Install ACK controllers (IAM, EKS, EC2, SecretsManager) + cmds: + - task: ack:install-controller + vars: { CONTROLLER: iam, VERSION: "{{.ACK_IAM_VERSION}}" } + - task: ack:install-controller + vars: { CONTROLLER: eks, VERSION: "{{.ACK_EKS_VERSION}}" } + - task: ack:install-controller + vars: { CONTROLLER: ec2, VERSION: "{{.ACK_EC2_VERSION}}" } + - task: ack:install-controller + vars: { CONTROLLER: secretsmanager, VERSION: "{{.ACK_SECRETSMANAGER_VERSION}}" } + - printf '{{.C_OK}}✓ ACK controllers installed.{{.C_RESET}}\n' + + ack:install-controller: + internal: true + status: + - helm status ack-{{.CONTROLLER}}-controller -n {{.ACK_NAMESPACE}} 2>/dev/null | grep -q deployed + cmds: + - helm install ack-{{.CONTROLLER}}-controller + oci://public.ecr.aws/aws-controllers-k8s/{{.CONTROLLER}}-chart + --namespace {{.ACK_NAMESPACE}} --create-namespace + --version {{.VERSION}} + --set aws.region={{.AWS_REGION}} + --set aws.credentials.secretName=aws-credentials + --set aws.credentials.secretKey=credentials + --set serviceAccount.name=ack-{{.CONTROLLER}}-controller + --wait --timeout 5m + + kro:install: + desc: Install KRO controller + status: + - helm status kro -n {{.KRO_NAMESPACE}} 2>/dev/null | grep -q deployed + cmds: + - helm install kro oci://registry.k8s.io/kro/charts/kro + --namespace {{.KRO_NAMESPACE}} --create-namespace + --version {{.KRO_VERSION}} + --wait --timeout 5m + - printf '{{.C_OK}}✓ KRO installed.{{.C_RESET}}\n' + + kro:install-eso-bootstrap: + desc: Install External Secrets Operator on the bootstrap kind cluster (required for RGD schema validation) + vars: + ESO_VERSION: + sh: yq '.external-secrets.defaultVersion' {{.GITOPS_ROOT}}/addons/registry/core.yaml + status: + - helm status external-secrets -n external-secrets 2>/dev/null | grep -q deployed + cmds: + - helm repo add external-secrets https://charts.external-secrets.io --force-update + - helm upgrade --install external-secrets external-secrets/external-secrets + --namespace external-secrets --create-namespace + --version {{.ESO_VERSION}} + --set serviceAccount.name=external-secrets-sa + --wait --timeout 5m + - printf '{{.C_OK}}✓ ESO installed on bootstrap cluster.{{.C_RESET}}\n' + + kro:install-cluster-secret-store-bootstrap: + desc: Apply aws-secrets-manager ClusterSecretStore on the bootstrap kind cluster (required for RGD ESO resources) + status: + - kubectl get clustersecretstore aws-secrets-manager 2>/dev/null | grep -q aws-secrets-manager + cmds: + - | + # Extract individual credential keys from the ACK credentials file and + # create a separate secret for ESO (ESO needs individual keys, not INI format) + CREDS=$(kubectl get secret aws-credentials -n ack-system -o jsonpath='{.data.credentials}' | base64 -d) + ACCESS_KEY=$(echo "$CREDS" | grep aws_access_key_id | awk '{print $3}') + SECRET_KEY=$(echo "$CREDS" | grep aws_secret_access_key | awk '{print $3}') + SESSION_TOKEN=$(echo "$CREDS" | grep aws_session_token | awk '{print $3}') + kubectl create secret generic eso-aws-credentials -n ack-system \ + --from-literal=access-key-id="$ACCESS_KEY" \ + --from-literal=secret-access-key="$SECRET_KEY" \ + --from-literal=session-token="$SESSION_TOKEN" \ + --dry-run=client -o yaml | kubectl apply -f - + - | + cat <<'EOF' | sed "s|AWS_REGION|{{.AWS_REGION}}|g" | kubectl apply -f - + apiVersion: external-secrets.io/v1 + kind: ClusterSecretStore + metadata: + name: aws-secrets-manager + spec: + provider: + aws: + service: SecretsManager + region: AWS_REGION + auth: + secretRef: + accessKeyIDSecretRef: + name: eso-aws-credentials + namespace: ack-system + key: access-key-id + secretAccessKeySecretRef: + name: eso-aws-credentials + namespace: ack-system + key: secret-access-key + sessionTokenSecretRef: + name: eso-aws-credentials + namespace: ack-system + key: session-token + EOF + - printf '{{.C_OK}}✓ ClusterSecretStore applied on bootstrap cluster.{{.C_RESET}}\n' + + kro:apply-rgds: + desc: Apply KRO ResourceGraphDefinitions for VPC and EKS + cmds: + - kubectl apply -f {{.GITOPS_ROOT}}/addons/charts/kro/resource-groups/manifests/eks/rg-vpc.yaml + - kubectl apply -f {{.GITOPS_ROOT}}/addons/charts/kro/resource-groups/manifests/eks/rg-eks.yaml + - kubectl apply -f {{.GITOPS_ROOT}}/addons/charts/kro/resource-groups/manifests/eks/rg-eks-vpc.yaml + - printf '{{.C_STEP}}▸ Waiting for RGDs to be ready...{{.C_RESET}}\n' + - kubectl wait --for=condition=Ready resourcegraphdefinition/vpc.kro.run --timeout=300s + - kubectl wait --for=condition=Ready resourcegraphdefinition/ekscluster.kro.run --timeout=300s + - kubectl wait --for=condition=Ready resourcegraphdefinition/eksclusterwithvpc.kro.run --timeout=300s + - printf '{{.C_OK}}✓ ResourceGraphDefinitions applied.{{.C_RESET}}\n' + + kro:reset-crds: + desc: Delete KRO-managed CRDs on the bootstrap kind cluster to allow breaking schema changes (run before kro:apply-rgds) + preconditions: + - sh: "! kubectl get ekscluster,eksclusterwithvpc -A --no-headers 2>/dev/null | grep -qv '^$'" + msg: "Live EksCluster/EksclusterWithVpc instances exist — delete them first (task kind-kro-ack:destroy) before resetting CRDs." + cmds: + - kubectl delete crd eksclusters.kro.run eksclusterwithvpcs.kro.run --ignore-not-found + - printf '{{.C_OK}}✓ KRO CRDs deleted — re-run task install.{{.C_RESET}}\n' + + idc:setup: + desc: Create IDC groups if they don't exist (idempotent) + vars: + STORE_ID: + sh: | + ID=$(yq '.identityCenter.identityStoreId // ""' {{.CONFIG_FILE}}) + if [ -z "$ID" ]; then + aws sso-admin list-instances --region {{.IDC_REGION}} --query 'Instances[0].IdentityStoreId' --output text 2>/dev/null || echo "" + else + echo "$ID" + fi + ADMIN_GROUP: + sh: | + EXISTING=$(yq '.identityCenter.adminGroupId // ""' {{.CONFIG_FILE}}) + if [ -n "$EXISTING" ]; then echo "$EXISTING"; exit 0; fi + STORE_ID=$(yq '.identityCenter.identityStoreId // ""' {{.CONFIG_FILE}}) + [ -z "$STORE_ID" ] && STORE_ID=$(aws sso-admin list-instances --region {{.IDC_REGION}} --query 'Instances[0].IdentityStoreId' --output text 2>/dev/null) + aws identitystore list-groups --identity-store-id "$STORE_ID" --region {{.IDC_REGION}} --filters '[{"AttributePath":"DisplayName","AttributeValue":"admin"}]' --query 'Groups[0].GroupId' --output text 2>/dev/null | grep -v None || echo "" + cmds: + - | + STORE_ID="{{.STORE_ID}}" + if [ -z "$STORE_ID" ]; then + echo "ERROR: No IDC Identity Store found. Set identityCenter.identityStoreId in config.local.yaml" + exit 1 + fi + printf '{{.C_STEP}}▸ IDC Identity Store: %s{{.C_RESET}}\n' "$STORE_ID" + # BYO IdC: if adminGroupId AND adminUserId are pre-populated in config, + # skip group/user/membership creation entirely (useful when IdC lives in + # a different AWS account and the bootstrap role lacks identitystore:Create* perms). + BYO_GROUP=$(yq '.identityCenter.adminGroupId // ""' {{.CONFIG_FILE}}) + BYO_USER=$(yq '.identityCenter.adminUserId // ""' {{.CONFIG_FILE}}) + if [ -n "$BYO_GROUP" ] && [ -n "$BYO_USER" ]; then + printf '{{.C_INFO}} BYO IdC detected (adminGroupId + adminUserId set) — skipping group/user/membership creation{{.C_RESET}}\n' + printf '{{.C_OK}}✓ IDC admin group: %s{{.C_RESET}}\n' "$BYO_GROUP" + printf '{{.C_OK}}✓ IDC admin user: %s{{.C_RESET}}\n' "$BYO_USER" + yq -i ".identityCenter.identityStoreId = \"$STORE_ID\"" {{.CONFIG_FILE}} + exit 0 + fi + # Create groups idempotently + for GROUP_NAME in admin editor viewer; do + EXISTING=$(aws identitystore list-groups --identity-store-id "$STORE_ID" --region {{.IDC_REGION}} \ + --filters "[{\"AttributePath\":\"DisplayName\",\"AttributeValue\":\"$GROUP_NAME\"}]" \ + --query 'Groups[0].GroupId' --output text 2>/dev/null | grep -v None || echo "") + if [ -n "$EXISTING" ]; then + printf '{{.C_INFO}} Group "%s" exists: %s{{.C_RESET}}\n' "$GROUP_NAME" "$EXISTING" + else + NEW_ID=$(aws identitystore create-group --identity-store-id "$STORE_ID" --region {{.IDC_REGION}} \ + --display-name "$GROUP_NAME" --description "Platform group for EKS ArgoCD Capability" \ + --query 'GroupId' --output text) + printf '{{.C_OK}} Created group "%s": %s{{.C_RESET}}\n' "$GROUP_NAME" "$NEW_ID" + fi + done + # Output admin group ID (prefer config value, fallback to lookup) + ADMIN_ID=$(yq '.identityCenter.adminGroupId // ""' {{.CONFIG_FILE}}) + if [ -z "$ADMIN_ID" ]; then + ADMIN_ID=$(aws identitystore list-groups --identity-store-id "$STORE_ID" --region {{.IDC_REGION}} \ + --filters '[{"AttributePath":"DisplayName","AttributeValue":"admin"}]' \ + --query 'Groups[0].GroupId' --output text) + fi + printf '{{.C_OK}}✓ IDC admin group: %s{{.C_RESET}}\n' "$ADMIN_ID" + # Create admin user if not exists + ADMIN_USER=$(aws identitystore list-users --identity-store-id "$STORE_ID" --region {{.IDC_REGION}} \ + --filters '[{"AttributePath":"UserName","AttributeValue":"eks-argocd-admin-user"}]' \ + --query 'Users[0].UserId' --output text 2>/dev/null | grep -v None || echo "") + if [ -z "$ADMIN_USER" ]; then + ADMIN_USER=$(aws identitystore create-user --identity-store-id "$STORE_ID" --region {{.IDC_REGION}} \ + --user-name "eks-argocd-admin-user" --display-name "EKS ArgoCD Admin" \ + --name '{"GivenName":"EKS Admin","FamilyName":"User"}' \ + --emails '[{"Value":"eks-argocd-admin@example.com","Primary":true}]' \ + --query 'UserId' --output text) + printf '{{.C_OK}} Created admin user: %s{{.C_RESET}}\n' "$ADMIN_USER" + else + printf '{{.C_INFO}} Admin user exists: %s{{.C_RESET}}\n' "$ADMIN_USER" + fi + # Add user to admin group if not already member + IS_MEMBER=$(aws identitystore is-member-in-groups --identity-store-id "$STORE_ID" --region {{.IDC_REGION}} \ + --member-id '{"UserId":"'"$ADMIN_USER"'"}' --group-ids "$ADMIN_ID" \ + --query 'Results[0].MembershipExists' --output text 2>/dev/null || echo "false") + if [ "$IS_MEMBER" != "True" ]; then + aws identitystore create-group-membership --identity-store-id "$STORE_ID" --region {{.IDC_REGION}} \ + --group-id "$ADMIN_ID" --member-id '{"UserId":"'"$ADMIN_USER"'"}' >/dev/null 2>&1 || true + printf '{{.C_OK}} Added user to admin group{{.C_RESET}}\n' + fi + # Update config.local.yaml with discovered values + yq -i ".identityCenter.adminGroupId = \"$ADMIN_ID\"" {{.CONFIG_FILE}} + yq -i ".identityCenter.identityStoreId = \"$STORE_ID\"" {{.CONFIG_FILE}} + + hub:claim: + desc: Apply hub cluster claim (EksclusterWithVpc instance) + cmds: + - kubectl create namespace {{.HUB_CLUSTER_NAME}} --dry-run=client -o yaml | kubectl apply -f - + - | + cat <<'EOF' | sed \ + -e "s|HUB_CLUSTER_NAME|{{.HUB_CLUSTER_NAME}}|g" \ + -e "s|AWS_REGION|{{.AWS_REGION}}|g" \ + -e "s|AWS_ACCOUNT_ID|{{.AWS_ACCOUNT_ID}}|g" \ + -e "s|HUB_K8S_VERSION|{{.HUB_K8S_VERSION}}|g" \ + -e "s|HUB_VPC_CIDR|{{.HUB_VPC_CIDR}}|g" \ + -e "s|HUB_VPC_PREFIX|{{.HUB_VPC_PREFIX}}|g" \ + -e "s|RESOURCE_PREFIX|{{.RESOURCE_PREFIX}}|g" \ + -e "s|CALLER_ROLE_ARN|{{.CALLER_ROLE_ARN}}|g" \ + -e "s|IDC_INSTANCE_ARN|{{.IDC_INSTANCE_ARN}}|g" \ + -e "s|IDC_REGION|{{.IDC_REGION}}|g" \ + -e "s|IDC_ADMIN_GROUP_ID|{{.IDC_ADMIN_GROUP_ID}}|g" \ + -e "s|ADMIN_ROLE_NAME|{{.ADMIN_ROLE_NAME}}|g" \ + -e "s|REPO_URL|{{.REPO_URL}}|g" \ + -e "s|REPO_REVISION|{{.REPO_REVISION}}|g" \ + -e "s|REPO_BASEPATH|{{.REPO_BASEPATH}}|g" \ + -e "s|DOMAIN|{{.DOMAIN}}|g" \ + | kubectl apply -f - + apiVersion: kro.run/v1alpha1 + kind: EksclusterWithVpc + metadata: + name: HUB_CLUSTER_NAME + namespace: HUB_CLUSTER_NAME + spec: + name: HUB_CLUSTER_NAME + tenant: platform + environment: control-plane + region: AWS_REGION + k8sVersion: "HUB_K8S_VERSION" + accountId: "AWS_ACCOUNT_ID" + managementAccountId: "AWS_ACCOUNT_ID" + argoCdHubRoleArn: "CALLER_ROLE_ARN" + adminRoleName: ADMIN_ROLE_NAME + fleetSecretManagerSecretNameSuffix: argocd-secret + domainName: DOMAIN + resourcePrefix: RESOURCE_PREFIX + argocdCapability: + enabled: "true" + idcInstanceArn: IDC_INSTANCE_ARN + idcRegion: IDC_REGION + adminGroupId: IDC_ADMIN_GROUP_ID + cidr: + vpcCidr: HUB_VPC_CIDR + publicSubnet1Cidr: "HUB_VPC_PREFIX.1.0/24" + publicSubnet2Cidr: "HUB_VPC_PREFIX.2.0/24" + privateSubnet1Cidr: "HUB_VPC_PREFIX.11.0/24" + privateSubnet2Cidr: "HUB_VPC_PREFIX.12.0/24" + gitops: + addonsRepoUrl: REPO_URL + addonsRepoRevision: REPO_REVISION + addonsRepoBasePath: "REPO_BASEPATH/addons/" + addonsRepoPath: bootstrap + fleetRepoUrl: REPO_URL + fleetRepoRevision: REPO_REVISION + fleetRepoBasePath: "REPO_BASEPATH/fleet/" + fleetRepoPath: bootstrap + platformRepoUrl: REPO_URL + platformRepoRevision: REPO_REVISION + platformRepoBasePath: "REPO_BASEPATH/platform/" + platformRepoPath: bootstrap + workloadRepoUrl: REPO_URL + workloadRepoRevision: REPO_REVISION + workloadRepoBasePath: "REPO_BASEPATH/apps/" + workloadRepoPath: "" + addons: + enable_ingress_class_alb: "true" + enable_argo_rollouts: "false" + enable_metrics_server: "false" + enable_external_secrets: "true" + enable_external_dns: "true" + enable_adot_collector: "false" + enable_cw_prometheus: "false" + enable_kyverno: "false" + enable_kyverno_policies: "false" + enable_kyverno_policy_reporter: "false" + enable_cni_metrics_helper: "false" + enable_prometheus_node_exporter: "false" + enable_kube_state_metrics: "false" + enable_opentelemetry_operator: "false" + enable_cert_manager: "true" + enable_aws_efs_csi_driver: "false" + enable_crossplane: "true" + enable_crossplane_aws: "true" + enable_flux: "false" + enable_ingress_nginx: "false" + enable_platform_manifests: "true" + enable_kubevela: "false" + adot_collector_namespace: adot-collector-kubeprometheus + adot_collector_service_account: adot-collector-kubeprometheus + external_dns_namespace: external-dns + external_dns_service_account: external-dns-sa + external_dns_policy: sync + external_secrets_namespace: external-secrets + external_secrets_service_account: external-secrets-sa + amp_endpoint_url: "" + EOF + + hub:wait-for-eks: + desc: Wait for the EksclusterWithVpc instance to reach ACTIVE state and the EKS cluster to exist in AWS + vars: + WAIT_TIMEOUT_SEC: '{{.WAIT_TIMEOUT_SEC | default "1800"}}' + WAIT_INTERVAL_SEC: '{{.WAIT_INTERVAL_SEC | default "30"}}' + cmds: + - | + printf '{{.C_STEP}}▸ Waiting for EksclusterWithVpc/{{.HUB_CLUSTER_NAME}} to reach ACTIVE (max {{.WAIT_TIMEOUT_SEC}}s)...{{.C_RESET}}\n' + END=$(( $(date +%s) + {{.WAIT_TIMEOUT_SEC}} )) + LAST_STATE="" + while [ "$(date +%s)" -lt "$END" ]; do + STATE=$(kubectl get eksclusterwithvpc {{.HUB_CLUSTER_NAME}} -n {{.HUB_CLUSTER_NAME}} -o jsonpath='{.status.state}' 2>/dev/null || echo "") + if [ "$STATE" != "$LAST_STATE" ]; then + printf '{{.C_INFO}} state: %s{{.C_RESET}}\n' "${STATE:-}" + LAST_STATE="$STATE" + fi + if [ "$STATE" = "ACTIVE" ]; then + break + fi + sleep {{.WAIT_INTERVAL_SEC}} + done + if [ "$STATE" != "ACTIVE" ]; then + printf '{{.C_ERR}}✗ EksclusterWithVpc not ACTIVE after {{.WAIT_TIMEOUT_SEC}}s (last state: %s){{.C_RESET}}\n' "$STATE" + kubectl get eksclusterwithvpc {{.HUB_CLUSTER_NAME}} -n {{.HUB_CLUSTER_NAME}} -o jsonpath='{.status.conditions}' 2>/dev/null || true + exit 1 + fi + printf '{{.C_OK}}✓ EksclusterWithVpc ACTIVE{{.C_RESET}}\n' + - | + printf '{{.C_STEP}}▸ Verifying EKS cluster {{.HUB_CLUSTER_NAME}} exists in AWS...{{.C_RESET}}\n' + END=$(( $(date +%s) + 300 )) + while [ "$(date +%s)" -lt "$END" ]; do + EKS_STATUS=$(aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.status' --output text 2>/dev/null || echo "") + if [ "$EKS_STATUS" = "ACTIVE" ]; then + printf '{{.C_OK}}✓ EKS cluster ACTIVE{{.C_RESET}}\n' + exit 0 + fi + printf '{{.C_INFO}} EKS status: %s{{.C_RESET}}\n' "${EKS_STATUS:-}" + sleep 15 + done + printf '{{.C_ERR}}✗ EKS cluster not ACTIVE in AWS after RGD reported ACTIVE{{.C_RESET}}\n' + exit 1 + + hub:seed: + desc: Seed hub cluster after EKS is ACTIVE + cmds: + - task: credentials:refresh + - printf '{{.C_STEP}}▸ Waiting for hub EksCluster to be ready...{{.C_RESET}}\n' + - | + STATUS="Pending" + for i in $(seq 1 120); do + STATUS=$(kubectl get ekscluster {{.HUB_CLUSTER_NAME}} -n {{.HUB_CLUSTER_NAME}} -o jsonpath='{.status.clusterState}' 2>/dev/null || echo "Pending") + printf '{{.C_INFO}} [%s/120] Cluster status: %s{{.C_RESET}}\n' "$i" "$STATUS" + if [ "$STATUS" = "ACTIVE" ]; then break; fi + sleep 15 + done + if [ "$STATUS" != "ACTIVE" ]; then echo "ERROR: cluster not ACTIVE"; exit 1; fi + - printf '{{.C_STEP}}▸ Creating cluster admin access entry...{{.C_RESET}}\n' + - cmd: | + CALLER_ARN=$(aws sts get-caller-identity --query "Arn" --output text) + if echo "$CALLER_ARN" | grep -q ":assumed-role/"; then + PRINCIPAL_ARN=$(echo "$CALLER_ARN" | sed 's|arn:aws:sts::|arn:aws:iam::|;s|:assumed-role/|:role/|;s|/[^/]*$||') + else + PRINCIPAL_ARN="$CALLER_ARN" + fi + aws eks create-access-entry --cluster-name {{.HUB_CLUSTER_NAME}} --principal-arn "$PRINCIPAL_ARN" --type STANDARD --region {{.AWS_REGION}} 2>/dev/null || true + aws eks associate-access-policy --cluster-name {{.HUB_CLUSTER_NAME}} --principal-arn "$PRINCIPAL_ARN" --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy --access-scope type=cluster --region {{.AWS_REGION}} 2>/dev/null || true + - task: secrets-manager:seed + - task: secrets-manager:seed-keycloak + - task: secrets-manager:seed-secrets + - task: hub:install-eso + - task: hub:cluster-secret-store + - task: hub:seed-secret + - task: hub:apply-root-appset + - task: hub:create-mgmt-roles + + hub:kubeconfig: + cmds: + - aws eks update-kubeconfig --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --kubeconfig {{.ROOT_DIR}}/private/hub-kubeconfig 2>&1 >/dev/null + + secrets-manager:seed: + desc: Seed hub config into AWS Secrets Manager + vars: + CLUSTER_ARN: + sh: aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.arn' --output text 2>/dev/null + VPC_ID: + sh: aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.vpcId' --output text + # Private subnets + cluster SG for the devlake RDS `vpc` object (see cmds). Mirrors the + # kind-crossplane seed so the aws-resources ExternalSecret / init-env-config Job can + # populate a non-empty vpc-config EnvironmentConfig. + PRIVATE_SUBNET_IDS: + sh: | + SUBNETS=$(aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.subnetIds' --output text | tr '\t' '\n') + aws ec2 describe-subnets --subnet-ids $SUBNETS --region {{.AWS_REGION}} --query 'Subnets[?MapPublicIpOnLaunch==`false`].SubnetId' --output text | tr '\t' ',' + CLUSTER_SG: + sh: aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text + cmds: + - | + SECRET_KEY="{{.HUB_CLUSTER_NAME}}/config" + GITLAB_CF_DOMAIN=$(cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain 2>/dev/null || yq '.cloudfront.gitlabDomain // ""' {{.CONFIG_FILE}} || true) + # The aws-resources ExternalSecret reads /config property `vpc` into + # peeks-hub-vpc-secret.vpc_data; the init-env-config Job then parses .id / .subnet_ids / + # .cluster_security_group_id from it to build the vpc-config EnvironmentConfig that the + # devlake RDS composition consumes. `vpc` MUST be a JSON object (a bare VPC-id string makes + # every jq lookup empty -> empty vpc-config -> RDS never provisions). Stored as a JSON + # STRING (--arg), matching kind-crossplane, so ESO's property extraction round-trips. + PRIVATE_SUBNETS_JSON=$(echo "{{.PRIVATE_SUBNET_IDS}}" | tr ',' '\n' | grep -v '^$' | jq -R . | jq -s .) + VPC_JSON=$(jq -n --arg id "{{.VPC_ID}}" --argjson subnet_ids "$PRIVATE_SUBNETS_JSON" \ + --arg cluster_security_group_id "{{.CLUSTER_SG}}" \ + '{id:$id, subnet_ids:$subnet_ids, cluster_security_group_id:$cluster_security_group_id}' | jq -c .) + SECRET_VALUE=$(jq -n \ + --arg metadata '{"addonsRepoURL":"{{.REPO_URL}}","addonsRepoRevision":"{{.REPO_REVISION}}","addonsRepoBasepath":"{{.REPO_BASEPATH}}","fleetRepoURL":"{{.REPO_URL}}","fleetRepoRevision":"{{.REPO_REVISION}}","fleetRepoBasepath":"{{.REPO_BASEPATH}}","aws_region":"{{.AWS_REGION}}","aws_account_id":"{{.AWS_ACCOUNT_ID}}","aws_cluster_name":"{{.HUB_CLUSTER_NAME}}","aws_vpc_id":"{{.VPC_ID}}","alb_controller_mode":"oss","ingress_domain_name":"{{.DOMAIN}}","ingress_name":"{{.HUB_CLUSTER_NAME}}-platform","ingress_security_groups":"{{.INGRESS_SECURITY_GROUPS}}","resource_prefix":"{{.RESOURCE_PREFIX}}","exposure_mode":"{{.EXPOSURE_MODE}}","insecure":"{{.INSECURE}}","certificate_arn":"{{.INGRESS_CERT_ARN}}","admin_role_name":"{{.ADMIN_ROLE_NAME}}","gitlab_domain_name":"'"$GITLAB_CF_DOMAIN"'","git_token":"{{.USER_PASSWORD}}","aws_argocd_url":""}' \ + --arg config '{"tlsClientConfig":{"insecure":false}}' \ + --arg server '{{.CLUSTER_ARN}}' \ + --arg addons '{"provider":"kro-ack"}' \ + --arg vpc "$VPC_JSON" \ + '{metadata: $metadata, config: $config, server: $server, addons: $addons, vpc: $vpc}') + aws secretsmanager restore-secret --secret-id "$SECRET_KEY" --region {{.AWS_REGION}} 2>/dev/null || true + aws secretsmanager create-secret \ + --name "$SECRET_KEY" --secret-string "$SECRET_VALUE" --region {{.AWS_REGION}} 2>/dev/null \ + || aws secretsmanager put-secret-value \ + --secret-id "$SECRET_KEY" --secret-string "$SECRET_VALUE" --region {{.AWS_REGION}} + + secrets-manager:seed-keycloak: + desc: Seed keycloak passwords + status: + - aws secretsmanager describe-secret --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} 2>/dev/null + cmds: + - | + ADMIN_PASS=$(openssl rand -base64 18 | tr -d '/+=' | head -c 24) + DB_PASS=$(openssl rand -base64 18 | tr -d '/+=' | head -c 24) + USER_PASS=$(openssl rand -base64 12 | tr -d '/+=' | head -c 16) + SECRET_VALUE=$(jq -n \ + --arg admin "$ADMIN_PASS" --arg db "$DB_PASS" --arg user "$USER_PASS" \ + '{keycloak_admin_password: $admin, keycloak_postgres_password: $db, user_password: $user}') + aws secretsmanager restore-secret --secret-id "{{.HUB_CLUSTER_NAME}}/keycloak" --region {{.AWS_REGION}} 2>/dev/null || true + aws secretsmanager create-secret \ + --name "{{.HUB_CLUSTER_NAME}}/keycloak" --secret-string "$SECRET_VALUE" --region {{.AWS_REGION}} 2>/dev/null \ + || aws secretsmanager put-secret-value \ + --secret-id "{{.HUB_CLUSTER_NAME}}/keycloak" --secret-string "$SECRET_VALUE" --region {{.AWS_REGION}} + + hub:install-argocd: + desc: Install ArgoCD on the hub cluster (OSS mode, skipped when EKS Capability is used) + status: + # Skip if argocdCapability is enabled in config OR if ArgoCD CRDs already exist on the hub + - test "$(yq '.argocdCapability.enabled // "false"' {{.CONFIG_FILE}} 2>/dev/null || echo false)" = "true" || KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl get crd applications.argoproj.io 2>/dev/null | grep -q applications + cmds: + - task: hub:kubeconfig + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig helm repo add argo https://argoproj.github.io/argo-helm --force-update + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig helm upgrade --install argocd argo/argo-cd + --namespace argocd --create-namespace + --version {{.ARGOCD_VERSION}} + --wait --timeout 5m + - printf '{{.C_OK}}✓ ArgoCD installed on hub (OSS mode).{{.C_RESET}}\n' + + hub:install-eso: + desc: Install External Secrets Operator on hub + vars: + ESO_VERSION: + sh: yq '.external-secrets.defaultVersion' {{.GITOPS_ROOT}}/addons/registry/core.yaml + status: + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig helm status external-secrets -n external-secrets 2>/dev/null | grep -q deployed + cmds: + - task: hub:kubeconfig + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig helm repo add external-secrets https://charts.external-secrets.io --force-update + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig helm upgrade --install external-secrets external-secrets/external-secrets + --namespace external-secrets --create-namespace + --version {{.ESO_VERSION}} + --set serviceAccount.name=external-secrets-sa + --wait --timeout 5m + + hub:cluster-secret-store: + desc: Apply ClusterSecretStore on hub + cmds: + - task: hub:kubeconfig + - | + cat <<'EOF' | sed "s|AWS_REGION|{{.AWS_REGION}}|g" | KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl apply -f - + apiVersion: external-secrets.io/v1 + kind: ClusterSecretStore + metadata: + name: aws-secrets-manager + spec: + provider: + aws: + service: SecretsManager + region: AWS_REGION + EOF + + hub:seed-secret: + desc: Create minimal seed cluster secret on the hub (repo coords + fleet labels only) + vars: + CLUSTER_ARN: + sh: aws eks describe-cluster --name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} --query 'cluster.arn' --output text 2>/dev/null || echo "arn:aws:eks:{{.AWS_REGION}}:{{.AWS_ACCOUNT_ID}}:cluster/{{.HUB_CLUSTER_NAME}}" + cmds: + - printf '{{.C_STEP}}▸ Getting hub kubeconfig...{{.C_RESET}}\n' + - task: hub:kubeconfig + - printf '{{.C_STEP}}▸ Creating seed cluster secret on hub...{{.C_RESET}}\n' + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl create namespace argocd --dry-run=client -o yaml | KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl apply -f - + - | + cat <<'EOF' | sed "s|CLUSTER_NAME|{{.HUB_CLUSTER_NAME}}|g; s|CLUSTER_ARN|{{.CLUSTER_ARN}}|g; s|FLEET_REPO_URL|{{.FLEET_REPO_URL}}|g; s|FLEET_REPO_REVISION|{{.FLEET_REPO_REVISION}}|g; s|FLEET_REPO_BASEPATH|{{.FLEET_REPO_BASEPATH}}|g; s|REPO_URL|{{.REPO_URL}}|g; s|REPO_REVISION|{{.REPO_REVISION}}|g; s|REPO_BASEPATH|{{.REPO_BASEPATH}}|g" | KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl apply -f - + apiVersion: v1 + kind: Secret + type: Opaque + metadata: + name: CLUSTER_NAME + namespace: argocd + annotations: + addonsRepoURL: "REPO_URL" + addonsRepoRevision: "REPO_REVISION" + addonsRepoBasepath: "REPO_BASEPATH" + fleetRepoURL: "FLEET_REPO_URL" + fleetRepoRevision: "FLEET_REPO_REVISION" + fleetRepoBasepath: "FLEET_REPO_BASEPATH" + aws_cluster_name: {{.HUB_CLUSTER_NAME}} + labels: + argocd.argoproj.io/secret-type: cluster + environment: control-plane + fleet_member: control-plane + stringData: + name: CLUSTER_NAME + server: "CLUSTER_ARN" + config: '{"tlsClientConfig":{"insecure":false}}' + project: default + EOF + - printf '{{.C_OK}}✓ Seed secret created on hub{{.C_RESET}}\n' + - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig + + hub:apply-root-appset: + desc: Apply root-appset to hub + cmds: + - task: hub:kubeconfig + # Gate: the ArgoCD EKS Capability reconciles asynchronously after the EksCluster is ACTIVE. + # root-appset.yaml contains an ApplicationSet, so we must wait until applicationsets.argoproj.io + # is registered and Established before applying, otherwise kubectl fails with + # "no matches for kind ApplicationSet in version argoproj.io/v1alpha1". + - printf '{{.C_STEP}}▸ Waiting for ArgoCD capability CRDs (ApplicationSet/Application) to be served...{{.C_RESET}}\n' + - | + KC={{.ROOT_DIR}}/private/hub-kubeconfig + for i in $(seq 1 60); do + if KUBECONFIG=$KC kubectl get crd applicationsets.argoproj.io applications.argoproj.io >/dev/null 2>&1; then + break + fi + printf '{{.C_INFO}} [%s/60] ArgoCD CRDs not ready yet...{{.C_RESET}}\n' "$i" + sleep 10 + done + if ! KUBECONFIG=$KC kubectl get crd applicationsets.argoproj.io >/dev/null 2>&1; then + printf '{{.C_ERR}}ERROR: ArgoCD ApplicationSet CRD never became available — the argocd EKS Capability has not reconciled.{{.C_RESET}}\n' + exit 1 + fi + KUBECONFIG=$KC kubectl wait --for=condition=Established \ + crd/applicationsets.argoproj.io crd/applications.argoproj.io --timeout=120s + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl apply -f {{.GITOPS_ROOT}}/bootstrap/root-appset.yaml + - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig + - printf '{{.C_OK}}✓ Root appset applied. Hub is now self-managing.{{.C_RESET}}\n' + + hub:force-eso-refresh: + desc: Force ESO to immediately refresh cluster secret so annotations are populated before ArgoCD wave 5 apps render + internal: true + cmds: + - task: hub:kubeconfig + - | + KC={{.ROOT_DIR}}/private/hub-kubeconfig + printf '{{.C_STEP}}▸ Forcing ESO refresh of cluster secret annotations...{{.C_RESET}}\n' + ES=$(KUBECONFIG=$KC kubectl -n argocd get externalsecret -o name 2>/dev/null | grep -i fleet-secret | grep {{.HUB_CLUSTER_NAME}} | head -1 || true) + if [ -n "$ES" ]; then + KUBECONFIG=$KC kubectl -n argocd annotate "$ES" \ + force-sync="$(date +%s)" --overwrite >/dev/null 2>&1 || true + for i in $(seq 1 12); do + VAL=$(KUBECONFIG=$KC kubectl get secret {{.HUB_CLUSTER_NAME}} -n argocd \ + -o jsonpath='{.metadata.annotations.exposure_mode}' 2>/dev/null) + [ -n "$VAL" ] && printf '{{.C_OK}}✓ Cluster secret annotations populated (exposure_mode=%s){{.C_RESET}}\n' "$VAL" && break + sleep 5 + done + else + printf '{{.C_INFO}} fleet-secret ESO not found yet — waiting 30s{{.C_RESET}}\n' + sleep 30 + fi + - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig + + hub:bootstrap-crossplane-identity: + desc: Seed hub Crossplane provider creds (provider-aws-iam/eks pod identities) so crossplane-base can provision + internal: true + status: + # Skip once Crossplane has already provisioned a downstream provider role — i.e. the + # providers are credentialed and working, so the bootstrap is done. + - aws iam get-role --role-name {{.HUB_CLUSTER_NAME}}-CrossplaneAMPProviderRole >/dev/null 2>&1 + cmds: + - printf '{{.C_STEP}}▸ Bootstrapping Crossplane provider pod identities (iam/eks)...{{.C_RESET}}\n' + - | + ROLE_NAME="{{.HUB_CLUSTER_NAME}}-crossplane-provider" + TRUST='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"pods.eks.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]}' + aws iam get-role --role-name "$ROLE_NAME" >/dev/null 2>&1 || \ + aws iam create-role --role-name "$ROLE_NAME" --assume-role-policy-document "$TRUST" \ + --description "Bootstrap creds for crossplane provider-aws-iam/eks" >/dev/null + aws iam attach-role-policy --role-name "$ROLE_NAME" \ + --policy-arn arn:aws:iam::aws:policy/AdministratorAccess >/dev/null 2>&1 || true + ROLE_ARN="arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/$ROLE_NAME" + for SA in provider-aws-iam provider-aws-eks; do + EXISTS=$(aws eks list-pod-identity-associations --cluster-name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} \ + --query "associations[?serviceAccount=='$SA'] | [0].associationId" --output text 2>/dev/null) + if [ -z "$EXISTS" ] || [ "$EXISTS" = "None" ]; then + aws eks create-pod-identity-association --cluster-name {{.HUB_CLUSTER_NAME}} --region {{.AWS_REGION}} \ + --namespace crossplane-system --service-account "$SA" --role-arn "$ROLE_ARN" >/dev/null + echo " created PIA crossplane-system/$SA" + else + echo " PIA crossplane-system/$SA already exists" + fi + done + # NOTE: We deliberately do NOT restart the Crossplane providers here. + # This task runs before crossplane-base is deployed, so the provider pods + # don't exist yet — they will start AFTER these PIAs exist and therefore + # pick up credentials at startup. The downstream providers (amp/rds/etc.) + # and any that started early are restarted (non-fatally) by the phase1 + # task hub:restart-identity-pods, which waits for all PIAs Ready first. + - printf '{{.C_OK}}✓ Crossplane provider pod identities created (providers credentialed on startup/phase1 restart).{{.C_RESET}}\n' + + hub:wait-for-sync: + desc: Wait for hub ArgoCD apps to sync + vars: + MAX_OUTOFSYNC: '{{.MAX_OUTOFSYNC | default "5"}}' + cmds: + - task: hub:kubeconfig + - printf '{{.C_STEP}}▸ Waiting for apps to appear...{{.C_RESET}}\n' + - | + TIMEOUT=300; ELAPSED=0 + while [ $ELAPSED -lt $TIMEOUT ]; do + TOTAL=$(KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl get applications.argoproj.io -n argocd --no-headers 2>/dev/null | wc -l | tr -d ' ') + [ "$TOTAL" -gt 3 ] && printf '{{.C_INFO}} %s apps detected{{.C_RESET}}\n' "$TOTAL" && break + printf '{{.C_INFO}} %s apps so far, waiting...{{.C_RESET}}\n' "$TOTAL" + sleep 10; ELAPSED=$((ELAPSED + 10)) + done + - printf '{{.C_STEP}}▸ Waiting for apps to sync...{{.C_RESET}}\n' + - | + TIMEOUT=1800; INTERVAL=15; ELAPSED=0 + while [ $ELAPSED -lt $TIMEOUT ]; do + APP_OUTPUT=$(KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl get applications.argoproj.io -n argocd --no-headers 2>/dev/null || echo "") + TOTAL=$(echo "$APP_OUTPUT" | awk '$2 != "Unknown" && NF>0' | wc -l | tr -d ' ') + SYNCED=$(echo "$APP_OUTPUT" | awk '$2 == "Synced"' | wc -l | tr -d ' ') + UNKNOWN=$(echo "$APP_OUTPUT" | awk '$2 == "Unknown"' | wc -l | tr -d ' ') + OUTOFSYNC=$((TOTAL - SYNCED)) + printf '{{.C_INFO}} Apps: %s/%s synced, %s unknown, %s out-of-sync (%ss elapsed){{.C_RESET}}\n' "$SYNCED" "$TOTAL" "$UNKNOWN" "$OUTOFSYNC" "$ELAPSED" + if [ "$TOTAL" -gt 0 ] && [ "$OUTOFSYNC" -le {{.MAX_OUTOFSYNC}} ]; then + printf '{{.C_OK}}✓ All %s apps synced (within %s tolerance, %s unknown).{{.C_RESET}}\n' "$SYNCED" "{{.MAX_OUTOFSYNC}}" "$UNKNOWN" + break + fi + sleep $INTERVAL; ELAPSED=$((ELAPSED + INTERVAL)) + done + [ $ELAPSED -ge $TIMEOUT ] && printf '{{.C_WARN}}⚠ Sync wait timed out after %ss — continuing.{{.C_RESET}}\n' "$TIMEOUT" || true + - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig + + status: + desc: Check bootstrap status + cmds: + - printf '{{.C_INFO}}=== Kind ==={{.C_RESET}}\n' && kind get clusters 2>/dev/null | grep "{{.KIND_CLUSTER_NAME}}" || echo "Not running" + - printf '{{.C_INFO}}=== Helm ==={{.C_RESET}}\n' && helm list -A 2>/dev/null || echo "None" + - printf '{{.C_INFO}}=== KRO RGDs ==={{.C_RESET}}\n' && kubectl get resourcegraphdefinitions 2>/dev/null || echo "None" + - printf '{{.C_INFO}}=== KRO Instances ==={{.C_RESET}}\n' && kubectl get eksclusterwithvpc -A 2>/dev/null || echo "None" + + destroy-kind: + desc: Delete Kind cluster only (hub persists) + status: + - "! kind get clusters 2>/dev/null | grep -q '^{{.KIND_CLUSTER_NAME}}$'" + cmds: + - kind delete cluster --name {{.KIND_CLUSTER_NAME}} + + destroy: + desc: Full teardown — delete KRO instances, wait for ACK to clean up AWS resources, delete Kind + vars: + CLEANUP_KIND: '{{.CLEANUP_KIND | default "false"}}' + cmds: + - kind get kubeconfig --name {{.KIND_CLUSTER_NAME}} > {{.KUBECONFIG}} + - task: credentials:refresh + # Delete the hub claim — KRO cascades to ACK which deletes AWS resources + - printf '{{.C_ERR}}▸ Deleting hub cluster claim...{{.C_RESET}}\n' + - kubectl delete eksclusterwithvpc {{.HUB_CLUSTER_NAME}} -n {{.HUB_CLUSTER_NAME}} --timeout=60s 2>/dev/null || true + - printf '{{.C_STEP}}▸ Waiting for EKS cluster deletion...{{.C_RESET}}\n' + - | + for i in $(seq 1 120); do + EXISTS=$(kubectl get ekscluster {{.HUB_CLUSTER_NAME}} -n {{.HUB_CLUSTER_NAME}} 2>/dev/null && echo "yes" || echo "no") + [ "$EXISTS" = "no" ] && break + printf '{{.C_INFO}} [%s/120] Waiting for deletion...{{.C_RESET}}\n' "$i" + sleep 15 + done + # Clean up Secrets Manager + - aws secretsmanager delete-secret --secret-id {{.HUB_CLUSTER_NAME}}/config --force-delete-without-recovery --region {{.AWS_REGION}} 2>/dev/null || true + - aws secretsmanager delete-secret --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --force-delete-without-recovery --region {{.AWS_REGION}} 2>/dev/null || true + - cmd: | + if [ "{{.CLEANUP_KIND}}" = "true" ]; then + kind delete cluster --name {{.KIND_CLUSTER_NAME}} + else + echo "Kind preserved. Run 'task destroy-kind' to remove." + fi + - printf '{{.C_OK}}✓ Destroy complete.{{.C_RESET}}\n' + + # ---- from kind-crossplane ---- + urls: + desc: Print platform URLs (alias for print-urls) + aliases: [print-urls] + vars: + INGRESS_DOMAIN: '{{.DOMAIN}}' + GITLAB_DOMAIN: + sh: | + if [ -f {{.ROOT_DIR}}/private/gitlab-cloudfront-domain ]; then + cat {{.ROOT_DIR}}/private/gitlab-cloudfront-domain + else + echo "{{.DOMAIN}}" + fi + KC_ADMIN_PASSWORD: + sh: aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text 2>/dev/null | jq -r '.keycloak_admin_password // empty' + USER_PASSWORD: + sh: aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text 2>/dev/null | jq -r '.user_password // "workshop-user-2026"' + ARGOCD_URL: + sh: aws eks describe-capability --cluster-name {{.HUB_CLUSTER_NAME}} --capability-name argocd --query 'capability.configuration.argoCd.serverUrl' --output text --region {{.AWS_REGION}} 2>/dev/null || echo "" + cmds: + - | + printf '{{.C_OK}}Platform URLs{{.C_RESET}}\n' + printf ' Backstage: https://{{.INGRESS_DOMAIN}}/backstage\n' + printf ' Keycloak: https://{{.INGRESS_DOMAIN}}/keycloak\n' + printf ' Argo WF: https://{{.INGRESS_DOMAIN}}/argo-workflows\n' + printf ' Kargo: https://{{.INGRESS_DOMAIN}}/\n' + printf ' Grafana: https://{{.INGRESS_DOMAIN}}/grafana\n' + printf ' ArgoCD: {{.ARGOCD_URL}}\n' + printf '\n' + printf '{{.C_INFO}}Credentials{{.C_RESET}}\n' + printf ' User: user1 / {{.USER_PASSWORD}}\n' + printf ' Keycloak: admin / {{.KC_ADMIN_PASSWORD}}\n' + + + + # ---- from kind-crossplane ---- + # ---- from kind-crossplane ---- + hub:restart-langfuse: + desc: Restart Langfuse to pick up Keycloak client secret (created by config job after initial deploy) + cmds: + - task: hub:kubeconfig + - printf '{{.C_STEP}}▸ Restarting Langfuse to pick up Keycloak client secret...{{.C_RESET}}\n' + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout restart deploy -n langfuse langfuse 2>/dev/null || true + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout status deploy -n langfuse langfuse --timeout=120s 2>/dev/null || true + - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig + - printf '{{.C_OK}}✓ Langfuse restarted with Keycloak SSO credentials.{{.C_RESET}}\n' + + + # ---- from kind-crossplane ---- + hub:wait-for-full-sync: + desc: Wait for ALL apps (including spokes) to sync after provider restart + vars: + MAX_OUTOFSYNC: '{{.MAX_OUTOFSYNC | default "5"}}' + cmds: + - task: hub:kubeconfig + - printf '{{.C_STEP}}▸ Waiting for all apps to sync (spoke clusters provisioning)...{{.C_RESET}}\n' + - | + TIMEOUT=2400 + INTERVAL=30 + ELAPSED=0 + while [ $ELAPSED -lt $TIMEOUT ]; do + APP_OUTPUT=$(KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl get applications.argoproj.io -n argocd --no-headers 2>/dev/null || echo "") + TOTAL=$(echo "$APP_OUTPUT" | awk 'NF>0' | wc -l | tr -d ' ') + SYNCED=$(echo "$APP_OUTPUT" | awk '$2 == "Synced"' | wc -l | tr -d ' ') + UNKNOWN=$(echo "$APP_OUTPUT" | awk '$2 == "Unknown"' | wc -l | tr -d ' ') + OUTOFSYNC=$((TOTAL - SYNCED - UNKNOWN)) + printf '{{.C_INFO}} Apps: %s/%s synced, %s unknown, %s out-of-sync (%ss elapsed){{.C_RESET}}\n' "$SYNCED" "$TOTAL" "$UNKNOWN" "$OUTOFSYNC" "$ELAPSED" + if [ "$TOTAL" -gt 0 ] && [ "$OUTOFSYNC" -le {{.MAX_OUTOFSYNC}} ]; then + printf '{{.C_OK}}✓ All %s apps synced (within %s tolerance, %s unknown).{{.C_RESET}}\n' "$SYNCED" "{{.MAX_OUTOFSYNC}}" "$UNKNOWN" + break + fi + sleep $INTERVAL + ELAPSED=$((ELAPSED + INTERVAL)) + done + if [ $ELAPSED -ge $TIMEOUT ]; then + printf '{{.C_ERR}}⚠ Timed out after %ss. %s app(s) still not synced:{{.C_RESET}}\n' "$TIMEOUT" "$OUTOFSYNC" + echo "$APP_OUTPUT" | awk '$2 != "Synced" && NF>0 {printf " - %s (%s/%s)\n", $1, $3, $2}' + printf '{{.C_STEP}}▸ Continuing — spoke clusters may still be provisioning.{{.C_RESET}}\n' + fi + - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig + + + # ---- from kind-crossplane ---- + hub:wait-for-providers: + desc: Wait for all Crossplane providers to be Healthy (ensures ec2/rds/etc. CRDs available before spoke provisioning) + cmds: + - task: hub:kubeconfig + - | + KC={{.ROOT_DIR}}/private/hub-kubeconfig + [ ! -f "$KC" ] && KC={{.ROOT_DIR}}/private/kubeconfig + printf '{{.C_STEP}}▸ Waiting for all Crossplane providers to be Healthy...{{.C_RESET}}\n' + for i in $(seq 1 60); do + UNHEALTHY=$(KUBECONFIG=$KC kubectl get providers.pkg.crossplane.io -o jsonpath='{.items[?(@.status.conditions[?(@.type=="Healthy")].status!="True")].metadata.name}' 2>/dev/null || true) + UNHEALTHY=$(echo "$UNHEALTHY" | wc -w | tr -d ' ') + TOTAL=$(KUBECONFIG=$KC kubectl get providers.pkg.crossplane.io --no-headers 2>/dev/null | wc -l | tr -d ' ' || echo "0") + if [ "$UNHEALTHY" = "0" ] && [ "$TOTAL" -gt 0 ]; then + printf '{{.C_OK}}✓ All %s Crossplane providers Healthy.{{.C_RESET}}\n' "$TOTAL" + break + fi + [ $((i % 4)) -eq 0 ] && printf '{{.C_INFO}} [%ds] %s/%s providers Healthy...{{.C_RESET}}\n' "$((i*15))" "$((TOTAL-UNHEALTHY))" "$TOTAL" || true + # After 5min, force-sync crossplane-base to unblock stuck ArgoCD retry loops + if [ "$i" -eq 20 ]; then + printf '{{.C_WARN}} Providers not all Healthy after 5min — force-syncing crossplane-base...{{.C_RESET}}\n' + KUBECONFIG=$KC kubectl patch application "crossplane-base-{{.HUB_CLUSTER_NAME}}" -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"revision":"HEAD"}}}' 2>/dev/null || true + fi + sleep 15 + done + printf '{{.C_WARN}} Providers wait complete.{{.C_RESET}}\n' + + + # ---- from kind-crossplane ---- + secrets-manager:seed-secrets: + desc: Seed platform user credentials into Secrets Manager for ExternalSecrets + # TODO(#757): gitlab_root_password and git_token should move to peeks-hub/gitlab + # (separate secret, gated on enable_gitlab) so the platform can run without GitLab. + # https://github.com/aws-samples/appmod-blueprints/issues/757 + cmds: + - | + SECRET_KEY="{{.HUB_CLUSTER_NAME}}/secrets" + USER_PASS=$(aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query SecretString --output text | jq -r .user_password) + # Merge into the existing secret so a re-run backfills any keys a previous + # (partial) install left out — and never rotates values live consumers + # depend on. First run: no secret yet, start from {} and create-then-put. + EXISTING=$(aws secretsmanager get-secret-value --secret-id "$SECRET_KEY" --region {{.AWS_REGION}} --query SecretString --output text 2>/dev/null || echo "{}") + # Reuse existing hash/key on re-runs (don't rotate them out from under Kargo etc.) + USER_PASS_HASH=$(echo "$EXISTING" | jq -r '.user_password_hash // ""') + if [ -z "$USER_PASS_HASH" ]; then + USER_PASS_HASH=$(echo -n "$USER_PASS" | python3 -c "import sys,bcrypt; print(bcrypt.hashpw(sys.stdin.buffer.read(), bcrypt.gensalt()).decode())" 2>/dev/null || echo "") + fi + USER_PASS_KEY=$(echo "$EXISTING" | jq -r '.user_password_key // ""') + if [ -z "$USER_PASS_KEY" ]; then + USER_PASS_KEY=$(echo -n "$USER_PASS" | base64) + fi + UPDATED=$(echo "$EXISTING" | jq \ + --arg user_password "$USER_PASS" \ + --arg user_password_hash "$USER_PASS_HASH" \ + --arg user_password_key "$USER_PASS_KEY" \ + '. + {user_password: $user_password, user_password_hash: $user_password_hash, user_password_key: $user_password_key}') + # DevLake credentials. These belong here (NOT in seed-observability) so the + # devlake-encryption-secret ExternalSecret resolves during the normal install, + # independent of the AMP/AMG observability flow — which waits up to 30 min for + # the AMP workspace and is frequently re-run separately (task observability-seed). + # Gating these behind observability left DevLake's ExternalSecret in + # SecretSyncedError whenever observability seeding didn't complete. + # Generate only if absent so re-runs don't break existing encrypted connections. + # - encryption secret: 128 uppercase chars (DevLake requirement) + if ! echo "$UPDATED" | jq -e 'has("devlake_encryption_secret")' >/dev/null; then + DEVLAKE_ENC=$(openssl rand -base64 2000 | tr -dc 'A-Z' | fold -w 128 | head -n 1) + UPDATED=$(echo "$UPDATED" | jq --arg p "$DEVLAKE_ENC" '. + {devlake_encryption_secret: $p}') + fi + if ! echo "$UPDATED" | jq -e 'has("devlake_mysql_password")' >/dev/null; then + DEVLAKE_MYSQL=$(openssl rand -base64 18 | tr -d '/+=' | head -c 24) + UPDATED=$(echo "$UPDATED" | jq --arg p "$DEVLAKE_MYSQL" '. + {devlake_mysql_password: $p}') + fi + # grafana_mysql_password: the devlake `mysql-passwords` ExternalSecret reads + # BOTH devlake_mysql_password AND grafana_mysql_password — ESO fails the whole + # ExternalSecret if either property is missing, so this must be seeded here too + # (it was previously only generated in the AMP-gated seed-observability flow, + # leaving mysql-passwords in SecretSyncedError and devlake-peeks-hub Degraded). + if ! echo "$UPDATED" | jq -e 'has("grafana_mysql_password")' >/dev/null; then + GRAFANA_MYSQL=$(openssl rand -base64 18 | tr -d '/+=' | head -c 24) + UPDATED=$(echo "$UPDATED" | jq --arg p "$GRAFANA_MYSQL" '. + {grafana_mysql_password: $p}') + fi + # backstage_postgres_password + git_token: the backstage-postgres and + # git-credentials ExternalSecrets read these from /secrets. They were + # previously only generated in the AMP-gated seed-observability flow, so on a + # fresh install where observability seeding hadn't completed Backstage stayed + # OutOfSync/Degraded ('key backstage_postgres_password does not exist') and + # $BACKSTAGE_URL returned 503 — blocking every Backstage-dependent module. + # Seed them here (in the always-runs early phase) instead. git_token mirrors + # the workshop user password (GitLab PAT == user password in this workshop). + if ! echo "$UPDATED" | jq -e 'has("backstage_postgres_password")' >/dev/null; then + BS_PG_PASS=$(openssl rand -base64 18 | tr -d '/+=' | head -c 24) + UPDATED=$(echo "$UPDATED" | jq --arg p "$BS_PG_PASS" '. + {backstage_postgres_password: $p}') + fi + # git_token is seeded by CDK at deploy time (root-) — no runtime seeding needed. + aws secretsmanager restore-secret --secret-id "$SECRET_KEY" --region {{.AWS_REGION}} 2>/dev/null || true + aws secretsmanager create-secret \ + --name "$SECRET_KEY" \ + --secret-string "$UPDATED" \ + --region {{.AWS_REGION}} 2>/dev/null \ + || aws secretsmanager put-secret-value \ + --secret-id "$SECRET_KEY" \ + --secret-string "$UPDATED" \ + --region {{.AWS_REGION}} + echo "Seeded Secrets Manager: $SECRET_KEY" + + + # ---- from kind-crossplane ---- + secrets-manager:seed-observability: + desc: Seed observability credentials (AMP endpoint, AMG API key) into Secrets Manager + vars: + KC_DOMAIN: '{{.DOMAIN}}' + cmds: + - | + set -e + SECRET_KEY="{{.HUB_CLUSTER_NAME}}/secrets" + + # Poll for AMP workspace (up to 60 min — Crossplane provisioning of AMP/AMG + # can take well over 30 min on busy accounts; the previous 30 min ceiling + # was too short and left Grafana unseeded on slow installs). + # Use crossplane-name tag to identify the correct workspace (avoids ambiguity with duplicates) + AMP_WORKSPACE_ID="" + printf '{{.C_STEP}}▸ Waiting for AMP workspace...{{.C_RESET}}\n' + for i in $(seq 1 120); do + for WS_ID in $(aws amp list-workspaces --region {{.AWS_REGION}} --query "workspaces[?alias=='{{.RESOURCE_PREFIX}}-observability-amp'].workspaceId" --output text 2>/dev/null); do + TAG_VAL=$(aws amp list-tags-for-resource --resource-arn "arn:aws:aps:{{.AWS_REGION}}:{{.AWS_ACCOUNT_ID}}:workspace/$WS_ID" --region {{.AWS_REGION}} --query "tags.\"crossplane-name\"" --output text 2>/dev/null || echo "") + if [ "$TAG_VAL" = "{{.RESOURCE_PREFIX}}-amp" ]; then + AMP_WORKSPACE_ID="$WS_ID" + break + fi + done + if [ -n "$AMP_WORKSPACE_ID" ]; then + break + fi + printf ' [%d/120] AMP workspace not ready yet...\n' "$i" + sleep 30 + done + if [ -z "$AMP_WORKSPACE_ID" ] || [ "$AMP_WORKSPACE_ID" = "None" ]; then + echo "" + echo "============================================================" + echo "WARN: AMP workspace not created after 60 min." + echo " The peeks-observability-seed.timer will keep retrying this" + echo " automatically; or run 'task observability-seed' manually once" + echo " the observability-aws ArgoCD app is Synced and AMP/AMG are ACTIVE." + echo "============================================================" + exit 0 + fi + AMP_ENDPOINT=$(aws amp describe-workspace --workspace-id "$AMP_WORKSPACE_ID" --region {{.AWS_REGION}} --query "workspace.prometheusEndpoint" --output text) + echo "Found AMP endpoint: $AMP_ENDPOINT" + + # Poll for AMG workspace (up to 15 min) + # Use crossplane-name tag to identify the correct workspace + AMG_WORKSPACE_ID="" + printf '{{.C_STEP}}▸ Waiting for AMG workspace...{{.C_RESET}}\n' + for i in $(seq 1 30); do + for WS in $(aws grafana list-workspaces --region {{.AWS_REGION}} --query "workspaces[?name=='{{.RESOURCE_PREFIX}}-observability' && status=='ACTIVE'].id" --output text 2>/dev/null); do + TAG_VAL=$(aws grafana describe-workspace --workspace-id "$WS" --region {{.AWS_REGION}} --query "workspace.tags.\"crossplane-name\"" --output text 2>/dev/null || echo "") + if [ "$TAG_VAL" = "{{.RESOURCE_PREFIX}}-amg" ]; then + AMG_WORKSPACE_ID="$WS" + break + fi + done + if [ -n "$AMG_WORKSPACE_ID" ]; then + break + fi + printf ' [%d/30] AMG workspace not ready yet...\n' "$i" + sleep 30 + done + if [ -z "$AMG_WORKSPACE_ID" ] || [ "$AMG_WORKSPACE_ID" = "None" ]; then + echo "WARN: AMG workspace not found. Run 'task observability-seed' later." + exit 0 + fi + AMG_ENDPOINT=$(aws grafana describe-workspace --workspace-id "$AMG_WORKSPACE_ID" --region {{.AWS_REGION}} --query "workspace.endpoint" --output text) + echo "Found AMG endpoint: $AMG_ENDPOINT" + + # Create AMG service account + token (admin, 30 days) — API keys removed in Grafana 12 + SA_NAME="crossplane-$(date +%s)" + SA_ID=$(aws grafana create-workspace-service-account --workspace-id "$AMG_WORKSPACE_ID" --grafana-role ADMIN --name "$SA_NAME" --region {{.AWS_REGION}} --query "id" --output text 2>/dev/null || echo "") + if [ -z "$SA_ID" ] || [ "$SA_ID" = "None" ]; then + echo "WARN: Could not create AMG service account. Skipping." + exit 0 + fi + AMG_API_KEY=$(aws grafana create-workspace-service-account-token --workspace-id "$AMG_WORKSPACE_ID" --service-account-id "$SA_ID" --name "token-$SA_NAME" --seconds-to-live 2592000 --region {{.AWS_REGION}} --query "serviceAccountToken.key" --output text 2>/dev/null || echo "") + if [ -z "$AMG_API_KEY" ]; then + echo "WARN: Could not create AMG service account token. Skipping." + exit 0 + fi + + # Install required Grafana plugins (X-Ray for trace visualization) + printf '{{.C_STEP}}▸ Installing AMG plugins (X-Ray)...{{.C_RESET}}\n' + curl -s -X POST -H "Authorization: Bearer $AMG_API_KEY" \ + -H "Content-Type: application/json" \ + "https://$AMG_ENDPOINT/api/plugins/grafana-x-ray-datasource/install" 2>/dev/null || true + echo "X-Ray plugin installed" + + # Merge into existing secret (preserves other keys). On first run + # the secret doesn't exist yet — `get-secret-value` returns empty, + # we start from `{}`, and `put-secret-value` would fail with + # ResourceNotFoundException, so create-then-put. + EXISTING=$(aws secretsmanager get-secret-value --secret-id "$SECRET_KEY" --region {{.AWS_REGION}} --query "SecretString" --output text 2>/dev/null || echo "{}") + # Pull user_password from keycloak secret for backstage ExternalSecrets + KC_SECRET=$(aws secretsmanager get-secret-value --secret-id "{{.HUB_CLUSTER_NAME}}/keycloak" --region {{.AWS_REGION}} --query "SecretString" --output text 2>/dev/null || echo "{}") + USER_PASS=$(echo "$KC_SECRET" | jq -r '.user_password // "workshop-user-2026"') + # Generate backstage_postgres_password if not already present + if ! echo "$EXISTING" | jq -e 'has("backstage_postgres_password")' >/dev/null 2>&1; then + BS_PG_PASS=$(openssl rand -base64 18 | tr -d '/+=' | head -c 24) + else + BS_PG_PASS=$(echo "$EXISTING" | jq -r '.backstage_postgres_password') + fi + # Kargo admin credentials: bcrypt hash of user password + random signing key. + # Reuse existing values on re-runs so we don't rotate them out from under Kargo. + KARGO_HASH=$(echo "$EXISTING" | jq -r '.user_password_hash // ""') + if [ -z "$KARGO_HASH" ]; then + pip install bcrypt --quiet 2>/dev/null || true + KARGO_HASH=$(python3 -c "import bcrypt,sys; print(bcrypt.hashpw(sys.argv[1].encode(), bcrypt.gensalt(rounds=10)).decode())" "$USER_PASS" 2>/dev/null) + fi + KARGO_KEY=$(echo "$EXISTING" | jq -r '.user_password_key // ""') + if [ -z "$KARGO_KEY" ]; then + KARGO_KEY=$(openssl rand -base64 30 | tr -d '/+=' | head -c 40) + fi + UPDATED=$(echo "$EXISTING" | jq \ + --arg amp_url "${AMP_ENDPOINT}api/v1/remote_write" \ + --arg amp_query_url "$AMP_ENDPOINT" \ + --arg amp_region "{{.AWS_REGION}}" \ + --arg grafana_api_key "$AMG_API_KEY" \ + --arg grafana_url "$AMG_ENDPOINT" \ + --arg user_password "$USER_PASS" \ + --arg backstage_postgres_password "$BS_PG_PASS" \ + --arg user_password_hash "$KARGO_HASH" \ + --arg user_password_key "$KARGO_KEY" \ + '. + {amp_endpoint_url: $amp_query_url, amp_region: $amp_region, grafana_api_key: $grafana_api_key, grafana_url: $grafana_url, user_password: $user_password, backstage_postgres_password: $backstage_postgres_password, user_password_hash: $user_password_hash, user_password_key: $user_password_key}') + # Generate the grafana_mysql_password the grafana-dashboards + # ExternalSecret expects, but only if it isn't already there + # (so re-runs don't rotate it out from under live consumers). + if ! echo "$UPDATED" | jq -e 'has("grafana_mysql_password")' >/dev/null; then + GRAFANA_MYSQL_PASSWORD=$(openssl rand -base64 18 | tr -d '/+=' | head -c 24) + UPDATED=$(echo "$UPDATED" | jq --arg p "$GRAFANA_MYSQL_PASSWORD" '. + {grafana_mysql_password: $p}') + fi + # DevLake encryption secret (128 uppercase chars, per DevLake requirement). + # Reuse on re-runs so we don't break existing encrypted connections. + if ! echo "$UPDATED" | jq -e 'has("devlake_encryption_secret")' >/dev/null; then + DEVLAKE_ENC=$(openssl rand -base64 2000 | tr -dc 'A-Z' | fold -w 128 | head -n 1) + UPDATED=$(echo "$UPDATED" | jq --arg p "$DEVLAKE_ENC" '. + {devlake_encryption_secret: $p}') + fi + if ! echo "$UPDATED" | jq -e 'has("devlake_mysql_password")' >/dev/null; then + DEVLAKE_MYSQL=$(openssl rand -base64 18 | tr -d '/+=' | head -c 24) + UPDATED=$(echo "$UPDATED" | jq --arg p "$DEVLAKE_MYSQL" '. + {devlake_mysql_password: $p}') + fi + aws secretsmanager restore-secret --secret-id "$SECRET_KEY" --region {{.AWS_REGION}} 2>/dev/null || true + aws secretsmanager create-secret \ + --name "$SECRET_KEY" \ + --secret-string "$UPDATED" \ + --region {{.AWS_REGION}} 2>/dev/null \ + || aws secretsmanager put-secret-value \ + --secret-id "$SECRET_KEY" \ + --secret-string "$UPDATED" \ + --region {{.AWS_REGION}} + echo "Seeded observability credentials into $SECRET_KEY" + + # Spoke amp-workspace-secrets ExternalSecrets (Argo Rollouts metric queries on the + # spokes) read a SEPARATE secret /platform/amp with properties + # 'amp-workspace' (the AMP query endpoint) and 'amp-region'. In the terraform flow + # this was created by common/secrets.tf; the kind-crossplane flow never created it, + # so every spoke amp-workspace-secrets-* ExternalSecret stayed in SecretSyncedError. + # Create/refresh it here now that AMP_ENDPOINT is known. + AMP_SECRET_KEY="{{.RESOURCE_PREFIX}}/platform/amp" + AMP_SECRET_VALUE=$(jq -n --arg r "{{.AWS_REGION}}" --arg w "$AMP_ENDPOINT" '{"amp-region": $r, "amp-workspace": $w}') + aws secretsmanager create-secret \ + --name "$AMP_SECRET_KEY" \ + --secret-string "$AMP_SECRET_VALUE" \ + --region {{.AWS_REGION}} 2>/dev/null \ + || aws secretsmanager put-secret-value \ + --secret-id "$AMP_SECRET_KEY" \ + --secret-string "$AMP_SECRET_VALUE" \ + --region {{.AWS_REGION}} + echo "Seeded AMP workspace secret: $AMP_SECRET_KEY" + + # Update hub cluster config secret with AMG URL for Keycloak SAML and grafana-dashboards + CONFIG_KEY="{{.HUB_CLUSTER_NAME}}/config" + CONFIG_EXISTING=$(aws secretsmanager get-secret-value --secret-id "$CONFIG_KEY" --region {{.AWS_REGION}} --query "SecretString" --output text) + CONFIG_UPDATED=$(echo "$CONFIG_EXISTING" | jq \ + --arg grafana_url "$AMG_ENDPOINT" \ + '.metadata = (.metadata | fromjson | . + {aws_grafana_url: $grafana_url} | tojson)') + aws secretsmanager put-secret-value \ + --secret-id "$CONFIG_KEY" \ + --secret-string "$CONFIG_UPDATED" \ + --region {{.AWS_REGION}} + echo "Updated hub config with AMG URL: $AMG_ENDPOINT" + + # Populate spoke scraper values into hub Secrets Manager secret (dynamic, no file commits needed) + CONFIG_KEY="{{.HUB_CLUSTER_NAME}}/config" + CONFIG_EXISTING=$(aws secretsmanager get-secret-value --secret-id "$CONFIG_KEY" --region {{.AWS_REGION}} --query "SecretString" --output text) + SPOKE_DEV_ARN="" + SPOKE_DEV_SUBNETS="" + SPOKE_DEV_SG="" + SPOKE_PROD_ARN="" + SPOKE_PROD_SUBNETS="" + SPOKE_PROD_SG="" + for CLUSTER in spoke-dev spoke-prod; do + SPOKE_ARN=$(aws eks describe-cluster --name "$CLUSTER" --region {{.AWS_REGION}} --query 'cluster.arn' --output text 2>/dev/null || echo "") + if [ -n "$SPOKE_ARN" ] && [ "$SPOKE_ARN" != "None" ]; then + SPOKE_ALL_SUBNETS=$(aws eks describe-cluster --name "$CLUSTER" --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.subnetIds[]' --output text) + SPOKE_PRIVATE_SUBNETS=$(aws ec2 describe-subnets --subnet-ids $SPOKE_ALL_SUBNETS --region {{.AWS_REGION}} --query 'Subnets[?MapPublicIpOnLaunch==`false`].SubnetId' --output text | tr '\t' '\n' | awk '!seen[$0]++' | head -2 | paste -sd ',' -) + if [ -z "$SPOKE_PRIVATE_SUBNETS" ]; then + SPOKE_PRIVATE_SUBNETS=$(echo "$SPOKE_ALL_SUBNETS" | tr '\t' '\n' | head -2 | paste -sd ',' -) + fi + SPOKE_SG=$(aws eks describe-cluster --name "$CLUSTER" --region {{.AWS_REGION}} --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text) + echo "Spoke $CLUSTER: arn=$SPOKE_ARN subnets=$SPOKE_PRIVATE_SUBNETS sg=$SPOKE_SG" + if [ "$CLUSTER" = "spoke-dev" ]; then + SPOKE_DEV_ARN="$SPOKE_ARN" + SPOKE_DEV_SUBNETS="$SPOKE_PRIVATE_SUBNETS" + SPOKE_DEV_SG="$SPOKE_SG" + else + SPOKE_PROD_ARN="$SPOKE_ARN" + SPOKE_PROD_SUBNETS="$SPOKE_PRIVATE_SUBNETS" + SPOKE_PROD_SG="$SPOKE_SG" + fi + else + echo "WARN: Cluster $CLUSTER not found in {{.AWS_REGION}}. Scraper skipped." + fi + done + if [ -n "$SPOKE_DEV_ARN" ] || [ -n "$SPOKE_PROD_ARN" ]; then + CONFIG_UPDATED=$(echo "$CONFIG_EXISTING" | jq \ + --arg sda "$SPOKE_DEV_ARN" \ + --arg sds "$SPOKE_DEV_SUBNETS" \ + --arg sdsg "$SPOKE_DEV_SG" \ + --arg spa "$SPOKE_PROD_ARN" \ + --arg sps "$SPOKE_PROD_SUBNETS" \ + --arg spsg "$SPOKE_PROD_SG" \ + '.metadata = (.metadata | fromjson | . + {spoke_dev_cluster_arn: $sda, spoke_dev_private_subnet_ids: $sds, spoke_dev_security_group_id: $sdsg, spoke_prod_cluster_arn: $spa, spoke_prod_private_subnet_ids: $sps, spoke_prod_security_group_id: $spsg} | tojson)') + aws secretsmanager put-secret-value \ + --secret-id "$CONFIG_KEY" \ + --secret-string "$CONFIG_UPDATED" \ + --region {{.AWS_REGION}} + echo "Updated hub config with spoke scraper data" + fi + + # Update Keycloak SAML client with AMG URL + printf '{{.C_STEP}}▸ Waiting for Keycloak to be reachable...{{.C_RESET}}\n' + for i in $(seq 1 30); do + if curl -sf -o /dev/null "https://{{.KC_DOMAIN}}/keycloak/realms/master/.well-known/openid-configuration" 2>/dev/null; then + echo "Keycloak is reachable" + break + fi + echo " [$i/30] Keycloak not reachable yet (DNS/ALB propagating)..." + sleep 20 + done + printf '{{.C_STEP}}▸ Updating Keycloak SAML client with AMG URL...{{.C_RESET}}\n' + # Best-effort, trailing block: relax errexit so a Keycloak/AMG hiccup + # (missing perms, duplicate SAML clients yielding a multi-line client id -> + # malformed curl URL, transient API error) only WARNs instead of aborting + # the entire install. This is the last block in the task, so set +e is safe. + set +e + KC_PASS=$(aws secretsmanager get-secret-value --secret-id {{.HUB_CLUSTER_NAME}}/keycloak --region {{.AWS_REGION}} --query 'SecretString' --output text | jq -r '.keycloak_admin_password') + KC_TOKEN=$(curl -s -X POST "https://{{.KC_DOMAIN}}/keycloak/realms/master/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=password&client_id=admin-cli&username=admin&password=$KC_PASS" | jq -r '.access_token') + if [ -n "$KC_TOKEN" ] && [ "$KC_TOKEN" != "null" ]; then + KC_CLIENT_ID=$(curl -s "https://{{.KC_DOMAIN}}/keycloak/admin/realms/platform/clients" \ + -H "Authorization: Bearer $KC_TOKEN" | jq -r '.[] | select(.name=="amazon-managed-grafana" and .protocol=="saml") | .id') + if [ -n "$KC_CLIENT_ID" ]; then + KC_CLIENT=$(curl -s "https://{{.KC_DOMAIN}}/keycloak/admin/realms/platform/clients/$KC_CLIENT_ID" \ + -H "Authorization: Bearer $KC_TOKEN") + KC_UPDATED=$(echo "$KC_CLIENT" | jq \ + --arg url "$AMG_ENDPOINT" \ + '.clientId = "https://\($url)/saml/metadata" | .adminUrl = "https://\($url)/login/saml" | .redirectUris = ["https://\($url)/saml/acs"]') + HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X PUT "https://{{.KC_DOMAIN}}/keycloak/admin/realms/platform/clients/$KC_CLIENT_ID" \ + -H "Authorization: Bearer $KC_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$KC_UPDATED") + if [ "$HTTP_CODE" = "204" ]; then + echo "Keycloak SAML client updated for AMG: $AMG_ENDPOINT" + else + echo "WARN: Keycloak SAML client update returned HTTP $HTTP_CODE" + fi + else + echo "WARN: Keycloak SAML client 'amazon-managed-grafana' not found" + fi + else + echo "WARN: Could not get Keycloak admin token. SAML client not updated." + fi + - | + # Nudge Grafana to converge now that grafana_api_key/AMG creds are seeded. + # The grafana-admin-credentials ExternalSecret -> Secret -> external-grafana + # instance chain only reconciles once these creds exist; and the + # grafana-operator caches its "available instances" registry, so + # GrafanaDashboards created before the instance was Ready keep reporting + # NoMatchingInstances until the operator re-evaluates. Force the + # ExternalSecrets to resync and restart the operator so dashboards attach + # and the grafana-dashboards Argo app reaches Healthy without manual steps. + # Best-effort: never fail the seed on this. + for es in grafana-admin-credentials grafana-amp-creds grafana-mysql-creds; do + kubectl annotate externalsecret "$es" -n grafana-operator force-sync="$(date +%s)" --overwrite 2>/dev/null || true + done + kubectl rollout restart deploy/grafana-operator -n grafana-operator 2>/dev/null || true + echo "Grafana ExternalSecrets resynced and grafana-operator restarted to pick up seeded credentials." + + + + hub:create-mgmt-roles: + desc: Create cluster-mgmt IAM roles assumed by ACK controllers (single-account prereq) + vars: + ACK_CAPABILITY_ROLE: + sh: aws eks describe-capability --cluster-name {{.HUB_CLUSTER_NAME}} --capability-name ack --region {{.AWS_REGION}} --query 'capability.roleArn' --output text 2>/dev/null | sed 's|.*/||' || echo "{{.HUB_CLUSTER_NAME}}-argocd-capability-role" + cmds: + - printf '{{.C_STEP}}▸ Creating cluster-mgmt IAM roles for ACK...{{.C_RESET}}\n' + - | + TRUST_POLICY=$(cat </dev/null 2>&1; then + aws iam update-assume-role-policy --role-name "$ROLE_NAME" --policy-document "$TRUST_POLICY" + else + aws iam create-role --role-name "$ROLE_NAME" --assume-role-policy-document "$TRUST_POLICY" --no-cli-pager >/dev/null + fi + for arn in $(get_policies $service); do + aws iam attach-role-policy --role-name "$ROLE_NAME" --policy-arn "$arn" 2>/dev/null || true + done + echo " ✓ $ROLE_NAME" + done + # Inline policies for EKS role: AccessEntry, PodIdentity, PassRole + EKS_ROLE="{{.RESOURCE_PREFIX}}-cluster-mgmt-eks" + aws iam put-role-policy --role-name "$EKS_ROLE" --policy-name eks-access-management --policy-document '{ + "Version": "2012-10-17", + "Statement": [ + {"Effect": "Allow", "Action": ["eks:*"], "Resource": "*"}, + {"Effect": "Allow", "Action": ["iam:PassRole","iam:GetRole"], "Resource": ["arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/{{.RESOURCE_PREFIX}}-*","arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/*-microservice-*"]} + ] + }' + # Allow ACK capability role to assume cluster-mgmt roles + aws iam put-role-policy --role-name "{{.ACK_CAPABILITY_ROLE}}" --policy-name ack-assume-role --policy-document '{ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": ["sts:AssumeRole", "sts:TagSession"], + "Resource": "arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/{{.RESOURCE_PREFIX}}-cluster-mgmt-*" + }] + }' + - printf '{{.C_OK}}✓ cluster-mgmt roles ready.{{.C_RESET}}\n' + + + hub:restart-identity-pods: + desc: Restart pods that depend on Pod Identity (external-dns, LBC, Crossplane providers) after identities are ready + status: + - | + KC={{.ROOT_DIR}}/private/hub-kubeconfig + # Skip if all providers are Healthy and have been running > 5min (pod identity already active) + UNHEALTHY=$(KUBECONFIG=$KC kubectl get providers.pkg.crossplane.io -o jsonpath='{.items[?(@.status.conditions[?(@.type=="Healthy")].status!="True")].metadata.name}' -n crossplane-system 2>/dev/null | wc -w) + [ "$UNHEALTHY" = "0" ] && OLDEST=$(KUBECONFIG=$KC kubectl get pods -n crossplane-system -l pkg.crossplane.io/revision --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[0].metadata.creationTimestamp}' 2>/dev/null) && [ -n "$OLDEST" ] && AGE=$(( $(date +%s) - $(date -d "$OLDEST" +%s 2>/dev/null || date -j -f "%Y-%m-%dT%H:%M:%SZ" "$OLDEST" +%s 2>/dev/null) )) && [ "$AGE" -gt 300 ] + cmds: + - task: hub:kubeconfig + - printf '{{.C_STEP}}▸ Terminating any stuck ArgoCD sync operations on pod-identities apps...{{.C_RESET}}\n' + - | + # A stuck PreSync hook on pod-identities-* (e.g. crd-wait Job whose + # resources were already deleted) leaves operationState.phase=Running + # indefinitely. This prevents PodIdentityAssociations from ever being + # created, which in turn means LBC/external-dns never get AWS creds. + # Terminate any such stuck operations before waiting. + KC="KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig" + eval "$KC" kubectl get applications -n argocd -o json 2>/dev/null | \ + jq -r '.items[] | select( + (.metadata.name | startswith("pod-identities")) and + (.status.operationState.phase == "Running") + ) | .metadata.name' | \ + while read -r app; do + printf ' ⚠ Terminating stuck sync on %s\n' "$app" + eval "$KC" kubectl patch application "$app" -n argocd \ + --type='json' -p='[{"op":"remove","path":"/operation"}]' 2>/dev/null || true + sleep 3 + eval "$KC" kubectl patch application "$app" -n argocd --type merge \ + -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"revision":"HEAD"}}}' \ + 2>/dev/null || true + done + - printf '{{.C_STEP}}▸ Waiting for pod identity associations to be Ready...{{.C_RESET}}\n' + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl -n crossplane-system wait --for=condition=Ready podidentityassociations.eks.aws.upbound.io --all --timeout=300s 2>/dev/null || true + - printf '{{.C_STEP}}▸ Restarting Crossplane providers to pick up pod identity credentials...{{.C_RESET}}\n' + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl delete pod -n crossplane-system -l pkg.crossplane.io/revision --force --grace-period=0 2>/dev/null || true + - sleep 15 + - printf '{{.C_STEP}}▸ Restarting external-dns and LBC to pick up pod identity credentials...{{.C_RESET}}\n' + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout restart deploy -n kube-system -l app.kubernetes.io/name=external-dns 2>/dev/null || true + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout restart deploy -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller 2>/dev/null || true + - printf '{{.C_STEP}}▸ Restarting LiteLLM to pick up pod identity credentials...{{.C_RESET}}\n' + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout restart deploy -n kagent litellm 2>/dev/null || true + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout restart deploy -n otel otel-collector 2>/dev/null || true + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout status deploy -n kube-system -l app.kubernetes.io/name=external-dns --timeout=60s 2>/dev/null || true + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout status deploy -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller --timeout=60s 2>/dev/null || true + - KUBECONFIG={{.ROOT_DIR}}/private/hub-kubeconfig kubectl rollout status deploy -n kagent litellm --timeout=120s 2>/dev/null || true + - rm -f {{.ROOT_DIR}}/private/hub-kubeconfig + - printf '{{.C_OK}}✓ Identity-dependent pods restarted.{{.C_RESET}}\n' + # Wait 60s for EKS Pod Identity agent to inject credentials into new pods + # before ArgoCD syncs crossplane-base and tries to create IAM roles + - printf '{{.C_INFO}} Waiting 90s for Pod Identity credentials to be injected...{{.C_RESET}}\n' + - sleep 90 + - printf '{{.C_OK}}✓ Pod Identity credentials ready.{{.C_RESET}}\n' + + + hub:update: + desc: Update hub infra via Kind+Crossplane + vars: + MNG_SETS: + sh: | + if [ "$(yq '.hub.managedNodeGroup.enabled // false' {{.CONFIG_FILE}})" = "true" ]; then + SETS="--set clusters.hub.managedNodeGroup.enabled=true" + IDX=0 + for TYPE in $(yq '.hub.managedNodeGroup.instanceTypes // ["m5.large"] | .[]' {{.CONFIG_FILE}}); do + SETS="$SETS --set clusters.hub.managedNodeGroup.instanceTypes[$IDX]=$TYPE" + IDX=$((IDX+1)) + done + SETS="$SETS --set clusters.hub.managedNodeGroup.desiredSize=$(yq '.hub.managedNodeGroup.desiredSize // 2' {{.CONFIG_FILE}})" + SETS="$SETS --set clusters.hub.managedNodeGroup.minSize=$(yq '.hub.managedNodeGroup.minSize // 1' {{.CONFIG_FILE}})" + SETS="$SETS --set clusters.hub.managedNodeGroup.maxSize=$(yq '.hub.managedNodeGroup.maxSize // 5' {{.CONFIG_FILE}})" + SETS="$SETS --set clusters.hub.managedNodeGroup.diskSize=$(yq '.hub.managedNodeGroup.diskSize // 50' {{.CONFIG_FILE}})" + SETS="$SETS --set clusters.hub.managedNodeGroup.capacityType=$(yq '.hub.managedNodeGroup.capacityType // "ON_DEMAND"' {{.CONFIG_FILE}})" + echo "$SETS" + fi + cmds: + - task: init + - task: kind:create + - task: credentials:setup + - task: crossplane:helm + - task: crossplane:providers + - task: crossplane:provider-config + - kubectl apply -f {{.GITOPS_ROOT}}/abstractions/crossplane/platform-cluster/templates/ 2>/dev/null || true + - helm template hub-cluster {{.GITOPS_ROOT}}/abstractions/crossplane/platform-cluster + --set clusters.hub.region={{.AWS_REGION}} + --set clusters.hub.clusterName={{.HUB_CLUSTER_NAME}} + --set clusters.hub.kubernetesVersion={{.HUB_K8S_VERSION}} + --set clusters.hub.vpcCidr={{.HUB_VPC_CIDR}} + --set clusters.hub.resourcePrefix={{.RESOURCE_PREFIX}} + {{.MNG_SETS}} + | kubectl apply -f - + - kubectl apply -f claims/ + - kubectl -n crossplane-system wait --for=condition=Ready --timeout=3500s platformclusters.platform.gitops.io/{{.HUB_CLUSTER_NAME}} + - task: destroy-kind + + + spokes:seed-provider-identity: + desc: >- + Seed the bootstrap Crossplane provider IAM roles + EKS Pod Identity + associations for spoke clusters (same account). The iam/eks providers are + createIdentity:false in the registry, so crossplane-base does NOT create + their roles; without this seed a spoke's iam/eks providers cannot + authenticate (chicken-and-egg). Mirrors the hub bootstrap. Idempotent. + Run after spokes are provisioned (ACTIVE). Cross-account spokes are NOT + handled here - their roles must be bootstrapped in the target account. + cmds: + - cmd: | + # provider service-account suffix -> IAM role name suffix + PROVIDERS="iam:CrossplaneIAMProviderRole eks:CrossplaneEKSProviderRole" + # Discover spokes from fleet members, excluding the hub. + SPOKES=$(ls "{{.GITOPS_ROOT}}/fleet/members" 2>/dev/null | grep -vx "{{.HUB_CLUSTER_NAME}}" || true) + if [ -z "$SPOKES" ]; then + printf '{{.C_INFO}} No spoke members under fleet/members; nothing to seed.{{.C_RESET}}\n' + exit 0 + fi + for SPOKE in $SPOKES; do + STATUS=$(aws eks describe-cluster --name "$SPOKE" --region {{.AWS_REGION}} --query 'cluster.status' --output text 2>/dev/null || echo "NONE") + if [ "$STATUS" != "ACTIVE" ]; then + printf '{{.C_INFO}} Skipping %s (cluster status: %s){{.C_RESET}}\n' "$SPOKE" "$STATUS" + continue + fi + for ENTRY in $PROVIDERS; do + SVC="${ENTRY%%:*}" # iam | eks + ROLE_NAME="${SPOKE}-${ENTRY##*:}" + SA="provider-aws-${SVC}" + ROLE_ARN="arn:aws:iam::{{.AWS_ACCOUNT_ID}}:role/${ROLE_NAME}" + # 1. IAM role trusting EKS Pod Identity (idempotent) + if ! aws iam get-role --role-name "$ROLE_NAME" >/dev/null 2>&1; then + aws iam create-role --role-name "$ROLE_NAME" \ + --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"pods.eks.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]}' \ + --tags Key=managed-by,Value=platform-bootstrap Key=purpose,Value="${SVC}-provider" >/dev/null + printf '{{.C_OK}} ✓ %s: created role %s{{.C_RESET}}\n' "$SPOKE" "$ROLE_NAME" + else + printf '{{.C_INFO}} %s: role %s exists{{.C_RESET}}\n' "$SPOKE" "$ROLE_NAME" + fi + # 2. AdministratorAccess (attach is idempotent) + aws iam attach-role-policy --role-name "$ROLE_NAME" \ + --policy-arn arn:aws:iam::aws:policy/AdministratorAccess >/dev/null + # 3. Pod Identity association on the spoke (idempotent) + EXISTING=$(aws eks list-pod-identity-associations --cluster-name "$SPOKE" --region {{.AWS_REGION}} \ + --namespace crossplane-system --service-account "$SA" \ + --query 'associations[0].associationId' --output text 2>/dev/null || echo "None") + if [ "$EXISTING" = "None" ] || [ -z "$EXISTING" ]; then + aws eks create-pod-identity-association --cluster-name "$SPOKE" --region {{.AWS_REGION}} \ + --namespace crossplane-system --service-account "$SA" --role-arn "$ROLE_ARN" >/dev/null + printf '{{.C_OK}} ✓ %s: pod identity %s -> %s{{.C_RESET}}\n' "$SPOKE" "$SA" "$ROLE_NAME" + else + printf '{{.C_INFO}} %s: pod identity for %s exists{{.C_RESET}}\n' "$SPOKE" "$SA" + fi + done + done + printf '{{.C_OK}}✓ Spoke provider identity seeding complete.{{.C_RESET}}\n' + + diff --git a/cluster-providers/kind-kro-ack/kind.yaml b/cluster-providers/kind-kro-ack/kind.yaml new file mode 100644 index 000000000..f64b87e8e --- /dev/null +++ b/cluster-providers/kind-kro-ack/kind.yaml @@ -0,0 +1,4 @@ +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: + - role: control-plane diff --git a/cluster-providers/kind-crossplane/manifests/argocd/create-capability.yaml b/cluster-providers/kind-kro-ack/manifests/argocd/create-capability.yaml similarity index 54% rename from cluster-providers/kind-crossplane/manifests/argocd/create-capability.yaml rename to cluster-providers/kind-kro-ack/manifests/argocd/create-capability.yaml index 14ff3991d..3594bdfaf 100644 --- a/cluster-providers/kind-crossplane/manifests/argocd/create-capability.yaml +++ b/cluster-providers/kind-kro-ack/manifests/argocd/create-capability.yaml @@ -1,17 +1,15 @@ # Job: Creates EKS ArgoCD capability after cluster is ACTIVE. -# Env vars injected from argocd-capability-config ConfigMap -# (created by Taskfile from config.yaml + Crossplane outputs). +# Env vars injected from argocd-capability-config ConfigMap. apiVersion: batch/v1 kind: Job metadata: name: create-argocd-capability - namespace: crossplane-system + namespace: ack-system spec: backoffLimit: 3 ttlSecondsAfterFinished: 600 template: spec: - serviceAccountName: crossplane restartPolicy: OnFailure volumes: - name: aws-creds @@ -97,66 +95,4 @@ spec: --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \ --access-scope type=cluster 2>/dev/null || true - echo "=== Create KRO capability ===" - KRO_STATUS=$(aws eks describe-capability \ - --cluster-name "$CLUSTER_NAME" \ - --capability-name "kro" \ - --region "$AWS_REGION" \ - --query 'capability.status' \ - --output text 2>/dev/null || echo "NOT_FOUND") - if [ "$KRO_STATUS" = "NOT_FOUND" ]; then - aws eks create-capability \ - --region "$AWS_REGION" \ - --cluster-name "$CLUSTER_NAME" \ - --capability-name "kro" \ - --type KRO \ - --role-arn "$CAPABILITY_ROLE_ARN" \ - --delete-propagation-policy RETAIN - echo "KRO capability created" - else - echo "KRO capability exists: $KRO_STATUS" - fi - - echo "=== Create ACK capability ===" - ACK_STATUS=$(aws eks describe-capability \ - --cluster-name "$CLUSTER_NAME" \ - --capability-name "ack" \ - --region "$AWS_REGION" \ - --query 'capability.status' \ - --output text 2>/dev/null || echo "NOT_FOUND") - if [ "$ACK_STATUS" = "NOT_FOUND" ]; then - aws eks create-capability \ - --region "$AWS_REGION" \ - --cluster-name "$CLUSTER_NAME" \ - --capability-name "ack" \ - --type ACK \ - --role-arn "$CAPABILITY_ROLE_ARN" \ - --delete-propagation-policy RETAIN - echo "ACK capability created" - else - echo "ACK capability exists: $ACK_STATUS" - fi - - echo "=== Wait for KRO and ACK ACTIVE ===" - for CAP in kro ack; do - for i in $(seq 1 40); do - S=$(aws eks describe-capability \ - --cluster-name "$CLUSTER_NAME" \ - --capability-name "$CAP" \ - --region "$AWS_REGION" \ - --query 'capability.status' \ - --output text 2>/dev/null || echo "UNKNOWN") - [ "$S" = "ACTIVE" ] && echo "$CAP: ACTIVE" && break - echo "$CAP: $S ($i/40)" - sleep 15 - done - done - echo "=== Done ===" - echo "ARN: arn:aws:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}" - aws eks describe-capability \ - --cluster-name "$CLUSTER_NAME" \ - --capability-name "$CAPABILITY_NAME" \ - --region "$AWS_REGION" \ - --query 'capability.configuration.argoCd.serverUrl' \ - --output text diff --git a/cluster-providers/kind-crossplane/manifests/argocd/delete-capability.yaml b/cluster-providers/kind-kro-ack/manifests/argocd/delete-capability.yaml similarity index 96% rename from cluster-providers/kind-crossplane/manifests/argocd/delete-capability.yaml rename to cluster-providers/kind-kro-ack/manifests/argocd/delete-capability.yaml index 686f43c19..864e3852d 100644 --- a/cluster-providers/kind-crossplane/manifests/argocd/delete-capability.yaml +++ b/cluster-providers/kind-kro-ack/manifests/argocd/delete-capability.yaml @@ -2,13 +2,12 @@ apiVersion: batch/v1 kind: Job metadata: name: delete-argocd-capability - namespace: crossplane-system + namespace: ack-system spec: backoffLimit: 3 ttlSecondsAfterFinished: 600 template: spec: - serviceAccountName: crossplane restartPolicy: OnFailure volumes: - name: aws-creds diff --git a/cluster-providers/terraform/helm.tf b/cluster-providers/terraform/helm.tf index e0a7c62fb..0bc25377e 100644 --- a/cluster-providers/terraform/helm.tf +++ b/cluster-providers/terraform/helm.tf @@ -23,19 +23,19 @@ locals { # (createIdentity: false in registry); others get identity from ArgoCD post-bootstrap. crossplane_bootstrap_providers = { iam = { - package = "xpkg.upbound.io/upbound/provider-aws-iam:v2.5.3" + package = "xpkg.upbound.io/upbound/provider-aws-iam:v2.6.1" service_account = "provider-aws-iam" } eks = { - package = "xpkg.upbound.io/upbound/provider-aws-eks:v2.5.3" + package = "xpkg.upbound.io/upbound/provider-aws-eks:v2.6.1" service_account = "provider-aws-eks" } ec2 = { - package = "xpkg.upbound.io/upbound/provider-aws-ec2:v2.5.3" + package = "xpkg.upbound.io/upbound/provider-aws-ec2:v2.6.1" service_account = "provider-aws-ec2" } secretsmanager = { - package = "xpkg.upbound.io/upbound/provider-aws-secretsmanager:v2.5.3" + package = "xpkg.upbound.io/upbound/provider-aws-secretsmanager:v2.6.1" service_account = "provider-aws-secretsmanager" } } @@ -121,7 +121,7 @@ resource "kubectl_manifest" "crossplane_provider_family" { name = "provider-family-aws" } spec = { - package = "xpkg.upbound.io/upbound/provider-family-aws:v2.5.3" + package = "xpkg.upbound.io/upbound/provider-family-aws:v2.6.1" } }) diff --git a/config.schema.json b/config.schema.json index e3fe665fa..86a5c24c4 100644 --- a/config.schema.json +++ b/config.schema.json @@ -3,17 +3,29 @@ "title": "Platform Bootstrap Configuration", "description": "Configuration for the GitOps addon management platform. Copy to config.local.yaml and customize.", "type": "object", - "required": ["hub", "aws", "repo", "domain"], + "required": [ + "hub", + "aws", + "repo", + "domain" + ], "properties": { "clusterProvider": { "type": "string", - "enum": ["kind-crossplane", "byoc"], + "enum": [ + "kind-crossplane", + "byoc" + ], "default": "kind-crossplane", "description": "Which cluster provider to use for hub provisioning" }, "repo": { "type": "object", - "required": ["url", "revision", "basepath"], + "required": [ + "url", + "revision", + "basepath" + ], "properties": { "url": { "type": "string", @@ -34,7 +46,9 @@ }, "hub": { "type": "object", - "required": ["clusterName"], + "required": [ + "clusterName" + ], "properties": { "clusterName": { "type": "string", @@ -63,8 +77,12 @@ }, "instanceTypes": { "type": "array", - "items": { "type": "string" }, - "default": ["m5.large"], + "items": { + "type": "string" + }, + "default": [ + "m5.large" + ], "description": "EC2 instance types for the node group" }, "desiredSize": { @@ -90,7 +108,10 @@ }, "capacityType": { "type": "string", - "enum": ["ON_DEMAND", "SPOT"], + "enum": [ + "ON_DEMAND", + "SPOT" + ], "default": "ON_DEMAND" } } @@ -99,7 +120,10 @@ }, "aws": { "type": "object", - "required": ["region", "accountId"], + "required": [ + "region", + "accountId" + ], "properties": { "region": { "type": "string", @@ -121,7 +145,17 @@ "domain": { "type": "string", "pattern": "^[a-z0-9][a-z0-9.-]+[a-z0-9]$", - "description": "Base domain for ingress hostnames. Must be unique per stack — two stacks sharing a domain will have DNS and TLS conflicts." + "description": "Ingress hostname for the platform, provisioned by the CONSUMER (Route53, CloudFront, etc.). The platform never creates it. Used for ingress host rules and printed URLs. Must be unique per stack." + }, + "insecure": { + "type": "boolean", + "default": false, + "description": "When true, platform ingress serves plain HTTP (no cert) — the consumer terminates TLS upstream (e.g. CloudFront). When false (default), the ALB serves HTTPS and uses an ACM cert (auto-discovered by host, or certificateArn if set)." + }, + "certificateArn": { + "type": "string", + "default": "", + "description": "Optional ACM certificate ARN for the ALB HTTPS listener when insecure=false. If empty, the AWS Load Balancer Controller auto-discovers an ACM cert matching the domain." }, "resourcePrefix": { "type": "string", @@ -138,9 +172,17 @@ "default": "", "description": "Comma-separated security group IDs for the ALB" }, + "adminRoleName": { + "type": "string", + "description": "IAM role name added as a cluster-admin access entry and used by Backstage templates for cluster access." + }, "identityCenter": { "type": "object", - "required": ["instanceArn", "region", "adminGroupId"], + "required": [ + "instanceArn", + "region", + "adminGroupId" + ], "description": "AWS Identity Center config for EKS ArgoCD Capability", "properties": { "instanceArn": { diff --git a/config.yaml b/config.yaml index 097435741..2d86d24d9 100644 --- a/config.yaml +++ b/config.yaml @@ -35,7 +35,17 @@ aws: profile: "default" # AWS CLI profile (local dev only) # Domain and networking -domain: "" # e.g. idp.example.com — leave empty for CloudFront mode (auto-created) +# `domain` is the ingress hostname, provisioned by YOU (Route53, CloudFront, etc.). +# The platform never creates it — it only uses it for ingress host rules and URLs. +domain: "" # e.g. idp.example.com + +# TLS / exposure: +# insecure: false (default) — ALB serves HTTPS; uses an ACM cert (auto-discovered +# by host, or set certificateArn). You bring the domain + cert. +# insecure: true — ALB serves plain HTTP; YOU terminate TLS upstream +# (e.g. CloudFront in front of the ALB). No cert managed by the platform. +insecure: false +certificateArn: "" # optional ACM ARN for the HTTPS listener (insecure=false) resourcePrefix: "" # Used for KRO tenant-facing resources only ingressName: "" # Defaults to {clusterName}-ingress if empty ingressSecurityGroups: "" diff --git a/gitops/abstractions/crossplane/platform-cluster/templates/claim.yaml b/gitops/abstractions/crossplane/platform-cluster/templates/claim.yaml index a9a7d388d..fb6db3a5c 100644 --- a/gitops/abstractions/crossplane/platform-cluster/templates/claim.yaml +++ b/gitops/abstractions/crossplane/platform-cluster/templates/claim.yaml @@ -33,6 +33,15 @@ spec: {{- else if $.Values.global.adminInstanceRoleArn }} adminInstanceRoleArn: {{ $.Values.global.adminInstanceRoleArn }} {{- end }} + {{- if $cluster.accountId }} + accountId: {{ $cluster.accountId | quote }} + {{- else if $.Values.global.accountId }} + accountId: {{ $.Values.global.accountId | quote }} + {{- end }} + {{- if $cluster.capabilities }} + capabilities: + {{- toYaml $cluster.capabilities | nindent 4 }} + {{- end }} {{- if and $cluster.managedNodeGroup $cluster.managedNodeGroup.enabled }} managedNodeGroup: enabled: true diff --git a/gitops/abstractions/crossplane/platform-cluster/templates/composition.yaml b/gitops/abstractions/crossplane/platform-cluster/templates/composition.yaml index 32d3eb60f..129105166 100644 --- a/gitops/abstractions/crossplane/platform-cluster/templates/composition.yaml +++ b/gitops/abstractions/crossplane/platform-cluster/templates/composition.yaml @@ -847,16 +847,228 @@ spec: fromFieldPath: spec.argoCDRoleArn toFieldPath: spec.forProvider.principalArn - # NOTE: KRO + ACK EKS Capabilities for Crossplane-provisioned spokes are - # created out-of-band by the IDE Taskfile task `spokes:create-capabilities` - # (install phase2-spoke-*), which authenticates with the IDE instance role - # and provisions per-spoke capability IAM roles - # (--kro/ack-capability-role). A previous in-composition - # `eks-capabilities-job` (an aws-cli Job on the `crossplane` SA) was removed: - # it assumed the crossplane SA had a Pod Identity association - # (CrossplaneCapabilitiesRole) that is never created on the hub, so it always - # failed with `NoCredentials` — a redundant, noisy second path while the - # Taskfile path already brings both capabilities to ACTIVE. + # ===== EKS CAPABILITIES (native provider-aws-eks Capability MRs) ===== + # Replaces the imperative spokes:create-capabilities task (Crossplane spokes) + # and create-capability.yaml Job (hub). Each capability + its IAM role is + # gated by spec.capabilities..enabled via the function-cel-filter step. + # Roles mirror the kro-ack RGD (rg-eks.yaml) exactly: capabilities.eks.amazonaws.com + # trust; KRO=AmazonEKSClusterPolicy; ACK=inline AssumeWorkloadRoles+ManageIRSARoles. + # The provider (AdministratorAccess pod identity) creates these once the Cluster + # is ACTIVE — no foreground wait, fully declarative/async. + + # --- KRO capability role (managed AmazonEKSClusterPolicy) --- + - name: kro-capability-role + base: + apiVersion: iam.aws.upbound.io/v1beta1 + kind: Role + metadata: + labels: + capability: kro + spec: + forProvider: + assumeRolePolicy: | + { + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": {"Service": "capabilities.eks.amazonaws.com"}, + "Action": ["sts:AssumeRole", "sts:TagSession"] + }] + } + patches: + - type: FromCompositeFieldPath + fromFieldPath: spec.clusterName + toFieldPath: metadata.annotations[crossplane.io/external-name] + transforms: + - type: string + string: + type: Format + fmt: "%s-kro-capability-role" + - type: PatchSet + patchSetName: tags + + - name: kro-capability-policy + base: + apiVersion: iam.aws.upbound.io/v1beta1 + kind: RolePolicyAttachment + spec: + forProvider: + policyArn: "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy" + roleSelector: + matchControllerRef: true + matchLabels: + capability: kro + + - name: kro-capability + base: + apiVersion: eks.aws.upbound.io/v1beta1 + kind: Capability + spec: + forProvider: + capabilityName: kro + type: KRO + deletePropagationPolicy: RETAIN + clusterNameSelector: + matchControllerRef: true + roleArnSelector: + matchControllerRef: true + matchLabels: + capability: kro + patches: + - type: PatchSet + patchSetName: common + + # --- ACK capability role (inline AssumeWorkloadRoles + ManageIRSARoles) --- + - name: ack-capability-role + base: + apiVersion: iam.aws.upbound.io/v1beta1 + kind: Role + metadata: + labels: + capability: ack + spec: + forProvider: + assumeRolePolicy: | + { + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": {"Service": "capabilities.eks.amazonaws.com"}, + "Action": ["sts:AssumeRole", "sts:TagSession"] + }] + } + patches: + - type: FromCompositeFieldPath + fromFieldPath: spec.clusterName + toFieldPath: metadata.annotations[crossplane.io/external-name] + transforms: + - type: string + string: + type: Format + fmt: "%s-ack-capability-role" + - type: PatchSet + patchSetName: tags + + # inline AssumeWorkloadRoles: sts:AssumeRole/TagSession on -cluster-mgmt-* + - name: ack-capability-policy-assume + base: + apiVersion: iam.aws.upbound.io/v1beta1 + kind: RolePolicy + metadata: + labels: + capability: ack + spec: + forProvider: + roleSelector: + matchControllerRef: true + matchLabels: + capability: ack + patches: + - type: FromCompositeFieldPath + fromFieldPath: spec.clusterName + toFieldPath: metadata.annotations[crossplane.io/external-name] + transforms: + - type: string + string: + type: Format + fmt: "%s-ack-AssumeWorkloadRoles" + - type: CombineFromComposite + combine: + variables: + - fromFieldPath: spec.accountId + - fromFieldPath: spec.resourcePrefix + strategy: string + string: + fmt: '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["sts:AssumeRole","sts:TagSession"],"Resource":["arn:aws:iam::%s:role/%s-cluster-mgmt-*"]}]}' + toFieldPath: spec.forProvider.policy + + # inline ManageIRSARoles: IRSA verbs on -* + - name: ack-capability-policy-irsa + base: + apiVersion: iam.aws.upbound.io/v1beta1 + kind: RolePolicy + metadata: + labels: + capability: ack + spec: + forProvider: + roleSelector: + matchControllerRef: true + matchLabels: + capability: ack + patches: + - type: FromCompositeFieldPath + fromFieldPath: spec.clusterName + toFieldPath: metadata.annotations[crossplane.io/external-name] + transforms: + - type: string + string: + type: Format + fmt: "%s-ack-ManageIRSARoles" + - type: CombineFromComposite + combine: + variables: + - fromFieldPath: spec.accountId + - fromFieldPath: spec.clusterName + strategy: string + string: + fmt: '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["iam:GetRole","iam:CreateRole","iam:DeleteRole","iam:TagRole","iam:UntagRole","iam:UpdateRole","iam:UpdateAssumeRolePolicy","iam:AttachRolePolicy","iam:DetachRolePolicy","iam:ListAttachedRolePolicies","iam:ListRolePolicies","iam:ListRoleTags","iam:ListInstanceProfilesForRole","iam:PutRolePolicy","iam:DeleteRolePolicy","iam:GetRolePolicy"],"Resource":["arn:aws:iam::%s:role/%s-*"]}]}' + toFieldPath: spec.forProvider.policy + + - name: ack-capability + base: + apiVersion: eks.aws.upbound.io/v1beta1 + kind: Capability + spec: + forProvider: + capabilityName: ack + type: ACK + deletePropagationPolicy: RETAIN + clusterNameSelector: + matchControllerRef: true + roleArnSelector: + matchControllerRef: true + matchLabels: + capability: ack + patches: + - type: PatchSet + patchSetName: common + + # --- ArgoCD capability (hub only) — reuses the existing argocd-capability-role + # (spec.argoCDRoleArn); needs IDC config for SSO ADMIN RBAC mapping. --- + - name: argocd-capability + base: + apiVersion: eks.aws.upbound.io/v1beta1 + kind: Capability + spec: + forProvider: + capabilityName: argocd + type: ARGOCD + deletePropagationPolicy: RETAIN + clusterNameSelector: + matchControllerRef: true + configuration: + argoCd: + awsIdc: {} + rbacRoleMapping: + - role: ADMIN + identity: + - type: SSO_GROUP + patches: + - type: PatchSet + patchSetName: common + - type: FromCompositeFieldPath + fromFieldPath: spec.argoCDRoleArn + toFieldPath: spec.forProvider.roleArn + - type: FromCompositeFieldPath + fromFieldPath: spec.capabilities.argocd.idcInstanceArn + toFieldPath: spec.forProvider.configuration.argoCd.awsIdc.idcInstanceArn + - type: FromCompositeFieldPath + fromFieldPath: spec.capabilities.argocd.idcRegion + toFieldPath: spec.forProvider.configuration.argoCd.awsIdc.idcRegion + - type: FromCompositeFieldPath + fromFieldPath: spec.capabilities.argocd.adminGroupId + toFieldPath: spec.forProvider.configuration.argoCd.rbacRoleMapping[0].identity[0].id # EKS Cluster with Auto Mode enabled - name: eks-cluster @@ -945,7 +1157,12 @@ spec: # Uses provider-kubernetes Object with Observe+Update so the Secret must # already exist (created by the fleet-secret Helm chart) — the Object only # patches the single annotation via Server-Side Apply. + # readinessCheck: none — the argocd namespace is created by the EKS ArgoCD + # Capability *after* the XR reaches Ready. Blocking XR readiness on this + # Object creates a circular dependency that stalls the install for 90+ min. - name: cluster-secret-vpc-annotation + readinessChecks: + - type: None base: apiVersion: kubernetes.crossplane.io/v1alpha2 kind: Object @@ -1154,4 +1371,22 @@ spec: expression: >- has(observed.composite.resource.spec.adminInstanceRoleArn) && observed.composite.resource.spec.adminInstanceRoleArn != "" + # EKS Capabilities — only compose each capability (+ its IAM role/policies) + # when spec.capabilities..enabled is true. Hub gets kro+ack+argocd; + # Crossplane spokes get kro+ack; unset/false => not composed. + - name: "^kro-capability(-role|-policy)?$" + expression: >- + has(observed.composite.resource.spec.capabilities) && + has(observed.composite.resource.spec.capabilities.kro) && + observed.composite.resource.spec.capabilities.kro.enabled == true + - name: "^ack-capability(-role|-policy-assume|-policy-irsa)?$" + expression: >- + has(observed.composite.resource.spec.capabilities) && + has(observed.composite.resource.spec.capabilities.ack) && + observed.composite.resource.spec.capabilities.ack.enabled == true + - name: "^argocd-capability$" + expression: >- + has(observed.composite.resource.spec.capabilities) && + has(observed.composite.resource.spec.capabilities.argocd) && + observed.composite.resource.spec.capabilities.argocd.enabled == true {{- end }} diff --git a/gitops/abstractions/crossplane/platform-cluster/templates/xrd.yaml b/gitops/abstractions/crossplane/platform-cluster/templates/xrd.yaml index d834dbb81..c0f4c8f66 100644 --- a/gitops/abstractions/crossplane/platform-cluster/templates/xrd.yaml +++ b/gitops/abstractions/crossplane/platform-cluster/templates/xrd.yaml @@ -48,6 +48,44 @@ spec: resourcePrefix: type: string description: Prefix for resource tagging and identification + accountId: + type: string + description: AWS account ID (used to build capability-role inline policy ARNs, e.g. -cluster-mgmt-*) + capabilities: + type: object + description: | + Native EKS Capabilities (KRO / ACK / ArgoCD) created for this cluster via + provider-aws-eks Capability MRs. Replaces the imperative + `spokes:create-capabilities` / create-capability.yaml Job path. + properties: + kro: + type: object + properties: + enabled: + type: boolean + default: false + ack: + type: object + properties: + enabled: + type: boolean + default: false + argocd: + type: object + description: ArgoCD capability (hub only). Needs IDC config for SSO RBAC. + properties: + enabled: + type: boolean + default: false + idcInstanceArn: + type: string + description: IAM Identity Center instance ARN + idcRegion: + type: string + description: IAM Identity Center region + adminGroupId: + type: string + description: IDC group id mapped to the ArgoCD ADMIN role argoCDRoleArn: type: string description: ARN of the hub ArgoCD capability role for spoke access diff --git a/gitops/abstractions/crossplane/platform-cluster/values.yaml b/gitops/abstractions/crossplane/platform-cluster/values.yaml index 8d2026105..51cf9c770 100644 --- a/gitops/abstractions/crossplane/platform-cluster/values.yaml +++ b/gitops/abstractions/crossplane/platform-cluster/values.yaml @@ -4,6 +4,9 @@ global: argoCDRoleArn: "" + # AWS account ID, used to build ACK capability-role inline-policy ARNs + # (-cluster-mgmt-* and -*). Per-cluster accountId takes precedence. + accountId: "" # Default AWS region for clusters that don't set their own. # Injected by the bootstrap ApplicationSet from the hub cluster's # aws_region annotation. Per-cluster `region` in diff --git a/gitops/abstractions/kro/kro-clusters/templates/clusters.yaml b/gitops/abstractions/kro/kro-clusters/templates/clusters.yaml index 73acbc3eb..bd09ac921 100644 --- a/gitops/abstractions/kro/kro-clusters/templates/clusters.yaml +++ b/gitops/abstractions/kro/kro-clusters/templates/clusters.yaml @@ -13,6 +13,7 @@ spec: name: {{ $name }} tenant: {{ $cluster.tenant | default "tenant1" | quote }} environment: {{ $cluster.environment | default "staging" | quote }} + provider: {{ $cluster.provider | default "" | quote }} region: {{ $cluster.region | default "us-west-2" | quote }} k8sVersion: {{ $cluster.k8sVersion | default "1.34" | quote }} accountId: {{ $cluster.accountId | quote }} diff --git a/gitops/abstractions/resource-groups-kro/platform-cluster-kro/Chart.yaml b/gitops/abstractions/resource-groups-kro/platform-cluster-kro/Chart.yaml new file mode 100644 index 000000000..e215ec9a0 --- /dev/null +++ b/gitops/abstractions/resource-groups-kro/platform-cluster-kro/Chart.yaml @@ -0,0 +1,5 @@ +apiVersion: v2 +name: platform-cluster-kro +description: KRO/ACK-based EKS cluster provisioning (alternative to Crossplane platform-cluster) +type: application +version: 0.1.0 diff --git a/gitops/abstractions/resource-groups-kro/platform-cluster-kro/templates/claim.yaml b/gitops/abstractions/resource-groups-kro/platform-cluster-kro/templates/claim.yaml new file mode 100644 index 000000000..0640bde52 --- /dev/null +++ b/gitops/abstractions/resource-groups-kro/platform-cluster-kro/templates/claim.yaml @@ -0,0 +1,48 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + name: {{ .Values.clusterName }} +--- +apiVersion: kro.run/v1alpha1 +kind: EksclusterWithVpc +metadata: + name: {{ .Values.clusterName }} + namespace: {{ .Values.clusterName }} +spec: + name: {{ .Values.clusterName }} + tenant: {{ .Values.tenant | default "workshop" }} + environment: {{ .Values.environment | default "dev" }} + provider: kro-ack + region: {{ .Values.region | default $.Values.global.region | default "us-west-2" }} + k8sVersion: {{ .Values.kubernetesVersion | default "1.32" | quote }} + accountId: {{ .Values.accountId | default $.Values.global.accountId | quote }} + managementAccountId: {{ .Values.managementAccountId | default $.Values.global.accountId | quote }} + argoCdHubRoleArn: {{ .Values.argoCdHubRoleArn | default $.Values.global.argoCDRoleArn | default "" }} + adminRoleName: {{ .Values.adminRoleName | default $.Values.global.adminRoleName | default "Admin" }} + fleetSecretManagerSecretNameSuffix: {{ .Values.fleetSecretManagerSecretNameSuffix | default "argocd-secret" }} + domainName: {{ .Values.domainName | default $.Values.global.domainName | default "example.com" }} + resourcePrefix: {{ .Values.resourcePrefix | default $.Values.global.resourcePrefix | default "peeks" }} + cidr: + vpcCidr: {{ .Values.vpcCidr | default "10.0.0.0/16" | quote }} + publicSubnet1Cidr: {{ .Values.publicSubnet1Cidr | default "10.0.1.0/24" | quote }} + publicSubnet2Cidr: {{ .Values.publicSubnet2Cidr | default "10.0.2.0/24" | quote }} + privateSubnet1Cidr: {{ .Values.privateSubnet1Cidr | default "10.0.11.0/24" | quote }} + privateSubnet2Cidr: {{ .Values.privateSubnet2Cidr | default "10.0.12.0/24" | quote }} + gitops: + addonsRepoUrl: {{ .Values.addonsRepoUrl | default $.Values.global.repoUrl }} + addonsRepoRevision: {{ .Values.addonsRepoRevision | default $.Values.global.repoRevision }} + addonsRepoBasePath: {{ .Values.addonsRepoBasePath | default $.Values.global.repoBasePath | default "gitops/addons/" }} + addonsRepoPath: bootstrap + fleetRepoUrl: {{ .Values.fleetRepoUrl | default $.Values.global.repoUrl }} + fleetRepoRevision: {{ .Values.fleetRepoRevision | default $.Values.global.repoRevision }} + fleetRepoBasePath: {{ .Values.fleetRepoBasePath | default $.Values.global.repoBasePath | default "gitops/fleet/" }} + fleetRepoPath: bootstrap + platformRepoUrl: {{ .Values.platformRepoUrl | default $.Values.global.repoUrl }} + platformRepoRevision: {{ .Values.platformRepoRevision | default $.Values.global.repoRevision }} + platformRepoBasePath: {{ .Values.platformRepoBasePath | default $.Values.global.repoBasePath | default "gitops/platform/" }} + platformRepoPath: bootstrap + workloadRepoUrl: {{ .Values.workloadRepoUrl | default $.Values.global.repoUrl }} + workloadRepoRevision: {{ .Values.workloadRepoRevision | default $.Values.global.repoRevision }} + workloadRepoBasePath: {{ .Values.workloadRepoBasePath | default $.Values.global.repoBasePath | default "gitops/apps/" }} + workloadRepoPath: "" diff --git a/gitops/addons/charts/argo-workflows/templates/ingress.yaml b/gitops/addons/charts/argo-workflows/templates/ingress.yaml index 3f91a9f11..519825ce7 100644 --- a/gitops/addons/charts/argo-workflows/templates/ingress.yaml +++ b/gitops/addons/charts/argo-workflows/templates/ingress.yaml @@ -1,3 +1,6 @@ +{{- /* insecure: HTTP-only ingress, no cert. Explicit .Values.insecure wins; + falls back to the legacy exposure_mode == "cloudfront" during transition. */ -}} +{{- $insecure := or .Values.insecure (eq (default "domain" .Values.exposure_mode) "cloudfront") -}} apiVersion: networking.k8s.io/v1 kind: Ingress metadata: @@ -5,7 +8,7 @@ metadata: namespace: argo annotations: alb.ingress.kubernetes.io/target-type: ip -{{- if eq (default "domain" .Values.exposure_mode) "cloudfront" }} +{{- if $insecure }} alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]' {{- else }} alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' @@ -16,7 +19,7 @@ metadata: spec: ingressClassName: platform rules: -{{- if eq (default "domain" .Values.exposure_mode) "cloudfront" }} +{{- if $insecure }} - http: {{- else }} - host: {{ .Values.ingress_domain_name }} diff --git a/gitops/addons/charts/argo-workflows/templates/install.yaml b/gitops/addons/charts/argo-workflows/templates/install.yaml index 10e4119ee..6bc9658ea 100644 --- a/gitops/addons/charts/argo-workflows/templates/install.yaml +++ b/gitops/addons/charts/argo-workflows/templates/install.yaml @@ -3050,15 +3050,15 @@ metadata: data: config: |- sso: - # insecureSkipVerify: true - issuer: https://{{ .Values.ingress_domain_name }}/keycloak/realms/platform + insecureSkipVerify: true + issuer: https://{{ .Values.ingress_domain_name | default .Values.gitlab_domain_name }}/keycloak/realms/platform clientId: name: keycloak-oidc key: client-id clientSecret: name: keycloak-oidc key: secret-key - redirectUrl: https://{{ .Values.ingress_domain_name }}/argo-workflows/oauth2/callback + redirectUrl: https://{{ .Values.ingress_domain_name | default .Values.gitlab_domain_name }}/argo-workflows/oauth2/callback rbac: enabled: true scopes: @@ -3109,7 +3109,9 @@ spec: - server - --configmap=workflow-controller-configmap - --auth-mode=client + {{- if ne (default "domain" .Values.exposure_mode) "cloudfront" }} - --auth-mode=sso + {{- end }} - "--base-href=/argo-workflows/" - "--secure=false" - "--loglevel" diff --git a/gitops/addons/charts/argocd-gitlab-repo-creds/Chart.yaml b/gitops/addons/charts/argocd-gitlab-repo-creds/Chart.yaml new file mode 100644 index 000000000..f96ce0bfa --- /dev/null +++ b/gitops/addons/charts/argocd-gitlab-repo-creds/Chart.yaml @@ -0,0 +1,12 @@ +apiVersion: v2 +name: argocd-gitlab-repo-creds +description: >- + ExternalSecrets that keep ArgoCD's GitLab repository / repo-creds credentials + (fleet-config-repo, gitlab-user-repo-creds) in sync with the canonical + peeks-hub/secrets.git_token PAT. Fixes the stale-PAT problem where the + imperatively-seeded ArgoCD repo secrets never refreshed after the PAT rotated + (#7). Deployed from the GitHub defaults path (public, no GitLab creds needed) + so ArgoCD can reconcile it without a chicken-and-egg on the very credentials + it provisions. +type: application +version: 0.1.0 diff --git a/gitops/addons/charts/argocd-gitlab-repo-creds/templates/externalsecrets.yaml b/gitops/addons/charts/argocd-gitlab-repo-creds/templates/externalsecrets.yaml new file mode 100644 index 000000000..723872e0c --- /dev/null +++ b/gitops/addons/charts/argocd-gitlab-repo-creds/templates/externalsecrets.yaml @@ -0,0 +1,94 @@ +{{- /* + Two ArgoCD secrets, sourced from Secrets Manager (peeks-hub/secrets.git_token) + via ESO so they always track the canonical PAT and self-heal when it rotates: + + 1. fleet-config-repo (secret-type: repository) — the private overlay repo + 2. gitlab-user-repo-creds (secret-type: repo-creds) — credential TEMPLATE that + matches every repo under https:/// (Backstage-created app repos) + + The imperative `secrets-manager:seed` step still creates these for cold-start; + ESO adopts (creationPolicy: Owner) and keeps them fresh thereafter. + + NOTE: this chart is deployed from the GitHub *defaults* path (public), NOT the + GitLab overlay — otherwise ArgoCD would need these very credentials to pull the + chart that provisions them. +*/}} +{{- $ns := .Values.namespace | default "argocd" }} +{{- $store := .Values.secretStoreName | default "aws-secrets-manager" }} +{{- $key := printf "%s/secrets" .Values.aws_cluster_name }} +{{- $gitlab := .Values.gitlab_domain_name }} +{{- $user := .Values.git_username | default "user1" }} +--- +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: fleet-config-repo + namespace: {{ $ns }} + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + secretStoreRef: + name: {{ $store }} + kind: ClusterSecretStore + refreshInterval: "5m" + target: + name: fleet-config-repo + creationPolicy: Owner + deletionPolicy: Retain + template: + engineVersion: v2 + mergePolicy: Replace + metadata: + labels: + argocd.argoproj.io/secret-type: repository + data: + type: git + url: "https://{{ $gitlab }}/{{ $user }}/fleet-config.git" + username: "{{ $user }}" + password: "{{ `{{ .GIT_PASSWORD }}` }}" + data: + - secretKey: GIT_PASSWORD + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: {{ $key }} + metadataPolicy: None + nullBytePolicy: Ignore + property: git_token +--- +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: gitlab-user-repo-creds + namespace: {{ $ns }} + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + secretStoreRef: + name: {{ $store }} + kind: ClusterSecretStore + refreshInterval: "5m" + target: + name: gitlab-user-repo-creds + creationPolicy: Owner + deletionPolicy: Retain + template: + engineVersion: v2 + mergePolicy: Replace + metadata: + labels: + argocd.argoproj.io/secret-type: repo-creds + data: + type: git + url: "https://{{ $gitlab }}/{{ $user }}" + username: "{{ $user }}" + password: "{{ `{{ .GIT_PASSWORD }}` }}" + data: + - secretKey: GIT_PASSWORD + remoteRef: + conversionStrategy: Default + decodingStrategy: None + key: {{ $key }} + metadataPolicy: None + nullBytePolicy: Ignore + property: git_token diff --git a/gitops/addons/charts/argocd-gitlab-repo-creds/values.yaml b/gitops/addons/charts/argocd-gitlab-repo-creds/values.yaml new file mode 100644 index 000000000..0e8ede5c1 --- /dev/null +++ b/gitops/addons/charts/argocd-gitlab-repo-creds/values.yaml @@ -0,0 +1,6 @@ +namespace: argocd +secretStoreName: aws-secrets-manager +# Populated from cluster-secret annotations via the addon registry valuesObject. +aws_cluster_name: "" +gitlab_domain_name: "" +git_username: "user1" diff --git a/gitops/addons/charts/backstage/templates/argocd-secrets.yaml b/gitops/addons/charts/backstage/templates/argocd-secrets.yaml index e5ede8ca2..1d8b55982 100644 --- a/gitops/addons/charts/backstage/templates/argocd-secrets.yaml +++ b/gitops/addons/charts/backstage/templates/argocd-secrets.yaml @@ -95,7 +95,7 @@ spec: secretStoreRef: name: aws-secrets-manager kind: ClusterSecretStore - refreshInterval: "1h" + refreshInterval: "5m" target: name: git-credentials creationPolicy: Owner diff --git a/gitops/addons/charts/backstage/templates/install.yaml b/gitops/addons/charts/backstage/templates/install.yaml index ce3c7943a..c1d8fdeb6 100644 --- a/gitops/addons/charts/backstage/templates/install.yaml +++ b/gitops/addons/charts/backstage/templates/install.yaml @@ -785,13 +785,16 @@ spec: matchLabels: app: backstage --- +{{- /* insecure: HTTP-only ALB ingress, no cert. Explicit .Values.insecure wins; + falls back to the legacy exposure_mode == "cloudfront" during transition. */ -}} +{{- $insecure := or .Values.insecure (eq (default "domain" .Values.exposure_mode) "cloudfront") -}} apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: backstage namespace: backstage annotations: -{{- if eq (default "domain" .Values.exposure_mode) "cloudfront" }} +{{- if $insecure }} alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]' alb.ingress.kubernetes.io/healthcheck-path: /backstage/health/v1/readiness @@ -807,7 +810,7 @@ metadata: nginx.ingress.kubernetes.io/session-cookie-path: "/backstage" {{- end }} spec: -{{- if eq (default "domain" .Values.exposure_mode) "cloudfront" }} +{{- if $insecure }} ingressClassName: "platform" rules: - http: diff --git a/gitops/addons/charts/crossplane-base/templates/provider-credential-restart.yaml b/gitops/addons/charts/crossplane-base/templates/provider-credential-restart.yaml new file mode 100644 index 000000000..dbb45edf8 --- /dev/null +++ b/gitops/addons/charts/crossplane-base/templates/provider-credential-restart.yaml @@ -0,0 +1,104 @@ +{{- /* + Provider Pod Identity credential bootstrap (#2). + + EKS Pod Identity injects credentials into a pod's environment via a webhook at + pod CREATION time only. The Crossplane AWS provider pods (provider-aws-dynamodb, + -rds, -ec2, ...) are started by the package manager at wave 3, but their + PodIdentityAssociations (iam.yaml, same chart) only become active a bit later. + A provider pod that started before its PIA was active runs credential-less + ("cannot retrieve the AWS credentials: no EC2 IMDS role found") and every + managed resource it owns (e.g. a DynamoDB Table) stays Synced=False forever. + + The hub fixes this with `task hub:restart-identity-pods`, but spokes are pure + GitOps with no Taskfile run against them. This PostSync hook is the spoke-side + equivalent: it deletes any provider pod that is missing the injected + AWS_CONTAINER_CREDENTIALS_FULL_URI env var so the Deployment recreates it and + the webhook injects the (now active) credentials. + + NOTE: the shared pod-identity-restart-hook cannot be reused here — it does + `kubectl rollout restart deployment -l