feat: move GitLab from EKS to EC2 docker-compose

.gitignore

@@ -175,3 +175,9 @@ private/
 
 # Local platform config (user-specific, not committed)
 config.local.yaml
+
+# Local scratch / chat session artifacts (never commit)
+.local/
+
+# Pinned platform artifact clone (OAP-style consumption)
+.platform/

.kiro/steering/troubleshooting.md

@@ -266,6 +266,86 @@ annotations:
 For Kargo specifically this lives in
 `gitops/overlays/environments/control-plane/kargo/values.yaml` under `api.ingress.annotations`.
 
+### CloudFront platform URLs hang (curl 000): VPC origin points to a stale/recreated ALB
+
+**Symptom**: In cloudfront exposure mode, every platform URL (`https://$CF/keycloak/...`,
+`/backstage`, `/grafana`, `/argo-workflows`, ...) times out with curl **`000`** — not `404`,
+not `5xx`. `idc:configure` loops forever on "Keycloak not ready" (the SAML descriptor never
+returns 200). A **direct in-cluster** curl to the internal ALB DNS returns `200/302`, proving
+the ALB, targets and LBC listener rules are healthy — only the **CloudFront → ALB** hop fails.
+
+**Root cause**: The platform ALB (`<hub>-platform`) was **deleted and recreated** (new
+ARN/DNS), but the CloudFront **VPC Origin is still bound to the OLD (deleted) ALB ARN**.
+`hub-distribution` only acts when no `<hub>-platform` distribution exists yet, so on a re-run it
+**skips** and never re-points the VPC origin → CloudFront keeps sending traffic to a dead ALB →
+the connection hangs → `000`.
+
+Who recreates the ALB: the **AWS Load Balancer Controller**. When the pre-created ALB's
+*immutable* attributes don't match the `platform` IngressClassParams, the LBC can't modify them
+in place, so it **deletes and recreates** the ALB. The classic trigger is a **scheme mismatch**:
+`create-alb` builds the ALB `internal` (required for a CloudFront VPC-Origin backend), but if
+`IngressClassParams.spec.scheme` is `internet-facing`, scheme is immutable → LBC delete+recreate
+loop. Every recreation orphans the VPC origin. (CloudTrail shows `DeleteLoadBalancer` by
+`<hub>-LBCPodIdentityRole` with userAgent `elbv2.k8s.aws/...`.)
+
+**Diagnosis**:
+```bash
+HUB=peeks-hub   # adjust to your resource prefix
+
+# 1. Direct in-cluster hit to the internal ALB (bypasses CloudFront). 200/302 => ALB is fine,
+#    so the break is the CloudFront->ALB hop (stale VPC origin), NOT the ALB/targets/rules.
+ALB_DNS=$(aws elbv2 describe-load-balancers --names "$HUB-platform" --query 'LoadBalancers[0].DNSName' --output text)
+kubectl --context "$HUB" -n keycloak run curltest --rm -i --restart=Never --image=curlimages/curl --command -- \
+  curl -s -o /dev/null -w '%{http_code}\n' --max-time 10 "http://$ALB_DNS/keycloak/realms/master"
+
+# 2. THE check: does the VPC origin's ARN match the CURRENT ALB ARN? (mismatch = stale = the bug)
+ALB_ARN=$(aws elbv2 describe-load-balancers --names "$HUB-platform" --query 'LoadBalancers[0].LoadBalancerArn' --output text)
+DIST=$(aws cloudfront list-distributions --query "DistributionList.Items[?Comment=='$HUB-platform'].Id" --output text)
+VO=$(aws cloudfront get-distribution --id "$DIST" --query 'Distribution.DistributionConfig.Origins.Items[0].VpcOriginConfig.VpcOriginId' --output text)
+echo "VPC origin ARN : $(aws cloudfront get-vpc-origin --id "$VO" --query 'VpcOrigin.VpcOriginEndpointConfig.Arn' --output text)"
+echo "current ALB ARN: $ALB_ARN"     # if these differ, CloudFront points at a dead ALB
+
+# 3. Confirm the LBC is the one deleting/recreating the ALB (and when)
+aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteLoadBalancer \
+  --max-results 5 --query 'Events[].{Time:EventTime,User:Username}' --output table
+```
+
+**Fix**:
+- **Stop the churn at the source**: ensure the `platform` IngressClassParams `scheme` is
+  `internal` in cloudfront mode so the LBC **adopts** the pre-created internal ALB instead of
+  delete-recreating it (`gitops/addons/registry/core.yaml` `ingress-class-alb.valuesObject.scheme`
+  + `gitops/addons/charts/ingress-class-alb/templates/ingressclass.yaml`). After this, CloudTrail
+  should show **no recurring** `DeleteLoadBalancer` events and the ALB ARN stays stable.
+- **If a stale VPC origin already exists**, re-point CloudFront at the current ALB: create a NEW
+  VPC origin for the current ALB ARN → update the distribution's origin to the new `VpcOriginId`
+  and current ALB DNS → wait for `Deployed` → delete the old VPC origin. Note a VPC origin
+  **cannot be updated in place while attached to a distribution** (`CannotUpdateEntityWhileInUse`),
+  so you must create-new / swap / delete-old rather than edit its ARN.
+
+**Prevention (root cause) — implemented**:
+- `IngressClassParams.spec.scheme = internal` in cloudfront mode so the LBC adopts the internal
+  ALB in place instead of delete-recreating it (scheme is immutable).
+- `create-alb` selects PRIVATE subnets (by `kubernetes.io/role/internal-elb` tag, then by
+  `MapPublicIpOnLaunch==false`) and **tags them `kubernetes.io/role/internal-elb=1`**, so the LBC
+  discovers the same subnet set and does not `SetSubnets`/churn the ALB. See
+  `cluster-providers/common/Taskfile.cloudfront.yaml` (`create-alb`).
+
+**Future hardening (NOT yet implemented) — `cloudfront:sync-vpc-origin` reconcile**:
+If an ALB is ever recreated despite the above (re-runs, other immutable-attr drift), a reconcile
+task would self-heal the stale origin. Proposed design (idempotent; no-op when the ARN already
+matches, so cheap on healthy installs):
+1. Guard on `EXPOSURE_MODE == cloudfront`; resolve current `<hub>-platform` ALB ARN + DNS; resolve
+   the distribution (`Comment == <hub>-platform`) and its origin `VpcOriginId` → that VPC origin's ARN.
+2. If `VPC-origin ARN == current ALB ARN` → **no-op** (done).
+3. Else drift → create-new / swap / delete-old (a VPC origin can't be edited in place while
+   attached): `create-vpc-origin` for the current ALB → poll `Deployed` → `update-distribution`
+   origin to the new `VpcOriginId` + current ALB DNS → poll `Deployed` → `delete-vpc-origin` (old)
+   → re-authorize `CloudFront-VPCOrigins-Service-SG → <current ALB SG>:80`.
+Wire into `install:phase1-cloudfront` (both providers) after `sync-domain`. Cost is one or two
+CloudFront deploys (~5–15 min each) ONLY on drift; a no-op otherwise. Deliberately left
+unimplemented for now — the prevention above should keep the ALB stable; add this only if churn
+recurs.
+
 ## ACK (AWS Controllers for Kubernetes) Issues
 
 ### ACK "scheduled for deletion" loop
@@ -316,6 +396,46 @@ The `services.k8s.aws/force-reconcile` annotation does NOT always work (especial
 
 **Fix**: Delete the CRD manually (`kubectl delete crd <name>.kro.run`), then sync to let KRO recreate it. **Warning**: This deletes all instances of that CRD — may trigger resource deletion in AWS. Only do this when safe.
 
+### Hub kro/ack capability RBAC-denied on ACK resources (kro-provisioned spokes)
+**Symptoms**: A spoke provisioned via the **KRO path** never gets its EKS cluster. The
+`EksclusterWithVpc` instance on the hub shows a reconcile error like:
+```
+vpcs.ec2.services.k8s.aws "<spoke>-vpc" is forbidden: User
+"arn:aws:sts::<acct>:assumed-role/<hub>-kro-capability-role/KRO" cannot get resource
+"vpcs" in API group "ec2.services.k8s.aws" in the namespace "<spoke>"
+```
+
+**Root cause**: A kro-provisioned spoke's `EksclusterWithVpc` claim is reconciled by the
+**hub's** kro capability (`<hub>-kro-capability-role/KRO`), which renders the ACK
+VPC/subnet/EKS CRs. The hub's kro/ack capability roles are NOT cluster-admin (they get
+`AmazonEKSClusterPolicy` / inline ACK policies, mirroring the kro-ack RGD). The
+`eks-capabilities-rbac` ClusterRole that grants the ACK API groups only targeted
+`enable_kro_manifests` (spokes); the hub uses `enable_kro_manifests_hub`, so the hub never
+got the RBAC. The chart comment "redundant on the hub because the capability role has
+ClusterAdminPolicy" is wrong — the hub kro/ack cap is not cluster-admin in either the
+crossplane or kro-ack flow. This is a **shared latent gap**; it surfaces on whichever flow
+first exercises a KRO-provisioned spoke (crossplane: `spoke-prod`).
+
+**Fix (GitOps)**: `eks-capabilities-rbac-hub` registry entry (`gitops/addons/registry/platform.yaml`),
+gated on `enable_kro_manifests_hub`, deploys the same `eks-capabilities-kro` ClusterRole+Binding
+on the hub, bound to `<hub>-kro-capability-role/KRO` and `<hub>-ack-capability-role/ACK`.
+
+**Applying to an already-running hub**: the hub's `addonsRepoRevision` tracks the branch, so a
+hard-refresh of the `cluster-addons` ApplicationSet regenerates it and syncs
+`eks-capabilities-rbac-hub-<hub>` automatically:
+```bash
+kubectl -n argocd annotate applicationset cluster-addons argocd.argoproj.io/refresh=hard --overwrite
+```
+(Or apply the rendered ClusterRole/Binding directly for an immediate unblock.)
+
+**Manual recovery — KRO does NOT self-heal the stuck ACK object**: after the RBAC is fixed,
+KRO will not reconcile *over* the ACK resource that was left half-created during the denied
+window. Delete the stuck ACK object so KRO recreates it cleanly:
+```bash
+kubectl --context <hub> delete vpcs.ec2.services.k8s.aws <spoke>-vpc -n <spoke>
+# KRO recreates it (now permitted) and the spoke provisioning resumes.
+```
+
 ## Crossplane Issues
 
 ### NAT Gateway reference resolution race condition

ITERATION_PLAN.md

@@ -162,17 +162,19 @@ The root Taskfile delegates to providers but each provider implements tasks diff
 
 ---
 
-### 10. Remove ArgoCD capability create/delete Job when Crossplane supports it
+### 10. Remove ArgoCD capability create/delete Job when Crossplane supports it ✅ DONE
 
 **Priority:** Low
 **Labels:** enhancement, tech-debt
 **Ref:** https://github.com/crossplane-contrib/provider-upjet-aws/pull/2015
 
-The `create-capability.yaml` and `delete-capability.yaml` Jobs use AWS CLI to manage the EKS ArgoCD Capability because Crossplane's EKS provider doesn't support it yet. Once `provider-upjet-aws` adds `EKSCapability` support:
-- Replace Jobs with Crossplane managed resources
-- Remove `manifests/argocd/create-capability.yaml` and `delete-capability.yaml`
-- Remove `argocd:capability` and `argocd:delete-capability` tasks
-- Add capability to the PlatformCluster composition or as a separate claim
+**Done** (provider-aws-eks v2.6.1 adds the `Capability` MR). The EKS Capabilities
+(KRO/ACK/ArgoCD) are now created declaratively:
+- Crossplane path: native `Capability` MRs in the `platform-cluster` Composition,
+  gated by `spec.capabilities.<type>.enabled` (hub + spokes).
+- The imperative `create-capability.yaml` / `delete-capability.yaml` Jobs and the
+  `argocd:capability` / `argocd:delete-capability` / `spokes:create-capabilities`
+  tasks were removed; `hub:seed` waits for the Capability MRs to be Ready.
 
 ---
 

README.md

@@ -132,13 +132,13 @@ helper:
 
 ```bash
 # Idempotent: skips if a valid config.local.yaml already exists
-./create-config.sh
+workshop/create-config.sh
 
 # Force overwrite
-FORCE=true ./create-config.sh
+FORCE=true workshop/create-config.sh
 
 # Override the branch (e.g. when WORKSHOP_GIT_BRANCH points to a release tag)
-REPO_REVISION=feature/cloudfront-on-agent-platform ./create-config.sh
+REPO_REVISION=... workshop/create-config.sh
 ```
 
 This is a standalone script (not a Taskfile task) because every task reads

Taskfile.yaml

@@ -25,6 +25,12 @@ includes:
     vars:
       CONFIG_FILE: "{{.CONFIG_FILE}}"
     optional: true
+  kind-kro-ack:
+    taskfile: ./cluster-providers/kind-kro-ack/Taskfile.yaml
+    dir: ./cluster-providers/kind-kro-ack
+    vars:
+      CONFIG_FILE: "{{.CONFIG_FILE}}"
+    optional: true
 
 tasks:
   default:

applications/java/deployment/templates/kro/application.yaml

@@ -21,7 +21,6 @@ spec:
     enabled: true
     path: /java-app
     rewrite: true
-    healthcheckPath: /java-app/
   functionalGate:
     enabled: true
     image: "httpd:alpine"

applications/java/kargo/deploy-kargo.sh

@@ -33,9 +33,8 @@ if ! aws iam get-role --role-name "$KARGO_ROLE" --region "$AWS_REGION" >/dev/nul
   echo "  ✓ IAM role $KARGO_ROLE created"
 fi
 KARGO_ROLE_ARN=$(aws iam get-role --role-name "$KARGO_ROLE" --query 'Role.Arn' --output text --region "$AWS_REGION")
-
 if ! aws eks list-pod-identity-associations --cluster-name "$HUB_CLUSTER" --region "$AWS_REGION" \
-    --query "associations[?namespace=='kargo'&&serviceAccount=='kargo-controller']" --output text 2>/dev/null | grep -q .; then
+  --query "associations[?namespace=='kargo'&&serviceAccount=='kargo-controller']" --output text 2>/dev/null | grep -q .; then
   aws eks create-pod-identity-association \
     --cluster-name "$HUB_CLUSTER" --namespace kargo --service-account kargo-controller \
     --role-arn "$KARGO_ROLE_ARN" --region "$AWS_REGION" >/dev/null
@@ -45,7 +44,6 @@ if ! aws eks list-pod-identity-associations --cluster-name "$HUB_CLUSTER" --regi
   kubectl rollout status deployment -n kargo --timeout=60s 2>/dev/null || true
 fi
 
-
 # Deploy the project (creates namespace)
 echo "Creating Kargo project..."
 envsubst < project.yaml | kubectl apply -f -

applications/java/kargo/project.yaml

@@ -5,6 +5,13 @@ metadata:
   annotations:
     argocd.argoproj.io/sync-wave: "-1"
 ---
+# Auto-promotion enabled for both stages. This is required because Kargo v1.7.5
+# has a webhook bug that blocks manually-created Promotion resources when stages
+# use inline `uses:` steps (the webhook incorrectly reports "Stage defines no
+# promotion steps"). The controller bypasses the webhook for auto-promotions,
+# so enabling auto-promotion is the reliable path.
+#
+# Freight flows: Warehouse → dev (auto) → prod (auto, after dev verifies).
 apiVersion: kargo.akuity.io/v1alpha1
 kind: ProjectConfig
 metadata:
@@ -14,4 +21,7 @@ spec:
   promotionPolicies:
   - stageSelector:
       name: dev
-    autoPromotionEnabled: true # Enable auto-promotion for dev
+    autoPromotionEnabled: true
+  - stageSelector:
+      name: prod
+    autoPromotionEnabled: true

applications/java/kargo/promotiontask.yaml

@@ -1,3 +1,8 @@
+# NOTE: This PromotionTask is kept for reference only. It is NOT referenced by
+# the Stage resources (stages.yaml uses inline steps instead). Kargo v1.7.5's
+# admission webhook incorrectly rejects Stages that only reference a task, and
+# the yaml-update step cannot address annotation keys with dots. See stages.yaml
+# for the working inline implementation.
 apiVersion: kargo.akuity.io/v1alpha1
 kind: PromotionTask
 metadata:
@@ -32,7 +37,7 @@ spec:
       updates:
       - key: spec.components.0.properties.image
         value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }}
-      - key: metadata.annotations.app\.oam\.dev/publishVersion
+      - key: metadata.annotations["app.oam.dev/publishVersion"]
         value: ${{ imageFrom(vars.image).Tag }}
   - uses: git-commit
     config:

applications/java/kargo/stages.yaml

@@ -1,11 +1,23 @@
+# Kargo Stages for the Java app.
+#
+# Promotion steps are inlined (not referenced via PromotionTask) because
+# Kargo v1.7.5 admission webhook rejects Stages that only carry a task
+# reference ("Stage defines no promotion steps"). Inline steps work reliably
+# with both auto-promotion (controller-initiated) and manual promotion.
+#
+# The publishVersion annotation update is intentionally omitted — Kargo v1.7.5
+# yaml-update does not support keys containing dots (neither bracket notation
+# nor dot-escaping works). The image field update alone is sufficient to trigger
+# reconciliation for both kro (spec.image) and kubevela (spec.components.0.properties.image)
+# paths. Both steps use continueOnError so the promotion succeeds regardless of
+# which manifest format is present.
 apiVersion: kargo.akuity.io/v1alpha1
 kind: Stage
 metadata:
   name: dev
   namespace: java-app-kargo
   annotations:
     kargo.akuity.io/color: green
-    kargo.akuity.io/auto-promotion: "false"
 spec:
   requestedFreight:
   - origin:
@@ -15,9 +27,50 @@ spec:
       direct: true
   promotionTemplate:
     spec:
+      vars:
+      - name: gitURL
+        value: ${GITLAB_URL}/${GIT_USERNAME}/java.git
+      - name: image
+        value: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/peeks/java
       steps:
-      - task:
-          name: update-image
+      - uses: git-clone
+        config:
+          repoURL: ${{ vars.gitURL }}
+          checkout:
+          - branch: main
+            path: ./src
+      - uses: yaml-update
+        as: update-image-kro
+        continueOnError: true
+        config:
+          path: ./src/deployment/${{ ctx.stage }}/application.yaml
+          updates:
+          - key: spec.image
+            value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }}
+      - uses: yaml-update
+        as: update-image-kubevela
+        continueOnError: true
+        config:
+          path: ./src/deployment/${{ ctx.stage }}/application.yaml
+          updates:
+          - key: spec.components.0.properties.image
+            value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }}
+      - uses: git-commit
+        config:
+          path: ./src
+          message: "Promote ${{ ctx.stage }} to ${{ imageFrom(vars.image).Tag }}"
+          author:
+            name: Kargo
+            email: kargo@akuity.io
+      - uses: git-push
+        config:
+          path: ./src
+      - uses: argocd-update
+        config:
+          apps:
+          - name: java-${{ ctx.stage }}-cd
+            sources:
+            - repoURL: ${{ vars.gitURL }}
 ---
 apiVersion: kargo.akuity.io/v1alpha1
 kind: Stage
@@ -36,6 +89,47 @@ spec:
       - dev
   promotionTemplate:
     spec:
+      vars:
+      - name: gitURL
+        value: ${GITLAB_URL}/${GIT_USERNAME}/java.git
+      - name: image
+        value: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/peeks/java
       steps:
-      - task:
-          name: update-image
+      - uses: git-clone
+        config:
+          repoURL: ${{ vars.gitURL }}
+          checkout:
+          - branch: main
+            path: ./src
+      - uses: yaml-update
+        as: update-image-kro
+        continueOnError: true
+        config:
+          path: ./src/deployment/${{ ctx.stage }}/application.yaml
+          updates:
+          - key: spec.image
+            value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }}
+      - uses: yaml-update
+        as: update-image-kubevela
+        continueOnError: true
+        config:
+          path: ./src/deployment/${{ ctx.stage }}/application.yaml
+          updates:
+          - key: spec.components.0.properties.image
+            value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }}
+      - uses: git-commit
+        config:
+          path: ./src
+          message: "Promote ${{ ctx.stage }} to ${{ imageFrom(vars.image).Tag }}"
+          author:
+            name: Kargo
+            email: kargo@akuity.io
+      - uses: git-push
+        config:
+          path: ./src
+      - uses: argocd-update
+        config:
+          apps:
+          - name: java-${{ ctx.stage }}-cd
+            sources:
+            - repoURL: ${{ vars.gitURL }}