KFLUXINFRA-4238: (onboarding) Migrate staging etcd-shield AppSet to ring deployments ArgoCD

[enkeefe00]

Summary of Changes

• Removed auto-sync policy from etcd-shield
• Created a new ApplicationSet to take ownership of the etcd-shield resources on the new ArgoCD staging instance on kflux-c-stg-i01
• Created a duplicate of the etcd-shield/ components directory to restructure etcd-shield in accordance with the new component standards

Risk Level

Risk Level: Medium
Reasoning: No new component resources are being created; the new ApplicationSet references the etcd-shield-rd/ directory, which is only a restructuring of the existing etcd-shield/ directory
Rollback Strategy: Do a manual sync for the original etcd-shield ApplicationSet, as it refrences the original structuring of etcd-shield.

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 6:58 PM UTC · Ended 7:01 PM UTC
Commit: 3041455 · View workflow run →

[github-actions]

Kustomize Render Diff

Comparing cf212d3c6 → ed8a455b8

No render differences detected.

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:02 PM UTC · Ended 7:13 PM UTC
Commit: 3041455 · View workflow run →

[qodo-for-redhat-appstudio]

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 3 rules

1. Missing base-snapshot resource 🐞 Bug ≡ Correctness

Description

The ring base kustomizations reference a base-snapshot resource that does not exist, so kustomize
rendering (and therefore ArgoCD sync) will fail for any cluster kustomization that includes
../base (e.g. ring-1 staging clusters).

Code

components/etcd-shield-rd/rings/ring-1/base/kustomization.yaml[R1-4]

+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- base-snapshot

Relevance

⭐⭐⭐ High

Missing/incorrect kustomize resources causing render/sync failure are typically fixed; similar
“missing path/empty-base” fix accepted (#13067).

PR-#13067
PR-#10549

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The ring base kustomization includes base-snapshot, and ring-1 cluster kustomizations include
../base, which pulls in that missing resource during build/sync.

components/etcd-shield-rd/rings/ring-1/base/kustomization.yaml[1-4]
components/etcd-shield-rd/rings/ring-0/base/kustomization.yaml[1-4]
components/etcd-shield-rd/rings/ring-1/stone-stage-p01/kustomization.yaml[1-4]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`components/etcd-shield-rd/rings/ring-{0,1}/base/kustomization.yaml` lists `base-snapshot` as a resource, but no such directory/resource exists in the repo. Any ring cluster overlay that includes `../base` will fail to build.

### Issue Context
Ring-1 staging cluster kustomizations include `../base`, so this breaks the intended deployment targets for rd-staging.

### Fix Focus Areas
- components/etcd-shield-rd/rings/ring-1/base/kustomization.yaml[1-4]
- components/etcd-shield-rd/rings/ring-0/base/kustomization.yaml[1-4]
- components/etcd-shield-rd/rings/ring-1/stone-stage-p01/kustomization.yaml[1-4]

### Suggested fix
Either:
1) Create `components/etcd-shield-rd/rings/ring-*/base/base-snapshot/` with a `kustomization.yaml` that includes the actual base (e.g. `resources: ["../../../../base"]` depending on desired layout), **or**
2) Replace `- base-snapshot` with a correct relative path to `components/etcd-shield-rd/base` (e.g. `- ../../../base`).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Broken metrics TLS configuration ✗ Dismissed 🐞 Bug ☼ Reliability

Description

ServiceMonitor/etcd-shield-metrics is configured for HTTPS and requires service-ca.crt from the
metrics-reader Secret, but that Secret is a service-account token and the metrics Service has no
serving-cert annotation, so Prometheus scrapes are expected to fail.

Code

components/etcd-shield-rd/base/metrics/monitor.yaml[R9-23]

+    - interval: 15s
+      scheme: https
+      path: /metrics
+      port: metrics
+      authorization:
+        credentials:
+          key: token
+          name: metrics-reader
+      tlsConfig:
+        ca:
+          secret:
+            key: service-ca.crt
+            name: metrics-reader
+            optional: false
+        serverName: etcd-shield.etcd-shield.svc

Relevance

⭐⭐⭐ High

Team has accepted fixing ServiceMonitor TLS/CA misconfigs that break Prometheus scrapes (e.g., PRs
#11694, #11283).

PR-#11694
PR-#11283

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The ServiceMonitor requires HTTPS + a CA key from metrics-reader, but metrics-reader is defined
as a service-account token Secret and the metrics Service definition has no serving-cert annotation
to back an HTTPS scrape.

components/etcd-shield-rd/base/metrics/monitor.yaml[8-23]
components/etcd-shield-rd/base/metrics/serviceaccount.yaml[1-7]
components/etcd-shield-rd/base/metrics/metrics-service.yaml[1-16]
components/etcd-shield-rd/base/deployment.yaml[28-33]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The ServiceMonitor is configured for `scheme: https` with a TLS CA sourced from `Secret/metrics-reader` key `service-ca.crt`. The referenced Secret is a `kubernetes.io/service-account-token` and the metrics Service lacks an OpenShift serving-cert annotation, so the expected CA material/cert chain is not established.

### Issue Context
The Deployment exposes metrics on port 9100 (`-metrics-addr=:9100`), and the metrics Service targets port 9100.

### Fix Focus Areas
- components/etcd-shield-rd/base/metrics/monitor.yaml[8-23]
- components/etcd-shield-rd/base/metrics/serviceaccount.yaml[1-13]
- components/etcd-shield-rd/base/metrics/metrics-service.yaml[1-16]
- components/etcd-shield-rd/base/deployment.yaml[28-33]

### Suggested fix
Choose one supported model and make all resources consistent:
- **If metrics are plain HTTP (common):** set ServiceMonitor `scheme: http` and remove `tlsConfig` and `authorization` (and then remove the now-unused `metrics-reader` Secret/SA/RBAC if no longer needed).
- **If metrics must be HTTPS:** add an OpenShift serving-cert annotation to the *metrics* Service, mount the serving cert into the pod (or configure the app to use it), and reference the correct CA bundle Secret/key in `tlsConfig.ca` (do not use the SA token Secret as a CA store).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Malformed ApplicationSet nesting ✓ Resolved 🐞 Bug ≡ Correctness

Description

etcd-shield-appset.yaml has incorrectly indented/nested fields (e.g.,
spec.generators[*].merge.generators, selector.matchLabels, and spec.template), producing an
invalid ApplicationSet structure that will be rejected or will not generate the intended
Applications.

Code

argo-cd-apps/overlays/rd-staging/etcd-shield/etcd-shield-appset.yaml[R8-44]

+    generators:
+        - merge:
+            mergeKeys:
+                - nameNormalized
+            generators:
+            - clusters:
+                selector:
+                    matchLabels:
+                    appstudio.redhat.com/is-tenant-cluster: "true"
+                values:
+                    sourceRoot: components/etcd-shield-rd
+                    ring: "empty-base"
+                    clusterDir: "empty-base"
+            - list:
+                elements: []
+template:
+    metadata:
+    name: etcd-shield-rd-{{nameNormalized}}
+    spec:
+    project: default
+    source:
+        path: '{{values.sourceRoot}}/rings/{{values.ring}}/{{values.clusterDir}}'
+        repoURL: https://github.com/redhat-appstudio/infra-deployments.git
+        targetRevision: main
+    destination:
+        namespace: etcd-shield
+        server: '{{server}}'
+    syncPolicy:
+        # automated:
+        #   prune: true
+        #   selfHeal: true
+        syncOptions:
+        - CreateNamespace=true
+        retry:
+        limit: 50
+        backoff:
+            duration: 15s

Relevance

⭐⭐⭐ High

Broken ApplicationSet YAML would block ArgoCD; team maintains correct AppSet structure in overlay
refactors (PR 13039).

PR-#13039

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The rd-staging etcd-shield ApplicationSet places template: at the top level and mis-indents
matchLabels and the template subtree; the rd-staging authentication ApplicationSet shows the
correct structure to follow.

argo-cd-apps/overlays/rd-staging/etcd-shield/etcd-shield-appset.yaml[8-44]
argo-cd-apps/overlays/rd-staging/authentication/authentication-appset.yaml[8-45]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new ApplicationSet manifest for rd-staging etcd-shield is not structured like a valid `ApplicationSet`: `template` is not nested under `spec`, `matchLabels` is not a map, and multiple fields are at the wrong indentation level.

## Issue Context
A working reference exists in the same overlay (`authentication-appset.yaml`), which uses the expected nesting for `spec.generators[].merge.generators[]`, `selector.matchLabels`, and `spec.template.metadata/spec`.

## Fix Focus Areas
- argo-cd-apps/overlays/rd-staging/etcd-shield/etcd-shield-appset.yaml[8-44]
- argo-cd-apps/overlays/rd-staging/authentication/authentication-appset.yaml[8-45]

## Implementation notes
- Ensure `spec.generators[0].merge.generators` is a properly indented YAML list containing `clusters` and `list` generators.
- Ensure `selector.matchLabels` contains `appstudio.redhat.com/is-tenant-cluster: "true"` as a child key.
- Move and indent `template:` under `spec:` as `spec.template:` and nest `metadata.name` and `spec.*` correctly.
- Fix list indentation under `syncOptions:` and indentation under `retry:` (`limit`, `backoff.duration`, etc.).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Wrong Prometheus CA file ✗ Dismissed 🐞 Bug ☼ Reliability

Description

components/etcd-shield-rd/base/config.yaml sets prometheus.config.tls_config.ca_file to
/var/tls/tls.crt, which is sourced from etcd-shield’s own serving-cert secret rather than the
OpenShift service CA used by prometheus-k8s. This can cause TLS verification failures when
etcd-shield queries Prometheus, preventing it from correctly evaluating EtcdShieldDenyAdmission.

Code

components/etcd-shield-rd/base/config.yaml[R3-11]

+prometheus:
+  address: https://prometheus-k8s.openshift-monitoring.svc:9091
+  alertName: EtcdShieldDenyAdmission
+  config:
+    authorization:
+      type: Bearer
+      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+    tls_config:
+      ca_file: /var/tls/tls.crt

Relevance

⭐⭐⭐ High

Team previously fixed TLS CA bundles for Prometheus scraping (use openshift-service-ca.crt) in PRs
#11283/#11294.

PR-#11283
PR-#11294

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The etcd-shield-rd Prometheus client CA is configured to read from /var/tls/tls.crt, while
/var/tls is mounted from the etcd-shield-tls serving-cert secret created for the etcd-shield
Service. Repo examples that talk to prometheus-k8s trust it using the serviceaccount
service-ca.crt bundle instead.

components/etcd-shield-rd/base/config.yaml[3-12]
components/etcd-shield-rd/base/deployment.yaml[76-90]
components/etcd-shield-rd/base/service.yaml[3-6]
components/monitoring/exporters/cardinality/staging/base/resources/auth-config.yaml[9-13]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`etcd-shield-rd` config uses `/var/tls/tls.crt` (etcd-shield’s serving cert file) as the CA bundle for connecting to `https://prometheus-k8s.openshift-monitoring.svc:9091`. This CA is not guaranteed to validate the Prometheus serving cert, so Prometheus queries can fail and the admission-gating logic may not trigger as intended.

### Issue Context
Other in-repo Prometheus clients trust OpenShift service endpoints using the serviceaccount-projected `service-ca.crt` bundle.

### Fix Focus Areas
- components/etcd-shield-rd/base/config.yaml[3-12]

### Suggested fix
Update `prometheus.config.tls_config.ca_file` to the OpenShift service CA bundle path (commonly):
- `/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt`

If your target clusters don’t provide that file, instead mount the OpenShift service CA ConfigMap/Secret used elsewhere in the repo (e.g., `openshift-service-ca.crt`) and point `ca_file` to that mounted path.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Automated sync removed ✗ Dismissed 🐞 Bug ☼ Reliability

Description

The member etcd-shield ApplicationSet template no longer sets syncPolicy.automated
(prune/selfHeal), so etcd-shield will not auto-sync or auto-prune/self-heal drift unless manually
synced or patched elsewhere.

Code

argo-cd-apps/base/member/infra-deployments/etcd-shield/etcd-shield.yaml[R51-54]

      syncPolicy:
-        automated:
-          prune: true
-          selfHeal: true
        syncOptions:
          - CreateNamespace=true
        retry:

Relevance

⭐⭐⭐ High

Team repeatedly turns ON automated prune/selfHeal for AppSets (PRs 11407, 11822, 13039).

PR-#11407
PR-#11822
PR-#13039

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The updated etcd-shield template lacks syncPolicy.automated, while a peer member ApplicationSet
(etcd-defrag) still includes it, demonstrating the behavior change and divergence from established
patterns.

argo-cd-apps/base/member/infra-deployments/etcd-shield/etcd-shield.yaml[51-59]
argo-cd-apps/base/member/infra-deployments/etcd-defrag/etcd-defrag.yaml[51-62]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The etcd-shield ApplicationSet template in `argo-cd-apps/base/member/infra-deployments/etcd-shield/etcd-shield.yaml` removed `syncPolicy.automated` (including `prune` and `selfHeal`). This changes operational behavior: ArgoCD will not automatically apply desired-state changes or prune/self-heal drift.

## Issue Context
Other member infra-deployments ApplicationSets in this repo commonly keep `syncPolicy.automated` enabled (example: `etcd-defrag`). If the intent is to keep etcd-shield aligned with those conventions, automated sync should be restored.

## Fix Focus Areas
- argo-cd-apps/base/member/infra-deployments/etcd-shield/etcd-shield.yaml[51-59]
- argo-cd-apps/base/member/infra-deployments/etcd-defrag/etcd-defrag.yaml[51-62]

## Implementation notes
- Add back:
 - `syncPolicy.automated.prune: true`
 - `syncPolicy.automated.selfHeal: true`
- If automated sync is intentionally disabled for a migration window, document that intent inline and/or ensure there is a follow-up step to re-enable it after cutover.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

6. Legacy SA token secret 🐞 Bug ⛨ Security

Description

The PR explicitly creates a kubernetes.io/service-account-token Secret for metrics-reader, which
is a legacy long-lived credential mechanism and may be disallowed on hardened/newer clusters; if
allowed, it increases the exposure window of a bearer token used for scraping.

Code

components/etcd-shield-rd/base/metrics/serviceaccount.yaml[R1-7]

+apiVersion: v1
+kind: Secret
+metadata:
+  annotations:
+    kubernetes.io/service-account.name: metrics-reader
+  name: metrics-reader
+type: kubernetes.io/service-account-token

Relevance

⭐ Low

Repo repeatedly adds explicit service-account-token Secrets (PR #10645); enforcement to deny these
was rejected (PR #11377).

PR-#10645
PR-#11377

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The metrics-reader Secret is explicitly declared as a service-account token, and the ServiceMonitor
uses that Secret as an authorization credential source.

components/etcd-shield-rd/base/metrics/serviceaccount.yaml[1-7]
components/etcd-shield-rd/base/metrics/monitor.yaml[13-17]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
A `kubernetes.io/service-account-token` Secret is being created explicitly for `metrics-reader`. This is the legacy token Secret mechanism and can be blocked by cluster policy; if populated, it yields a persistent bearer token that is then used by the ServiceMonitor.

### Issue Context
The ServiceMonitor references `Secret/metrics-reader` as an authorization credential source.

### Fix Focus Areas
- components/etcd-shield-rd/base/metrics/serviceaccount.yaml[1-13]
- components/etcd-shield-rd/base/metrics/monitor.yaml[13-17]
- components/etcd-shield-rd/base/metrics/prometheus-crb.yaml[1-24]

### Suggested fix
Prefer eliminating the need for a long-lived scrape token:
- If feasible, remove `authorization` from the ServiceMonitor and avoid creating the token Secret entirely (control access via namespace/network policies instead).
- If authentication is required, switch to an auth pattern that doesn’t depend on legacy SA token Secrets (e.g., a dedicated, rotated credential Secret managed by your platform tooling, or an in-cluster proxy/authenticator approach that supports short-lived tokens).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Previous review results

Review updated until commit 5585006 ⚖️ Balanced

Results up to commit 8575daf

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Malformed ApplicationSet nesting ✓ Resolved 🐞 Bug ≡ Correctness

Description

etcd-shield-appset.yaml has incorrectly indented/nested fields (e.g.,
spec.generators[*].merge.generators, selector.matchLabels, and spec.template), producing an
invalid ApplicationSet structure that will be rejected or will not generate the intended
Applications.

Code

argo-cd-apps/overlays/rd-staging/etcd-shield/etcd-shield-appset.yaml[R8-44]

+    generators:
+        - merge:
+            mergeKeys:
+                - nameNormalized
+            generators:
+            - clusters:
+                selector:
+                    matchLabels:
+                    appstudio.redhat.com/is-tenant-cluster: "true"
+                values:
+                    sourceRoot: components/etcd-shield-rd
+                    ring: "empty-base"
+                    clusterDir: "empty-base"
+            - list:
+                elements: []
+template:
+    metadata:
+    name: etcd-shield-rd-{{nameNormalized}}
+    spec:
+    project: default
+    source:
+        path: '{{values.sourceRoot}}/rings/{{values.ring}}/{{values.clusterDir}}'
+        repoURL: https://github.com/redhat-appstudio/infra-deployments.git
+        targetRevision: main
+    destination:
+        namespace: etcd-shield
+        server: '{{server}}'
+    syncPolicy:
+        # automated:
+        #   prune: true
+        #   selfHeal: true
+        syncOptions:
+        - CreateNamespace=true
+        retry:
+        limit: 50
+        backoff:
+            duration: 15s

Relevance

⭐⭐⭐ High

Broken ApplicationSet YAML would block ArgoCD; team maintains correct AppSet structure in overlay
refactors (PR 13039).

PR-#13039

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The rd-staging etcd-shield ApplicationSet places template: at the top level and mis-indents
matchLabels and the template subtree; the rd-staging authentication ApplicationSet shows the
correct structure to follow.

argo-cd-apps/overlays/rd-staging/etcd-shield/etcd-shield-appset.yaml[8-44]
argo-cd-apps/overlays/rd-staging/authentication/authentication-appset.yaml[8-45]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new ApplicationSet manifest for rd-staging etcd-shield is not structured like a valid `ApplicationSet`: `template` is not nested under `spec`, `matchLabels` is not a map, and multiple fields are at the wrong indentation level.

## Issue Context
A working reference exists in the same overlay (`authentication-appset.yaml`), which uses the expected nesting for `spec.generators[].merge.generators[]`, `selector.matchLabels`, and `spec.template.metadata/spec`.

## Fix Focus Areas
- argo-cd-apps/overlays/rd-staging/etcd-shield/etcd-shield-appset.yaml[8-44]
- argo-cd-apps/overlays/rd-staging/authentication/authentication-appset.yaml[8-45]

## Implementation notes
- Ensure `spec.generators[0].merge.generators` is a properly indented YAML list containing `clusters` and `list` generators.
- Ensure `selector.matchLabels` contains `appstudio.redhat.com/is-tenant-cluster: "true"` as a child key.
- Move and indent `template:` under `spec:` as `spec.template:` and nest `metadata.name` and `spec.*` correctly.
- Fix list indentation under `syncOptions:` and indentation under `retry:` (`limit`, `backoff.duration`, etc.).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Automated sync removed ✗ Dismissed 🐞 Bug ☼ Reliability

Description

The member etcd-shield ApplicationSet template no longer sets syncPolicy.automated
(prune/selfHeal), so etcd-shield will not auto-sync or auto-prune/self-heal drift unless manually
synced or patched elsewhere.

Code

argo-cd-apps/base/member/infra-deployments/etcd-shield/etcd-shield.yaml[R51-54]

      syncPolicy:
-        automated:
-          prune: true
-          selfHeal: true
        syncOptions:
          - CreateNamespace=true
        retry:

Relevance

⭐⭐⭐ High

Team repeatedly turns ON automated prune/selfHeal for AppSets (PRs 11407, 11822, 13039).

PR-#11407
PR-#11822
PR-#13039

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The updated etcd-shield template lacks syncPolicy.automated, while a peer member ApplicationSet
(etcd-defrag) still includes it, demonstrating the behavior change and divergence from established
patterns.

argo-cd-apps/base/member/infra-deployments/etcd-shield/etcd-shield.yaml[51-59]
argo-cd-apps/base/member/infra-deployments/etcd-defrag/etcd-defrag.yaml[51-62]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The etcd-shield ApplicationSet template in `argo-cd-apps/base/member/infra-deployments/etcd-shield/etcd-shield.yaml` removed `syncPolicy.automated` (including `prune` and `selfHeal`). This changes operational behavior: ArgoCD will not automatically apply desired-state changes or prune/self-heal drift.

## Issue Context
Other member infra-deployments ApplicationSets in this repo commonly keep `syncPolicy.automated` enabled (example: `etcd-defrag`). If the intent is to keep etcd-shield aligned with those conventions, automated sync should be restored.

## Fix Focus Areas
- argo-cd-apps/base/member/infra-deployments/etcd-shield/etcd-shield.yaml[51-59]
- argo-cd-apps/base/member/infra-deployments/etcd-defrag/etcd-defrag.yaml[51-62]

## Implementation notes
- Add back:
 - `syncPolicy.automated.prune: true`
 - `syncPolicy.automated.selfHeal: true`
- If automated sync is intentionally disabled for a migration window, document that intent inline and/or ensure there is a follow-up step to re-enable it after cutover.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Results up to commit 7e811f2

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Wrong Prometheus CA file ✗ Dismissed 🐞 Bug ☼ Reliability

Description

components/etcd-shield-rd/base/config.yaml sets prometheus.config.tls_config.ca_file to
/var/tls/tls.crt, which is sourced from etcd-shield’s own serving-cert secret rather than the
OpenShift service CA used by prometheus-k8s. This can cause TLS verification failures when
etcd-shield queries Prometheus, preventing it from correctly evaluating EtcdShieldDenyAdmission.

Code

components/etcd-shield-rd/base/config.yaml[R3-11]

+prometheus:
+  address: https://prometheus-k8s.openshift-monitoring.svc:9091
+  alertName: EtcdShieldDenyAdmission
+  config:
+    authorization:
+      type: Bearer
+      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+    tls_config:
+      ca_file: /var/tls/tls.crt

Relevance

⭐⭐⭐ High

Team previously fixed TLS CA bundles for Prometheus scraping (use openshift-service-ca.crt) in PRs
#11283/#11294.

PR-#11283
PR-#11294

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The etcd-shield-rd Prometheus client CA is configured to read from /var/tls/tls.crt, while
/var/tls is mounted from the etcd-shield-tls serving-cert secret created for the etcd-shield
Service. Repo examples that talk to prometheus-k8s trust it using the serviceaccount
service-ca.crt bundle instead.

components/etcd-shield-rd/base/config.yaml[3-12]
components/etcd-shield-rd/base/deployment.yaml[76-90]
components/etcd-shield-rd/base/service.yaml[3-6]
components/monitoring/exporters/cardinality/staging/base/resources/auth-config.yaml[9-13]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`etcd-shield-rd` config uses `/var/tls/tls.crt` (etcd-shield’s serving cert file) as the CA bundle for connecting to `https://prometheus-k8s.openshift-monitoring.svc:9091`. This CA is not guaranteed to validate the Prometheus serving cert, so Prometheus queries can fail and the admission-gating logic may not trigger as intended.

### Issue Context
Other in-repo Prometheus clients trust OpenShift service endpoints using the serviceaccount-projected `service-ca.crt` bundle.

### Fix Focus Areas
- components/etcd-shield-rd/base/config.yaml[3-12]

### Suggested fix
Update `prometheus.config.tls_config.ca_file` to the OpenShift service CA bundle path (commonly):
- `/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt`

If your target clusters don’t provide that file, instead mount the OpenShift service CA ConfigMap/Secret used elsewhere in the repo (e.g., `openshift-service-ca.crt`) and point `ca_file` to that mounted path.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Results up to commit dc735b0 ⚖️ Balanced

🐞 Bugs (2) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Missing base-snapshot resource 🐞 Bug ≡ Correctness

Description

The ring base kustomizations reference a base-snapshot resource that does not exist, so kustomize
rendering (and therefore ArgoCD sync) will fail for any cluster kustomization that includes
../base (e.g. ring-1 staging clusters).

Code

components/etcd-shield-rd/rings/ring-1/base/kustomization.yaml[R1-4]

+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- base-snapshot

Relevance

⭐⭐⭐ High

Missing/incorrect kustomize resources causing render/sync failure are typically fixed; similar
“missing path/empty-base” fix accepted (#13067).

PR-#13067
PR-#10549

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The ring base kustomization includes base-snapshot, and ring-1 cluster kustomizations include
../base, which pulls in that missing resource during build/sync.

components/etcd-shield-rd/rings/ring-1/base/kustomization.yaml[1-4]
components/etcd-shield-rd/rings/ring-0/base/kustomization.yaml[1-4]
components/etcd-shield-rd/rings/ring-1/stone-stage-p01/kustomization.yaml[1-4]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`components/etcd-shield-rd/rings/ring-{0,1}/base/kustomization.yaml` lists `base-snapshot` as a resource, but no such directory/resource exists in the repo. Any ring cluster overlay that includes `../base` will fail to build.

### Issue Context
Ring-1 staging cluster kustomizations include `../base`, so this breaks the intended deployment targets for rd-staging.

### Fix Focus Areas
- components/etcd-shield-rd/rings/ring-1/base/kustomization.yaml[1-4]
- components/etcd-shield-rd/rings/ring-0/base/kustomization.yaml[1-4]
- components/etcd-shield-rd/rings/ring-1/stone-stage-p01/kustomization.yaml[1-4]

### Suggested fix
Either:
1) Create `components/etcd-shield-rd/rings/ring-*/base/base-snapshot/` with a `kustomization.yaml` that includes the actual base (e.g. `resources: ["../../../../base"]` depending on desired layout), **or**
2) Replace `- base-snapshot` with a correct relative path to `components/etcd-shield-rd/base` (e.g. `- ../../../base`).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Broken metrics TLS configuration ✗ Dismissed 🐞 Bug ☼ Reliability

Description

ServiceMonitor/etcd-shield-metrics is configured for HTTPS and requires service-ca.crt from the
metrics-reader Secret, but that Secret is a service-account token and the metrics Service has no
serving-cert annotation, so Prometheus scrapes are expected to fail.

Code

components/etcd-shield-rd/base/metrics/monitor.yaml[R9-23]

+    - interval: 15s
+      scheme: https
+      path: /metrics
+      port: metrics
+      authorization:
+        credentials:
+          key: token
+          name: metrics-reader
+      tlsConfig:
+        ca:
+          secret:
+            key: service-ca.crt
+            name: metrics-reader
+            optional: false
+        serverName: etcd-shield.etcd-shield.svc

Relevance

⭐⭐⭐ High

Team has accepted fixing ServiceMonitor TLS/CA misconfigs that break Prometheus scrapes (e.g., PRs
#11694, #11283).

PR-#11694
PR-#11283

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The ServiceMonitor requires HTTPS + a CA key from metrics-reader, but metrics-reader is defined
as a service-account token Secret and the metrics Service definition has no serving-cert annotation
to back an HTTPS scrape.

components/etcd-shield-rd/base/metrics/monitor.yaml[8-23]
components/etcd-shield-rd/base/metrics/serviceaccount.yaml[1-7]
components/etcd-shield-rd/base/metrics/metrics-service.yaml[1-16]
components/etcd-shield-rd/base/deployment.yaml[28-33]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The ServiceMonitor is configured for `scheme: https` with a TLS CA sourced from `Secret/metrics-reader` key `service-ca.crt`. The referenced Secret is a `kubernetes.io/service-account-token` and the metrics Service lacks an OpenShift serving-cert annotation, so the expected CA material/cert chain is not established.

### Issue Context
The Deployment exposes metrics on port 9100 (`-metrics-addr=:9100`), and the metrics Service targets port 9100.

### Fix Focus Areas
- components/etcd-shield-rd/base/metrics/monitor.yaml[8-23]
- components/etcd-shield-rd/base/metrics/serviceaccount.yaml[1-13]
- components/etcd-shield-rd/base/metrics/metrics-service.yaml[1-16]
- components/etcd-shield-rd/base/deployment.yaml[28-33]

### Suggested fix
Choose one supported model and make all resources consistent:
- **If metrics are plain HTTP (common):** set ServiceMonitor `scheme: http` and remove `tlsConfig` and `authorization` (and then remove the now-unused `metrics-reader` Secret/SA/RBAC if no longer needed).
- **If metrics must be HTTPS:** add an OpenShift serving-cert annotation to the *metrics* Service, mount the serving cert into the pod (or configure the app to use it), and reference the correct CA bundle Secret/key in `tlsConfig.ca` (do not use the SA token Secret as a CA store).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Legacy SA token secret 🐞 Bug ⛨ Security

Description

The PR explicitly creates a kubernetes.io/service-account-token Secret for metrics-reader, which
is a legacy long-lived credential mechanism and may be disallowed on hardened/newer clusters; if
allowed, it increases the exposure window of a bearer token used for scraping.

Code

components/etcd-shield-rd/base/metrics/serviceaccount.yaml[R1-7]

+apiVersion: v1
+kind: Secret
+metadata:
+  annotations:
+    kubernetes.io/service-account.name: metrics-reader
+  name: metrics-reader
+type: kubernetes.io/service-account-token

Relevance

⭐ Low

Repo repeatedly adds explicit service-account-token Secrets (PR #10645); enforcement to deny these
was rejected (PR #11377).

PR-#10645
PR-#11377

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The metrics-reader Secret is explicitly declared as a service-account token, and the ServiceMonitor
uses that Secret as an authorization credential source.

components/etcd-shield-rd/base/metrics/serviceaccount.yaml[1-7]
components/etcd-shield-rd/base/metrics/monitor.yaml[13-17]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
A `kubernetes.io/service-account-token` Secret is being created explicitly for `metrics-reader`. This is the legacy token Secret mechanism and can be blocked by cluster policy; if populated, it yields a persistent bearer token that is then used by the ServiceMonitor.

### Issue Context
The ServiceMonitor references `Secret/metrics-reader` as an authorization credential source.

### Fix Focus Areas
- components/etcd-shield-rd/base/metrics/serviceaccount.yaml[1-13]
- components/etcd-shield-rd/base/metrics/monitor.yaml[13-17]
- components/etcd-shield-rd/base/metrics/prometheus-crb.yaml[1-24]

### Suggested fix
Prefer eliminating the need for a long-lived scrape token:
- If feasible, remove `authorization` from the ServiceMonitor and avoid creating the token Secret entirely (control access via namespace/network policies instead).
- If authentication is required, switch to an auth pattern that doesn’t depend on legacy SA token Secrets (e.g., a dedicated, rotated credential Secret managed by your platform tooling, or an in-cluster proxy/authenticator approach that supports short-lived tokens).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:13 PM UTC · Completed 7:23 PM UTC
Commit: 3041455 · View workflow run →

[konflux-ci-qe-bot]

🤖 Pipeline Failure Analysis

Category: Infrastructure

The pipeline failed because the Konflux installation timed out due to the ArgoCD application policies-in-cluster-local being unable to sync, as critical ValidatingAdmissionPolicy resources were missing from the cluster.

📋 Technical Details

Immediate Cause

The konflux-ci-install-konflux step timed out after one hour. This timeout occurred because the policies-in-cluster-local ArgoCD application remained in an OutOfSync state, failing to reconcile essential ValidatingAdmissionPolicy resources.

Contributing Factors

The core contributing factor was the inability of the Kubernetes cluster to provision or reconcile the required ValidatingAdmissionPolicy objects, specifically deny-on-validation-error.kueue.konflux-ci.dev, deny-serviceaccount-token-secrets.konflux-ci.dev, and konflux-rbac-restrict-bindings-serviceaccounts-create.konflux-ci.dev. This indicates an underlying issue within the cluster's API server or admission controller configuration.

Impact

The failure to install Konflux successfully blocked the entire appstudio-e2e-tests job, preventing any subsequent e2e tests from executing. The installation step is a prerequisite for a functional test environment.

🔍 Evidence

appstudio-e2e-tests/konflux-ci-install-konflux

Category: infrastructure
Root Cause: The installation timed out because the policies-in-cluster-local ArgoCD application could not successfully sync and apply its resources. It reported "Missing" ValidatingAdmissionPolicy resources, indicating an underlying issue with the Kubernetes cluster's ability to provision or reconcile these policy objects.

Logs:

artifacts/appstudio-e2e-tests/konflux-ci-install-konflux/build-log.txt

[2026-07-21 17:59:06] [INFO] Pending: policies-in-cluster-local

artifacts/appstudio-e2e-tests/konflux-ci-install-konflux/build-log.txt

[2026-07-21 18:01:12] [INFO]   ├─ Sync Status: OutOfSync

artifacts/appstudio-e2e-tests/konflux-ci-install-konflux/build-log.txt

[2026-07-21 18:01:12] [WARN]   │  ├─ Degraded/Missing resources:
[2026-07-21 18:01:12] [WARN]   │  │  └─ ValidatingAdmissionPolicy/deny-on-validation-error.kueue.konflux-ci.dev: Missing
[2026-07-21 18:01:12] [WARN]   │  │  └─ ValidatingAdmissionPolicy/deny-serviceaccount-token-secrets.konflux-ci.dev: Missing
[2026-07-21 18:01:12] [WARN]   │  │  └─ ValidatingAdmissionPolicy/konflux-rbac-restrict-bindings-serviceaccounts-create.konflux-ci.dev: Missing

artifacts/appstudio-e2e-tests/konflux-ci-install-konflux/build-log.txt

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 1h0m0s timeout","severity":"error","time":"2026-07-21T18:35:06Z"}

artifacts/appstudio-e2e-tests/konflux-ci-install-konflux/build-log.txt

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:267","func":"sigs.k8s.io/prow/pkg/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 1m0s grace period","severity":"error","time":"2026-07-21T18:36:06Z"}

---

Analysis powered by prow-failure-analysis | Build: 2079617739193847808

[fullsend-ai-review]

Review

Verdict: comment · 1 medium, 1 low finding

This PR migrates the staging etcd-shield ApplicationSet from the legacy base/member AppSet model to the ring-deployment model (rd-staging), following the pattern established by the authentication component. The new components/etcd-shield-rd/ directory correctly mirrors the existing components/etcd-shield/base/ content (same image tag 94885d0, same deployment config, same RBAC, same webhook, same alerting rules), with the RBAC refactored from a single rbac.yaml into a cleaner directory structure.

The ring deployment structure is well-formed: empty-base for unmatched clusters, ring-0 (empty, for development), and ring-1 with per-cluster overlays for stone-stage-p01 and stone-stg-rh01. The kflux-stg-es01 cluster correctly gets an empty kustomization since etcd-shield is not deployed to EaaS clusters.

Findings

1. Removing automated sync from base AppSet affects production clusters — medium

File: argo-cd-apps/base/member/infra-deployments/etcd-shield/etcd-shield.yaml

The automated sync policy (prune: true, selfHeal: true) is removed from the base AppSet. This base is included by both staging-downstream and production-downstream overlays. While the new rd-staging AppSet re-adds automated sync for staging clusters, production clusters served by the production-downstream overlay will lose auto-sync and self-heal since no corresponding rd-production etcd-shield AppSet is being created in this PR.

The infra/hold-production label and multi-environment labels suggest the team is aware and managing this as a staged migration. However, until the production ring-deployment migration is completed, production clusters relying on the old AppSet will require manual sync operations.

Remediation: Consider whether the automated sync removal could be deferred to a later PR that simultaneously introduces the rd-production etcd-shield AppSet, or document the interim manual-sync requirement for production clusters in the PR description.

2. PR body is empty — low

File: (PR metadata)

The PR body is null. Per project conventions in AGENTS.md, production PRs should include ## Risk Assessment (level, description, rollback plan) and ## Validation (staging evidence) sections. While an AI-generated risk assessment exists in the comments (from @eedri), the PR itself has no description explaining the migration strategy or the intent behind removing automated sync from the base.

Previous run

Review

Verdict: Approve

This PR migrates the staging etcd-shield ApplicationSet from the legacy multi-environment base AppSet to the ring deployment pattern in rd-staging, following the same structure already established by the authentication component.

What was reviewed

• Base AppSet change (argo-cd-apps/base/member/.../etcd-shield.yaml): Removes syncPolicy.automated (prune + selfHeal). Confirmed intentional by the author — this disables auto-sync on the legacy AppSet so the new ring deployment AppSet takes over for staging.
• New rd-staging AppSet (argo-cd-apps/overlays/rd-staging/etcd-shield/): Creates a ring-deployment-style ApplicationSet with member label, merge generator, and empty-base default. Structure matches the authentication reference AppSet exactly.
• Member clusters patch: Assigns stone-stage-p01 and stone-stg-rh01 to ring-1. Correctly excludes kflux-stg-es01 (EaaS cluster) from the member patch.
• New component (components/etcd-shield-rd/): Faithful copy of the existing components/etcd-shield/base/ manifests (deployment, service, webhook, config, alerts, metrics, RBAC) reorganized into a ring overlay structure. RBAC split from single file into rbac/ directory; metrics files renamed but content identical.
• Ring structure: empty-base/ (no-op default), ring-0/base/ (canary placeholder), ring-1/ with per-cluster directories including kflux-stg-es01 with resources: [] to document the intentional exclusion.
• OWNERS: Correctly assigned to konflux-infra-team.

Correctness

• The new AppSet template path {{values.sourceRoot}}/rings/{{values.ring}}/{{values.clusterDir}} resolves correctly for all cluster configurations.
• Kustomization reference chains are valid: per-cluster dirs → ring base → component base.
• The image tag (94885d051db284f214f43d76df2f9bd4c9fe993f) matches the existing base.
• Retry settings (limit: 50, duration: 15s) align with the authentication AppSet convention.

Security

• RBAC content is identical to the existing etcd-shield/base/rbac.yaml — no privilege escalation.
• ClusterRoles scoped to tokenreviews, subjectaccessreviews, cluster-monitoring-view, and /metrics — all required for admission webhook and monitoring functionality.
• Metrics reader ServiceAccount token secret follows existing pattern.
• Webhook failurePolicy: Fail matches existing configuration.

Observations (low severity)

1. Missing PR description: The PR body is empty. Project conventions call for What/Why/Validation sections on all PRs, and a Risk Assessment section for production-touching PRs. The infra/hold-production label acknowledges the production scope, but a description would help reviewers and future archaeology.

2. Auto-sync removal affects all base consumers: The base AppSet serves staging-downstream, production-downstream, development, and konflux-public-production overlays. Only staging is migrated to the ring deployment AppSet in this PR. Production and other environments lose auto-sync with no ring deployment replacement yet. This appears to be an intentional migration step (disable old before enabling new), but should be tracked as follow-up work.

[qodo-for-redhat-appstudio]

Code review by qodo was updated up to the latest commit 7e811f2

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.11%. Comparing base (3041455) to head (5585006).
⚠️ Report is 50 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #13112   +/-   ##
=======================================
  Coverage   57.11%   57.11%           
=======================================
  Files          23       23           
  Lines        1455     1455           
=======================================
  Hits          831      831           
  Misses        548      548           
  Partials       76       76           

Flag	Coverage Δ
go	57.11% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[eedri]

Risk Assessment

AI calculated Risk — based on code changes and PR description.
Override this by adding your own ## Risk Assessment section to the PR description.

Risk Level: High
Description: This PR deploys a new etcd-shield validating webhook to two staging clusters using an ArgoCD ApplicationSet. The webhook has failurePolicy: Fail for PipelineRun creations, meaning any issue with the webhook could block all PipelineRuns in the affected clusters, making this a high-risk change despite being in staging.
Rollback: Revert this PR to remove the ApplicationSet and the etcd-shield resources, which should disable the validating webhook and restore PipelineRun functionality.

[mcada]

@enkeefe00 this PR modifies files that affect multiple environments simultaneously, which blocks production approval.

Changes must be merged in order:

1. development — open a PR with only the development environment file changes
2. staging — once merged, open a PR for the staging environment changes and let it soak for at least 1 day before proceeding
3. production — only then update this PR to contain only the production environment changes

The environment/* labels are generated automatically from the files changed — splitting the files across separate PRs will result in each PR having only its target environment label.

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:32 PM UTC · Ended 7:34 PM UTC
Commit: cf212d3 · View workflow run →

[fullsend-ai-review]

🤖 Review · ❌ Terminated · Started 7:35 PM UTC · Ended 7:43 PM UTC
Commit: cf212d3 · View workflow run →

[qodo-for-redhat-appstudio]

Code review by qodo was updated up to the latest commit dc735b0

[fullsend-ai-review]

See the review comment for full details.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:35 PM UTC · Completed 7:43 PM UTC
Commit: cf212d3 · View workflow run →

[enkeefe00]

/agentic_review

[qodo-for-redhat-appstudio]

Code review by qodo was updated up to the latest commit 5585006

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enkeefe00, sadlerap

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [enkeefe00,sadlerap]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[enkeefe00]

/retest

[enkeefe00]

@enkeefe00 this PR modifies files that affect multiple environments simultaneously, which blocks production approval.

Changes must be merged in order:

1. development — open a PR with only the development environment file changes
2. staging — once merged, open a PR for the staging environment changes and let it soak for at least 1 day before proceeding
3. production — only then update this PR to contain only the production environment changes

The environment/* labels are generated automatically from the files changed — splitting the files across separate PRs will result in each PR having only its target environment label.

@mcada I don't think there is a way to write this PR to separate out environments. I updated the description with the changes I made. The only production level change is turning off auto-sync for all etcd apps, which is fine since can manually sync them.