Skip to content

KFLUXINFRA-3980: add type guards to build-platforms CEL expressions (…#13096

Merged
openshift-merge-bot[bot] merged 1 commit into
mainfrom
KFLUXINFRA-3980-prod-ring2
Jul 20, 2026
Merged

KFLUXINFRA-3980: add type guards to build-platforms CEL expressions (…#13096
openshift-merge-bot[bot] merged 1 commit into
mainfrom
KFLUXINFRA-3980-prod-ring2

Conversation

@oswcab

@oswcab oswcab commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

…ring-2)

What

Adds type(value) == list guards to the build-platforms CEL expressions in tekton-kueue (resource-request and AWS-IP expressions), plus a validation-error annotation expression and a temporary tenant-scoped ValidatingAdmissionPolicy that denies PipelineRuns carrying that annotation.

Applies the fix to ring 2 production clusters: kflux-osp-p01, kflux-prd-rh03, stone-prod-p02.

  • stone-prod-p01 and kflux-prd-rh02 previously inherited production/base/tekton-kueue/config.yaml; this adds temporary per-cluster config.yaml overrides (configMapGenerator behavior: replace) to scope the change to ring 1 only.
    kflux-ocp-p01 already had its own override, updated directly.
  • Adds the temporary tenant-scoped ValidatingAdmissionPolicy denying PipelineRuns carrying the validation-error annotation. Wired only into the 3 ring-2 cluster kustomizations, not the shared policies/kueue kustomization, to keep ring isolation.
  • kflux-osp-p01 and kflux-prd-rh03 previously inherited production/base/tekton-kueue/config.yaml; this adds temporary per-cluster config.yaml overrides (configMapGenerator behavior: replace) so the change is scoped to ring 2 only. stone-prod-p02 already had its own override, updated directly.
  • Adds ring-2 test cases for string-typed build-platforms to hack/test-tekton-kueue-config.py.
  • Ring 3 will fold both the CEL change into base/tekton-kueue/config.yaml and the VAP into the shared policies/kueue/kustomization.yaml, removing the temporary overrides — same pattern as the KFLUXINFRA-3973 rollout.
  • Adds a validation-error annotation expression for wrong-typed build-platforms, emitted as kueue.konflux-ci.dev/validation-error (matching the VAP from the start, unlike the original dev/staging rollout — see the ring-1 PR KFLUXINFRA-3980: add type guards to build-platforms CEL expressions (… #13059 and the dev/staging fix for that mismatch).

KFLUXINFRA-3980

Why

The tekton-kueue webhook crashes with HTTP 500 (failurePolicy: Fail) when a PipelineRun submits build-platforms as a string instead of an array, rejecting the PipelineRun outright. Observed on kflux-prd-rh02 on 2026-06-09 (18:12 and 18:14 UTC, 2 rejections). The guard makes the expression tolerant of the wrong type instead of crashing, and the VAP prevents malformed PipelineRuns from silently bypassing the queue while the no-skip-kueue VAP rollout is still in progress.

Validation

  • kustomize build passes cleanly for all affected overlays.
  • Confirmed via rendered diff that only these 3 clusters pick up the CEL guards and the new VAP/VAPB; the ring-3 production clusters render unchanged.
  • python hack/test-tekton-kueue-config.py passes, including new cases covering string-typed build-platforms against the ring-2 configs.
  • Already deployed to dev/staging with no issues: [KFLUXINFRA-3980] Add type guards to build-platforms CEL expressions #12893.

Risk Assessment

Risk Level: Low
What could go wrong: The guard only changes behavior for the previously-crashing case (string-typed build-platforms); array-typed values take the same code path as before. The VAP is scoped to tenant namespaces and only denies PipelineRuns already carrying the validation-error annotation, so it cannot affect any currently-passing PipelineRun.
Rollback: Revert this PR.
Blast radius: 3 named production clusters (ring 2 of 3); no changes to production/base or the shared policy kustomization, so no other production cluster is touched.

@openshift-ci
openshift-ci Bot requested review from eisraeli and glevi-rh July 17, 2026 10:31
@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Kustomize Render Diff

Comparing 91965a555c7e1ef38e

Component Environment Changes
components/kueue/production/kflux-osp-p01 production +12 -2
components/kueue/production/kflux-prd-rh03 production +12 -2
components/kueue/production/stone-prod-p02 production +12 -2
components/policies/production/kflux-osp-p01 production +35 -0
components/policies/production/kflux-prd-rh03 production +35 -0
components/policies/production/stone-prod-p02 production +37 -0

Total: 6 components, +143 -6 lines

📋 Full diff available in the workflow summary and as a downloadable artifact.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 17, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 10:31 AM UTC · Ended 10:36 AM UTC
Commit: 91965a5 · View workflow run →

…ring-2)

The tekton-kueue webhook CEL expressions for multi-platform resource
requests call .value.map() on the build-platforms pipeline parameter,
assuming it is an array. When a PipelineRun submits build-platforms as
a string, the CEL runtime crashes with "got 'types.String', expected
iterable type", causing HTTP 500 and rejecting the PipelineRun entirely
(failurePolicy: Fail). Observed on kflux-prd-rh02 on 2026-06-09.

Applies the fix to ring 2 production clusters: kflux-osp-p01,
kflux-prd-rh03, stone-prod-p02.

- Adds type(value) == list guards to both build-platforms CEL
  expressions (resource request and AWS IP), so they return []
  instead of crashing when the value is a string.
- Adds a validation-error annotation expression for wrong-typed
  build-platforms, emitted as kueue.konflux-ci.dev/validation-error
  (matching the VAP from the start, unlike the original dev/staging
  rollout — see the ring-1 PR #13059 and the dev/staging fix for
  that mismatch).
- Adds the temporary tenant-scoped ValidatingAdmissionPolicy denying
  PipelineRuns carrying the validation-error annotation. Wired only
  into the 3 ring-2 cluster kustomizations, not the shared
  policies/kueue kustomization, to keep ring isolation.
- kflux-osp-p01 and kflux-prd-rh03 previously inherited
  production/base/tekton-kueue/config.yaml; this adds temporary
  per-cluster config.yaml overrides (configMapGenerator behavior:
  replace) so the change is scoped to ring 2 only. stone-prod-p02
  already had its own override, updated directly.
- Adds ring-2 test cases for string-typed build-platforms to
  hack/test-tekton-kueue-config.py.

Already deployed to dev/staging and ring 1 of production (#13059)
with no issues.

Ring 3 will fold the CEL change into base/tekton-kueue/config.yaml
and the VAP into the shared policies/kueue/kustomization.yaml,
removing the temporary per-cluster overrides created in rings 1 and
2, same as the KFLUXINFRA-3973 token-resource rollout.

Contributes to: KFLUXINFRA-3980

Assisted-by: Claude Code (claude-sonnet-5)
Signed-off-by: Hector Oswaldo Caballero <hcaballe@redhat.com>
@oswcab
oswcab force-pushed the KFLUXINFRA-3980-prod-ring2 branch from ef6e986 to 78e07f1 Compare July 17, 2026 10:36
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 17, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 10:37 AM UTC · Completed 10:44 AM UTC
Commit: 91965a5 · View workflow run →

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review — KFLUXINFRA-3980: build-platforms CEL type guards (ring 2)

Verdict: Approve ✅

Summary

This PR extends the ring-2 production rollout of the build-platforms CEL type guard fix to three clusters: kflux-osp-p01, kflux-prd-rh03, and stone-prod-p02. The change prevents the tekton-kueue webhook from crashing (HTTP 500, failurePolicy: Fail) when a PipelineRun submits build-platforms as a string instead of an array. It also adds a kueue.konflux-ci.dev/validation-error annotation expression and wires in the deny-on-validation-error ValidatingAdmissionPolicy for these three clusters.

Analysis

Correctness

  • The type(value) == list guards correctly wrap the two build-platforms CEL expressions (resource-request and AWS-IP), causing string-typed values to fall through to [] instead of crashing the webhook. ✅
  • The validation-error annotation expression correctly fires when build-platforms is present but not a list (!= list), using the kueue.konflux-ci.dev/validation-error annotation key that matches the VAP from the start (avoiding the mismatch noted in the dev/staging rollout). ✅
  • New config.yaml files for kflux-osp-p01 and kflux-prd-rh03 are exact copies of production/base/tekton-kueue/config.yaml plus the three new/modified CEL guard expressions, preserving the unconditional token mechanism and priority logic these clusters previously inherited from base. ✅
  • stone-prod-p02 already had its own config.yaml override (with the conditional internal-services token exception and konflux-release resource); the diff correctly patches only the build-platforms guards and inserts the validation-error expression without disturbing the existing cluster-specific logic. ✅
  • configMapGenerator with behavior: replace in the two new cluster kustomization.yaml files follows the exact pattern already established by stone-prod-p02 and the ring-1 clusters. ✅
  • Tests: Three new test cases added for all three ring-2 configs using the multiplatform_new_string_param fixture (string-typed build-platforms). Expected outputs are correct — validation-error annotation, token, queue name, and priority all verified against the actual config logic. ✅

Ring isolation

  • Ring-3 clusters (kflux-fedora-01, kflux-rhel-p01, stone-prd-rh01) remain unaffected — no changes to production/base/tekton-kueue/config.yaml or the shared policies/kueue/ kustomization. Verified by inspecting the base config and confirming it still lacks the type guard. ✅
  • Ring-1 clusters (kflux-ocp-p01, stone-prod-p01, kflux-prd-rh02) already have the type guard and deny-on-validation-error policy — not touched by this PR. ✅

Security

  • The ValidatingAdmissionPolicy (deny-on-validation-error.kueue.konflux-ci.dev) uses failurePolicy: Fail (safe default) and the VAPB restricts scope to tenant namespaces via namespaceSelector.matchLabels: konflux-ci.dev/type: tenant. No privilege escalation or overly broad scope. ✅
  • The deny-on-validation-error resources are wired directly into the three ring-2 cluster policy kustomizations, not the shared policies/kueue/ kustomization, maintaining ring isolation as expected. ✅

Intent & coherence

Documentation

  • No in-repo documentation made stale by this change. ✅

No findings at or above the severity threshold.

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Jul 17, 2026
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.11%. Comparing base (91965a5) to head (78e07f1).
⚠️ Report is 37 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main   #13096   +/-   ##
=======================================
  Coverage   57.11%   57.11%           
=======================================
  Files          23       23           
  Lines        1455     1455           
=======================================
  Hits          831      831           
  Misses        548      548           
  Partials       76       76           
Flag Coverage Δ
go 57.11% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hugares hugares left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hugares, oswcab

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mcada

mcada commented Jul 20, 2026

Copy link
Copy Markdown

Production Approval Record

Field Value
Action APPROVED
Reviewer @mcada
Timestamp 2026-07-20T15:12:56.708Z
Risk Level low

@konflux-ci-qe-bot

konflux-ci-qe-bot commented Jul 20, 2026

Copy link
Copy Markdown

🤖 Pipeline Failure Analysis

Category: Configuration

The pipeline failed during Konflux installation because the ArgoCD application 'vector-kubearchive-log-collector-in-cluster-local' became stuck in an "Unknown" state.

📋 Technical Details

Immediate Cause

The appstudio-e2e-tests/konflux-ci-install-konflux step failed because the ArgoCD application vector-kubearchive-log-collector-in-cluster-local could not transition to a healthy and synced state, getting perpetually stuck in an "Unknown" status during the Konflux deployment process.

Contributing Factors

The failure is attributed to a configuration problem, likely within the application's deployment manifests or the specific runtime environment on the cluster, preventing the ArgoCD application from progressing. No further specific contributing factors were identified from the provided diagnostic artifacts.

Impact

The inability to install and synchronize the vector-kubearchive-log-collector-in-cluster-local application directly prevented the successful completion of the Konflux installation, leading to the overall failure of the appstudio-e2e-tests pipeline.

🔍 Evidence

appstudio-e2e-tests/konflux-ci-install-konflux

Category: configuration
Root Cause: The ArgoCD application 'vector-kubearchive-log-collector-in-cluster-local' failed to transition to a healthy and synced state, getting stuck in an "Unknown" status during the Konflux installation. This indicates a configuration problem within the application's deployment manifests or its runtime environment on the cluster.

Logs:

artifacts/appstudio-e2e-tests/konflux-ci-install-konflux/build-log.txt line 273
[WARN] Found applications in Unknown state (not Progressing), investigating...
artifacts/appstudio-e2e-tests/konflux-ci-install-konflux/build-log.txt line 274
[ERROR] Application 'vector-kubearchive-log-collector-in-cluster-local' is in Unknown state without 'context deadline exceeded'

Analysis powered by prow-failure-analysis | Build: 2079274778413240320

@oswcab

oswcab commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

/test appstudio-e2e-tests

1 similar comment
@oswcab

oswcab commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

/test appstudio-e2e-tests

@qodo-for-redhat-appstudio

qodo-for-redhat-appstudio Bot commented Jul 20, 2026

Copy link
Copy Markdown

PR-Agent: could not find a component named appstudio-e2e-tests in a supported language in this PR.

@openshift-merge-bot
openshift-merge-bot Bot merged commit d9d8aa0 into main Jul 20, 2026
79 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants