Skip to content

PSHP-316: update golang and base image for Hypershift Operator CVE remediation#8823

Closed
BraeTroutman wants to merge 2 commits into
openshift:mainfrom
BraeTroutman:PSHP-316/june-cve
Closed

PSHP-316: update golang and base image for Hypershift Operator CVE remediation#8823
BraeTroutman wants to merge 2 commits into
openshift:mainfrom
BraeTroutman:PSHP-316/june-cve

Conversation

@BraeTroutman

@BraeTroutman BraeTroutman commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Updates the ubi-minimal base image used for hypershift operator and other images to make use of ubi-minimal:9.8.

Go mod version is bumped to 1.25.11. Since there is no 1.25.11 tag in the go-toolset image, build images are updated to 1.26.3, which works due to golang compilation backwards compatibility guarantees

these bumps address the below CVEs

CVE-2026-33846
CVE-2026-33845
CVE-2026-42009
CVE-2026-42010
CVE-2026-4878
CVE-2026-33811
CVE-2026-39836
CVE-2026-27145
CVE-2026-42499
CVE-2026-33814
CVE-2026-42504
CVE-2026-39820

Which issue(s) this PR fixes:

Fixes #PSHP-316

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

Summary by CodeRabbit

  • Chores
    • Updated operator and control-plane container base images to newer supported tags (builder and runtime).
    • Refreshed the Go toolchain version used for builds (including the Go toolchain specified in the build tooling).
    • Updated the CLI build stage to use a newer Go builder tag while keeping produced release artifacts the same.
    • Updated development and end-to-end container build images to a newer OpenShift/Golang combination without changing build or runtime behavior.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 24, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 24, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 24, 2026

Copy link
Copy Markdown

@BraeTroutman: This pull request references PSHP-316 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the vulnerability to target the "5.0.0" version, but no target version was set.

Details

In response to this:

addressing the below CVEs with golang minor version update, and bump to 9.8 ubi-minimal base image. ticket here

CVE-2026-33846
CVE-2026-33845
CVE-2026-42009
CVE-2026-42010
CVE-2026-4878
CVE-2026-33811
CVE-2026-39836
CVE-2026-27145
CVE-2026-42499
CVE-2026-33814
CVE-2026-42504
CVE-2026-39820

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This pull request updates the Go directive in go.mod from 1.25.7 to 1.25.11 and bumps base image tags in Containerfile.cli, Containerfile.control-plane, Containerfile.operator, Dockerfile.dev, and Dockerfile.e2e. No other build steps, runtime instructions, module dependencies, or code logic are changed.


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name Status Explanation Resolution
No-Weak-Crypto ❌ Error The diff adds weak-crypto names in vendored AWS S3 enums (ChecksumAlgorithmMd5, ChecksumAlgorithmSha1), which triggers the check. Remove or justify the vendored additions, or exempt generated/vendor dependency updates if these checksum enums are intentional and not app crypto.
✅ Passed checks (10 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: updating Go and base images for CVE remediation.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Changed Ginkgo names are static/descriptive only; no titles include dynamic values like namespaces, pods, UUIDs, or timestamps.
Test Structure And Quality ✅ Passed Only Containerfile/Dockerfile/go.mod files changed; no Ginkgo test code or test patterns were touched, so this test-quality check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed Only image/base-tag files and go.mod changed; no manifests, controllers, or scheduling constraints were modified.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No Ginkgo e2e test files were added or modified; the commit only changes build images and go.mod, so this compatibility check is not applicable.
Container-Privileges ✅ Passed Only image/toolchain tags changed in Dockerfiles/Containerfiles and go.mod; no privileged, host*, SYS_ADMIN, or allowPrivilegeEscalation settings were added.
No-Sensitive-Data-In-Logs ✅ Passed Changed files only bump image tags and go version; no new logging or secret/PII-bearing output was introduced.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.28%. Comparing base (4d087b3) to head (b528b0b).
⚠️ Report is 19 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8823      +/-   ##
==========================================
+ Coverage   43.19%   43.28%   +0.08%     
==========================================
  Files         767      771       +4     
  Lines       94910    95503     +593     
==========================================
+ Hits        40997    41335     +338     
- Misses      51050    51284     +234     
- Partials     2863     2884      +21     

see 15 files with indirect coverage changes

Flag Coverage Δ
cmd-support 36.67% <ø> (+0.23%) ⬆️
cpo-hostedcontrolplane 45.31% <ø> (ø)
cpo-other 45.10% <ø> (ø)
hypershift-operator 53.59% <ø> (+0.06%) ⬆️
other 31.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@go.mod`:
- Line 3: The Go toolchain version is now pinned to 1.25.11 in go.mod, but the
build images still reference older toolchains, so update the pinned Go version
in Containerfile.operator, Containerfile.control-plane, and
Dockerfile.github-actions-runner to 1.25.11 or newer. Make sure the changes are
consistent across these image definitions so any go build or related build step
runs with the same required toolchain version.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 532c08d2-0f91-4683-8dbd-9c194c3134aa

📥 Commits

Reviewing files that changed from the base of the PR and between bc3bda9 and 85e7081.

📒 Files selected for processing (2)
  • Containerfile.operator
  • go.mod

Comment thread go.mod Outdated
@BraeTroutman BraeTroutman force-pushed the PSHP-316/june-cve branch 2 times, most recently from ece88ee to c6e575c Compare June 24, 2026 15:00
@openshift-ci openshift-ci Bot added area/ci-tooling Indicates the PR includes changes for CI or tooling and removed do-not-merge/needs-area labels Jun 24, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Containerfile.operator`:
- Line 1: The Red Hat base image in the Containerfile is pinned to a
build-specific tag, so updates will not flow automatically. Update the FROM
reference for the builder stage, and also the runtime base image in the same
Containerfile, to use floating Red Hat tags instead of the fixed versions. Keep
the existing stage names and image sources intact while changing only the tag
values.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: bb672032-e585-4e42-bb10-1e50e0026bd5

📥 Commits

Reviewing files that changed from the base of the PR and between ece88ee and c6e575c.

⛔ Files ignored due to path filters (1)
  • hack/workspace/go.work is excluded by !**/*.work
📒 Files selected for processing (3)
  • Containerfile.control-plane
  • Containerfile.operator
  • go.mod
✅ Files skipped from review due to trivial changes (2)
  • Containerfile.control-plane
  • go.mod

Comment thread Containerfile.operator Outdated
@BraeTroutman BraeTroutman marked this pull request as ready for review June 24, 2026 16:09
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 24, 2026
@openshift-ci openshift-ci Bot requested review from Nirshal and clebs June 24, 2026 16:11

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Containerfile.control-plane`:
- Line 1: The builder and runtime base images are pinned to specific Red Hat
build tags, which blocks automatic Red Hat security updates. Update the
Containerfile.control-plane FROM directives for both the builder stage and the
ubi-minimal runtime stage to use floating Red Hat tags instead of build-specific
version suffixes, following the Red Hat image policy while keeping non-Red Hat
images pinned by digest.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 2294a0aa-a572-47a5-9ff5-2501443bd361

📥 Commits

Reviewing files that changed from the base of the PR and between f7f4c3f and 5d772c6.

⛔ Files ignored due to path filters (1)
  • hack/workspace/go.work is excluded by !**/*.work
📒 Files selected for processing (6)
  • Containerfile.cli
  • Containerfile.control-plane
  • Containerfile.operator
  • Dockerfile.dev
  • Dockerfile.e2e
  • go.mod
✅ Files skipped from review due to trivial changes (1)
  • Dockerfile.dev
🚧 Files skipped from review as they are similar to previous changes (1)
  • go.mod

Comment thread Containerfile.control-plane Outdated
@@ -1,4 +1,4 @@
FROM registry.access.redhat.com/ubi9/go-toolset:1.25.9-1778054913 AS builder
FROM registry.access.redhat.com/ubi9/go-toolset:1.26.3-1782305929 AS builder

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Use floating Red Hat base image tags.

Both the go-toolset builder (Line 1) and the ubi-minimal runtime (Line 11) are pinned to build-specific tags (1.26.3-1782305929, 9.8-1782191395). Pinning these Red Hat bases prevents Red Hat-managed security updates from flowing automatically. Use floating Red Hat tags instead.

🔒 Proposed change
-FROM registry.access.redhat.com/ubi9/go-toolset:1.26.3-1782305929 AS builder
+FROM registry.access.redhat.com/ubi9/go-toolset:1.26 AS builder
-FROM registry.access.redhat.com/ubi9/ubi-minimal:9.8-1782191395
+FROM registry.access.redhat.com/ubi9/ubi-minimal:9.8
As per path instructions: "Red Hat images: use floating tags (Red Hat manages updates); non-RH images: pin by digest".

Also applies to: 11-11

🧰 Tools
🪛 Trivy (0.69.3)

[error] 1-1: Image user should not be 'root'

Specify at least 1 USER command in Dockerfile with non-root user as argument

Rule: DS-0002

Learn more

(IaC/Dockerfile)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Containerfile.control-plane` at line 1, The builder and runtime base images
are pinned to specific Red Hat build tags, which blocks automatic Red Hat
security updates. Update the Containerfile.control-plane FROM directives for
both the builder stage and the ubi-minimal runtime stage to use floating Red Hat
tags instead of build-specific version suffixes, following the Red Hat image
policy while keeping non-Red Hat images pinned by digest.

Source: Path instructions

@BraeTroutman BraeTroutman force-pushed the PSHP-316/june-cve branch 3 times, most recently from 52f66b5 to a7c96d9 Compare June 24, 2026 17:30
@openshift-ci openshift-ci Bot added the area/api Indicates the PR includes changes for the API label Jun 24, 2026
@openshift-ci

openshift-ci Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

@BraeTroutman: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@BraeTroutman BraeTroutman marked this pull request as draft June 24, 2026 19:04
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 24, 2026
@clebs

clebs commented Jun 26, 2026

Copy link
Copy Markdown
Member

/approve

@openshift-ci

openshift-ci Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: BraeTroutman, clebs
Once this PR has been reviewed and has the lgtm label, please assign csrwng for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

golangci lint must be built on the same or higher version of golang
as the project it is linting: update the github actions
to use existing tools if their version matches, otherwise build fresh
to ensure compatibility. The golang version of the linter image is also
updated to the `1.26.4`
@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Now I have a complete understanding of the root cause. Let me produce the final report.

Test Failure Analysis Complete

Job Information

  • Prow Job: Red Hat Konflux / hypershift-cli-mce-50-on-pull-request
  • Build ID: hypershift-cli-mce-50-on-pull-request-wb5z8
  • Pipeline Run: hypershift-cli-mce-50-on-pull-request-wb5z8
  • Namespace: crt-redhat-acm-tenant
  • Failed Step: build-images (1 minute, vs ~7+ minutes for a successful build)
  • PR: #8823PSHP-316: update golang and base image for Hypershift Operator CVE remediation

Test Failure Analysis

Error

build-images step failed after ~1 minute in hermetic Konflux pipeline.
Builder image brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_golang_1.26
ships Go 1.26.3 (or earlier patch), but go.mod requires go 1.26.4.
Go's GOTOOLCHAIN=auto (the default) attempts to download go1.26.4 from the internet,
which fails because the build runs in hermetic mode (network-isolated).

Summary

The build-images step in the Konflux hermetic pipeline fails because of a Go toolchain version mismatch. PR #8823 sets go 1.26.4 in go.mod but uses brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_golang_1.26 as the builder image in Containerfile.cli. This floating image tag currently ships Go 1.26.3 (confirmed by stackrox/stackrox successfully using the same image digest with go 1.26.3 in their go.mod). Since Go 1.21, the go directive in go.mod acts as a minimum toolchain version — when the installed Go (1.26.3) is older than the required version (1.26.4), Go's default GOTOOLCHAIN=auto behavior tries to download the newer toolchain from the internet. In the hermetic (network-isolated) Konflux build, this download fails. The sibling Konflux builds (hypershift-operator-main and hypershift-release-mce-50) succeed because their Containerfile.operator uses registry.access.redhat.com/ubi9/go-toolset:1.26.4-1782736563, which ships exactly Go 1.26.4.

Root Cause

Go toolchain version mismatch in hermetic build: builder image ships Go 1.26.3, but go.mod requires Go 1.26.4.

The failure chain is:

  1. PR PSHP-316: update golang and base image for Hypershift Operator CVE remediation #8823 bumps go.mod from go 1.25.7 to go 1.26.4 and updates Containerfile.cli to use brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_golang_1.26.

  2. The image tag rhel_9_golang_1.26 is a floating tag (no patch version pinned). It currently resolves to a build containing Go 1.26.3 — not 1.26.4. Evidence: stackrox/stackrox uses the same image (pinned by digest sha256:8bca01ace...) with go 1.26.3 in their go.mod and their builds succeed.

  3. Since Go 1.21, the go directive in go.mod acts as a minimum required toolchain version. When the installed Go version (1.26.3) is lower than the required version (1.26.4), Go's default GOTOOLCHAIN=auto behavior attempts to download the matching toolchain from proxy.golang.org.

  4. The Konflux pipeline runs with hermetic: "true" — all network access is blocked during the build step. The toolchain download request fails immediately.

  5. No GOTOOLCHAIN=local is set in the Makefile or environment, and no toolchain directive exists in go.mod to override this behavior.

  6. The build fails within ~1 minute (compared to the ~7+ minutes for the successful operator build), consistent with an immediate network/version failure rather than a compilation error.

Why other Konflux builds on the same PR succeed:

  • hypershift-release-mce-50 and hypershift-operator-main use Containerfile.operator, which references registry.access.redhat.com/ubi9/go-toolset:1.26.4-1782736563 — this image ships exactly Go 1.26.4, matching the go.mod requirement. No toolchain download is needed.
Recommendations
  1. Align go.mod with the builder image's Go version (quickest fix): Downgrade go.mod to go 1.26.3 to match the Go version currently shipped in openshift-golang-builder:rhel_9_golang_1.26. This avoids the toolchain auto-download entirely.

  2. Pin the builder image by digest with a Go 1.26.4 build: Once the openshift-golang-builder:rhel_9_golang_1.26 image is rebuilt with Go 1.26.4, update Containerfile.cli to pin the digest (like stackrox does: @sha256:...). This prevents future floating-tag drift.

  3. Add GOTOOLCHAIN=local to the Makefile (defensive fix): Add GOTOOLCHAIN=local to the GO variable in the Makefile: GO=GO111MODULE=on GOWORK=off GOFLAGS=-mod=vendor GOTOOLCHAIN=local go. This prevents Go from attempting toolchain downloads in hermetic environments. Note: this will cause a clear error message ("go: go.mod requires go >= 1.26.4") instead of a confusing network timeout if the versions don't match.

  4. Request a rebuild of the openshift-golang-builder image: File a request with the ART/OSBS team to rebuild openshift-golang-builder:rhel_9_golang_1.26 with Go 1.26.4, so the floating tag resolves to the correct patch version.

  5. Switch Containerfile.cli to use go-toolset with an exact version tag (like the other Containerfiles): FROM registry.access.redhat.com/ubi9/go-toolset:1.26.4-1782736563 AS builder. This is the pattern that already works for the operator and release builds.

Evidence
Evidence Detail
Failing builder image brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_golang_1.26 (floating tag, ships Go ≤1.26.3)
Succeeding builder image registry.access.redhat.com/ubi9/go-toolset:1.26.4-1782736563 (exact pin, ships Go 1.26.4)
go.mod requirement go 1.26.4 (patch version higher than builder image)
Hermetic mode hermetic: "true" in .tekton/hypershift-cli-mce-50-pull-request.yaml — network blocked during build
GOTOOLCHAIN setting Not set (defaults to auto → attempts toolchain download)
Build duration (failure) ~1 minute for build-images step; 13 min total pipeline (vs 19–20 min for successful builds)
Sibling build results hypershift-operator-main-on-pull-request: ✅ SUCCESS; hypershift-release-mce-50-on-pull-request: ✅ SUCCESS; hypershift-cli-mce-50-on-pull-request: ❌ FAILURE
StackRox reference Uses same image digest sha256:8bca01ace... with go 1.26.3 in go.mod → builds succeed, confirming image ships Go 1.26.3
Main branch (pre-PR) go 1.25.7 + openshift-golang-builder:rhel_9_golang_1.25 → SUCCESS (versions match)
No code changes PR only modifies go.mod versions, Dockerfiles/Containerfiles, and CI config — no Go source changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api Indicates the PR includes changes for the API area/ci-tooling Indicates the PR includes changes for CI or tooling do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants