fix(kubernetes): eliminate Critical/High container CVEs (slim runtime base, Go 1.25, otel 1.43, Dependabot docker)#966
Open
dvaldivia wants to merge 4 commits into
Conversation
… Go bumps
The controller/operator image (kubernetes/Dockerfile) shipped the full
`golang:1.24` build toolchain image as its *runtime* stage, dragging in the
entire Debian 13 userland (python3.13, perl, gnutls), the build toolchain
(binutils, gcc, git, gnupg, libc6-dev) and kernel headers (linux-libc-dev).
Trivy flagged ~340 Critical/High CVEs against it; the same Dockerfile also
builds the task-executor image, which carried the identical surface.
Four changes fix it:
1. Runtime base golang:1.24 -> debian:13-slim (digest-pinned) installing
ONLY util-linux-extra (nsenter; it left util-linux in trixie) and
ca-certificates, with `apt-get upgrade` for security patches and a
`command -v nsenter` build assertion so sidecar task execution can't
silently break.
2. Builder golang:1.24 -> golang:1.25 (digest-pinned). The binary was built
with Go stdlib 1.24.13, which is EOL for the stdlib CVEs (fixed only in
1.25.10+).
3. otel cluster -> v1.43.0 (go.mod/go.sum); grpc resolved to v1.80.0. This
moves the go.mod directive to go 1.25.0.
4. Dockerfile.image-committer builder golang:1.24-alpine -> golang:1.25-alpine
(digest-pinned) so it can still compile the now-go-1.25.0 module; its
runtime alpine:3.19 -> alpine:3.21 to match execd/ingress.
Result on the scanned image: Critical 19 -> 2, High 321 -> 7. The 9 residuals
have no upstream fix yet and sit on essential base packages (perl-base,
libcap2, ncurses); they will clear automatically on rebuild once Debian ships
patches.
CRITICAL fixed (17):
- CVE-2026-33186 grpc -> rebuild (go.mod grpc v1.80.0)
- CVE-2026-31789 openssl/libssl3 -> apt upgrade to 3.5.5-1~deb13u2
- CVE-2026-23112 linux-libc-dev -> removed (kernel headers not in runtime)
- CVE-2026-7210 python3.13 (x4) -> removed (no python in slim runtime)
- CVE-2026-8376 perl/-modules -> removed (full perl not in slim runtime)
- CVE-2026-42496 perl/-modules -> removed (full perl not in slim runtime)
- CVE-2026-33845 libgnutls30t64 -> removed (gnutls not in slim runtime)
- CVE-2026-42010 libgnutls30t64 -> removed (gnutls not in slim runtime)
HIGH fixed (314):
- 12 Go stdlib (built w/ 1.24.13) -> golang:1.25 builder (>=1.25.10):
CVE-2026-25679, -32280, -32281, -32283, -33811, -33814, -39820, -39823,
-39825, -39826, -39836, -42499
- 2 otel/sdk -> v1.43.0: CVE-2026-24051 (fix 1.40.0), CVE-2026-39883 (1.43.0)
- 12 openssl family -> apt upgrade 3.5.5-1~deb13u2: CVE-2026-28387, -28388,
-28389, -28390
- 288 toolchain/trixie packages removed with the slim base, dominated by
linux-libc-dev (239 kernel-header CVEs), plus python3.13, full perl,
gnutls, curl/krb5, gnupg, binutils, openssh-client, libexpat1, libncursesw6
RESIDUAL (no upstream fix; monitor):
- perl-base: CVE-2026-8376, CVE-2026-42496 (Critical);
CVE-2026-42497, CVE-2026-48962, CVE-2026-9538 (High)
- libcap2: CVE-2026-4878 (High)
- ncurses (libtinfo6/ncurses-base/ncurses-bin): CVE-2025-69720 (High)
Verified: `go build ./...`, `go mod verify`, and docker build of the
controller (uid 65532), task-executor (uid 0) and image-committer images;
nsenter + /bin/sh confirmed present in the built runtime images.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stale container base images were the root cause of the bulk of the
Critical/High OS-package CVEs Trivy found: Go modules were being bumped by
Dependabot, but no `docker` ecosystem was configured, so base images were
never refreshed and images were rarely rebuilt (the scanned binary still
embedded grpc v1.68.1 while go.mod was already at v1.79.3).
- Add .github/dependabot.yml (previously UI-only, not in-tree). Covers
gomod, pip and npm to preserve existing coverage, adds github-actions,
and — critically — adds the `docker` ecosystem across every Dockerfile
directory so base images stay patched. Updates are grouped to keep PR
volume manageable.
- Pin components/execd and components/ingress from the floating
`alpine:latest` to `alpine:3.21@sha256:...` for reproducible builds and
so Dependabot can track them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dvaldivia
requested review from
Generalwin,
Pangjiping,
Spground,
fengcone,
hittyt,
jwx0925,
kevinlynx and
ninan-nn
as code owners
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Updates the Kubernetes module and container build setup to newer Go/dependencies and improves supply-chain/security posture by pinning base images and adding in-repo Dependabot configuration.
Changes:
- Bump Go directive and several Go module dependencies in
kubernetes(including OpenTelemetry, grpc-gateway, andgolang.org/x/*). - Pin container base images by tag+digest and move the Kubernetes runtime image to a slimmer Debian base with explicit runtime deps.
- Add a repository-owned Dependabot config including Docker ecosystem updates.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| kubernetes/go.mod | Updates Go version directive and bumps indirect dependencies. |
| kubernetes/go.sum | Refreshes module checksums to match updated dependency graph. |
| kubernetes/Dockerfile.image-committer | Moves builder to Go 1.25 Alpine and pins builder/runtime images by digest. |
| kubernetes/Dockerfile | Pins Go builder by digest; switches runtime to debian:13-slim with explicit deps and health check for nsenter/sh. |
| components/ingress/Dockerfile | Pins runtime Alpine base image by tag+digest. |
| components/execd/Dockerfile | Pins runtime Alpine base image by tag+digest. |
| .github/dependabot.yml | Adds Dependabot configuration (gomod, docker, pip, npm, actions) with grouping. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c70c7350e4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
kubernetes/Dockerfile:
- Pin the builder to an explicit patch tag golang:1.25.10 (the digest already
resolves to go1.25.10), so the tag, digest and the ">=1.25.10" comment are
consistent. (Copilot)
- Drop `apt-get upgrade` for reproducible builds. The pinned debian:13-slim
digest already ships libssl3t64 3.5.6-1~deb13u1 (newer than the
3.5.5-1~deb13u2 OpenSSL fix), so the OpenSSL CVEs stay fixed via the base
digest; OS patches now come from Dependabot bumping the pinned base rather
than a non-deterministic upgrade. Verified the rebuilt image still carries
libssl3t64 3.5.6 + nsenter, and the Critical/High residual is unchanged.
(Copilot)
.github/dependabot.yml:
- Switch the Python block from pip to the uv ecosystem; all six dirs pin deps
with uv.lock (no requirements.txt), which pip would not maintain. (Codex)
- Drop the overlapping npm child dirs. /sdks is a pnpm workspace root whose
pnpm-lock.yaml already covers sdks/{code-interpreter,sandbox}/javascript, so
listing them alongside /sdks violated Dependabot's non-overlap rule. (Codex)
- Keep the plural `directories:` key (GA-supported since 2024). (Copilot)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pangjiping
added
component/k8s
The go directive in kubernetes/go.mod was raised to 1.25.0 (otel v1.43.0
requires it), but the kubernetes CI still set up Go 1.24 with GOTOOLCHAIN=local:
- Controller E2E jobs failed at `make manifests`:
go.mod requires go >= 1.25.0 (running go 1.24.13; GOTOOLCHAIN=local)
- the `test` job failed at `make lint`: golangci-lint (installed via
`go install`) was built with go1.24, lower than the targeted 1.25.0
Bump actions/setup-go to 1.25 in kubernetes-test.yml (E2E matrix GO_VERSION +
the test job) and in kubernetes-nightly-build.yml. golangci-lint is then
rebuilt with 1.25, satisfying its build-version guard. Verified locally on
Go >=1.25: make manifests, make lint (0 issues), make build all pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A Trivy scan of the Kubernetes controller/operator image surfaced ~340 Critical/High CVEs. The root cause is that
kubernetes/Dockerfileuses the fullgolang:1.24build-toolchain image as its runtime stage (originally chosen fornsenter/shell). That ships the entire Debian 13 userland (python3.13,perl,gnutls), the build toolchain (binutils,gcc,git,gnupg,libc6-dev) and kernel headers (linux-libc-dev) in the runtime. The sameDockerfilealso builds thetask-executorimage, so the surface is duplicated across two control-plane images.This PR slims the runtime, refreshes the Go toolchain/dependencies, and adds the missing prevention layer.
Changes
fix(kubernetes)golang:1.24→debian:13-slim(digest-pinned), installing onlyutil-linux-extra(fornsenter— it moved out ofutil-linuxin trixie) andca-certificates, withapt-get upgradefor security patches and acommand -v nsenterbuild assertion so sidecar task execution can't silently break.distrolessis unusable here because the task-executor needs bothnsenterand/bin/sh.golang:1.24→golang:1.25(digest-pinned). The binary was built with Go stdlib1.24.13, which is EOL for the stdlib CVEs (fixed only in1.25.10+).v1.43.0,grpcresolved tov1.80.0(go.mod/go.sum);godirective moves to1.25.0.Dockerfile.image-committerbuilder →golang:1.25-alpine(to match thego 1.25.0module), runtimealpine:3.19→alpine:3.21.chore(ci).github/dependabot.ymlcoveringgomod,pip,npm,github-actionsand — the missing piece — thedockerecosystem across every Dockerfile directory. Stale base images were never refreshed (the scanned binary still embeddedgrpc v1.68.1whilego.modwas already onv1.79.3), which is how the OS-package CVEs accumulated.components/execdandcomponents/ingressfrom floatingalpine:latesttoalpine:3.21@sha256:….CVE impact (scanned controller image)
apt upgrade(3.5.5-1~deb13u2); the no-fix Criticals onlinux-libc-dev,python3.13, fullperlandgnutlsare removed with the slim runtime.stdlib(Go 1.25 builder), 2otel/sdk(v1.43.0), 12 OpenSSL (apt upgrade), and 288 toolchain/trixie packages removed — dominated bylinux-libc-dev(239 kernel-header CVEs).Residual (no upstream fix yet — monitored, will clear on rebuild once Debian ships patches)
perl-base: CVE-2026-8376, CVE-2026-42496 (Critical); CVE-2026-42497, CVE-2026-48962, CVE-2026-9538 (High)libcap2: CVE-2026-4878 (High)ncurses(libtinfo6/ncurses-base/ncurses-bin): CVE-2025-69720 (High)Verification
go build ./...andgo mod verifypass inkubernetes/.docker buildof the controller (uid 65532), task-executor (uid 0) and image-committer images succeeds;nsenter+/bin/shconfirmed present in the built runtime images.linux/amd64to confirm the slimmed bases assemble cleanly.Notes
USERIDbuild-args, same Debian-13 family.