Skip to content

NE-2739: Migrate operator to native metrics protection with centralized TLS#313

Open
bentito wants to merge 8 commits into
openshift:mainfrom
bentito:feature/NE-2739-native-metrics
Open

NE-2739: Migrate operator to native metrics protection with centralized TLS#313
bentito wants to merge 8 commits into
openshift:mainfrom
bentito:feature/NE-2739-native-metrics

Conversation

@bentito

@bentito bentito commented Jul 15, 2026

Copy link
Copy Markdown

Migrates the aws-load-balancer-operator from utilizing kube-rbac-proxy sidecars to the native secure serving metrics server introduced in newer versions of controller-runtime.

Key Changes:

  • main.go: Integrated secure serving on port :8443 using the native auth and authz filters. Added startup resolution of the OpenShift APIServer configuration named cluster to parse and apply the centralized TLSSecurityProfile (MinVersion, CipherSuites, and elliptic curves/groups, including post-quantum X25519MLKEM768). Restressed HTTP/2 by disabling it on the metrics server to match External DNS Operator (EDO) PR 502's security posture.
  • config/: Deleted kube-rbac-proxy patches, updated manager deployment configuration to expose port 8443 directly, and commented out the auth proxy references in default Kustomizations.
  • bundle/ & vendor/: Regenerated the OLM bundle using make bundle and validated it successfully with operator-sdk. Resolved the vendored dependencies and updated the test CRD fixtures to keep test pipelines completely green.

Fully validated and verified locally using make verify and make lint with zero issues found.

@openshift-ci-robot

openshift-ci-robot commented Jul 15, 2026

Copy link
Copy Markdown

@bentito: This pull request references NE-2739 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Migrates the aws-load-balancer-operator from utilizing kube-rbac-proxy sidecars to the native secure serving metrics server introduced in newer versions of controller-runtime.

Key Changes:

  • main.go: Integrated secure serving on port :8443 using the native auth and authz filters. Added startup resolution of the OpenShift APIServer configuration named cluster to parse and apply the centralized TLSSecurityProfile (MinVersion, CipherSuites, and elliptic curves/groups, including post-quantum X25519MLKEM768). Restressed HTTP/2 by disabling it on the metrics server to match External DNS Operator (EDO) PR 502's security posture.
  • config/: Deleted kube-rbac-proxy patches, updated manager deployment configuration to expose port 8443 directly, and commented out the auth proxy references in default Kustomizations.
  • bundle/ & vendor/: Regenerated the OLM bundle using make bundle and validated it successfully with operator-sdk. Resolved the vendored dependencies and updated the test CRD fixtures to keep test pipelines completely green.

Fully validated and verified locally using make verify and make lint with zero issues found.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 15, 2026
@openshift-ci

openshift-ci Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign miciah for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@bentito, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 8 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7d0644de-73dd-4270-b714-27f9cb39a187

📥 Commits

Reviewing files that changed from the base of the PR and between 5b48f30 and 5750ba9.

📒 Files selected for processing (2)
  • hack/generate-iam-from-credrequest.sh
  • main.go
📝 Walkthrough

Walkthrough

The change moves controller metrics to HTTPS on port 8443 using the cluster TLS security profile, removes the metrics authentication proxy, and updates manifests. It adds ALB deregistration documentation and a demonstration script, refreshes Go dependencies, and expands Infrastructure schema validation and cloud load balancer configuration for AWS, Azure, and GCP.

Suggested reviewers: rfredette, grzpiotrowski


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The new demo script echoes cluster context/node names and dumps raw kubectl YAML/logs, which can expose internal hostnames or other cluster data. Redact or avoid raw kubectl logs/YAML and sanitize echoed cluster identifiers; only print minimal, non-sensitive status messages.
Docstring Coverage ⚠️ Warning Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (13 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: moving metrics protection to native serving with centralized TLS.
Description check ✅ Passed The description clearly matches the changeset and objectives, covering native metrics serving, TLS config, and config updates.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The patch only changes docs/manifests; no *_test.go or Ginkgo titles are present in the changed files.
Test Structure And Quality ✅ Passed No Ginkgo test code changed in the commit; only manifests/policy files were modified, so the checklist is not applicable.
Microshift Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; the only non-vendor Go change is main.go, with no It/Describe/Context/When declarations.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo e2e tests were added in the diff; only docs/config/script changes appear, so SNO multi-node compatibility is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The commit only removes kube-rbac-proxy patch lines, fixes a Kustomize index, and renames IAM JSON keys; no node selectors, affinity, replicas, or topology logic changed.
Ote Binary Stdout Contract ✅ Passed main.go is the only changed process-level binary entrypoint, and it has no stdout prints; its messages use controller-runtime logging instead.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new *_test.go/Ginkgo e2e tests were added in this PR, so there are no IPv4 or external-connectivity test assumptions to flag.
No-Weak-Crypto ✅ Passed No weak crypto or secret comparisons found; main.go uses OpenShift TLS profiles and SecureTLSConfig, with no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB usage.
Container-Privileges ✅ Passed Changed deployment manifests keep runAsNonRoot true and allowPrivilegeEscalation false, with no hostNetwork/hostPID/hostIPC/privileged/SYS_ADMIN settings.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (4)
main.go (2)

351-361: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider logging dropped/unsupported TLS groups.

Groups without a Go CurveID mapping are silently skipped. A debug/info log when a configured TLSGroup has no equivalent would make misconfigurations easier to diagnose.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@main.go` around lines 351 - 361, Update the TLS group processing around
TLSGroupToCurveID to log each configured group that lacks a CurveID mapping
before skipping it. Use the existing logging mechanism and include the
unsupported group value, while preserving the current behavior for supported
groups and empty curve lists.

301-364: 🩺 Stability & Availability | 🔵 Trivial

TLS profile is resolved once at startup; cluster profile changes require a pod restart to take effect.

There's no watch/reconcile loop reacting to updates on the APIServer object's TLSSecurityProfile, so an admin changing the cluster TLS profile won't be picked up by the running metrics server until the operator pod restarts. This may be an acceptable initial tradeoff, but worth confirming it's the intended behavior for this release given other OpenShift components implement a watcher for exactly this scenario.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@main.go` around lines 301 - 364, Confirm and preserve the intended
startup-only behavior for TLS profile resolution in getTLSSecurityProfile and
getTLSConfigFromProfile, or add a watch/reconcile path that refreshes the
metrics server’s TLS configuration when the APIServer object’s
TLSSecurityProfile changes. Ensure runtime profile updates take effect without
requiring an operator pod restart if dynamic behavior is required for this
release.
demonstrate-deregistration-gap.sh (2)

68-92: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

diagnose_failure is defined but never invoked.

The detailed diagnostics helper (CR status, CredentialsRequest, deployments/pods, manager logs) would be most useful right when the script falls back to SIMULATION_MODE=true, but nothing calls it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@demonstrate-deregistration-gap.sh` around lines 68 - 92, Invoke
diagnose_failure at the point where the script falls back to
SIMULATION_MODE=true, before or alongside the fallback message. Keep the
existing diagnostic collection and simulation behavior unchanged, ensuring the
CR, CredentialsRequest, workload, and manager log checks run automatically when
the fallback occurs.

1-14: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing project copyright header on all three new files. Contribution guidelines require new files to carry the project's copyright template at the top, updated to the current year; none of these new additions include one.

  • demonstrate-deregistration-gap.sh#L1-L14: add the copyright header comment block before the shebang/comment banner.
  • OCPBUGS-97941-analysis.md#L1-L2: add the copyright header before the # Technical Analysis... title.
  • OCPBUGS-97941-short-analysis.md#L1-L2: add the copyright header before the # OCPBUGS-97941... title.

Based on learnings, "New files must include the project copyright message template at the top as a comment, with the year updated."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@demonstrate-deregistration-gap.sh` around lines 1 - 14, Add the project’s
standard copyright header, updated to the current year, at the top of
demonstrate-deregistration-gap.sh before the shebang/comment banner, and at the
top of OCPBUGS-97941-analysis.md and OCPBUGS-97941-short-analysis.md before
their titles.

Source: Learnings

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@go.mod`:
- Line 286: Update the google.golang.org/grpc dependency in go.mod from v1.72.2
to the patched v1.79.3 or later release, then regenerate go.sum so the module
checksums match the upgraded dependency.
- Around line 258-262: Upgrade the OpenTelemetry dependency set represented by
the go.mod entries for otel, exporters, metric, and sdk to a consistent current
version that includes the fixes from otel v1.41.0 and sdk v1.40.0. Regenerate
the Go module graph so go.mod and go.sum reflect the coordinated updates.

In `@main.go`:
- Around line 128-134: Replace context.TODO() in the startup call to
getTLSSecurityProfile with a bounded context.Context, creating and canceling a
startup timeout context before the fetch. Pass that context through
getTLSSecurityProfile so its client.Get call observes the deadline, while
preserving the existing error logging and exit behavior.

---

Nitpick comments:
In `@demonstrate-deregistration-gap.sh`:
- Around line 68-92: Invoke diagnose_failure at the point where the script falls
back to SIMULATION_MODE=true, before or alongside the fallback message. Keep the
existing diagnostic collection and simulation behavior unchanged, ensuring the
CR, CredentialsRequest, workload, and manager log checks run automatically when
the fallback occurs.
- Around line 1-14: Add the project’s standard copyright header, updated to the
current year, at the top of demonstrate-deregistration-gap.sh before the
shebang/comment banner, and at the top of OCPBUGS-97941-analysis.md and
OCPBUGS-97941-short-analysis.md before their titles.

In `@main.go`:
- Around line 351-361: Update the TLS group processing around TLSGroupToCurveID
to log each configured group that lacks a CurveID mapping before skipping it.
Use the existing logging mechanism and include the unsupported group value,
while preserving the current behavior for supported groups and empty curve
lists.
- Around line 301-364: Confirm and preserve the intended startup-only behavior
for TLS profile resolution in getTLSSecurityProfile and getTLSConfigFromProfile,
or add a watch/reconcile path that refreshes the metrics server’s TLS
configuration when the APIServer object’s TLSSecurityProfile changes. Ensure
runtime profile updates take effect without requiring an operator pod restart if
dynamic behavior is required for this release.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

Comment thread go.mod
Comment thread go.mod
Comment thread main.go
@bentito

bentito commented Jul 15, 2026

Copy link
Copy Markdown
Author

/jira refresh

@openshift-ci-robot

openshift-ci-robot commented Jul 15, 2026

Copy link
Copy Markdown

@bentito: This pull request references NE-2739 which is a valid jira issue.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@config/manifests/kustomization.yaml`:
- Line 22: Update the volumeMount removal patch in the kustomization
configuration to target the cert mount rather than index 0, which still refers
to bound-sa-token; identify the mount by name where supported, or reorder the
patch operations so the cert mount is removed reliably.

In `@hack/operator-permission-policy.json`:
- Around line 5-24: Update the IAM policy entries shown so the keys use AWS’s
canonical casing: change every action, effect, and resource key to Action,
Effect, and Resource, preserving all existing values and permissions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 524ec681-e2d8-4faa-b402-836bc2c4c086

📥 Commits

Reviewing files that changed from the base of the PR and between f7530de and 5b48f30.

📒 Files selected for processing (3)
  • config/default/manager_webhook_patch.yaml
  • config/manifests/kustomization.yaml
  • hack/operator-permission-policy.json
💤 Files with no reviewable changes (1)
  • config/default/manager_webhook_patch.yaml

Comment thread config/manifests/kustomization.yaml
Comment thread hack/operator-permission-policy.json Outdated
@bentito

bentito commented Jul 15, 2026

Copy link
Copy Markdown
Author

Unsupported TLS groups logging added and accidental files removed in commit 26970fc.

@bentito

bentito commented Jul 15, 2026

Copy link
Copy Markdown
Author

/assign @alebedev87
This is pretty similar to your work on EDO openshift/external-dns-operator#502 so hoping you could review this. Thanks!

@openshift-ci

openshift-ci Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@bentito: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-operator 5750ba9 link true /test e2e-aws-operator
ci/prow/e2e-aws-rosa-operator 5750ba9 link true /test e2e-aws-rosa-operator

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants