Skip to content

OCPBUGS-85680: Re-fetch infraConfig on every periodic loop iteration#1458

Open
Thealisyed wants to merge 1 commit into
openshift:masterfrom
Thealisyed:OCPBUGS-85680/refetch-infraconfig
Open

OCPBUGS-85680: Re-fetch infraConfig on every periodic loop iteration#1458
Thealisyed wants to merge 1 commit into
openshift:masterfrom
Thealisyed:OCPBUGS-85680/refetch-infraconfig

Conversation

@Thealisyed

@Thealisyed Thealisyed commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Problem

The Infrastructure config (which includes ControlPlaneTopology) is fetched once during Operator.Start() and reused on every 1 minute reconciliation tick. If the topology changes at runtime, the operator never detects it and continues using stale configuration until it is restarted.

This was identified as problematic

We follow the pattern of the Ingress config where it is already re-fetched inside the periodic loop.

Solution

  • Move the Infrastructure config fetch inside the periodic wait.Until loop, matching the pattern already used for the Ingress config
  • The ExternalTopologyMode check is now evaluated each tick, allowing the operator to respond to topology transitions without a restart
  • Add unit tests for DetermineReplicas() which had no prior test coverage

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 2, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@Thealisyed: This pull request references Jira Issue OCPBUGS-85680, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • Move the Infrastructure config fetch inside the periodic wait.Until loop in Operator.Start(), matching the pattern already used for the Ingress config
  • Previously infraConfig was fetched once at startup and reused on every 1-minute tick, so topology changes at runtime (e.g., ControlPlaneTopology transitioning to/from ExternalTopologyMode) were never detected
  • The ExternalTopologyMode check is now evaluated each tick, allowing the operator to respond to topology transitions without a restart
  • Add unit tests for DetermineReplicas() which had no test coverage

Test plan

  • go build ./pkg/operator/... compiles cleanly
  • go test ./pkg/operator/controller/ingress/ -run TestDetermineReplicas — all 5 cases pass
  • make test — full unit test suite
  • make verify — gofmt, CRD, deps checks
  • E2E: deploy operator on a cluster and verify default IngressController is created with correct replica count

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jun 2, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

This pull request adds test coverage for replica determination logic and refactors the Operator.Start method to use periodic reconciliation. The test validates DetermineReplicas behavior across different infrastructure and control plane topology combinations. The Operator.Start refactor replaces one-time infrastructure fetching with an unconditional wait.Until goroutine that continuously syncs the cache, re-fetches both Infrastructure and Ingress configurations on each iteration, and skips reconciliation for external topology mode while logging errors instead of failing the Start method.

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: moving the infrastructure config fetch inside the periodic loop to detect topology changes at runtime.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The project uses standard Go testing, not Ginkgo. The custom check for "Stable and Deterministic Ginkgo Test Names" is not applicable to this PR.
Test Structure And Quality ✅ Passed Test uses standard Go patterns matching the codebase (not Ginkgo), employs table-driven subtests with single responsibility, has appropriate assertions consistent with existing tests.
Microshift Test Compatibility ✅ Passed No Ginkgo e2e tests are added in this PR; only a standard Go unit test (replicas_test.go) is added, which is outside the scope of this MicroShift compatibility check.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No Ginkgo e2e tests added. The new test (TestDetermineReplicas) is a standard Go unit test using the testing package, not a Ginkgo e2e test. The check does not apply.
Topology-Aware Scheduling Compatibility ✅ Passed DetermineReplicas handles topology modes. Pod anti-affinity guarded by singleReplica(). Operator refetches infraConfig for topology transitions. No topology-incompatible constraints.
Ote Binary Stdout Contract ✅ Passed PR changes do not violate OTE Binary Stdout Contract. The new test file is a standard unit test with no stdout writes, and operator.go changes don't affect OTE binary output handling.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No Ginkgo e2e tests were added. The PR adds only a standard Go unit test (replicas_test.go using testing.T) in pkg/operator/controller/ingress/, not in test/e2e/ directory. Check not applicable.
No-Weak-Crypto ✅ Passed No weak cryptography patterns found. PR adds test coverage and refactors config fetching without cryptographic operations or insecure comparisons.
Container-Privileges ✅ Passed PR modifies only Go source code files (unit test and operator implementation), not container/K8s manifests or Dockerfiles. Custom check for container privileges is not applicable.
No-Sensitive-Data-In-Logs ✅ Passed Logging statements only emit generic messages and safe config fields (names, namespace, topology modes), never full objects or credentials.
Description check ✅ Passed The PR description clearly explains the problem (stale Infrastructure config), the solution (move fetch into periodic loop), and includes test additions.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from miheer and rikatz June 2, 2026 13:03
@openshift-ci

openshift-ci Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign gcs278 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Previously, Operator.Start() fetched the Infrastructure config once
at startup and reused the stale object in the periodic loop that
ensures the default IngressController exists. If topology changed at
runtime, the operator would never react.

We are moving the Infrastructure config fetch inside the loop to match the
pattern already used for the Ingress config. The ExternalTopologyMode
check is also evaluated on each tick, allowing the operator to respond
to topology transitions without requiring a restart.

Additionally we also add unit tests for DetermineReplicas() which had no
test coverage.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
@Thealisyed Thealisyed force-pushed the OCPBUGS-85680/refetch-infraconfig branch from 4916fca to 02161d5 Compare June 2, 2026 13:04
@Thealisyed

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 2, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@Thealisyed: This pull request references Jira Issue OCPBUGS-85680, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @melvinjoseph86

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot requested a review from melvinjoseph86 June 2, 2026 13:05
@Thealisyed

Copy link
Copy Markdown
Contributor Author

do a review on my PR please @coderabbitai :)

@coderabbitai

coderabbitai Bot commented Jun 2, 2026

Copy link
Copy Markdown

@Thealisyed: Sure! I'll kick off a full review of your PR now. 🚀

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
pkg/operator/operator.go (1)

389-392: 💤 Low value

Consider using log.Info or providing error context for cache sync failure.

WaitForCacheSync returns a boolean, not an error. Passing nil to log.Error is semantically unclear. Consider either using log.Info with an appropriate message, or providing context about why the cache sync failed.

Suggested alternative
 		if !o.manager.GetCache().WaitForCacheSync(ctx) {
-			log.Error(nil, "failed to sync cache before ensuring default ingresscontroller")
+			log.Info("cache sync incomplete before ensuring default ingresscontroller, will retry")
 			return
 		}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/operator/operator.go` around lines 389 - 392, The current cache sync
failure uses log.Error(nil, ...) which passes nil instead of an error; update
the block where o.manager.GetCache().WaitForCacheSync(ctx) is checked (the
WaitForCacheSync call) to either log an informational message with log.Info
describing the sync failure (e.g., "cache not synced before ensuring default
ingresscontroller") or construct a contextual error and pass it to log.Error so
the call is not given nil; ensure you update the message to reference the
operation (ensuring default ingresscontroller) for clarity and keep the return
behavior the same.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/operator/operator.go`:
- Around line 394-407: The client.Get calls inside Start() are using
context.TODO() which prevents cancellation propagation; update those calls to
use the ctx parameter passed into Start() so the fetches respect shutdown
signals—specifically replace context.TODO() in the o.client.Get calls that fetch
operatorcontroller.InfrastructureClusterConfigName() into infraConfig and
operatorcontroller.IngressClusterConfigName() into ingressConfig with the
Start() ctx variable.

---

Nitpick comments:
In `@pkg/operator/operator.go`:
- Around line 389-392: The current cache sync failure uses log.Error(nil, ...)
which passes nil instead of an error; update the block where
o.manager.GetCache().WaitForCacheSync(ctx) is checked (the WaitForCacheSync
call) to either log an informational message with log.Info describing the sync
failure (e.g., "cache not synced before ensuring default ingresscontroller") or
construct a contextual error and pass it to log.Error so the call is not given
nil; ensure you update the message to reference the operation (ensuring default
ingresscontroller) for clarity and keep the return behavior the same.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: e467154e-ec52-4b2d-8ae7-09944914a2a6

📥 Commits

Reviewing files that changed from the base of the PR and between 53b8a64 and 02161d5.

📒 Files selected for processing (2)
  • pkg/operator/controller/ingress/replicas_test.go
  • pkg/operator/operator.go

Comment thread pkg/operator/operator.go
@bentito

bentito commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

/assign @grzpiotrowski

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@Thealisyed: This pull request references Jira Issue OCPBUGS-85680, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @melvinjoseph86

Details

In response to this:

Problem

The Infrastructure config (which includes ControlPlaneTopology) is fetched once during Operator.Start() and reused on every subsequent 1-minute reconciliation tick. If the topology changes at runtime (e.g., ControlPlaneTopology transitioning to/from ExternalTopologyMode), the operator never detects it and continues using stale configuration until it is restarted.

This was identified as finding H2: "Topology configuration is fetched once and forgotten. Fetch it not only during startup, but when required."

The Ingress config does not have this problem because it is already re-fetched inside the periodic loop.

Solution

  • Move the Infrastructure config fetch inside the periodic wait.Until loop, matching the pattern already used for the Ingress config
  • The ExternalTopologyMode check is now evaluated each tick, allowing the operator to respond to topology transitions without a restart
  • Add unit tests for DetermineReplicas() which had no test coverage

Fixes: https://issues.redhat.com/browse/OCPBUGS-85680

Test plan

  • go build ./pkg/operator/... compiles cleanly
  • go test ./pkg/operator/controller/ingress/ -run TestDetermineReplicas — all 5 cases pass
  • make test — full unit test suite
  • make verify — gofmt, CRD, deps checks
  • E2E: deploy operator on a cluster and verify default IngressController is created with correct replica count

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@Thealisyed

Copy link
Copy Markdown
Contributor Author

Infra failures unrelated to our changes

/retest

@openshift-ci

openshift-ci Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

@Thealisyed: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@Thealisyed

Copy link
Copy Markdown
Contributor Author

/assign @davidesalerno

Comment thread pkg/operator/operator.go
// Periodically ensure the default controller exists.
go wait.Until(func() {
if !o.manager.GetCache().WaitForCacheSync(ctx) {
log.Error(nil, "failed to sync cache before ensuring default ingresscontroller")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that since there is no actual error, log.Info(...) is cleaner than log.Error(nil, ...).

@davidesalerno

Copy link
Copy Markdown
Contributor

The main fix looks good to me — moving infra fetch into the periodic loop correctly addresses the stale topology problem. Only one minor nit related to logging.

@melvinjoseph86

Copy link
Copy Markdown

marking as verified since it ingress unit test is passing
/verified by e2e

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 25, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@melvinjoseph86: This PR has been marked as verified by e2e.

Details

In response to this:

marking as verified since it ingress unit test is passing
/verified by e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants