Skip to content

OCPBUGS-77056: Make external cert validation asynchronous#745

Open
bentito wants to merge 7 commits into
openshift:masterfrom
bentito:OCPBUGS-77056-async-sar
Open

OCPBUGS-77056: Make external cert validation asynchronous#745
bentito wants to merge 7 commits into
openshift:masterfrom
bentito:OCPBUGS-77056-async-sar

Conversation

@bentito

@bentito bentito commented Mar 3, 2026

Copy link
Copy Markdown
Contributor

This PR implements asynchronous and cached SubjectAccessReview checks for spec.tls.externalCertificate during router startup.

Previously, each external certificate triggered synchronous API validations that blocked the main thread. When scaling to thousands of external cert routes, the router hit O(N * API_latency) delays and would hit CrashLoopBackOff from failing liveness probes.

This fix maintains full RBAC security checks but executes them asynchronously in the background. A global sync.Map acts as an asyncSARCache to drastically speed up repeated checks for the same underlying secret.

Fixes: OCPBUGS-77056

Summary by CodeRabbit

  • New Features

    • Cache-backed, semaphore-throttled validation for external certificates with configurable concurrent SAR checks.
    • Configurable parallelism for route status writer lease workers.
  • Improvements

    • Hybrid secret informer strategy (faster where allowed; more selective when listing is forbidden).
    • Higher Kubernetes client request throughput for router startup.
    • Architecture-aware debug image builds and more robust route condition/status comparisons.
  • Bug Fixes

    • Preserve “ignored” admission condition details and record completion of external-certificate access checks.
  • Tests

    • Updated and added unit tests for secret/SAR caching behavior and status logic.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 3, 2026
@openshift-ci

openshift-ci Bot commented Mar 3, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Mar 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@bentito: This pull request references Jira Issue OCPBUGS-77056, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lihongan

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This PR implements asynchronous and cached SubjectAccessReview checks for spec.tls.externalCertificate during router startup.

Previously, each external certificate triggered synchronous API validations that blocked the main thread. When scaling to thousands of external cert routes, the router hit O(N * API_latency) delays and would hit CrashLoopBackOff from failing liveness probes.

This fix maintains full RBAC security checks but executes them asynchronously in the background. A global sync.Map acts as an asyncSARCache to drastically speed up repeated checks for the same underlying secret.

Fixes: OCPBUGS-77056

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 3, 2026
@openshift-ci
openshift-ci Bot requested a review from lihongan March 3, 2026 23:38
@bentito bentito changed the title WIP: OCPBUGS-77056: Make external cert validation asynchronous OCPBUGS-77056: Make external cert validation asynchronous Mar 3, 2026
@bentito
bentito force-pushed the OCPBUGS-77056-async-sar branch from 325f313 to 214b603 Compare March 3, 2026 23:41
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 3, 2026
@bentito
bentito marked this pull request as ready for review March 3, 2026 23:41
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 3, 2026
@bentito

bentito commented Mar 3, 2026

Copy link
Copy Markdown
Contributor Author

/hold

@openshift-ci
openshift-ci Bot requested review from Miciah and grzpiotrowski March 3, 2026 23:42
@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 3, 2026
@coderabbitai

coderabbitai Bot commented Mar 9, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Semaphore-throttled, cached SAR validation for external TLS; new RBAC-aware SharedSecretManager with hybrid informer strategy; SAR-completed route status events and condition equality fixes; WriterLease gains worker concurrency; synchronous test refactors and small build/client wiring tweaks.

Changes

External Certificate Validation, Secret Management, and Route Status

Layer / File(s) Summary
SAR validation with caching and throttling
pkg/router/routeapihelpers/validation.go, pkg/router/routeapihelpers/validation_test.go
Reimplement ValidateTLSExternalCertificate as synchronous, semaphore-throttled, short-TTL-cached SubjectAccessReview checks (get, watch, list); simplify ExtendedValidateRoute to use tls.X509KeyPair directly; remove internal PEM validation helper; add MaxConcurrentSARChecks, cache invalidation, and test helpers; adjust test expectations.
SharedSecretManager RBAC-aware informer strategy
pkg/router/controller/shared_secret_manager.go, pkg/router/controller/shared_secret_manager_test.go
Probe Secrets(list) per namespace to determine RBAC restrictions; create namespace-wide or per-secret informers with matching selectors; support idempotent route registration; normalize and dispatch secret events to matching routes; deduplicate informers; fall back to direct API reads when cache unavailable; add unit tests for hybrid and multi-route scenarios.
RouteSecretManager cache invalidation and status events
pkg/router/controller/route_secret_manager.go, pkg/router/controller/route_secret_manager_test.go
Record ExternalCertificateSARCompleted status after successful downstream handling; synchronously revalidate and repopulate on certain watch.Modified events; invalidate async SAR cache on secret Add/Update/Delete with proper tombstone extraction; refactor tests to use mutex-protected statusRecorder and direct handler invocation instead of informer/channel waiting.
Status equality and ignored condition reasons
pkg/router/controller/contention.go, pkg/router/controller/status.go, pkg/router/controller/status_test.go
Add ExternalCertificateSARCompleted to ignored ingress condition reasons; fix conditionsEqual to require Type/Status match before applying ignore-reason short-circuit; preserve existing ignored Reason/Message in recordIngressCondition when incoming Reason is empty; remove lock from HandleRoute admission path; change conflict handling to return writerlease.None; add test case for ignored reason preservation.
WriterLease multi-worker support
pkg/router/writerlease/writerlease.go, pkg/router/writerlease/writerlease_test.go
Add workers parameter (minimum 1) to New and NewWithBackoff constructors; launch l.workers goroutines in Run coordinated by sync.WaitGroup with graceful shutdown; replace follower-mode sleep with AddAfter requeue; update all tests to pass workers argument.
Wiring and build configuration
go.mod, hack/Makefile.debug, pkg/cmd/infra/router/clientcmd.go, pkg/cmd/infra/router/template.go, pkg/router/router_test.go, pkg/router/template/configmanager/haproxy/testing/haproxy.go
Add go.mod replace for bentito/library-go fork; export GOARCH make variable and pass to image builder; increase Kubernetes client QPS to 50 and Burst to 100; wire SharedSecretManager in template.go and pass MaxConcurrentSARChecks to WriterLease; update test harness; adjust fake HAProxy socket path and error handling.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error Lines 332 and 337 in pkg/router/controller/route_secret_manager.go log "obj" and "tombstone.Obj" parameters which contain entire Kubernetes Secret objects including sensitive Data field with creden... Log only safe metadata like dynamic type name (e.g., fmt.Sprintf("%T", obj)) instead of the entire Secret object; follow the proposed fix in review comments lines 327-339.
Docstring Coverage ⚠️ Warning Docstring coverage is 48.28% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'OCPBUGS-77056: Make external cert validation asynchronous' accurately reflects the primary objective of the changeset—implementing asynchronous SubjectAccessReview checks for external certificate validation to improve router startup performance.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All test names in modified test files use static, descriptive strings. Table-driven tests use static struct field names from hardcoded scenarios, with no dynamic content like timestamps, UUIDs, or...
Test Structure And Quality ✅ Passed The custom check targets Ginkgo tests (It/Describe/BeforeEach/AfterEach patterns), but this PR contains only standard Go unit tests using testing.T. The tests follow good practices: single responsi...
Microshift Test Compatibility ✅ Passed PR adds only standard Go unit tests (testing package), not Ginkgo e2e tests. Custom check for Ginkgo e2e test MicroShift compatibility is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No Ginkgo e2e tests were added in this PR. All test modifications use standard Go testing (testing.T), not Ginkgo framework. The check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed No topology-aware scheduling constraints introduced. PR contains only Go source code changes (controllers, validation, lease management) with no pod specs, affinity rules, replicas calculations, no...
Ote Binary Stdout Contract ✅ Passed This PR contains only library code (no main() functions) and introduces no new process-level code (init(), TestMain(), BeforeSuite(), AfterSuite(), or top-level var initializers) that writes to std...
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR does not add any Ginkgo e2e tests. All changes are to production code and standard Go unit tests, which are outside the scope of this IPv6/disconnected network compatibility check.
No-Weak-Crypto ✅ Passed PR introduces no weak cryptography. All cryptographic operations use secure standard library (crypto/tls, crypto/x509, crypto/rsa, crypto/ecdsa). Test data with weak certs (MD5/SHA1) is used approp...
Container-Privileges ✅ Passed PR introduces no new privileged container settings. deploy/router.yaml's existing hostNetwork: true is justified (ingress router binds ports 80/443); container runs as non-root (USER 1001); no priv...
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/router/controller/route_secret_manager.go`:
- Around line 288-298: The current Add handler unconditionally calls
RecordRouteRejection when a secret is replayed; change it to mirror the
UpdateFunc/SAR-completion logic: check the route's admission (via
populateRouteTLSFromSecret result or route.Status.Conditions) and if the route
is already admitted emit a normal update event instead of calling
RecordRouteRejection, otherwise record the rejection; extract the string
"ExternalCertificateSecretLoaded" into a shared constant (e.g.,
ExtCrtStatusReasonSecretLoaded) and add that constant to the
ignoreIngressConditionReason set in contention.go so the admission-ignoring
logic treats this replay reason as benign. Ensure you update references to
RecordRouteRejection, populateRouteTLSFromSecret, UpdateFunc,
ExternalCertificateSecretLoaded, and ignoreIngressConditionReason accordingly.

In `@pkg/router/routeapihelpers/validation.go`:
- Around line 503-556: The async SAR flow stores only a single pending error
(pendingErr) in asyncSARCache for a cacheKey, so subsequent calls (in validate
flow) that hit the cache drop their onComplete callback and never get
re-enqueued; change the cache value to include a list of waiting callbacks (or a
small struct with errs + []onComplete callbacks) keyed by asyncSARCache so that
when you detect a pending entry you append the current onComplete for
route.Namespace/secretName, and in the goroutine after
asyncSARCache.Store(cacheKey, errs) iterate and invoke all stored callbacks (not
just the first) and then replace the cache entry with errs only; update code
paths that read the cache (the cache-hit branch and the goroutine completion) to
handle this new struct and ensure onComplete is invoked for every waiting route.
- Around line 485-490: The global asyncSARCache currently stores completed
validation results forever; change it to avoid permanent sticky entries by (a)
replacing the raw sync.Map value with a small struct that includes the cached
field.ErrorList plus an expiration timestamp or a source type flag, (b) only
writing non-transient successful validation results (or results with a short
TTL) into asyncSARCache, and (c) adding explicit invalidation helpers (e.g.,
InvalidateAsyncSARCache(namespace, secretName) and
InvalidateAllAsyncSARCacheForSecret(namespace, secretName)) and call them from
the secret add/update/delete/recreate paths in the secret manager
(route_secret_manager.go) so secret changes force revalidation; also update
ClearAsyncSARCacheForTest to clear the new structure. Ensure references to
asyncSARCache and ClearAsyncSARCacheForTest (and the code paths mentioned around
the 503-556 region) are updated to use the new value type and invalidation APIs.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e07f375c-945a-4e58-b5ec-3347eaff109c

📥 Commits

Reviewing files that changed from the base of the PR and between d8ed355 and 093ad83.

📒 Files selected for processing (5)
  • pkg/router/controller/contention.go
  • pkg/router/controller/route_secret_manager.go
  • pkg/router/controller/route_secret_manager_test.go
  • pkg/router/routeapihelpers/validation.go
  • pkg/router/routeapihelpers/validation_test.go

Comment thread pkg/router/controller/route_secret_manager.go Outdated
Comment thread pkg/router/routeapihelpers/validation.go Outdated
Comment thread pkg/router/routeapihelpers/validation.go Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (2)
pkg/router/routeapihelpers/validation.go (1)

485-497: ⚠️ Potential issue | 🟠 Major

Completed SAR entries are still permanent.

Only secret add/update/delete clears this cache. A finished allow/deny result for namespace/secretName survives route deletion, later RBAC grants/revocations, and transient SAR/API failures, so a new route that reuses the same secret name can inherit a stale decision without performing a fresh check. Please put completed entries behind a TTL/generation and avoid caching transient internal errors indefinitely.

Also applies to: 593-605

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/router/routeapihelpers/validation.go` around lines 485 - 497, The cache
currently stores completed asyncSARResult entries forever; update asyncSARResult
(and uses of asyncSARCache) to include a timestamp or generation counter and
enforce a TTL/generation check before returning cached results so that completed
allow/deny decisions expire after a short interval or when generation changes.
Ensure only stable allow/deny outcomes are cached long-term; if
asyncSARResult.errs contains transient/internal errors, set a much shorter TTL
(or do not mark done) so they trigger fresh SARs. Update the cache read path to
validate timestamp/generation and the write path to record it, and keep
InvalidateAsyncSARCache(namespace, secretName) as-is to support manual
invalidation.
pkg/router/controller/route_secret_manager.go (1)

262-304: ⚠️ Potential issue | 🟠 Major

Don't clear the fresh SAR result on the initial SecretLoaded replay.

RegisterRoute() is only reached after validate() has already completed successfully, so this non-deleted AddFunc path is replaying an already-validated secret. Clearing asyncSARCache here throws away that fresh result, defeats the per-secret cache for shared secrets, and can bounce the route back into a second "authorization check pending" / ExternalCertificateValidationFailed cycle on the SecretLoaded requeue. Keep invalidation on actual secret changes (recreated/updated/deleted), but not on the initial load replay.

Suggested shape
 		AddFunc: func(obj interface{}) {
 			secret := obj.(*kapi.Secret)
 			log.V(4).Info("Secret added for route", "namespace", namespace, "secret", secret.Name, "route", routeName)
-			routeapihelpers.InvalidateAsyncSARCache(namespace, secret.Name)
 
 			// Secret re-creation scenario
 			// Check if the route key exists in the deletedSecrets map, indicating that the secret was previously deleted for this route.
 			// If it exists, it means the secret is being recreated. Remove the key from the map and proceed with handling the route.
@@
 			key := generateKey(namespace, routeName)
 			if _, deleted := p.deletedSecrets.LoadAndDelete(key); deleted {
+				routeapihelpers.InvalidateAsyncSARCache(namespace, secret.Name)
 				log.V(4).Info("Secret recreated for route", "namespace", namespace, "secret", secret.Name, "route", routeName)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/router/controller/route_secret_manager.go` around lines 262 - 304, The
AddFunc for secret events currently calls
routeapihelpers.InvalidateAsyncSARCache unconditionally, which discards fresh
per-secret async SAR results during the initial cache-sync replay; to fix,
remove the unconditional call to routeapihelpers.InvalidateAsyncSARCache from
the top of the AddFunc in RegisterRoute's secret handler and only invoke
routeapihelpers.InvalidateAsyncSARCache when the secret is actually
changed/recreated/deleted (e.g., inside the p.deletedSecrets.LoadAndDelete
"recreated" branch and in the Update/Delete handlers), using
generateKey(namespace, routeName) and p.deletedSecrets to determine real
recreation so you don't invalidate the SAR cache on the initial load.
🧹 Nitpick comments (1)
pkg/router/controller/route_secret_manager_test.go (1)

1355-1372: Put a timeout around these async waits.

These bare channel receives will hang the whole suite on any missed informer callback. Now that the tests intentionally depend on async completion, a tiny helper with select + time.After will turn those deadlocks into normal test failures.

Suggested helper
+func waitForRecorderEvent(t *testing.T, ch <-chan struct{}) {
+	t.Helper()
+	select {
+	case <-ch:
+	case <-time.After(2 * time.Second):
+		t.Fatal("timeout waiting for recorder event")
+	}
+}

Then replace each bare receive with:

-<-recorder.doneCh
+waitForRecorderEvent(t, recorder.doneCh)

Also applies to: 1434-1444, 1496-1516

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/router/controller/route_secret_manager_test.go` around lines 1355 - 1372,
Replace bare channel receives on recorder.doneCh with a timed wait helper to
avoid test hangs: add a small helper (e.g., waitForDone(ctxTimeout, ch) or
waitWithTimeout(t, ch, duration)) that uses select with the channel and
time.After to fail the test on timeout, then update every occurrence of
"<-recorder.doneCh" in route_secret_manager_test.go (including the occurrences
around the Secret Update flow that use kubeClient.CoreV1().Secrets(...).Update
and the initial Add wait) to call that helper instead so missed informer
callbacks produce a deterministic test failure.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@pkg/router/controller/route_secret_manager.go`:
- Around line 262-304: The AddFunc for secret events currently calls
routeapihelpers.InvalidateAsyncSARCache unconditionally, which discards fresh
per-secret async SAR results during the initial cache-sync replay; to fix,
remove the unconditional call to routeapihelpers.InvalidateAsyncSARCache from
the top of the AddFunc in RegisterRoute's secret handler and only invoke
routeapihelpers.InvalidateAsyncSARCache when the secret is actually
changed/recreated/deleted (e.g., inside the p.deletedSecrets.LoadAndDelete
"recreated" branch and in the Update/Delete handlers), using
generateKey(namespace, routeName) and p.deletedSecrets to determine real
recreation so you don't invalidate the SAR cache on the initial load.

In `@pkg/router/routeapihelpers/validation.go`:
- Around line 485-497: The cache currently stores completed asyncSARResult
entries forever; update asyncSARResult (and uses of asyncSARCache) to include a
timestamp or generation counter and enforce a TTL/generation check before
returning cached results so that completed allow/deny decisions expire after a
short interval or when generation changes. Ensure only stable allow/deny
outcomes are cached long-term; if asyncSARResult.errs contains
transient/internal errors, set a much shorter TTL (or do not mark done) so they
trigger fresh SARs. Update the cache read path to validate timestamp/generation
and the write path to record it, and keep InvalidateAsyncSARCache(namespace,
secretName) as-is to support manual invalidation.

---

Nitpick comments:
In `@pkg/router/controller/route_secret_manager_test.go`:
- Around line 1355-1372: Replace bare channel receives on recorder.doneCh with a
timed wait helper to avoid test hangs: add a small helper (e.g.,
waitForDone(ctxTimeout, ch) or waitWithTimeout(t, ch, duration)) that uses
select with the channel and time.After to fail the test on timeout, then update
every occurrence of "<-recorder.doneCh" in route_secret_manager_test.go
(including the occurrences around the Secret Update flow that use
kubeClient.CoreV1().Secrets(...).Update and the initial Add wait) to call that
helper instead so missed informer callbacks produce a deterministic test
failure.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c800f527-b5e1-4386-b6cf-e20eb4726a31

📥 Commits

Reviewing files that changed from the base of the PR and between 093ad83 and 8c6f16a.

📒 Files selected for processing (4)
  • pkg/router/controller/contention.go
  • pkg/router/controller/route_secret_manager.go
  • pkg/router/controller/route_secret_manager_test.go
  • pkg/router/routeapihelpers/validation.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • pkg/router/controller/contention.go

@lihongan

Copy link
Copy Markdown

/retest-required

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/router/routeapihelpers/validation.go`:
- Around line 561-568: The semaphore (asyncSARSemaphore) is being acquired on
the caller path before launching the goroutine, causing the caller to block when
the limit is reached; move the acquire into the goroutine so the caller can
spawn all per-secret goroutines without blocking, then release the token inside
that goroutine (use defer to ensure release). Concretely: start the goroutine
immediately, perform asyncSARSemaphore <- struct{}{} at the top of the goroutine
body, and keep the existing defer func() { <-asyncSARSemaphore }() to guarantee
release; ensure any early returns in the goroutine still release the token.
- Around line 532-535: The current guard that returns nil when sarc or
secretsGetter is nil hides wiring bugs and skips RBAC/secret validation;
instead, change the conditional in validation.go so that if sarc == nil ||
secretsGetter == nil you return an internal error (e.g., fmt.Errorf or the
package's internal/server error type) describing "missing validation clients"
rather than nil, or alternatively gate this behind a test-only seam/flag so
production never silently succeeds; update callers to propagate/handle the
returned error as needed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 31462f9b-48bb-466f-8d90-5c399b5fab8b

📥 Commits

Reviewing files that changed from the base of the PR and between 8c6f16a and 871f184.

📒 Files selected for processing (1)
  • pkg/router/routeapihelpers/validation.go

Comment thread pkg/router/routeapihelpers/validation.go Outdated
Comment thread pkg/router/routeapihelpers/validation.go Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/router/routeapihelpers/validation.go`:
- Around line 526-529: The code currently returns a synthetic pending error on
cache miss without registering a callback if onComplete is nil; change the logic
in the validation path (the block that manipulates cached.callbacks and returns
cached.errs) so that onComplete must never be nil: either (A) make onComplete a
required parameter at the API boundary, or (B) when onComplete == nil execute
the async validation synchronously (drive the same validation routine on the
current goroutine, wait for completion, populate cached.errs, and then return
the real errors) instead of just returning the pending placeholder; if you
choose (B) ensure you reuse the same validation function that enqueues callbacks
(so cached state is updated) and that cached.callbacks is appended only when
onComplete is non-nil or after synchronous completion you return the finalized
errors.
- Around line 572-599: Wrap each SAR and secret GET with a per-request timeout
context (use context.WithTimeout and defer cancel) instead of context.TODO();
for the SAR checks stop using authorizationutil.Authorize and call the SAR
client Create() directly (e.g., sarClient.Create) with that timeout context so
you can inspect both (response, err) separately, treat API/transport errors as
retryable/internal (append field.InternalError or a distinct cached error) and
only append field.Forbidden when the SAR response explicitly denies, and call
secretsGetter.Secrets(route.Namespace).Get with the same bounded context and map
Get errors to NotFound vs InternalError accordingly. Ensure you still reference
routerServiceAccount, secretName, fldPath and preserve existing
field.Forbidden/field.InternalError/field.NotFound usages when classifying the
outcome.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: bbebc97e-5cff-40e0-80d7-bff7bf16a68c

📥 Commits

Reviewing files that changed from the base of the PR and between 871f184 and c2f6381.

📒 Files selected for processing (1)
  • pkg/router/routeapihelpers/validation.go

Comment thread pkg/router/routeapihelpers/validation.go Outdated
Comment thread pkg/router/routeapihelpers/validation.go Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
pkg/router/routeapihelpers/validation.go (1)

637-639: Consider recovering from callback panics to ensure all callbacks are invoked.

If a callback panics, the remaining callbacks in the slice won't be invoked. This is unlikely but could leave some routes stuck in a pending state if one route's callback handler has a bug.

🛡️ Proposed defensive callback invocation
 		for _, cb := range callbacks {
-			cb(route.Namespace, secretName)
+			func() {
+				defer func() {
+					if r := recover(); r != nil {
+						// Log panic but continue invoking remaining callbacks
+					}
+				}()
+				cb(route.Namespace, secretName)
+			}()
 		}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/router/routeapihelpers/validation.go` around lines 637 - 639, Wrap each
callback invocation in a deferred recover to prevent a single panic from
aborting the loop: inside the loop over callbacks (callbacks, cb) call each cb
via a small closure that uses defer + recover to catch any panic, log or report
the panic along with the callback context (e.g., route.Namespace and secretName)
and continue to the next cb; ensure the recovered panic does not re-panic so all
callbacks are invoked.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@pkg/router/routeapihelpers/validation.go`:
- Around line 637-639: Wrap each callback invocation in a deferred recover to
prevent a single panic from aborting the loop: inside the loop over callbacks
(callbacks, cb) call each cb via a small closure that uses defer + recover to
catch any panic, log or report the panic along with the callback context (e.g.,
route.Namespace and secretName) and continue to the next cb; ensure the
recovered panic does not re-panic so all callbacks are invoked.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c9c78ea5-dba5-483e-8d75-13dc6f9905d7

📥 Commits

Reviewing files that changed from the base of the PR and between c2f6381 and dc6b383.

📒 Files selected for processing (1)
  • pkg/router/routeapihelpers/validation.go

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 17, 2026
@bentito
bentito force-pushed the OCPBUGS-77056-async-sar branch from 716e042 to 0a99ee0 Compare March 17, 2026 20:15
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 17, 2026
@bentito
bentito force-pushed the OCPBUGS-77056-async-sar branch from 0a99ee0 to 4e1eae4 Compare March 23, 2026 13:51
@bentito

bentito commented Mar 27, 2026

Copy link
Copy Markdown
Contributor Author

Depends-On: openshift/library-go#2132

@bentito

bentito commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

/hold cancel

@openshift-ci openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 14, 2026
@jcmoraisjr

Copy link
Copy Markdown
Member

Concerns addressed or commented. Just the unlock during the GetList() leaves the code more complex and error prone on future updates, but I understand it addresses a latency sensitive code running on huge environments.

/approve
/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 16, 2026
@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcmoraisjr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 16, 2026
@bentito

bentito commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

/acknowledge-critical-fixes-only

@melvinjoseph86

Copy link
Copy Markdown

/label acknowledge-critical-fixes-only
based on #745 (comment)

@openshift-ci openshift-ci Bot added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Jul 17, 2026
@melvinjoseph86

Copy link
Copy Markdown

/retest-required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 63387d7 and 2 for PR HEAD 1b9d32f in total

@bentito

bentito commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

/retest-required

@gcs278

gcs278 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

/payload-job periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-upgrade

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@gcs278: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/357d5bc0-81f3-11f1-8145-96d019be3322-0

@gcs278

gcs278 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@melvinjoseph86 @bentito mind reviewing https://pr-payload-tests.ci.openshift.org/runs/ci/357d5bc0-81f3-11f1-8145-96d019be3322-0 when it finishes before merging? We have a pretty big CI gap on upgrade E2E that this covers (see openshift/release#82020 for the fix).

@bentito
bentito force-pushed the OCPBUGS-77056-async-sar branch from 1b9d32f to 604900f Compare July 17, 2026 20:54
@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Jul 17, 2026
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 17, 2026
@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@bentito

bentito commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

@gcs278 okay I rebased (even though github could not see it was needed)
going to run that payload test again

@bentito

bentito commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

/payload-job periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-upgrade

@bentito

bentito commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

@jcmoraisjr please re-lgtm -- just a rebase to allow the payload job to run
@melvinjoseph86 please re-verify -- just a rebase to allow the payload job to run

@openshift-ci

openshift-ci Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

@bentito: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-serial 2611961 link true /test e2e-aws-serial
ci/prow/e2e-aws-fips 604900f link true /test e2e-aws-fips

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants