WIP: OCPBUGS-77056: add external certs benchmark scripts

[bentito]

This PR adds an orchestrator (00-run-all.sh) and supporting scripts to benchmark the OpenShift router's startup time when handling spec.tls.externalCertificate routes, aimed at validating fixes for OCPBUGS-77056.

The benchmark harness tests how quickly the router can load N external certificates, comparing the existing bug (where a global write lock causes serial registrations) against a patched image that allows concurrent processing.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rikatz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@bentito: This pull request references Jira Issue OCPBUGS-77056, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lihongan

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This PR adds an orchestrator (00-run-all.sh) and supporting scripts to benchmark the OpenShift router's startup time when handling spec.tls.externalCertificate routes, aimed at validating fixes for OCPBUGS-77056.

The benchmark harness tests how quickly the router can load N external certificates, comparing the existing bug (where a global write lock causes serial registrations) against a patched image that allows concurrent processing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bentito]

The fix for OCPBUGS-77056 that enables faster startup behavior benchmarked in this PR is implemented and proposed in the following linked pulls:

• OCPBUGS-77056: Release lock before WaitForCacheSync in secret monitor library-go#2132 (Async secret registration / non-blocking cache sync)
• OCPBUGS-77056: Make external cert validation asynchronous router#745 (Asynchronous and cached SubjectAccessReview evaluation)

[bentito]

This PR is primarily to tie together changes in router and library-go and to illustrate a benchmarking scheme for the Perf/Scale team.

@SachinNinganure please let me know if the benchmarking scripts helps test changes to the router made in the linked PRs.

We are hoping you're able to work testing with these PRs in place into some of your work on https://issues.redhat.com/browse/CORENET-6842

Primarily we're looking to make sure that parallelizing these calls to speed up the router's handling of adding routes with external certs, is not simply shifting the problem to the API server or elsewhere. So when we save time in the router pod are we causing CPU or memory problems elsewhere on the cluster. Thanks!

[SachinNinganure]

This PR is primarily to tie together changes in router and library-go and to illustrate a benchmarking scheme for the Perf/Scale team.

@SachinNinganure please let me know if the benchmarking scripts helps test changes to the router made in the linked PRs.

We are hoping you're able to work testing with these PRs in place into some of your work on https://issues.redhat.com/browse/CORENET-6842

Primarily we're looking to make sure that parallelizing these calls to speed up the router's handling of adding routes with external certs, is not simply shifting the problem to the API server or elsewhere. So when we save time in the router pod are we causing CPU or memory problems elsewhere on the cluster. Thanks!

I have started putting some note in this doc. https://docs.google.com/document/d/1tEZBNEaKLvXI8l1KIcHKCQgbAzkv0_YQSzkXcrlq56Y/edit?tab=t.0

I have just tried to understand the benchmark scripts and yes they directly test both the pr-s.
the flow is ...1>It Creates 200 routes with external certificates (configurable) 2>Restarts router with current code → measures startup time 3>Deploys patched router with async changes → measures startup time...4> Compares the perf! I will additionally monitor the resources and their utilization.

for 6842 I will be running our ingress perf test, you may take a look at doc for more info

I will be running the benchmark scripts you created for the linked pr verification and additionally , create the perf-scale-grafana boards for the resource utilization and other verification. Thank you