OCPBUGS-84254: Redeploy console pods upon cert rotation

[jhadvig]

Root cause

When the console-serving-cert secret is rotated (by service-ca operator), the console-operator does not detect the change. The console pods continue
running with the old certificate mounted at /var/serving-cert, leading to TLS errors and console unavailability until pods are manually restarted.

Solution

Register an informer for the console-serving-cert secret so the operator watches it for changes. When the secret's resourceVersion changes, the
operator stamps the new value as a console.openshift.io/serving-cert-secret-version annotation on the console Deployment, which triggers an
automatic pod rollout with the new certificate.

Backport of #1093

Test cases

Scenario 1: Verify secret informer registration

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
36546
$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
36546

Expected: The deployment annotation matches the secret's resourceVersion.

Scenario 2: Automatic pod restart on certificate rotation

Delete the serving cert to simulate rotation

$ oc delete secret console-serving-cert -n openshift-console

Wait for service-ca to regenerate it, then verify

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
38672

$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
38672

$ oc get pods -n openshift-console
NAME READY STATUS RESTARTS AGE
console-75d8bfcdf6-6w2wf 1/1 Running 0 55s <-- redeployed
console-75d8bfcdf6-9zn7z 1/1 Running 0 55s <-- redeployed
downloads-6d94488dc7-rnqts 1/1 Running 0 52m
downloads-6d94488dc7-tb4bz 1/1 Running 0 52m

Expected: The annotation updates to the new resourceVersion and console pods are automatically redeployed. Downloads pods remain untouched.

/assign @Leo6Leo

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8a0b70c5-0ab1-456d-9345-d66625a287e3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci-robot]

@jhadvig: This pull request references Jira Issue OCPBUGS-63502, which is invalid:

• expected the bug to target either version "4.20." or "openshift-4.20.", but it targets "4.22.0" instead
• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead
• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected Jira Issue OCPBUGS-63502 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Root cause

When the console-serving-cert secret is rotated (by service-ca operator), the console-operator does not detect the change. The console pods continue
running with the old certificate mounted at /var/serving-cert, leading to TLS errors and console unavailability until pods are manually restarted.

Solution

Register an informer for the console-serving-cert secret so the operator watches it for changes. When the secret's resourceVersion changes, the
operator stamps the new value as a console.openshift.io/serving-cert-secret-version annotation on the console Deployment, which triggers an
automatic pod rollout with the new certificate.

Backport of #1093

Test cases

Scenario 1: Verify secret informer registration

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
36546
$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
36546

Expected: The deployment annotation matches the secret's resourceVersion.

Scenario 2: Automatic pod restart on certificate rotation

Delete the serving cert to simulate rotation

$ oc delete secret console-serving-cert -n openshift-console

Wait for service-ca to regenerate it, then verify

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
38672

$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
38672

$ oc get pods -n openshift-console
NAME READY STATUS RESTARTS AGE
console-75d8bfcdf6-6w2wf 1/1 Running 0 55s <-- redeployed
console-75d8bfcdf6-9zn7z 1/1 Running 0 55s <-- redeployed
downloads-6d94488dc7-rnqts 1/1 Running 0 52m
downloads-6d94488dc7-tb4bz 1/1 Running 0 52m

Expected: The annotation updates to the new resourceVersion and console pods are automatically redeployed. Downloads pods remain untouched.

/assign @Leo6Leo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jhadvig

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jhadvig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@jhadvig: This pull request references Jira Issue OCPBUGS-84254, which is invalid:

• expected Jira Issue OCPBUGS-84254 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Root cause

When the console-serving-cert secret is rotated (by service-ca operator), the console-operator does not detect the change. The console pods continue
running with the old certificate mounted at /var/serving-cert, leading to TLS errors and console unavailability until pods are manually restarted.

Solution

Register an informer for the console-serving-cert secret so the operator watches it for changes. When the secret's resourceVersion changes, the
operator stamps the new value as a console.openshift.io/serving-cert-secret-version annotation on the console Deployment, which triggers an
automatic pod rollout with the new certificate.

Backport of #1093

Test cases

Scenario 1: Verify secret informer registration

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
36546
$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
36546

Expected: The deployment annotation matches the secret's resourceVersion.

Scenario 2: Automatic pod restart on certificate rotation

Delete the serving cert to simulate rotation

$ oc delete secret console-serving-cert -n openshift-console

Wait for service-ca to regenerate it, then verify

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
38672

$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
38672

$ oc get pods -n openshift-console
NAME READY STATUS RESTARTS AGE
console-75d8bfcdf6-6w2wf 1/1 Running 0 55s <-- redeployed
console-75d8bfcdf6-9zn7z 1/1 Running 0 55s <-- redeployed
downloads-6d94488dc7-rnqts 1/1 Running 0 52m
downloads-6d94488dc7-tb4bz 1/1 Running 0 52m

Expected: The annotation updates to the new resourceVersion and console pods are automatically redeployed. Downloads pods remain untouched.

/assign @Leo6Leo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@jhadvig: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-operator	1532fca	link	true	/test e2e-aws-operator
ci/prow/e2e-aws-console	1532fca	link	true	/test e2e-aws-console

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.