OCPBUGS-84332: Fix ssh and password rollbacks#5884
OCPBUGS-84332: Fix ssh and password rollbacks#5884openshift-merge-bot[bot] merged 1 commit intoopenshift:mainfrom
Conversation
This change fixes the issue in SSH keys and user passwords that made the rollback useless as it tried to apply the new configuration instead of the previous one. Signed-off-by: Pablo Rodriguez Nava <git@amail.pablintino.eu>
openshift-ci-robot
added
jira/severity-low
|
@pablintino: This pull request references Jira Issue OCPBUGS-84332, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
WalkthroughA bug fix in the Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 11 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (11 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
openshift-ci
Bot
added
the
approved
|
/lgtm |
|
Scheduling tests matching the |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: pablintino, umohnani8 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@isabella-janssen: This pull request references Jira Issue OCPBUGS-84332, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
HarshwardhanPatil07
left a comment
HarshwardhanPatil07
left a comment
There was a problem hiding this comment.
Pre-merge Verification: PASSED
Cluster version: 4.22.0-0-2026-04-29-050426-test-ci-ln-5rzgwi2-latest
Steps Performed:
- Apply the first MC
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ cat <<'EOF' | oc apply -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: mc-tc-88821-rollback
spec:
config:
ignition:
version: 3.5.0
passwd:
users:
- name: core
passwordHash: $6$rounds=5000$saltsalt$abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSv1
sshAuthorizedKeys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtest1234567890 testv1@mco
storage:
files:
- contents:
source: data:,mco-test-content-v1%0A
mode: 0644
overwrite: true
path: /etc/mco-test-file.conf
systemd:
units:
- contents: |
[Unit]
Description=MCO test unit
[Service]
Type=oneshot
ExecStart=/bin/true
Environment="MCO_TEST_UNIT=v1"
[Install]
WantedBy=multi-user.target
enabled: true
name: mco-test.service
dropins:
- contents: |
[Service]
Environment="MCO_TEST=v1"
name: 99-mco-test.conf
- name: wait-for-ipsec-connect.service
- contents: |
[Unit]
Description=Snapshot all tracked files on change
[Service]
Type=forking
ExecStart=/bin/bash -c '\
( dir=/tmp/drift-snapshots/$(date +%%Y%%m%%d-%%H%%M%%S.%%N); \
mkdir -p $dir; \
for f in \
/etc/mco-test-file.conf \
/etc/mco-test-new-file.conf \
/etc/systemd/system/mco-test.service \
/etc/systemd/system/mco-test-new.service \
/etc/systemd/system/wait-for-ipsec-connect.service.d/99-mco-test.conf \
/etc/systemd/system/wait-for-ipsec-connect.service.d/99-mco-test-new.conf \
/home/core/.ssh/authorized_keys.d/ignition \
/etc/shadow; \
do \
name=$(echo $f | sed "s|/|_|g"); \
if [ -f "$f" ]; then \
cp "$f" "$dir/$name"; \
else \
echo "ABSENT" > "$dir/$name"; \
fi; \
done ) &'
name: mco-drift-snapshot.service
- contents: |
[Unit]
Description=Watch existing tracked files for changes
[Path]
PathChanged=/etc/mco-test-file.conf
PathChanged=/etc/systemd/system/mco-test.service
PathChanged=/etc/systemd/system/wait-for-ipsec-connect.service.d/99-mco-test.conf
PathChanged=/home/core/.ssh/authorized_keys.d/ignition
PathChanged=/etc/shadow
[Install]
WantedBy=multi-user.target
enabled: true
name: mco-drift-snapshot.path
- contents: |
[Unit]
Description=Watch for new files being created
[Path]
PathExists=/etc/mco-test-new-file.conf
PathExists=/etc/systemd/system/mco-test-new.service
PathExists=/etc/systemd/system/wait-for-ipsec-connect.service.d/99-mco-test-new.conf
[Install]
WantedBy=multi-user.target
enabled: true
name: mco-drift-snapshot-new.path
- contents: |
[Unit]
Description=Snapshot when new files appear then watch for removal
[Service]
Type=oneshot
ExecStart=/bin/bash -c '\
/usr/bin/systemctl start mco-drift-snapshot.service; \
/usr/bin/systemctl start mco-drift-snapshot-removal.path'
name: mco-drift-snapshot-new.service
- contents: |
[Unit]
Description=Watch for new files being removed
[Path]
PathChanged=/etc/mco-test-new-file.conf
PathChanged=/etc/systemd/system/mco-test-new.service
PathChanged=/etc/systemd/system/wait-for-ipsec-connect.service.d/99-mco-test-new.conf
[Install]
WantedBy=multi-user.target
enabled: true
name: mco-drift-snapshot-removal.path
- contents: |
[Unit]
Description=Snapshot when new files are removed
[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl start mco-drift-snapshot.service
name: mco-drift-snapshot-removal.service
EOF
machineconfig.machineconfiguration.openshift.io/mc-tc-88821-rollback created
- Wait for mcp to get updated
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-ea2b91bd318f4982d593dcec2c5a17f8 True False False 3 3 3 0 81m
worker rendered-worker-ab6a569937433aac6f603675126fa01b True False False 3 3 3 0 81m
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ oc get mc mc-tc-88821-rollback
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
mc-tc-88821-rollback 3.5.0 20m
- Apply the second MC
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ cat <<'EOF' | oc apply -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: mc-tc-88821-rollback-v2
spec:
config:
ignition:
version: 3.5.0
passwd:
users:
- name: core
passwordHash: $6$rounds=5000$saltsalt$zyxwvutsrqponmlkjihgfedcba9876543210ZYXWVUTSRQPONMLKJIv2
sshAuthorizedKeys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmodified9876543210 testv2@mco
storage:
files:
- contents:
source: data:,mco-test-content-v2%0A
mode: 0644
overwrite: true
path: /etc/mco-test-file.conf
- contents:
source: data:,mco-test-new-file-v2%0A
mode: 0644
overwrite: true
path: /etc/mco-test-new-file.conf
systemd:
units:
- contents: |
[Unit]
Description=MCO test unit
[Service]
Type=oneshot
ExecStart=/bin/true
Environment="MCO_TEST_UNIT=v2"
[Install]
WantedBy=multi-user.target
enabled: true
name: mco-test.service
- contents: |
[Unit]
Description=MCO test new unit
[Service]
Type=oneshot
ExecStart=/bin/true
Environment="MCO_TEST_NEW_UNIT=v2"
[Install]
WantedBy=multi-user.target
enabled: true
name: mco-test-new.service
- dropins:
- contents: |
[Service]
Environment="MCO_TEST=v2"
name: 99-mco-test.conf
- contents: |
[Service]
Environment="MCO_TEST_NEW_DROPIN=v2"
name: 99-mco-test-new.conf
name: wait-for-ipsec-connect.service
extensions: []
kernelArguments: []
osImageURL: quay.io/openshifttest/tc88821fakeimage:latest
EOF
machineconfig.machineconfiguration.openshift.io/mc-tc-88821-rollback-v2 created
- Wait for mcp to degrade
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-ea2b91bd318f4982d593dcec2c5a17f8 True False False 3 3 3 0 117m
worker rendered-worker-ab6a569937433aac6f603675126fa01b False True True 3 0 0 1 117m
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ oc get mcp worker -o jsonpath='{.status.conditions[?(@.type=="Degraded")].message}' | fold -s
Node ip-10-0-16-3.us-east-2.compute.internal is reporting: "Node
ip-10-0-16-3.us-east-2.compute.internal upgrade failure. Failed to update OS to
quay.io/openshifttest/tc88821fakeimage:latest after retries: timed out waiting
for the condition", Node ip-10-0-16-3.us-east-2.compute.internal is reporting:
"Failed to update OS to quay.io/openshifttest/tc88821fakeimage:latest after
retries: timed out waiting for the condition"
- Check all rollback snapshots
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ NODE=$(oc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{.items[0].metadata.name}')
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ echo "=== /etc/mco-test-file.conf ==="
oc debug -q node/$NODE -- chroot /host bash -c 'for d in $(ls -d /tmp/drift-snapshots/* 2>/dev/null | sort); do echo "$(basename $d): $(cat $d/_etc_mco-test-file.conf 2>/dev/null)"; done' 2>/dev/null
=== /etc/mco-test-file.conf ===
---
20260429-070555.563971269: mco-test-content-v2
20260429-070555.632212375: mco-test-content-v2
20260429-070556.311659463: mco-test-content-v2
20260429-070717.501814852: mco-test-content-v1
20260429-070718.182021502: mco-test-content-v1
20260429-070718.472947360: mco-test-content-v1
20260429-070719.565260763: mco-test-content-v2
20260429-070719.622025046: mco-test-content-v2
20260429-070720.273521457: mco-test-content-v2
---
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ echo ""
echo "=== mco-test.service ==="
oc debug -q node/$NODE -- chroot /host bash -c 'for d in $(ls -d /tmp/drift-snapshots/* 2>/dev/null | sort); do echo "$(basename $d): $(grep MCO_TEST_UNIT $d/_etc_systemd_system_mco-test.service 2>/dev/null || cat $d/_etc_systemd_system_mco-test.service 2>/dev/null)"; done' 2>/dev/null
=== mco-test.service ===
---
20260429-070432.343292042: Environment="MCO_TEST_UNIT=v2"
20260429-070553.913947099: Environment="MCO_TEST_UNIT=v1"
20260429-070554.533559300: Environment="MCO_TEST_UNIT=v1"
20260429-070555.563971269: Environment="MCO_TEST_UNIT=v1"
20260429-070555.632212375: Environment="MCO_TEST_UNIT=v1"
20260429-070556.311659463: Environment="MCO_TEST_UNIT=v2"
---
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ echo ""
echo "=== 99-mco-test.conf dropin ==="
oc debug -q node/$NODE -- chroot /host bash -c 'for d in $(ls -d /tmp/drift-snapshots/* 2>/dev/null | sort); do echo "$(basename $d): $(grep MCO_TEST $d/_etc_systemd_system_wait-for-ipsec-connect.service.d_99-mco-test.conf 2>/dev/null || cat $d/_etc_systemd_system_wait-for-ipsec-connect.service.d_99-mco-test.conf 2>/dev/null)"; done' 2>/dev/null
=== 99-mco-test.conf dropin ===
---
20260429-070305.944383893: Environment="MCO_TEST=v2"
20260429-070306.277951834: ABSENT
20260429-070306.587866784: ABSENT
20260429-070307.635692857: ABSENT
20260429-070307.701316420: Environment="MCO_TEST=v2"
---
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ echo ""
echo "=== /etc/mco-test-new-file.conf (new in v2) ==="
oc debug -q node/$NODE -- chroot /host bash -c 'for d in $(ls -d /tmp/drift-snapshots/* 2>/dev/null | sort); do echo "$(basename $d): $(cat $d/_etc_mco-test-new-file.conf 2>/dev/null)"; done' 2>/dev/null
=== /etc/mco-test-new-file.conf (new in v2) ===
---
20260429-070141.917470975: mco-test-new-file-v2
20260429-070142.277812836: ABSENT
20260429-070142.571690444: ABSENT
20260429-070143.674182930: mco-test-new-file-v2
---
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ echo ""
echo "=== mco-test-new.service (new in v2) ==="
oc debug -q node/$NODE -- chroot /host bash -c 'for d in $(ls -d /tmp/drift-snapshots/* 2>/dev/null | sort); do echo "$(basename $d): $(grep MCO_TEST_NEW_UNIT $d/_etc_systemd_system_mco-test-new.service 2>/dev/null || cat $d/_etc_systemd_system_mco-test-new.service 2>/dev/null)"; done' 2>/dev/null
=== mco-test-new.service (new in v2) ===
---
20260429-070142.277812836: Environment="MCO_TEST_NEW_UNIT=v2"
20260429-070142.571690444: ABSENT
20260429-070143.674182930: ABSENT
20260429-070143.749306372: Environment="MCO_TEST_NEW_UNIT=v2"
---
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ echo ""
echo "=== 99-mco-test-new.conf dropin (new in v2) ==="
oc debug -q node/$NODE -- chroot /host bash -c 'for d in $(ls -d /tmp/drift-snapshots/* 2>/dev/null | sort); do echo "$(basename $d): $(grep MCO_TEST_NEW_DROPIN $d/_etc_systemd_system_wait-for-ipsec-connect.service.d_99-mco-test-new.conf 2>/dev/null || cat $d/_etc_systemd_system_wait-for-ipsec-connect.service.d_99-mco-test-new.conf 2>/dev/null)"; done' 2>/dev/null
=== 99-mco-test-new.conf dropin (new in v2) ===
---
20260429-070141.917470975: Environment="MCO_TEST_NEW_DROPIN=v2"
20260429-070142.277812836: ABSENT
20260429-070142.571690444: ABSENT
20260429-070143.674182930: ABSENT
20260429-070143.749306372: Environment="MCO_TEST_NEW_DROPIN=v2"
---
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ echo ""
echo "=== authorized_keys (ssh) ==="
oc debug -q node/$NODE -- chroot /host bash -c 'for d in $(ls -d /tmp/drift-snapshots/* 2>/dev/null | sort); do echo "$(basename $d): $(grep -o "test[^ ]*@mco[^ ]*" $d/_home_core_.ssh_authorized_keys.d_ignition 2>/dev/null)"; done' 2>/dev/null
=== authorized_keys (ssh) ===
---
20260429-070012.229867354: testv1@mco
testv2@mco
20260429-070012.273334900: testv1@mco
testv2@mco
20260429-070141.917470975: testv1@mco
20260429-070142.277812836: testv1@mco
---
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ echo ""
echo "=== shadow (core password) ==="
oc debug -q node/$NODE -- chroot /host bash -c 'for d in $(ls -d /tmp/drift-snapshots/* 2>/dev/null | sort); do echo "$(basename $d): $(grep core $d/_etc_shadow 2>/dev/null | cut -d: -f2 | grep -o "..$")"; done' 2>/dev/null
=== shadow (core password) ===
---
20260429-070012.229867354: v2
20260429-070012.273334900: v2
20260429-070141.917470975: v1
20260429-070142.277812836: v1
20260429-070142.571690444: v1
---
- Cleanup
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ oc delete mc mc-tc-88821-rollback-v2
machineconfig.machineconfiguration.openshift.io "mc-tc-88821-rollback-v2" deleted
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ oc patch node "$NODE" -p "{\"metadata\":{\"annotations\":{\"machineconfiguration.openshift.io/desiredConfig\":\"$(oc get node "$NODE" -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/currentConfig}')\"}}}"
node/ip-10-0-16-3.us-east-2.compute.internal patched
harshpat@harshpat-thinkpadp1gen4i:~/Downloads$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-ea2b91bd318f4982d593dcec2c5a17f8 True False False 3 3 3 0 136m
worker rendered-worker-ab6a569937433aac6f603675126fa01b True False False 3 3 3 0 136m
|
/verified by @HarshwardhanPatil07 here |
openshift-ci-robot
added
the
verified
|
@HarshwardhanPatil07: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/test e2e-gcp-op-single-node |
|
/test e2e-hypershift |
|
/test e2e-aws-ovn-upgrade |
|
/hold Revision 3d98512 was retested 3 times: holding |
openshift-ci
Bot
added
the
do-not-merge/hold
|
/test e2e-aws-ovn-upgrade |
|
/unhold |
openshift-ci
Bot
removed
the
do-not-merge/hold
|
@pablintino: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@pablintino: Jira Issue Verification Checks: Jira Issue OCPBUGS-84332 Jira Issue OCPBUGS-84332 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/cherry-pick release-4.22 |
|
@pablintino: new pull request created: #5906 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Fix included in release 5.0.0-0.nightly-2026-05-01-010944 |
7 participants
Closes: #OCPBUGS-84332
- What I did
This change fixes the issue in SSH keys and user passwords that made the rollback useless as it tried to apply the new configuration instead of the previous one.
- How to verify it
TBD
- Description for the changelog
This change fixes the issue in SSH keys and user passwords that made the rollback useless as it tried to apply the new configuration instead of the previous one.
Summary by CodeRabbit