kola: Add soft-reboot support for external tests by jmarrero · Pull Request #4119 · coreos/coreos-assembler

jmarrero · 2025-05-30T19:10:13Z

Implements soft-reboot capabilities for Kola,
it enables tests to use systemd's soft-reboot functionality.

The implementation follows the same pattern as regular reboots but for systemctl soft-reboot, tracks systemd boot
timestamps rather than kernel boot IDs for state detection.

openshift-ci · 2025-05-30T19:10:17Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

depends on: coreos/coreos-assembler#4119

dustymabe · 2025-05-30T19:17:01Z

Implements soft-reboot capabilities for Kola,
it enables tests to use systemd's soft-reboot functionality.

Is there a case where a soft-reboot would be a better test than a hard reboot?

Is the goal here to reduce test time?

jmarrero · 2025-05-30T19:21:09Z

Right now soft-reboots are not really supported by ostree but we are implementing soft-reboots for ostree on : ostreedev/ostree#3420
We wanted to use Kola for the tests as we already depend on it. The idea is to reduce reboot time and we only would support if no Kernel is present on the deployment we are going to soft-reboot to. But it's still in progress. Just that having Kola support it would make our testing cycle easier too.

dustymabe · 2025-05-30T19:22:17Z

Ok, Just comment here when you move this out of draft

cgwalters

Looks good to me overall! Main issue is code duplication with the "hard" reboot path but that's probably hard to fix and not worth doing.

cgwalters · 2025-05-30T19:27:35Z

+    echo "test beginning"
+    # Check that boot_id stays the same across soft-reboot
+    INITIAL_BOOT_ID=$(cat /proc/sys/kernel/random/boot_id)
+    echo "Initial boot ID: $INITIAL_BOOT_ID" | logger


Don't need the |logger, we always log tests to the journal

cgwalters · 2025-05-30T19:28:30Z

+    # Verify boot_id is the same (soft-reboot should not change it)
+    CURRENT_BOOT_ID=$(cat /proc/sys/kernel/random/boot_id)
+    echo "Current boot ID after soft-reboot: $CURRENT_BOOT_ID" | logger


How is this verified though? I think we need to save the boot id to a persistent file like /var/cache/kola-boot-id or something in the first boot and then compare it here.

cgwalters · 2025-05-30T19:29:01Z

+  mark2) 
+    echo "test in mark2"
+    FINAL_BOOT_ID=$(cat /proc/sys/kernel/random/boot_id)
+    echo "Final boot ID after forced soft-reboot: $FINAL_BOOT_ID" | logger


jmarrero · 2025-06-03T14:23:18Z

OK test now dies with a timeout which is the same behavior I see when I do systemctl soft-reboot without any of our changes to ostree. I will try to see if I can get farther with our changes plus this test.

edit: looking at the console log the VM appears to come back. It's just cosa that loses the ability to ssh back in, even during a cosa run. I been trying to figure out why.

jmarrero · 2025-06-04T22:38:38Z

OK needed to modify qemu.go to get the test to comeback too.

Now need to figureout this failed service.

× coreos-ignition-firstboot-complete.service - CoreOS Mark Ignition Boot Complete
     Loaded: loaded (/usr/lib/systemd/system/coreos-ignition-firstboot-complete.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Wed 2025-06-04 22:34:52 UTC; 18s ago
   Duration: 11.967s
 Invocation: 97458820b4e8426184180f42cd04bf13
       Docs: https://docs.fedoraproject.org/en-US/fedora-coreos/
    Process: 2729 ExecStart=/usr/libexec/coreos-ignition-firstboot-complete (code=exited, status=1/FAILURE)
   Main PID: 2729 (code=exited, status=1/FAILURE)
   Mem peak: 2.7M
        CPU: 11ms

Jun 04 22:34:52 cosa-devsh systemd[1]: Starting coreos-ignition-firstboot-complete.service - CoreOS Mark Ignition Boot Complete...
Jun 04 22:34:52 cosa-devsh coreos-ignition-firstboot-complete[2755]: rm: cannot remove '/boot/ignition.firstboot': No such file or directory
Jun 04 22:34:52 cosa-devsh systemd[1]: coreos-ignition-firstboot-complete.service: Main process exited, code=exited, status=1/FAILURE
Jun 04 22:34:52 cosa-devsh systemd[1]: coreos-ignition-firstboot-complete.service: Failed with result 'exit-code'.
Jun 04 22:34:52 cosa-devsh systemd[1]: Failed to start coreos-ignition-firstboot-complete.service - CoreOS Mark Ignition Boot Complete.

jmarrero · 2025-06-04T23:21:06Z

https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14

I am not sure of the story of that service but it looks like we could ignore that failed service without any real issues.

cgwalters · 2025-06-05T00:45:49Z

My offhand guess as to what's happening here is that service isn't prepared for soft reboots. https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/lib/systemd/system/coreos-ignition-firstboot-complete.service#L4 is still triggered because we didn't change the kernel commandline across the soft reboot.

I think CoreOS probably does need to fix this because soft rebooting from the initial boot is a sane thing to do and I see real world use cases for it. It is perhaps as simple as ConditionPathExists=!/boot/ignition.firstboot in the unit.

On coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14

coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14

cgwalters · 2025-06-05T21:04:58Z

Holy cow that was painful to figure out but fixes in #4133 get us improved error message handling to the point where I now see

2025-06-05T20:53:19Z kola: dropping to shell: kolet failed: kolet run-test-unit failed:  mkfifo: cannot create fifo '/run/kolet-reboot': File exists

Which was the real bug here - with soft reboots /run persists (this will be a big trap!) and so we need to change the mkfifo to be idempotent, which I'll do after rebasing this PR on the prep one.

coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14

coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14 Co-authored-by: Colin Walters <walters@verbum.org>

…t-reboots coreos/coreos-assembler#4119 surfaced that this service would fail on soft-reboots, it's non fatal but would make the Kola tests fail. It looks like originally this was set to fail instead of risking not running it: https://github.com/coreos/fedora-coreos-config/blob/20feb176f19c3142b7256c1eb5bf1cb7c53b29b9/overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete#L14 Co-authored-by: Colin Walters <walters@verbum.org>

coreos/coreos-assembler#4119 surfaced that this services using ConditionKernelCommandLine=ignition.firstboot would fail on soft-reboots, it's non fatal but would make the Kola tests fail. Co-authored-by: Colin Walters <walters@verbum.org>

jmarrero · 2025-06-08T01:07:49Z

/test rhcos

coreos/coreos-assembler#4119 surfaced that this services using ConditionKernelCommandLine=ignition.firstboot would fail on soft-reboots, it's non fatal but would make the Kola tests fail. Co-authored-by: Jonathan Lebon <jonathan@jlebon.com> Co-authored-by: Colin Walters <walters@verbum.org>

jmarrero · 2025-06-16T14:09:15Z

/retest-required

cgwalters

I don't think we had any blockers left right?

Implements soft-reboot capabilities for Kola, it enables tests to use systemd's soft-reboot functionality. The implementation follows the same pattern as regular reboots but for `systemctl soft-reboot`, tracks systemd boot timestamps rather than kernel boot IDs for state detection. Co-Authored-By: Colin Walters <walters@verbum.org> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Colin Walters <walters@verbum.org> Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com>

cgwalters · 2025-06-23T17:20:01Z

+	// For soft-reboot, we don't change the boot ID, but the systemd SoftRebootsCount should increment
+	// We use systemd's SoftRebootsCount property to detect when the system has soft-rebooted
+	c := make(chan error)
+	go func() {


Completely unrelated to this but just for the future...I am increasingly of the opinion that the way tests should work is for the target machine to run autonomously, and report its status instead of being polled. IOW the kubernetes model, not the ansible model. It's a bit ironic that we ended up with a test system that follows more closely to the latter.

At least the external tests always run in a systemd unit. If we instead had an "agent reports status" model then this poll/retry code wouldn't be necessary. In fact the harness wouldn't even need to be aware of soft reboots in general!

But anyways, the test suite is what it is now.

cgwalters

Still LGTM (good catches on the further review!)

openshift-ci Bot added the do-not-merge/work-in-progress label May 30, 2025

jmarrero added a commit to jmarrero/ostree that referenced this pull request May 30, 2025

add kola test

1c51dcf

depends on: coreos/coreos-assembler#4119

cgwalters requested changes May 30, 2025

View reviewed changes

jmarrero force-pushed the soft-reboot branch from 23c3678 to 025b07e Compare June 2, 2025 19:38

jmarrero force-pushed the soft-reboot branch 2 times, most recently from 4944160 to d0743f6 Compare June 4, 2025 23:20

jmarrero mentioned this pull request Jun 5, 2025

overlay.d: Update ignition.firstboot services for soft-reboots coreos/fedora-coreos-config#3536

Merged

cgwalters mentioned this pull request Jun 5, 2025

Soft reboot prep #4133

Merged

cgwalters force-pushed the soft-reboot branch from d0743f6 to 01dc0a6 Compare June 5, 2025 21:20

cgwalters force-pushed the soft-reboot branch from 01dc0a6 to 1800086 Compare June 6, 2025 15:20

jmarrero force-pushed the soft-reboot branch from 1800086 to 5c37650 Compare June 6, 2025 16:26

jmarrero marked this pull request as ready for review June 6, 2025 17:36

openshift-ci Bot removed the do-not-merge/work-in-progress label Jun 6, 2025

jmarrero force-pushed the soft-reboot branch from 5c37650 to 6796892 Compare June 6, 2025 18:40

cgwalters previously approved these changes Jun 16, 2025

View reviewed changes

jlebon reviewed Jun 17, 2025

View reviewed changes

jmarrero dismissed cgwalters’s stale review via df29ea1 June 23, 2025 16:14

jmarrero force-pushed the soft-reboot branch from 6796892 to df29ea1 Compare June 23, 2025 16:14

cgwalters reviewed Jun 23, 2025

View reviewed changes

cgwalters requested a review from jlebon June 23, 2025 17:20

jlebon approved these changes Jun 23, 2025

View reviewed changes

jlebon enabled auto-merge (rebase) June 23, 2025 18:08

jlebon merged commit c65199b into coreos:main Jun 23, 2025
5 checks passed

jmarrero mentioned this pull request Aug 14, 2025

Leverage CoreOS test framework for our CI bootc-dev/bootc#1514

Open

cgwalters mentioned this pull request Nov 6, 2025

support soft reboots teemtee/tmt#4298

Closed

Conversation

jmarrero commented May 30, 2025

Uh oh!

openshift-ci Bot commented May 30, 2025

Uh oh!

dustymabe commented May 30, 2025

Uh oh!

jmarrero commented May 30, 2025

Uh oh!

dustymabe commented May 30, 2025

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

cgwalters May 30, 2025

Choose a reason for hiding this comment

Uh oh!

cgwalters May 30, 2025

Choose a reason for hiding this comment

Uh oh!

cgwalters May 30, 2025

Choose a reason for hiding this comment

Uh oh!

jmarrero commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmarrero commented Jun 4, 2025

Uh oh!

jmarrero commented Jun 4, 2025

Uh oh!

cgwalters commented Jun 5, 2025

Uh oh!

cgwalters commented Jun 5, 2025

Uh oh!

jmarrero commented Jun 8, 2025

Uh oh!

jmarrero commented Jun 16, 2025

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgwalters Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jmarrero commented Jun 3, 2025 •

edited

Loading