adaptor: add sandbox persistence and recovery across CAA restarts#3055
Open
thejasn wants to merge 15 commits into
Open
adaptor: add sandbox persistence and recovery across CAA restarts#3055thejasn wants to merge 15 commits into
thejasn wants to merge 15 commits into
Conversation
thejasn
force-pushed
the
thn/sandbox-persistence
branch
from
May 13, 2026 13:32
5a7f60d to
209b74e
Compare
thejasn
changed the title
thejasn
marked this pull request as ready for review
May 14, 2026 04:32
thejasn
force-pushed
the
thn/sandbox-persistence
branch
from
May 14, 2026 05:29
209b74e to
10f23af
Compare
Introduce a file-based state manager that persists sandbox metadata
(instance ID, network config, forwarder port, etc.) to disk so that
CAA can recover running sandboxes after a restart without destroying
the underlying VMs.
State is saved atomically under /run/peerpod/pods/{sandboxID}/state.json.
Signed-off-by: Thejas N <thn@redhat.com>
Add LoadOrCreateTLSMaterial() which saves the CA certificate and client key pair to disk on first run, and reloads them on subsequent starts. This allows a restarted CAA to re-establish mutually authenticated TLS connections to forwarders in existing pod VMs. Signed-off-by: Thejas N <thn@redhat.com>
A restarted CAA generates a new CA and client key pair. Existing forwarders in running pod VMs still hold the old CA's server certificate, so the new CAA cannot establish mutually authenticated TLS connections to them, breaking reconnection to surviving sandboxes. Persist the TLS material to disk on first run and reload it on subsequent starts so both sides of the mTLS handshake agree on the same CA. Signed-off-by: Thejas N <thn@redhat.com>
The pod index counter resets to zero on CAA restart. Recovered sandboxes retain their original VXLAN IDs (derived from pod index at creation time), so new sandboxes allocated from a reset counter collide on the same VXLAN IDs, breaking network connectivity for both old and new pods. Allow the recovery path to advance the counter past all restored sandboxes before any new ones are created. Signed-off-by: Thejas N <thn@redhat.com>
Without recovery, a CAA restart orphans all running pod VMs. The shim holds the sandbox open, but no agent proxy exists to forward ttrpc requests, so exec, logs, and lifecycle operations hang until timeout. On startup, scan persisted state files and attempt to reconnect to each sandbox that was in the Running phase. For sandboxes in any other phase, tear down host-side network resources and delete stale state. Signed-off-by: Thejas N <thn@redhat.com>
During CAA shutdown, the agent proxy closes immediately, severing any in-flight ttrpc request between the shim and kata-agent. The shim sees a broken pipe and may mark the container as failed. Wait up to 30 seconds for pending requests to complete before closing the connection to the forwarder. Signed-off-by: Thejas N <thn@redhat.com>
Teardown extracts VXLAN parameters by reading the device from the pod network namespace. After a CAA restart the pod netns may be inaccessible (stale mount or cleaned-up namespace), causing teardown to fail and leak iptables rules on the host. Persist the destination IP during Setup so teardown can clean up host-side iptables rules using only persisted state, even when the pod netns is gone. Signed-off-by: Thejas N <thn@redhat.com>
The restart-time check requires every peerpod's Ready condition to have transitioned after CAA's start time. With sandbox persistence, pods survive the restart with their original transition time, so this check never passes and the startup probe blocks indefinitely. Remove the check. The remaining Ready condition check is sufficient to verify existing pods are healthy after recovery. Signed-off-by: Thejas N <thn@redhat.com>
The startup probe dials the hypervisor socket to verify the ttrpc server is accepting connections. This couples probe liveness to ttrpc server readiness, which is unnecessary — the kubelet's own container startup timeout already covers the case where the server never comes up. Signed-off-by: Thejas N <thn@redhat.com>
When CAA restarts, it opens a new TCP connection to the forwarder. The old connection's ttrpc Serve goroutine is still blocked in a read loop. When the shim sends a request, ttrpc may route it to the stale session whose underlying connection is half-closed, returning "ttrpc: closed". The kata-agent receives a SIGKILL (exit code 9) and the container restarts. Close the previous connection when a new one arrives so the old Serve goroutine exits cleanly. The forwarder-to-kata-agent Unix socket is unaffected, only the CAA-to-forwarder TCP leg is replaced. Signed-off-by: Thejas N <thn@redhat.com>
After a CAA restart, the new process cannot bind the probe port because the old socket lingers in TIME_WAIT. Set SO_REUSEPORT so the new process can bind immediately. The previous implementation also leaked the probe goroutine on context cancellation because http.ListenAndServe ignores cancellation. Use a context-aware listener so the probe server stops with the main process. Signed-off-by: Thejas N <thn@redhat.com>
When CAA shuts down (e.g., pod restart during rolling update), RemoveExtendedResources() removes the kata.peerpods.io/vm capacity from the node. This causes the kubelet to consider existing peerpod pods as unschedulable and evict them, defeating sandbox persistence across CAA restarts. Remove the RemoveExtendedResources() call from Shutdown() so that the node retains its peerpod capacity. The extended resources are re-advertised on startup via AdvertiseExtendedResources(), making the removal on shutdown unnecessary. Signed-off-by: Thejas N <thn@redhat.com>
Add missing configs in helm charts to support caa restarts Signed-off-by: Thejas N <thn@redhat.com>
During a rolling update, the old and new CAA pods share the same hostPath (/run/peerpod/). When the old pod's ttrpc.Shutdown() calls listener.Close(), it unlinks the socket file that the new pod already created, leaving the shim unable to connect. Set SetUnlinkOnClose(false) on the Unix listener so the old pod's shutdown does not remove the socket inode. The new pod's startup already handles cleanup via os.RemoveAll before creating a fresh socket. Signed-off-by: Thejas N <thn@redhat.com>
parser.ParseDir was deprecated in Go 1.25 as it ignores build tags, which can cause incorrect package-to-file associations. Signed-off-by: Thejas N <thn@redhat.com>
thejasn
force-pushed
the
thn/sandbox-persistence
branch
from
May 20, 2026 16:21
a764f6b to
8150ed8
Compare
Contributor
Author
|
rebased and pushed CI failure fixes (lint issues) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CAA stores all sandbox state in an in-memory map. When the CAA pod restarts, this state is lost. The new CAA instance has no record of existing sandboxes and does not recreate the agent proxy. The shim's ttrpc connection to agent.ttrpc goes stale, the shim's health monitor detects the broken connection and exits, and the pod restarts. A routine CAA daemonset update disrupts every peer pod workload on the node.
Solution
This PR persists sandbox state to disk so that a restarted CAA can reconnect to existing pod VMs instead of destroying them. After recovery, the shim's ttrpc requests resume flowing through the new agent proxy to the original kata-agent — containers continue running with zero restarts.
State is saved at each lifecycle transition: CreateVM persists initial metadata, StartVM updates instance info and network config, StopVM deletes state. On startup, CAA scans
/run/peerpod/pods/*for state files and reconnects to sandboxes that were in the Running phase. Sandboxes in any other phase are cleaned up.Additional Changes
Several subsystems assumed fresh-start semantics and needed fixes:
Testing
Tested on libvirt with a two-node kcli cluster (control-plane + worker) using a debug podvm image with a custom kata-agent and shim (includes changes from kata-containers/kata-containers#12945)
Test flow:
i=0; while true; do echo counter=$i; i=$((i+1)); sleep 1; done)Unit and e2e Tests
Fixes: rhjira#KATA-4915