Skip to content

Commit fcbcf54

Browse files
shayonjgvisor-bot
authored andcommitted
hostinet: support checkpoint/restore
Host networking did not have checkpoint/restore support. A sandbox running with `--network=host` could not be checkpointed because `hostinet.Stack` was not savable, and restore did not define what should happen to open host sockets. This change makes the hostinet stack savable and defines that restore contract. Host socket fds cannot cross a checkpoint boundary, so `Socket.fd` is tagged `state:"nosave"` and save does not touch host sockets. With checkpoint `--leave-running`, the original sandbox keeps serving traffic on its existing connections. If save fails, the sandbox is left unharmed. Without `--leave-running`, the sandbox exits after save and the host kernel closes its sockets. That sends `FIN` or `RST` so remote peers do not hang on half-open connections. In the restored sandbox, `afterLoad` sets each socket fd to `-1`. Host syscalls then fail with `EBADF`. `Readiness` reports hangup and error so `epoll` waiters wake immediately, and tasks that were blocked in I/O see `EBADF` when their syscall restarts. `State()` reports no TCP state for these sockets rather than logging a warning on every `/proc/net/tcp` read. Applications are expected to reconnect after restore. The stack's host-derived state is not saved. This includes `/proc/net/dev`, `/proc/net/snmp`, TCP buffer sizes, and allowed socket types. During restore, runsc configures a fresh hostinet stack before seccomp filters are installed. `ReplaceConfig` then copies the destination host configuration into the deserialized stack and transfers ownership of the fresh proc net handles. The restored sandbox therefore uses the destination host configuration and limits. A hostinet checkpoint contains no netstack state and a netstack checkpoint contains no hostinet state, so the checkpoint now records the network type in its metadata. Restore fails with a clear error when the network stack kind differs between checkpoint and restore, instead of panicking inside `ReplaceConfig` during kernel load. Sandbox and none networking both use netstack and remain interchangeable. Checkpoints taken before the network type was recorded skip this check. The hostinet `save_resume` syscall tests are enabled because save does not touch host sockets. The save and restore syscall tests remain excluded because they expect sockets to keep working after restore. Container tests cover restored socket behavior for TCP, UDP, `epoll`, and a blocked accept. They also cover remote peer close after checkpoint, continued service after checkpoint `--leave-running`, and the error when restoring a host networking checkpoint with sandbox networking. The focused hostinet unit tests and the hostinet `save_resume` syscall tests pass on `x86_64`. The checkpoint/restore user guide documents the restored socket behavior. Fixes #6243 FUTURE_COPYBARA_INTEGRATE_REVIEW=#13404 from shayonj:hostinet-save-restore-v2 734480e PiperOrigin-RevId: 940513420
1 parent 2153438 commit fcbcf54

15 files changed

Lines changed: 1179 additions & 31 deletions

File tree

g3doc/user_guide/checkpoint_restore.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,25 @@ docker start --checkpoint <checkpoint-name> <container-name>
150150
`--checkpoint-dir` flag but this will be required when restoring from a
151151
checkpoint made in another container.
152152

153+
## Networking
154+
155+
Checkpoint/restore is supported with `--network=sandbox` (default),
156+
`--network=none`, and `--network=host`.
157+
158+
With `--network=host`, host sockets cannot be saved, so:
159+
160+
- Checkpoint with `--leave-running` does not touch the running sandbox's
161+
sockets. It keeps using them as before.
162+
- TCP listening sockets are re-created during restore and keep accepting new
163+
connections. Connections that were pending in the backlog at checkpoint time
164+
are lost. If the listen address cannot be bound on the restoring host, the
165+
restore fails.
166+
- Sockets that were connected at checkpoint time return `ECONNABORTED`, and
167+
`epoll_wait` on them returns `EPOLLERR | EPOLLHUP` immediately. Applications
168+
must reconnect after restore.
169+
- Network configuration visible inside the sandbox (interface statistics, TCP
170+
buffer sizes) reflects the host the sandbox was restored on.
171+
153172
## Checkpoint & Restore with different CPU features
154173

155174
When restoring a state file, gVisor verifies that the target host machine

pkg/sentry/socket/hostinet/BUILD

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,27 @@
1-
load("//tools:defs.bzl", "go_library")
1+
load("//tools:defs.bzl", "go_library", "go_test")
22

33
package(
44
default_applicable_licenses = ["//:license"],
55
licenses = ["notice"],
66
)
77

8+
go_test(
9+
name = "hostinet_test",
10+
size = "small",
11+
srcs = ["stack_test.go"],
12+
library = ":hostinet",
13+
deps = [
14+
"//pkg/abi/linux",
15+
"@org_golang_x_sys//unix:go_default_library",
16+
],
17+
)
18+
819
go_library(
920
name = "hostinet",
1021
srcs = [
1122
"hostinet.go",
1223
"netlink.go",
24+
"save_restore.go",
1325
"socket.go",
1426
"socket_unsafe.go",
1527
"sockopt.go",
Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
// Copyright 2026 The gVisor Authors.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package hostinet
16+
17+
import (
18+
"context"
19+
"fmt"
20+
21+
"golang.org/x/sys/unix"
22+
"gvisor.dev/gvisor/pkg/fdnotifier"
23+
"gvisor.dev/gvisor/pkg/log"
24+
"gvisor.dev/gvisor/pkg/sync"
25+
)
26+
27+
// Host socket fds cannot cross a checkpoint boundary, so Socket.fd is not
28+
// saved and the checkpointed sandbox retains ownership of it.
29+
//
30+
// Listening sockets carry no connection state, so beforeSave records what is
31+
// needed to re-create them and Stack.Restore re-creates the host socket with
32+
// socket/bind/listen. Connections that were pending in the backlog are lost.
33+
//
34+
// All other sockets are restored with fd -1, so host socket operations fail
35+
// with ECONNABORTED and Readiness reports hangup/error to wake pollers
36+
// immediately.
37+
38+
// listenerState describes a listening host socket so restore can re-create it.
39+
//
40+
// +stateify savable
41+
type listenerState struct {
42+
addr []byte
43+
backlog int32
44+
reuseAddr int32
45+
reusePort int32
46+
v6Only int32
47+
bindToDevice string
48+
}
49+
50+
var restoredListeners struct {
51+
mu sync.Mutex
52+
sockets []*Socket
53+
}
54+
55+
// beforeSave is invoked by stateify.
56+
func (s *Socket) beforeSave() {
57+
s.savedListener = nil
58+
if s.fd < 0 {
59+
return
60+
}
61+
accepting, err := unix.GetsockoptInt(s.fd, unix.SOL_SOCKET, unix.SO_ACCEPTCONN)
62+
if err != nil || accepting == 0 {
63+
return
64+
}
65+
addr, err := getsockname(s.fd)
66+
if err != nil {
67+
panic(fmt.Sprintf("getsockname on listening host socket failed during save: %v", err))
68+
}
69+
l := &listenerState{
70+
addr: addr,
71+
backlog: s.listenBacklog.Load(),
72+
}
73+
if v, err := unix.GetsockoptInt(s.fd, unix.SOL_SOCKET, unix.SO_REUSEADDR); err == nil {
74+
l.reuseAddr = int32(v)
75+
}
76+
if v, err := unix.GetsockoptInt(s.fd, unix.SOL_SOCKET, unix.SO_REUSEPORT); err == nil {
77+
l.reusePort = int32(v)
78+
}
79+
if s.family == unix.AF_INET6 {
80+
if v, err := unix.GetsockoptInt(s.fd, unix.IPPROTO_IPV6, unix.IPV6_V6ONLY); err == nil {
81+
l.v6Only = int32(v)
82+
}
83+
}
84+
if dev, err := unix.GetsockoptString(s.fd, unix.SOL_SOCKET, unix.SO_BINDTODEVICE); err == nil {
85+
l.bindToDevice = dev
86+
}
87+
s.savedListener = l
88+
}
89+
90+
// afterLoad is invoked by stateify.
91+
func (s *Socket) afterLoad(context.Context) {
92+
s.fd = -1
93+
if s.savedListener != nil {
94+
restoredListeners.mu.Lock()
95+
restoredListeners.sockets = append(restoredListeners.sockets, s)
96+
restoredListeners.mu.Unlock()
97+
}
98+
}
99+
100+
// restoreListeners re-creates the host sockets for saved listening sockets.
101+
func restoreListeners() {
102+
restoredListeners.mu.Lock()
103+
sockets := restoredListeners.sockets
104+
restoredListeners.sockets = nil
105+
restoredListeners.mu.Unlock()
106+
for _, s := range sockets {
107+
if err := s.restoreListener(); err != nil {
108+
log.Warningf("Failed to restore listening host socket (family=%d, addr=%x): %v", s.family, s.savedListener.addr, err)
109+
s.savedListener = nil
110+
}
111+
}
112+
}
113+
114+
// restoreListener attempts to re-create a saved listening socket. If any host
115+
// operation fails, the socket is left unrestored with fd -1 and restore
116+
// continues with other sockets.
117+
func (s *Socket) restoreListener() error {
118+
l := s.savedListener
119+
fd, err := unix.Socket(s.family, int(s.stype)|unix.SOCK_NONBLOCK|unix.SOCK_CLOEXEC, s.protocol)
120+
if err != nil {
121+
return fmt.Errorf("creating socket: %w", err)
122+
}
123+
if l.reuseAddr != 0 {
124+
if err := unix.SetsockoptInt(fd, unix.SOL_SOCKET, unix.SO_REUSEADDR, int(l.reuseAddr)); err != nil {
125+
_ = unix.Close(fd)
126+
return fmt.Errorf("setting SO_REUSEADDR: %w", err)
127+
}
128+
}
129+
if l.reusePort != 0 {
130+
if err := unix.SetsockoptInt(fd, unix.SOL_SOCKET, unix.SO_REUSEPORT, int(l.reusePort)); err != nil {
131+
_ = unix.Close(fd)
132+
return fmt.Errorf("setting SO_REUSEPORT: %w", err)
133+
}
134+
}
135+
if s.family == unix.AF_INET6 {
136+
if err := unix.SetsockoptInt(fd, unix.IPPROTO_IPV6, unix.IPV6_V6ONLY, int(l.v6Only)); err != nil {
137+
_ = unix.Close(fd)
138+
return fmt.Errorf("setting IPV6_V6ONLY: %w", err)
139+
}
140+
}
141+
if l.bindToDevice != "" {
142+
if err := unix.SetsockoptString(fd, unix.SOL_SOCKET, unix.SO_BINDTODEVICE, l.bindToDevice); err != nil {
143+
_ = unix.Close(fd)
144+
return fmt.Errorf("setting SO_BINDTODEVICE to %q: %w", l.bindToDevice, err)
145+
}
146+
}
147+
if err := bind(fd, l.addr); err != nil {
148+
_ = unix.Close(fd)
149+
return fmt.Errorf("binding: %w", err)
150+
}
151+
if err := unix.Listen(fd, int(l.backlog)); err != nil {
152+
_ = unix.Close(fd)
153+
return fmt.Errorf("listening: %w", err)
154+
}
155+
if err := fdnotifier.AddFD(int32(fd), &s.queue); err != nil {
156+
_ = unix.Close(fd)
157+
return fmt.Errorf("registering with fdnotifier: %w", err)
158+
}
159+
s.fd = fd
160+
s.savedListener = nil
161+
return nil
162+
}

0 commit comments

Comments
 (0)