Skip to content

fix: Move /var/run/dstack.sock to /var/run/dstack/dstack.sock directory#471

Merged
kvinwang merged 2 commits into
masterfrom
fix/socket-path-for-docker-mount
Jan 27, 2026
Merged

fix: Move /var/run/dstack.sock to /var/run/dstack/dstack.sock directory#471
kvinwang merged 2 commits into
masterfrom
fix/socket-path-for-docker-mount

Conversation

@kvinwang

@kvinwang kvinwang commented Jan 27, 2026

Copy link
Copy Markdown
Collaborator

Problem

When containers mount the socket file directly (e.g., -v /var/run/dstack.sock:/var/run/dstack.sock), the socket becomes inaccessible after guest-agent restarts.

Root Cause

Docker's bind mount locks the inode at container start time, not the path. When guest-agent restarts:

  1. Rocket's Unix socket handling acquires an exclusive lock on <path>.lock
  2. If reuse=true (default), it deletes the existing socket file via remove_file()
  3. Creates a new socket via bind() - this creates a NEW inode
  4. The container still holds the old inode, which no longer has any listener

Why Rocket Deletes the Socket

Unlike TCP sockets, Unix Domain Sockets do not support SO_REUSEADDR. The bind() syscall requires the path to not exist - if the socket file already exists, bind() fails with EADDRINUSE.

Rocket's reuse option is a workaround that simulates TCP-like behavior by:

  1. Acquiring a file lock to prevent races
  2. Deleting the existing socket file
  3. Creating a new socket

This "delete and recreate" approach is necessary for UDS but breaks Docker's file-based mounts.

Relevant Rocket code (unix.rs:32-72):

pub async fn bind<P: AsRef<Path>>(path: P, reuse: bool) -> io::Result<Self> {
    let lock = if reuse {
        // ...acquire exclusive lock...
        if path.exists() {
            tokio::fs::remove_file(&path).await?;  // <-- Deletes old socket!
        }
        Some(lock_file)
    };
    let listener = tokio::net::UnixListener::bind(&path);  // <-- Creates new inode
    // ...
}

Solution

1. Move guest-agent to listen on new paths

Old Path New Path
/var/run/dstack.sock /var/run/dstack/dstack.sock
/var/run/tappd.sock /var/run/dstack/tappd.sock

2. Systemd socket proxy for backward compatibility

For containers that mount the old paths, we use systemd socket activation with systemd-socket-proxyd:

Container → /var/run/dstack.sock (systemd-owned) → proxy → /var/run/dstack/dstack.sock (guest-agent)

Key insight: systemd.socket creates and owns the socket file, not the service process. This means:

  • The socket file persists across proxy service restarts
  • The inode never changes
  • Containers mounting /var/run/dstack.sock continue to work even after guest-agent restarts

Files added:

  • dstack-socket.socket + dstack-socket.service
  • tappd-socket.socket + tappd-socket.service

3. SDK Fallback

All SDKs (Rust, Python, JS, Go) try the new path first, falling back to the old path for compatibility with older guest-agent versions.

User Impact

None - Apps can continue using /var/run/dstack.sock and /var/run/tappd.sock as before. The systemd socket proxy handles the forwarding transparently.

Changes

  • guest-agent/dstack.toml: Updated socket paths to /var/run/dstack/
  • dstack-util/src/system_setup.rs: Create /var/run/dstack/ directory
  • basefiles/: Added systemd socket + proxy service files for backward compatibility
  • SDK updates: Rust, Python, JS, Go clients try new path first with fallback

@kvinwang kvinwang changed the title fix: Move socket to /var/run/dstack/ directory for Docker mount compatibility fix: Move /var/run/dstack.sock to /var/run/dstack/dstack.sock directory Jan 27, 2026
@kvinwang
kvinwang force-pushed the fix/socket-path-for-docker-mount branch 2 times, most recently from a3b76d1 to 34483a0 Compare January 27, 2026 04:14
…tibility

## Problem

When containers mount the socket file directly (e.g., `-v /var/run/dstack.sock:/var/run/dstack.sock`),
the socket becomes inaccessible after guest-agent restarts. This is because:

1. Docker's bind mount locks the inode at container start time
2. When guest-agent restarts, Rocket's Unix socket handling deletes and recreates
   the socket file, creating a NEW inode
3. The container still holds the old inode, which no longer has any listener

### Why Rocket Deletes the Socket

Unix Domain Sockets do NOT support `SO_REUSEADDR` like TCP sockets. The `bind()`
syscall requires the path to not exist - if the socket file already exists, `bind()`
fails with `EADDRINUSE`. Rocket's `reuse` option simulates TCP-like behavior by
deleting the existing socket before creating a new one.

## Solution

Move sockets to a dedicated directory `/var/run/dstack/`:
- `/var/run/dstack/dstack.sock`
- `/var/run/dstack/tappd.sock`

All SDKs (Rust, Python, JS, Go) try the new path first, falling back to the old
path for backward compatibility with older guest-agent versions.
Add systemd socket activation files to provide backward compatibility
for containers mounting /var/run/dstack.sock directly.

- dstack-socket.socket/service: Proxy for /var/run/dstack.sock
- tappd-socket.socket/service: Proxy for /var/run/tappd.sock

Uses systemd-socket-proxyd to forward connections from the legacy paths
to the new paths in /var/run/dstack/. Since systemd owns the socket file
(not the service), the inode persists across guest-agent restarts.
@kvinwang
kvinwang force-pushed the fix/socket-path-for-docker-mount branch from fcc9e52 to d685e07 Compare January 27, 2026 06:16
@kvinwang
kvinwang enabled auto-merge January 27, 2026 06:17
@kvinwang
kvinwang merged commit c92c2b4 into master Jan 27, 2026
11 checks passed
@kvinwang
kvinwang deleted the fix/socket-path-for-docker-mount branch July 20, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant