Skip to content

spacetime start refuses to boot ("Port X is already in use") on Kubernetes/OpenShift and SELinux-confined Docker — port check fails closed when sock_diag is denied #5556

Description

@goern

Summary

Since v2.4.0, spacetime start runs is_port_available()
(crates/standalone/src/subcommands/start.rs) before binding. The check
enumerates sockets via the netstat2 crate, which on Linux uses a netlink
sock_diag (inet_diag) dump. Under SELinux container_t — the default for
OpenShift (any SCC, including restricted-v2), and for Docker/Podman on
RHEL-family hosts with typical policies — the dump is denied with EACCES.

is_port_available treats enumeration failure as "port in use":

Err(_) => return false, // if we can't inspect sockets, fail closed

so the requested port and all 100 "nearby" fallback ports appear taken, and
the server exits:

Error: Port 3000 is already in use and could not find an available port nearby.
Please free up the port or specify a different port with --listen-addr.

…even in a fresh pod with nothing listening. v2.2.1 (pre-check) boots fine in
the same environment.

Reproduction

OpenShift (or any Kubernetes with SELinux-enforcing nodes / container_t):

containers:
  - image: docker.io/clockworklabs/spacetime:v2.6.1
    args: ["start", "--data-dir=/data", "--listen-addr=0.0.0.0:3000"]
    env: [{name: HOME, value: /tmp}]

→ CrashLoopBackOff with the error above; /proc/net/tcp in the pod shows no
listener on 3000.

Minimal probe confirming the denial (same pod context, Python):
socket(AF_NETLINK, SOCK_RAW, NETLINK_SOCK_DIAG) succeeds, but the
SOCK_DIAG_BY_FAMILY dump request returns EACCES.

Suggested fix

Fail open when socket enumeration is unavailable — the authoritative check
is the TcpListener::bind a few lines later, which already produces a clear
error when the port is genuinely taken (the code even notes the check is
racy and bind is the backstop):

Err(_) => return true, // can't inspect sockets (e.g. SELinux denies sock_diag); let bind decide

Same applies to the parse_host None arm if the host string is unusual.

Environment

  • SpacetimeDB v2.6.1 (also inspected: check present v2.4.0 → master)
  • OpenShift 4.x / RHCOS, SELinux enforcing, restricted-v2 SCC
  • Image: docker.io/clockworklabs/spacetime:v2.6.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions