feat(connect): detect existing instance before install (disconnect + reinstall or cancel)

CHANGELOG.md

@@ -290,6 +290,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   of the canonical BBO identity, so a count-only change at an unchanged price/size is a distinct quote.
 
 ### Changed
+- Installer (`scripts/connect*.sh`) now detects an **existing edge-connect instance** before
+  installing: if a container named `$DZ_NAME` (default `doublezero-edge-connect`) already exists on
+  the host, the installer warns (naming the instance's env/image, since all three installers share
+  `$DZ_NAME` — so e.g. the testnet installer flags a live *mainnet* container) and prompts to
+  **reinstall or cancel** instead of silently colliding with the live tunnel/ports. On reinstall it
+  prints "Uninstalling existing instance..." and, for a running instance, tears it down **gracefully
+  via `docker stop`** — the container entrypoint's SIGTERM trap runs a bounded `doublezero
+  disconnect` (releasing the GRE tunnel/routes/on-chain session) before doublezerod is killed —
+  rather than a raw, unbounded `docker exec … disconnect`; the stop is `timeout`-guarded so a wedged
+  or restarting container can't hang the installer. After removal it verifies the `doublezero1`
+  tunnel interface is actually gone from the host netns and warns loudly if it lingers (orphaned
+  session). Interactively, declining aborts and leaves the instance untouched; non-interactively
+  (`DZ_ASSUME_YES=1`, or no usable TTY) it reinstalls, preserving the previous silent-reinstall
+  behaviour for automation. TTY detection probes an actual `/dev/tty` open rather than trusting
+  `-r`, so a headless run with no controlling terminal is classified correctly. The env/image
+  labelling is best-effort — a `docker inspect` that fails mid-teardown (container removed between
+  detection and inspect, or a daemon blip) no longer aborts the installer under `set -o pipefail`.
+  Applied identically to all three installers; covered by `tests/scripts/reinstall_existing.bats`.
 - `dz_depth_dropped_total` now carries a `publisher` label (the dropped copy's source class),
   symmetric with `dz_depth_admitted_total`, so a lagging publisher losing the book race is
   directly visible (#66). This changes the label set of an existing series — exact-label matchers

scripts/connect-devnet.sh

@@ -362,6 +362,72 @@ if ! command -v docker >/dev/null 2>&1; then
 fi
 $SUDO docker info >/dev/null 2>&1 || die "Docker is installed but the daemon isn't reachable. Start it (e.g. 'sudo systemctl start docker') and re-run."
 
+# ----------------------------------------------------------------------------
+# 3b. existing instance? reinstall (graceful teardown) or cancel
+# ----------------------------------------------------------------------------
+# A prior run leaves a long-lived container (--restart unless-stopped) that holds
+# the DoubleZero tunnel and the WS sink. Re-running the installer over a live
+# instance would collide (same container name, same tunnel, same host ports), so
+# detect one up front and tear it down cleanly first.
+reinstall_existing_instance() {
+  $SUDO docker ps -a -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . || return 0
+
+  local running="" existing_env="" existing_img=""
+  $SUDO docker ps -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . && running=1
+  # Label the victim: all three installers share DZ_NAME, so e.g. the testnet
+  # installer can find a live *mainnet* container -- name the network/image so the
+  # operator isn't asked to destroy an unidentified instance.
+  existing_env="$($SUDO docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$DZ_NAME" 2>/dev/null | sed -n 's/^DZ_ENV=//p' | head -1 || true)"
+  existing_img="$($SUDO docker inspect -f '{{.Config.Image}}' "$DZ_NAME" 2>/dev/null || true)"
+  warn "An edge-connect instance ('$DZ_NAME'${existing_env:+, env=$existing_env}${existing_img:+, image=$existing_img}) already exists on this host${running:+ and is running}."
+
+  # Decide whether to reinstall, keeping the three cases distinct so we neither
+  # break headless automation nor mislabel a genuine decline:
+  #   DZ_ASSUME_YES=1     -> reinstall (skip the prompt)
+  #   interactive decline -> abort (the operator said no)
+  #   no usable TTY, !yes -> reinstall, but say so (pre-3b behaviour was a silent
+  #                          reinstall; keep automation working rather than abort)
+  # A readable /dev/tty inode (-r) can still fail to OPEN with no controlling
+  # terminal (cron/systemd/`curl|bash` without a tty), so probe an actual open
+  # rather than trusting -r (and skip confirm()'s own tty read in that case).
+  if [ "$DZ_ASSUME_YES" = 1 ]; then
+    :
+  elif { : <"$TTY"; } 2>/dev/null; then
+    confirm "Reinstall? This disconnects and removes the existing instance" \
+      || die "Cancelled: leaving the existing instance in place (manage it with 'sudo docker logs $DZ_NAME')."
+  else
+    warn "No terminal to prompt on and DZ_ASSUME_YES is unset; reinstalling to preserve non-interactive behaviour (set DZ_ASSUME_YES=1 to silence this, or run interactively to be asked first)."
+  fi
+
+  info "Uninstalling existing instance..."
+  if [ -n "$running" ]; then
+    # Graceful: `docker stop` sends SIGTERM, which the container entrypoint traps to
+    # run a bounded `doublezero disconnect` (only if a tunnel is up) before tearing
+    # doublezerod down -- releasing the GRE tunnel/routes/on-chain session cleanly.
+    # The container's --stop-timeout (60s) bounds the stop; the outer `timeout`
+    # guards a wedged docker CLI / restarting container so we can never hang forever.
+    info "Stopping it gracefully (disconnecting the DoubleZero tunnel)..."
+    local stop_ok=1
+    if command -v timeout >/dev/null 2>&1; then
+      $SUDO timeout 90 docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    else
+      $SUDO docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    fi
+    [ "$stop_ok" = 1 ] || warn "Could not stop the existing container cleanly (timed out or errored); forcing removal. Its GRE tunnel/routes may be orphaned in the host network namespace -- check 'doublezero status' / 'ip link' and disconnect manually if connectivity is off."
+  fi
+  $SUDO docker rm -f "$DZ_NAME" >/dev/null 2>&1 || true
+
+  # Version-independent confirmation that the tunnel actually came down: the
+  # container ran with --network host, so a failed disconnect leaves the doublezero1
+  # interface (and its routes/on-chain session) orphaned in the host netns. Warn
+  # loudly if it lingers -- the fresh connect below usually recreates it, but a
+  # leftover old tunnel means the previous session wasn't released.
+  if command -v ip >/dev/null 2>&1 && $SUDO ip link show doublezero1 >/dev/null 2>&1; then
+    warn "The DoubleZero tunnel interface (doublezero1) is still present after teardown; the previous instance may not have disconnected cleanly. Its on-chain session/routes could be orphaned -- verify with 'doublezero status'."
+  fi
+}
+reinstall_existing_instance
+
 # ----------------------------------------------------------------------------
 # 4. host kernel / network prep (host-side; safe to attempt)
 # ----------------------------------------------------------------------------

scripts/connect-testnet.sh

@@ -357,6 +357,72 @@ if ! command -v docker >/dev/null 2>&1; then
 fi
 $SUDO docker info >/dev/null 2>&1 || die "Docker is installed but the daemon isn't reachable. Start it (e.g. 'sudo systemctl start docker') and re-run."
 
+# ----------------------------------------------------------------------------
+# 3b. existing instance? reinstall (graceful teardown) or cancel
+# ----------------------------------------------------------------------------
+# A prior run leaves a long-lived container (--restart unless-stopped) that holds
+# the DoubleZero tunnel and the WS sink. Re-running the installer over a live
+# instance would collide (same container name, same tunnel, same host ports), so
+# detect one up front and tear it down cleanly first.
+reinstall_existing_instance() {
+  $SUDO docker ps -a -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . || return 0
+
+  local running="" existing_env="" existing_img=""
+  $SUDO docker ps -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . && running=1
+  # Label the victim: all three installers share DZ_NAME, so e.g. the testnet
+  # installer can find a live *mainnet* container -- name the network/image so the
+  # operator isn't asked to destroy an unidentified instance.
+  existing_env="$($SUDO docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$DZ_NAME" 2>/dev/null | sed -n 's/^DZ_ENV=//p' | head -1 || true)"
+  existing_img="$($SUDO docker inspect -f '{{.Config.Image}}' "$DZ_NAME" 2>/dev/null || true)"
+  warn "An edge-connect instance ('$DZ_NAME'${existing_env:+, env=$existing_env}${existing_img:+, image=$existing_img}) already exists on this host${running:+ and is running}."
+
+  # Decide whether to reinstall, keeping the three cases distinct so we neither
+  # break headless automation nor mislabel a genuine decline:
+  #   DZ_ASSUME_YES=1     -> reinstall (skip the prompt)
+  #   interactive decline -> abort (the operator said no)
+  #   no usable TTY, !yes -> reinstall, but say so (pre-3b behaviour was a silent
+  #                          reinstall; keep automation working rather than abort)
+  # A readable /dev/tty inode (-r) can still fail to OPEN with no controlling
+  # terminal (cron/systemd/`curl|bash` without a tty), so probe an actual open
+  # rather than trusting -r (and skip confirm()'s own tty read in that case).
+  if [ "$DZ_ASSUME_YES" = 1 ]; then
+    :
+  elif { : <"$TTY"; } 2>/dev/null; then
+    confirm "Reinstall? This disconnects and removes the existing instance" \
+      || die "Cancelled: leaving the existing instance in place (manage it with 'sudo docker logs $DZ_NAME')."
+  else
+    warn "No terminal to prompt on and DZ_ASSUME_YES is unset; reinstalling to preserve non-interactive behaviour (set DZ_ASSUME_YES=1 to silence this, or run interactively to be asked first)."
+  fi
+
+  info "Uninstalling existing instance..."
+  if [ -n "$running" ]; then
+    # Graceful: `docker stop` sends SIGTERM, which the container entrypoint traps to
+    # run a bounded `doublezero disconnect` (only if a tunnel is up) before tearing
+    # doublezerod down -- releasing the GRE tunnel/routes/on-chain session cleanly.
+    # The container's --stop-timeout (60s) bounds the stop; the outer `timeout`
+    # guards a wedged docker CLI / restarting container so we can never hang forever.
+    info "Stopping it gracefully (disconnecting the DoubleZero tunnel)..."
+    local stop_ok=1
+    if command -v timeout >/dev/null 2>&1; then
+      $SUDO timeout 90 docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    else
+      $SUDO docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    fi
+    [ "$stop_ok" = 1 ] || warn "Could not stop the existing container cleanly (timed out or errored); forcing removal. Its GRE tunnel/routes may be orphaned in the host network namespace -- check 'doublezero status' / 'ip link' and disconnect manually if connectivity is off."
+  fi
+  $SUDO docker rm -f "$DZ_NAME" >/dev/null 2>&1 || true
+
+  # Version-independent confirmation that the tunnel actually came down: the
+  # container ran with --network host, so a failed disconnect leaves the doublezero1
+  # interface (and its routes/on-chain session) orphaned in the host netns. Warn
+  # loudly if it lingers -- the fresh connect below usually recreates it, but a
+  # leftover old tunnel means the previous session wasn't released.
+  if command -v ip >/dev/null 2>&1 && $SUDO ip link show doublezero1 >/dev/null 2>&1; then
+    warn "The DoubleZero tunnel interface (doublezero1) is still present after teardown; the previous instance may not have disconnected cleanly. Its on-chain session/routes could be orphaned -- verify with 'doublezero status'."
+  fi
+}
+reinstall_existing_instance
+
 # ----------------------------------------------------------------------------
 # 4. host kernel / network prep (host-side; safe to attempt)
 # ----------------------------------------------------------------------------

scripts/connect.sh

@@ -357,6 +357,72 @@ if ! command -v docker >/dev/null 2>&1; then
 fi
 $SUDO docker info >/dev/null 2>&1 || die "Docker is installed but the daemon isn't reachable. Start it (e.g. 'sudo systemctl start docker') and re-run."
 
+# ----------------------------------------------------------------------------
+# 3b. existing instance? reinstall (graceful teardown) or cancel
+# ----------------------------------------------------------------------------
+# A prior run leaves a long-lived container (--restart unless-stopped) that holds
+# the DoubleZero tunnel and the WS sink. Re-running the installer over a live
+# instance would collide (same container name, same tunnel, same host ports), so
+# detect one up front and tear it down cleanly first.
+reinstall_existing_instance() {
+  $SUDO docker ps -a -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . || return 0
+
+  local running="" existing_env="" existing_img=""
+  $SUDO docker ps -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . && running=1
+  # Label the victim: all three installers share DZ_NAME, so e.g. the testnet
+  # installer can find a live *mainnet* container -- name the network/image so the
+  # operator isn't asked to destroy an unidentified instance.
+  existing_env="$($SUDO docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$DZ_NAME" 2>/dev/null | sed -n 's/^DZ_ENV=//p' | head -1 || true)"
+  existing_img="$($SUDO docker inspect -f '{{.Config.Image}}' "$DZ_NAME" 2>/dev/null || true)"
+  warn "An edge-connect instance ('$DZ_NAME'${existing_env:+, env=$existing_env}${existing_img:+, image=$existing_img}) already exists on this host${running:+ and is running}."
+
+  # Decide whether to reinstall, keeping the three cases distinct so we neither
+  # break headless automation nor mislabel a genuine decline:
+  #   DZ_ASSUME_YES=1     -> reinstall (skip the prompt)
+  #   interactive decline -> abort (the operator said no)
+  #   no usable TTY, !yes -> reinstall, but say so (pre-3b behaviour was a silent
+  #                          reinstall; keep automation working rather than abort)
+  # A readable /dev/tty inode (-r) can still fail to OPEN with no controlling
+  # terminal (cron/systemd/`curl|bash` without a tty), so probe an actual open
+  # rather than trusting -r (and skip confirm()'s own tty read in that case).
+  if [ "$DZ_ASSUME_YES" = 1 ]; then
+    :
+  elif { : <"$TTY"; } 2>/dev/null; then
+    confirm "Reinstall? This disconnects and removes the existing instance" \
+      || die "Cancelled: leaving the existing instance in place (manage it with 'sudo docker logs $DZ_NAME')."
+  else
+    warn "No terminal to prompt on and DZ_ASSUME_YES is unset; reinstalling to preserve non-interactive behaviour (set DZ_ASSUME_YES=1 to silence this, or run interactively to be asked first)."
+  fi
+
+  info "Uninstalling existing instance..."
+  if [ -n "$running" ]; then
+    # Graceful: `docker stop` sends SIGTERM, which the container entrypoint traps to
+    # run a bounded `doublezero disconnect` (only if a tunnel is up) before tearing
+    # doublezerod down -- releasing the GRE tunnel/routes/on-chain session cleanly.
+    # The container's --stop-timeout (60s) bounds the stop; the outer `timeout`
+    # guards a wedged docker CLI / restarting container so we can never hang forever.
+    info "Stopping it gracefully (disconnecting the DoubleZero tunnel)..."
+    local stop_ok=1
+    if command -v timeout >/dev/null 2>&1; then
+      $SUDO timeout 90 docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    else
+      $SUDO docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    fi
+    [ "$stop_ok" = 1 ] || warn "Could not stop the existing container cleanly (timed out or errored); forcing removal. Its GRE tunnel/routes may be orphaned in the host network namespace -- check 'doublezero status' / 'ip link' and disconnect manually if connectivity is off."
+  fi
+  $SUDO docker rm -f "$DZ_NAME" >/dev/null 2>&1 || true
+
+  # Version-independent confirmation that the tunnel actually came down: the
+  # container ran with --network host, so a failed disconnect leaves the doublezero1
+  # interface (and its routes/on-chain session) orphaned in the host netns. Warn
+  # loudly if it lingers -- the fresh connect below usually recreates it, but a
+  # leftover old tunnel means the previous session wasn't released.
+  if command -v ip >/dev/null 2>&1 && $SUDO ip link show doublezero1 >/dev/null 2>&1; then
+    warn "The DoubleZero tunnel interface (doublezero1) is still present after teardown; the previous instance may not have disconnected cleanly. Its on-chain session/routes could be orphaned -- verify with 'doublezero status'."
+  fi
+}
+reinstall_existing_instance
+
 # ----------------------------------------------------------------------------
 # 4. host kernel / network prep (host-side; safe to attempt)
 # ----------------------------------------------------------------------------