feat(connect): detect existing instance before install (disconnect + reinstall or cancel) (#76)

juan-malbeclabs · web-flow · commit 705015c590b0 · 2026-07-10T14:09:42.000-04:00
* feat(connect): detect existing instance before install; disconnect + reinstall or cancel

Re-running the one-liner over a live edge-connect container would collide with
the existing tunnel, container name, and host ports. The installers now detect a
container named $DZ_NAME before any host prep or docker run, warn that an
instance already exists, and prompt to reinstall or cancel. On reinstall they
print "Uninstalling existing instance...", run `doublezero disconnect` inside the
old container to drop the tunnel cleanly, then remove it. On cancel they abort
and leave the running instance untouched. Non-interactive runs (DZ_ASSUME_YES=1)
reinstall. Applied identically to all three installers.

* fix(connect): graceful, bounded teardown of an existing instance; robust headless detection

Address review of the existing-instance guard:

- Graceful, bounded teardown: replace the raw unbounded `docker exec … doublezero
  disconnect` with `docker stop`, so the container entrypoint's SIGTERM trap runs
  its own bounded disconnect (releasing the GRE tunnel/routes/on-chain session)
  before doublezerod is killed. The stop is timeout-guarded so a wedged or
  restarting container can't hang the installer forever.
- Warn loudly on orphaned state: after removal, verify the doublezero1 tunnel
  interface is gone from the host netns (the container is --network host) and warn
  if it lingers, and warn if the stop timed out/errored before the forced removal.
- Preserve headless automation: a no-TTY run with DZ_ASSUME_YES unset now
  reinstalls (with a notice) instead of aborting, matching pre-guard behaviour;
  an interactive decline still aborts, with distinct messaging.
- Robust TTY detection: probe an actual /dev/tty open rather than trusting `-r`
  (a readable inode still fails to open with no controlling terminal), which also
  sidesteps confirm()'s tty read in that case.
- Network-aware prompt: name the existing instance's env/image (all three
  installers share DZ_NAME, so the testnet installer can find a live mainnet one).
- Add tests/scripts/reinstall_existing.bats: running (stop-before-rm ordering),
  stopped (no stop), no-instance (no-op), and headless (reinstall, not abort).

* fix(connect): guard the env-label command sub against set -o pipefail abort

The existing-instance env label built its value with a bare
`docker inspect … | sed | head -1` command substitution, missing the `|| true`
its DZ_IMAGE sibling has. Under `set -euo pipefail` a failing docker inspect
(container removed between the `ps -a` detection and the inspect, or a daemon
blip) — or `head` closing the pipe early on multi-line env — makes the pipeline
non-zero and aborts the whole installer before the "already exists" warning even
prints. Add `|| true` so the label stays best-effort.

Also harden the bats stub: `docker inspect` now emits realistic multi-line env +
an image (exercising the DZ_ENV extraction and the head-closes-pipe path) and can
be told to fail via STUB_INSPECT_FAIL. New test asserts a mid-teardown inspect
failure still detects, warns, and reinstalls; the running-instance test now
asserts the warning names the victim's env/image.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -290,6 +290,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   of the canonical BBO identity, so a count-only change at an unchanged price/size is a distinct quote.
 
 ### Changed
+- Installer (`scripts/connect*.sh`) now detects an **existing edge-connect instance** before
+  installing: if a container named `$DZ_NAME` (default `doublezero-edge-connect`) already exists on
+  the host, the installer warns (naming the instance's env/image, since all three installers share
+  `$DZ_NAME` — so e.g. the testnet installer flags a live *mainnet* container) and prompts to
+  **reinstall or cancel** instead of silently colliding with the live tunnel/ports. On reinstall it
+  prints "Uninstalling existing instance..." and, for a running instance, tears it down **gracefully
+  via `docker stop`** — the container entrypoint's SIGTERM trap runs a bounded `doublezero
+  disconnect` (releasing the GRE tunnel/routes/on-chain session) before doublezerod is killed —
+  rather than a raw, unbounded `docker exec … disconnect`; the stop is `timeout`-guarded so a wedged
+  or restarting container can't hang the installer. After removal it verifies the `doublezero1`
+  tunnel interface is actually gone from the host netns and warns loudly if it lingers (orphaned
+  session). Interactively, declining aborts and leaves the instance untouched; non-interactively
+  (`DZ_ASSUME_YES=1`, or no usable TTY) it reinstalls, preserving the previous silent-reinstall
+  behaviour for automation. TTY detection probes an actual `/dev/tty` open rather than trusting
+  `-r`, so a headless run with no controlling terminal is classified correctly. The env/image
+  labelling is best-effort — a `docker inspect` that fails mid-teardown (container removed between
+  detection and inspect, or a daemon blip) no longer aborts the installer under `set -o pipefail`.
+  Applied identically to all three installers; covered by `tests/scripts/reinstall_existing.bats`.
 - `dz_depth_dropped_total` now carries a `publisher` label (the dropped copy's source class),
   symmetric with `dz_depth_admitted_total`, so a lagging publisher losing the book race is
   directly visible (#66). This changes the label set of an existing series — exact-label matchers
diff --git a/scripts/connect-devnet.sh b/scripts/connect-devnet.sh
@@ -362,6 +362,72 @@ if ! command -v docker >/dev/null 2>&1; then
 fi
 $SUDO docker info >/dev/null 2>&1 || die "Docker is installed but the daemon isn't reachable. Start it (e.g. 'sudo systemctl start docker') and re-run."
 
+# ----------------------------------------------------------------------------
+# 3b. existing instance? reinstall (graceful teardown) or cancel
+# ----------------------------------------------------------------------------
+# A prior run leaves a long-lived container (--restart unless-stopped) that holds
+# the DoubleZero tunnel and the WS sink. Re-running the installer over a live
+# instance would collide (same container name, same tunnel, same host ports), so
+# detect one up front and tear it down cleanly first.
+reinstall_existing_instance() {
+  $SUDO docker ps -a -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . || return 0
+
+  local running="" existing_env="" existing_img=""
+  $SUDO docker ps -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . && running=1
+  # Label the victim: all three installers share DZ_NAME, so e.g. the testnet
+  # installer can find a live *mainnet* container -- name the network/image so the
+  # operator isn't asked to destroy an unidentified instance.
+  existing_env="$($SUDO docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$DZ_NAME" 2>/dev/null | sed -n 's/^DZ_ENV=//p' | head -1 || true)"
+  existing_img="$($SUDO docker inspect -f '{{.Config.Image}}' "$DZ_NAME" 2>/dev/null || true)"
+  warn "An edge-connect instance ('$DZ_NAME'${existing_env:+, env=$existing_env}${existing_img:+, image=$existing_img}) already exists on this host${running:+ and is running}."
+
+  # Decide whether to reinstall, keeping the three cases distinct so we neither
+  # break headless automation nor mislabel a genuine decline:
+  #   DZ_ASSUME_YES=1     -> reinstall (skip the prompt)
+  #   interactive decline -> abort (the operator said no)
+  #   no usable TTY, !yes -> reinstall, but say so (pre-3b behaviour was a silent
+  #                          reinstall; keep automation working rather than abort)
+  # A readable /dev/tty inode (-r) can still fail to OPEN with no controlling
+  # terminal (cron/systemd/`curl|bash` without a tty), so probe an actual open
+  # rather than trusting -r (and skip confirm()'s own tty read in that case).
+  if [ "$DZ_ASSUME_YES" = 1 ]; then
+    :
+  elif { : <"$TTY"; } 2>/dev/null; then
+    confirm "Reinstall? This disconnects and removes the existing instance" \
+      || die "Cancelled: leaving the existing instance in place (manage it with 'sudo docker logs $DZ_NAME')."
+  else
+    warn "No terminal to prompt on and DZ_ASSUME_YES is unset; reinstalling to preserve non-interactive behaviour (set DZ_ASSUME_YES=1 to silence this, or run interactively to be asked first)."
+  fi
+
+  info "Uninstalling existing instance..."
+  if [ -n "$running" ]; then
+    # Graceful: `docker stop` sends SIGTERM, which the container entrypoint traps to
+    # run a bounded `doublezero disconnect` (only if a tunnel is up) before tearing
+    # doublezerod down -- releasing the GRE tunnel/routes/on-chain session cleanly.
+    # The container's --stop-timeout (60s) bounds the stop; the outer `timeout`
+    # guards a wedged docker CLI / restarting container so we can never hang forever.
+    info "Stopping it gracefully (disconnecting the DoubleZero tunnel)..."
+    local stop_ok=1
+    if command -v timeout >/dev/null 2>&1; then
+      $SUDO timeout 90 docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    else
+      $SUDO docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    fi
+    [ "$stop_ok" = 1 ] || warn "Could not stop the existing container cleanly (timed out or errored); forcing removal. Its GRE tunnel/routes may be orphaned in the host network namespace -- check 'doublezero status' / 'ip link' and disconnect manually if connectivity is off."
+  fi
+  $SUDO docker rm -f "$DZ_NAME" >/dev/null 2>&1 || true
+
+  # Version-independent confirmation that the tunnel actually came down: the
+  # container ran with --network host, so a failed disconnect leaves the doublezero1
+  # interface (and its routes/on-chain session) orphaned in the host netns. Warn
+  # loudly if it lingers -- the fresh connect below usually recreates it, but a
+  # leftover old tunnel means the previous session wasn't released.
+  if command -v ip >/dev/null 2>&1 && $SUDO ip link show doublezero1 >/dev/null 2>&1; then
+    warn "The DoubleZero tunnel interface (doublezero1) is still present after teardown; the previous instance may not have disconnected cleanly. Its on-chain session/routes could be orphaned -- verify with 'doublezero status'."
+  fi
+}
+reinstall_existing_instance
+
 # ----------------------------------------------------------------------------
 # 4. host kernel / network prep (host-side; safe to attempt)
 # ----------------------------------------------------------------------------
diff --git a/scripts/connect-testnet.sh b/scripts/connect-testnet.sh
@@ -357,6 +357,72 @@ if ! command -v docker >/dev/null 2>&1; then
 fi
 $SUDO docker info >/dev/null 2>&1 || die "Docker is installed but the daemon isn't reachable. Start it (e.g. 'sudo systemctl start docker') and re-run."
 
+# ----------------------------------------------------------------------------
+# 3b. existing instance? reinstall (graceful teardown) or cancel
+# ----------------------------------------------------------------------------
+# A prior run leaves a long-lived container (--restart unless-stopped) that holds
+# the DoubleZero tunnel and the WS sink. Re-running the installer over a live
+# instance would collide (same container name, same tunnel, same host ports), so
+# detect one up front and tear it down cleanly first.
+reinstall_existing_instance() {
+  $SUDO docker ps -a -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . || return 0
+
+  local running="" existing_env="" existing_img=""
+  $SUDO docker ps -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . && running=1
+  # Label the victim: all three installers share DZ_NAME, so e.g. the testnet
+  # installer can find a live *mainnet* container -- name the network/image so the
+  # operator isn't asked to destroy an unidentified instance.
+  existing_env="$($SUDO docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$DZ_NAME" 2>/dev/null | sed -n 's/^DZ_ENV=//p' | head -1 || true)"
+  existing_img="$($SUDO docker inspect -f '{{.Config.Image}}' "$DZ_NAME" 2>/dev/null || true)"
+  warn "An edge-connect instance ('$DZ_NAME'${existing_env:+, env=$existing_env}${existing_img:+, image=$existing_img}) already exists on this host${running:+ and is running}."
+
+  # Decide whether to reinstall, keeping the three cases distinct so we neither
+  # break headless automation nor mislabel a genuine decline:
+  #   DZ_ASSUME_YES=1     -> reinstall (skip the prompt)
+  #   interactive decline -> abort (the operator said no)
+  #   no usable TTY, !yes -> reinstall, but say so (pre-3b behaviour was a silent
+  #                          reinstall; keep automation working rather than abort)
+  # A readable /dev/tty inode (-r) can still fail to OPEN with no controlling
+  # terminal (cron/systemd/`curl|bash` without a tty), so probe an actual open
+  # rather than trusting -r (and skip confirm()'s own tty read in that case).
+  if [ "$DZ_ASSUME_YES" = 1 ]; then
+    :
+  elif { : <"$TTY"; } 2>/dev/null; then
+    confirm "Reinstall? This disconnects and removes the existing instance" \
+      || die "Cancelled: leaving the existing instance in place (manage it with 'sudo docker logs $DZ_NAME')."
+  else
+    warn "No terminal to prompt on and DZ_ASSUME_YES is unset; reinstalling to preserve non-interactive behaviour (set DZ_ASSUME_YES=1 to silence this, or run interactively to be asked first)."
+  fi
+
+  info "Uninstalling existing instance..."
+  if [ -n "$running" ]; then
+    # Graceful: `docker stop` sends SIGTERM, which the container entrypoint traps to
+    # run a bounded `doublezero disconnect` (only if a tunnel is up) before tearing
+    # doublezerod down -- releasing the GRE tunnel/routes/on-chain session cleanly.
+    # The container's --stop-timeout (60s) bounds the stop; the outer `timeout`
+    # guards a wedged docker CLI / restarting container so we can never hang forever.
+    info "Stopping it gracefully (disconnecting the DoubleZero tunnel)..."
+    local stop_ok=1
+    if command -v timeout >/dev/null 2>&1; then
+      $SUDO timeout 90 docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    else
+      $SUDO docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    fi
+    [ "$stop_ok" = 1 ] || warn "Could not stop the existing container cleanly (timed out or errored); forcing removal. Its GRE tunnel/routes may be orphaned in the host network namespace -- check 'doublezero status' / 'ip link' and disconnect manually if connectivity is off."
+  fi
+  $SUDO docker rm -f "$DZ_NAME" >/dev/null 2>&1 || true
+
+  # Version-independent confirmation that the tunnel actually came down: the
+  # container ran with --network host, so a failed disconnect leaves the doublezero1
+  # interface (and its routes/on-chain session) orphaned in the host netns. Warn
+  # loudly if it lingers -- the fresh connect below usually recreates it, but a
+  # leftover old tunnel means the previous session wasn't released.
+  if command -v ip >/dev/null 2>&1 && $SUDO ip link show doublezero1 >/dev/null 2>&1; then
+    warn "The DoubleZero tunnel interface (doublezero1) is still present after teardown; the previous instance may not have disconnected cleanly. Its on-chain session/routes could be orphaned -- verify with 'doublezero status'."
+  fi
+}
+reinstall_existing_instance
+
 # ----------------------------------------------------------------------------
 # 4. host kernel / network prep (host-side; safe to attempt)
 # ----------------------------------------------------------------------------
diff --git a/scripts/connect.sh b/scripts/connect.sh
@@ -357,6 +357,72 @@ if ! command -v docker >/dev/null 2>&1; then
 fi
 $SUDO docker info >/dev/null 2>&1 || die "Docker is installed but the daemon isn't reachable. Start it (e.g. 'sudo systemctl start docker') and re-run."
 
+# ----------------------------------------------------------------------------
+# 3b. existing instance? reinstall (graceful teardown) or cancel
+# ----------------------------------------------------------------------------
+# A prior run leaves a long-lived container (--restart unless-stopped) that holds
+# the DoubleZero tunnel and the WS sink. Re-running the installer over a live
+# instance would collide (same container name, same tunnel, same host ports), so
+# detect one up front and tear it down cleanly first.
+reinstall_existing_instance() {
+  $SUDO docker ps -a -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . || return 0
+
+  local running="" existing_env="" existing_img=""
+  $SUDO docker ps -q --filter "name=^${DZ_NAME}$" 2>/dev/null | grep -q . && running=1
+  # Label the victim: all three installers share DZ_NAME, so e.g. the testnet
+  # installer can find a live *mainnet* container -- name the network/image so the
+  # operator isn't asked to destroy an unidentified instance.
+  existing_env="$($SUDO docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$DZ_NAME" 2>/dev/null | sed -n 's/^DZ_ENV=//p' | head -1 || true)"
+  existing_img="$($SUDO docker inspect -f '{{.Config.Image}}' "$DZ_NAME" 2>/dev/null || true)"
+  warn "An edge-connect instance ('$DZ_NAME'${existing_env:+, env=$existing_env}${existing_img:+, image=$existing_img}) already exists on this host${running:+ and is running}."
+
+  # Decide whether to reinstall, keeping the three cases distinct so we neither
+  # break headless automation nor mislabel a genuine decline:
+  #   DZ_ASSUME_YES=1     -> reinstall (skip the prompt)
+  #   interactive decline -> abort (the operator said no)
+  #   no usable TTY, !yes -> reinstall, but say so (pre-3b behaviour was a silent
+  #                          reinstall; keep automation working rather than abort)
+  # A readable /dev/tty inode (-r) can still fail to OPEN with no controlling
+  # terminal (cron/systemd/`curl|bash` without a tty), so probe an actual open
+  # rather than trusting -r (and skip confirm()'s own tty read in that case).
+  if [ "$DZ_ASSUME_YES" = 1 ]; then
+    :
+  elif { : <"$TTY"; } 2>/dev/null; then
+    confirm "Reinstall? This disconnects and removes the existing instance" \
+      || die "Cancelled: leaving the existing instance in place (manage it with 'sudo docker logs $DZ_NAME')."
+  else
+    warn "No terminal to prompt on and DZ_ASSUME_YES is unset; reinstalling to preserve non-interactive behaviour (set DZ_ASSUME_YES=1 to silence this, or run interactively to be asked first)."
+  fi
+
+  info "Uninstalling existing instance..."
+  if [ -n "$running" ]; then
+    # Graceful: `docker stop` sends SIGTERM, which the container entrypoint traps to
+    # run a bounded `doublezero disconnect` (only if a tunnel is up) before tearing
+    # doublezerod down -- releasing the GRE tunnel/routes/on-chain session cleanly.
+    # The container's --stop-timeout (60s) bounds the stop; the outer `timeout`
+    # guards a wedged docker CLI / restarting container so we can never hang forever.
+    info "Stopping it gracefully (disconnecting the DoubleZero tunnel)..."
+    local stop_ok=1
+    if command -v timeout >/dev/null 2>&1; then
+      $SUDO timeout 90 docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    else
+      $SUDO docker stop "$DZ_NAME" >/dev/null 2>&1 || stop_ok=0
+    fi
+    [ "$stop_ok" = 1 ] || warn "Could not stop the existing container cleanly (timed out or errored); forcing removal. Its GRE tunnel/routes may be orphaned in the host network namespace -- check 'doublezero status' / 'ip link' and disconnect manually if connectivity is off."
+  fi
+  $SUDO docker rm -f "$DZ_NAME" >/dev/null 2>&1 || true
+
+  # Version-independent confirmation that the tunnel actually came down: the
+  # container ran with --network host, so a failed disconnect leaves the doublezero1
+  # interface (and its routes/on-chain session) orphaned in the host netns. Warn
+  # loudly if it lingers -- the fresh connect below usually recreates it, but a
+  # leftover old tunnel means the previous session wasn't released.
+  if command -v ip >/dev/null 2>&1 && $SUDO ip link show doublezero1 >/dev/null 2>&1; then
+    warn "The DoubleZero tunnel interface (doublezero1) is still present after teardown; the previous instance may not have disconnected cleanly. Its on-chain session/routes could be orphaned -- verify with 'doublezero status'."
+  fi
+}
+reinstall_existing_instance
+
 # ----------------------------------------------------------------------------
 # 4. host kernel / network prep (host-side; safe to attempt)
 # ----------------------------------------------------------------------------
diff --git a/tests/scripts/reinstall_existing.bats b/tests/scripts/reinstall_existing.bats
