daemon: add configurable max-age to recycle RPC connections

Long-lived daemon RPC connections stay pinned to a single backend for
their whole lifetime. When electrs connects through a load balancer such
as a Kubernetes ClusterSetIP (`*.clusterset.local`), a connection
established before a node rotation keeps routing to the original backend
via the existing TCP/conntrack flow, even after healthier/closer
backends become available. The connection is only re-established on
error, so a still-working-but-stale endpoint is never rebalanced.

Add a `--daemon-rpc-conn-max-age` option (seconds). When a connection
exceeds the configured age it is proactively recycled before the next
request, re-establishing the TCP connection so the load balancer can
re-select a backend. Defaults to 0 = unlimited (never recycle), so
behavior is unchanged unless explicitly enabled. The age check is also
applied to the per-thread connections used for parallel RPC requests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

daemon: make max-age connection recycling best-effort

Proactive recycling previously called the infinite-retry reconnect path
while holding the daemon connection mutex, before sending the request.
A transient "new connections fail" event at the load balancer could
therefore block all requests on that connection instead of continuing to
use the existing, still-healthy socket -- turning an LB hiccup into an
electrs outage when --daemon-rpc-conn-max-age is enabled.

Split tcp_connect() into a single-attempt tcp_connect_once() (primary
then fallback, no retry/backoff) and keep the looping tcp_connect() for
startup and post-failure reconnects, where there is no usable socket to
fall back to. Max-age recycling now uses try_reconnect_once(): on
success the connection is swapped, on failure we log and keep the
existing connection, retrying recycling on a later request. Real
send/recv failures still go through the existing infinite reconnect.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

daemon: address Copilot review nits

- config: parse --daemon-rpc-conn-max-age via value_t_or_exit! for
  consistent clap error handling instead of a manual parse + panic!.
- daemon: store the actually-connected address (primary or fallback) on
  Connection and log it when recycling, so diagnostics aren't misleading
  when connected to the fallback daemon.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

daemon: rate-limit failed recycles, add metric + tests

Address review feedback on proactive max-age recycling:

- Blocker: a failed recycle attempt kept the existing connection (good)
  but did not update any timestamp, so is_expired() stayed true and every
  subsequent request re-attempted the recycle first -- each failed
  attempt blocking up to DAEMON_CONNECTION_TIMEOUT under the connection
  mutex. During a sustained "new connections fail" event this turned
  every fast RPC into a request paying a full connect timeout. Now a
  failed attempt records last_recycle_attempt and a cooldown
  (DAEMON_CONN_RECYCLE_COOLDOWN, default 30s) gates retries, so the old
  socket keeps serving requests at full speed between attempts.

- Extract the recycle decision into a pure `recycle_due()` helper and
  cover it with unit tests (max-age boundary, None, and cooldown).

- Add a daemon_rpc_conn_recycled{result="ok|failed"} counter so recycle
  behavior is observable in prod.

- tcp_connect_once no longer warns per-attempt; it returns one
  descriptive error that callers log, avoiding double log lines on the
  recycle path. The startup/error loop logs that error + backoff.

- Document in --daemon-rpc-conn-max-age help that the reconnect is inline
  on the request path, so the value should be generous (minutes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: DeviaVir

src/bin/electrs.rs

@@ -74,6 +74,7 @@ fn run_server(config: Arc<Config>, salt_rwlock: Arc<RwLock<String>>) -> Result<(
         config.network_type,
         signal.clone(),
         &metrics,
+        config.daemon_conn_max_age,
     )?);
     info!("opening database at {}", config.db_path.display());
     let store = Arc::new(Store::open(&config, &metrics, true));

src/bin/tx-fingerprint-stats.rs

@@ -40,6 +40,7 @@ fn main() {
             config.network_type,
             signal,
             &metrics,
+            config.daemon_conn_max_age,
         )
         .unwrap(),
     );

src/config.rs

@@ -5,6 +5,7 @@ use std::net::SocketAddr;
 use std::net::ToSocketAddrs;
 use std::path::{Path, PathBuf};
 use std::sync::Arc;
+use std::time::Duration;
 use stderrlog;
 
 use crate::chain::Network;
@@ -27,6 +28,7 @@ pub struct Config {
     pub daemon_rpc_addr: SocketAddr,
     pub daemon_rpc_fallback_addr: Option<SocketAddr>,
     pub daemon_parallelism: usize,
+    pub daemon_conn_max_age: Option<Duration>,
     pub cookie: Option<String>,
     pub electrum_rpc_addr: SocketAddr,
     pub http_addr: SocketAddr,
@@ -177,6 +179,13 @@ impl Config {
                     .help("Number of JSONRPC requests to send in parallel")
                     .default_value("4")
             )
+            .arg(
+                Arg::with_name("daemon_rpc_conn_max_age")
+                    .long("daemon-rpc-conn-max-age")
+                    .help("Max age (in seconds) of a daemon RPC TCP connection before it is proactively recycled. Recycling re-establishes the connection, letting a load balancer (e.g. a Kubernetes ClusterSetIP) re-select a backend after node rotations. The reconnect happens inline on the next request, so prefer a generous value (minutes, not seconds) to avoid periodic latency spikes. 0 = unlimited / never recycle (default)")
+                    .default_value("0")
+                    .takes_value(true),
+            )
             .arg(
                 Arg::with_name("monitoring_addr")
                     .long("monitoring-addr")
@@ -425,6 +434,12 @@ impl Config {
             .value_of("daemon_rpc_fallback_addr")
             .map(|e| str_to_socketaddr(e, "Bitcoin Fallback RPC"));
 
+        let daemon_conn_max_age: Option<Duration> =
+            match value_t_or_exit!(m, "daemon_rpc_conn_max_age", u64) {
+                0 => None, // 0 = unlimited / never recycle
+                secs => Some(Duration::from_secs(secs)),
+            };
+
         let electrum_rpc_addr: SocketAddr = str_to_socketaddr(
             m.value_of("electrum_rpc_addr")
                 .unwrap_or(&format!("127.0.0.1:{}", default_electrum_port)),
@@ -494,6 +509,7 @@ impl Config {
             daemon_rpc_addr,
             daemon_rpc_fallback_addr,
             daemon_parallelism: value_t_or_exit!(m, "daemon_parallelism", usize),
+            daemon_conn_max_age,
             cookie,
             utxos_limit: value_t_or_exit!(m, "utxos_limit", usize),
             electrum_rpc_addr,