fix(qwp): foreground role-reject retry backs off exponentially, not a fixed-interval TLS storm

The cursor I/O loop's role-reject branch reset backoff to
reconnect_initial_backoff_millis, parked exactly that interval (no
jitter, no growth), then `continue`d -- bypassing the shared capped
exponential backoff-with-jitter block. Once an all-replica failover
window (or a misconfigured replicas-only address list, now surfaced
here as a retriable role reject rather than a terminal durable-ack
mismatch) reaches this branch, every sweep pinned at ~100ms forever:
a fixed ~10-20 handshakes/s storm of fresh WebSocketClient (2x64KB
native) + fresh SSLContext (no session resumption, trust-store
re-read) per endpoint. That broke the documented
capped-exponential-backoff contract and was ~50x more aggressive than
the orphan drainer, which already grows to reconnect_max_backoff_millis.

Fall through to the shared backoff block so role-reject grows to
reconnect_max_backoff_millis (5s default) like every other reconnect
branch, and emit a throttled WARN naming the all-replica condition.
close() still unparks the loop via LockSupport, so shutdown stays
prompt under the longer park (QwpRoleRejectCloseRaceTest green).

Adds QwpRoleRejectBackoffGrowthTest as a red-first guard: on the old
behaviour the inter-attempt gaps are flat ([52,51,51,51,51,51]ms);
after the fix they grow exponentially.

Also lands accompanying connect-timeout / Invariant-B doc and comment
refinements across Sender, QwpWebSocketSender, BackgroundDrainer,
SenderConnectionEvent, QwpVersionMismatchException, and the design
notes, plus .gitignore entries for root /target and .pi-subagents/.

Author: bluestreak01

.gitignore

@@ -29,4 +29,9 @@ core/CMakeCache.txt
 **/build
 **/CMakeFiles
 .envrc
-.vscode
+.vscode
+# Root-level Maven build output
+/target
+
+# pi subagents runtime artifacts
+.pi-subagents/

core/src/main/java/io/questdb/client/QuestDBBuilder.java

@@ -130,7 +130,7 @@ public QuestDBBuilder connectionListener(SenderConnectionListener listener) {
     /**
      * Sets the async error handler applied to every pooled ingest
      * {@link Sender}. The handler receives terminal/async ingest errors
-     * (connect-budget exhaustion, terminal upgrade failures, write errors)
+     * (terminal upgrade failures, write errors)
      * from across the whole sender pool; notifications are delivered on the
      * senders' I/O threads, so the handler must be thread-safe and must not
      * block. Pass {@code null} (the default) to keep each sender's

core/src/main/java/io/questdb/client/Sender.java

@@ -791,11 +791,12 @@ default Sender uuidColumn(CharSequence name, long lo, long hi) {
      *       unconnected sender; the I/O thread runs the same retry loop in
      *       the background. The user thread can call {@code at()} /
      *       {@code flush()} immediately; rows accumulate in the cursor SF
-     *       engine until the wire is up. A connect-budget exhaustion or a
-     *       terminal upgrade failure is delivered to the async error inbox
-     *       as a {@link io.questdb.client.SenderError} (no synchronous
-     *       throw on the user call site). Wire {@code error_handler=...}
-     *       to observe these.</li>
+     *       engine until the wire is up. Connect failures are retried
+     *       indefinitely in the background; a terminal upgrade failure
+     *       (auth reject, capability mismatch) is delivered to the async
+     *       error inbox as a {@link io.questdb.client.SenderError} (no
+     *       synchronous throw on the user call site). Wire
+     *       {@code error_handler=...} to observe these.</li>
      * </ul>
      * <p>
      * Default resolution when the caller does not pick a value:
@@ -2388,15 +2389,16 @@ public LineSenderBuilder reconnectMaxBackoffMillis(long millis) {
         }
 
         /**
-         * Per-outage cap on the cursor I/O loop's reconnect retry budget.
-         * Once a wire failure occurs, the loop retries with exponential
-         * backoff until either reconnect succeeds (timer resets) or this
-         * many millis elapse since the first failure of this outage —
-         * whichever comes first. On budget exhaustion, the next user
-         * thread API call throws.
+         * Cap on the blocking initial-connect retry budget when
+         * {@code initial_connect_retry=sync}. {@code fromConfig} retries
+         * with exponential backoff until connect succeeds or this many
+         * millis elapse, then throws. The background reconnect loop
+         * (mid-stream outages and async initial connect) does NOT consult
+         * this value: it retries indefinitely and halts only on a terminal
+         * auth/upgrade error or {@code close()}.
          * <p>
-         * Default {@code 300_000} (5 minutes). Lower for fail-fast services;
-         * higher for tolerating long maintenance windows. WebSocket only.
+         * Default {@code 300_000} (5 minutes). Lower for fail-fast startup;
+         * higher for tolerating a slow server boot. WebSocket only.
          */
         public LineSenderBuilder reconnectMaxDurationMillis(long millis) {
             if (protocol != PARAMETER_NOT_SET_EXPLICITLY && protocol != PROTOCOL_WEBSOCKET) {

core/src/main/java/io/questdb/client/SenderConnectionEvent.java

@@ -223,8 +223,10 @@ public enum Kind {
         /**
          * Every endpoint in the configured address list was attempted and none
          * accepted the connection in this sweep. The client will back off and
-         * retry the sweep until the reconnect budget is exhausted. Fired once
-         * per failed sweep.
+         * retry the sweep — bounded by {@code reconnect_max_duration_millis}
+         * during a blocking (sync) initial connect, indefinitely otherwise
+         * (Invariant B: the background loop never gives up on a wall-clock
+         * budget). Fired once per failed sweep.
          */
         ALL_ENDPOINTS_UNREACHABLE,
 

core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpVersionMismatchException.java

@@ -31,10 +31,11 @@
  * {@code X-QWP-Version} outside the client's supported range. Treated as
  * transient at every layer per sf-client.md section 13.3: the per-endpoint
  * round walks to the next host (rolling upgrade can leave one node ahead of
- * or behind its peers), and a full round of mismatches consumes the per-outage
- * reconnect budget. Only after the budget exhausts does the connect loop
- * surface a terminal error -- as {@code PROTOCOL_VIOLATION} via the natural
- * giveup path, not {@code SECURITY_ERROR}.
+ * or behind its peers). The background reconnect loop retries a full round
+ * of mismatches indefinitely (Invariant B: no wall-clock give-up); the
+ * blocking (sync) initial connect consumes its retry budget and surfaces a
+ * {@code LineSenderException} from {@code fromConfig} on exhaustion. Never
+ * classified as {@code SECURITY_ERROR}.
  */
 public final class QwpVersionMismatchException extends HttpClientException {
     public QwpVersionMismatchException(int serverVersion, int clientMaxVersion) {

core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java

@@ -229,13 +229,14 @@ public class QwpWebSocketSender implements Sender {
     // point in buildAndConnect.
     private boolean hasEverConnected;
     // OFF   → startup connect failure is immediately terminal (default).
-    // SYNC  → startup connect goes through the same retry-with-backoff
-    //         loop as in-flight reconnect; auth failures still terminal.
+    // SYNC  → startup connect retries with backoff on the user thread,
+    //         bounded by reconnect_max_duration_millis; auth failures
+    //         still terminal.
     // ASYNC → user thread does not connect at all. The I/O thread runs
-    //         the same retry loop in the background; terminal failures
-    //         (auth/upgrade reject, budget exhaustion) are delivered
-    //         to the SenderError dispatcher rather than thrown from the
-    //         constructor.
+    //         the reconnect loop in the background, indefinitely
+    //         (Invariant B); terminal failures (auth/upgrade reject)
+    //         are delivered to the SenderError dispatcher rather than
+    //         thrown from the constructor.
     private Sender.InitialConnectMode initialConnectMode = Sender.InitialConnectMode.OFF;
     private boolean ownsCursorEngine;
     private long pendingBytes;
@@ -2316,7 +2317,7 @@ public QwpWebSocketSender uuidColumn(CharSequence columnName, long lo, long hi)
      * True iff this sender has at least once installed a live (connected
      * + upgraded) WebSocket. Sticky — once true, stays true even after a
      * subsequent disconnect. Lets a {@link SenderErrorHandler}
-     * disambiguate a "never reached the server" budget exhaustion (likely
+     * disambiguate a "never reached the server" terminal failure (likely
      * a config typo or firewall block) from a "lost connection after we
      * were up" failure (likely transient). Returns {@code false} if no
      * I/O loop is running.
@@ -2891,8 +2892,9 @@ private void ensureConnected() {
                 // version today). Frames written before the first successful
                 // connect commit to V1 because cursor segments are immutable;
                 // a future version bump must account for that. Auth/upgrade
-                // rejects and budget exhaustion are surfaced via the error
-                // inbox by the I/O thread, not thrown here.
+                // rejects are surfaced via the error inbox by the I/O
+                // thread, not thrown here; plain connect failures retry
+                // indefinitely (Invariant B).
                 client = null;
                 break;
             case OFF:

core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/BackgroundDrainer.java

@@ -30,6 +30,7 @@
 import io.questdb.client.cutlass.qwp.client.QwpDurableAckMismatchException;
 import io.questdb.client.cutlass.qwp.client.QwpIngressRoleRejectedException;
 import io.questdb.client.cutlass.qwp.client.QwpRoleMismatchException;
+import io.questdb.client.cutlass.qwp.client.QwpVersionMismatchException;
 import org.jetbrains.annotations.TestOnly;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -336,14 +337,29 @@ public WebSocketClient connectWithDurableAckRetry() {
                 LOG.warn("drainer slot {} attempt {}: durable-ack unavailable, retrying after backoff",
                         slotPath, capabilityGapAttempts);
             } catch (Throwable t) {
+                if (t instanceof Error) {
+                    // java.lang.Error (OOM, LinkageError, StackOverflowError)
+                    // is a JVM/programming failure, not a transport outage:
+                    // retrying cannot clear it, and spinning here would pin
+                    // the slot .lock forever with no .failed sentinel and only
+                    // a throttled, possibly-null-message WARN as a trace.
+                    // Rethrow: run()'s outer catch quarantines the slot
+                    // (markFailed + FAILED) and its finally releases the lock
+                    // -- quarantine-and-exit, exactly as genuine terminals do.
+                    throw (Error) t;
+                }
                 // INVARIANT B: a transport failure -- the whole cluster is
                 // unreachable right now (server down, network partition) -- is
                 // TRANSIENT, exactly as the live sender's background loop treats
                 // it. The server will come back; keep retrying (capped backoff)
                 // until it does, stopRequested, or SF exhaustion. NEVER quarantine
                 // the slot on a transport error. Genuine terminals (auth /
                 // non-421 upgrade / durable-ack capability gap) are handled by the
-                // catches above and still fail fast.
+                // catches above and still fail fast. A QWP version mismatch also
+                // reaches here (it extends HttpClientException, not
+                // WebSocketUpgradeException) and is intentionally retried under
+                // Invariant B -- but it is NOT a transport outage, so log it
+                // truthfully below rather than mislabelling it "cluster unreachable".
                 lastErrorMessage = t.getMessage();
                 // Pause the episode wall clock: the gap-to-gap interval this
                 // window interrupts is never charged. Attempts and elapsed
@@ -352,8 +368,24 @@ public WebSocketClient connectWithDurableAckRetry() {
                 lastCapabilityGapNanos = 0L;
                 long nowWarn = System.nanoTime();
                 if (nowWarn - lastTransportWarnNanos >= 5_000_000_000L) {
-                    LOG.warn("drainer slot {}: cluster unreachable ({}), retrying after backoff",
-                            slotPath, t.getMessage());
+                    if (t instanceof QwpVersionMismatchException) {
+                        // The cluster IS reachable: every endpoint completed the
+                        // WebSocket upgrade but advertised a QWP protocol version
+                        // this client cannot speak. A rolling upgrade clears this
+                        // once peers converge, so Invariant B keeps retrying -- but
+                        // if it persists the client binary is version-incompatible
+                        // with the whole cluster and an operator must intervene
+                        // (upgrade the client or the servers). Name the real
+                        // condition so it is diagnosable, not hidden behind a
+                        // network-outage message.
+                        LOG.warn("drainer slot {}: every reachable endpoint advertises an unsupported "
+                                        + "QWP protocol version ({}); retrying (rolling-upgrade window) -- "
+                                        + "if this persists the client is version-incompatible with the cluster",
+                                slotPath, t.getMessage());
+                    } else {
+                        LOG.warn("drainer slot {}: cluster unreachable ({}), retrying after backoff",
+                                slotPath, t.getMessage());
+                    }
                     lastTransportWarnNanos = nowWarn;
                 }
             }