Fix OIDC blank-refresh gate and correct docs

tryRefresh's hasRequiredToken gated the served kind on length() > 0
while storeTokens folds a whitespace-only token to null via
Chars.isBlank. A non-conformant 2xx refresh with a blank access/id
token therefore reported success, cached a token storeTokens then
nulled, and made signIn() throw "no access_token" instead of falling
back to the interactive device flow. Gate hasRequiredToken on
!Chars.isBlank so the refresh gate and the cache agree, and add
testBlankTokenFromRefreshFallsBackToInteractiveFlow, which fails
without the change with the exact "no access_token" error.

Correct three stale docs left by the SF-drainer-terminal fix:
- QwpCredentialUnavailableException's javadoc described a
  reconnectMaxDurationMillis-bounded terminate that no path
  implements; the running store-and-forward drainer retries a
  credential-unavailable failure indefinitely under Invariant B, and
  only the foreground/SYNC initial connect fails fast.
- Sender.httpTokenProvider's javadoc claimed a WebSocket sender
  terminates on a sustained token outage; a running SF-backed sender
  retries token-pull failures indefinitely and recovers, as
  testPersistentlyThrowingProviderOnReconnect... already proves.
- FileTokenStore's createLockFile/acquireLock/stealIfStale comments
  claimed a single atomic create+stamp with no gap; writeNewFile
  opens then writes separately, so a GC pause can land in the empty
  window, which EMPTY_LOCK_STEAL_GRACE_MILLIS covers.

Add testProviderTokenReResolvedOnFailoverReconnect, which drives a
real QwpQueryClient failover reconnect and asserts reconnectViaTracker
re-resolves the token provider so a rotated token reaches the
reconnect upgrade, closing the one untested cross-context provider
path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: glasstiger

core/src/main/java/io/questdb/client/Sender.java

@@ -2111,9 +2111,13 @@ public LineSenderBuilder httpToken(String token) {
          * started, then once per flush. Over WebSocket the initial connection handshake runs during
          * {@code build()} and queries the provider once for it, then again once per reconnect handshake - so a
          * refreshed token is presented each time the link is (re)established; an already-established WebSocket
-         * is not re-authenticated mid-stream. The two transports differ on a sustained token outage: over HTTP
-         * a failed pull is retried on the next row, but over WebSocket a pull that keeps failing past the
-         * reconnect budget terminates the sender for good, like any persistent reconnect failure.
+         * is not re-authenticated mid-stream. The two transports differ in mechanism but both keep the producer
+         * alive across a sustained token outage: over HTTP a failed pull leaves the request token-pending and is
+         * retried on the next row; over WebSocket the token must be obtainable when {@code build()} runs (the
+         * initial handshake fails fast otherwise), after which a pull that keeps failing on later reconnects is
+         * retried indefinitely, with the buffered rows held in store-and-forward, until a token is available
+         * again. A token outage does not terminate a running WebSocket sender, just as a persistent transport
+         * reconnect failure does not (store-and-forward Invariant B).
          * <br>
          * Over HTTP the token is pulled once per request and written into the request buffer ahead of the
          * buffered rows, so a token refresh is picked up on the next new batch after a successful flush. A

core/src/main/java/io/questdb/client/cutlass/auth/FileTokenStore.java

@@ -352,11 +352,13 @@ long getLockStaleMillis() {
     }
 
     private static void createLockFile(Path lock, String nonce) throws IOException {
-        // Exclusively create the lock (O_CREAT|O_EXCL via CREATE_NEW) AND write the owner nonce in a single
-        // open, so there is NO create->stamp gap for a GC/safepoint pause (or a cross-machine clock skew) to
-        // straddle and make our freshly-created lock look empty-and-stale to a peer. FileAlreadyExists means a
-        // peer already holds it. releaseLock and stealIfStale verify this nonce before deleting. Keep the
-        // owner-only perms (and the non-POSIX fallback) to match the store's other files.
+        // Exclusively create the lock (O_CREAT|O_EXCL via CREATE_NEW), then write the owner nonce into that same
+        // open channel before closing it. The file exists empty only for the tiny window between the create and
+        // the stamp; a GC/safepoint pause (or a cross-machine clock skew) CAN land in that window, so what keeps
+        // our freshly-created lock from being stolen as empty-and-stale is EMPTY_LOCK_STEAL_GRACE_MILLIS sitting
+        // well above it, not the absence of the window. FileAlreadyExists means a peer already holds it.
+        // releaseLock and stealIfStale verify this nonce before deleting. Keep the owner-only perms (and the
+        // non-POSIX fallback) to match the store's other files.
         final byte[] bytes = nonce.getBytes(StandardCharsets.UTF_8);
         try {
             writeNewFile(lock, bytes, FILE_ATTRS);
@@ -701,8 +703,9 @@ private String acquireLock(Path lock) {
         final long deadline = System.currentTimeMillis() + lockAcquireBudgetMillis;
         while (true) {
             try {
-                // atomic exclusive-create + stamp in one call: no create->stamp gap, so a GC/safepoint pause
-                // can no longer make our freshly-created lock look empty-and-stale to a peer mid-acquisition
+                // exclusive-create then stamp on the same open channel: the empty-file window between the two is
+                // tiny and covered by EMPTY_LOCK_STEAL_GRACE_MILLIS, so a GC/safepoint pause mid-acquisition
+                // cannot get our freshly-created lock stolen as empty-and-stale
                 createLockFile(lock, nonce);
                 return nonce;
             } catch (FileAlreadyExistsException e) {
@@ -792,11 +795,12 @@ private void stealIfStale(Path lock) {
                 return;
             }
         } else if (!isOlderThan(lock, Math.min(EMPTY_LOCK_STEAL_GRACE_MILLIS, lockStaleMillis))) {
-            // an empty/unreadable lock is never a validly-held lock: acquireLock creates the lock and writes
-            // the owner nonce in ONE atomic call (createLockFile via CREATE_NEW), so a live lock always carries
-            // a stamp. An empty lock therefore means a crash mid-write - the exclusive create succeeded but the
-            // nonce write did not - a rare, narrow window with NO Java-level create->stamp gap for a GC/safepoint
-            // pause to straddle (the pause would have to land inside the single write call). Steal it on the
+            // an empty/unreadable lock is almost never a validly-held lock: acquireLock creates the lock and
+            // stamps the owner nonce onto the same open channel (createLockFile via CREATE_NEW), so a live lock
+            // carries its stamp within the tiny create->stamp window. An empty lock therefore means either a
+            // crash mid-write (the exclusive create succeeded but the nonce write did not) or a peer momentarily
+            // caught in that narrow window - a GC/safepoint pause CAN land there, which is exactly why the grace
+            // exists. Steal it on the
             // short empty-lock grace rather than the full staleness window, so a crash orphan stops wedging peers
             // for the whole window; the capture-verify below still confirms the lock is unchanged before
             // completing the steal. (A cross-machine clock skew wider than the grace could still pre-empt such a

core/src/main/java/io/questdb/client/cutlass/auth/OidcDeviceAuth.java

@@ -1601,12 +1601,15 @@ private boolean tryRefresh() {
             return false;
         }
         // succeed only on a clean 2xx (no OAuth error) returning the token getToken() actually serves (the
-        // id token when groups are encoded in it, the access token otherwise). A refresh that omits the id
-        // token - which RFC 6749 permits and many providers do - or carries an error or a non-2xx status
-        // must fall back to the interactive flow rather than be cached (and later fail in selectToken())
+        // id token when groups are encoded in it, the access token otherwise). A refresh that omits the served
+        // kind - which RFC 6749 permits and many providers do - or returns it blank/whitespace-only, or carries
+        // an error or a non-2xx status, must fall back to the interactive flow rather than be cached. Test the
+        // served kind with Chars.isBlank, the SAME contract storeTokens/adopt use to fold a blank token to null:
+        // gating on length() > 0 here would pass a whitespace-only token, which storeTokens then nulls, so
+        // tryRefresh would report success while selectToken() throws "no token" instead of falling back.
         boolean hasRequiredToken = (groupsInToken
-                ? tokenParser.idToken.length() > 0
-                : tokenParser.accessToken.length() > 0)
+                ? !Chars.isBlank(tokenParser.idToken)
+                : !Chars.isBlank(tokenParser.accessToken))
                 && isHttpStatusSuccess()
                 && tokenParser.error.length() == 0;
         if (hasRequiredToken) {

core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java

@@ -32,13 +32,15 @@
  * returning a token -- a failed silent refresh, or no sign-in yet.
  * <p>
  * Distinct from {@link QwpAuthFailedException}, which means the server rejected a
- * credential the client did present. Neither is a transport outage, so neither heals
- * by retrying alone: a reconnect loop cannot conjure a credential it is unable to
- * acquire. The cursor send loop therefore retries this class only for as long as a
- * transient refresh could still recover -- bounded by {@code reconnectMaxDurationMillis}
- * -- and then terminates the sender with the provider's own message, rather than
- * reconnect-looping forever against a dead credential (Invariant B, which governs
- * genuine transport outages, stays untouched).
+ * credential the client did present (a terminal auth failure). A credential the client
+ * cannot ACQUIRE is instead handled by connection phase, exactly like a transport outage:
+ * the RUNNING store-and-forward drainer retries it indefinitely with capped backoff under
+ * Invariant B -- the IdP becomes reachable again, or the user completes an interactive
+ * sign-in -- holding the un-acked rows in SF meanwhile, and NEVER bounds it by
+ * {@code reconnectMaxDurationMillis} nor latches a terminal (either would drop a producer
+ * store-and-forward promised to keep alive). Only the foreground/SYNC initial connect
+ * fails fast, because a connectivity error is the caller's to see during initialization,
+ * not after the drainer is running.
  * <p>
  * This is an internal marker that carries the provider's own exception: it exists so
  * the send loop can tell "the provider failed" apart from "the network failed". A

core/src/test/java/io/questdb/client/test/cutlass/auth/OidcDeviceAuthTest.java

@@ -1775,6 +1775,41 @@ public void testBlankServedTokenFromWireIsNotServed() throws Exception {
         });
     }
 
+    @Test(timeout = 30_000)
+    public void testBlankTokenFromRefreshFallsBackToInteractiveFlow() throws Exception {
+        assertMemoryLeak(() -> {
+            // a non-conformant IdP answers a SILENT REFRESH with a 2xx carrying a whitespace-only access token.
+            // The refresh gate (hasRequiredToken) must treat it as absent with the same Chars.isBlank contract
+            // storeTokens uses, so tryRefresh() reports failure and signIn() falls back to the interactive device
+            // flow - rather than caching a token storeTokens then nulls, which would make signIn() throw "no
+            // access_token" while a fresh interactive sign-in was still possible.
+            AtomicInteger deviceCodePolls = new AtomicInteger();
+            MockOidcServer.Handler handler = (method, path, body) -> {
+                if (DEVICE_PATH.equals(path)) {
+                    return MockOidcServer.json(200, deviceAuthorizationJson(1, 300));
+                }
+                if (body.contains("grant_type=refresh_token")) {
+                    // blank served token on refresh: the gate must fall back, not cache-and-serve it
+                    return MockOidcServer.json(200, tokenJson("   ", null, null, 3600));
+                }
+                // the device-code grant: the first poll mints the initial short-lived token; the second is the
+                // interactive fallback after the blank refresh and mints a fresh, usable one
+                return deviceCodePolls.incrementAndGet() == 1
+                        ? MockOidcServer.json(200, tokenJson("ACCESS-1", null, "REFRESH-1", 60))
+                        : MockOidcServer.json(200, tokenJson("ACCESS-FALLBACK", null, "REFRESH-2", 3600));
+            };
+            try (MockOidcServer server = new MockOidcServer(handler);
+                 OidcDeviceAuth auth = newAuth(server, false, noopPrompt())) {
+                Assert.assertEquals("ACCESS-1", auth.signIn());
+                expireCachedToken(auth); // force the silent-refresh path on the next sign-in
+                // with the blank-refresh gate fixed, signIn() falls back to the device flow instead of throwing
+                Assert.assertEquals("ACCESS-FALLBACK", auth.signIn());
+                Assert.assertEquals("the interactive device flow must run again as the fallback",
+                        2, deviceCodePolls.get());
+            }
+        });
+    }
+
     @Test(timeout = 30_000)
     public void testGroupsInTokenButNoIdTokenFails() throws Exception {
         assertMemoryLeak(() -> {

core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientTokenProviderTest.java

@@ -26,18 +26,28 @@
 
 import io.questdb.client.cutlass.http.client.HttpClientException;
 import io.questdb.client.cutlass.line.LineSenderException;
+import io.questdb.client.cutlass.qwp.client.QwpColumnBatch;
+import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler;
+import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind;
 import io.questdb.client.cutlass.qwp.client.QwpQueryClient;
+import io.questdb.client.cutlass.qwp.protocol.QwpConstants;
+import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer;
 import org.junit.Assert;
 import org.junit.Test;
 
+import java.io.IOException;
 import java.io.OutputStream;
 import java.net.InetAddress;
 import java.net.ServerSocket;
 import java.net.Socket;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
 
 /**
  * Unit coverage for {@link QwpQueryClient#withBearerTokenProvider}: header
@@ -53,6 +63,20 @@
  */
 public class QwpQueryClientTokenProviderTest {
 
+    private static final QwpColumnBatchHandler NOOP_BATCH_HANDLER = new QwpColumnBatchHandler() {
+        @Override
+        public void onBatch(QwpColumnBatch batch) {
+        }
+
+        @Override
+        public void onEnd(long totalRows) {
+        }
+
+        @Override
+        public void onError(byte status, String message) {
+        }
+    };
+
     @Test
     public void testProviderConflictsWithBasicAuth() {
         try (QwpQueryClient c = QwpQueryClient.newPlainText("localhost", 9000).withBearerTokenProvider(() -> "tok")) {
@@ -126,6 +150,54 @@ public void testProviderSynthesizesBearerHeader() {
         }
     }
 
+    @Test(timeout = 20_000)
+    public void testProviderTokenReResolvedOnFailoverReconnect() throws Exception {
+        // The failover reconnect path (reconnectViaTracker) resolves the Authorization header once before its
+        // endpoint walk, exactly as connect() does, so a rotating token reaches the reconnect upgrade. This
+        // pins that a regression dropping the re-resolve from the reconnect path would be caught: bind endpoint
+        // A on the initial connect (capturing tok-0), drop it, then run a query - the failover reconnect to
+        // endpoint B must upgrade with a FRESHLY resolved token, not the stale connect-time one.
+        AtomicInteger calls = new AtomicInteger();
+        TestWebSocketServer a = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() {
+        });
+        a.setSendServerInfo(true);
+        TestWebSocketServer b = new TestWebSocketServer(new ExecDoneQueryServer());
+        b.setSendServerInfo(true);
+        try {
+            a.start();
+            b.start();
+            Assert.assertTrue(a.awaitStart(5, TimeUnit.SECONDS));
+            Assert.assertTrue(b.awaitStart(5, TimeUnit.SECONDS));
+
+            try (QwpQueryClient client = QwpQueryClient.fromConfig(
+                    "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + ";auth_timeout_ms=2000;")
+                    .withBearerTokenProvider(() -> "tok-" + calls.getAndIncrement())) {
+                client.connect();
+                Assert.assertTrue("client must bind the first endpoint on connect", client.isConnected());
+                String aHeader = a.pollAuthorizationHeader(5, TimeUnit.SECONDS);
+                Assert.assertEquals("the initial connect upgrade must carry the first resolved token",
+                        "Bearer tok-0", aHeader);
+
+                // drop endpoint A so the next execute() cannot use its connection and must fail over
+                a.close();
+
+                // the query fails on the dead A connection, drives the failover loop -> reconnectViaTracker,
+                // which re-resolves the header and upgrades B; B answers EXEC_DONE so execute() returns
+                client.execute("SELECT 1", NOOP_BATCH_HANDLER, false);
+
+                String bHeader = b.pollAuthorizationHeader(5, TimeUnit.SECONDS);
+                Assert.assertNotNull("the failover reconnect must upgrade endpoint B", bHeader);
+                Assert.assertTrue("the reconnect upgrade must carry a Bearer token, was: " + bHeader,
+                        bHeader.startsWith("Bearer tok-"));
+                Assert.assertNotEquals("the failover reconnect must RE-RESOLVE the provider, not reuse the "
+                        + "connect-time token", aHeader, bHeader);
+            }
+        } finally {
+            a.close();
+            b.close();
+        }
+    }
+
     @Test(timeout = 15_000)
     public void testProviderTokenSentOnRealUpgrade() throws Exception {
         // drive the REAL connect path (connect() -> resolveAuthorizationHeader -> runUpgradeWithTimeout),
@@ -235,4 +307,34 @@ public void testThrowingProviderFailsConnect() throws Exception {
             }
         }
     }
+
+    private static byte[] buildExecDone(byte[] queryRequest) {
+        int bodyLen = 1 + 8 + 1 + 1; // msg_kind + request_id + op_type + rows_affected varint
+        byte[] frame = new byte[QwpConstants.HEADER_SIZE + bodyLen];
+        ByteBuffer bb = ByteBuffer.wrap(frame).order(ByteOrder.LITTLE_ENDIAN);
+        bb.put((byte) 'Q').put((byte) 'W').put((byte) 'P').put((byte) '1');
+        bb.put((byte) 1);       // version
+        bb.put((byte) 0);       // flags
+        bb.putShort((short) 0); // table_count
+        bb.putInt(bodyLen);     // payload_length
+        bb.put(QwpEgressMsgKind.EXEC_DONE);
+        bb.put(queryRequest, 1, 8); // echo request_id verbatim
+        bb.put((byte) 0);       // op_type
+        bb.put((byte) 0);       // rows_affected = 0
+        return frame;
+    }
+
+    private static final class ExecDoneQueryServer implements TestWebSocketServer.WebSocketServerHandler {
+        @Override
+        public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) {
+            if (data.length == 0 || data[0] != QwpEgressMsgKind.QUERY_REQUEST) {
+                return;
+            }
+            try {
+                client.sendBinary(buildExecDone(data));
+            } catch (IOException e) {
+                // best-effort: a failed reply surfaces to the client as a transport error
+            }
+        }
+    }
 }