Fix flaky token-store steal-contention test

testConcurrentStealContentionPreservesMutualExclusion asserted a hard
mutual-exclusion invariant (overlaps == 0, maxInside == 1) that the
FileTokenStore cross-process lock does not provide. stealIfStale is
best-effort by design and documents a three-actor residual - a peer
recreating the lock while a second captures that fresh live lock and a
third claims the freed path - under which two holders briefly run at
once, degrading only to one extra token refresh (Layer-1 atomic rename
keeps the credential intact). Under four-way contention that residual is
reachable, so the test failed intermittently on CI.

A stress harness confirmed the mechanism: 8 contenders overlapped in 12
of 700 iterations (maxInside 2), while 2 contenders never overlapped in
800 iterations.

Split the one over-strict test in two:

- testConcurrentStealContentionTwoWayPreservesMutualExclusion runs two
  contenders and keeps the strict overlaps == 0 / maxInside == 1
  assertions. The three-actor residual cannot arise with two threads, so
  this is deterministic and still pins the exclusion the atomic capture
  guarantees.

- testConcurrentStealContentionDegradesCleanly runs four contenders and
  asserts only the benign invariants - every contender runs its section
  and no capture temp leaks - tolerating the documented residual.

No production change. Full FileTokenStoreTest is 40/40 green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: glasstiger

core/src/test/java/io/questdb/client/test/cutlass/auth/FileTokenStoreTest.java

@@ -185,7 +185,7 @@ public void testClearOnEmptyStoreIsNoOp() throws Exception {
     }
 
     @Test
-    public void testConcurrentStealContentionPreservesMutualExclusion() throws Exception {
+    public void testConcurrentStealContentionDegradesCleanly() throws Exception {
         assertMemoryLeak(() -> {
             Path dir = storeDir();
             Files.createDirectories(dir);
@@ -195,14 +195,64 @@ public void testConcurrentStealContentionPreservesMutualExclusion() throws Excep
             Files.write(lock, "crashed-holder-stamp".getBytes(StandardCharsets.UTF_8));
             Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 600_000));
 
-            // several "processes" race to steal the one abandoned lock. The staleness window (60s) far exceeds
-            // each ~100ms hold, so a live holder's lock is never judged stale: only the abandoned lock is
-            // stealable, and the atomic, stamp-verified steal must admit exactly one holder at a time. This is a
-            // mutual-exclusion invariant guard under steal contention: it exercises the steal path concurrently
-            // and fails on any gross loss of exclusion. It does not deterministically reproduce the narrow
-            // isStale->delete race the atomic steal closes - that needs a timing seam this final class does not
-            // expose - so the fix itself rests on the atomic capture + stamp re-check, not on this test alone.
+            // several "processes" race to steal the one abandoned lock. This exercises the steal path under
+            // N-way contention and asserts it degrades CLEANLY: every contender eventually runs its critical
+            // section (none is starved or wedged) and no atomic-capture temp file leaks. It deliberately does
+            // NOT assert strict mutual exclusion. stealIfStale is best-effort by design and documents a
+            // three-actor residual - a peer recreating the lock in the isStale->capture gap while a second
+            // captures that fresh live lock and a third claims the momentarily-free path, all at once - under
+            // which two holders can briefly run concurrently. That residual needs three or more contenders and
+            // degrades only to one extra token refresh (a re-prompt on a rotating-refresh-token IdP), never a
+            // torn or forged credential, since the Layer-1 atomic-rename write is independent of the lock.
+            // testConcurrentStealContentionTwoWayPreservesMutualExclusion pins the exclusion the atomic capture
+            // does guarantee, deterministically, with two contenders.
             final int threads = 4;
+            AtomicInteger ran = new AtomicInteger();
+            TokenStore.CriticalSection section = () -> {
+                Os.sleep(100);
+                ran.incrementAndGet();
+                return true;
+            };
+
+            Thread[] ts = new Thread[threads];
+            for (int i = 0; i < threads; i++) {
+                // a generous acquire budget so a contender waits for the lock rather than giving up early
+                FileTokenStore store = new FileTokenStore(dir, 30_000, 60_000);
+                ts[i] = new Thread(() -> store.inLock(key, section));
+            }
+            for (Thread t : ts) {
+                t.start();
+            }
+            for (Thread t : ts) {
+                t.join();
+            }
+
+            Assert.assertEquals("every contender must run its critical section", threads, ran.get());
+            assertNoCaptureTempFiles(dir, key);
+        });
+    }
+
+    @Test
+    public void testConcurrentStealContentionTwoWayPreservesMutualExclusion() throws Exception {
+        assertMemoryLeak(() -> {
+            Path dir = storeDir();
+            Files.createDirectories(dir);
+            TokenStoreKey key = sampleKey();
+            // a lock abandoned by a crashed holder, backdated well past the staleness window
+            Path lock = lockFile(dir, key);
+            Files.write(lock, "crashed-holder-stamp".getBytes(StandardCharsets.UTF_8));
+            Files.setLastModifiedTime(lock, FileTime.fromMillis(System.currentTimeMillis() - 600_000));
+
+            // TWO processes race to steal the one abandoned lock. With exactly two contenders the steal is
+            // deterministically mutually exclusive: the atomic capture (rename) lets exactly one steal the
+            // abandoned lock, and once a winner holds a freshly-stamped lock the loser reads that live stamp,
+            // judges it not stale (the 60s window far exceeds each ~100ms hold) and waits rather than stealing
+            // it; the empty-lock grace likewise stops the loser stealing the winner's lock in its brief
+            // create->stamp gap. The three-actor residual stealIfStale documents - a peer recreating the lock
+            // while a second captures it AND a third claims the freed path - structurally cannot arise with two
+            // threads, so exclusion holds exactly here (the N-way best-effort path is
+            // testConcurrentStealContentionDegradesCleanly).
+            final int threads = 2;
             AtomicInteger inside = new AtomicInteger();
             AtomicInteger maxInside = new AtomicInteger();
             AtomicInteger overlaps = new AtomicInteger();