Re-fold recovery at baseline 0 when discarding the dictionary

The full-dict-fallback branch discards a persisted .symbol-dict whose
frames carry their whole dictionary inline (maxSymbolDeltaStart == 0 and
recoveredMaxSymbolId >= size). But the recovery analysis was folded with
the dictionary's size as its baseline, and every consumer of it presents
the baseline it derived the same way: once pd is null,
seedGlobalDictionaryFromPersisted computes baseline 0, and
checkedRecoveryAnalysis rejects a baseline that disagrees with the fold.
Discarding a dictionary that held entries (size > 0) therefore threw
IllegalStateException("recovery symbol baseline mismatch") out of build().

That is not an UnreplayableSlotException, so build()'s quarantine handler
could not set the slot aside. With a stable senderId every restart
re-recovered the same slot and threw again, so the application could never
construct a Sender -- it could not even buffer new rows. Exactly the
outage quarantineTornSlot exists to prevent, and on a slot that is fully
recoverable, since its frames need no dictionary.

It takes one transient plus one crash: a session whose .symbol-dict fails
to open (EIO, fd exhaustion, a Windows share lock) runs full-dict fallback
and, per the never-recreate contract, leaves the previous session's
populated side-file intact; if that session then crashes, this recovery
opens a dictionary with size > 0 beside self-sufficient frames that
out-reach it. The prior regression test covered only the size == 0 case.

Re-fold at baseline 0 after the discard so producer, mirror and replay
guard all anchor at 0 -- the full-dict mode the slot was written in. The
new test plants a populated survivor beside full-dict frames; reverting
the re-fold throws "recovery symbol baseline mismatch [expected=3,
actual=0]".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: glasstiger

core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java

@@ -404,6 +404,36 @@ private CursorSendEngine(String sfDir, long segmentSizeBytes, SegmentManager man
                         && recoveredMaxSymbolDeltaStart == 0L) {
                     persistedDictInProgress.close();
                     persistedDictInProgress = null;
+                    // Re-fold at baseline 0. The analysis above was keyed to the
+                    // dictionary's size, and every consumer of it presents the baseline it
+                    // derived the same way -- seedGlobalDictionaryFromPersisted computes
+                    // baseline 0 once pd is gone, and checkedRecoveryAnalysis rejects a
+                    // baseline that disagrees with the fold. Discarding a dictionary that
+                    // held entries (size > 0) therefore desynchronised the two and threw
+                    // IllegalStateException("recovery symbol baseline mismatch") out of
+                    // build(). That is NOT an UnreplayableSlotException, so build()'s
+                    // quarantine handler could not catch it and set the slot aside: with a
+                    // stable senderId every restart re-recovered the same slot and threw
+                    // again, so the application could never construct a Sender -- it could
+                    // not even BUFFER new rows. Exactly the outage quarantineTornSlot
+                    // exists to prevent, on a slot that is fully recoverable (its frames
+                    // carry their whole dictionary inline).
+                    //
+                    // Reachable on one transient plus one crash: a session whose
+                    // .symbol-dict fails to open (EIO, fd exhaustion, a Windows share
+                    // lock) falls back to full-dict frames and, per the never-recreate
+                    // contract, leaves the previous session's populated side-file intact;
+                    // if that session then crashes, this recovery opens a dictionary with
+                    // size > 0 next to self-sufficient frames that out-reach it.
+                    //
+                    // Re-folding rather than keeping the dictionary preserves the discard's
+                    // whole point -- the slot recovers in full-dict mode, exactly as it was
+                    // written, with producer, mirror and replay guard all anchored at 0.
+                    recoveredFrameAnalysisInProgress.close();
+                    recoveredFrameAnalysisInProgress = recovered.analyzeRecovery(0);
+                    this.recoveredCommitBoundaryFsn = recoveredFrameAnalysisInProgress.commitBoundaryFsn();
+                    this.recoveredMaxSymbolId = recoveredFrameAnalysisInProgress.maxDeltaEnd() - 1L;
+                    this.recoveredMaxSymbolDeltaStart = recoveredFrameAnalysisInProgress.maxDeltaStart();
                 }
             } else {
                 // Fresh start with no recovered segments. Any stale

core/src/test/java/io/questdb/client/test/cutlass/qwp/client/DeltaDictRecoveryTest.java

@@ -1352,6 +1352,101 @@ public void testTornDictTotalLossRebuildsFromSurvivingFrames() throws Exception
         });
     }
 
+    @Test
+    public void testFullDictFramesRecoverBesideASurvivingPopulatedDictionary() throws Exception {
+        // The populated-side-file twin of testFullDictFramesRecoverInFullDictModeInsteadOf-
+        // Bricking. Self-sufficient frames can also be recovered next to a .symbol-dict
+        // that HOLDS entries, and that slot must recover exactly as the empty-side-file
+        // one does.
+        //
+        // It takes one transient plus one crash. A delta session leaves a populated
+        // .symbol-dict. The next session's open of it fails transiently (EIO, fd
+        // exhaustion, a Windows share lock), so that session runs full-dict fallback --
+        // and PersistedSymbolDict's never-recreate contract deliberately leaves the
+        // older side-file intact rather than truncating the only copy of load-bearing
+        // state. That session writes self-sufficient frames out-reaching the stale
+        // dictionary, then crashes. This recovery opens the survivor: size() > 0,
+        // maxSymbolDeltaStart == 0, recoveredMaxSymbolId >= size().
+        //
+        // CursorSendEngine discards the dictionary here (the frames carry their own, so
+        // it is not needed) -- but the recovery analysis was folded with the dictionary's
+        // size as its baseline, while seedGlobalDictionaryFromPersisted computes baseline
+        // 0 once pd is gone. checkedRecoveryAnalysis then threw
+        // IllegalStateException("recovery symbol baseline mismatch"), which is NOT an
+        // UnreplayableSlotException, so build()'s quarantine handler could not set the
+        // slot aside. With a stable senderId every restart re-recovered the same slot and
+        // threw again: the application could never construct a Sender, so it could not
+        // even buffer new rows -- on a slot that is perfectly recoverable. The engine now
+        // re-folds at baseline 0 when it discards.
+        assertMemoryLeak(() -> {
+            java.nio.file.Path slot = Paths.get(sfDir, "default");
+            java.nio.file.Path dict = slot.resolve(".symbol-dict");
+            // Phase 1: force full-dict fallback with a planted directory, so the frames
+            // land self-sufficient (deltaStart=0) and no side-file is written.
+            java.nio.file.Files.createDirectories(dict);
+            java.nio.file.Path blocker = dict.resolve("blocker");
+            java.nio.file.Files.createFile(blocker);
+            try (TestWebSocketServer silent = new TestWebSocketServer(new SilentHandler())) {
+                int port = silent.getPort();
+                silent.start();
+                Assert.assertTrue(silent.awaitStart(5, TimeUnit.SECONDS));
+                String cfg = "ws::addr=localhost:" + port
+                        + ";sf_dir=" + sfDir
+                        + ";close_flush_timeout_millis=0;";
+                try (Sender s1 = Sender.fromConfig(cfg)) {
+                    for (int i = 0; i < DISTINCT_SYMBOLS; i++) {
+                        s1.table("m").symbol("s", "sym-" + i).longColumn("v", i).atNow();
+                        s1.flush();
+                    }
+                }
+                Assert.assertTrue("full-dict fallback: .symbol-dict must stay a directory",
+                        java.nio.file.Files.isDirectory(dict));
+            }
+
+            // Swap the planted directory for the POPULATED side-file the earlier delta
+            // session left behind. Fewer entries than the frames reference, so the
+            // discard's recoveredMaxSymbolId >= size() guard fires with size() > 0 --
+            // the case that bricked build().
+            java.nio.file.Files.delete(blocker);
+            java.nio.file.Files.delete(dict);
+            final int survivingEntries = 3;
+            Assert.assertTrue(survivingEntries < DISTINCT_SYMBOLS);
+            try (PersistedSymbolDict survivor = PersistedSymbolDict.openClean(slot.toString())) {
+                Assert.assertNotNull("the survivor dictionary must open", survivor);
+                for (int i = 0; i < survivingEntries; i++) {
+                    survivor.appendSymbol("sym-" + i);
+                }
+                Assert.assertEquals(survivingEntries, survivor.size());
+            }
+
+            // Phase 2: recover. build() must SUCCEED, and the self-sufficient frames must
+            // replay gap-free in full-dict mode -- the stale dictionary is discarded, not
+            // trusted, so it cannot misattribute an id.
+            DictReconstructingHandler handler = new DictReconstructingHandler();
+            try (TestWebSocketServer good = new TestWebSocketServer(handler)) {
+                int port = good.getPort();
+                good.start();
+                Assert.assertTrue(good.awaitStart(5, TimeUnit.SECONDS));
+                String cfg = "ws::addr=localhost:" + port + ";sf_dir=" + sfDir + ";";
+                try (Sender ignored = Sender.fromConfig(cfg)) { // must NOT throw
+                    long deadline = System.currentTimeMillis() + 5_000;
+                    while (System.currentTimeMillis() < deadline
+                            && handler.maxDictSize() < DISTINCT_SYMBOLS) {
+                        Thread.sleep(20);
+                    }
+                }
+                Assert.assertFalse("a discarded dictionary leaves full-dict mode, which is "
+                        + "catch-up-free", handler.sawCatchUpFrame);
+                List<String> reconstructed = handler.dictSnapshot();
+                Assert.assertEquals("reconstructed dictionary size",
+                        DISTINCT_SYMBOLS, reconstructed.size());
+                for (int i = 0; i < DISTINCT_SYMBOLS; i++) {
+                    Assert.assertEquals("dictionary id " + i, "sym-" + i, reconstructed.get(i));
+                }
+            }
+        });
+    }
+
     @Test
     public void testFullDictFramesRecoverInFullDictModeInsteadOfBricking() throws Exception {
         // M1 regression (counterpoint to testTornDictTotalLossFailsCleanOnResume): a