Truncate the torn tail when reopening the symbol dict

openExisting parsed complete entries and set appendOffset past the last
one, but left the file at its full length. A crash mid-append leaves a
torn trailing record; if the next append after recovery is SHORTER than
that torn tail, it overwrites only the tail's prefix and leaves residue
beyond its own end. A later recovery can then mis-parse that residue as a
ghost symbol, shifting every subsequent dense id -- so the "self-healing
tail" guarantee was not actually airtight.

open() now truncates the file to the end of the last complete entry
(ftruncate) so nothing survives past appendOffset. Best-effort: a failed
truncate falls back to the prior overwrite-from-appendOffset behaviour.

testTornTrailingEntrySelfHeals now asserts the file returns to its clean
length after the reopen; reverting the truncate fails it (19 vs 16 bytes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: glasstiger

core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/PersistedSymbolDict.java

@@ -328,8 +328,18 @@ private static PersistedSymbolDict openExisting(String filePath, long fileLen) {
             }
             Unsafe.free(buf, len, MemoryTag.NATIVE_DEFAULT);
             buf = 0L;
-            // appendOffset lands just past the last complete entry, so the next
-            // append overwrites any torn trailing bytes.
+            // Physically drop any torn/stale trailing bytes (a crash mid-append)
+            // so a LATER, shorter append cannot leave residue past its own end
+            // that a future recovery mis-parses as a ghost symbol -- which would
+            // shift every subsequent dense id. Without this, appendOffset already
+            // lands just past the last complete entry, so the next append
+            // overwrites FROM there, but only truncation guarantees nothing
+            // survives BEYOND it. Best-effort: a failed truncate just forgoes the
+            // hardening and falls back to the overwrite-from-appendOffset behaviour.
+            long validLen = HEADER_SIZE + entriesLen;
+            if (validLen < len) {
+                Files.truncate(fd, validLen);
+            }
             return new PersistedSymbolDict(fd, HEADER_SIZE + entriesLen, count, entriesAddr, entriesLen);
         } catch (Throwable t) {
             if (buf != 0L) {

core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/PersistedSymbolDictTest.java

@@ -229,17 +229,24 @@ public void testTornTrailingEntrySelfHeals() throws Exception {
                 d.close();
 
                 Path f = dir.resolve(".symbol-dict");
+                long cleanLen = Files.size(f); // header + "one" + "two", no tail
                 Files.write(f, new byte[]{(byte) 5, (byte) 'x', (byte) 'y'},
                         StandardOpenOption.APPEND);
+                Assert.assertEquals("torn tail present before reopen", cleanLen + 3, Files.size(f));
 
                 // Reopen: the torn tail is ignored; only the two complete entries load.
                 PersistedSymbolDict re = PersistedSymbolDict.open(dir.toString());
                 try {
+                    // open() physically truncates the torn tail: the file returns
+                    // to its clean length, so a later SHORTER append can never
+                    // leave residue past its end that a future recovery mis-parses
+                    // as a ghost symbol.
+                    Assert.assertEquals("torn tail physically dropped by open", cleanLen, Files.size(f));
                     Assert.assertEquals(2, re.size());
                     ObjList<String> s = re.readLoadedSymbols();
                     Assert.assertEquals("one", s.getQuick(0));
                     Assert.assertEquals("two", s.getQuick(1));
-                    // The next append overwrites the torn tail, keeping the file consistent.
+                    // The next append continues from the truncated tail cleanly.
                     re.appendSymbol("three");
                     Assert.assertEquals(3, re.size());
                 } finally {