Commit cbaf474
Add a per-entry CRC to the SF symbol dictionary
The .symbol-dict side-file was page-cache durable but carried no
integrity check, so a host-crash interior tear -- a lost page reading
back as zeroes, or a stale entry left past the end by a failed
truncate -- could shift the dense id->symbol map without changing the
entry count the send-loop replay guard checks, silently misattributing
symbol-column values on recovery. The SF segment frames are CRC-32C
protected; the dictionary they depend on was not.
Each entry now carries a trailing CRC-32C over its [len][utf8] bytes,
the same checksum MmapSegment uses. openExisting verifies every entry
and stops at the first torn or mismatched one, so recovery trusts only
the intact prefix and the guard then forces a resend of the rest --
turning a silent misattribution into a fail-clean "resend required".
open() strips the CRCs into the wire-shaped loaded-entries buffer, so
the catch-up mirror seed and appendRawEntries keep the file==wire
contract and the send loop stays untouched. The tail truncate is now
failure-checked: a file open() cannot trim is untrusted and recreated
empty rather than left exposing stale bytes.
Bumps the on-disk format to v2 with a version check; existing files
are recreated. A tear that leaves a CRC-matching byte run is still
undetected, a 1-in-2^32 collision per entry, no weaker than the frames.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent fef7203 commit cbaf474
2 files changed
Lines changed: 222 additions & 80 deletions
File tree
- core/src
- main/java/io/questdb/client/cutlass/qwp/client/sf/cursor
- test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor
0 commit comments