Commit 87320e5
fix(ilp): three SF recovery correctness fixes from PR-17 review
Three independent recovery-time bugs in SegmentLog that all let a
durability layer silently produce or operate on a wrong view of the
on-disk log. Each fix has a red regression test that fails on the
unfixed code and passes after the fix.
1. Mid-rotate crash recovery resets FSN sequence to 0.
rotate() has a window between ff.rename(.sfa → .sfs) and the
subsequent createActive(lastSeq + 1) where the process can die or
createActive can throw (allocNativePath OOM, openCleanRW failure,
etc.) leaving on disk: one or more sealed .sfs files, no .sfa.
openInternal saw active==null after scanDirectory and unconditionally
called createActive(FIRST_SEQ=0), restarting FSN assignment at 0
even though sealed segments on disk already covered 0..N. The new
active produced frames whose FSNs collided with sealed FSNs already
on disk, breaking ACK translation, trim, and replay against data
the recovery never saw. Fix derives the new active's baseSeq from
the highest sealed lastSeqOnDisk + 1 (segments is sorted by baseSeq
and sealed ranges are non-overlapping, so the last entry holds the
largest lastSeqOnDisk).
Tests:
- testMidRotateCrashRecoveryPreservesFsnMonotonicity (fault
injection: failNextActiveAllocNativePath inside rotate()).
- testRestartWithOnlySealedSegmentsRecoversCorrectly (independent
coverage via pure on-disk filesystem manipulation — write frames,
manually rename .sfa to .sfs — to exercise the open/recovery code
in isolation from rotate's failure handling, then verify the full
contract: nextSeq, oldestSeq, replay order, and post-restart
append).
2. oldestSeq() returned a removePending segment's baseSeq even though
replay() skips it.
trim() keeps an undeletable sealed segment in the in-memory list as
removePending; replay() correctly skips such segments so already-
acked frames are not re-shipped on reconnect. oldestSeq() returned
segments.getQuick(0).baseSeq unconditionally — including when the
first segment was removePending. WebSocketSendQueue pins
fsnAtZero = oldestSeq() in both the constructor (line 247-248) and
doReconnectCycle (line 925-926), then asserts fsn == fsnAtZero +
wireSeq inside the replay visitor (line 974). The mismatch threw
"SF replay FSN drift" on the first replayed frame; the catch
triggered failConnection(non-fatal); reconnectRequested fired; the
I/O loop re-entered doReconnectCycle, called oldestSeq() again with
the same stale return, and drift fired identically. Permanent
reconnect loop until either the FS issue cleared AND a non-reconnect
trim ran (it can't — the I/O thread is stuck reconnecting), or the
user closed the sender. Fix skips removePending in oldestSeq() the
same way replay() does.
Tests:
- testOldestSeqMustSkipRemovePendingToMatchReplay (unit-level: cross-
check oldestSeq() against the first FSN replay() actually visits).
- testReplaySucceedsWithRemovePendingSegmentAtHeadOfList (end-to-end
integration: real TestWebSocketServer + sender + RemoveFailingSf
Facade; verified pre-fix to reproduce the reconnect loop with "SF
replay FSN drift: fsn=2 expected=0", post-fix the 2 unacked frames
replay successfully and a fresh send reaches the server).
3. Directory scan errors silently treated as EOF / empty log.
Files.findNext()'s contract is 1=success, 0=EOF, -1=read error.
scanDirectory's while (rc > 0) loop exited identically on both 0 and
-1, conflating a real readdir failure (EIO/ESTALE on NFS, etc.) with
normal end-of-directory. Files.findFirst()==0 means either opendir
failed (errno set — transient EACCES/EMFILE/ESTALE/ENOMEM) or the
directory is empty; scanDirectory unconditionally treated it as
"nothing to scan." By the time scanDirectory runs, openInternal has
created the directory if missing and successfully opened+locked the
lock file inside it, so an empty listing is impossible — find==0
here can only mean opendir failed. The silent fallthrough let
openInternal proceed to createActive(...) on top of any unscanned
on-disk segments, aliasing or overwriting still-existing data — the
exact failure mode a durability layer must guard against. Fix
throws SfException in both branches; recovery refuses to proceed
from a partial / unknown view of its own log.
Tests:
- testScanDirectoryFailsWhenFindFirstReturnsZero (FilesFacade
forces findFirst to return 0; pre-fix open silently succeeded
with empty segments and nextSeq=0 over real on-disk data).
- testScanDirectoryFailsWhenFindNextReturnsError (FilesFacade
forces findNext to return -1; same shape, mid-scan readdir
failure is now fatal).
Full module suite: 1994/1994 green (1988 baseline + 6 new tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 83bb368 commit 87320e5
3 files changed
Lines changed: 971 additions & 10 deletions
File tree
- core/src
- main/java/io/questdb/client/cutlass/qwp/client/sf
- test/java/io/questdb/client/test/cutlass/qwp/client/sf
Lines changed: 59 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
394 | 394 | | |
395 | 395 | | |
396 | 396 | | |
397 | | - | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
398 | 406 | | |
399 | 407 | | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
405 | | - | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
406 | 420 | | |
407 | | - | |
| 421 | + | |
408 | 422 | | |
409 | 423 | | |
410 | 424 | | |
| |||
503 | 517 | | |
504 | 518 | | |
505 | 519 | | |
506 | | - | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
507 | 537 | | |
508 | 538 | | |
509 | 539 | | |
510 | 540 | | |
511 | 541 | | |
512 | 542 | | |
513 | 543 | | |
514 | | - | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
515 | 555 | | |
516 | 556 | | |
517 | 557 | | |
| |||
526 | 566 | | |
527 | 567 | | |
528 | 568 | | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
529 | 578 | | |
530 | 579 | | |
531 | 580 | | |
| |||
0 commit comments