Skip to content

fix(rekey): preserve every durable main page - #19

Merged
farhan-syah merged 2 commits into
NodeDB-Lab:mainfrom
presempathy-awb:codex/upstream/rekey-state-integrity
Jul 26, 2026
Merged

fix(rekey): preserve every durable main page#19
farhan-syah merged 2 commits into
NodeDB-Lab:mainfrom
presempathy-awb:codex/upstream/rekey-state-integrity

Conversation

@presempathy-awb

@presempathy-awb presempathy-awb commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Completes the main-database side of an online rekey before the source epoch is
retired, and stops the rekey protocol abandoning the pages its own transitions
supersede.

Three categories of durable page were still sealed under the source epoch when
rekey_db returned:

  1. reusable pages named by the current durable free list;
  2. the live catalog tree, apart from the copy-on-write path a rekey happens to
    write through; and
  3. pages superseded by the rekey's own catalog transitions, which reached no
    root and no free list.

Ordinary live-root reads looked healthy, because none of those pages is reached
from a live root. A physical integrity walk — which authenticates every non-zero
page in main.db — failed after reopen, since successful retirement had removed
the only key that could authenticate them.

No public API, persisted format, command-line interface, feature flag, or
dependency changes.

The catalog is the root that cannot be re-sealed early

Every other reader-visible root is re-sealed before the target header is
published. The catalog is the exception: until that header is durable it must
stay source-readable, because it carries the rekey intent that an open able to
verify only the stale A/B side uses to admit recovery.

So it is re-sealed after publication instead, while both keys are still
installed — using the same bounded, deduplicating traversal the data tree uses.
Work is proportional to unique reachable catalog pages, not to the size of the
file.

This was easy to miss because the default retention policy hides it. With
history retained, the newest retained row names the live catalog root, so the
retained-root walk covers the catalog incidentally. With retention disabled,
only the pages a transition writes through get re-sealed, and everything else in
a multi-page catalog stays under an epoch that retirement then destroys.

Superseded pages are reclaimed, not abandoned

commit_rekey_catalog_root had no freed-page parameter, so each durable stage —
intent, header publication, main-done, segments-pending, segment progress rows,
and the final clear — abandoned the catalog pages its copy-on-write rewrite
superseded. Unreachable from every root, absent from the free list, and once the
source epoch retired, unauthenticatable. The leak grew with every rekey.

Every rekey-time catalog rewrite now routes its leftovers through the durable
free list, exactly as ordinary commits do. Entries carry the current commit id,
so the reclamation floor still withholds any page a live reader or retained
history root can still name; chain pages carry the id that marks writer-only
metadata immediately recyclable, matching the ordinary commit path.

Fixing the accounting is also what makes the rest cheap: once the superseded
pages are on the free list, the free-list pass re-seals them, and no scan of the
whole file is needed to find them.

Free-list pages

The free-list traversal previously decoded the current chain but re-sealed only
the pages storing the chain records. Those records also name reusable pages
whose envelopes can represent any legal main.db page kind, and they are
intentionally unreachable from live roots.

They are re-sealed rather than cleared. A page on the free list can still be
pinned below the reclamation floor by a live reader, so zeroing it would corrupt
a snapshot that is still legally readable.

The page kind comes from the cleartext Format-A header so the caller can select
the AAD binding, but the pager still authenticates the full envelope under that
same kind before marking it dirty. A tampered kind byte cannot smuggle a page
into another role — it fails the tag.

Bounded memory

Dirty pages are never evicted, so a rekey that re-seals a whole database before
a single flush would pin the entire decrypted store in the buffer pool no matter
what budget the caller configured. rewrite_page_under_current_epoch now flushes
whenever the dirty set reaches that budget.

Flushing mid-walk is safe and idempotent: the active epoch is already the target,
so a page re-read after being re-sealed opens under the same key, and a crash
resumes from the durable intent with both keys installed.

Ordering

The catalog and free-list passes run after the target-authenticated header is
durable but before the intent advances from HeaderTargetPublished to
MainDone:

  • the target header is already a safe recovery anchor;
  • the source key is still installed, so residual source pages remain readable;
  • a crash during the passes reopens with the durable intent and both epoch keys,
    making them idempotent;
  • they are flushed before MainDone is published; and
  • from that point every catalog page is target-sealed, so later transitions can
    only supersede target-epoch pages.

Regression coverage

tests/rekey_basic.rs. Each new test was confirmed to fail against the
production change it pins.

  • rekey_reseals_a_catalog_larger_than_one_page — builds a catalog spanning
    several pages with retention disabled, rekeys, reopens, and runs the full deep
    physical integrity walk. Stubbing out the catalog traversal makes it fail.
  • rekey_leaves_no_leaked_pages — rekeys three times in a row and asserts on
    orphan_page_ids directly, because DeepWalkReport::is_clean() deliberately
    excludes orphans and so cannot see a leak. Without freed-page accounting it
    reports one orphan per rekey plus eleven authentication failures.
  • rekey_preserves_durable_free_list_across_reopen — deletes most records with
    retention disabled, proves the durable free list is populated, rekeys, reopens
    using only the live transition state, and walks the file.
  • rekey_preserves_preexisting_main_reader_snapshot_after_cache_evict — proves
    a reader opened before the rekey keeps its pinned logical snapshot after clean
    cache entries are evicted, so the expanded physical rewrite leaves epoch leases
    and retained snapshots alone.
  • rekey_rejects_non_advancing_epoch — a non-advancing epoch is rejected without
    persisting an intent or poisoning an otherwise healthy store.

The existing segment-reconciliation failure case is already covered by
rekey_segment_reconciliation_failure_poisoned_handle_reopens_with_readable_segment
in tests/durability/unpublished_commit.rs; it passes unchanged and is not
duplicated here.

Verification

On this branch, rebased onto main:

  • cargo nextest run --all-features --no-fail-fast: 536 passed, 5 skipped;
  • the same suite under PAGEDB_INVARIANT_CHECKS=1: 536 passed, 5 skipped;
  • cargo fmt --all --check: pass;
  • cargo clippy --all-targets --all-features -- -D warnings: pass;
  • cargo check --no-default-features: pass;
  • cargo check --target wasm32-unknown-unknown --features opfs --lib: pass;
  • cargo doc --no-deps --all-features: no warnings.

Review focus

  1. Does every path that retires the source epoch first flush the catalog and
    free-list passes?
  2. Can a crash at HeaderTargetPublished safely repeat them with both epoch keys
    installed?
  3. Is the current commit id the right reclamation floor for pages a rekey
    transition superseded?
  4. Are pages created after those passes necessarily target-epoch pages?
  5. Does re-sealing free-list pages, rather than clearing them, correctly respect
    readers pinned below the reclamation floor?

presempathy-awb and others added 2 commits July 27, 2026 06:26
Re-encrypt durable free-list entries and residual copy-on-write pages before retiring the source epoch. Add regression coverage for free-list integrity, pinned reader snapshots, and monotonic epoch validation.
Rekey previously re-sealed main.db by scanning every allocated page
between the reserved header and the allocation cursor, which skipped
the catalog until its own commit. It also discarded the pages each
catalog copy-on-write rewrite superseded instead of folding them into
the free list, leaking them permanently once the source epoch was
retired. Replace the residual-page scan with a bounded, deduplicating
walk of the live catalog tree, and route every rekey-time catalog
rewrite's freed pages through the durable free list before recording
the new allocation cursor.

Bound the memory a rekey walk pins by flushing the buffer pool mid-walk
once the dirty set reaches the configured budget, since dirty cache
entries are never evicted.
@farhan-syah
farhan-syah force-pushed the codex/upstream/rekey-state-integrity branch from d50192f to 3287ab0 Compare July 26, 2026 22:43
@farhan-syah
farhan-syah merged commit 22993a1 into NodeDB-Lab:main Jul 26, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants