[#290] Report leftover MVV mark bits in IntegrityCheck#291
Conversation
maximthomas
left a comment
There was a problem hiding this comment.
The detection half is solid — countMarkedVersions terminates safely (getLength() masks to 0x7FFF, so from advances by at least LENGTH_PER_VERSION), the dedicated scan is the right call over extending VersionVisitor, and MVVTest (40/40) + IntegrityCheckTest (10/10) are green on the branch.
The problem is the remediation the PR prescribes. Requesting changes on that.
icheck -p remediation silently resurrects deleted keys (high)
The PR says "icheck -p remains the remediation path … marks are reported first and then rewritten by the prune." That is not safe: MVV.prune uses the same bit as its private "keep this version" flag and never clears pre-existing marks before pass 1. A leftover mark is therefore read as prune's own decision to keep:
persistit/core/src/main/java/com/persistit/MVV.java:530-546— the primordial branch promotes the first marked version it finds. With a stale mark on an older version, the obsolete version wins. Note the branch is still entered:markedcounts only prune's ownmark()calls (1), while two versions actually carry the bit.persistit/core/src/main/java/com/persistit/MVV.java:508-524— pass 2 registers only!isMarkedversions inprunedVersionList. The marked version that pass 3 doesn't promote is therefore discarded without ever being registered, so its long-record page (lines 515-520) is never deallocated. Observed:prunedVersionListis empty in the stale-mark run vs. one entry in the control.
Reproduced on this branch at unit level (two committed versions, mark on the older one, MVV.prune called directly):
leftoverMark=false survivor=[22 22] prunedVersionList.size=1 <- correct, newest version
leftoverMark=true survivor=[11 11 11] prunedVersionList.size=0 <- stale version promoted
And end-to-end through the exact path the PR recommends — testPruneClearsLeftoverMarkBits plus a before/after snapshot of the visible key set, with a control run that calls corrupt5's ex.store() but skips MVV.mark:
>>> doMark=false markedVersions=0 faults=0 keysBefore=500 keysAfter=500
>>> doMark=true markedVersions=10 faults=10 keysBefore=500 keysAfter=505
icheck -p undeleted 5 transactionally removed keys. The mark sat on the value version, so prune promoted it instead of the AntiValue. The control rules out corrupt5's non-transactional store as the cause.
The bug lives in MVV.prune and predates this PR, but this PR is what tells operators to run icheck -p on affected volumes, and testPruneClearsLeftoverMarkBits asserts remediation "works" while only checking counters. Suggested:
- Make
prunedefensive about its own invariant — clear any pre-existing marks before pass 1 (you already have the scan):
// MVV.prune, before the first pass
if (countMarkedVersions(bytes, offset, length) > 0) {
int from = offset + 1;
while (from + LENGTH_PER_VERSION <= offset + length) {
final int vlength = getLength(bytes, from);
unmark(bytes, from);
from += vlength + LENGTH_PER_VERSION;
}
}- Add the data-preservation assertion to
testPruneClearsLeftoverMarkBitsin
persistit/core/src/test/java/com/persistit/IntegrityCheckTest.java:132— snapshot the visible key/value set before corruption, compare after remediation. That single assertion is what surfaces this.
icheck -P (pruneAndClear) regresses on affected volumes (medium)
_faults is never cleared (reset() at persistit/core/src/main/java/com/persistit/IntegrityCheck.java:786 doesn't touch it), and the new fault is added during buffer.verify — i.e. before the prune in the same verifyPage call. So the gate at persistit/core/src/main/java/com/persistit/IntegrityCheck.java:315 can never pass on an affected volume: resetMVVCounts is skipped even though the prune fully rewrote the marks.
Confirmed by running the real runTask() path (icheck trees=* -u -P -v), against a control that stores the same records without setting the mark bit:
>>> doMark=false marked=0 faults=0 -> "0 aborted transactions were cleared by pruning"
>>> doMark=true marked=10 faults=10 -> "PruneAndClear failed to remove all aborted MMVs"
Before this PR the same volume ran -P to completion.
Options: keep marked-version faults in a separate list excluded from the -P gate; re-verify after pruning and drop the fault when the count reaches zero; or at minimum document "run icheck -P twice".
Marked-version count runs on an unvalidated MVV (low)
persistit/core/src/main/java/com/persistit/IntegrityCheck.java:206 calls countMarkedVersions before visitAllVersions validates the record, so a structurally corrupt MVV is walked with garbage lengths. Confirmed — an MVV with only a damaged length field and no mark bit anywhere:
intact -> marked=0
corrupt -> marked=1
That yields a bogus MVV has 1 version with a leftover mark bit fault and an inflated MarkedVersions counter next to the genuine CorruptValueException fault — misleading exactly when an operator is using the counter to answer "am I affected by #286?". The inline comment says the count "must be counted before visitAllVersions", but there is no such constraint — visitAllVersions does not mutate bytes. Counting after a clean traversal (or sharing one traversal) fixes it.
Nits
- Doc drift:
persistit/doc/Management.rst:136documents theicheck -cCSV header verbatim and isn't updated withMarkedVersions. Worth noting in the PR description that both the CSV columns and the plain-report format changed — breaking for anything parsing icheck output. - Redundant page tracking:
verifyPagealready holdspageatpersistit/core/src/main/java/com/persistit/IntegrityCheck.java:1187; a field set there would remove thevisitPageoverride and the duplicated_currentPagestate. - Copyright header:
MVV.javaandMVVTest.javacarryPortions Copyrighted 2026 3A Systems, LLC;persistit/core/src/test/java/com/persistit/IntegrityCheckTest.javais modified without it (recent commits3f8c98909,cca1422d7do add it). corrupt5duplication: it is a byte-for-byte copy ofcorrupt2except for the mutation — a shared helper taking the mutation would be cleaner. Both also leakignoreMVCCFetch(true)if the loop throws (nofinally).- Test granularity:
testLeftoverMarkBitsReportedmarks one version per record, sofaults == markedVersionCountby construction. Marking two versions in one record would cover the counter/fault distinction and theplural()branch. - Line length:
persistit/core/src/main/java/com/persistit/IntegrityCheck.java:209is 121 chars, one over the file's 120-col wrap (cosmetic — no formatter config in the repo).
d9ed39f to
79b2cd7
Compare
|
Thanks for the thorough review — the prune reproduction especially. All points addressed; the branch is force-pushed as 79b2cd7 on a reordered stack.
Count on an unvalidated MVV (low) — fixed: Nits:
Local runs on the new head: |
|
@maximthomas ready for another look — all review points are addressed, see the point-by-point summary above (#291 (comment)). New head is 79b2cd7. Highlights:
#293 would need your review as well — it carries the actual |
83e50d6 to
b703ccb
Compare
maximthomas
left a comment
There was a problem hiding this comment.
One issue left.
-P gate can be defeated by fault-list truncation (medium)
// persistit/core/src/main/java/com/persistit/IntegrityCheck.java:319
if (_faults.size() == _markedVersionFaultCount && _counters._mvvPageCount == _counters._prunedPageCountaddFault drops faults past MAX_FAULTS (200) and returns false, so _markedVersionFaultCount only counts recorded ones:
// persistit/core/src/main/java/com/persistit/IntegrityCheck.java:361
final boolean recorded = _faults.size() < MAX_FAULTS;Once 200 mark faults fill the list, _faults.size() == _markedVersionFaultCount == 200 and every later fault is silently discarded — including genuine structural ones. The equality still holds, so resetMVVCounts clears the TransactionIndex on a volume with real corruption. _faults.isEmpty() could not be fooled this way.
Reproduced with icheck trees=* -u -P on identically built volumes — same corrupted key in both, only the distribution of marks differs:
marks on keys >=2000 markedRecords=999 faults=200 nonMarkRecorded=2 CLEARED=false FAILED=true
marks on all keys markedRecords=2999 faults=200 nonMarkRecorded=0 CLEARED=true FAILED=false
The first row proves the key corruption is real and detectable here (2 non-mark faults recorded, gate correctly refuses). In the second, 200 mark faults fill the list before the traversal reaches that page, the genuine faults are discarded, and -P clears the TransactionIndex.
Suggested fix: count non-mark faults independently of the cap rather than deriving them by subtraction — e.g. a _nonMarkFaultCount incremented in addFault before the recorded check (with the mark-bit call site using a separate path), and gate on it being zero.
Nits
- Sweep-only prune does not dirty the page —
Buffer.pruneMvvValuessetschangedonly whennewSize != oldSize(persistit/core/src/main/java/com/persistit/Buffer.java:3699) and dirties only whenchanged(Buffer.java:3625). Clearing a stale mark never changes the size, so if prune keeps exactly the versions already present the repair stays in the page image only. Verified at the unit level (pruneMvvValues changed = falsewhile the mark count went 1 → 0), but I could not construct an end-to-end case where a repair is actually lost — reaching it needs a concurrent transaction pinning every version of an already-marked page, and the quiesced path demonstrably persists. Worth a comment at most; mentioning it because it is the one way the-Prationale atIntegrityCheck.java:311-318could be untrue. _prunedPageCountcounts no-op prunes —persistit/core/src/main/java/com/persistit/IntegrityCheck.java:1204-1205ignorespruneMvvValues's return value and increments unconditionally, so the_mvvPageCount == _prunedPageCounthalf of the gate reduces to "prune did not throw". Pre-existing, but it is load-bearing for the gate this PR changes._markedVersionFaultCountnot cleared inreset()—persistit/core/src/main/java/com/persistit/IntegrityCheck.java:792. Consistent with_faults, so only a latent hazard for reused instances.
maximthomas
left a comment
There was a problem hiding this comment.
See
#291 (review) comment
|
All points addressed — on the branch (now rebased onto #293's tip) plus one follow-up commit, 9d68015.
While wiring that up I found and fixed an edge in the gate itself (9d68015): comparing Marked-version count on an unvalidated MVV (low) — the scan now runs after Nits —
|
A prune interrupted before the issue OpenIdentityPlatform#286 fix could leave 0x8000 mark bits behind in on-disk version length fields. Since PR OpenIdentityPlatform#288 the read paths strip the mark bit silently, so such versions read normally and icheck reported affected volumes as clean. Add MVV.countMarkedVersions and scan every data record during icheck: marked versions are counted in a new MarkedVersions counter (report, CSV and getMarkedVersionCount) and reported as a fault so affected volumes can be identified. The scan runs only after visitAllVersions has validated the version structure, so a structurally corrupt MVV cannot produce a bogus marked-version fault. icheck -p remains the remediation: pruning rewrites the marked versions (issue OpenIdentityPlatform#292 makes prune clear stale marks unconditionally). With icheck -P the marked-version faults are reported but do not block clearing the TransactionIndex, since the prune pass has already rewritten the marks. The icheck -c CSV example in Management.rst is updated for the new MarkedVersions column. Fixes OpenIdentityPlatform#290
The pruneAndClear gate excluded marked-version faults by comparing _faults.size() with the marked-fault count, but both stop moving once the fault list reaches MAX_FAULTS: on a volume with more than MAX_FAULTS marked-version records a genuine fault found afterwards (the garbage chain is checked after the trees) is dropped from the list and the arithmetic still converges, so -P cleared the TransactionIndex despite real corruption. Count every fault found - recorded in the list or not - and gate on the counts instead.
9d68015 to
a69c0b8
Compare
Summary
Implements the proposal from #290 (split out of PR #288, suggested by @maximthomas in the #288 review):
IntegrityChecknow detects and reports MVV versions whose length field still carries a leftover0x8000mark bit — the on-disk residue of a prune interrupted before the #286 fix. Since PR #288 the read paths strip the mark bit silently, so such versions read normally andicheckreported affected volumes as clean; the mark survived until some future prune rewrote that version.Changes
MVV.countMarkedVersions(bytes, offset, length): a dedicated scan counting versions with the mark bit set. A separate scan rather than an extension ofMVV.VersionVisitor: the interface has six implementations, andsawVersionhands out the post-header value offset (with no header at all for primordial values), so a visitor cannot see the mark anyway.IntegrityCheck:visitDataRecordscans each record for marks aftervisitAllVersionshas validated the version structure, so a structurally corrupt MVV cannot produce a bogus marked-version fault next to its genuineCorruptValueExceptionfault. Marked versions feed a newMarkedVersionscounter (plain report, CSV, and agetMarkedVersionCount()getter) and add aFault("MVV has N versions with a leftover mark bit") with the page address and record position, sohasFaults()flags affected volumes.icheck -premains the remediation path:verifyPagerunsbuffer.verifybefore pruning, so marks are reported first and then rewritten by the prune ([persistit] MVV.prune can promote a stale-marked old version during primordial conversion #292 makes prune clear stale marks up front).icheck -P(pruneAndClear): marked-version faults are reported but do not blockresetMVVCounts— by the time the run completes, the prune pass has already rewritten the marks, so only other faults and incomplete pruning count as failures. Without this,-Pcould never succeed on an affected volume.-Pgate compares fault counts, not theMAX_FAULTS-capped_faultslist: on a volume with more thanMAX_FAULTSmarked-version records the list holds only marked-version faults, and a genuine fault found afterwards (the garbage chain is checked after the trees) is dropped from the list — comparing list size against the marked-fault count would converge and clear the TransactionIndex despite real corruption.addFault/addGarbageFaultnow count every fault found, recorded or not.Tests
MVVTest:countMarkedVersions(mark/unmark transitions over a two-version MVV) andcountMarkedVersionsNonMVV(non-MVV, zero and negative lengths).IntegrityCheckTest:corrupt2/corrupt5share acorruptMVVshelper taking the mutation as a callback (parameterized by start key and record limit), withtry/finallyaroundignoreMVCCFetch.corrupt5marks every version of each corrupted MVV, so the marked-version count exceeds the fault count (one fault per record) and the plural fault message is exercised.testLeftoverMarkBitsReportedasserts a clean tree reports nothing and a corrupted tree reports exactly one fault per corrupted record plus the exact marked-version total.testBrokenMVVsadditionally asserts a structurally corrupt MVV does not inflate theMarkedVersionscounter.testPruneClearsLeftoverMarkBitssnapshots the visible key/value contents before corruption and asserts remediation (icheckwith pruning) leaves them unchanged, then that a subsequenticheckis clean.testPruneAndClearWithLeftoverMarkBitsruns the real task path (CLI.parseTask("icheck trees=* -u -P")) and asserts the faults are reported while the TransactionIndex is still cleared ("aborted transactions were cleared by pruning", not "PruneAndClear failed").testPruneAndClearBlockedByFaultBeyondFaultCapcorrupts more thanMAX_FAULTSrecords with leftover marks plus the garbage chain and asserts-Pstill refuses to clear ("PruneAndClear failed") — with the gate comparing the capped list instead of counts, this test fails with a wrongly cleared TransactionIndex.Local runs:
IntegrityCheckTest12/12,MVVTest45/45,MVCCBasicTest+MVCCPruneTest+MVCCPruneBufferTest+TransactionIndexTest+Bug1017957Testall green (1 pre-existing skip).Fixes #290