Skip to content

roachtest: skip revision history in OR recovery test#170682

Merged
trunk-io[bot] merged 1 commit into
cockroachdb:masterfrom
kev-cao:roachtest/deflake-or-recovery-revision
May 21, 2026
Merged

roachtest: skip revision history in OR recovery test#170682
trunk-io[bot] merged 1 commit into
cockroachdb:masterfrom
kev-cao:roachtest/deflake-or-recovery-revision

Conversation

@kev-cao

@kev-cao kev-cao commented May 20, 2026

Copy link
Copy Markdown
Contributor

The OR recovery roachtest would fail if revision history backups were
taken, as the link phase would ingest the SSTs instead of linking them.
Since #170148 moved deleting the SSTs to before the link phase, the link
phase would then fail due to being unable to find the SSTs. This patch
updates the test to avoid taking revision history backups.

Fixes: #170617

Release note: None

Co-Authored-By: roachdev-claude roachdev-claude-bot@cockroachlabs.com

@trunk-io

trunk-io Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

😎 Merged successfully - details.

@cockroach-teamcity

Copy link
Copy Markdown
Member

This change is Reviewable

@kev-cao kev-cao requested review from a team and msbutler and removed request for a team May 20, 2026 20:14
@kev-cao kev-cao force-pushed the roachtest/deflake-or-recovery-revision branch from 7065d9a to 36f9141 Compare May 21, 2026 14:25
@msbutler

msbutler commented May 21, 2026

Copy link
Copy Markdown
Collaborator

@kev-cao i think the commit message could be clearer here. Is this the story?

  • currently, the OR recovery roachtest deletes SSTs before the link phase
  • online restore of revision history backups necessarily have to read rev history backups ssts during the link phase, so this test fails before the download phase begins.

I know there was some discussion around why the rev history backups were even read, as they should have been compacted away. is there a tracking issue?

Uber nit here: In general, when referring to other issues in a PR message or issue write up, i think it is helpful to directly link to the precise comment that explains what is going on, instead of to the top level failure that has a bunch investigate output.

@kev-cao

kev-cao commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

@msbutler Yeah that's the correct story.

I know there was some discussion around why the rev history backups were even read, as they should have been compacted away. is there a tracking issue?

Maybe this is getting mixed with something else — even with compactions, we still will ingest instead of link the SSTs. In OR, we begin ingesting the moment we see a revision history backup, so if the full is a revision history backup (which it is when using our driver), we ingest all of the SSTs:

// Find the first layer with revision history. Once we hit a layer with
// revision history, that layer and all subsequent layers must be ingested
// (not linked) to maintain correct MVCC/LSM ordering.
firstRevHistoryLayer := len(backups) // default: no revision history layers
for i, b := range backups {
if !b.RevisionStartTime.IsEmpty() || b.MVCCFilter == backuppb.MVCCFilter_All {
firstRevHistoryLayer = i
break
}
}


That being said, I had some discussions with David, and I realized that the reason why we were seeing failures was likely not because Pebble was compacting away SSTs before the download phase could reach it. That wouldn't explain why the tests were failing so much more frequently recently.

It was because we enabled revision history for OR, which meant that 50% of the time, the tests were guaranteed to fail because the link phase would ingest the SSTs instead of linking them.

I'm thinking we merge this PR (skipping revision history is still needed), and then revert the change of only allowing database/table restores).

The OR recovery roachtest would fail if revision history backups were
taken, as the link phase would ingest the SSTs instead of linking them.
Since #170148 moved deleting the SSTs to before the link phase, the link
phase would then fail due to being unable to find the SSTs. This patch
updates the test to avoid taking revision history backups.

Fixes: #170617

Release note: None

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
@kev-cao kev-cao force-pushed the roachtest/deflake-or-recovery-revision branch from 36f9141 to 3ab09d7 Compare May 21, 2026 17:23
@kev-cao

kev-cao commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

TFTR!

/trunk merge

@blathers-crl

blathers-crl Bot commented May 21, 2026

Copy link
Copy Markdown

Detected infrastructure failure on trunk-merge branch (matched: self-hosted runner lost communication with the server). Automatically resubmitting to merge queue (attempt 1 of 2). (run link)

@blathers-crl

blathers-crl Bot commented May 21, 2026

Copy link
Copy Markdown

/trunk merge

@trunk-io trunk-io Bot merged commit 969083d into cockroachdb:master May 21, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

roachtest: backup-restore/online-restore-recovery failed

3 participants