roachtest: skip revision history in OR recovery test#170682
Conversation
|
😎 Merged successfully - details. |
7065d9a to
36f9141
Compare
|
@kev-cao i think the commit message could be clearer here. Is this the story?
I know there was some discussion around why the rev history backups were even read, as they should have been compacted away. is there a tracking issue? Uber nit here: In general, when referring to other issues in a PR message or issue write up, i think it is helpful to directly link to the precise comment that explains what is going on, instead of to the top level failure that has a bunch investigate output. |
|
@msbutler Yeah that's the correct story.
Maybe this is getting mixed with something else — even with compactions, we still will ingest instead of link the SSTs. In OR, we begin ingesting the moment we see a revision history backup, so if the full is a revision history backup (which it is when using our driver), we ingest all of the SSTs: cockroach/pkg/backup/restore_span_covering.go Lines 307 to 316 in 67034ac That being said, I had some discussions with David, and I realized that the reason why we were seeing failures was likely not because Pebble was compacting away SSTs before the download phase could reach it. That wouldn't explain why the tests were failing so much more frequently recently. It was because we enabled revision history for OR, which meant that 50% of the time, the tests were guaranteed to fail because the link phase would ingest the SSTs instead of linking them. I'm thinking we merge this PR (skipping revision history is still needed), and then revert the change of only allowing database/table restores). |
The OR recovery roachtest would fail if revision history backups were taken, as the link phase would ingest the SSTs instead of linking them. Since #170148 moved deleting the SSTs to before the link phase, the link phase would then fail due to being unable to find the SSTs. This patch updates the test to avoid taking revision history backups. Fixes: #170617 Release note: None Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
36f9141 to
3ab09d7
Compare
|
TFTR! /trunk merge |
|
Detected infrastructure failure on trunk-merge branch (matched: self-hosted runner lost communication with the server). Automatically resubmitting to merge queue (attempt 1 of 2). (run link) |
|
/trunk merge |
The OR recovery roachtest would fail if revision history backups were
taken, as the link phase would ingest the SSTs instead of linking them.
Since #170148 moved deleting the SSTs to before the link phase, the link
phase would then fail due to being unable to find the SSTs. This patch
updates the test to avoid taking revision history backups.
Fixes: #170617
Release note: None
Co-Authored-By: roachdev-claude roachdev-claude-bot@cockroachlabs.com