VTOrc: fix PrimaryIsReadOnly recovery deadlock against PrimarySemiSyncBlocked

[timvaillancourt]

Description

Extends #19925 to close one more variant of the same recheckPrimaryHealth deadlock pattern. #19925 fixed ReplicationStopped × PrimarySemiSyncBlocked; this PR adds:

1. PrimaryIsReadOnly × PrimarySemiSyncBlocked — issue #20011's primary report. A read-only primary on a semi-sync-blocked shard never recovers because recheckPrimaryHealth aborts fixPrimary every cycle

What changed

• PrimaryIsReadOnly declares BeforeAnalyses: [PrimarySemiSyncBlocked] — mirrors VTOrc: fix ReplicationStopped + PrimarySemiSyncBlocked recovery deadlock #19925's pattern. Routes fixPrimary through GetDetectionAnalysis's suppression bypass so recheckPrimaryHealth no longer aborts it when the shard also has PrimarySemiSyncBlocked. Once fixPrimary clears the read-only state, PrimarySemiSyncBlocked is handled by ERS on the next cycle if it still persists

That's the whole production change — one BeforeAnalyses declaration. The existing GetDetectionAnalysis machinery handles it unchanged

Why not PrimaryDiskStalled too?

An earlier version of this PR also widened BeforeAnalyses to include PrimaryDiskStalled (for both PrimaryIsReadOnly and ReplicationStopped). I dropped that after review feedback because:

• PrimaryDiskStalled only matches when !LastCheckValid && IsDiskStalled — i.e. the primary is unreachable. In that regime, fixPrimary's SetReadWrite and fixReplica's CHANGE REPLICATION SOURCE either fail outright (primary unreachable) or can't fix the underlying disk problem. ERS is the correct action
• Worse: same-tablet selection in GetDetectionAnalysis picks one problem per tablet. If the primary matches both PrimaryIsReadOnly and PrimaryDiskStalled, declaring PrimaryIsReadOnly Before PrimaryDiskStalled causes PrimaryIsReadOnly to be selected and PrimaryDiskStalled to be masked entirely on that tablet — recoverDeadPrimaryFunc is never dispatched. If fixPrimary then fails to clear read-only (plausible on an unreachable primary), ERS is never scheduled and the shard stays wedged
• The PR description's pair-2 motivation ("plausible after a stall recovers but a read-only state was set during it") doesn't actually trigger the pair: once the disk recovers, LastCheckValid flips true and PrimaryDiskStalled stops matching — so the cached read-only state would surface as plain PrimaryIsReadOnly, no pairing to deadlock on

Net: for any pair involving PrimaryDiskStalled, we want ERS to dispatch directly, which is exactly the pre-#20015 behaviour

Testing

Unit tests:

• TestDeclaresBefore / TestDeclaresAfter in analysis_dao_test.go pin down both directions: PrimaryIsReadOnly declares Before PrimarySemiSyncBlocked (positive), and does NOT declare Before PrimaryDiskStalled (negative — guards against accidentally re-introducing the masking issue above)
• TestRecheckPrimaryHealth in topology_recovery_test.go exercises the recheckPrimaryHealth path that was aborting fixPrimary for the new pairing

E2E:

• TestRecoveryDeadlocks in go/test/endtoend/vtorc/general/vtorc_test.go exercises the pairing on a real cluster. The test stops the acker's IO thread, hangs a write on the semi-sync wait, flips super_read_only=ON on the primary, and asserts SuccessfulRecoveries[FixPrimary] increments (pre-fix this counter never incremented because recheckPrimaryHealth aborted the recovery mid-flight) and RecoverDeadPrimary does not. The test inherits the same timing caveat as VTOrc: fix ReplicationStopped + PrimarySemiSyncBlocked recovery deadlock #19925's TestReplicationStoppedWithSemiSyncBlocked — PrimarySemiSyncBlocked is hard to assert deterministically because VTOrc usually fixes the replica faster than we can sustain a blocked write, but the FixPrimary counter increment is sufficient to distinguish pre-fix from post-fix behaviour when the dual condition does coincide during an analysis cycle

Why backport

Same range as #19925 — bug introduced by #18234 (recheckPrimaryHealth), first released in v0.23.0. Should be backported to at least release-23.0 and release-24.0

Related Issue(s)

Resolves: #20011
Related: #19925
Related: #19941

Checklist

• "Backport to:" labels have been added if this change should be back-ported to release branches
• If this change is to be back-ported to previous releases, a justification is included in the PR description
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on CI?
• Documentation was added or is not required

Deployment Notes

No new flags or configuration. The fix changes VTOrc's internal recovery ordering behaviour — when a primary is both PrimaryIsReadOnly and PrimarySemiSyncBlocked, fixPrimary now runs first instead of being indefinitely blocked by recheckPrimaryHealth. ERS still runs on the next cycle if the shard-wide problem persists

AI Disclosure

Claude Code assisted with implementation and testing; I committed the change manually after reviewing each step. Claude prepared this PR summary

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[codecov]

Codecov Report

❌ Patch coverage is 55.55556% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.85%. Comparing base (70c7a72) to head (534b375).
⚠️ Report is 242 commits behind head on main.

Files with missing lines	Patch %	Lines
go/vt/vtorc/inst/analysis_dao.go	55.55%	24 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #20015       +/-   ##
===========================================
- Coverage   69.67%   65.85%    -3.82%     
===========================================
  Files        1614       22     -1592     
  Lines      216793     3778   -213015     
===========================================
- Hits       151044     2488   -148556     
+ Misses      65749     1290    -64459     

Flag	Coverage Δ
partial	65.85% <55.55%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[mattlord]

LGTM. Thanks, @timvaillancourt ! ❤️