chore: use deterministic waits or latch where ever possible

[ash-thakur-rh]

Description

Fixes #7391

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change
• Chore (non-breaking change which doesn't affect codebase;
  test, version modification, documentation, etc.)

Checklist

• Code contributed by me aligns with current project license: Apache 2.0
• I Added CHANGELOG entry regarding this change
• I have implemented unit tests to cover my changes
• I have added/updated the javadocs and other documentation accordingly
• No new bugs, code smells, etc. in SonarCloud report
• I tested my code in Kubernetes
• I tested my code in OpenShift

[manusa]

Hi @ash-thakur-rh

I'm rescoping #7391 — the broader effort (deterministic gates / invariant-driven refactors, sometimes touching production code) is going to be tracked as follow-up work rather than as one sweeping change.

The changes already in this PR are exactly the ones I'd like to keep, since each one targets the actual invariant under test rather than just swapping a sleep for an await:

• UtilsTest.testSerialExecution — the two-latch pattern (taskStarted / taskCanComplete) deterministically holds the task open across the window where the scheduler might re-fire. Good.
• AbstractWatchManagerTest.reconnectRace — the scheduleReconnectDone latch is keyed on the event the race actually depends on, with a real timeout assertion. Good.
• SerialExecutorTest.clearInterrupt — new CountDownLatch(1).await() cleanly replaces the busy-spin loop and the behavior under test (interrupt propagation) is unchanged. Good.
• SerialExecutorTest.taskExecutedInOrderOfInsertion — gating task provide a callback based mutate API for easier load + mutate + save of resources #1 on allSubmitted is the right fix; the original Thread.sleep(100) was a wish that all 5 tasks would queue in time, the latch makes it a guarantee. Good.

Proposal: please keep this PR scoped exactly to these four changes, move it out of draft, and we can land it as-is. Anything else that was originally in #7391's scope (LeaderElectorTest, WatchIT, and whatever else turns up) we'll handle as separate follow-ups under the rescoped issue.

Thanks for the careful work here.

[manusa]

Quick follow-up: while rescoping #7391 I noticed one of the four changes in this PR (SerialExecutorTest.taskExecutedInOrderOfInsertion) was independently fixed in main since you opened the draft — commit 8e482f5 uses the same kind of start latch you proposed. After rebase that hunk becomes a no-op, so the in-scope set here is three changes, not four:

• UtilsTest.testSerialExecution (two-latch pattern around the re-fire window) — category C in the rescoped issue
• AbstractWatchManagerTest.reconnectRace (latch on scheduleReconnect completion) — category B
• SerialExecutorTest.clearInterrupt (new CountDownLatch(1).await() replaces the busy-spin) — category A

Categories D (LeaderElectorTest.runShouldAbortAfterRenewDeadlineExpired — needs a production-side clock hook) and E (WatchIT — withRequestConfig for a shorter timeout) will be separate follow-up PRs after this one lands.