Commit e31fe74
fix(compactor): de-flake sharded-ownership compactor tests
TestCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning
and its partition twin seed one shared bucket.ClientMock with 100
tenants (~3,208/~3,707 registered testify expectations). Every bucket
operation from both compactors and both blocks cleaners holds the
mock's single mutex through an O(expectations) reflective scan (~38%
of CPU in two independent profiles), so the first compaction cycle
alone takes ~20s under -race on an idle machine and 6-7x longer on
starved CI runners. Each per-tenant meta sync additionally runs under
a deadline equal to -compactor.compaction-interval (5s in these tests,
pkg/compactor/compactor.go:1085), so starved syncs turn whole runs
into failures and CompactionRunsCompleted can stay pinned at 0 longer
than ANY poll budget: in CI run 26632776611 the partition test failed
its 60s poll AND the non-partition sibling burned its full 120s poll,
in both attempts. The ring itself was ACTIVE and stable before the
poll started (starting() waits for both), correcting the issue's
original ring-convergence framing.
Fix (test-only):
- Cut numUsers from 100 to 20 in both tests (~25x less quadratic
mock-matching work; the per-user ownership assertions are
tenant-count independent).
- Align the partition test's run-completion poll budget with its
non-partition sibling (60s -> 120s) as a seatbelt.
- Set WaitActiveInstanceTimeout=30s in both tests (same hardening as
#7503): prepareConfig's 5s ACTIVE ceiling is within reach of the
observed CI starvation envelope.
The shuffle-sharding twins are deliberately untouched: their live
arm64 failure in the same run is an ownership-exclusivity violation
(compactor_test.go:1318), a different failure class that needs its own
root-causing.
Before/after on Apple Silicon, -race: 21.00s/24.01s -> 3.57s/3.55s.
Fixes #7607
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Sandy Chen <Yuxuan.Chen@morganstanley.com>1 parent 74185ef commit e31fe74
3 files changed
Lines changed: 6 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
| 60 | + | |
60 | 61 | | |
61 | 62 | | |
62 | 63 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1142 | 1142 | | |
1143 | 1143 | | |
1144 | 1144 | | |
1145 | | - | |
| 1145 | + | |
1146 | 1146 | | |
1147 | 1147 | | |
1148 | 1148 | | |
| |||
1192 | 1192 | | |
1193 | 1193 | | |
1194 | 1194 | | |
| 1195 | + | |
1195 | 1196 | | |
1196 | 1197 | | |
1197 | 1198 | | |
| |||
1214 | 1215 | | |
1215 | 1216 | | |
1216 | 1217 | | |
1217 | | - | |
| 1218 | + | |
1218 | 1219 | | |
1219 | 1220 | | |
1220 | 1221 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1091 | 1091 | | |
1092 | 1092 | | |
1093 | 1093 | | |
1094 | | - | |
| 1094 | + | |
1095 | 1095 | | |
1096 | 1096 | | |
1097 | 1097 | | |
| |||
1139 | 1139 | | |
1140 | 1140 | | |
1141 | 1141 | | |
| 1142 | + | |
1142 | 1143 | | |
1143 | 1144 | | |
1144 | 1145 | | |
| |||
0 commit comments