Fix idle range completion scan - #832
Conversation
lejeunerenard
left a comment
There was a problem hiding this comment.
This approach is more thorough when completing range requests, but it blocks seek requests when processing large amounts of ranges. Accounting for this seems like it would add more state at the replicator level for tracking where the cursor left off etc.
If we keep _updateNonPrimary's workload small but:
- make it pick random subset of ranges to avoid stacking unfulfillable ranges at the front.
- trigger non-primary updates more often, maybe when inflight goes idle
We can allow it to yield to seeks while being a minor change. I have the following competing PR with the above, which still doesn't pass atm. #841
|
One note, processing can get slow if there are many seeks at once or a very large backlog of ranges. Potentially optimization/perf improrvement can be made here but left it out to make the PR easier |
lejeunerenard
left a comment
There was a problem hiding this comment.
Did another pass. Still need to cover some of the tests but found a potential way that ranges now can be blocked instead of seeks.
lejeunerenard
left a comment
There was a problem hiding this comment.
Only one test i'm not sure what it's purpose is, but otherwise only notes about seeks vs ranges behavior for the PR in the future.
co-authored with AI