fix(test): stop the backend-trace specs racing the lossy trace channel - #11146
Merged
Conversation
RecordBackendTrace does a non-blocking send onto a 100-slot channel and drops when it is full, so tracing never stalls inference. The payload bounding specs pushed all 200 traces in one tight loop, which overruns that channel on a loaded machine: entries are dropped for good and the Eventually waiting for 200 can never be satisfied, no matter the timeout. CI hit this on master at 0a8a7fb, settling at 158/200. Feed the traces in chunks of 50, draining after each, so the channel is never overrun and the count stays exact. Reproduced with 60 busy loops on a 20-core box at GOMAXPROCS=2: 0/12 runs passed before, 12/12 after. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
testsworkflow failed on master at0a8a7fbb(run 30244703942):Found by the pre-release CI sweep for v4.8.0. The spec was added in #11056, so it is new in this release cycle.
Cause
This is not a slow test, and raising the timeout would not have fixed it.
RecordBackendTracedoes a non-blocking send ontobackendLogChan, which is buffered at 100, and drops on thedefault:branch when full:That is correct and deliberate: tracing must never stall inference. But the spec pushed 200 traces of ~16 KB in one tight loop. When the consumer goroutine does not keep up, entries are dropped permanently, and
Eventually(...).Should(Equal(200))can never be satisfied no matter how long it waits. On an idle machine the consumer keeps up and the spec passes, which is why this only shows up under CI load.Fix
Feed the traces in chunks of 50, draining after each, so the channel is never overrun and the count stays exact. Production code is unchanged: the lossy fast path is the right behavior, the test was wrong to depend on it.
Verification
Reproduced and fixed under identical synthetic load (60 busy loops on a 20-core box,
GOMAXPROCS=2), 12 runs each:Full package also green:
go test ./core/http/endpoints/localai/→ok (34.9s).