Skip to content

fix(test): stop the backend-trace specs racing the lossy trace channel - #11146

Merged
mudler merged 1 commit into
masterfrom
fix/traces-test-eventually-timeout
Jul 27, 2026
Merged

fix(test): stop the backend-trace specs racing the lossy trace channel#11146
mudler merged 1 commit into
masterfrom
fix/traces-test-eventually-timeout

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Problem

The tests workflow failed on master at 0a8a7fbb (run 30244703942):

[FAILED] Timed out after 1.000s.
Expected <int>: 158 to equal <int>: 200
In [BeforeEach] at core/http/endpoints/localai/traces_test.go:120

Found by the pre-release CI sweep for v4.8.0. The spec was added in #11056, so it is new in this release cycle.

Cause

This is not a slow test, and raising the timeout would not have fixed it.

RecordBackendTrace does a non-blocking send onto backendLogChan, which is buffered at 100, and drops on the default: branch when full:

select {
case backendLogChan <- &t:
default:
    xlog.Warn("Backend trace channel full, dropping trace")
}

That is correct and deliberate: tracing must never stall inference. But the spec pushed 200 traces of ~16 KB in one tight loop. When the consumer goroutine does not keep up, entries are dropped permanently, and Eventually(...).Should(Equal(200)) can never be satisfied no matter how long it waits. On an idle machine the consumer keeps up and the spec passes, which is why this only shows up under CI load.

Fix

Feed the traces in chunks of 50, draining after each, so the channel is never overrun and the count stays exact. Production code is unchanged: the lossy fast path is the right behavior, the test was wrong to depend on it.

Verification

Reproduced and fixed under identical synthetic load (60 busy loops on a 20-core box, GOMAXPROCS=2), 12 runs each:

Result Observed counts
Before 0 / 12 passed stuck at 164, 101 (CI saw 158)
After 12 / 12 passed 200 every run

Full package also green: go test ./core/http/endpoints/localai/ok (34.9s).

RecordBackendTrace does a non-blocking send onto a 100-slot channel and
drops when it is full, so tracing never stalls inference. The payload
bounding specs pushed all 200 traces in one tight loop, which overruns
that channel on a loaded machine: entries are dropped for good and the
Eventually waiting for 200 can never be satisfied, no matter the timeout.
CI hit this on master at 0a8a7fb, settling at 158/200.

Feed the traces in chunks of 50, draining after each, so the channel is
never overrun and the count stays exact. Reproduced with 60 busy loops on
a 20-core box at GOMAXPROCS=2: 0/12 runs passed before, 12/12 after.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
@mudler
mudler merged commit 05ff401 into master Jul 27, 2026
69 checks passed
@mudler
mudler deleted the fix/traces-test-eventually-timeout branch July 27, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants