Skip to content

fix(sse): preserve broadcast order and de-race message history#483

Open
walcz-de wants to merge 1 commit into
mudler:mainfrom
walcz-de:fix/sse-broadcast-ordering-upstream
Open

fix(sse): preserve broadcast order and de-race message history#483
walcz-de wants to merge 1 commit into
mudler:mainfrom
walcz-de:fix/sse-broadcast-ordering-upstream

Conversation

@walcz-de

@walcz-de walcz-de commented Jul 3, 2026

Copy link
Copy Markdown

What

The SSE broadcast manager runs a pool of N competing delivery workers (NewManager(5) in core/state/pool.go) on a single broadcast channel. Each message is handed to exactly one worker, so with N>1 the delivery order to connected clients depends on goroutine scheduling.

Why it matters

Token-level SSE streams (stream_event chat deltas emitted by the pool's WithStreamCallback) are only meaningful in send order. Under real streaming rates the worker pool visibly scrambles them — in the React UI this surfaces as interleaved/corrupted agent chat output (tokens woven out of order, seemingly "multiple answers braided together").

There are two additional defects in the same file:

  • messageHistory.Add() was called once per delivered client inside the per-client loop: with 2 clients each message was recorded twice (duplicated on replay after an EventSource reconnect), and with 0 clients nothing was recorded at all. It also raced unsynchronized across the 5 workers (slice append without a mutex).
  • history.Send() wrote to the client channel blocking, so a client with a full buffer could stall the connection handler.

Fix

  • Single delivery goroutine consuming a buffered channel: FIFO order is preserved end-to-end, Send() stays non-blocking in practice, and slow clients still drop messages instead of stalling the broadcast (unchanged semantics).
  • History records exactly once per message, under a mutex, independent of connected clients.
  • History replay snapshots under the lock and delivers non-blocking.

Tests

go test ./core/sse/... -race — adds order-preservation, history-dedup and history-without-clients tests. The order test fails against the previous worker-pool implementation.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KYp5FnAfsdGb6yjLmv8Mge

The broadcast manager ran a pool of N competing delivery workers on a
single channel. Each message is handed to exactly one worker, so with
N>1 the delivery order to clients depends on goroutine scheduling:
token-level SSE streams (agent chat deltas) arrive scrambled, which
surfaces as interleaved/corrupted chat output in the React UI.

Replace the pool with a single delivery goroutine consuming a buffered
channel: FIFO order is preserved end-to-end, Send() stays non-blocking
in practice, and slow clients still drop messages instead of stalling
the broadcast (unchanged semantics).

Also fix the message history:
- Add() was called once per *delivered client* (duplicates with >1
  client, nothing recorded with 0 clients) and raced unsynchronized
  across workers. It now records exactly once per message, under a
  mutex, delivery-independent.
- Send() now snapshots under the lock and replays non-blocking, so a
  full client buffer cannot deadlock replay against the broadcaster.

Adds order-preservation and history-dedup tests (pass with -race;
the order test fails against the previous worker-pool implementation).

Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
walcz-de added a commit to walcz-de/LocalAI that referenced this pull request Jul 3, 2026
Agent chat streams arrived visibly corrupted in the React UI (tokens of
one turn interleaved/scrambled, internal generations merged into one
bubble). Two fixes:

- Bump the walcz-de/LocalAGI override to ad5ef2e (fix/sse-broadcast-
  ordering, upstream PR mudler/LocalAGI#483): the SSE broadcast manager
  ran 5 competing delivery workers on one channel, so message order to
  clients was nondeterministic; history was recorded once per delivered
  client (duplicates on EventSource reconnect) and raced unsynchronized.
  Now a single FIFO delivery goroutine + mutexed, per-message history.

- AgentChat.jsx: reset accumulated stream content on 'done' stream
  events. One agent turn runs several internal LLM generations (tool
  selection, reasoning, final answer) over the same SSE channel; without
  the reset the internal generations' text merged into the final
  answer's live bubble.

Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant