fix(openresponses): make responses visible and cancellable across replicas

[localai-bot]

Fixes #10993.

Problem, reproduced on a live cluster

The Open Responses store is a sync.OnceValue over a process-local map (store.go:39,88-92). Behind a round-robin load balancer, any request landing on a replica other than the creator misses.

Measured on a 2-replica controller, same response id:

replica A (creator) -> HTTP 200
replica B (peer)    -> HTTP 404
cancel on B         -> HTTP 404

The cancel is the worst of it: it returned 404 without ever invoking CancelFunc, so generation ran to completion on A — consuming GPU — while the caller was told the response did not exist.

Approach: replicate, delegate, or refuse

StoredResponse was split by what can actually cross a process boundary.

Replicated — request, response resource, owner, timestamps, expiry, StreamEnabled/IsBackground, plus OwnerReplica. Carried in a syncstate.SyncedMap[string, *syncedResponse] (namespace responses.metadata) — the same component finetune, quantization and agent tasks use, per #10542. Get and FindItem fall back to it on a local miss and return a read-only view flagged Remote: true.

That fixes polling, previous_response_id chaining (the request is replicated, so the turn can be replayed) and item lookup.

No durable Postgres Store is wired: responses are ephemeral, TTL-bounded state that already does not survive a restart, so peers converge through deltas alone — which syncstate explicitly sanctions for ephemeral state. A late-joining replica does not learn about older responses, which is the same visibility a client had before and strictly better than the previous 404-from-every-peer.

Delegated — cancellation. CancelFunc is a function pointer; only the creating process has it. A cancel on a non-owner broadcasts responses.<id>.cancel, and every replica's wildcard subscriber applies it if it holds the response. Both the direct and delegated paths funnel through one cancelLocal, so terminal-state idempotence is identical.

Broadcast rather than addressing OwnerReplica directly, following galleryop's CancelOperation: a broadcast cannot misroute after a restart and needs no liveness knowledge. (Agent tasks were the other candidate pattern but do not record an owner at all — they delegate through dispatcher.Cancel.)

Refused — streaming resume. The resume buffer is a byte log plus a live channel; replicating it means shipping every token over NATS. New ErrResponseNotLocal → HTTP 409, naming the owning replica.

Deliberately not ErrOffsetLost: that means "the owner's buffer evicted your events" — a retention verdict, after which a client may reasonably stop. This means "ask the other replica" — a routing problem. Conflating them would mislead clients into giving up. GetEventsAfter/GetEventsChan return it too, so no caller can receive an empty slice that reads as a finished stream.

Dead owners

The delegating publish is fire-and-forget. A request/reply would block the HTTP handler for a full timeout precisely when the owner is gone. The delegating replica marks the replicated status cancelled without waiting — truthful either way, since a crashed owner's generation died with its process. Cleanup also reaps expired replicated entries whose owner never got to expire them.

Not fixed, and why

Cross-replica stream resume. Refused with a clear 409 rather than half-implemented. Making it work needs either the event buffer in a shared store — a real durability and throughput decision — or session affinity at the ingress. Both options are documented in docs/content/features/text-generation.md.

Compatibility

Standalone deployments never call EnableDistributed and are behaviourally unchanged.

Testing

Red first, functionally: with EnableDistributed stubbed to the pre-fix path, 7 of 42 specs failed on behaviour — response not found: resp_cross_replica, resp_dead_owner, resp_cancel_terminal, and errors.Is(err, ErrResponseNotLocal) returning false. No compile errors.

Green: all 42 pass. -race clean on openresponses, routes, messaging, syncstate. Specs cover cross-replica visibility, status propagation, delete propagation, cancel reaching the owner's CancelFunc, dead-owner non-blocking cancel, terminal-state idempotence, and the resume refusal.

go build ./core/... ./pkg/... OK; make lint 0 issues. Repo-wide grep including tests/e2e/ found nothing asserting single-store behaviour; the only 404 assertion (core/http/openresponses_test.go:927) is for a genuinely nonexistent id and remains correct.

---

🤖 Generated with Claude Code