feat(realtime): prewarm the classifier scoring prompt on registration

Swapping a session's classifier option list (a voice-switched command
mode, for instance) made the next turns pay a full re-prefill of the new
option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and
worse: on hybrid-memory models like LFM2.5, whose state cannot be
partially rewound (llama.cpp can only restore checkpoints), *every*
probe change re-prefilled from scratch whenever the last checkpoint
missed the probe boundary, so even same-list turns intermittently cost
full prefills.

Registering an option list (pipeline seed or session.update) now fires a
best-effort background prewarm: two throwaway scores with distinct
probes. The first prefills the new option-list prompt; the second,
diverging exactly where per-turn probe text starts, plants the backend's
rewind point (KV checkpoint) at the stable-prefix boundary that every
real turn reuses. The prewarm hides behind the canned mode-switch reply
— by the time it finishes speaking, the cache is warm. Idempotent per
option set, detached from the registering request's lifetime.

Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after
a mode switch 2374ms -> 340ms; intermittent same-list full prefills
(1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently,
options: [parallel:2] on the scoring model additionally keeps one slot
per list via prefix-similarity routing (+26MB RSS, unified KV).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

Author: richiejp

core/http/endpoints/openai/realtime.go

@@ -280,6 +280,11 @@ type Model interface {
 	// pipeline's scoring model (classifier.model, defaulting to the LLM) —
 	// no autoregressive decode happens.
 	ClassifyTurn(ctx context.Context, messages schema.Messages, options []types.ClassifierOption, normalization string) ([]router.LabelScore, error)
+	// PrewarmClassifier primes the scoring backend's prompt cache for a
+	// newly registered option list (fired async on registration) so the
+	// first turns after a session.update don't pay the option-list
+	// prefill. Best-effort and idempotent per option set.
+	PrewarmClassifier(ctx context.Context, options []types.ClassifierOption, normalization string)
 	// FillToolArguments completes the chosen option's argument slots with a
 	// short grammar-constrained completion that continues the exact scoring
 	// prompt (so the backend's prompt cache stays warm) and returns the
@@ -616,6 +621,10 @@ func runRealtimeSession(application *application.Application, t Transport, model
 		return
 	}
 	session.ModelInterface = m
+	// A pipeline-seeded option list gets its scoring prompt prewarmed
+	// alongside the model warm-up below, so the session's first turn
+	// doesn't pay the option-list prefill.
+	prewarmClassifier(session)
 
 	// The voice gate is built before the warm-up below so its
 	// speaker-recognition model can warm alongside the pipeline stages.
@@ -1269,6 +1278,7 @@ func updateSession(session *Session, update *types.SessionUnion, cl *config.Mode
 			return err
 		}
 		session.Classifier = rt.LocalAIClassifier
+		prewarmClassifier(session)
 	}
 
 	if rt.MaxOutputTokens != 0 {

core/http/endpoints/openai/realtime_classifier.go

@@ -83,6 +83,27 @@ func classifierConfigFromPipeline(p *config.PipelineClassifier) (*types.Classifi
 	return cc, nil
 }
 
+// prewarmClassifier primes the scoring prompt cache for the session's
+// current classifier config in the background: registration returns
+// immediately, and by the time the canned mode-switch reply finishes
+// speaking, the new option list's prompt (and, on hybrid/recurrent
+// models, a rewind checkpoint at the per-turn probe boundary) is already
+// in the backend's cache. The context is deliberately detached from the
+// registering request — the warmed cache belongs to the backend, not the
+// request.
+func prewarmClassifier(session *Session) {
+	cc := session.Classifier
+	if session.ModelInterface == nil || !cc.Active() {
+		return
+	}
+	options, normalization := cc.Options, cc.Normalization
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+		defer cancel()
+		session.ModelInterface.PrewarmClassifier(ctx, options, normalization)
+	}()
+}
+
 // resolveClassifier merges the session classifier config with a
 // response-level override: a non-nil override replaces the whole block
 // (same replace-not-merge semantics as tools), so {"enabled": false} runs

core/http/endpoints/openai/realtime_classifier_test.go

@@ -65,6 +65,32 @@ func replyTexts(t *fakeTransport) []string {
 	return out
 }
 
+var _ = Describe("prewarmClassifier", func() {
+	It("prewarms an active option list in the background", func() {
+		m := &fakeModel{}
+		session := classifierTestSession(m)
+		session.Classifier = classifierTestConfig(0.35, nil)
+
+		prewarmClassifier(session)
+
+		Eventually(func() int { n, _ := m.prewarmed(); return n }).Should(Equal(1))
+		_, opts := m.prewarmed()
+		Expect(opts).To(HaveLen(len(session.Classifier.Options)))
+	})
+
+	It("does nothing without an active classifier", func() {
+		m := &fakeModel{}
+		session := classifierTestSession(m)
+		prewarmClassifier(session)
+
+		off := false
+		session.Classifier = &types.ClassifierConfig{Enabled: &off, Options: classifierTestConfig(0.35, nil).Options}
+		prewarmClassifier(session)
+
+		Consistently(func() int { n, _ := m.prewarmed(); return n }, "150ms").Should(BeZero())
+	})
+})
+
 var _ = Describe("classifierConfigFromPipeline", func() {
 	It("returns nil for an absent block", func() {
 		cc, err := classifierConfigFromPipeline(nil)

core/http/endpoints/openai/realtime_doubles_test.go

@@ -3,6 +3,7 @@ package openai
 import (
 	"context"
 	"strings"
+	"sync"
 
 	"github.com/mudler/LocalAI/core/backend"
 	"github.com/mudler/LocalAI/core/config"
@@ -119,6 +120,12 @@ type fakeModel struct {
 	fillCalls      int
 	lastFillChosen *types.ClassifierOption
 
+	// PrewarmClassifier runs on a background goroutine, so its recording
+	// is mutex-guarded; specs poll prewarmCalls with Eventually.
+	prewarmMu          sync.Mutex
+	prewarmCalls       int
+	lastPrewarmOptions []types.ClassifierOption
+
 	// VAD scripting: vadFn, when set, decides per call (specs vary the
 	// answer across ticks or record the request); otherwise
 	// vadSegments/vadErr answer every call.
@@ -129,6 +136,19 @@ type fakeModel struct {
 	lastMessages schema.Messages
 }
 
+func (m *fakeModel) PrewarmClassifier(_ context.Context, options []types.ClassifierOption, _ string) {
+	m.prewarmMu.Lock()
+	defer m.prewarmMu.Unlock()
+	m.prewarmCalls++
+	m.lastPrewarmOptions = options
+}
+
+func (m *fakeModel) prewarmed() (int, []types.ClassifierOption) {
+	m.prewarmMu.Lock()
+	defer m.prewarmMu.Unlock()
+	return m.prewarmCalls, m.lastPrewarmOptions
+}
+
 func (m *fakeModel) FillToolArguments(_ context.Context, msgs schema.Messages, options []types.ClassifierOption, _ string, chosen *types.ClassifierOption) (string, map[string]string, error) {
 	m.fillCalls++
 	m.lastFillChosen = chosen

core/http/endpoints/openai/realtime_model.go

@@ -9,6 +9,7 @@ import (
 	"fmt"
 	"strings"
 	"sync"
+	"time"
 
 	"github.com/mudler/LocalAI/core/application"
 	"github.com/mudler/LocalAI/core/backend"
@@ -57,6 +58,9 @@ type wrappedModel struct {
 	classifier     *router.ScoreClassifier
 	classifierKey  string
 	classifierWarn sync.Once
+	// prewarmKey remembers the option set whose scoring prompt was last
+	// prewarmed, so re-pushing an unchanged config doesn't re-score.
+	prewarmKey string
 
 	// Routing — populated by newModel when the application wires routing
 	// deps in. nil-safe: with classifierRegistry == nil the per-turn
@@ -114,6 +118,9 @@ func (m *transcriptOnlyModel) FillToolArguments(ctx context.Context, messages sc
 	return "", nil, fmt.Errorf("classifier mode not supported in transcript-only mode")
 }
 
+func (m *transcriptOnlyModel) PrewarmClassifier(ctx context.Context, options []types.ClassifierOption, normalization string) {
+}
+
 func (m *transcriptOnlyModel) TTS(ctx context.Context, text, voice, language string) (string, *proto.Result, error) {
 	return "", nil, fmt.Errorf("TTS not supported in transcript-only mode")
 }
@@ -488,6 +495,44 @@ func (m *wrappedModel) classifierFor(options []types.ClassifierOption, normaliza
 	return m.classifier, nil
 }
 
+// PrewarmClassifier primes the scoring backend's prompt cache for a newly
+// registered option list so the first real turns don't pay the prefill.
+// Two throwaway scores with distinct probes run back to back: the first
+// prefills the new option-list prompt, and the second — diverging exactly
+// where the per-turn probe text starts — leaves the backend a rewind point
+// (a KV checkpoint on hybrid/recurrent models, which cannot rewind
+// arbitrarily) at the stable-prefix boundary every subsequent turn reuses.
+// Best-effort: errors are logged, never surfaced.
+func (m *wrappedModel) PrewarmClassifier(ctx context.Context, options []types.ClassifierOption, normalization string) {
+	classifier, err := m.classifierFor(options, normalization)
+	if err != nil {
+		xlog.Debug("realtime classifier: prewarm skipped", "error", err)
+		return
+	}
+	m.classifierMu.Lock()
+	key := m.classifierKey
+	done := m.prewarmKey == key
+	m.classifierMu.Unlock()
+	if done {
+		return
+	}
+	start := time.Now()
+	for _, probe := range []string{"warmup", "standing by"} {
+		if ctx.Err() != nil {
+			return
+		}
+		if _, err := classifier.Classify(ctx, router.Probe{Prompt: probe, Messages: []string{probe}}); err != nil {
+			xlog.Warn("realtime classifier: prewarm scoring failed", "error", err)
+			return
+		}
+	}
+	m.classifierMu.Lock()
+	m.prewarmKey = key
+	m.classifierMu.Unlock()
+	xlog.Debug("realtime classifier: prewarmed scoring prompt cache",
+		"options", len(options), "latency_ms", time.Since(start).Milliseconds())
+}
+
 func (m *wrappedModel) ClassifyTurn(ctx context.Context, messages schema.Messages, options []types.ClassifierOption, normalization string) ([]router.LabelScore, error) {
 	classifier, err := m.classifierFor(options, normalization)
 	if err != nil {

docs/content/features/openai-realtime.md

@@ -246,6 +246,8 @@ The option's spoken `reply` can reference the same placeholders — `reply: "Goi
 
 Slot declarations (and hints) are appended to the option's description in the shared system prompt, so they also inform scoring and cost no extra per-turn tokens. The `localai.classifier.result` event carries the final `arguments` and a `fill_latency_ms`. On an inference failure the slots' defaults apply (and template the reply); if any slot lacks a default the response fails (or falls through to generation with `fallback.mode: generate`). Slot filling requires `completion` in the scoring model's `known_usecases` alongside `score`.
 
+Registering an option list (pipeline seed or `session.update`) prewarms the scoring prompt in the background: two throwaway scores prefill the option-list prompt and leave the backend a reuse point at the per-turn probe boundary — on hybrid/recurrent models (which cannot rewind their state arbitrarily, only restore checkpoints) this is what keeps every turn fast, not just repeats of the same probe. A client that swaps option lists at runtime (e.g. voice-switched command modes) pays nothing on the first turn after a swap: the prewarm hides behind the acknowledgement reply. Swapping between lists frequently? `options: [parallel:2]` on the scoring model gives llama.cpp a slot per list (prefix-similarity routing picks the matching one) at a small KV memory cost.
+
 The concrete scoring model must declare `score` in `known_usecases`. A single llama.cpp model can serve ordinary inference and classification concurrently by declaring multiple use cases, for example `known_usecases: [chat, completion, score]`; LocalAI reserves the scoring slots only when `score` is present. Scoring also requires the unified KV cache, which is enabled by default, so a score-enabled model cannot set `kv_unified:false`.
 
 ## Transports