fix(distributed): reject wrong-model requests at the backend (#10970)

fix(distributed): reject wrong-model requests at the backend (#10952)

In distributed mode the controller caches a NodeModel row naming a backend's
host:port. A worker can recycle a stopped backend's gRPC port for a different
model's backend, and probeHealth verifies liveness rather than identity, so the
probe succeeds against whatever now occupies the port and the request is
dispatched to the wrong backend. The caller gets a silent wrong-model answer.

Nothing in the request could catch this: PredictOptions had no model field, so
model identity crossed the wire only in ModelOptions.Model at LoadModel time,
and the cached-hit path issues no LoadModel. Every backend's "model not loaded"
guard checks a nil handle, which a process holding a different model passes, so
the stale row was never dropped either.

Add PredictOptions.ModelIdentity and enforce it at the point of use:

  - The controller populates it in gRPCPredictOpts from ModelConfig.Model, the
    same expression ModelOptions feeds to model.WithModel and therefore the
    same value the backend received as ModelOptions.Model. Both are read from
    one config value in one function, so they are equal by construction and the
    comparison cannot false-reject.
  - Backends compare it against what they loaded and return NOT_FOUND with a
    fixed sentinel. Enforced in pkg/grpc/server.go (27 Go backends), an
    interceptor in backend/python/common (all 36 Python backends, no
    per-backend change), and the llama-cpp / ik-llama-cpp / ds4 C++ servers.
    That is every backend with real exposure: kokoros answers all four RPCs
    with unimplemented and privacy-filter implements none of them.
  - The router's reconcile drops the stale replica row on a mismatch, so the
    next request reloads somewhere correct.

Empty means "skip the check" on both sides: a controller that predates the
field sends nothing, a backend loaded by such a controller has nothing to
compare, and the C++ server synthesizes PredictOptions internally for ASR. That
keeps upgrades working in both directions.

Scoped to the four PredictOptions RPCs. TTSRequest.model and
SoundGenerationRequest.model are deliberately NOT validated: FileStagingClient
already rewrites them to worker-local absolute paths, so in distributed mode
they already differ from the load-time value and comparing them would reject
valid requests.

IsModelMismatch requires both the NOT_FOUND code and the sentinel, unlike the
neighbouring helpers which accept either. insightface's Embedding returns
NOT_FOUND "no face detected" on a PredictOptions RPC, and a code-only check
would drop a healthy replica row on every faceless image.


Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

Author: localai-bot

backend/backend.proto

@@ -315,6 +315,32 @@ message PredictOptions {
   int32 TopLogprobs = 51;  // Number of top logprobs to return per token (maps to OpenAI top_logprobs parameter)
   map<string, string> Metadata = 52;  // Generic per-request metadata (e.g., enable_thinking)
   float MinP = 53;  // Minimum probability sampling threshold (0.0 = disabled)
+
+  // ModelIdentity names the model this request is for, so a backend can reject
+  // a request that reached it by mistake instead of answering from whatever
+  // model it happens to hold. In distributed mode a worker can recycle a
+  // stopped backend's gRPC port for a different model's backend, and a
+  // liveness-only health probe cannot tell that apart from a valid cached
+  // route (#10952).
+  //
+  // The value is the controller's ModelConfig.Model, the SAME expression that
+  // produces ModelOptions.Model at LoadModel time, so the two are equal by
+  // construction rather than by convention.
+  //
+  // Empty means "no identity supplied": backends MUST skip the check. That
+  // keeps an old controller talking to a new backend working, and covers
+  // callers that legitimately synthesize a PredictOptions internally.
+  //
+  // Do NOT reuse TTSRequest.model or SoundGenerationRequest.model for this
+  // purpose. FileStagingClient already rewrites those to worker-local absolute
+  // paths (core/services/nodes/file_staging_client.go), so in distributed mode
+  // they already differ from the load-time value and comparing them would
+  // reject valid requests. Extending identity to those RPCs needs a separate
+  // field carrying the untranslated value.
+  string ModelIdentity = 54;
+
+  // 24 was never assigned; reserve it so it is not silently reused.
+  reserved 24;
 }
 
 // ToolCallDelta represents an incremental tool call update from the C++ parser.

backend/cpp/ds4/grpc-server.cpp

@@ -51,6 +51,11 @@ namespace {
 
 // Global state - ds4 is single-engine-per-process by design.
 std::mutex g_engine_mu;
+// The ModelOptions.Model this process loaded, compared against
+// PredictOptions.ModelIdentity so a request that arrived through a stale
+// distributed route is rejected rather than answered from the wrong model
+// (#10952). Guarded by g_engine_mu like the rest of the engine state.
+std::string g_loaded_model_identity;
 ds4_engine *g_engine = nullptr;
 ds4_session *g_session = nullptr;
 int g_ctx_size = 32768;
@@ -562,6 +567,24 @@ static void build_prompt(ds4_engine *engine, const backend::PredictOptions *requ
     ds4_chat_append_assistant_prefix(engine, out, think);
 }
 
+// check_model_identity mirrors pkg/grpc/server.go and
+// backend/python/common/model_identity.py. Either side empty means "skip": the
+// request side is empty for a controller that predates the field, the loaded
+// side when such a controller performed the load. A false rejection is worse
+// than the miss it prevents. Callers must already hold g_engine_mu.
+static GStatus check_model_identity(const backend::PredictOptions *request) {
+    if (request == nullptr || request->modelidentity().empty()) return GStatus::OK;
+    if (g_loaded_model_identity.empty() ||
+        g_loaded_model_identity == request->modelidentity()) {
+        return GStatus::OK;
+    }
+    // NOT_FOUND plus this exact sentinel is the cross-language contract the
+    // router matches on (grpcerrors.ModelMismatchSentinel).
+    return GStatus(StatusCode::NOT_FOUND,
+                   "ds4: model identity mismatch: loaded \"" + g_loaded_model_identity +
+                   "\", requested \"" + request->modelidentity() + "\"");
+}
+
 class DS4Backend final : public backend::Backend::Service {
 public:
     GStatus Health(ServerContext *, const backend::HealthMessage *,
@@ -716,6 +739,7 @@ class DS4Backend final : public backend::Backend::Service {
         }
 
         result->set_success(true);
+        g_loaded_model_identity = request->model();
         result->set_message("loaded " + model_path);
         return GStatus::OK;
     }
@@ -724,6 +748,7 @@ class DS4Backend final : public backend::Backend::Service {
                           backend::TokenizationResponse *response) override {
         std::lock_guard<std::mutex> lock(g_engine_mu);
         if (!g_engine) return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
+        if (GStatus id = check_model_identity(request); !id.ok()) return id;
         ds4_tokens out = {};
         ds4_tokenize_text(g_engine, request->prompt().c_str(), &out);
         for (int i = 0; i < out.len; ++i) response->add_tokens(out.v[i]);
@@ -738,6 +763,7 @@ class DS4Backend final : public backend::Backend::Service {
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
+        if (GStatus id = check_model_identity(request); !id.ok()) return id;
         if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
             return GStatus(StatusCode::UNAVAILABLE, route_err);
         }
@@ -837,6 +863,7 @@ class DS4Backend final : public backend::Backend::Service {
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
+        if (GStatus id = check_model_identity(request); !id.ok()) return id;
         if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
             return GStatus(StatusCode::UNAVAILABLE, route_err);
         }

backend/cpp/ik-llama-cpp/grpc-server.cpp

@@ -2412,7 +2412,33 @@ static void params_parse(const backend::ModelOptions* request,
 
 // GRPC Server start
 class BackendServiceImpl final : public backend::Backend::Service {
+private:
+  // The ModelOptions.Model this process was loaded with. Compared against
+  // PredictOptions.ModelIdentity so a request that reached us through a stale
+  // distributed route is rejected instead of answered from the wrong model
+  // (#10952).
+  std::string loaded_model_identity;
+
 public:
+  // checkModelIdentity mirrors pkg/grpc/server.go and
+  // backend/python/common/model_identity.py. Either side being empty means
+  // "skip": the request side is empty for a controller that predates the field,
+  // and the loaded side is empty when such a controller performed the load. A
+  // false rejection is worse than the miss it prevents.
+  grpc::Status checkModelIdentity(const backend::PredictOptions* request) {
+    if (request == nullptr || request->modelidentity().empty()) {
+      return grpc::Status::OK;
+    }
+    if (loaded_model_identity.empty() || loaded_model_identity == request->modelidentity()) {
+      return grpc::Status::OK;
+    }
+    // NOT_FOUND plus this exact sentinel is the cross-language contract the
+    // router matches on (grpcerrors.ModelMismatchSentinel).
+    return grpc::Status(grpc::StatusCode::NOT_FOUND,
+                        "ik-llama-cpp: model identity mismatch: loaded \"" + loaded_model_identity +
+                        "\", requested \"" + request->modelidentity() + "\"");
+  }
+
   grpc::Status Health(ServerContext* context, const backend::HealthMessage* request, backend::Reply* reply) {
     // Implement Health RPC
     reply->set_message("OK");
@@ -2438,9 +2464,12 @@ class BackendServiceImpl final : public backend::Backend::Service {
     result->set_message("Loading succeeded");
     result->set_success(true);
     loaded_model = true;
+    loaded_model_identity = request->model();
     return Status::OK;
   }
   grpc::Status PredictStream(grpc::ServerContext* context, const backend::PredictOptions* request, grpc::ServerWriter<backend::Reply>* writer) override {
+        auto identity = checkModelIdentity(request);
+        if (!identity.ok()) return identity;
         json data = parse_options(true, request, llama);
         const int task_id = llama.queue_tasks.get_new_id();
         llama.queue_results.add_waiting_task_id(task_id);
@@ -2495,6 +2524,8 @@ class BackendServiceImpl final : public backend::Backend::Service {
 
 
     grpc::Status Predict(ServerContext* context, const backend::PredictOptions* request, backend::Reply* reply) {
+        auto identity = checkModelIdentity(request);
+        if (!identity.ok()) return identity;
         json data = parse_options(false, request, llama);
         const int task_id = llama.queue_tasks.get_new_id();
         llama.queue_results.add_waiting_task_id(task_id);
@@ -2532,6 +2563,8 @@ class BackendServiceImpl final : public backend::Backend::Service {
 
     /// https://github.com/ggerganov/llama.cpp/blob/aa2341298924ac89778252015efcb792f2df1e20/examples/server/server.cpp#L2969
     grpc::Status Embedding(ServerContext* context, const backend::PredictOptions* request, backend::EmbeddingResult* embeddingResult) {
+        auto identity = checkModelIdentity(request);
+        if (!identity.ok()) return identity;
         json data = parse_options(false, request, llama);
         const int task_id = llama.queue_tasks.get_new_id();
         llama.queue_results.add_waiting_task_id(task_id);
@@ -2556,6 +2589,8 @@ class BackendServiceImpl final : public backend::Backend::Service {
     }
 
     grpc::Status TokenizeString(ServerContext* context, const backend::PredictOptions* request, backend::TokenizationResponse* response){
+        auto identity = checkModelIdentity(request);
+        if (!identity.ok()) return identity;
          json data = parse_options(false, request, llama);
 
          std::vector<llama_token> tokens = llama.tokenize(data["prompt"],false);

backend/cpp/llama-cpp/grpc-server.cpp

@@ -1401,10 +1401,36 @@ class BackendServiceImpl final : public backend::Backend::Service {
 private:
     server_context& ctx_server;
     common_params params_base; // Store copy of params_base, set after model load
+    // The ModelOptions.Model this process was loaded with. Compared against
+    // PredictOptions.ModelIdentity so a request that reached us through a stale
+    // distributed route is rejected instead of answered from the wrong model
+    // (#10952). Written under LoadModel, read by the inference RPCs.
+    std::string loaded_model_identity;
 
 public:
     BackendServiceImpl(server_context& ctx) : ctx_server(ctx) {}
 
+    // checkModelIdentity mirrors pkg/grpc/server.go and
+    // backend/python/common/model_identity.py. Either side being empty means
+    // "skip": the request side is empty for a controller that predates the
+    // field and for the synthetic PredictOptions this server builds internally
+    // for ASR, and the loaded side is empty when such a controller performed
+    // the load. A false rejection is worse than the miss it prevents.
+    grpc::Status checkModelIdentity(const backend::PredictOptions* request) {
+        if (request == nullptr || request->modelidentity().empty()) {
+            return grpc::Status::OK;
+        }
+        if (loaded_model_identity.empty() || loaded_model_identity == request->modelidentity()) {
+            return grpc::Status::OK;
+        }
+        // NOT_FOUND plus this exact sentinel is the cross-language contract the
+        // router matches on (grpcerrors.ModelMismatchSentinel). The code alone
+        // is not enough: NOT_FOUND is returned for unrelated reasons elsewhere.
+        return grpc::Status(grpc::StatusCode::NOT_FOUND,
+                            "llama-cpp: model identity mismatch: loaded \"" + loaded_model_identity +
+                            "\", requested \"" + request->modelidentity() + "\"");
+    }
+
     grpc::Status Health(ServerContext* context, const backend::HealthMessage* /*request*/, backend::Reply* reply) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
@@ -1535,6 +1561,7 @@ class BackendServiceImpl final : public backend::Backend::Service {
         result->set_message("Loading succeeded");
         result->set_success(true);
         loaded_model = true;
+        loaded_model_identity = request->model();
         // Store copy of params_base for use in parse_options and other methods
         params_base = params;
 
@@ -1616,6 +1643,8 @@ class BackendServiceImpl final : public backend::Backend::Service {
     grpc::Status PredictStream(grpc::ServerContext* context, const backend::PredictOptions* request, grpc::ServerWriter<backend::Reply>* writer) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
+        auto identity = checkModelIdentity(request);
+        if (!identity.ok()) return identity;
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
@@ -2183,6 +2212,8 @@ class BackendServiceImpl final : public backend::Backend::Service {
     grpc::Status Predict(ServerContext* context, const backend::PredictOptions* request, backend::Reply* reply) override {
          auto auth = checkAuth(context);
          if (!auth.ok()) return auth;
+         auto identity = checkModelIdentity(request);
+         if (!identity.ok()) return identity;
          if (params_base.model.path.empty()) {
              return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
          }
@@ -2715,6 +2746,8 @@ class BackendServiceImpl final : public backend::Backend::Service {
     grpc::Status Embedding(ServerContext* context, const backend::PredictOptions* request, backend::EmbeddingResult* embeddingResult) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
+        auto identity = checkModelIdentity(request);
+        if (!identity.ok()) return identity;
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
@@ -3108,6 +3141,8 @@ class BackendServiceImpl final : public backend::Backend::Service {
     grpc::Status TokenizeString(ServerContext* context, const backend::PredictOptions* request, backend::TokenizationResponse* response) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
+        auto identity = checkModelIdentity(request);
+        if (!identity.ok()) return identity;
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }

backend/python/common/grpc_auth.py

@@ -11,6 +11,7 @@
 
 import grpc
 
+from model_identity import AsyncModelIdentityInterceptor, ModelIdentityInterceptor
 from parent_watch import start_parent_death_watcher
 
 
@@ -64,13 +65,15 @@ async def intercept_service(self, continuation, handler_call_details):
 
 
 def get_auth_interceptors(*, aio: bool = False):
-    """Return a list of gRPC interceptors for bearer token auth.
+    """Return the gRPC server interceptors every LocalAI Python backend installs.
+
+    Always includes model-identity enforcement (model_identity.py), which is
+    unrelated to authentication. Bearer token auth is added on top only when
+    LOCALAI_GRPC_AUTH_TOKEN is set.
 
     Args:
         aio: If True, return async-compatible interceptors for grpc.aio.server().
              If False (default), return sync interceptors for grpc.server().
-
-    Returns an empty list when LOCALAI_GRPC_AUTH_TOKEN is not set.
     """
     # Arm the best-effort parent-death backstop here: this is the single helper
     # every LocalAI Python backend invokes exactly once while building its gRPC
@@ -79,9 +82,17 @@ def get_auth_interceptors(*, aio: bool = False):
     # unsupported platforms — see parent_watch.py.
     start_parent_death_watcher()
 
+    # Model-identity enforcement is independent of authentication and must be
+    # installed BEFORE the token check returns. gRPC auth is off by default, so
+    # an identity interceptor added below the early return would never be
+    # installed on any Python backend, and nothing would report it.
+    interceptors = [AsyncModelIdentityInterceptor()] if aio else [ModelIdentityInterceptor()]
+
     token = os.environ.get("LOCALAI_GRPC_AUTH_TOKEN", "")
     if not token:
-        return []
+        return interceptors
     if aio:
-        return [AsyncTokenAuthInterceptor(token)]
-    return [TokenAuthInterceptor(token)]
+        interceptors.append(AsyncTokenAuthInterceptor(token))
+    else:
+        interceptors.append(TokenAuthInterceptor(token))
+    return interceptors