fix(backend/python): don't await sync servicer behaviors in AsyncModelIdentityInterceptor

[walcz-de]

What

AsyncModelIdentityInterceptor (added in #10952, installed on every Python backend's gRPC server via get_auth_interceptors) awaits the wrapped servicer behavior unconditionally. A backend's servicer methods may be plain sync functions; grpc.aio normally adapts both sync and async behaviors, but this interceptor calls the behavior itself and bypasses that adaptation:

result = await original(request, context)                 # LoadModel
return await original_unary(request, context)             # guarded RPCs
async for response in original_stream(request, context):  # streaming

For a sync method, original(request, context) returns a message object, not a coroutine:

TypeError: object Result can't be used in 'await' expression

Impact

Any grpc.aio Python backend with sync servicer methods breaks on the guarded RPCs, on every platform. Concretely the transformers backend defines def LoadModel and def Embedding (not async def): on current master, LoadModel loads the model and then dies on return; the guarded Embedding fails the same way.

Reproduced end-to-end against the transformers backend:

Loading weights: 100%|██████████| 290/290
Loaded processor for /models/...
LoadModel RPC: StatusCode.UNKNOWN
  "Unexpected <class 'TypeError'>: object Result can't be used in 'await' expression"

CI didn't catch it because AsyncModelIdentityInterceptor had no behavioral test — only the "is it installed" assertion in TestInterceptorInstalled.

Fix

Await the behavior's result only when it is actually awaitable (inspect.isawaitable), mirroring grpc.aio's own sync/async adaptation. The streaming guard iterates a sync generator with for and an async generator with async for. The sync ModelIdentityInterceptor was already correct and is unchanged.

Tests

Adds TestAsyncInterceptorBehavior to model_identity_test.py, exercising the async interceptor with both sync and async LoadModel / guarded-unary / streaming behaviors. The sync cases fail on current master with the TypeError above and pass with this fix.

$ python -m unittest model_identity_test
Ran 23 tests ... OK

[localai-org-maint-bot]

The regression is real and the new focused suite passes locally (23/23), but this fix still bypasses an important part of grpc.aio sync-handler adaptation: the migration thread pool. original(request, context) is invoked before inspect.isawaitable, so a synchronous LoadModel or unary handler runs directly on the asyncio event-loop thread. Likewise, for response in stream calls a synchronous generator’s next() on the event loop. A slow model load/inference/stream can therefore freeze all aio RPC handling even though the TypeError is gone.

Please dispatch synchronous behavior through an executor (asyncio.to_thread / loop.run_in_executor) while continuing to await native async behavior. For a sync streaming iterator, each next() also needs executor dispatch (using a small helper that returns a done sentinel, since StopIteration must not escape through a Future). A regression test that records the handler thread ID and verifies it differs from the event-loop thread would cover this.

The awaitable-result check is useful for callable wrappers, but it must happen without first running potentially blocking sync application code on the event loop.

[walcz-de]

Thanks — good catch, addressed in de4d018b.

Sync behavior now goes through run_in_executor so it never runs on the event-loop thread, while native async behavior is still awaited directly. A callable wrapper that returns an awaitable is run in the thread and its awaitable awaited back on the loop (so the awaitable-result check no longer runs blocking app code on the loop first). Sync streaming pulls each item via the executor with a _STREAM_DONE sentinel, so StopIteration can't escape through a Future.

Added three regression tests that record the handler thread id and assert it differs from the event-loop thread (sync LoadModel, guarded unary, sync stream). Full suite: 26/26 green.