fix(server,client,docs): default-model alias, empty-content guard, truthful /v1/models, native-client contract

Author: ctrl-gaurav

docs/migration.md

@@ -1,5 +1,92 @@
 # Migration Guide
 
+## Coming from the OpenAI SDK / LangChain
+
+effGen ships an OpenAI-compatible HTTP server, so most code that already talks to
+the OpenAI API works by changing only the `base_url`. See
+[`server/openai-compat.md`](server/openai-compat.md) for the full endpoint,
+alias, streaming, and error-status reference.
+
+### Point the official `openai` client at effGen
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="YOUR_EFFGEN_API_KEY")
+
+resp = client.chat.completions.create(
+    model="openai:gpt-5-nano",                 # route by provider:model
+    messages=[{"role": "user", "content": "Summarize the CAP theorem."}],
+)
+print(resp.choices[0].message.content)
+```
+
+- **Routing.** Send `provider:model` (`groq:llama-3.1-8b-instant`,
+  `gemini:gemini-3.1-flash-lite`) or `provider/model`; a bare local id
+  (`transformers:Qwen/Qwen2.5-1.5B-Instruct`) also loads. `effgen-default`
+  routes to the server's configured default model. OpenAI flagship names
+  (`gpt-4o-mini`, `gpt-3.5-turbo`) resolve to local models; the response's
+  non-standard `effgen` object reports `resolved_model` and `alias_applied`.
+- **Streaming** works unchanged, including `stream_options={"include_usage":
+  True}` for a final usage chunk. `response_format={"type": "json_object"}`,
+  legacy `/v1/completions`, and `/v1/embeddings` are supported.
+- **Errors** map to the OpenAI status/type contract: unknown provider → 400,
+  bad key → 401, unknown model → 404, rate limit → 429, upstream key
+  missing/rejected → 503/502. `except openai.APIStatusError` code carries over.
+- **Cost** rides along on each response as the `effgen` extension
+  (`resp.effgen["cost_usd"]` for priced models); OpenAI-only clients ignore it.
+
+### Tools run server-side
+
+This is the one place the protocol differs. effGen executes its **own**
+registered tools on the server and returns the final answer; it does **not**
+forward client-defined function tools for the caller to run, and it does not
+emit client-side `tool_calls` deltas. Request a registered tool by name:
+
+```python
+resp = client.chat.completions.create(
+    model="groq:llama-3.1-8b-instant",
+    messages=[{"role": "user", "content": "What is 17 * 23?"}],
+    tools=[{"type": "function", "function": {"name": "calculator"}}],
+)
+```
+
+An unregistered tool name is refused with a 400 that points at
+`effgen tools list`.
+
+### Native client
+
+For a lighter dependency than the `openai` SDK, `effgen.client.EffGenClient`
+speaks the same server:
+
+```python
+from effgen.client import EffGenClient
+
+c = EffGenClient(base_url="http://localhost:8000", api_key="YOUR_EFFGEN_API_KEY")
+print(c.chat("Hello").content)                              # default model
+print(c.chat("What is 17*23?", tools=["calculator"]).content)  # tool by name
+```
+
+### Coming from LangChain
+
+Point `ChatOpenAI` at the effGen server the same way:
+
+```python
+from langchain_openai import ChatOpenAI
+
+llm = ChatOpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="YOUR_EFFGEN_API_KEY",
+    model="openai:gpt-5-nano",
+)
+```
+
+Chains and prompt templates that call the model over the OpenAI protocol keep
+working. Move any client-side LangChain tool that must run on the server into an
+effGen registered tool (see `effgen tools list` and the tool authoring guide).
+
+---
+
 ## v0.1.x → v0.2.0
 
 ### Breaking Changes

docs/server/openai-compat.md

@@ -34,7 +34,7 @@ curl -H "Authorization: Bearer $EFFGEN_API_KEY" http://127.0.0.1:8000/v1/models
 |---|---|---|
 | POST | `/v1/chat/completions` | Chat completions (streaming + non-streaming) |
 | POST | `/v1/completions` | Legacy text completions |
-| GET | `/v1/models` | List the model aliases |
+| GET | `/v1/models` | List the aliases + ids served this run (not exhaustive) |
 | POST | `/v1/embeddings` | Text embeddings (local SentenceTransformer) |
 
 ## Python (official `openai` client)
@@ -64,6 +64,11 @@ OpenAI names map to concrete effGen models so OpenAI-only clients work:
 |---|---|
 | `gpt-4`, `gpt-4-turbo`, `gpt-4o` | `Qwen/Qwen2.5-7B-Instruct` |
 | `gpt-4o-mini`, `gpt-3.5-turbo` | `Qwen/Qwen2.5-3B-Instruct` |
+| `effgen-default`, `default` | the server's default model (`EFFGEN_DEFAULT_MODEL`, else `Qwen/Qwen2.5-3B-Instruct`) |
+
+`GET /v1/models` lists these aliases plus every id the server has served a
+successful response for this run. The list is **not** exhaustive: any
+`provider:model` id the server can reach is callable whether or not it appears.
 
 Aliasing is never silent. The response `model` field reports the model that
 actually ran, and a non-standard `effgen` object documents the mapping (OpenAI

effgen/api/openai_compat.py

@@ -30,6 +30,7 @@
 from __future__ import annotations
 
 import json
+import os
 import re
 import time
 import uuid
@@ -59,9 +60,28 @@ def Field(default=None, **kwargs):  # type: ignore
     "gpt-3.5-turbo-instruct": "Qwen/Qwen2.5-3B-Instruct",
 }
 
+# Names that route to the server's configured default model. A caller that has
+# no particular model in mind (including the native client's zero-argument
+# ``chat("hi")``) can send ``"effgen-default"`` or ``"default"`` and get an
+# answer without knowing a concrete id. The target is read from
+# ``EFFGEN_DEFAULT_MODEL`` at request time, falling back to a small local model.
+DEFAULT_MODEL_ALIASES: frozenset[str] = frozenset({"effgen-default", "default"})
+_FALLBACK_DEFAULT_MODEL = "Qwen/Qwen2.5-3B-Instruct"
+
+
+def default_model_id() -> str:
+    """Return the model id the ``effgen-default``/``default`` names resolve to.
+
+    Controlled by ``EFFGEN_DEFAULT_MODEL``; defaults to a small local model when
+    the variable is unset or blank.
+    """
+    return os.environ.get("EFFGEN_DEFAULT_MODEL", "").strip() or _FALLBACK_DEFAULT_MODEL
+
 
 def resolve_model_alias(model: str) -> str:
     """Resolve an OpenAI model name to a local effGen model id."""
+    if model in DEFAULT_MODEL_ALIASES:
+        return default_model_id()
     return MODEL_ALIASES.get(model, model)
 
 
@@ -501,9 +521,10 @@ def _effgen_meta(requested: str, resolved: str) -> dict[str, Any]:
     return {
         "requested_model": requested,
         "resolved_model": resolved,
-        # True only when an OpenAI compatibility alias (gpt-4 → local model) was
-        # applied — not for internal provider/model routing normalization.
-        "alias_applied": requested in MODEL_ALIASES,
+        # True when a compatibility alias (gpt-4 → local model) or the
+        # effgen-default/default name was applied — not for internal
+        # provider/model routing normalization.
+        "alias_applied": requested in MODEL_ALIASES or requested in DEFAULT_MODEL_ALIASES,
     }
 
 
@@ -526,6 +547,32 @@ def _messages_to_prompt(messages: list[Any]) -> str:
     return "\n".join(parts)
 
 
+def _has_actionable_content(messages: list[Any]) -> bool:
+    """Return True if there is anything for the model to act on.
+
+    A request whose every message has empty/whitespace/``None``/absent text
+    content — and no image parts, no ``tool_calls``, and no tool result — gives
+    the model nothing to answer. Such a request is rejected with a 400 before a
+    billed model call, matching the Agent layer's empty-task guard and OpenAI's
+    own handling. A message counts as actionable when it has non-blank text, a
+    non-empty multimodal content list, ``tool_calls``, or is a ``tool`` result.
+    """
+    for msg in messages:
+        is_dict = isinstance(msg, dict)
+        role = msg.get("role") if is_dict else getattr(msg, "role", None)
+        content = msg.get("content") if is_dict else getattr(msg, "content", None)
+        tool_calls = msg.get("tool_calls") if is_dict else getattr(msg, "tool_calls", None)
+        if isinstance(content, str) and content.strip():
+            return True
+        if isinstance(content, list) and content:
+            return True
+        if tool_calls:
+            return True
+        if role == "tool" and content is not None:
+            return True
+    return False
+
+
 def create_openai_router(
     runner: Runner, *, extra_models: Callable[[], list[str]] | None = None
 ) -> Any:
@@ -559,15 +606,32 @@ def create_openai_router(
     def _error_response(exc: Exception) -> Any:
         """Return an OpenAI-style, redacted error JSONResponse for *exc*."""
         status, err_type, code = _classify_http(exc)
+        message = str(exc)
+        # A bare/unknown model id falls through to the local loader and fails
+        # with a Transformers "not a valid model identifier" message. Add a
+        # one-line hint pointing at the id shapes the server accepts.
+        low = message.lower()
+        if code == "model_not_found" and (
+            "not a valid model identifier" in low or "is not a local folder" in low
+        ):
+            message += (
+                " Pass a provider-prefixed model id (e.g. 'openai:gpt-5-nano', "
+                "'groq:llama-3.1-8b-instant'), a valid local model id, or "
+                "'effgen-default'."
+            )
         return JSONResponse(
             status_code=status,
-            content=_error_payload(str(exc), err_type, code),
+            content=_error_payload(message, err_type, code),
         )
 
     router = APIRouter(prefix="/v1", tags=["openai-compat"])
 
     @router.get("/models")
     async def list_models() -> dict[str, Any]:
+        # The list is the drop-in aliases plus the ids this process has actually
+        # served a successful response for this run. It is not exhaustive: any
+        # `provider:model` id the server can reach (e.g. "openai:gpt-5-nano",
+        # "groq:llama-3.1-8b-instant") is callable whether or not it appears here.
         now = _now()
         data = [
             {
@@ -579,10 +643,19 @@ async def list_models() -> dict[str, Any]:
             }
             for alias, target in MODEL_ALIASES.items()
         ]
+        # The names that route to the server's configured default model.
+        for name in sorted(DEFAULT_MODEL_ALIASES):
+            data.append({
+                "id": name,
+                "object": "model",
+                "created": now,
+                "owned_by": "effgen",
+                "root": default_model_id(),
+            })
         # Alongside the drop-in legacy aliases, list the ids the server has
-        # actually loaded and served this run (e.g. "openai:gpt-5-nano"),
-        # so a client discovers real, currently-servable models instead of
-        # only the 6 hardcoded OpenAI-flagship aliases.
+        # actually served this run (e.g. "openai:gpt-5-nano"), so a client
+        # discovers real, currently-servable models instead of only the
+        # hardcoded aliases.
         if extra_models is not None:
             try:
                 served = extra_models()
@@ -611,6 +684,17 @@ async def chat_completions(request: ChatCompletionRequest) -> Any:
                     400, "messages must not be empty", code="empty_messages"
                 ),
             )
+        # A request whose messages carry no text, no image, no tool_calls, and no
+        # tool result gives the model nothing to answer. Reject it with a 400
+        # before any billed model call rather than returning a paid-for
+        # non-answer at 200.
+        if not _has_actionable_content(request.messages):
+            return JSONResponse(
+                status_code=400,
+                content=error_envelope(
+                    400, "message content must not be empty", code="empty_content"
+                ),
+            )
         resolved = resolve_model_alias(request.model)
         model = resolved  # echoed in the response 'model' field (what actually ran)
         prompt = _messages_to_prompt(request.messages)

effgen/client/client.py

@@ -110,7 +110,14 @@ def _raise_for_status(self, status: int, payload: Any) -> None:
             return
         msg = f"HTTP {status}"
         if isinstance(payload, dict):
-            msg = payload.get("error") or payload.get("message") or msg
+            err = payload.get("error")
+            if isinstance(err, dict):
+                # OpenAI-style envelope: {"error": {"message": ..., "type": ...}}.
+                msg = err.get("message") or err.get("code") or msg
+            elif isinstance(err, str):
+                msg = err
+            else:
+                msg = payload.get("message") or msg
         if status in (401, 403):
             raise EffGenAuthError(msg, status_code=status, payload=payload)
         if status == 429:
@@ -218,25 +225,30 @@ async def _request_async(
     def chat(
         self,
         message: str,
-        tools: list[str] | None = None,
+        tools: list[str | dict] | None = None,
         model: str | None = None,
         **kwargs: Any,
     ) -> ChatResponse:
-        """Send a single-turn chat message and return the response."""
+        """Send a single-turn chat message and return the response.
+
+        ``tools`` accepts registered tool names (``["calculator"]``) or full
+        OpenAI tool specs; names are expanded to the server's tool-spec shape.
+        The server runs the tool and returns the final answer.
+        """
         body: dict = {
             "model": model or "effgen-default",
             "messages": [{"role": "user", "content": message}],
         }
         if tools is not None:
-            body["tools"] = tools
+            body["tools"] = _tools_to_payload(tools)
         body.update(kwargs)
         payload = self._request_sync("POST", "/v1/chat/completions", body)
         return self._parse_chat(payload)
 
     def embed(
         self,
         texts: list[str],
-        model: str = "text-embedding-small",
+        model: str = "text-embedding-3-small",
     ) -> list[list[float]]:
         """Compute embeddings for a list of input texts."""
         payload = self._request_sync(
@@ -287,7 +299,7 @@ def chat_stream_sync(self, message: str, model: str | None = None) -> Iterator[s
     async def achat(
         self,
         message: str,
-        tools: list[str] | None = None,
+        tools: list[str | dict] | None = None,
         model: str | None = None,
         **kwargs: Any,
     ) -> ChatResponse:
@@ -296,13 +308,13 @@ async def achat(
             "messages": [{"role": "user", "content": message}],
         }
         if tools is not None:
-            body["tools"] = tools
+            body["tools"] = _tools_to_payload(tools)
         body.update(kwargs)
         payload = await self._request_async("POST", "/v1/chat/completions", body)
         return self._parse_chat(payload)
 
     async def aembed(
-        self, texts: list[str], model: str = "text-embedding-small"
+        self, texts: list[str], model: str = "text-embedding-3-small"
     ) -> list[list[float]]:
         payload = await self._request_async(
             "POST", "/v1/embeddings", {"model": model, "input": texts}
@@ -369,6 +381,27 @@ def _parse_chat(payload: Any) -> ChatResponse:
         )
 
 
+def _tools_to_payload(tools: list[Any]) -> list[dict]:
+    """Normalize tool references into the server's OpenAI tool-spec shape.
+
+    A registered tool may be named by a plain string (``"calculator"``); the
+    server expects each tool as ``{"type": "function", "function": {"name":
+    ...}}``. A dict is passed through unchanged so callers can send a full spec.
+    """
+    payload: list[dict] = []
+    for tool in tools:
+        if isinstance(tool, str):
+            payload.append({"type": "function", "function": {"name": tool}})
+        elif isinstance(tool, dict):
+            payload.append(tool)
+        else:
+            raise EffGenAPIError(
+                f"Unsupported tool reference {tool!r}: pass a registered tool "
+                "name (str) or an OpenAI tool spec (dict)"
+            )
+    return payload
+
+
 def _parse_sse_line(line: str) -> str | None:
     """Parse a single SSE line and return the text delta, or None."""
     if not line or not line.startswith("data:"):