feat(client,spec)!: express the AI surface that exists, retire the declarations that never did (#3718) (#3840)

`client.ai` and the real AI service were disjoint sets: three methods whose
URLs no repo has ever mounted (deleted in v17) against 12 mounted routes with
no SDK expression. v17 closed the first half; this closes the second.

The SDK now expresses every AI route meant to be tenant surface — ai.chat /
ai.chatStream (POST /ai/chat, JSON | UI Message Stream), ai.complete,
ai.models, and ai.conversations.{create,list,get,update,delete,addMessage}.
`chat` forces `stream: false` because the endpoint streams by default;
`chatStream` returns a promise for an async iterable so the request (and any
HTTP error) happens on call, not on first iteration.

Breaking, and none of it had an implementation anywhere: the Ai{Nlq,Suggest,
Insights} schemas/types are replaced by the wire shapes of the real routes,
DEFAULT_AI_ROUTES is deleted (8 default registrations, not 9), and AiProtocol
is deleted — the real server contract is IAIService + IAIConversationService.

/api/v1/ai/ becomes a bounded prefix exemption in the capstone (#3642): only
`ai.*` may use it and the namespace must still be reaching it. The reachability
check lives where the routes are, in cloud's ai-route-ledger conformance test,
which reads buildAIRoutes() and drives this SDK against it. The wildcard-only
bound stays 0.

Refs #3718, #3708, #3642, #3563

Author: os-zhuang

.changeset/ai-namespace-expresses-real-surface.md

@@ -0,0 +1,65 @@
+---
+"@objectstack/client": major
+"@objectstack/spec": major
+---
+
+feat(client,spec)!: the SDK's `ai` namespace now expresses the AI surface that exists (#3718)
+
+`client.ai` and the AI service were **disjoint sets**. The namespace held three
+methods — `nlq`, `suggest`, `insights` — whose URLs no repo has ever mounted
+(removed in v17), while `service-ai` mounted 12 routes the SDK could not reach
+at all. v17 closed the first half by deleting the dead methods. This closes the
+second: the SDK now reaches every route that is meant to be tenant API surface.
+
+| SDK | Route |
+|---|---|
+| `ai.chat(request)` | `POST /api/v1/ai/chat` — forces `stream: false`, so the JSON mode is what you get |
+| `ai.chatStream(request)` | `POST /api/v1/ai/chat` — `AsyncIterable` of UI Message Stream frames |
+| `ai.complete(request)` | `POST /api/v1/ai/complete` |
+| `ai.models()` | `GET /api/v1/ai/models` — the ADR-0028 plan-filtered picker list |
+| `ai.conversations.create/list/get/update/delete/addMessage` | the six `/api/v1/ai/conversations` routes |
+
+`ai.chatStream` returns a promise for an async iterable rather than being an
+async generator, so the request is issued — and an HTTP error thrown — when you
+call it, not when you first iterate.
+
+**Where the server is.** `service-ai` is a Cloud/EE package in the `cloud`
+repo; this repo only proxies `/api/v1/ai/**` and 404s `AI service is not
+configured` without it. Check `discovery.services` before calling, exactly as
+for any other plugin-provided namespace. For a React chat UI, `useChat()`
+(`@ai-sdk/react`) is still the better client — it speaks the same protocol
+`ai.chatStream` parses and owns message state; these methods are for callers
+that are not components.
+
+**Breaking — the spec's dead AI declarations are retired.** All three had no
+implementation anywhere and no runtime consumer:
+
+- `Ai{Nlq,Suggest,Insights}{Request,Response}[Schema]` → replaced by the wire
+  shapes of the real routes: `AiChat{Request,Response}`, `AiStreamChunk`,
+  `AiCompleteRequest`, `AiModelsResponse`, `AiConversation`, `AiMessage`,
+  `{Create,List,Update}AiConversation*`. The six retired JSON Schemas are
+  dropped from `json-schema.manifest.json` (deliberate retirement, #2978).
+- `DEFAULT_AI_ROUTES` → deleted, and `getDefaultRouteRegistrations()` returns 8
+  groups instead of 9. It declared the three phantom endpoints and had no
+  runtime consumer; re-declaring the real ones here would recreate the same
+  illusion, since they are mounted from another repo.
+- `AiProtocol` (`aiNlq?` / `aiSuggest?` / `aiInsights?`) → deleted. Nothing
+  implemented it and nothing dispatched through it. The real server contract is
+  `IAIService` + `IAIConversationService` in `@objectstack/spec/contracts`.
+
+**The guard.** `/api/v1/ai/` becomes a bounded prefix exemption in the capstone
+(#3642) alongside the control plane — bounded from both ends: only `ai.*` may
+use it, and the namespace must still be reaching it. That is not a
+wave-through. The reachability check lives where the routes are:
+`cloud`'s `packages/service-ai/src/ai-route-ledger.conformance.test.ts` reads
+the table `buildAIRoutes()` returns and drives this SDK against it, so an
+`ai.*` URL that stops resolving fails a test in the repo that mounts it. The
+wildcard-only bound stays **0** — these URLs never touch the `* /ai/**` row,
+which is what certified three dead methods for years.
+
+The four replaced client tests are worth naming: they mocked `fetch` and
+asserted the URL the client *built*, never that anything answered it, and
+passed for years against endpoints that did not exist. The new ones assert only
+what this repo can honestly know — verb, path, and the body decisions the SDK
+makes for you (`stream: false` on `chat`, the 204 on `delete`, SSE frame
+parsing) — and leave "does it resolve" to the ledger next to the routes.

content/docs/api/client-sdk.mdx

@@ -128,7 +128,7 @@ The `@objectstack/client` SDK aims to implement the ObjectStack API protocol spe
 | **storage** | ✅ | 2 | File upload & download |
 | **i18n** | ✅ | 3 | Internationalization |
 | **notifications** | ✅ | 3 | List, mark-read, mark-all-read (inbox/receipt spine, ADR-0030) |
-| **ai** | ✅ | 3 | AI services (NLQ, suggest, insights) |
+| **ai** | ✅ | 10 | Chat (JSON + streaming), completion, model picker, conversation CRUD — the surface `service-ai` (Cloud/EE) mounts |
 
 The former `permissions`, `views`, `workflow`, and `realtime` namespaces (and
 the notifications device/preference helpers) were removed in #3612: no server
@@ -312,21 +312,40 @@ await client.notifications.markAllRead();
 // through sys_inbox_message and tracks read-state in sys_notification_receipt.
 // These helpers will be repointed during the objectui bell cut-over.
 
-// AI — the `client.ai` namespace was REMOVED in v17 (#3718).
-//
-// It held `nlq`, `suggest` and `insights`, which built
-// /api/v1/ai/{nlq,suggest,insights}. No server in any repo ever mounted those
-// paths, so every call 404ed for the whole life of the namespace. They were
-// typed and shipped, which is exactly why they looked usable.
-//
-// The AI surface that DOES exist is served by `service-ai` (Cloud/EE):
-// POST /api/v1/ai/chat and /chat/stream, POST /complete, GET /models, and six
-// /conversations routes. The SDK has no method for any of them yet — that is
-// tracked on #3718 as new API, not as a rename of what was removed.
+// AI — served by `service-ai` (Cloud/EE); 404s "AI service is not configured"
+// when the plugin is absent, so check `discovery.services` first.
+const answer = await client.ai.chat({
+  messages: [{ role: 'user', content: 'How many open orders this quarter?' }],
+  conversationId,                        // omit to have one created and echoed back
+});
+answer.content;                          // string
+answer.usage?.totalTokens;
+
+// Streaming — the Vercel UI Message Stream Protocol, frame by frame.
+for await (const frame of await client.ai.chatStream({ messages })) {
+  if (frame.type === 'text-delta') process.stdout.write(frame.delta as string);
+}
+
+await client.ai.complete({ prompt: 'Summarise this account in one line:' });
+await client.ai.models();                // plan-filtered picker list (ADR-0028)
+
+// Conversations — all six routes, scoped to the authenticated user server-side.
+const conv = await client.ai.conversations.create({ title: 'Q3 pipeline' });
+await client.ai.conversations.list({ limit: 20 });
+await client.ai.conversations.get(conv.id);
+await client.ai.conversations.addMessage(conv.id, { role: 'user', content: 'hi' });
+await client.ai.conversations.update(conv.id, { title: 'Renamed' });
+await client.ai.conversations.delete(conv.id);
+
+// In a React chat UI prefer `useChat()` (`@ai-sdk/react`) over `ai.chatStream`:
+// it speaks the same protocol and owns message state. These methods are for
+// everything that is not a component — server code, jobs, CLIs, tests.
 //
-// For chat, call the endpoint directly with the Vercel AI SDK
-// (`useChat()` from `@ai-sdk/react`); it speaks the Data Stream Protocol that
-// POST /api/v1/ai/chat serves.
+// #3718 history: `client.ai` used to hold `nlq`, `suggest` and `insights`,
+// building /api/v1/ai/{nlq,suggest,insights}. No server in any repo ever
+// mounted those paths, so every call 404ed for the whole life of the
+// namespace. They were typed and shipped, which is exactly why they looked
+// usable. v17 removed them; the methods above are the surface that exists.
 
 // i18n — Internationalization
 await client.i18n.getLocales();

content/docs/api/plugin-endpoints.mdx

@@ -95,19 +95,19 @@ The core dispatcher implements only the list / read / read-all routes above. Dev
 
 ### AI (`/ai`) — Plugin Required
 
-These are the routes `service-ai` mounts:
-
-| Method | Endpoint | Description |
-|:-------|:---------|:------------|
-| POST | `/ai/chat` | Chat completion (Vercel Data Stream or JSON) |
-| POST | `/ai/chat/stream` | SSE streaming chat |
-| POST | `/ai/complete` | Text completion |
-| GET | `/ai/models` | Models this environment offers (ADR-0028) |
-| GET | `/ai/status` | Active adapter provenance |
-| GET | `/ai/effective-model` | Resolved model ids and their source |
-| POST / GET | `/ai/conversations` | Create / list conversations |
-| GET / PATCH / DELETE | `/ai/conversations/:id` | Read / update / delete |
-| POST | `/ai/conversations/:id/messages` | Append a message |
+These are the routes `service-ai` mounts, and the SDK method that reaches each:
+
+| Method | Endpoint | SDK | Description |
+|:-------|:---------|:----|:------------|
+| POST | `/ai/chat` | `ai.chat` / `ai.chatStream` | Chat completion — JSON with `stream: false`, otherwise the Vercel UI Message Stream |
+| POST | `/ai/chat/stream` | — | Generic-SSE twin of `/ai/chat`: the same completion without the tool loop or persistence |
+| POST | `/ai/complete` | `ai.complete` | Text completion |
+| GET | `/ai/models` | `ai.models` | Models this environment offers (ADR-0028) |
+| GET | `/ai/status` | — | Active adapter provenance (console diagnostics) |
+| GET | `/ai/effective-model` | — | Resolved model ids and their source (console diagnostics) |
+| POST / GET | `/ai/conversations` | `ai.conversations.create` / `.list` | Create / list conversations |
+| GET / PATCH / DELETE | `/ai/conversations/:id` | `ai.conversations.get` / `.update` / `.delete` | Read / update / delete |
+| POST | `/ai/conversations/:id/messages` | `ai.conversations.addMessage` | Append a message |
 
 <Callout type="warn">
 **This table used to be inverted (#3718).** It listed `/ai/nlq`,
@@ -116,12 +116,16 @@ and its callout stated "there is no `/ai/chat` route", which was wrong.
 
 The three phantom routes were declared in `DEFAULT_AI_ROUTES` and called by
 `client.ai.nlq/suggest/insights`; every call 404ed. **v17 removed that SDK
-namespace** rather than build endpoints for it, so nothing calls them now.
-
-No route above has a client-SDK method yet — reach them directly, or with
-`useChat()` (`@ai-sdk/react`) for chat, which speaks the Data Stream Protocol
-`POST /ai/chat` serves. Reviewed dispositions for all 12 live in the `cloud`
-repo, `packages/service-ai/src/ai-route-ledger.ts`.
+namespace** rather than build endpoints for it, and the same issue then gave
+the SDK the surface that does exist — the `ai.*` column above.
+
+Reviewed dispositions for all 12 routes live in the `cloud` repo,
+`packages/service-ai/src/ai-route-ledger.ts`, whose conformance test reads
+`buildAIRoutes()` and drives the SDK against it. The three rows with no SDK
+method are deliberate: `/status` and `/effective-model` are operator
+diagnostics, and `/chat/stream` is superseded by `/chat`'s streaming mode.
+For a React chat UI prefer `useChat()` (`@ai-sdk/react`) — it speaks the same
+protocol `ai.chatStream` parses and owns message state for you.
 </Callout>
 
 ### i18n (`/i18n`) — Plugin Required