feat: Zod-validated tool responses and stable/experimental field taxonomy (#151)

* feat: Zod-validated tool responses and stable/experimental field taxonomy

Nest experimental MCP fields (degraded, decision_trace, etc.) under
experimental, add response-schemas.ts for all 9 tools with runtime
validation, and document the stable/experimental boundary.

Fixes #139, #140

* addressed review agent results

* addressed reviews

---------

Co-authored-by: zho <jornathanm910923@gmail.com>

Author: jonathanMLDev

CHANGELOG.md

@@ -10,11 +10,14 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release
 
 ### Changed
 
+- **Breaking (MCP):** Experimental tool response fields are nested under `experimental` on success payloads. Affected tools: `query`, `query_documents`, `guided_query`. Fields moved: `degraded`, `degradation_reason`, `hybrid_leg_failed`, `rerank_skipped_reason` (query-shaped tools); `decision_trace` (`guided_query`). Stable fields (`status`, `results`, `namespace`, etc.) are unchanged. See [MIGRATION.md](docs/MIGRATION.md#unreleased-stable-vs-experimental-response-fields).
 - **Breaking (core):** `resolveConfig` requires a Pinecone index name and no longer applies Alliance index/rerank defaults. Removed exported `DEFAULT_INDEX_NAME` and `DEFAULT_RERANK_MODEL` from the package root. Rerank is opt-in when `PINECONE_RERANK_MODEL` / `rerankModel` is unset.
 - **Breaking (core):** `setupCoreServer` MCP `instructions` use `CORE_SERVER_INSTRUCTIONS` (no `guided_query` / `suggest_query_params`). `resolveConfig` defaults `disableSuggestFlow` to `true` so `query` / `count` / `query_documents` work without Alliance tools. Alliance CLI / `resolveAllianceConfig` unchanged: gate on by default, `ALLIANCE_SERVER_INSTRUCTIONS`.
 - **Alliance CLI / `resolveAllianceConfig`:** When index or rerank env/CLI values are omitted, defaults remain `rag-hybrid` and `bge-reranker-v2-m3` (API-key-only MCP configs unchanged). See [examples/alliance/.env.example](examples/alliance/.env.example).
 ### Added
 
+- Zod schemas for all nine MCP tool success responses (`queryResponseSchema`, `guidedQueryResponseSchema`, etc.) exported from the package root for client-side validation. Success payloads are runtime-validated before return.
+- Stable vs experimental response field taxonomy documented in [docs/TOOLS.md](docs/TOOLS.md) and [docs/deprecation-policy.md](docs/deprecation-policy.md#stable-vs-experimental-mcp-response-fields).
 - Formal deprecation policy ([docs/deprecation-policy.md](docs/deprecation-policy.md)) and breaking-change release notes template ([docs/templates/breaking-change-release-notes.md](docs/templates/breaking-change-release-notes.md)).
 
 ## [0.2.0] - 2026-05-29

README.md

@@ -104,7 +104,7 @@ To try the server on **your own** Pinecone project (free tier, no Alliance index
 The codebase is split into two layers:
 
 - **`src/core/`** — generic MCP–Pinecone bridge (`PineconeClient`, `resolveConfig`, core MCP tools). Import from `@will-cppa/pinecone-read-only-mcp` (package root).
-- **`src/alliance/`** — C++ Alliance app tools (`suggest_query_params`, `guided_query`, Boost/Slack URL builtins). Import from `@will-cppa/pinecone-read-only-mcp/alliance`; embed via `resolveAllianceConfig` → `createServer` / `setClient` → `setupAllianceServer({ context: ctx })` (see [Library embedding](#library-embedding)).
+- **`src/alliance/`** — C++ Alliance app tools (`suggest_query_params`, `guided_query`, Boost/Slack URL builtins). Import from `@will-cppa/pinecone-read-only-mcp/alliance` and use `createServer(config)` → `ctx.setClient(...)` → `setupAllianceServer({ context: ctx })` for the full tool surface (what the CLI uses); see [Library embedding](#library-embedding) below.
 
 ## Configuration
 
@@ -198,7 +198,7 @@ A fuller embedding sample lives in [examples/alliance/custom-url-generator.ts](e
 | File | Description |
 | ---- | ----------- |
 | [examples/alliance/suggest-flow-demo.ts](examples/alliance/suggest-flow-demo.ts) | Manual **suggest_query_params → query** flow |
-| [examples/alliance/guided-query-demo.ts](examples/alliance/guided-query-demo.ts) | **guided_query** and `decision_trace` |
+| [examples/alliance/guided-query-demo.ts](examples/alliance/guided-query-demo.ts) | **guided_query** and `experimental.decision_trace` |
 | [examples/alliance/library-embedding-demo.ts](examples/alliance/library-embedding-demo.ts) | **setupAllianceServer** without the CLI |
 | [examples/alliance/custom-url-generator.ts](examples/alliance/custom-url-generator.ts) | Custom **URL generator** registration |
 
@@ -424,7 +424,7 @@ Single orchestrator tool that runs the full flow in one call:
 2. query param suggestion,
 3. execution via `count` or hybrid `query` (`fast` / `detailed` / `full` presets).
 
-It returns both the final result and a `decision_trace` for transparency.
+It returns both the final result and `experimental.decision_trace` for transparency.
 
 **Parameters:**
 
@@ -437,7 +437,7 @@ It returns both the final result and a `decision_trace` for transparency.
 | `preferred_tool`  | enum    | No       | `auto`  | One of `auto`, `count`, `fast`, `detailed`, `full`                                  |
 | `enrich_urls`     | boolean | No       | `true`  | Auto-generate URLs for `mailing` and `slack-Cpplang` when `metadata.url` is missing |
 
-**Returns:** JSON containing `decision_trace` and `result`.
+**Returns:** JSON containing `experimental.decision_trace` and `result`.
 
 ### `generate_urls`
 

docs/CONFIGURATION.md

@@ -28,7 +28,7 @@ Configuration is built from **CLI flags** (when using the binary), **environment
 
 **Throws** if `apiKey` or `indexName` is missing after trim.
 
-For the full Alliance tool surface (including `suggest_query_params`, `guided_query`, and built-in URL generators), import from `@will-cppa/pinecone-read-only-mcp/alliance` and follow the [Library embedding](#library-embedding) flow: `resolveAllianceConfig` → `createServer` / `setClient` → `setupAllianceServer({ context: ctx })`.
+For the full Alliance tool surface (including `suggest_query_params`, `guided_query`, and built-in URL generators), import from `@will-cppa/pinecone-read-only-mcp/alliance` and use the three-step instance-first recipe at [Library embedding](#library-embedding) below.
 
 ### Core vs Alliance resolvers
 

docs/CONTRIBUTING.md

@@ -49,6 +49,14 @@ Follow [deprecation-policy.md](./deprecation-policy.md). In summary:
 - **Breaking** a release (especially while `0.y.z`): use labeled bullets — `**Breaking (MCP):**`, `**Breaking (types):**`, etc. — each with what changed, who is affected, and a link to a MIGRATION anchor.
 - **Removing** deprecated APIs: add `### Removed` only after the deprecation window; keep migration text in MIGRATION.md.
 
+## Response field stability
+
+When changing MCP tool **success** response shapes:
+
+- New fields default to the **`experimental`** nested object unless explicitly promoted (see [deprecation-policy.md § Stable vs experimental](./deprecation-policy.md#stable-vs-experimental-mcp-response-fields)).
+- Update Zod schemas in `src/core/server/response-schemas.ts` and export from `src/core/index.ts`.
+- Update [TOOLS.md](./TOOLS.md) stable/experimental tables for the affected tool(s).
+
 ## Documentation
 
 Authoritative reference lives under [`docs/`](./README.md). When adding tools or config knobs, update `docs/TOOLS.md` and `docs/CONFIGURATION.md` in the same PR when possible.

docs/MIGRATION.md

@@ -6,6 +6,69 @@ This guide is for **library and MCP client authors** upgrading from earlier **0.
 
 Under [semver 0.y.z](https://semver.org/spec/v2.0.0.html#spec-item-4), **0.1.x → 0.2.0 is a breaking minor** — pin `@0.2.0` only after reading this guide.
 
+## Unreleased: Stable vs experimental response fields
+
+**Rationale:** Tool success payloads mixed stable contract fields with experimental diagnostics (`degraded`, `decision_trace`, etc.) at the top level. Experimental fields are now nested under `experimental` so consumers know which fields are safe across minor version bumps.
+
+**Who is affected:** MCP clients and integrators parsing `query`, `query_documents`, or `guided_query` success JSON.
+
+**Before (`query`):**
+
+```json
+{
+  "status": "success",
+  "results": [],
+  "degraded": true,
+  "degradation_reason": "rerank_failed: timeout",
+  "hybrid_leg_failed": "dense"
+}
+```
+
+**After (`query`):**
+
+```json
+{
+  "status": "success",
+  "results": [],
+  "experimental": {
+    "degraded": true,
+    "degradation_reason": "rerank_failed: timeout",
+    "hybrid_leg_failed": "dense"
+  }
+}
+```
+
+**Before (`guided_query`):**
+
+```json
+{
+  "status": "success",
+  "decision_trace": { "selected_namespace": "mailing" },
+  "result": { "status": "success", "results": [], "degraded": false }
+}
+```
+
+**After (`guided_query`):**
+
+```json
+{
+  "status": "success",
+  "experimental": {
+    "decision_trace": { "selected_namespace": "mailing", "rerank_status": "success" }
+  },
+  "result": {
+    "status": "success",
+    "results": []
+  }
+}
+```
+
+When no experimental fields apply, the `experimental` key is **omitted** (not an empty object).
+
+**Client-side validation:** Import Zod schemas from the package root, e.g. `import { queryResponseSchema } from '@will-cppa/pinecone-read-only-mcp'`.
+
+**Promotion:** Moving a field from `experimental` to stable requires CHANGELOG, TOOLS.md, and schema updates per [deprecation-policy.md § Stable vs experimental](./deprecation-policy.md#stable-vs-experimental-mcp-response-fields).
+
 ## Unreleased: `ServerContext` instance APIs (phase 1)
 
 **Rationale:** Process-global singletons (Pinecone client slot, config, URL registry, suggest-flow gate, namespaces cache) complicate testing and multi-tenant embedding. Phase 1 introduces an opt-in **`ServerContext`** without removing legacy getters.

docs/TOOLS.md

@@ -2,6 +2,24 @@
 
 Unless noted, failures return MCP `isError: true` with JSON matching `ToolError` (see [MIGRATION.md](./MIGRATION.md) and [README error table](../README.md#error-responses)).
 
+## Response field stability
+
+Success payloads separate **stable** fields (safe across minor bumps after `1.0.0`) from **experimental** fields (may change before `1.0.0`). Experimental fields are nested under `experimental` when present; the key is omitted when empty.
+
+| Tool | Stable | Experimental |
+| ---- | ------ | ------------ |
+| `list_namespaces` | `status`, `cache_hit`, `cache_ttl_seconds`, `expires_at_iso`, `count`, `namespaces` | _(none)_ |
+| `namespace_router` | `status`, `cache_hit`, `user_query`, `suggestions`, `recommended_namespace` | _(none)_ |
+| `suggest_query_params` | `status`, `cache_hit`, `suggested_fields`, `recommended_tool`, `use_count_tool`, `explanation`, `namespace_found` | _(none)_ |
+| `count` | `status`, `count`, `truncated`, `namespace`, `metadata_filter` | _(none)_ |
+| `query` | `status`, `mode`, `query`, `namespace`, `metadata_filter`, `result_count`, `results`, `fields` | `experimental.degraded`, `experimental.degradation_reason`, `experimental.hybrid_leg_failed`, `experimental.rerank_skipped_reason` |
+| `keyword_search` | `status`, `query`, `namespace`, `index`, `metadata_filter`, `result_count`, `results`, `fields` | _(none)_ |
+| `query_documents` | `status`, `query`, `namespace`, `metadata_filter`, `result_count`, `documents` | Same experimental degradation fields as `query` |
+| `guided_query` | `status`, `result` (count or query-shaped stable fields) | `experimental.decision_trace`; query-path `result.experimental` degradation fields |
+| `generate_urls` | `status`, `namespace`, `count`, `results` | _(none)_ |
+
+Promotion process: [deprecation-policy.md § Stable vs experimental](./deprecation-policy.md#stable-vs-experimental-mcp-response-fields).
+
 ## Core vs Alliance tool surface
 
 | Setup | Tools | MCP instructions |
@@ -108,7 +126,7 @@ When **`disableSuggestFlow`** is **`true`** (core default via `resolveConfig`),
 | `metadata_filter` | object | no | Metadata filter |
 | `fields` | string[] | no | Pinecone fields to return |
 
-**Success (`QueryResponse`):** `{ status: 'success', mode?: 'query' \| 'query_fast' \| 'query_detailed', query, namespace, metadata_filter?, result_count, results[], fields?, degraded?, degradation_reason?, hybrid_leg_failed? }`.
+**Success (`QueryResponse`):** `{ status: 'success', mode?, query, namespace, metadata_filter?, result_count, results[], fields?, experimental?: { degraded?, degradation_reason?, hybrid_leg_failed?, rerank_skipped_reason? } }`.
 
 Each row: `document_id`, `paper_number` (deprecated alias), `title`, `author`, `url`, `content`, `score`, `reranked`, optional `metadata`.
 
@@ -125,17 +143,17 @@ Each row: `document_id`, `paper_number` (deprecated alias), `title`, `author`, `
 
 ### Rerank and hybrid degradation
 
-When reranking is requested but the rerank API fails, the server still returns **`status: 'success'`** with rows where `reranked: false`, plus envelope fields:
+When reranking is requested but the rerank API fails, the server still returns **`status: 'success'`** with rows where `reranked: false`, plus **experimental** envelope fields:
 
 | Field | When set | Meaning |
 | ----- | -------- | ------- |
-| `degraded` | `true` | Rerank was attempted and failed (or another degradation path fired) |
-| `degradation_reason` | string | Human-readable detail for MCP/LLM clients (e.g. `rerank_failed: timeout after 5000ms`) |
-| `hybrid_leg_failed` | `'dense'` \| `'sparse'` \| omitted / `null` | Exactly one hybrid search leg failed while the other returned hits |
+| `experimental.degraded` | `true` | Rerank was attempted and failed (or another degradation path fired) |
+| `experimental.degradation_reason` | string | Human-readable detail for MCP/LLM clients (e.g. `rerank_failed: timeout after 5000ms`) |
+| `experimental.hybrid_leg_failed` | `'dense'` \| `'sparse'` | Exactly one hybrid search leg failed while the other returned hits |
 
-Treat **`degraded: true`** as lower confidence even when `status` is `success`. Combine with per-row `reranked`, `preset`, and `use_reranking`. Structured stderr logs may include additional detail.
+Treat **`experimental.degraded: true`** as lower confidence even when `status` is `success`. Combine with per-row `reranked`, `preset`, and `use_reranking`. Structured stderr logs may include additional detail.
 
-`query_documents` propagates the same flags on its nested query payload when applicable.
+`query_documents` propagates the same experimental flags when applicable.
 
 ---
 
@@ -167,7 +185,7 @@ Treat **`degraded: true`** as lower confidence even when `status` is `success`.
 | `metadata_filter` | object | no | Filter |
 | `max_chunks_per_document` | int | no | Cap merged chunks per doc (default 200, max 500) |
 
-**Success:** `{ status: 'success', query, namespace, metadata_filter?, result_count, documents: [{ document_id, merged_content, metadata, chunk_count, best_score }] }`.
+**Success:** `{ status: 'success', query, namespace, metadata_filter?, result_count, documents[], experimental?: { degraded?, degradation_reason?, hybrid_leg_failed?, rerank_skipped_reason? } }`.
 
 ---
 
@@ -184,11 +202,11 @@ Treat **`degraded: true`** as lower confidence even when `status` is `success`.
 | `preferred_tool` | `auto` \| `count` \| `fast` \| `detailed` \| `full` | no | Override automated tool choice |
 | `enrich_urls` | boolean | no (default true) | Run URL generator when metadata lacks `url` |
 
-**Success:** `{ status: 'success', decision_trace, result }` where `result` is either a count payload or a `QueryResponse`-shaped query payload.
+**Success:** `{ status: 'success', experimental: { decision_trace }, result }` where `result` is either a count payload or a `QueryResponse`-shaped query payload.
 
-**`decision_trace` fields (non-exhaustive):** `cache_hit`, `input_namespace`, `routed_namespace`, `selected_namespace`, `ranked_namespaces`, `suggested_fields`, `suggested_tool`, `selected_tool`, `explanation`, `enrich_urls`, `rerank_status` (`success` \| `skipped` \| `failed`).
+**`experimental.decision_trace` fields (non-exhaustive):** `cache_hit`, `input_namespace`, `routed_namespace`, `selected_namespace`, `ranked_namespaces`, `suggested_fields`, `suggested_tool`, `selected_tool`, `explanation`, `enrich_urls`, `rerank_status` (`success` \| `skipped` \| `skipped_no_model` \| `failed`).
 
-When the inner query path runs, `result` includes the same `degraded`, `degradation_reason`, and `hybrid_leg_failed` fields as `query` (see [Rerank and hybrid degradation](#rerank-and-hybrid-degradation)).
+When the inner query path runs, `result.experimental` includes the same degradation fields as `query` (see [Rerank and hybrid degradation](#rerank-and-hybrid-degradation)).
 
 **Example:**
 
@@ -227,4 +245,4 @@ Single-shot:
   guided_query
 ```
 
-Canonical Zod schemas live beside each handler under `src/server/tools/*.ts`.
+Canonical Zod input schemas live beside each handler under `src/core/server/tools/*.ts` and `src/alliance/tools/*.ts`. Success response schemas are in `src/core/server/response-schemas.ts` and exported from the package root (e.g. `queryResponseSchema`, `guidedQueryResponseSchema`).

docs/deprecation-policy.md

@@ -64,6 +64,33 @@ While **`0.y.z`**, a minor release **may** ship breaking changes without a prior
 
 Security fixes may break behavior when required; document impact in CHANGELOG and MIGRATION.
 
+## Stable vs experimental MCP response fields
+
+MCP tool **success** responses separate fields into two tiers:
+
+| Tier | Meaning | Location in JSON |
+| ---- | ------- | ---------------- |
+| **Stable** | Safe to depend on across minor version bumps after `1.0.0` | Top-level keys (`status`, `results`, `namespace`, `count`, etc.) |
+| **Experimental** | May change or be removed before `1.0.0` | Nested under `experimental` when present |
+
+Current experimental fields (see [TOOLS.md](./TOOLS.md) per tool):
+
+- `degraded`, `degradation_reason`, `hybrid_leg_failed`, `rerank_skipped_reason` — query-shaped tools (`query`, `query_documents`, and `guided_query.result` when applicable)
+- `decision_trace` — `guided_query` only
+
+The `experimental` key is **omitted** when no experimental fields are set.
+
+### Promoting experimental → stable
+
+To promote a field from experimental to stable:
+
+1. **CHANGELOG** — `### Changed` entry describing the promotion and target release.
+2. **TOOLS.md** — Move the field from the Experimental to Stable table for the affected tool(s).
+3. **Schema** — Move the field out of `experimental` in `response-schemas.ts` and update handler builders.
+4. **MIGRATION.md** — Document before/after for integrators.
+
+New response fields added before `1.0.0` default to `experimental` unless explicitly promoted through this process.
+
 ## Future instance APIs (`ServerContext`)
 
 Phase 1 of the **`ServerContext`** / **`createServer(config)`** refactor is available while legacy module-level getters remain supported. See [MIGRATION.md § ServerContext instance APIs](./MIGRATION.md#unreleased-servercontext-instance-apis-phase-1) for upgrade steps.

examples/alliance/README.md

@@ -3,7 +3,7 @@
 These examples target the **full** MCP surface via `setupAllianceServer` from `@will-cppa/pinecone-read-only-mcp/alliance`:
 
 - `suggest_query_params` and the suggest-flow gate
-- `guided_query` with `decision_trace`
+- `guided_query` with `experimental.decision_trace`
 - Built-in URL generators for `mailing` and `slack-Cpplang`
 
 They assume a Pinecone index you control with compatible data (not necessarily the C++ Alliance production index). To bootstrap a **neutral** index from scratch, start with [examples/quickstart/README.md](../quickstart/README.md).
@@ -24,7 +24,7 @@ Optional: `PINECONE_RERANK_MODEL`, `PINECONE_SPARSE_INDEX_NAME`, etc. See [docs/
 | File | Description |
 | ---- | ----------- |
 | [suggest-flow-demo.ts](./suggest-flow-demo.ts) | Manual **suggest_query_params → query** flow |
-| [guided-query-demo.ts](./guided-query-demo.ts) | **guided_query** and `decision_trace` |
+| [guided-query-demo.ts](./guided-query-demo.ts) | **guided_query** and `experimental.decision_trace` |
 | [library-embedding-demo.ts](./library-embedding-demo.ts) | Programmatic **setupAllianceServer** wiring |
 | [custom-url-generator.ts](./custom-url-generator.ts) | Custom **URL generator** registration |
 | [demo-mock-pinecone-client.ts](./demo-mock-pinecone-client.ts) | Mock client with `mailing` namespace (no network) |