Add per-source descriptions and per-namespace declared schemas from p…

CHANGELOG.md

@@ -8,8 +8,15 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release
 
 ## [Unreleased]
 
+### Added
+
+- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE` only — not inline `PINECONE_SOURCES`). Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md](docs/CONFIGURATION.md#json-config-file).
+- **`list_namespaces`:** optional per-namespace `schema_source` (`declared` | `sampled`) and top-level `config_warnings` when private config declarations do not match live Pinecone data.
+
 ### Changed
 
+- **Breaking (`list_sources`):** response shape `sources: string[]` → `sources: { name, description? }[]`. Migration: [MIGRATION.md § Unreleased list_sources](docs/MIGRATION.md#unreleased-list_sources-response-shape).
+- **Breaking (library API):** `PineconeClient.listNamespacesWithMetadata()` now returns `{ namespaces, warnings }` instead of a bare array. Migration: [MIGRATION.md § Unreleased PineconeClient.listNamespacesWithMetadata](docs/MIGRATION.md#unreleased-pineconeclientlistnamespaceswithmetadata-return-shape).
 - **Instructions:** Trimmed operator/install/deploy content (env-var setup, misconfiguration note, Alliance CLI index/rerank defaults, stderr logging config) from `CORE_SERVER_INSTRUCTIONS` and `ALLIANCE_INSTRUCTIONS_APPENDIX` — reduces per-session token cost; no behavior change. Replaced colliding Alliance appendix steps 4–5 with unnumbered "Manual Alliance flow" bullets (includes `PINECONE_DISABLE_SUGGEST_FLOW=true` escape clause). Full detail remains in [docs/CONFIGURATION.md](docs/CONFIGURATION.md).
 
 ## [0.4.0] - 2026-06-24

benchmarks/latency.ts

@@ -233,7 +233,10 @@ function createBenchPineconeMock(): PineconeClient {
       return { count: 42, truncated: false };
     },
     async listNamespacesWithMetadata() {
-      return namespaces;
+      return {
+        namespaces: namespaces.map((n) => ({ ...n, schema_source: 'sampled' as const })),
+        warnings: [],
+      };
     },
     async listNamespacesFromKeywordIndex() {
       return namespaces.map((n) => ({ namespace: n.namespace, recordCount: n.recordCount }));

docs/CONFIGURATION.md

@@ -75,6 +75,17 @@ Set `PINECONE_CONFIG_FILE` (or `--config-file`) to a path such as [examples/mult
 
 Values support `${ENV_VAR}` indirection (resolved at startup). Per-source `sparseIndexName` and `rerankModel` are optional; Alliance defaults apply when omitted.
 
+Optional **config-file-only** fields (not supported in inline `PINECONE_SOURCES`):
+
+| Field | Scope | Purpose |
+| ----- | ----- | ------- |
+| `description` | Per source | Short corpus/content hint surfaced via `list_sources` (helps LLM routing across sources or MCP entries) |
+| `namespaces` | Per source | Map of namespace name → `{ description?, metadata_schema? }` |
+
+`metadata_schema` is a flat `fieldName → type` map (same vocabulary as `list_namespaces` → `metadata_fields`, e.g. `"title": "string"`). When declared for a live namespace, the server **skips live sampling** for that namespace and trusts the declared schema until the config file changes. Namespaces declared in config but absent from Pinecone produce a non-fatal `config_warnings` entry in `list_namespaces` (never a startup failure).
+
+**Never** commit real corpus descriptions, namespace names, or internal field names to the open-source repo — use generic placeholders in examples only. Real values belong in staff-machine private config files per [Deployment profiles](#deployment-profiles). See [SECURITY.md](./SECURITY.md).
+
 ### MCP tools and routing
 
 | Tool | `source` parameter |
@@ -131,7 +142,7 @@ Multi-source mode supports two operational profiles. **Never** ship a merged int
 }
 ```
 
-Prefer `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over inline API keys in `PINECONE_SOURCES`. See [SECURITY.md](./SECURITY.md).
+Prefer `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over inline API keys in `PINECONE_SOURCES`. For internal deployments, add optional `description` and `namespaces` declarations in the JSON config file on staff machines only — never in public examples or committed constants. See [SECURITY.md](./SECURITY.md).
 
 ### Architecture decision
 

docs/MIGRATION.md

@@ -6,6 +6,56 @@ This guide is for **library and MCP client authors** upgrading from earlier **0.
 
 Under [semver 0.y.z](https://semver.org/spec/v2.0.0.html#spec-item-4), **0.1.x → 0.2.0 is a breaking minor** — pin `@0.2.0` only after reading this guide.
 
+## Unreleased: `list_sources` response shape
+
+**Who is affected:** MCP clients parsing `list_sources` success JSON in multi-source mode.
+
+**Before:**
+
+```json
+{
+  "status": "success",
+  "sources": ["api_key_1", "api_key_2"],
+  "default": "api_key_1"
+}
+```
+
+**After:**
+
+```json
+{
+  "status": "success",
+  "sources": [
+    { "name": "api_key_1", "description": "Optional corpus hint from private config" },
+    { "name": "api_key_2" }
+  ],
+  "default": "api_key_1"
+}
+```
+
+Read `sources[].name` instead of treating `sources` as `string[]`. `description` is omitted when not configured.
+
+## Unreleased: `PineconeClient.listNamespacesWithMetadata` return shape
+
+**Who is affected:** Direct library consumers of `PineconeClient` (not MCP tool clients — `list_namespaces` tool response shape is additive only).
+
+**Before:**
+
+```ts
+const rows = await client.listNamespacesWithMetadata();
+// rows: Array<{ namespace, recordCount, metadata }>
+```
+
+**After:**
+
+```ts
+const { namespaces, warnings } = await client.listNamespacesWithMetadata(declaredSchemas);
+// namespaces: Array<{ namespace, recordCount, metadata, schema_source }>
+// warnings: string[] — e.g. stale declared namespace not found live
+```
+
+Optional `declaredSchemas` argument skips live sampling for namespaces with a declared schema.
+
 ## Unreleased: Stable vs experimental response fields
 
 **Rationale:** Tool success payloads mixed stable contract fields with experimental diagnostics (`degraded`, `decision_trace`, etc.) at the top level. Experimental fields are now nested under `experimental` so consumers know which fields are safe across minor version bumps.

docs/SECURITY.md

@@ -23,6 +23,10 @@ Tool responses returned to MCP clients (and LLM consumers) are sanitized in `src
 
 This covers tool error payloads, hybrid degradation reasons, and SDK error text surfaced in DEBUG log mode — not only stderr logs.
 
+## Private config content (descriptions and schemas)
+
+Per-source `description` and per-namespace `namespaces` (including `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles).
+
 ## Docker image
 
 The multi-stage [`Dockerfile`](../Dockerfile):

docs/TOOLS.md

@@ -8,8 +8,8 @@ Success payloads separate **stable** fields (safe across minor bumps after `1.0.
 
 | Tool | Stable | Experimental |
 | ---- | ------ | ------------ |
-| `list_sources` | `status`, `sources`, `default` | _(none)_ |
-| `list_namespaces` | `status`, `cache_hit`, `cache_ttl_seconds`, `expires_at_iso`, `count`, `namespaces`, optional `source_errors` | _(none)_ |
+| `list_sources` | `status`, `sources` (`{ name, description? }[]`), `default` | _(none)_ |
+| `list_namespaces` | `status`, `cache_hit`, `cache_ttl_seconds`, `expires_at_iso`, `count`, `namespaces`, optional `source_errors`, optional `config_warnings`, optional per-row `schema_source` | _(none)_ |
 | `namespace_router` | `status`, `cache_hit`, `user_query`, `suggestions`, `recommended_namespace`, optional `recommended_source` | _(none)_ |
 | `suggest_query_params` | `status`, `cache_hit`, `suggested_fields`, `recommended_tool`, `use_count_tool`, `explanation`, `namespace_found`, optional `source` | _(none)_ |
 | `count` | `status`, `count`, `truncated`, `namespace`, `metadata_filter`, optional `source` | _(none)_ |
@@ -73,7 +73,7 @@ Registered only when more than one Pinecone source is configured.
 | | |
 | --- | --- |
 | **Input** | _(empty object)_ |
-| **Success** | `{ status: 'success', sources: string[], default: string }` |
+| **Success** | `{ status: 'success', sources: { name, description? }[], default: string }` |
 | **Errors** | `LIFECYCLE` when not in multi-source mode |
 
 **Example:**
@@ -91,7 +91,7 @@ Registered only when more than one Pinecone source is configured.
 | | |
 | --- | --- |
 | **Input** | Optional `source` — filter to one configured project |
-| **Success** | `{ status: 'success', cache_hit, cache_ttl_seconds, expires_at_iso, count, namespaces: [{ name, record_count, metadata_fields, source? }], source_errors? }` |
+| **Success** | `{ status: 'success', cache_hit, cache_ttl_seconds, expires_at_iso, count, namespaces: [{ name, record_count, metadata_fields, source?, schema_source? }], source_errors?, config_warnings? }` |
 | **Errors** | `PINECONE_ERROR`, `TIMEOUT`, etc. |
 
 **Example (multi-source, all projects):**

examples/alliance/demo-mock-pinecone-client.ts

@@ -36,21 +36,33 @@ export class DemoMockPineconeClient extends PineconeClient {
     super({ apiKey: '00000000-0000-0000-0000-000000000000', indexName: 'demo-index' });
   }
 
-  override async listNamespacesWithMetadata(): Promise<
-    Array<{ namespace: string; recordCount: number; metadata: Record<string, string> }>
-  > {
-    return [
-      {
-        namespace: DEMO_NAMESPACE,
-        recordCount: 42,
-        metadata: {
-          document_number: 'string',
-          title: 'string',
-          chunk_text: 'string',
-          url: 'string',
+  override async listNamespacesWithMetadata(
+    _declaredSchemas?: Record<string, Record<string, string>>
+  ): Promise<{
+    namespaces: Array<{
+      namespace: string;
+      recordCount: number;
+      metadata: Record<string, string>;
+      schema_source: 'declared' | 'sampled';
+    }>;
+    warnings: string[];
+  }> {
+    return {
+      namespaces: [
+        {
+          namespace: DEMO_NAMESPACE,
+          recordCount: 42,
+          metadata: {
+            document_number: 'string',
+            title: 'string',
+            chunk_text: 'string',
+            url: 'string',
+          },
+          schema_source: 'sampled',
         },
-      },
-    ];
+      ],
+      warnings: [],
+    };
   }
 
   override async listNamespacesFromKeywordIndex(): Promise<KeywordIndexNamespacesResult> {

examples/multi-source/pinecone-sources.json.example

@@ -3,7 +3,17 @@
   "sources": {
     "api_key_1": {
       "apiKey": "${PINECONE_API_KEY_1}",
-      "indexName": "index_name_1"
+      "indexName": "index_name_1",
+      "description": "<optional: describe this corpus in your PRIVATE staff config, not here>",
+      "namespaces": {
+        "example-namespace": {
+          "description": "<optional: describe this namespace in your PRIVATE staff config, not here>",
+          "metadata_schema": {
+            "field_a": "string",
+            "field_b": "number"
+          }
+        }
+      }
     },
     "api_key_2": {
       "apiKey": "${PINECONE_API_KEY_2}",

scripts/test-search.ts

@@ -36,8 +36,8 @@ async function test() {
     // Test 1: List namespaces with metadata
     console.log('\n📋 Test 1: Listing namespaces with metadata...');
     const namespacesInfo = await client.listNamespacesWithMetadata();
-    console.log(`✅ Found ${namespacesInfo.length} namespace(s):`);
-    namespacesInfo.forEach((ns) => {
+    console.log(`✅ Found ${namespacesInfo.namespaces.length} namespace(s):`);
+    namespacesInfo.namespaces.forEach((ns) => {
       console.log(`   - ${ns.namespace}: ${ns.recordCount} records`);
       if (Object.keys(ns.metadata).length > 0) {
         console.log(`     Metadata fields:`, Object.keys(ns.metadata));

src/alliance/tools/isolated-context.context.test.ts

@@ -7,6 +7,7 @@ import {
   isolateFromDefaultContext,
   makeHybridQueryResult,
   makeSearchResult,
+  mockNamespacesWithMetadataResult,
   parseToolJson,
 } from '../../core/server/tools/test-helpers.js';
 import {
@@ -27,18 +28,22 @@ describe('isolated ServerContext with zero default context', () => {
   });
 
   it('guided_query enrich_urls uses ctx builtins, not default registry', async () => {
-    const listNamespacesWithMetadata = vi.fn().mockResolvedValue([
-      {
-        namespace: 'mailing',
-        recordCount: 42,
-        metadata: {
-          document_number: 'string',
-          title: 'string',
-          author: 'string',
-          chunk_text: 'string',
-        },
-      },
-    ]);
+    const listNamespacesWithMetadata = vi
+      .fn()
+      .mockResolvedValue(
+        mockNamespacesWithMetadataResult([
+          {
+            namespace: 'mailing',
+            recordCount: 42,
+            metadata: {
+              document_number: 'string',
+              title: 'string',
+              author: 'string',
+              chunk_text: 'string',
+            },
+          },
+        ])
+      );
     const query = vi.fn().mockResolvedValue(
       makeHybridQueryResult({
         results: [

src/alliance/tools/suggest-query-params-tool.context.test.ts

@@ -7,6 +7,7 @@ import {
   createTestServerContext,
   expectMatchesResponseSchema,
   makeHybridQueryResult,
+  mockNamespacesWithMetadataResult,
   parseToolJson,
 } from '../../core/server/tools/test-helpers.js';
 
@@ -22,7 +23,11 @@ function mockNamespacesClient() {
   return {
     listNamespacesWithMetadata: vi
       .fn()
-      .mockResolvedValue([{ namespace: 'wg21', recordCount: 42, metadata: namespaceMetadata }]),
+      .mockResolvedValue(
+        mockNamespacesWithMetadataResult([
+          { namespace: 'wg21', recordCount: 42, metadata: namespaceMetadata },
+        ])
+      ),
   };
 }
 
@@ -32,19 +37,23 @@ describe('suggest_query_params tool handler (ServerContext instance path)', () =
   });
 
   it('marks suggest-flow on injected context when namespace exists', async () => {
-    const listNamespacesWithMetadata = vi.fn().mockResolvedValue([
-      {
-        namespace: 'wg21',
-        recordCount: 42,
-        metadata: {
-          document_number: 'string',
-          title: 'string',
-          url: 'string',
-          author: 'string',
-          chunk_text: 'string',
-        },
-      },
-    ]);
+    const listNamespacesWithMetadata = vi
+      .fn()
+      .mockResolvedValue(
+        mockNamespacesWithMetadataResult([
+          {
+            namespace: 'wg21',
+            recordCount: 42,
+            metadata: {
+              document_number: 'string',
+              title: 'string',
+              url: 'string',
+              author: 'string',
+              chunk_text: 'string',
+            },
+          },
+        ])
+      );
     const ctx = createTestServerContext({
       client: { listNamespacesWithMetadata } as never,
     });

src/constants.test.ts

@@ -1,3 +1,5 @@
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
 import { describe, expect, it } from 'vitest';
 import {
   ALLIANCE_INSTRUCTIONS_APPENDIX,
@@ -55,4 +57,26 @@ describe('server instructions', () => {
   it('SERVER_INSTRUCTIONS aliases Alliance instructions', () => {
     expect(SERVER_INSTRUCTIONS).toBe(ALLIANCE_SERVER_INSTRUCTIONS);
   });
+
+  it('example multi-source config uses only generic placeholder descriptions and schemas', () => {
+    const examplePath = join(
+      process.cwd(),
+      'examples/multi-source/pinecone-sources.json.example'
+    );
+    const raw = readFileSync(examplePath, 'utf8');
+    expect(raw).toContain(
+      '<optional: describe this corpus in your PRIVATE staff config, not here>'
+    );
+    expect(raw).toContain(
+      '<optional: describe this namespace in your PRIVATE staff config, not here>'
+    );
+    expect(raw).toContain('"field_a": "string"');
+    expect(raw).toContain('"field_b": "number"');
+    expect(CORE_SERVER_INSTRUCTIONS).not.toMatch(
+      /<optional: describe this corpus in your PRIVATE staff config, not here>/
+    );
+    expect(ALLIANCE_SERVER_INSTRUCTIONS).not.toMatch(
+      /<optional: describe this corpus in your PRIVATE staff config, not here>/
+    );
+  });
 });

src/core/pinecone-client.ts

@@ -77,15 +77,19 @@ export class PineconeClient {
     return this.indexSession.listNamespacesFromKeywordIndex();
   }
 
-  /** Dense index namespaces with sampled metadata field types. */
-  async listNamespacesWithMetadata(): Promise<
-    Array<{
+  /** Dense index namespaces with sampled or declared metadata field types. */
+  async listNamespacesWithMetadata(
+    declaredSchemas?: Record<string, Record<string, string>>
+  ): Promise<{
+    namespaces: Array<{
       namespace: string;
       recordCount: number;
       metadata: Record<string, string>;
-    }>
-  > {
-    return this.indexSession.listNamespacesWithMetadata();
+      schema_source: 'declared' | 'sampled';
+    }>;
+    warnings: string[];
+  }> {
+    return this.indexSession.listNamespacesWithMetadata(declaredSchemas);
   }
 
   /** Probe dense + sparse indexes (describeIndexStats) for startup checks. */