Add per-source descriptions and per-namespace declared schemas from p… (#204)

* Add per-source descriptions and per-namespace declared schemas from private config

* fixed format errors

* Address PR review: fix test-search script, tighten description validation, harden tests/docs

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix stale-namespace warnings and surface namespace descriptions on list_namespaces

* addressed reviews: not.toContain(corpusPlaceholder) - clearest intent: "this exact text must not be in the instructions."

* created namespace-cache.ts for removing the duplication

---------

Co-authored-by: zho <jornathanm910923@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Author: 3 people

CHANGELOG.md

@@ -8,8 +8,15 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release
 
 ## [Unreleased]
 
+### Added
+
+- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE` only — not inline `PINECONE_SOURCES`). Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md](docs/CONFIGURATION.md#json-config-file).
+- **`list_namespaces`:** optional per-namespace `schema_source` (`declared` | `sampled`), optional per-namespace `description` (from private config), and top-level `config_warnings` when private config declarations do not match live Pinecone data.
+
 ### Changed
 
+- **Breaking (`list_sources`):** response shape `sources: string[]` → `sources: { name, description? }[]`. Migration: [MIGRATION.md § Unreleased list_sources](docs/MIGRATION.md#unreleased-list_sources-response-shape).
+- **Breaking (library API):** `PineconeClient.listNamespacesWithMetadata()` now returns `{ namespaces, warnings }` instead of a bare array. Migration: [MIGRATION.md § Unreleased PineconeClient.listNamespacesWithMetadata](docs/MIGRATION.md#unreleased-pineconeclientlistnamespaceswithmetadata-return-shape).
 - **Instructions:** Trimmed operator/install/deploy content (env-var setup, misconfiguration note, Alliance CLI index/rerank defaults, stderr logging config) from `CORE_SERVER_INSTRUCTIONS` and `ALLIANCE_INSTRUCTIONS_APPENDIX` — reduces per-session token cost; no behavior change. Replaced colliding Alliance appendix steps 4–5 with unnumbered "Manual Alliance flow" bullets (includes `PINECONE_DISABLE_SUGGEST_FLOW=true` escape clause). Full detail remains in [docs/CONFIGURATION.md](docs/CONFIGURATION.md).
 
 ## [0.4.0] - 2026-06-24

benchmarks/latency.ts

@@ -233,7 +233,10 @@ function createBenchPineconeMock(): PineconeClient {
       return { count: 42, truncated: false };
     },
     async listNamespacesWithMetadata() {
-      return namespaces;
+      return {
+        namespaces: namespaces.map((n) => ({ ...n, schema_source: 'sampled' as const })),
+        warnings: [],
+      };
     },
     async listNamespacesFromKeywordIndex() {
       return namespaces.map((n) => ({ namespace: n.namespace, recordCount: n.recordCount }));

docs/CONFIGURATION.md

@@ -75,6 +75,17 @@ Set `PINECONE_CONFIG_FILE` (or `--config-file`) to a path such as [examples/mult
 
 Values support `${ENV_VAR}` indirection (resolved at startup). Per-source `sparseIndexName` and `rerankModel` are optional; Alliance defaults apply when omitted.
 
+Optional **config-file-only** fields (not supported in inline `PINECONE_SOURCES`):
+
+| Field | Scope | Purpose |
+| ----- | ----- | ------- |
+| `description` | Per source | Short corpus/content hint surfaced via `list_sources` (helps LLM routing across sources or MCP entries) |
+| `namespaces` | Per source | Map of namespace name → `{ description?, metadata_schema? }`; per-namespace `description` is surfaced via `list_namespaces` on matching live rows |
+
+`metadata_schema` is a flat `fieldName → type` map (same vocabulary as `list_namespaces` → `metadata_fields`, e.g. `"title": "string"`). When declared for a live namespace, the server **skips live sampling** for that namespace and trusts the declared schema until the config file changes. Namespaces declared in config but absent from Pinecone produce a non-fatal `config_warnings` entry in `list_namespaces` (never a startup failure).
+
+**Never** commit real corpus descriptions, namespace names, or internal field names to the open-source repo — use generic placeholders in examples only. Real values belong in staff-machine private config files per [Deployment profiles](#deployment-profiles). See [SECURITY.md](./SECURITY.md).
+
 ### MCP tools and routing
 
 | Tool | `source` parameter |
@@ -131,7 +142,7 @@ Multi-source mode supports two operational profiles. **Never** ship a merged int
 }
 ```
 
-Prefer `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over inline API keys in `PINECONE_SOURCES`. See [SECURITY.md](./SECURITY.md).
+Prefer `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over inline API keys in `PINECONE_SOURCES`. For internal deployments, add optional `description` and `namespaces` declarations in the JSON config file on staff machines only — never in public examples or committed constants. See [SECURITY.md](./SECURITY.md).
 
 ### Architecture decision
 

docs/MIGRATION.md

@@ -6,6 +6,56 @@ This guide is for **library and MCP client authors** upgrading from earlier **0.
 
 Under [semver 0.y.z](https://semver.org/spec/v2.0.0.html#spec-item-4), **0.1.x → 0.2.0 is a breaking minor** — pin `@0.2.0` only after reading this guide.
 
+## Unreleased: `list_sources` response shape
+
+**Who is affected:** MCP clients parsing `list_sources` success JSON in multi-source mode.
+
+**Before:**
+
+```json
+{
+  "status": "success",
+  "sources": ["api_key_1", "api_key_2"],
+  "default": "api_key_1"
+}
+```
+
+**After:**
+
+```json
+{
+  "status": "success",
+  "sources": [
+    { "name": "api_key_1", "description": "Optional corpus hint from private config" },
+    { "name": "api_key_2" }
+  ],
+  "default": "api_key_1"
+}
+```
+
+Read `sources[].name` instead of treating `sources` as `string[]`. `description` is omitted when not configured.
+
+## Unreleased: `PineconeClient.listNamespacesWithMetadata` return shape
+
+**Who is affected:** Direct library consumers of `PineconeClient` (not MCP tool clients — `list_namespaces` tool response shape is additive only).
+
+**Before:**
+
+```ts
+const rows = await client.listNamespacesWithMetadata();
+// rows: Array<{ namespace, recordCount, metadata }>
+```
+
+**After:**
+
+```ts
+const { namespaces, warnings } = await client.listNamespacesWithMetadata(declaredSchemas);
+// namespaces: Array<{ namespace, recordCount, metadata, schema_source }>
+// warnings: string[] — e.g. stale declared namespace not found live
+```
+
+Optional `declaredSchemas` argument skips live sampling for namespaces with a declared schema.
+
 ## Unreleased: Stable vs experimental response fields
 
 **Rationale:** Tool success payloads mixed stable contract fields with experimental diagnostics (`degraded`, `decision_trace`, etc.) at the top level. Experimental fields are now nested under `experimental` so consumers know which fields are safe across minor version bumps.

docs/SECURITY.md

@@ -23,6 +23,10 @@ Tool responses returned to MCP clients (and LLM consumers) are sanitized in `src
 
 This covers tool error payloads, hybrid degradation reasons, and SDK error text surfaced in DEBUG log mode — not only stderr logs.
 
+## Private config content (descriptions and schemas)
+
+Per-source `description` and per-namespace `namespaces` (including namespace names and `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles).
+
 ## Docker image
 
 The multi-stage [`Dockerfile`](../Dockerfile):

docs/TOOLS.md

@@ -8,8 +8,8 @@ Success payloads separate **stable** fields (safe across minor bumps after `1.0.
 
 | Tool | Stable | Experimental |
 | ---- | ------ | ------------ |
-| `list_sources` | `status`, `sources`, `default` | _(none)_ |
-| `list_namespaces` | `status`, `cache_hit`, `cache_ttl_seconds`, `expires_at_iso`, `count`, `namespaces`, optional `source_errors` | _(none)_ |
+| `list_sources` | `status`, `sources` (`{ name, description? }[]`), `default` | _(none)_ |
+| `list_namespaces` | `status`, `cache_hit`, `cache_ttl_seconds`, `expires_at_iso`, `count`, `namespaces`, optional `source_errors`, optional `config_warnings`, optional per-row `schema_source` | _(none)_ |
 | `namespace_router` | `status`, `cache_hit`, `user_query`, `suggestions`, `recommended_namespace`, optional `recommended_source` | _(none)_ |
 | `suggest_query_params` | `status`, `cache_hit`, `suggested_fields`, `recommended_tool`, `use_count_tool`, `explanation`, `namespace_found`, optional `source` | _(none)_ |
 | `count` | `status`, `count`, `truncated`, `namespace`, `metadata_filter`, optional `source` | _(none)_ |
@@ -73,7 +73,7 @@ Registered only when more than one Pinecone source is configured.
 | | |
 | --- | --- |
 | **Input** | _(empty object)_ |
-| **Success** | `{ status: 'success', sources: string[], default: string }` |
+| **Success** | `{ status: 'success', sources: { name, description? }[], default: string }` |
 | **Errors** | `LIFECYCLE` when not in multi-source mode |
 
 **Example:**
@@ -91,7 +91,7 @@ Registered only when more than one Pinecone source is configured.
 | | |
 | --- | --- |
 | **Input** | Optional `source` — filter to one configured project |
-| **Success** | `{ status: 'success', cache_hit, cache_ttl_seconds, expires_at_iso, count, namespaces: [{ name, record_count, metadata_fields, source? }], source_errors? }` |
+| **Success** | `{ status: 'success', cache_hit, cache_ttl_seconds, expires_at_iso, count, namespaces: [{ name, record_count, metadata_fields, source?, schema_source?, description? }], source_errors?, config_warnings? }` |
 | **Errors** | `PINECONE_ERROR`, `TIMEOUT`, etc. |
 
 **Example (multi-source, all projects):**

examples/alliance/demo-mock-pinecone-client.ts

@@ -10,6 +10,7 @@ import {
   type HybridQueryResult,
   type KeywordIndexNamespacesResult,
   type KeywordSearchParams,
+  type NamespacesWithMetadataResult,
   type PineconeMetadataValue,
   type QueryParams,
   type SearchResult,
@@ -36,21 +37,26 @@ export class DemoMockPineconeClient extends PineconeClient {
     super({ apiKey: '00000000-0000-0000-0000-000000000000', indexName: 'demo-index' });
   }
 
-  override async listNamespacesWithMetadata(): Promise<
-    Array<{ namespace: string; recordCount: number; metadata: Record<string, string> }>
-  > {
-    return [
-      {
-        namespace: DEMO_NAMESPACE,
-        recordCount: 42,
-        metadata: {
-          document_number: 'string',
-          title: 'string',
-          chunk_text: 'string',
-          url: 'string',
+  override async listNamespacesWithMetadata(
+    _declaredSchemas?: Record<string, Record<string, string>>,
+    _declaredNamespaceNames?: string[]
+  ): Promise<NamespacesWithMetadataResult> {
+    return {
+      namespaces: [
+        {
+          namespace: DEMO_NAMESPACE,
+          recordCount: 42,
+          metadata: {
+            document_number: 'string',
+            title: 'string',
+            chunk_text: 'string',
+            url: 'string',
+          },
+          schema_source: 'sampled',
         },
-      },
-    ];
+      ],
+      warnings: [],
+    };
   }
 
   override async listNamespacesFromKeywordIndex(): Promise<KeywordIndexNamespacesResult> {

examples/multi-source/pinecone-sources.json.example

@@ -3,7 +3,17 @@
   "sources": {
     "api_key_1": {
       "apiKey": "${PINECONE_API_KEY_1}",
-      "indexName": "index_name_1"
+      "indexName": "index_name_1",
+      "description": "<optional: describe this corpus in your PRIVATE staff config, not here>",
+      "namespaces": {
+        "example-namespace": {
+          "description": "<optional: describe this namespace in your PRIVATE staff config, not here>",
+          "metadata_schema": {
+            "field_a": "string",
+            "field_b": "number"
+          }
+        }
+      }
     },
     "api_key_2": {
       "apiKey": "${PINECONE_API_KEY_2}",

scripts/test-search.ts

@@ -36,22 +36,27 @@ async function test() {
     // Test 1: List namespaces with metadata
     console.log('\n📋 Test 1: Listing namespaces with metadata...');
     const namespacesInfo = await client.listNamespacesWithMetadata();
-    console.log(`✅ Found ${namespacesInfo.length} namespace(s):`);
-    namespacesInfo.forEach((ns) => {
+    console.log(`✅ Found ${namespacesInfo.namespaces.length} namespace(s):`);
+    namespacesInfo.namespaces.forEach((ns) => {
       console.log(`   - ${ns.namespace}: ${ns.recordCount} records`);
       if (Object.keys(ns.metadata).length > 0) {
         console.log(`     Metadata fields:`, Object.keys(ns.metadata));
       }
     });
 
-    if (namespacesInfo.length === 0) {
+    if (namespacesInfo.warnings.length > 0) {
+      console.log('⚠️  Config warnings:');
+      namespacesInfo.warnings.forEach((warning) => console.log(`   - ${warning}`));
+    }
+
+    if (namespacesInfo.namespaces.length === 0) {
       console.log('⚠️  No namespaces found in your index');
       console.log('   Make sure your Pinecone index has data');
       return;
     }
 
     // Test 2: Query without reranking (faster)
-    const testNamespace = namespacesInfo[0].namespace;
+    const testNamespace = namespacesInfo.namespaces[0].namespace;
     console.log(`\n🔍 Test 2: Query WITHOUT reranking (faster)`);
     console.log(`   Namespace: "${testNamespace}"`);
     console.log(`   Query: "test query"`);
@@ -95,7 +100,7 @@ async function test() {
     }
 
     // Test 4: Query with metadata filter on wg21-papers namespace
-    const wg21Namespace = namespacesInfo.find((ns) => ns.namespace === 'wg21-papers');
+    const wg21Namespace = namespacesInfo.namespaces.find((ns) => ns.namespace === 'wg21-papers');
     let duration3: number | undefined;
 
     if (wg21Namespace) {
@@ -149,7 +154,7 @@ async function test() {
     } else {
       console.log(`\n⚠️  Test 4 skipped: "wg21-papers" namespace not found`);
       console.log(
-        `   Available namespaces: ${namespacesInfo.map((ns) => ns.namespace).join(', ')}`
+        `   Available namespaces: ${namespacesInfo.namespaces.map((ns) => ns.namespace).join(', ')}`
       );
     }
 

src/alliance/tools/isolated-context.context.test.ts

@@ -7,6 +7,7 @@ import {
   isolateFromDefaultContext,
   makeHybridQueryResult,
   makeSearchResult,
+  mockNamespacesWithMetadataResult,
   parseToolJson,
 } from '../../core/server/tools/test-helpers.js';
 import {
@@ -27,18 +28,20 @@ describe('isolated ServerContext with zero default context', () => {
   });
 
   it('guided_query enrich_urls uses ctx builtins, not default registry', async () => {
-    const listNamespacesWithMetadata = vi.fn().mockResolvedValue([
-      {
-        namespace: 'mailing',
-        recordCount: 42,
-        metadata: {
-          document_number: 'string',
-          title: 'string',
-          author: 'string',
-          chunk_text: 'string',
+    const listNamespacesWithMetadata = vi.fn().mockResolvedValue(
+      mockNamespacesWithMetadataResult([
+        {
+          namespace: 'mailing',
+          recordCount: 42,
+          metadata: {
+            document_number: 'string',
+            title: 'string',
+            author: 'string',
+            chunk_text: 'string',
+          },
         },
-      },
-    ]);
+      ])
+    );
     const query = vi.fn().mockResolvedValue(
       makeHybridQueryResult({
         results: [