Load per-source schema manifests from Pinecone _mcp_config at startup

CHANGELOG.md

@@ -10,7 +10,8 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release
 
 ### Added
 
-- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE` only — not inline `PINECONE_SOURCES`). Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md](docs/CONFIGURATION.md#json-config-file).
+- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE`), or loaded automatically from the `_mcp_config` Pinecone namespace when not declared locally. Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md § Multi-source mode](docs/CONFIGURATION.md#multi-source-mode).
+- **Remote `_mcp_config` schema manifest:** automatic per-source loading of description and namespace declarations from Pinecone at startup (opt out via `PINECONE_DISABLE_REMOTE_SCHEMA` / `--disable-remote-schema`); non-fatal on failure.
 - **`list_namespaces`:** optional per-namespace `schema_source` (`declared` | `sampled`), optional per-namespace `description` (from private config), and top-level `config_warnings` when private config declarations do not match live Pinecone data.
 - **Tests:** MCP verification harness ([src/__tests__/mcp-rc-readiness.test.ts](src/__tests__/mcp-rc-readiness.test.ts)) that drives the real server over an in-memory transport with the SDK client (initialize and protocol negotiation, the full registered tool surface, and a round-trip tool call), so a future SDK or MCP protocol bump is verified in one place. Protocol assertions key off the SDK's `LATEST_PROTOCOL_VERSION`, so pinning the RC SDK re-runs the check with no test edits. (#202)
 

docs/SECURITY.md

@@ -25,7 +25,7 @@ This covers tool error payloads, hybrid degradation reasons, and SDK error text
 
 ## Private config content (descriptions and schemas)
 
-Per-source `description` and per-namespace `namespaces` (including namespace names and `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles).
+Per-source `description` and per-namespace `namespaces` (including namespace names and `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` or the `_mcp_config` schema manifest are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Manifests in `_mcp_config` inherit the same access boundary as the rest of that Pinecone index; do not publish sensitive descriptions on indexes shared more broadly than the private data warrants. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles).
 
 ## Docker image
 

docs/TOOLS.md

@@ -64,6 +64,19 @@ Or single-shot: `guided_query` (routes across sources when `namespace` is omitte
 
 **Suggest-flow in multi-source mode:** gate state uses compound keys `source:namespace`. Pass the same `source` + `namespace` pair for `suggest_query_params` and gated tools when the namespace exists on multiple sources.
 
+### Design notes
+
+**Chosen:** multi-source server with optional `source` parameter on tools (`SourceRegistry` + `PINECONE_SOURCES` / JSON config), over:
+
+- **Cursor routing rules only:** doesn't fix the UX problem when users forget which MCP entry to use; no code changes.
+- **Thin proxy MCP:** extra package and latency; duplicates routing logic already in `ServerContext`.
+
+**Rationale:** Pinecone SDK v8 supports multiple `Pinecone({ apiKey })` instances per process; MCP has no barrier to aggregating backends. Security is enforced by **deployment profiles** (public-only vs merged config), not per-query MCP authorization. All multi-source results include `source` for LLM provenance.
+
+**Execution semantics:** discovery tools aggregate all sources when `source` is omitted. Execution tools (`query`, `count`, etc.) call `resolveSource`: infer source when the namespace exists on exactly one project; return `VALIDATION` when ambiguous. They do **not** fan out one query to all sources. `guided_query` without `namespace` routes via the aggregated namespace list and sets `selected_source` in `decision_trace`.
+
+**Audit logging:** when a tool resolves a specific source, stderr logs `toolname [source=name]` at INFO (execution tools and discovery tools that pass an explicit `source` filter). Aggregated discovery without `source` does not log per-source lines.
+
 ---
 
 ## `list_sources` (multi-source only)

src/cli.test.ts

@@ -36,4 +36,12 @@ describe('parseCli', () => {
       expect(r.overrides.defaultTopK).toBe(25);
     }
   });
+
+  it('parses --disable-remote-schema', () => {
+    const r = parseCli(['--disable-remote-schema']);
+    expect(r.kind).toBe('config');
+    if (r.kind === 'config') {
+      expect(r.overrides.disableRemoteSchema).toBe(true);
+    }
+  });
 });

src/cli.ts

@@ -117,6 +117,9 @@ export function parseCli(argv: string[] = process.argv.slice(2)): ParseCliResult
       case '--check-indexes':
         overrides.checkIndexes = true;
         break;
+      case '--disable-remote-schema':
+        overrides.disableRemoteSchema = true;
+        break;
       case '--sources': {
         const v = readOptionValue(argv, i);
         if (v !== undefined) {
@@ -160,7 +163,8 @@ Options:
   --request-timeout-ms N      Per Pinecone call timeout [env: PINECONE_REQUEST_TIMEOUT_MS]
   --disable-suggest-flow      Bypass suggest_query_params gate (PINECONE_DISABLE_SUGGEST_FLOW)
   --check-indexes             Verify dense + sparse indexes then exit 0/1 (PINECONE_CHECK_INDEXES)
-  --sources TEXT              Multi-source inline config (PINECONE_SOURCES)
+  --disable-remote-schema     Skip loading schema manifest from _mcp_config (PINECONE_DISABLE_REMOTE_SCHEMA)
+  --sources TEXT              Multi-source inline config: name:apiKey:indexName;... (PINECONE_SOURCES)
   --config-file PATH          Multi-source JSON config file (PINECONE_CONFIG_FILE)
   --help, -h                  Show this message
   --version, -v               Print package version

src/core/config.ts

@@ -57,6 +57,8 @@ export interface ServerConfigBase {
   disableSuggestFlow: boolean;
   /** When true, on-startup probe verifies dense + sparse indexes exist. */
   checkIndexes: boolean;
+  /** When true, skip loading schema manifest from `_mcp_config` at startup. */
+  disableRemoteSchema: boolean;
   /** Named Pinecone sources when multi-project mode is enabled. */
   sources?: SourceDefinition[];
   /** Default source name when multi-source mode is enabled. */
@@ -150,6 +152,7 @@ export interface ConfigOverrides {
   requestTimeoutMs?: number;
   disableSuggestFlow?: boolean;
   checkIndexes?: boolean;
+  disableRemoteSchema?: boolean;
   sources?: string;
   configFile?: string;
 }
@@ -196,6 +199,8 @@ export function resolveConfig(
   const disableSuggestFlow =
     overrides.disableSuggestFlow ?? asBool(env['PINECONE_DISABLE_SUGGEST_FLOW'], true);
   const checkIndexes = overrides.checkIndexes ?? asBool(env['PINECONE_CHECK_INDEXES'], false);
+  const disableRemoteSchema =
+    overrides.disableRemoteSchema ?? asBool(env['PINECONE_DISABLE_REMOTE_SCHEMA'], false);
 
   const multi = resolveSourceDefinitions(overrides, env, parseSourcesOptions);
   if (multi) {
@@ -217,6 +222,7 @@ export function resolveConfig(
       requestTimeoutMs,
       disableSuggestFlow,
       checkIndexes,
+      disableRemoteSchema,
       sources: multi.sources,
       defaultSource: multi.defaultSource,
     });
@@ -254,6 +260,7 @@ export function resolveConfig(
     requestTimeoutMs,
     disableSuggestFlow,
     checkIndexes,
+    disableRemoteSchema,
   });
 }
 

src/core/pinecone-client.test.ts

@@ -69,6 +69,21 @@ describe('PineconeClient', () => {
     });
   });
 
+  describe('fetchRecordFields', () => {
+    it('delegates to indexSession.fetchRecordFields', async () => {
+      const fields = { chunk_text: '{"ok":true}' };
+      const fetchMock = vi.fn().mockResolvedValue(fields);
+      Object.assign(client, {
+        indexSession: { fetchRecordFields: fetchMock },
+      });
+
+      const result = await client.fetchRecordFields('_mcp_config', 'schema_manifest');
+
+      expect(fetchMock).toHaveBeenCalledWith('_mcp_config', 'schema_manifest');
+      expect(result).toEqual(fields);
+    });
+  });
+
   describe('query', () => {
     it('should throw error for empty query', async () => {
       await expect(

src/core/pinecone-client.ts

@@ -15,6 +15,7 @@ import type {
   HybridLegFailed,
 } from '../types.js';
 import { DEFAULT_TOP_K, MAX_TOP_K, COUNT_TOP_K, COUNT_FIELDS } from '../constants.js';
+import { DEFAULT_REQUEST_TIMEOUT_MS } from './config.js';
 import { PineconeIndexSession, type NamespacesWithMetadataResult } from './pinecone/indexes.js';
 import {
   countUniqueDocumentsFromHits,
@@ -39,7 +40,8 @@ export class PineconeClient {
     this.indexSession = new PineconeIndexSession(
       config.apiKey,
       config.indexName,
-      config.sparseIndexName
+      config.sparseIndexName,
+      config.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS
     );
     const normalizedRerankModel = config.rerankModel?.trim();
     this.rerankModel = normalizedRerankModel ? normalizedRerankModel : undefined;
@@ -90,6 +92,11 @@ export class PineconeClient {
     return this.indexSession.checkIndexes();
   }
 
+  /** Fetch record fields from the dense index (metadata + top-level scalars). */
+  async fetchRecordFields(namespace: string, id: string): Promise<Record<string, unknown> | null> {
+    return this.indexSession.fetchRecordFields(namespace, id);
+  }
+
   private async searchIndex(
     index: SearchableIndex,
     query: string,

src/core/pinecone/indexes.test.ts

@@ -270,4 +270,83 @@ describe('PineconeIndexSession', () => {
       expect(result.errors.some((e) => e.includes('no client'))).toBe(true);
     });
   });
+
+  describe('fetchRecordFields', () => {
+    class FetchFieldsSession extends PineconeIndexSession {
+      constructor(
+        private readonly fetchResponse: {
+          records?: Record<string, { metadata?: Record<string, unknown>; [key: string]: unknown }>;
+        }
+      ) {
+        super('test-api-key', 'test-index');
+      }
+
+      override ensureClient() {
+        return {
+          index: () => ({
+            fetch: vi.fn().mockResolvedValue(this.fetchResponse),
+          }),
+        } as never;
+      }
+    }
+
+    it('merges metadata and top-level scalar fields', async () => {
+      const session = new FetchFieldsSession({
+        records: {
+          schema_manifest: {
+            id: 'schema_manifest',
+            chunk_text: '{"ok":true}',
+            metadata: { extra: 'meta' },
+          },
+        },
+      });
+
+      const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest');
+      expect(fields).toMatchObject({
+        extra: 'meta',
+        chunk_text: '{"ok":true}',
+        id: 'schema_manifest',
+      });
+    });
+
+    it('returns null when record is missing', async () => {
+      const session = new FetchFieldsSession({ records: {} });
+      const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest');
+      expect(fields).toBeNull();
+    });
+
+    it('returns metadata-only fields when chunk_text is only in metadata', async () => {
+      const session = new FetchFieldsSession({
+        records: {
+          schema_manifest: {
+            metadata: { chunk_text: '{"from":"metadata"}' },
+          },
+        },
+      });
+
+      const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest');
+      expect(fields?.chunk_text).toBe('{"from":"metadata"}');
+    });
+
+    it('rejects when fetch exceeds requestTimeoutMs', async () => {
+      class HangingFetchSession extends PineconeIndexSession {
+        constructor() {
+          super('test-api-key', 'test-index', undefined, 50);
+        }
+
+        override ensureClient() {
+          return {
+            index: () => ({
+              fetch: () => new Promise(() => {}),
+            }),
+          } as never;
+        }
+      }
+
+      const session = new HangingFetchSession();
+      await expect(session.fetchRecordFields('_mcp_config', 'schema_manifest')).rejects.toThrow(
+        /Timeout after 50ms while waiting for fetchRecordFields/
+      );
+    });
+  });
 });

src/core/pinecone/indexes.ts

@@ -3,6 +3,8 @@
  */
 
 import { Pinecone } from '@pinecone-database/pinecone';
+import { DEFAULT_REQUEST_TIMEOUT_MS } from '../config.js';
+import { withTimeout } from '../server/retry.js';
 import { error as logError, info as logInfo } from '../../logger.js';
 import type {
   KeywordIndexNamespacesResult,
@@ -46,7 +48,8 @@ export class PineconeIndexSession {
   constructor(
     private readonly apiKey: string,
     private readonly indexName: string,
-    private readonly sparseIndexName?: string
+    private readonly sparseIndexName?: string,
+    private readonly requestTimeoutMs: number = DEFAULT_REQUEST_TIMEOUT_MS
   ) {}
 
   /** Sparse index name; defaults to `{indexName}-sparse`. */
@@ -229,6 +232,34 @@ export class PineconeIndexSession {
     }
   }
 
+  /**
+   * Fetch a record by id and return a flat field bag (metadata merged with top-level scalar fields).
+   * Mirrors Python `_extract_record_fields` for integrated-embedding indexes where `chunk_text`
+   * may appear on the record or inside metadata.
+   */
+  async fetchRecordFields(namespace: string, id: string): Promise<Record<string, unknown> | null> {
+    return withTimeout(
+      async (_signal) => {
+        const pc = this.ensureClient();
+        const response = await pc.index(this.indexName).fetch({ ids: [id], namespace });
+        const record = response.records?.[id] as
+          (Record<string, unknown> & { metadata?: Record<string, unknown> }) | undefined;
+        if (!record) {
+          return null;
+        }
+        const metadata = record.metadata ?? {};
+        const merged: Record<string, unknown> = { ...metadata };
+        for (const [k, v] of Object.entries(record)) {
+          if (k !== 'metadata' && k !== 'values' && k !== 'sparseValues' && !(k in merged)) {
+            merged[k] = v;
+          }
+        }
+        return merged;
+      },
+      { timeoutMs: this.requestTimeoutMs, label: 'fetchRecordFields' }
+    );
+  }
+
   /**
    * Verify dense and sparse indexes are reachable (describeIndexStats).
    * Used by `--check-indexes` / `PINECONE_CHECK_INDEXES` before the server starts.