Load per-source schema manifests from Pinecone _mcp_config at startup

CHANGELOG.md

@@ -10,7 +10,8 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release
 
 ### Added
 
-- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE` only — not inline `PINECONE_SOURCES`). Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md](docs/CONFIGURATION.md#json-config-file).
+- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE`), or loaded automatically from the `_mcp_config` Pinecone namespace when not declared locally. Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md § Multi-source mode](docs/CONFIGURATION.md#multi-source-mode).
+- **Remote `_mcp_config` schema manifest:** automatic per-source loading of description and namespace declarations from Pinecone at startup (opt out via `PINECONE_DISABLE_REMOTE_SCHEMA` / `--disable-remote-schema`); non-fatal on failure.
 - **`list_namespaces`:** optional per-namespace `schema_source` (`declared` | `sampled`), optional per-namespace `description` (from private config), and top-level `config_warnings` when private config declarations do not match live Pinecone data.
 - **Tests:** MCP verification harness ([src/__tests__/mcp-rc-readiness.test.ts](src/__tests__/mcp-rc-readiness.test.ts)) that drives the real server over an in-memory transport with the SDK client (initialize and protocol negotiation, the full registered tool surface, and a round-trip tool call), so a future SDK or MCP protocol bump is verified in one place. Protocol assertions key off the SDK's `LATEST_PROTOCOL_VERSION`, so pinning the RC SDK re-runs the check with no test edits. (#202)
 

docs/SECURITY.md

@@ -25,7 +25,7 @@ This covers tool error payloads, hybrid degradation reasons, and SDK error text
 
 ## Private config content (descriptions and schemas)
 
-Per-source `description` and per-namespace `namespaces` (including namespace names and `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles).
+Per-source `description` and per-namespace `namespaces` (including namespace names and `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` or the `_mcp_config` schema manifest are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Manifests in `_mcp_config` inherit the same access boundary as the rest of that Pinecone index; do not publish sensitive descriptions on indexes shared more broadly than the private data warrants. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles).
 
 ## Docker image
 

docs/TOOLS.md

@@ -64,6 +64,19 @@ Or single-shot: `guided_query` (routes across sources when `namespace` is omitte
 
 **Suggest-flow in multi-source mode:** gate state uses compound keys `source:namespace`. Pass the same `source` + `namespace` pair for `suggest_query_params` and gated tools when the namespace exists on multiple sources.
 
+### Design notes
+
+**Chosen:** multi-source server with optional `source` parameter on tools (`SourceRegistry` + `PINECONE_SOURCES` / JSON config), over:
+
+- **Cursor routing rules only:** doesn't fix the UX problem when users forget which MCP entry to use; no code changes.
+- **Thin proxy MCP:** extra package and latency; duplicates routing logic already in `ServerContext`.
+
+**Rationale:** Pinecone SDK v8 supports multiple `Pinecone({ apiKey })` instances per process; MCP has no barrier to aggregating backends. Security is enforced by **deployment profiles** (public-only vs merged config), not per-query MCP authorization. All multi-source results include `source` for LLM provenance.
+
+**Execution semantics:** discovery tools aggregate all sources when `source` is omitted. Execution tools (`query`, `count`, etc.) call `resolveSource`: infer source when the namespace exists on exactly one project; return `VALIDATION` when ambiguous. They do **not** fan out one query to all sources. `guided_query` without `namespace` routes via the aggregated namespace list and sets `selected_source` in `decision_trace`.
+
+**Audit logging:** when a tool resolves a specific source, stderr logs `toolname [source=name]` at INFO (execution tools and discovery tools that pass an explicit `source` filter). Aggregated discovery without `source` does not log per-source lines.
+
 ---
 
 ## `list_sources` (multi-source only)

src/cli.ts

@@ -117,6 +117,9 @@ export function parseCli(argv: string[] = process.argv.slice(2)): ParseCliResult
       case '--check-indexes':
         overrides.checkIndexes = true;
         break;
+      case '--disable-remote-schema':
+        overrides.disableRemoteSchema = true;
+        break;
       case '--sources': {
         const v = readOptionValue(argv, i);
         if (v !== undefined) {
@@ -160,7 +163,8 @@ Options:
   --request-timeout-ms N      Per Pinecone call timeout [env: PINECONE_REQUEST_TIMEOUT_MS]
   --disable-suggest-flow      Bypass suggest_query_params gate (PINECONE_DISABLE_SUGGEST_FLOW)
   --check-indexes             Verify dense + sparse indexes then exit 0/1 (PINECONE_CHECK_INDEXES)
-  --sources TEXT              Multi-source inline config (PINECONE_SOURCES)
+  --disable-remote-schema     Skip loading schema manifest from _mcp_config (PINECONE_DISABLE_REMOTE_SCHEMA)
+  --sources TEXT              Multi-source inline config: name:apiKey:indexName;... (PINECONE_SOURCES)
   --config-file PATH          Multi-source JSON config file (PINECONE_CONFIG_FILE)
   --help, -h                  Show this message
   --version, -v               Print package version

src/core/config.ts

@@ -57,6 +57,8 @@ export interface ServerConfigBase {
   disableSuggestFlow: boolean;
   /** When true, on-startup probe verifies dense + sparse indexes exist. */
   checkIndexes: boolean;
+  /** When true, skip loading schema manifest from `_mcp_config` at startup. */
+  disableRemoteSchema: boolean;
   /** Named Pinecone sources when multi-project mode is enabled. */
   sources?: SourceDefinition[];
   /** Default source name when multi-source mode is enabled. */
@@ -150,6 +152,7 @@ export interface ConfigOverrides {
   requestTimeoutMs?: number;
   disableSuggestFlow?: boolean;
   checkIndexes?: boolean;
+  disableRemoteSchema?: boolean;
   sources?: string;
   configFile?: string;
 }
@@ -196,6 +199,8 @@ export function resolveConfig(
   const disableSuggestFlow =
     overrides.disableSuggestFlow ?? asBool(env['PINECONE_DISABLE_SUGGEST_FLOW'], true);
   const checkIndexes = overrides.checkIndexes ?? asBool(env['PINECONE_CHECK_INDEXES'], false);
+  const disableRemoteSchema =
+    overrides.disableRemoteSchema ?? asBool(env['PINECONE_DISABLE_REMOTE_SCHEMA'], false);
 
   const multi = resolveSourceDefinitions(overrides, env, parseSourcesOptions);
   if (multi) {
@@ -217,6 +222,7 @@ export function resolveConfig(
       requestTimeoutMs,
       disableSuggestFlow,
       checkIndexes,
+      disableRemoteSchema,
       sources: multi.sources,
       defaultSource: multi.defaultSource,
     });
@@ -254,6 +260,7 @@ export function resolveConfig(
     requestTimeoutMs,
     disableSuggestFlow,
     checkIndexes,
+    disableRemoteSchema,
   });
 }
 

src/core/pinecone-client.ts

@@ -90,6 +90,11 @@ export class PineconeClient {
     return this.indexSession.checkIndexes();
   }
 
+  /** Fetch record fields from the dense index (metadata + top-level scalars). */
+  async fetchRecordFields(namespace: string, id: string): Promise<Record<string, unknown> | null> {
+    return this.indexSession.fetchRecordFields(namespace, id);
+  }
+
   private async searchIndex(
     index: SearchableIndex,
     query: string,

src/core/pinecone/indexes.test.ts

@@ -270,4 +270,62 @@ describe('PineconeIndexSession', () => {
       expect(result.errors.some((e) => e.includes('no client'))).toBe(true);
     });
   });
+
+  describe('fetchRecordFields', () => {
+    class FetchFieldsSession extends PineconeIndexSession {
+      constructor(
+        private readonly fetchResponse: {
+          records?: Record<string, { metadata?: Record<string, unknown>; [key: string]: unknown }>;
+        }
+      ) {
+        super('test-api-key', 'test-index');
+      }
+
+      override ensureClient() {
+        return {
+          index: () => ({
+            fetch: vi.fn().mockResolvedValue(this.fetchResponse),
+          }),
+        } as never;
+      }
+    }
+
+    it('merges metadata and top-level scalar fields', async () => {
+      const session = new FetchFieldsSession({
+        records: {
+          schema_manifest: {
+            id: 'schema_manifest',
+            chunk_text: '{"ok":true}',
+            metadata: { extra: 'meta' },
+          },
+        },
+      });
+
+      const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest');
+      expect(fields).toMatchObject({
+        extra: 'meta',
+        chunk_text: '{"ok":true}',
+        id: 'schema_manifest',
+      });
+    });
+
+    it('returns null when record is missing', async () => {
+      const session = new FetchFieldsSession({ records: {} });
+      const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest');
+      expect(fields).toBeNull();
+    });
+
+    it('returns metadata-only fields when chunk_text is only in metadata', async () => {
+      const session = new FetchFieldsSession({
+        records: {
+          schema_manifest: {
+            metadata: { chunk_text: '{"from":"metadata"}' },
+          },
+        },
+      });
+
+      const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest');
+      expect(fields?.chunk_text).toBe('{"from":"metadata"}');
+    });
+  });
 });

src/core/pinecone/indexes.ts

@@ -229,6 +229,29 @@ export class PineconeIndexSession {
     }
   }
 
+  /**
+   * Fetch a record by id and return a flat field bag (metadata merged with top-level scalar fields).
+   * Mirrors Python `_extract_record_fields` for integrated-embedding indexes where `chunk_text`
+   * may appear on the record or inside metadata.
+   */
+  async fetchRecordFields(namespace: string, id: string): Promise<Record<string, unknown> | null> {
+    const pc = this.ensureClient();
+    const response = await pc.index(this.indexName).fetch({ ids: [id], namespace });
+    const record = response.records?.[id] as
+      (Record<string, unknown> & { metadata?: Record<string, unknown> }) | undefined;
+    if (!record) {
+      return null;
+    }
+    const metadata = record.metadata ?? {};
+    const merged: Record<string, unknown> = { ...metadata };
+    for (const [k, v] of Object.entries(record)) {
+      if (k !== 'metadata' && k !== 'values' && k !== 'sparseValues' && !(k in merged)) {
+        merged[k] = v;
+      }
+    }
+    return merged;
+  }
+
   /**
    * Verify dense and sparse indexes are reachable (describeIndexStats).
    * Used by `--check-indexes` / `PINECONE_CHECK_INDEXES` before the server starts.

src/core/server/remote-schema.test.ts

@@ -0,0 +1,163 @@
+import { describe, expect, it, vi } from 'vitest';
+import type { PineconeClient } from '../pinecone-client.js';
+import type { SourceDefinition } from './source-config.js';
+import {
+  SCHEMA_MANIFEST_NAMESPACE,
+  SCHEMA_MANIFEST_RECORD_ID,
+  loadRemoteSchemaForSource,
+  loadRemoteSchemaForSources,
+} from './remote-schema.js';
+
+function baseDefinition(overrides?: Partial<SourceDefinition>): SourceDefinition {
+  return {
+    name: 'api_key_1',
+    apiKey: 'key',
+    indexName: 'rag-hybrid',
+    sparseIndexName: 'rag-hybrid-sparse',
+    ...overrides,
+  };
+}
+
+function mockClient(
+  fetchImpl: (namespace: string, id: string) => Promise<Record<string, unknown> | null>
+): PineconeClient {
+  return {
+    fetchRecordFields: vi.fn(fetchImpl),
+  } as unknown as PineconeClient;
+}
+
+describe('remote-schema', () => {
+  it('merges description and namespaces from chunk_text in metadata', async () => {
+    const manifest = {
+      description: 'Corpus hint',
+      namespaces: {
+        mailing: {
+          description: 'Mailing threads',
+          metadata_schema: { doc_id: 'string' },
+          record_count: 100,
+        },
+      },
+    };
+    const client = mockClient(async () => ({
+      chunk_text: JSON.stringify(manifest),
+    }));
+
+    const { definition, warning } = await loadRemoteSchemaForSource(client, baseDefinition());
+
+    expect(warning).toBeUndefined();
+    expect(definition.description).toBe('Corpus hint');
+    expect(definition.namespaces?.mailing).toMatchObject({
+      description: 'Mailing threads',
+      metadata_schema: { doc_id: 'string' },
+    });
+    expect(client.fetchRecordFields).toHaveBeenCalledWith(
+      SCHEMA_MANIFEST_NAMESPACE,
+      SCHEMA_MANIFEST_RECORD_ID
+    );
+  });
+
+  it('reads chunk_text when returned as a top-level field', async () => {
+    const manifest = {
+      namespaces: {
+        docs: { metadata_schema: { title: 'string' } },
+      },
+    };
+    const client = mockClient(async () => ({
+      id: SCHEMA_MANIFEST_RECORD_ID,
+      chunk_text: JSON.stringify(manifest),
+    }));
+
+    const { definition } = await loadRemoteSchemaForSource(client, baseDefinition());
+
+    expect(definition.namespaces?.docs?.metadata_schema).toEqual({ title: 'string' });
+  });
+
+  it('returns unchanged when record is missing (no warning)', async () => {
+    const client = mockClient(async () => null);
+    const input = baseDefinition();
+
+    const { definition, warning } = await loadRemoteSchemaForSource(client, input);
+
+    expect(warning).toBeUndefined();
+    expect(definition).toEqual(input);
+  });
+
+  it('returns warning on malformed JSON in chunk_text', async () => {
+    const client = mockClient(async () => ({ chunk_text: '{not json' }));
+    const input = baseDefinition();
+
+    const { definition, warning } = await loadRemoteSchemaForSource(client, input);
+
+    expect(definition).toEqual(input);
+    expect(warning).toMatch(/malformed \(invalid JSON/);
+  });
+
+  it('returns warning when chunk_text is valid JSON but namespaces has the wrong shape', async () => {
+    const client = mockClient(async () => ({
+      chunk_text: JSON.stringify({ namespaces: ['not', 'an', 'object'] }),
+    }));
+    const input = baseDefinition();
+
+    const { definition, warning } = await loadRemoteSchemaForSource(client, input);
+
+    expect(definition).toEqual(input);
+    expect(warning).toMatch(/malformed \(manifest "namespaces" must be an object\)/);
+  });
+
+  it('returns warning when fetch throws', async () => {
+    const client = mockClient(async () => {
+      throw new Error('network down');
+    });
+    const input = baseDefinition();
+
+    const { definition, warning } = await loadRemoteSchemaForSource(client, input);
+
+    expect(definition).toEqual(input);
+    expect(warning).toMatch(/failed to fetch/);
+  });
+
+  it('skips fetch when local namespaces are already set', async () => {
+    const fetchFn = vi.fn(async () => ({ chunk_text: '{}' }));
+    const client = mockClient(fetchFn);
+    const input = baseDefinition({
+      namespaces: { local_ns: { description: 'local' } },
+    });
+
+    const { definition, warning } = await loadRemoteSchemaForSource(client, input);
+
+    expect(fetchFn).not.toHaveBeenCalled();
+    expect(warning).toBeUndefined();
+    expect(definition).toEqual(input);
+  });
+
+  it('keeps local description when merging remote namespaces', async () => {
+    const client = mockClient(async () => ({
+      chunk_text: JSON.stringify({
+        description: 'remote description',
+        namespaces: { ns1: { metadata_schema: { a: 'string' } } },
+      }),
+    }));
+    const input = baseDefinition({ description: 'local description' });
+
+    const { definition } = await loadRemoteSchemaForSource(client, input);
+
+    expect(definition.description).toBe('local description');
+    expect(definition.namespaces?.ns1?.metadata_schema).toEqual({ a: 'string' });
+  });
+
+  it('loadRemoteSchemaForSources processes multiple entries', async () => {
+    const client1 = mockClient(async () => ({
+      chunk_text: JSON.stringify({ description: 'source one' }),
+    }));
+    const client2 = mockClient(async () => null);
+
+    const result = await loadRemoteSchemaForSources([
+      { definition: baseDefinition({ name: 's1' }), client: client1 },
+      { definition: baseDefinition({ name: 's2' }), client: client2 },
+    ]);
+
+    expect(result.definitions).toHaveLength(2);
+    expect(result.definitions[0]?.description).toBe('source one');
+    expect(result.warnings).toHaveLength(0);
+  });
+});