From 27feb692182bf6c5a3dbd8a96930287a41569b32 Mon Sep 17 00:00:00 2001 From: zho Date: Tue, 14 Jul 2026 00:19:57 +0800 Subject: [PATCH 1/8] Support JSON-shaped PINECONE_SOURCES with descriptions and namespaces --- CHANGELOG.md | 3 +- docs/CONFIGURATION.md | 66 +++++++++++++--- docs/SECURITY.md | 2 +- src/cli.ts | 2 +- src/core/server/source-config.test.ts | 94 ++++++++++++++++++++++ src/core/server/source-config.ts | 108 ++++++++++++++++++-------- 6 files changed, 228 insertions(+), 47 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ad9a2d3..ad6fd2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,8 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release ### Added -- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE` only — not inline `PINECONE_SOURCES`). Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md](docs/CONFIGURATION.md#json-config-file). +- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE`) or JSON-shaped `PINECONE_SOURCES` (not the colon-delimited inline format). Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md](docs/CONFIGURATION.md#json-config-file). +- **JSON-shaped `PINECONE_SOURCES`:** inline rich multi-source config (descriptions, namespaces, optional `defaultSource`) in MCP `env` or `--sources` without a separate config file; `apiKey` defaults to `${sourceName}` when omitted. - **`list_namespaces`:** optional per-namespace `schema_source` (`declared` | `sampled`), optional per-namespace `description` (from private config), and top-level `config_warnings` when private config declarations do not match live Pinecone data. - **Tests:** MCP verification harness ([src/__tests__/mcp-rc-readiness.test.ts](src/__tests__/mcp-rc-readiness.test.ts)) that drives the real server over an in-memory transport with the SDK client (initialize and protocol negotiation, the full registered tool surface, and a round-trip tool call), so a future SDK or MCP protocol bump is verified in one place. Protocol assertions key off the SDK's `LATEST_PROTOCOL_VERSION`, so pinning the RC SDK re-runs the check with no test edits. (#202) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index a107c22..09c85d1 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -49,7 +49,7 @@ C++ Alliance deployers can copy [examples/alliance/.env.example](../examples/all Use **one MCP server entry** with multiple Pinecone API keys / projects when `PINECONE_SOURCES`, `--sources`, or a JSON config file (`PINECONE_CONFIG_FILE` / `--config-file`) is set. -### Inline format (`PINECONE_SOURCES` / `--sources`) +### Inline colon format (`PINECONE_SOURCES` / `--sources`) Semicolon-separated entries: `name:apiKey:indexName` @@ -59,6 +59,47 @@ PINECONE_SOURCES=api_key_1:${PINECONE_API_KEY_1}:index_name_1;api_key_2:${PINECO API keys may contain colons; the parser treats the last `:` segment as `indexName` and everything between `name:` and `:indexName` as the key. +This colon format does **not** support `description` or `namespaces` — use JSON-shaped `PINECONE_SOURCES` or a JSON config file for those fields. + +### Inline JSON format (`PINECONE_SOURCES` / `--sources`) + +When the value starts with `{`, it is parsed as JSON. Two shapes are accepted: + +1. **Sources map** (MCP `env` friendly) — keys are source names; omit `apiKey` to default to `${sourceName}` (resolved from a sibling env var with the same name): + +```json +{ + "api_key_1": { + "indexName": "rag-hybrid", + "description": "Public corpora", + "namespaces": { + "example_ns": { + "description": "Namespace hint", + "metadata_schema": { "field_a": "string" } + } + } + }, + "api_key_2": { "indexName": "rag-hybrid" } +} +``` + +2. **Full file shape** — same as a JSON config file (`defaultSource` optional, `sources` required): + +```json +{ + "defaultSource": "api_key_1", + "sources": { + "api_key_1": { "apiKey": "${PINECONE_API_KEY_1}", "indexName": "index_name_1" } + } +} +``` + +At process runtime, env values are always strings. MCP hosts (including Cursor) typically JSON-stringify nested `env` objects when spawning the server; a pre-stringified JSON value also works. + +**Precedence:** `PINECONE_CONFIG_FILE` / `--config-file` wins over `PINECONE_SOURCES` when both are set. Within `PINECONE_SOURCES`, JSON (starts with `{`) is detected before the colon format. + +Staff inline example: [data/cppa-pinecone-mcp-setting.json](../data/cppa-pinecone-mcp-setting.json). + ### JSON config file Set `PINECONE_CONFIG_FILE` (or `--config-file`) to a path such as [examples/multi-source/pinecone-sources.json.example](../examples/multi-source/pinecone-sources.json.example): @@ -73,18 +114,18 @@ Set `PINECONE_CONFIG_FILE` (or `--config-file`) to a path such as [examples/mult } ``` -Values support `${ENV_VAR}` indirection (resolved at startup). Per-source `sparseIndexName` and `rerankModel` are optional; Alliance defaults apply when omitted. +Values support `${ENV_VAR}` indirection (resolved at startup). Per-source `sparseIndexName` and `rerankModel` are optional; Alliance defaults apply when omitted. When `apiKey` is omitted in JSON, it defaults to `${sourceName}` (e.g. source `api_key_1` → `${api_key_1}`). -Optional **config-file-only** fields (not supported in inline `PINECONE_SOURCES`): +Optional rich fields (supported in JSON config files **and** JSON-shaped `PINECONE_SOURCES`; **not** in the colon-delimited inline format): | Field | Scope | Purpose | | ----- | ----- | ------- | | `description` | Per source | Short corpus/content hint surfaced via `list_sources` (helps LLM routing across sources or MCP entries) | | `namespaces` | Per source | Map of namespace name → `{ description?, metadata_schema? }`; per-namespace `description` is surfaced via `list_namespaces` on matching live rows | -`metadata_schema` is a flat `fieldName → type` map (same vocabulary as `list_namespaces` → `metadata_fields`, e.g. `"title": "string"`). When declared for a live namespace, the server **skips live sampling** for that namespace and trusts the declared schema until the config file changes. Namespaces declared in config but absent from Pinecone produce a non-fatal `config_warnings` entry in `list_namespaces` (never a startup failure). +`metadata_schema` is a flat `fieldName → type` map (same vocabulary as `list_namespaces` → `metadata_fields`, e.g. `"title": "string"`). When declared for a live namespace, the server **skips live sampling** for that namespace and trusts the declared schema until the config changes. Namespaces declared in config but absent from Pinecone produce a non-fatal `config_warnings` entry in `list_namespaces` (never a startup failure). -**Never** commit real corpus descriptions, namespace names, or internal field names to the open-source repo — use generic placeholders in examples only. Real values belong in staff-machine private config files per [Deployment profiles](#deployment-profiles). See [SECURITY.md](./SECURITY.md). +**Never** commit real corpus descriptions, namespace names, or internal field names to the open-source repo — use generic placeholders in examples only. Real values belong in staff-machine private config (JSON file or MCP `env`) per [Deployment profiles](#deployment-profiles). See [SECURITY.md](./SECURITY.md). ### MCP tools and routing @@ -124,7 +165,7 @@ Multi-source mode supports two operational profiles. **Never** ship a merged int } ``` -**Internal MCP config (merged sources):** +**Internal MCP config (merged sources, JSON inline):** ```json { @@ -133,16 +174,21 @@ Multi-source mode supports two operational profiles. **Never** ship a merged int "command": "npx", "args": ["-y", "@will-cppa/pinecone-read-only-mcp"], "env": { - "PINECONE_SOURCES": "api_key_1:${PINECONE_API_KEY_1}:index_name_1;api_key_2:${PINECONE_API_KEY_2}:index_name_2", - "PINECONE_API_KEY_1": "pcsk_...", - "PINECONE_API_KEY_2": "pcsk_..." + "api_key_1": "pcsk_...", + "api_key_2": "pcsk_...", + "PINECONE_SOURCES": { + "api_key_1": { "indexName": "rag-hybrid", "description": "..." }, + "api_key_2": { "indexName": "rag-hybrid" } + } } } } } ``` -Prefer `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over inline API keys in `PINECONE_SOURCES`. For internal deployments, add optional `description` and `namespaces` declarations in the JSON config file on staff machines only — never in public examples or committed constants. See [SECURITY.md](./SECURITY.md). +Colon format remains supported: `"PINECONE_SOURCES": "api_key_1:${PINECONE_API_KEY_1}:index_name_1;..."`. + +Prefer JSON-shaped `PINECONE_SOURCES` or `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over embedding raw API keys. For internal deployments, add optional `description` and `namespaces` in JSON on staff machines only — never in public examples or committed constants. See [SECURITY.md](./SECURITY.md). ### Architecture decision diff --git a/docs/SECURITY.md b/docs/SECURITY.md index dcd338f..dd39e7e 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -25,7 +25,7 @@ This covers tool error payloads, hybrid degradation reasons, and SDK error text ## Private config content (descriptions and schemas) -Per-source `description` and per-namespace `namespaces` (including namespace names and `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles). +Per-source `description` and per-namespace `namespaces` (including namespace names and `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` or JSON-shaped `PINECONE_SOURCES` are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles). ## Docker image diff --git a/src/cli.ts b/src/cli.ts index 8654647..980a29e 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -160,7 +160,7 @@ Options: --request-timeout-ms N Per Pinecone call timeout [env: PINECONE_REQUEST_TIMEOUT_MS] --disable-suggest-flow Bypass suggest_query_params gate (PINECONE_DISABLE_SUGGEST_FLOW) --check-indexes Verify dense + sparse indexes then exit 0/1 (PINECONE_CHECK_INDEXES) - --sources TEXT Multi-source inline config (PINECONE_SOURCES) + --sources TEXT Multi-source inline config: JSON object or name:apiKey:indexName;... (PINECONE_SOURCES) --config-file PATH Multi-source JSON config file (PINECONE_CONFIG_FILE) --help, -h Show this message --version, -v Print package version diff --git a/src/core/server/source-config.test.ts b/src/core/server/source-config.test.ts index fee1f92..cf97533 100644 --- a/src/core/server/source-config.test.ts +++ b/src/core/server/source-config.test.ts @@ -216,6 +216,100 @@ describe('source-config', () => { expect(sources[0]).not.toHaveProperty('namespaces'); }); + it('resolveSourceDefinitions parses JSON sources-map with description and namespaces', () => { + const inline = JSON.stringify({ + api_key_1: { + indexName: 'index_name_1', + description: 'Staff corpus hint', + namespaces: { + example_ns: { + description: 'Namespace hint', + metadata_schema: { field_a: 'string' }, + }, + }, + }, + }); + const env = { api_key_1: 'api-1', PINECONE_SOURCES: inline }; + const parsed = resolveSourceDefinitions({}, env); + expect(parsed?.defaultSource).toBe('api_key_1'); + expect(parsed?.sources[0]).toMatchObject({ + name: 'api_key_1', + apiKey: 'api-1', + indexName: 'index_name_1', + description: 'Staff corpus hint', + }); + expect(parsed?.sources[0]?.namespaces?.example_ns).toMatchObject({ + description: 'Namespace hint', + metadata_schema: { field_a: 'string' }, + }); + }); + + it('resolveSourceDefinitions parses full JSON file shape inline', () => { + const inline = JSON.stringify({ + defaultSource: 'api_key_2', + sources: { + api_key_1: { apiKey: '${K1}', indexName: 'index_name_1' }, + api_key_2: { apiKey: '${K2}', indexName: 'index_name_2' }, + }, + }); + const env = { K1: 'api-1', K2: 'api-2', PINECONE_SOURCES: inline }; + const parsed = resolveSourceDefinitions({}, env); + expect(parsed?.defaultSource).toBe('api_key_2'); + expect(parsed?.sources).toHaveLength(2); + expect(parsed?.sources.find((s) => s.name === 'api_key_2')?.apiKey).toBe('api-2'); + }); + + it('resolveSourceDefinitions colon format still works (regression)', () => { + const env = { K1: 'api-1', PINECONE_SOURCES: 'api_key_1:${K1}:index_name_1' }; + const parsed = resolveSourceDefinitions({}, env); + expect(parsed?.sources[0]).toMatchObject({ + name: 'api_key_1', + apiKey: 'api-1', + indexName: 'index_name_1', + }); + expect(parsed?.sources[0]).not.toHaveProperty('description'); + }); + + it('resolveSourceDefinitions throws on invalid JSON PINECONE_SOURCES', () => { + const env = { PINECONE_SOURCES: '{not valid json' }; + expect(() => resolveSourceDefinitions({}, env)).toThrow( + /Failed to parse PINECONE_SOURCES JSON/ + ); + }); + + it('resolveSourceDefinitions prefers config file over JSON PINECONE_SOURCES', () => { + const dir = mkdtempSync(join(tmpdir(), 'pinecone-sources-')); + const filePath = join(dir, 'sources.json'); + writeFileSync( + filePath, + JSON.stringify({ + defaultSource: 'from_file', + sources: { + from_file: { apiKey: 'file-key', indexName: 'file-index' }, + }, + }) + ); + const env = { + PINECONE_SOURCES: JSON.stringify({ + from_inline: { apiKey: 'inline-key', indexName: 'inline-index' }, + }), + PINECONE_CONFIG_FILE: filePath, + }; + const parsed = resolveSourceDefinitions({}, env); + expect(parsed?.defaultSource).toBe('from_file'); + expect(parsed?.sources[0]?.indexName).toBe('file-index'); + }); + + it('resolveSourceDefinitions throws when defaulted apiKey env is missing', () => { + const inline = JSON.stringify({ + api_key_1: { indexName: 'index_name_1' }, + }); + const env = { PINECONE_SOURCES: inline }; + expect(() => resolveSourceDefinitions({}, env)).toThrow( + /Environment variable api_key_1 is not set/ + ); + }); + it('resolveConfig uses PINECONE_SOURCES when set', () => { vi.stubEnv('PINECONE_SOURCES', 'api_key_1:sk-test:my-index'); vi.stubEnv('PINECONE_API_KEY', 'ignored'); diff --git a/src/core/server/source-config.ts b/src/core/server/source-config.ts index ad04f49..43d2b50 100644 --- a/src/core/server/source-config.ts +++ b/src/core/server/source-config.ts @@ -6,7 +6,7 @@ import { readFileSync } from 'node:fs'; import { resolve } from 'node:path'; import { trimOptional } from '../config.js'; -/** Per-namespace declaration loaded from private JSON config (config-file only). */ +/** Per-namespace declaration loaded from JSON config file or JSON-shaped PINECONE_SOURCES. */ export interface NamespaceDeclaration { description?: string; metadata_schema?: Record; @@ -19,9 +19,9 @@ export interface SourceDefinition { indexName: string; sparseIndexName?: string; rerankModel?: string; - /** Optional corpus-level description (config-file only; never from inline PINECONE_SOURCES). */ + /** Optional corpus-level description (JSON config file or JSON-shaped PINECONE_SOURCES only). */ description?: string; - /** Optional per-namespace declarations (config-file only). */ + /** Optional per-namespace declarations (JSON config file or JSON-shaped PINECONE_SOURCES only). */ namespaces?: Record; } @@ -208,52 +208,48 @@ export function parseInlineSources( return sources; } -type JsonSourceFile = { +type JsonSourceEntry = { + apiKey?: string; + indexName: string; + sparseIndexName?: string; + rerankModel?: string; + description?: string; + namespaces?: Record; +}; + +export type JsonSourceFile = { defaultSource?: string; - sources: Record< - string, - { - apiKey: string; - indexName: string; - sparseIndexName?: string; - rerankModel?: string; - description?: string; - namespaces?: Record; - } - >; + sources: Record; }; -/** Parse JSON config file for multi-source setup. */ -export function parseSourcesConfigFile( - filePath: string, +function defaultApiKeyForSource(name: string, rawApiKey: unknown): string { + const trimmed = trimOptional(rawApiKey != null ? String(rawApiKey) : undefined); + return trimmed ?? `\${${name}}`; +} + +/** Parse a JSON sources object (config file or inline PINECONE_SOURCES JSON). */ +export function parseSourcesConfigObject( + parsed: JsonSourceFile, env: NodeJS.ProcessEnv = process.env, - options?: ParseSourcesOptions + options?: ParseSourcesOptions, + originLabel = 'config file' ): { sources: SourceDefinition[]; defaultSource: string } { - const absolute = resolve(filePath); - let parsed: JsonSourceFile; - try { - const raw = readFileSync(absolute, 'utf8'); - parsed = JSON.parse(raw) as JsonSourceFile; - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - throw new Error(`Failed to read PINECONE config file "${filePath}": ${message}`); - } - if (!parsed.sources || typeof parsed.sources !== 'object') { - throw new Error(`PINECONE config file "${filePath}" must contain a "sources" object.`); + if (!parsed.sources || typeof parsed.sources !== 'object' || Array.isArray(parsed.sources)) { + throw new Error(`PINECONE ${originLabel} must contain a "sources" object.`); } const entries = Object.entries(parsed.sources); if (entries.length === 0) { - throw new Error(`PINECONE config file "${filePath}" has no sources.`); + throw new Error(`PINECONE ${originLabel} has no sources.`); } const sources: SourceDefinition[] = []; const seen = new Set(); for (const [name, cfg] of entries) { if (seen.has(name)) { - throw new Error(`Duplicate source name "${name}" in config file.`); + throw new Error(`Duplicate source name "${name}" in ${originLabel}.`); } seen.add(name); if (!cfg || typeof cfg !== 'object') { - throw new Error(`Source "${name}" in config file must be an object.`); + throw new Error(`Source "${name}" in ${originLabel} must be an object.`); } let description: string | undefined; if (cfg.description != null) { @@ -267,7 +263,7 @@ export function parseSourcesConfigFile( normalizeSourceEntry( name, { - apiKey: String(cfg.apiKey ?? ''), + apiKey: defaultApiKeyForSource(name, cfg.apiKey), indexName: String(cfg.indexName ?? ''), ...(cfg.sparseIndexName != null ? { sparseIndexName: String(cfg.sparseIndexName) } : {}), ...(cfg.rerankModel != null ? { rerankModel: String(cfg.rerankModel) } : {}), @@ -288,6 +284,47 @@ export function parseSourcesConfigFile( return { sources, defaultSource }; } +function parseJsonSourcesInline( + inline: string, + env: NodeJS.ProcessEnv, + options?: ParseSourcesOptions +): { sources: SourceDefinition[]; defaultSource: string } { + let parsed: unknown; + try { + parsed = JSON.parse(inline); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse PINECONE_SOURCES JSON: ${message}`); + } + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { + throw new Error('PINECONE_SOURCES JSON must be a JSON object.'); + } + const obj = parsed as Record; + const fileShape = + obj['sources'] != null && typeof obj['sources'] === 'object' && !Array.isArray(obj['sources']) + ? (parsed as JsonSourceFile) + : ({ sources: parsed } as JsonSourceFile); + return parseSourcesConfigObject(fileShape, env, options, 'PINECONE_SOURCES JSON'); +} + +/** Parse JSON config file for multi-source setup. */ +export function parseSourcesConfigFile( + filePath: string, + env: NodeJS.ProcessEnv = process.env, + options?: ParseSourcesOptions +): { sources: SourceDefinition[]; defaultSource: string } { + const absolute = resolve(filePath); + let parsed: JsonSourceFile; + try { + const raw = readFileSync(absolute, 'utf8'); + parsed = JSON.parse(raw) as JsonSourceFile; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to read PINECONE config file "${filePath}": ${message}`); + } + return parseSourcesConfigObject(parsed, env, options, `config file "${filePath}"`); +} + /** Resolve sources from overrides/env/file; config file wins over inline when both are set. */ export function resolveSourceDefinitions( overrides: { sources?: string; configFile?: string }, @@ -302,6 +339,9 @@ export function resolveSourceDefinitions( return parseSourcesConfigFile(configFile, env, options); } if (inline) { + if (inline.startsWith('{')) { + return parseJsonSourcesInline(inline, env, options); + } const sources = parseInlineSources(inline, env, options); return { sources, defaultSource: sources[0]!.name }; } From 1b96561e768f86bcdaee28a54df9b2daccf679e4 Mon Sep 17 00:00:00 2001 From: zho Date: Tue, 14 Jul 2026 00:27:54 +0800 Subject: [PATCH 2/8] remove a staff example line --- docs/CONFIGURATION.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 09c85d1..48400e4 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -98,8 +98,6 @@ At process runtime, env values are always strings. MCP hosts (including Cursor) **Precedence:** `PINECONE_CONFIG_FILE` / `--config-file` wins over `PINECONE_SOURCES` when both are set. Within `PINECONE_SOURCES`, JSON (starts with `{`) is detected before the colon format. -Staff inline example: [data/cppa-pinecone-mcp-setting.json](../data/cppa-pinecone-mcp-setting.json). - ### JSON config file Set `PINECONE_CONFIG_FILE` (or `--config-file`) to a path such as [examples/multi-source/pinecone-sources.json.example](../examples/multi-source/pinecone-sources.json.example): From 38d67fa6d74a955973704aafe4bf26d164a358e4 Mon Sep 17 00:00:00 2001 From: zho Date: Tue, 14 Jul 2026 01:41:13 +0800 Subject: [PATCH 3/8] updated CHANGELOG.md --- CHANGELOG.md | 2 +- docs/CONFIGURATION.md | 246 ++++++++++++-------------- docs/TOOLS.md | 13 ++ src/core/server/source-config.test.ts | 30 ++++ src/core/server/source-config.ts | 18 +- 5 files changed, 175 insertions(+), 134 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ad6fd2a..b778c93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release ### Added -- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE`) or JSON-shaped `PINECONE_SOURCES` (not the colon-delimited inline format). Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md](docs/CONFIGURATION.md#json-config-file). +- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE`) or JSON-shaped `PINECONE_SOURCES` (not the colon-delimited inline format). Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md § Multi-source mode](docs/CONFIGURATION.md#multi-source-mode). - **JSON-shaped `PINECONE_SOURCES`:** inline rich multi-source config (descriptions, namespaces, optional `defaultSource`) in MCP `env` or `--sources` without a separate config file; `apiKey` defaults to `${sourceName}` when omitted. - **`list_namespaces`:** optional per-namespace `schema_source` (`declared` | `sampled`), optional per-namespace `description` (from private config), and top-level `config_warnings` when private config declarations do not match live Pinecone data. - **Tests:** MCP verification harness ([src/__tests__/mcp-rc-readiness.test.ts](src/__tests__/mcp-rc-readiness.test.ts)) that drives the real server over an in-memory transport with the SDK client (initialize and protocol negotiation, the full registered tool surface, and a round-trip tool call), so a future SDK or MCP protocol bump is verified in one place. Protocol assertions key off the SDK's `LATEST_PROTOCOL_VERSION`, so pinning the RC SDK re-runs the check with no test edits. (#202) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 48400e4..d09f1ab 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -1,45 +1,82 @@ # Configuration -Configuration is built from **CLI flags** (when using the binary), **environment variables**, and **defaults**. Library callers use `resolveConfig(overrides)` (core) or `resolveAllianceConfig(overrides)` (Alliance CLI / `setupAllianceServer`) with optional `ConfigOverrides`. +Configuration comes from **CLI flags** (binary), **environment variables**, or **`ConfigOverrides`** (library). Precedence: **CLI / `ConfigOverrides` > environment variables > built-in defaults.** -## Precedence +This doc is organized for two audiences: -**CLI / `ConfigOverrides` > environment variables > built-in defaults.** - -`resolveConfig` in `src/core/config.ts` applies this order for each field. +- **Running the server** (MCP host config, env vars, CLI flags) — read top to bottom. +- **Embedding the library** — jump to [Library embedding](#library-embedding). --- -## `ServerConfig` (resolved shape) +## Quick start: single Pinecone project + +Most setups need just an API key and index name. Add this to your MCP host config (e.g. Cursor `mcp.json`, `claude_desktop_config.json`): + +```json +{ + "mcpServers": { + "pinecone-search": { + "command": "npx", + "args": ["-y", "@will-cppa/pinecone-read-only-mcp"], + "env": { + "PINECONE_API_KEY": "your-api-key-here", + "PINECONE_INDEX_NAME": "your-index-name" + } + } + } +} +``` + +Or via CLI flags instead of env: + +```json +{ + "mcpServers": { + "pinecone-search": { + "command": "npx", + "args": ["-y", "@will-cppa/pinecone-read-only-mcp", "--api-key", "your-api-key-here", "--index-name", "your-index-name"] + } + } +} +``` -| Field | Source | Default / notes | -| ----- | ------ | --------------- | -| `apiKey` | `apiKey` / `PINECONE_API_KEY` | **Required** (non-empty after trim) | -| `indexName` | `indexName` / `PINECONE_INDEX_NAME` | **Required** (non-empty after trim) | -| `sparseIndexName` | `sparseIndexName` / `PINECONE_SPARSE_INDEX_NAME` | `{indexName}-sparse` | -| `rerankModel` | `rerankModel` / `PINECONE_RERANK_MODEL` | **Core:** omitted when unset (rerank disabled). **Alliance CLI:** `bge-reranker-v2-m3` when unset | -| `defaultTopK` | `defaultTopK` / `PINECONE_TOP_K` | `10` (positive int) | -| `logLevel` | `logLevel` / `PINECONE_READ_ONLY_MCP_LOG_LEVEL` | `INFO` (`DEBUG`–`ERROR`) | -| `logFormat` | `logFormat` / `PINECONE_READ_ONLY_MCP_LOG_FORMAT` | `text` or `json` | -| `cacheTtlMs` | `cacheTtlSeconds` / `PINECONE_CACHE_TTL_SECONDS` | `1800` seconds → ms | -| `requestTimeoutMs` | `requestTimeoutMs` / `PINECONE_REQUEST_TIMEOUT_MS` | `15000` | -| `disableSuggestFlow` | `disableSuggestFlow` / `PINECONE_DISABLE_SUGGEST_FLOW` | **Core `resolveConfig`:** `true` (gate off). **Alliance `resolveAllianceConfig` / CLI:** `false` (gate on). Bool parsing: true/1/yes/on | -| `checkIndexes` | `checkIndexes` / `PINECONE_CHECK_INDEXES` | `false` | -| `sources` | `sources` / `PINECONE_SOURCES` or JSON config file | Omitted in single-key mode; see [Multi-source mode](#multi-source-mode) | -| `defaultSource` | JSON config `defaultSource` only | First source when using inline `PINECONE_SOURCES` | +Need more than one Pinecone project in the same MCP entry? See [Multi-source mode](#multi-source-mode). -**Throws** if `apiKey` or `indexName` is missing after trim (single-key mode) — this happens at server startup, not as an MCP tool error. In multi-source mode, `PINECONE_API_KEY` is ignored when `PINECONE_SOURCES` or a config file is set; credentials come from each source entry. +--- -For the full Alliance tool surface (including `suggest_query_params`, `guided_query`, and built-in URL generators), import from `@will-cppa/pinecone-read-only-mcp/alliance` and use the three-step instance-first recipe at [Library embedding](#library-embedding) below. +## All settings (`ServerConfig`) + +| Field | CLI flag | Env var | Default / notes | +| ----- | -------- | ------- | ---------------- | +| `apiKey` | `--api-key` | `PINECONE_API_KEY` | **Required** in single-key mode (non-empty after trim) | +| `indexName` | `--index-name` | `PINECONE_INDEX_NAME` | **Required** in single-key mode (non-empty after trim) | +| `sparseIndexName` | `--sparse-index-name` | `PINECONE_SPARSE_INDEX_NAME` | `{indexName}-sparse` | +| `rerankModel` | `--rerank-model` | `PINECONE_RERANK_MODEL` | **Core:** omitted when unset (rerank disabled). **Alliance CLI:** `bge-reranker-v2-m3` when unset | +| `defaultTopK` | `--top-k` | `PINECONE_TOP_K` | `10` (positive int) | +| `logLevel` | `--log-level` | `PINECONE_READ_ONLY_MCP_LOG_LEVEL` | `INFO` (`DEBUG`–`ERROR`) | +| `logFormat` | `--log-format` | `PINECONE_READ_ONLY_MCP_LOG_FORMAT` | `text` or `json` | +| `cacheTtlMs` | `--cache-ttl-seconds` | `PINECONE_CACHE_TTL_SECONDS` | `1800` seconds → ms | +| `requestTimeoutMs` | `--request-timeout-ms` | `PINECONE_REQUEST_TIMEOUT_MS` | `15000` | +| `disableSuggestFlow` | `--disable-suggest-flow` | `PINECONE_DISABLE_SUGGEST_FLOW` | **Core:** `true` (gate off). **Alliance:** `false` (gate on). Bool parsing: `true`/`1`/`yes`/`on` | +| `checkIndexes` | `--check-indexes` | `PINECONE_CHECK_INDEXES` | `false` | +| `sources` | `--sources` | `PINECONE_SOURCES` | Multi-source only; see below | +| `configFile` | `--config-file` | `PINECONE_CONFIG_FILE` | Multi-source only; see below | +| — | `--help`, `-h` | — | Print help and exit | +| — | `--version`, `-v` | — | Print version and exit | + +**Throws** if `apiKey` or `indexName` is missing after trim (single-key mode) — at server startup, not as an MCP tool error. In multi-source mode, `PINECONE_API_KEY` is ignored; credentials come from each source entry. ### Core vs Alliance resolvers +Library callers use `resolveConfig(overrides)` (core) or `resolveAllianceConfig(overrides)` (Alliance CLI / `setupAllianceServer`). They differ only in defaults: + | Resolver | When to use | Index when unset | Rerank when unset | Suggest gate when unset | -| -------- | ------------- | ---------------- | ----------------- | ----------------------- | +| -------- | ----------- | ----------------- | ------------------ | ------------------------ | | `resolveConfig` | Package root, `setupCoreServer`, quickstart | **Throws** | Omitted (no rerank) | Off (`disableSuggestFlow: true`) | | `resolveAllianceConfig` | Published CLI, `setupAllianceServer` | `rag-hybrid` | `bge-reranker-v2-m3` | On (`disableSuggestFlow: false`) | -**Warning:** Switching between `resolveConfig` / `setupCoreServer` (package root) and `resolveAllianceConfig` / `setupAllianceServer` changes suggest-flow gate behavior. Core defaults bypass the gate; Alliance defaults enforce it. Use `guided_query` (registered in both setups) for single-call retrieval without manual `suggest_query_params`, or align `disableSuggestFlow` explicitly when migrating between entry points. +**Warning:** switching between the two changes suggest-flow gate behavior. Use `guided_query` (registered in both) for single-call retrieval without manual `suggest_query_params`, or set `disableSuggestFlow` explicitly when migrating between entry points. C++ Alliance deployers can copy [examples/alliance/.env.example](../examples/alliance/.env.example). Constants: `ALLIANCE_DEFAULT_INDEX_NAME` / `ALLIANCE_DEFAULT_RERANK_MODEL` from `@will-cppa/pinecone-read-only-mcp/alliance`. @@ -47,88 +84,85 @@ C++ Alliance deployers can copy [examples/alliance/.env.example](../examples/all ## Multi-source mode -Use **one MCP server entry** with multiple Pinecone API keys / projects when `PINECONE_SOURCES`, `--sources`, or a JSON config file (`PINECONE_CONFIG_FILE` / `--config-file`) is set. +Use **one MCP server entry** with multiple Pinecone projects by setting `PINECONE_SOURCES` / `--sources`, or `PINECONE_CONFIG_FILE` / `--config-file`. + +**Pick a format:** + +| I want to... | Use | +| ------------ | --- | +| List a few `name:key:index` pairs, nothing else | [Colon format](#colon-format-keys-only) | +| Add per-source `description` or per-namespace `metadata_schema` | [JSON format](#json-format-descriptions--namespaces) — inline or file, same schema | +| Keep source config in a separate file from the MCP settings | JSON via `PINECONE_CONFIG_FILE` | +| Keep everything in the MCP host's `env` block | JSON via `PINECONE_SOURCES` | -### Inline colon format (`PINECONE_SOURCES` / `--sources`) +**Precedence:** `PINECONE_CONFIG_FILE` / `--config-file` wins when both are set. Within `PINECONE_SOURCES`, a value starting with `{` is parsed as JSON; otherwise it is parsed as the colon format. -Semicolon-separated entries: `name:apiKey:indexName` +### Colon format (keys only) + +Semicolon-separated `name:apiKey:indexName` entries: ```bash PINECONE_SOURCES=api_key_1:${PINECONE_API_KEY_1}:index_name_1;api_key_2:${PINECONE_API_KEY_2}:index_name_2 ``` -API keys may contain colons; the parser treats the last `:` segment as `indexName` and everything between `name:` and `:indexName` as the key. - -This colon format does **not** support `description` or `namespaces` — use JSON-shaped `PINECONE_SOURCES` or a JSON config file for those fields. - -### Inline JSON format (`PINECONE_SOURCES` / `--sources`) +API keys may contain colons; the parser treats the last `:` segment as `indexName` and everything between `name:` and `:indexName` as the key. This format never carries `description` or `namespaces` — use JSON for those. -When the value starts with `{`, it is parsed as JSON. Two shapes are accepted: +### JSON format (descriptions + namespaces) -1. **Sources map** (MCP `env` friendly) — keys are source names; omit `apiKey` to default to `${sourceName}` (resolved from a sibling env var with the same name): - -```json -{ - "api_key_1": { - "indexName": "rag-hybrid", - "description": "Public corpora", - "namespaces": { - "example_ns": { - "description": "Namespace hint", - "metadata_schema": { "field_a": "string" } - } - } - }, - "api_key_2": { "indexName": "rag-hybrid" } -} -``` - -2. **Full file shape** — same as a JSON config file (`defaultSource` optional, `sources` required): +One schema, loadable two ways — as the `PINECONE_SOURCES` value (inline) or as a `PINECONE_CONFIG_FILE` file: ```json { "defaultSource": "api_key_1", "sources": { - "api_key_1": { "apiKey": "${PINECONE_API_KEY_1}", "indexName": "index_name_1" } + "api_key_1": { + "apiKey": "${PINECONE_API_KEY_1}", + "indexName": "index_name_1", + "description": "Short corpus hint shown by list_sources", + "namespaces": { + "example_ns": { + "description": "Namespace hint shown by list_namespaces", + "metadata_schema": { "field_a": "string" } + } + } + }, + "api_key_2": { "apiKey": "${PINECONE_API_KEY_2}", "indexName": "index_name_2" } } } ``` -At process runtime, env values are always strings. MCP hosts (including Cursor) typically JSON-stringify nested `env` objects when spawning the server; a pre-stringified JSON value also works. +- `defaultSource` is optional (defaults to the first key); required only to be a valid source name when present. +- `apiKey` is optional per source — when omitted, it defaults to `${sourceName}` (e.g. source `api_key_1` → env var `api_key_1`). Use this to keep keys as sibling `env` vars without repeating the source name. +- `sparseIndexName` / `rerankModel` are optional per source (same defaults as single-key mode). +- All string values support `${ENV_VAR}` indirection, resolved at startup. -**Precedence:** `PINECONE_CONFIG_FILE` / `--config-file` wins over `PINECONE_SOURCES` when both are set. Within `PINECONE_SOURCES`, JSON (starts with `{`) is detected before the colon format. - -### JSON config file - -Set `PINECONE_CONFIG_FILE` (or `--config-file`) to a path such as [examples/multi-source/pinecone-sources.json.example](../examples/multi-source/pinecone-sources.json.example): +**Inline in `PINECONE_SOURCES`**, you can also drop the `sources` wrapper and use the map directly (handy for MCP `env` blocks — Cursor's `mcp.json` example below uses this shape): ```json { - "defaultSource": "api_key_1", - "sources": { - "api_key_1": { "apiKey": "${PINECONE_API_KEY_1}", "indexName": "index_name_1" }, - "api_key_2": { "apiKey": "${PINECONE_API_KEY_2}", "indexName": "index_name_2" } - } + "api_key_1": { "indexName": "rag-hybrid", "description": "..." }, + "api_key_2": { "indexName": "rag-hybrid" } } ``` -Values support `${ENV_VAR}` indirection (resolved at startup). Per-source `sparseIndexName` and `rerankModel` are optional; Alliance defaults apply when omitted. When `apiKey` is omitted in JSON, it defaults to `${sourceName}` (e.g. source `api_key_1` → `${api_key_1}`). +Both shapes (wrapped with `sources`, or the bare map) are accepted whenever `PINECONE_SOURCES` starts with `{`. + +**As a file**, set `PINECONE_CONFIG_FILE=./pinecone-sources.json` (or `--config-file`) pointing at the wrapped shape above — see [examples/multi-source/pinecone-sources.json.example](../examples/multi-source/pinecone-sources.json.example). -Optional rich fields (supported in JSON config files **and** JSON-shaped `PINECONE_SOURCES`; **not** in the colon-delimited inline format): +**Passing JSON through `env`:** env values are always strings at the OS level. The safest option is to pass an already-stringified JSON string as the `PINECONE_SOURCES` value. Some MCP hosts (including Cursor) also accept a nested JSON object in `mcp.json` and stringify it before spawning the server — if your host does this, a raw object works too; if you see a parse error, stringify it explicitly instead. -| Field | Scope | Purpose | -| ----- | ----- | ------- | -| `description` | Per source | Short corpus/content hint surfaced via `list_sources` (helps LLM routing across sources or MCP entries) | -| `namespaces` | Per source | Map of namespace name → `{ description?, metadata_schema? }`; per-namespace `description` is surfaced via `list_namespaces` on matching live rows | +**`metadata_schema`** is a flat `fieldName → type` map (same vocabulary as `list_namespaces` → `metadata_fields`, e.g. `"title": "string"`). When declared for a live namespace, the server **skips live sampling** for that namespace and trusts the declared schema until the config changes. Namespaces declared but absent from Pinecone produce a non-fatal `config_warnings` entry in `list_namespaces` (never a startup failure). -`metadata_schema` is a flat `fieldName → type` map (same vocabulary as `list_namespaces` → `metadata_fields`, e.g. `"title": "string"`). When declared for a live namespace, the server **skips live sampling** for that namespace and trusts the declared schema until the config changes. Namespaces declared in config but absent from Pinecone produce a non-fatal `config_warnings` entry in `list_namespaces` (never a startup failure). +**Never** commit real corpus descriptions, namespace names, or internal field names to the open-source repo — use generic placeholders in examples only. Real values belong in staff-machine private config (file or MCP `env`). See [SECURITY.md](./SECURITY.md). -**Never** commit real corpus descriptions, namespace names, or internal field names to the open-source repo — use generic placeholders in examples only. Real values belong in staff-machine private config (JSON file or MCP `env`) per [Deployment profiles](#deployment-profiles). See [SECURITY.md](./SECURITY.md). +> **Caveat — source names with hyphens:** the default-`apiKey` shortcut (`apiKey` omitted → `${sourceName}`) only works when the source name is also a valid environment-variable identifier (letters, digits, underscore; no leading digit). A name like `internal-corpus` triggers a startup error — supply `apiKey` explicitly for hyphenated source names. + +> **Note — bare-map with source named `sources`:** a payload like `{"sources": {"indexName": "rag-hybrid"}}` is interpreted as a sources-map (one source named `sources`). For multi-source file layout or an explicit `defaultSource`, use the wrapped `{ "defaultSource"?, "sources": { ... } }` shape. ### MCP tools and routing | Tool | `source` parameter | -| ---- | ------------------ | +| ---- | ------------------- | | `list_sources` | Registered only when more than one source is configured | | `list_namespaces`, `namespace_router` | Omit to aggregate all sources; results include `source` when tagged | | `query`, `count`, `query_documents`, `keyword_search`, `generate_urls`, `suggest_query_params`, `guided_query` | Omit when the namespace uniquely identifies one source; required when the same namespace exists on multiple sources | @@ -137,31 +171,18 @@ Discovery responses may include `source_errors` when one project fails but other Single-key deployments (`PINECONE_API_KEY` + `PINECONE_INDEX_NAME` only) are unchanged — no `source` field on responses and no `list_sources` tool. +See [TOOLS.md § Multi-source mode](./TOOLS.md#multi-source-mode) for the full per-tool routing reference and design notes. + ### Deployment profiles Multi-source mode supports two operational profiles. **Never** ship a merged internal config through the same channel used for external partners. | Profile | Who | Config | Risk if mis-shared | -| ------- | --- | ------ | ------------------ | +| ------- | --- | ------ | -------------------- | | **External (public-only)** | External companies, public MCP distribution | `PINECONE_API_KEY` + `PINECONE_INDEX_NAME`, or `PINECONE_SOURCES` with **one** entry | Low — single public key only | | **Internal (merged)** | Staff machines with access to private data | `PINECONE_SOURCES` or JSON config with **two+** entries | **High** — private API key and private namespace names exposed | -**External MCP config (unchanged):** - -```json -{ - "mcpServers": { - "pinecone-search": { - "command": "npx", - "args": ["-y", "@will-cppa/pinecone-read-only-mcp"], - "env": { - "PINECONE_API_KEY": "your-public-key", - "PINECONE_INDEX_NAME": "rag-hybrid" - } - } - } -} -``` +**External MCP config (single source, unchanged):** see [Quick start](#quick-start-single-pinecone-project) above. **Internal MCP config (merged sources, JSON inline):** @@ -184,48 +205,9 @@ Multi-source mode supports two operational profiles. **Never** ship a merged int } ``` -Colon format remains supported: `"PINECONE_SOURCES": "api_key_1:${PINECONE_API_KEY_1}:index_name_1;..."`. - -Prefer JSON-shaped `PINECONE_SOURCES` or `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over embedding raw API keys. For internal deployments, add optional `description` and `namespaces` in JSON on staff machines only — never in public examples or committed constants. See [SECURITY.md](./SECURITY.md). - -### Architecture decision - -**Chosen:** Option A — multi-source server with optional `source` parameter on tools (`SourceRegistry` + `PINECONE_SOURCES` / JSON config). - -**Rejected:** - -- **Option B (Cursor routing rules only):** Does not fix the UX problem when users forget which MCP entry to use; no code changes. -- **Option C (thin proxy MCP):** Extra package and latency; duplicates routing logic already in `ServerContext`. - -**Rationale:** Pinecone SDK v8 supports multiple `Pinecone({ apiKey })` instances per process; MCP has no barrier to aggregating backends. Security is enforced by **deployment profiles** (public-only vs merged config), not per-query MCP authorization. All multi-source results include `source` for LLM provenance. - -**Execution semantics:** Discovery tools aggregate all sources when `source` is omitted. Execution tools (`query`, `count`, etc.) call `resolveSource`: infer source when the namespace exists on exactly one project; return `VALIDATION` when ambiguous. They do **not** fan out one query to all sources. `guided_query` without `namespace` routes via the aggregated namespace list and sets `selected_source` in `decision_trace`. - -**Audit logging:** When a tool resolves a specific source, stderr logs `toolname [source=name]` at INFO (execution tools and discovery tools that pass an explicit `source` filter). Aggregated discovery without `source` does not log per-source lines. - -See also [TOOLS.md § Multi-source mode](./TOOLS.md#multi-source-mode). - ---- +Colon format also works here: `"PINECONE_SOURCES": "api_key_1:${PINECONE_API_KEY_1}:index_name_1;..."`. -## CLI flags (`parseCli` / `src/cli.ts`) - -| Flag | Maps to | -| ---- | ------- | -| `--api-key` | `apiKey` | -| `--index-name` | `indexName` | -| `--sparse-index-name` | `sparseIndexName` | -| `--rerank-model` | `rerankModel` | -| `--top-k` | `defaultTopK` | -| `--log-level` | `logLevel` | -| `--log-format` | `logFormat` | -| `--cache-ttl-seconds` | `cacheTtlSeconds` | -| `--request-timeout-ms` | `requestTimeoutMs` | -| `--disable-suggest-flow` | `disableSuggestFlow: true` | -| `--check-indexes` | `checkIndexes: true` | -| `--sources` | `sources` (inline multi-source string) | -| `--config-file` | `configFile` / `PINECONE_CONFIG_FILE` | -| `--help` / `-h` | Print help and exit | -| `--version` / `-v` | Print version and exit | +Prefer JSON-shaped `PINECONE_SOURCES` or `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over embedding raw API keys directly. For internal deployments, add optional `description` and `namespaces` in JSON on staff machines only — never in public examples or committed constants. See [SECURITY.md](./SECURITY.md). --- diff --git a/docs/TOOLS.md b/docs/TOOLS.md index 8f7195e..e5fb9ca 100644 --- a/docs/TOOLS.md +++ b/docs/TOOLS.md @@ -64,6 +64,19 @@ Or single-shot: `guided_query` (routes across sources when `namespace` is omitte **Suggest-flow in multi-source mode:** gate state uses compound keys `source:namespace`. Pass the same `source` + `namespace` pair for `suggest_query_params` and gated tools when the namespace exists on multiple sources. +### Design notes + +**Chosen:** multi-source server with optional `source` parameter on tools (`SourceRegistry` + `PINECONE_SOURCES` / JSON config), over: + +- **Cursor routing rules only:** doesn't fix the UX problem when users forget which MCP entry to use; no code changes. +- **Thin proxy MCP:** extra package and latency; duplicates routing logic already in `ServerContext`. + +**Rationale:** Pinecone SDK v8 supports multiple `Pinecone({ apiKey })` instances per process; MCP has no barrier to aggregating backends. Security is enforced by **deployment profiles** (public-only vs merged config), not per-query MCP authorization. All multi-source results include `source` for LLM provenance. + +**Execution semantics:** discovery tools aggregate all sources when `source` is omitted. Execution tools (`query`, `count`, etc.) call `resolveSource`: infer source when the namespace exists on exactly one project; return `VALIDATION` when ambiguous. They do **not** fan out one query to all sources. `guided_query` without `namespace` routes via the aggregated namespace list and sets `selected_source` in `decision_trace`. + +**Audit logging:** when a tool resolves a specific source, stderr logs `toolname [source=name]` at INFO (execution tools and discovery tools that pass an explicit `source` filter). Aggregated discovery without `source` does not log per-source lines. + --- ## `list_sources` (multi-source only) diff --git a/src/core/server/source-config.test.ts b/src/core/server/source-config.test.ts index cf97533..37c6861 100644 --- a/src/core/server/source-config.test.ts +++ b/src/core/server/source-config.test.ts @@ -16,6 +16,12 @@ describe('source-config', () => { expect(resolveEnvIndirection('${PINECONE_API_KEY_1}', env)).toBe('key-one'); }); + it('throws on malformed env indirection reference', () => { + expect(() => resolveEnvIndirection('${internal-corpus}', {})).toThrow( + /Invalid environment variable reference/ + ); + }); + it('parses inline sources', () => { const env = { K1: 'api-1', @@ -310,6 +316,30 @@ describe('source-config', () => { ); }); + it('resolveSourceDefinitions throws when hyphenated source name uses defaulted apiKey', () => { + const inline = JSON.stringify({ + 'internal-corpus': { indexName: 'index_name_1' }, + }); + const env = { PINECONE_SOURCES: inline }; + expect(() => resolveSourceDefinitions({}, env)).toThrow( + /Invalid environment variable reference "\$\{internal-corpus\}"/ + ); + }); + + it('resolveSourceDefinitions treats bare map with source named sources as sources-map', () => { + const inline = JSON.stringify({ + sources: { indexName: 'rag-hybrid' }, + }); + const env = { sources: 'api-key-for-sources', PINECONE_SOURCES: inline }; + const parsed = resolveSourceDefinitions({}, env); + expect(parsed?.sources).toHaveLength(1); + expect(parsed?.sources[0]).toMatchObject({ + name: 'sources', + apiKey: 'api-key-for-sources', + indexName: 'rag-hybrid', + }); + }); + it('resolveConfig uses PINECONE_SOURCES when set', () => { vi.stubEnv('PINECONE_SOURCES', 'api_key_1:sk-test:my-index'); vi.stubEnv('PINECONE_API_KEY', 'ignored'); diff --git a/src/core/server/source-config.ts b/src/core/server/source-config.ts index 43d2b50..609d121 100644 --- a/src/core/server/source-config.ts +++ b/src/core/server/source-config.ts @@ -38,8 +38,14 @@ const SOURCE_NAME_PATTERN = /^[a-zA-Z0-9_-]+$/; /** Resolve `${ENV_VAR}` references in a string value. */ export function resolveEnvIndirection(value: string, env: NodeJS.ProcessEnv): string { const trimmed = value.trim(); + const looksLikeIndirection = /^\$\{.*\}$/.test(trimmed); const match = /^\$\{([A-Za-z_][A-Za-z0-9_]*)\}$/.exec(trimmed); if (!match) { + if (looksLikeIndirection) { + throw new Error( + `Invalid environment variable reference "${trimmed}": names must contain only letters, digits, and underscores (no leading digit).` + ); + } return trimmed; } const envKey = match[1]!; @@ -284,6 +290,10 @@ export function parseSourcesConfigObject( return { sources, defaultSource }; } +function looksLikeSourceEntry(v: unknown): boolean { + return !!v && typeof v === 'object' && !Array.isArray(v); +} + function parseJsonSourcesInline( inline: string, env: NodeJS.ProcessEnv, @@ -300,8 +310,14 @@ function parseJsonSourcesInline( throw new Error('PINECONE_SOURCES JSON must be a JSON object.'); } const obj = parsed as Record; + const sourcesCandidate = obj['sources']; + const hasFileShapeSourcesField = + sourcesCandidate != null && + typeof sourcesCandidate === 'object' && + !Array.isArray(sourcesCandidate) && + Object.values(sourcesCandidate as Record).every(looksLikeSourceEntry); const fileShape = - obj['sources'] != null && typeof obj['sources'] === 'object' && !Array.isArray(obj['sources']) + obj['defaultSource'] != null || hasFileShapeSourcesField ? (parsed as JsonSourceFile) : ({ sources: parsed } as JsonSourceFile); return parseSourcesConfigObject(fileShape, env, options, 'PINECONE_SOURCES JSON'); From f8cd3232cb4d9a40b80646442e54a95acd2f97d9 Mon Sep 17 00:00:00 2001 From: zho Date: Wed, 15 Jul 2026 01:58:24 +0800 Subject: [PATCH 4/8] Load per-source schema manifests from Pinecone _mcp_config at startup --- CHANGELOG.md | 4 +- docs/CONFIGURATION.md | 53 +++--- docs/SECURITY.md | 2 +- src/cli.ts | 6 +- src/core/config.ts | 7 + src/core/pinecone-client.ts | 5 + src/core/pinecone/indexes.test.ts | 58 +++++++ src/core/pinecone/indexes.ts | 23 +++ src/core/server/remote-schema.test.ts | 163 ++++++++++++++++++ src/core/server/remote-schema.ts | 128 ++++++++++++++ .../server/server-context.composition.test.ts | 29 ++++ src/core/server/server-context.ts | 10 +- src/core/server/source-config.test.ts | 109 ++++-------- src/core/server/source-config.ts | 49 +----- src/index.ts | 70 +++++++- 15 files changed, 564 insertions(+), 152 deletions(-) create mode 100644 src/core/server/remote-schema.test.ts create mode 100644 src/core/server/remote-schema.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b778c93..aa8b260 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,8 +10,8 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release ### Added -- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE`) or JSON-shaped `PINECONE_SOURCES` (not the colon-delimited inline format). Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md § Multi-source mode](docs/CONFIGURATION.md#multi-source-mode). -- **JSON-shaped `PINECONE_SOURCES`:** inline rich multi-source config (descriptions, namespaces, optional `defaultSource`) in MCP `env` or `--sources` without a separate config file; `apiKey` defaults to `${sourceName}` when omitted. +- **Per-source and per-namespace private config:** optional `description` (source-level) and `namespaces` map (`description` + declared `metadata_schema`) in JSON config files (`PINECONE_CONFIG_FILE`), or loaded automatically from the `_mcp_config` Pinecone namespace when not declared locally. Declared schemas skip live sampling; stale declared namespaces surface as `config_warnings` in `list_namespaces`. See [CONFIGURATION.md § Multi-source mode](docs/CONFIGURATION.md#multi-source-mode). +- **Remote `_mcp_config` schema manifest:** automatic per-source loading of description and namespace declarations from Pinecone at startup (opt out via `PINECONE_DISABLE_REMOTE_SCHEMA` / `--disable-remote-schema`); non-fatal on failure. - **`list_namespaces`:** optional per-namespace `schema_source` (`declared` | `sampled`), optional per-namespace `description` (from private config), and top-level `config_warnings` when private config declarations do not match live Pinecone data. - **Tests:** MCP verification harness ([src/__tests__/mcp-rc-readiness.test.ts](src/__tests__/mcp-rc-readiness.test.ts)) that drives the real server over an in-memory transport with the SDK client (initialize and protocol negotiation, the full registered tool surface, and a round-trip tool call), so a future SDK or MCP protocol bump is verified in one place. Protocol assertions key off the SDK's `LATEST_PROTOCOL_VERSION`, so pinning the RC SDK re-runs the check with no test edits. (#202) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index d09f1ab..9379fee 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -60,6 +60,7 @@ Need more than one Pinecone project in the same MCP entry? See [Multi-source mod | `requestTimeoutMs` | `--request-timeout-ms` | `PINECONE_REQUEST_TIMEOUT_MS` | `15000` | | `disableSuggestFlow` | `--disable-suggest-flow` | `PINECONE_DISABLE_SUGGEST_FLOW` | **Core:** `true` (gate off). **Alliance:** `false` (gate on). Bool parsing: `true`/`1`/`yes`/`on` | | `checkIndexes` | `--check-indexes` | `PINECONE_CHECK_INDEXES` | `false` | +| `disableRemoteSchema` | `--disable-remote-schema` | `PINECONE_DISABLE_REMOTE_SCHEMA` | `false` (remote `_mcp_config` loading on) | | `sources` | `--sources` | `PINECONE_SOURCES` | Multi-source only; see below | | `configFile` | `--config-file` | `PINECONE_CONFIG_FILE` | Multi-source only; see below | | — | `--help`, `-h` | — | Print help and exit | @@ -91,11 +92,10 @@ Use **one MCP server entry** with multiple Pinecone projects by setting `PINECON | I want to... | Use | | ------------ | --- | | List a few `name:key:index` pairs, nothing else | [Colon format](#colon-format-keys-only) | -| Add per-source `description` or per-namespace `metadata_schema` | [JSON format](#json-format-descriptions--namespaces) — inline or file, same schema | -| Keep source config in a separate file from the MCP settings | JSON via `PINECONE_CONFIG_FILE` | -| Keep everything in the MCP host's `env` block | JSON via `PINECONE_SOURCES` | +| Add per-source `description` or per-namespace `metadata_schema` locally | [JSON config file](#json-config-file-descriptions--namespaces) | +| Let Pinecone supply descriptions/schemas automatically | [Remote schema manifest](#remote-schema-manifest-_mcp_config) (default when no local `namespaces`) | -**Precedence:** `PINECONE_CONFIG_FILE` / `--config-file` wins when both are set. Within `PINECONE_SOURCES`, a value starting with `{` is parsed as JSON; otherwise it is parsed as the colon format. +**Precedence:** `PINECONE_CONFIG_FILE` / `--config-file` wins over colon `PINECONE_SOURCES` when both are set. Per-source local `namespaces` in a config file win over the remote `_mcp_config` manifest for that source. ### Colon format (keys only) @@ -107,9 +107,9 @@ PINECONE_SOURCES=api_key_1:${PINECONE_API_KEY_1}:index_name_1;api_key_2:${PINECO API keys may contain colons; the parser treats the last `:` segment as `indexName` and everything between `name:` and `:indexName` as the key. This format never carries `description` or `namespaces` — use JSON for those. -### JSON format (descriptions + namespaces) +### JSON config file (descriptions + namespaces) -One schema, loadable two ways — as the `PINECONE_SOURCES` value (inline) or as a `PINECONE_CONFIG_FILE` file: +Set `PINECONE_CONFIG_FILE=./pinecone-sources.json` (or `--config-file`) with this shape: ```json { @@ -136,28 +136,28 @@ One schema, loadable two ways — as the `PINECONE_SOURCES` value (inline) or as - `sparseIndexName` / `rerankModel` are optional per source (same defaults as single-key mode). - All string values support `${ENV_VAR}` indirection, resolved at startup. -**Inline in `PINECONE_SOURCES`**, you can also drop the `sources` wrapper and use the map directly (handy for MCP `env` blocks — Cursor's `mcp.json` example below uses this shape): +See [examples/multi-source/pinecone-sources.json.example](../examples/multi-source/pinecone-sources.json.example). -```json -{ - "api_key_1": { "indexName": "rag-hybrid", "description": "..." }, - "api_key_2": { "indexName": "rag-hybrid" } -} -``` +> **Caveat — source names with hyphens:** the default-`apiKey` shortcut (`apiKey` omitted → `${sourceName}`) only works when the source name is also a valid environment-variable identifier (letters, digits, underscore; no leading digit). A name like `internal-corpus` triggers a startup error — supply `apiKey` explicitly for hyphenated source names. -Both shapes (wrapped with `sources`, or the bare map) are accepted whenever `PINECONE_SOURCES` starts with `{`. +**`metadata_schema`** is a flat `fieldName → type` map (same vocabulary as `list_namespaces` → `metadata_fields`, e.g. `"title": "string"`). When declared for a live namespace, the server **skips live sampling** for that namespace and trusts the declared schema until the config changes. Namespaces declared but absent from Pinecone produce a non-fatal `config_warnings` entry in `list_namespaces` (never a startup failure). -**As a file**, set `PINECONE_CONFIG_FILE=./pinecone-sources.json` (or `--config-file`) pointing at the wrapped shape above — see [examples/multi-source/pinecone-sources.json.example](../examples/multi-source/pinecone-sources.json.example). +**Never** commit real corpus descriptions, namespace names, or internal field names to the open-source repo — use generic placeholders in examples only. Real values belong in staff-machine private config (file or MCP `env`). See [SECURITY.md](./SECURITY.md). -**Passing JSON through `env`:** env values are always strings at the OS level. The safest option is to pass an already-stringified JSON string as the `PINECONE_SOURCES` value. Some MCP hosts (including Cursor) also accept a nested JSON object in `mcp.json` and stringify it before spawning the server — if your host does this, a raw object works too; if you see a parse error, stringify it explicitly instead. +### Remote schema manifest (`_mcp_config`) -**`metadata_schema`** is a flat `fieldName → type` map (same vocabulary as `list_namespaces` → `metadata_fields`, e.g. `"title": "string"`). When declared for a live namespace, the server **skips live sampling** for that namespace and trusts the declared schema until the config changes. Namespaces declared but absent from Pinecone produce a non-fatal `config_warnings` entry in `list_namespaces` (never a startup failure). +When a Pinecone project has a schema manifest upserted into the reserved `_mcp_config` namespace (record id `schema_manifest`), the MCP server **loads it automatically at startup** for: -**Never** commit real corpus descriptions, namespace names, or internal field names to the open-source repo — use generic placeholders in examples only. Real values belong in staff-machine private config (file or MCP `env`). See [SECURITY.md](./SECURITY.md). +- **Multi-source mode:** each source that does not already have local `namespaces` in `PINECONE_CONFIG_FILE`. +- **Single-key mode:** when using only `PINECONE_API_KEY` + `PINECONE_INDEX_NAME`. -> **Caveat — source names with hyphens:** the default-`apiKey` shortcut (`apiKey` omitted → `${sourceName}`) only works when the source name is also a valid environment-variable identifier (letters, digits, underscore; no leading digit). A name like `internal-corpus` triggers a startup error — supply `apiKey` explicitly for hyphenated source names. +The manifest supplies optional source-level `description` and per-namespace `description` + `metadata_schema` (same effect as a config file). Local declarations always win; remote loading is skipped when local `namespaces` are set for that source. -> **Note — bare-map with source named `sources`:** a payload like `{"sources": {"indexName": "rag-hybrid"}}` is interpreted as a sources-map (one source named `sources`). For multi-source file layout or an explicit `defaultSource`, use the wrapped `{ "defaultSource"?, "sources": { ... } }` shape. +- **Opt out:** set `PINECONE_DISABLE_REMOTE_SCHEMA=true` or pass `--disable-remote-schema`. +- **Failures are non-fatal:** missing manifest, network errors, or malformed JSON log a warning and the server falls back to live namespace sampling (same as having no declared schema). +- **Publishing:** Alliance ingestion (`cloud_rag/schema_update.py`) rebuilds and upserts manifests independently of document ingestion. + +Colon-format `PINECONE_SOURCES` plus remote manifest loading is the recommended staff setup — API keys and index names in `mcp.json`, descriptions and schemas in Pinecone. ### MCP tools and routing @@ -184,7 +184,7 @@ Multi-source mode supports two operational profiles. **Never** ship a merged int **External MCP config (single source, unchanged):** see [Quick start](#quick-start-single-pinecone-project) above. -**Internal MCP config (merged sources, JSON inline):** +**Internal MCP config (merged sources, colon format + remote schema):** ```json { @@ -195,19 +195,18 @@ Multi-source mode supports two operational profiles. **Never** ship a merged int "env": { "api_key_1": "pcsk_...", "api_key_2": "pcsk_...", - "PINECONE_SOURCES": { - "api_key_1": { "indexName": "rag-hybrid", "description": "..." }, - "api_key_2": { "indexName": "rag-hybrid" } - } + "PINECONE_SOURCES": "api_key_1:${api_key_1}:rag-hybrid;api_key_2:${api_key_2}:rag-hybrid" } } } } ``` -Colon format also works here: `"PINECONE_SOURCES": "api_key_1:${PINECONE_API_KEY_1}:index_name_1;..."`. +Descriptions and namespace schemas are loaded from each project's `_mcp_config` namespace at startup (see [Remote schema manifest](#remote-schema-manifest-_mcp_config)). Use `PINECONE_CONFIG_FILE` instead when you need local overrides. + +Colon format also supports explicit env indirection: `"PINECONE_SOURCES": "api_key_1:${PINECONE_API_KEY_1}:index_name_1;..."`. -Prefer JSON-shaped `PINECONE_SOURCES` or `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over embedding raw API keys directly. For internal deployments, add optional `description` and `namespaces` in JSON on staff machines only — never in public examples or committed constants. See [SECURITY.md](./SECURITY.md). +Prefer `${ENV_VAR}` indirection over embedding raw API keys directly. For internal deployments, optional `description` and `namespaces` in a JSON config file on staff machines only — never in public examples or committed constants. See [SECURITY.md](./SECURITY.md). --- diff --git a/docs/SECURITY.md b/docs/SECURITY.md index dd39e7e..5330067 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -25,7 +25,7 @@ This covers tool error payloads, hybrid degradation reasons, and SDK error text ## Private config content (descriptions and schemas) -Per-source `description` and per-namespace `namespaces` (including namespace names and `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` or JSON-shaped `PINECONE_SOURCES` are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles). +Per-source `description` and per-namespace `namespaces` (including namespace names and `metadata_schema` field names) loaded from `PINECONE_CONFIG_FILE` or the `_mcp_config` schema manifest are **High**-risk if committed to the open-source repo or shared through public distribution channels — they can disclose what private corpora contain and which internal metadata fields exist. Manifests in `_mcp_config` inherit the same access boundary as the rest of that Pinecone index; do not publish sensitive descriptions on indexes shared more broadly than the private data warrants. Keep real values on staff machines only; use generic placeholders in examples and tests. Same deployment-profile separation as API keys — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles). ## Docker image diff --git a/src/cli.ts b/src/cli.ts index 980a29e..2320447 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -117,6 +117,9 @@ export function parseCli(argv: string[] = process.argv.slice(2)): ParseCliResult case '--check-indexes': overrides.checkIndexes = true; break; + case '--disable-remote-schema': + overrides.disableRemoteSchema = true; + break; case '--sources': { const v = readOptionValue(argv, i); if (v !== undefined) { @@ -160,7 +163,8 @@ Options: --request-timeout-ms N Per Pinecone call timeout [env: PINECONE_REQUEST_TIMEOUT_MS] --disable-suggest-flow Bypass suggest_query_params gate (PINECONE_DISABLE_SUGGEST_FLOW) --check-indexes Verify dense + sparse indexes then exit 0/1 (PINECONE_CHECK_INDEXES) - --sources TEXT Multi-source inline config: JSON object or name:apiKey:indexName;... (PINECONE_SOURCES) + --disable-remote-schema Skip loading schema manifest from _mcp_config (PINECONE_DISABLE_REMOTE_SCHEMA) + --sources TEXT Multi-source inline config: name:apiKey:indexName;... (PINECONE_SOURCES) --config-file PATH Multi-source JSON config file (PINECONE_CONFIG_FILE) --help, -h Show this message --version, -v Print package version diff --git a/src/core/config.ts b/src/core/config.ts index 0f59a3d..b49affc 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -57,6 +57,8 @@ export interface ServerConfigBase { disableSuggestFlow: boolean; /** When true, on-startup probe verifies dense + sparse indexes exist. */ checkIndexes: boolean; + /** When true, skip loading schema manifest from `_mcp_config` at startup. */ + disableRemoteSchema: boolean; /** Named Pinecone sources when multi-project mode is enabled. */ sources?: SourceDefinition[]; /** Default source name when multi-source mode is enabled. */ @@ -150,6 +152,7 @@ export interface ConfigOverrides { requestTimeoutMs?: number; disableSuggestFlow?: boolean; checkIndexes?: boolean; + disableRemoteSchema?: boolean; sources?: string; configFile?: string; } @@ -196,6 +199,8 @@ export function resolveConfig( const disableSuggestFlow = overrides.disableSuggestFlow ?? asBool(env['PINECONE_DISABLE_SUGGEST_FLOW'], true); const checkIndexes = overrides.checkIndexes ?? asBool(env['PINECONE_CHECK_INDEXES'], false); + const disableRemoteSchema = + overrides.disableRemoteSchema ?? asBool(env['PINECONE_DISABLE_REMOTE_SCHEMA'], false); const multi = resolveSourceDefinitions(overrides, env, parseSourcesOptions); if (multi) { @@ -217,6 +222,7 @@ export function resolveConfig( requestTimeoutMs, disableSuggestFlow, checkIndexes, + disableRemoteSchema, sources: multi.sources, defaultSource: multi.defaultSource, }); @@ -254,6 +260,7 @@ export function resolveConfig( requestTimeoutMs, disableSuggestFlow, checkIndexes, + disableRemoteSchema, }); } diff --git a/src/core/pinecone-client.ts b/src/core/pinecone-client.ts index 1f7cd89..f86ceb6 100644 --- a/src/core/pinecone-client.ts +++ b/src/core/pinecone-client.ts @@ -90,6 +90,11 @@ export class PineconeClient { return this.indexSession.checkIndexes(); } + /** Fetch record fields from the dense index (metadata + top-level scalars). */ + async fetchRecordFields(namespace: string, id: string): Promise | null> { + return this.indexSession.fetchRecordFields(namespace, id); + } + private async searchIndex( index: SearchableIndex, query: string, diff --git a/src/core/pinecone/indexes.test.ts b/src/core/pinecone/indexes.test.ts index ac7deb2..e4d388f 100644 --- a/src/core/pinecone/indexes.test.ts +++ b/src/core/pinecone/indexes.test.ts @@ -270,4 +270,62 @@ describe('PineconeIndexSession', () => { expect(result.errors.some((e) => e.includes('no client'))).toBe(true); }); }); + + describe('fetchRecordFields', () => { + class FetchFieldsSession extends PineconeIndexSession { + constructor( + private readonly fetchResponse: { + records?: Record; [key: string]: unknown }>; + } + ) { + super('test-api-key', 'test-index'); + } + + override ensureClient() { + return { + index: () => ({ + fetch: vi.fn().mockResolvedValue(this.fetchResponse), + }), + } as never; + } + } + + it('merges metadata and top-level scalar fields', async () => { + const session = new FetchFieldsSession({ + records: { + schema_manifest: { + id: 'schema_manifest', + chunk_text: '{"ok":true}', + metadata: { extra: 'meta' }, + }, + }, + }); + + const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest'); + expect(fields).toMatchObject({ + extra: 'meta', + chunk_text: '{"ok":true}', + id: 'schema_manifest', + }); + }); + + it('returns null when record is missing', async () => { + const session = new FetchFieldsSession({ records: {} }); + const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest'); + expect(fields).toBeNull(); + }); + + it('returns metadata-only fields when chunk_text is only in metadata', async () => { + const session = new FetchFieldsSession({ + records: { + schema_manifest: { + metadata: { chunk_text: '{"from":"metadata"}' }, + }, + }, + }); + + const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest'); + expect(fields?.chunk_text).toBe('{"from":"metadata"}'); + }); + }); }); diff --git a/src/core/pinecone/indexes.ts b/src/core/pinecone/indexes.ts index c519c63..f97abed 100644 --- a/src/core/pinecone/indexes.ts +++ b/src/core/pinecone/indexes.ts @@ -229,6 +229,29 @@ export class PineconeIndexSession { } } + /** + * Fetch a record by id and return a flat field bag (metadata merged with top-level scalar fields). + * Mirrors Python `_extract_record_fields` for integrated-embedding indexes where `chunk_text` + * may appear on the record or inside metadata. + */ + async fetchRecordFields(namespace: string, id: string): Promise | null> { + const pc = this.ensureClient(); + const response = await pc.index(this.indexName).fetch({ ids: [id], namespace }); + const record = response.records?.[id] as + (Record & { metadata?: Record }) | undefined; + if (!record) { + return null; + } + const metadata = record.metadata ?? {}; + const merged: Record = { ...metadata }; + for (const [k, v] of Object.entries(record)) { + if (k !== 'metadata' && k !== 'values' && k !== 'sparseValues' && !(k in merged)) { + merged[k] = v; + } + } + return merged; + } + /** * Verify dense and sparse indexes are reachable (describeIndexStats). * Used by `--check-indexes` / `PINECONE_CHECK_INDEXES` before the server starts. diff --git a/src/core/server/remote-schema.test.ts b/src/core/server/remote-schema.test.ts new file mode 100644 index 0000000..e4a5dc0 --- /dev/null +++ b/src/core/server/remote-schema.test.ts @@ -0,0 +1,163 @@ +import { describe, expect, it, vi } from 'vitest'; +import type { PineconeClient } from '../pinecone-client.js'; +import type { SourceDefinition } from './source-config.js'; +import { + SCHEMA_MANIFEST_NAMESPACE, + SCHEMA_MANIFEST_RECORD_ID, + loadRemoteSchemaForSource, + loadRemoteSchemaForSources, +} from './remote-schema.js'; + +function baseDefinition(overrides?: Partial): SourceDefinition { + return { + name: 'api_key_1', + apiKey: 'key', + indexName: 'rag-hybrid', + sparseIndexName: 'rag-hybrid-sparse', + ...overrides, + }; +} + +function mockClient( + fetchImpl: (namespace: string, id: string) => Promise | null> +): PineconeClient { + return { + fetchRecordFields: vi.fn(fetchImpl), + } as unknown as PineconeClient; +} + +describe('remote-schema', () => { + it('merges description and namespaces from chunk_text in metadata', async () => { + const manifest = { + description: 'Corpus hint', + namespaces: { + mailing: { + description: 'Mailing threads', + metadata_schema: { doc_id: 'string' }, + record_count: 100, + }, + }, + }; + const client = mockClient(async () => ({ + chunk_text: JSON.stringify(manifest), + })); + + const { definition, warning } = await loadRemoteSchemaForSource(client, baseDefinition()); + + expect(warning).toBeUndefined(); + expect(definition.description).toBe('Corpus hint'); + expect(definition.namespaces?.mailing).toMatchObject({ + description: 'Mailing threads', + metadata_schema: { doc_id: 'string' }, + }); + expect(client.fetchRecordFields).toHaveBeenCalledWith( + SCHEMA_MANIFEST_NAMESPACE, + SCHEMA_MANIFEST_RECORD_ID + ); + }); + + it('reads chunk_text when returned as a top-level field', async () => { + const manifest = { + namespaces: { + docs: { metadata_schema: { title: 'string' } }, + }, + }; + const client = mockClient(async () => ({ + id: SCHEMA_MANIFEST_RECORD_ID, + chunk_text: JSON.stringify(manifest), + })); + + const { definition } = await loadRemoteSchemaForSource(client, baseDefinition()); + + expect(definition.namespaces?.docs?.metadata_schema).toEqual({ title: 'string' }); + }); + + it('returns unchanged when record is missing (no warning)', async () => { + const client = mockClient(async () => null); + const input = baseDefinition(); + + const { definition, warning } = await loadRemoteSchemaForSource(client, input); + + expect(warning).toBeUndefined(); + expect(definition).toEqual(input); + }); + + it('returns warning on malformed JSON in chunk_text', async () => { + const client = mockClient(async () => ({ chunk_text: '{not json' })); + const input = baseDefinition(); + + const { definition, warning } = await loadRemoteSchemaForSource(client, input); + + expect(definition).toEqual(input); + expect(warning).toMatch(/malformed \(invalid JSON/); + }); + + it('returns warning when chunk_text is valid JSON but namespaces has the wrong shape', async () => { + const client = mockClient(async () => ({ + chunk_text: JSON.stringify({ namespaces: ['not', 'an', 'object'] }), + })); + const input = baseDefinition(); + + const { definition, warning } = await loadRemoteSchemaForSource(client, input); + + expect(definition).toEqual(input); + expect(warning).toMatch(/malformed \(manifest "namespaces" must be an object\)/); + }); + + it('returns warning when fetch throws', async () => { + const client = mockClient(async () => { + throw new Error('network down'); + }); + const input = baseDefinition(); + + const { definition, warning } = await loadRemoteSchemaForSource(client, input); + + expect(definition).toEqual(input); + expect(warning).toMatch(/failed to fetch/); + }); + + it('skips fetch when local namespaces are already set', async () => { + const fetchFn = vi.fn(async () => ({ chunk_text: '{}' })); + const client = mockClient(fetchFn); + const input = baseDefinition({ + namespaces: { local_ns: { description: 'local' } }, + }); + + const { definition, warning } = await loadRemoteSchemaForSource(client, input); + + expect(fetchFn).not.toHaveBeenCalled(); + expect(warning).toBeUndefined(); + expect(definition).toEqual(input); + }); + + it('keeps local description when merging remote namespaces', async () => { + const client = mockClient(async () => ({ + chunk_text: JSON.stringify({ + description: 'remote description', + namespaces: { ns1: { metadata_schema: { a: 'string' } } }, + }), + })); + const input = baseDefinition({ description: 'local description' }); + + const { definition } = await loadRemoteSchemaForSource(client, input); + + expect(definition.description).toBe('local description'); + expect(definition.namespaces?.ns1?.metadata_schema).toEqual({ a: 'string' }); + }); + + it('loadRemoteSchemaForSources processes multiple entries', async () => { + const client1 = mockClient(async () => ({ + chunk_text: JSON.stringify({ description: 'source one' }), + })); + const client2 = mockClient(async () => null); + + const result = await loadRemoteSchemaForSources([ + { definition: baseDefinition({ name: 's1' }), client: client1 }, + { definition: baseDefinition({ name: 's2' }), client: client2 }, + ]); + + expect(result.definitions).toHaveLength(2); + expect(result.definitions[0]?.description).toBe('source one'); + expect(result.warnings).toHaveLength(0); + }); +}); diff --git a/src/core/server/remote-schema.ts b/src/core/server/remote-schema.ts new file mode 100644 index 0000000..441ab11 --- /dev/null +++ b/src/core/server/remote-schema.ts @@ -0,0 +1,128 @@ +/** + * Load per-source description and namespace declarations from the reserved + * `_mcp_config` Pinecone namespace (schema manifest record). + */ + +import type { PineconeClient } from '../pinecone-client.js'; +import { trimOptional } from '../config.js'; +import type { SourceDefinition } from './source-config.js'; +import { validateNamespaces } from './source-config.js'; + +export const SCHEMA_MANIFEST_NAMESPACE = '_mcp_config'; +export const SCHEMA_MANIFEST_RECORD_ID = 'schema_manifest'; + +type SchemaManifest = { + description?: string; + namespaces?: Record; +}; + +type ManifestParseResult = { ok: true; manifest: SchemaManifest } | { ok: false; reason: string }; + +function parseManifestChunkText(chunkText: string): ManifestParseResult { + let parsed: unknown; + try { + parsed = JSON.parse(chunkText); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + return { ok: false, reason: `invalid JSON: ${message}` }; + } + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { + return { ok: false, reason: 'manifest must be a JSON object' }; + } + const obj = parsed as Record; + const description = + typeof obj['description'] === 'string' ? trimOptional(obj['description']) : undefined; + const namespacesRaw = obj['namespaces']; + if ( + namespacesRaw != null && + (typeof namespacesRaw !== 'object' || Array.isArray(namespacesRaw)) + ) { + return { ok: false, reason: 'manifest "namespaces" must be an object' }; + } + return { + ok: true, + manifest: { + ...(description !== undefined ? { description } : {}), + ...(namespacesRaw != null ? { namespaces: namespacesRaw as Record } : {}), + }, + }; +} + +export async function loadRemoteSchemaForSource( + client: PineconeClient, + definition: SourceDefinition +): Promise<{ definition: SourceDefinition; warning?: string }> { + if (definition.namespaces && Object.keys(definition.namespaces).length > 0) { + return { definition }; + } + + let fields: Record | null; + try { + fields = await client.fetchRecordFields(SCHEMA_MANIFEST_NAMESPACE, SCHEMA_MANIFEST_RECORD_ID); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + return { + definition, + warning: `Source "${definition.name}": failed to fetch _mcp_config schema manifest: ${message}`, + }; + } + + if (!fields) { + return { definition }; + } + + const chunkText = fields['chunk_text']; + if (typeof chunkText !== 'string' || !chunkText.trim()) { + return { definition }; + } + + const parseResult = parseManifestChunkText(chunkText); + if (!parseResult.ok) { + return { + definition, + warning: `Source "${definition.name}": _mcp_config schema manifest in chunk_text is malformed (${parseResult.reason}).`, + }; + } + const manifest = parseResult.manifest; + + let namespaces: SourceDefinition['namespaces']; + if (manifest.namespaces != null) { + try { + namespaces = validateNamespaces(definition.name, manifest.namespaces); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + return { + definition, + warning: `Source "${definition.name}": _mcp_config schema manifest namespaces invalid: ${message}`, + }; + } + } + + const description = + definition.description !== undefined ? definition.description : manifest.description; + + const enriched: SourceDefinition = { + ...definition, + ...(description !== undefined ? { description } : {}), + ...(namespaces !== undefined ? { namespaces } : {}), + }; + + return { definition: enriched }; +} + +export async function loadRemoteSchemaForSources( + entries: { definition: SourceDefinition; client: PineconeClient }[] +): Promise<{ definitions: SourceDefinition[]; warnings: string[] }> { + const results = await Promise.all( + entries.map(async ({ definition, client }) => loadRemoteSchemaForSource(client, definition)) + ); + const warnings: string[] = []; + const definitions: SourceDefinition[] = []; + for (const result of results) { + definitions.push(result.definition); + if (result.warning) { + warnings.push(result.warning); + } + } + return { definitions, warnings }; +} diff --git a/src/core/server/server-context.composition.test.ts b/src/core/server/server-context.composition.test.ts index 4de0258..2ee7ea8 100644 --- a/src/core/server/server-context.composition.test.ts +++ b/src/core/server/server-context.composition.test.ts @@ -51,6 +51,35 @@ describe('ServerContext composition API', () => { expect(flow.ok).toBe(true); }); + it('declaredNamespaces composition seed is used in single-key getNamespacesWithCache', async () => { + const listNamespaces = vi.fn().mockResolvedValue( + mockNamespacesWithMetadataResult([ + { + namespace: 'mailing', + recordCount: 10, + metadata: { doc_id: 'string' }, + schema_source: 'declared', + }, + ]) + ); + const client = { listNamespacesWithMetadata: listNamespaces } as unknown as PineconeClient; + + const ctx = new ServerContext(testConfig(), { + client, + declaredNamespaces: { + mailing: { + description: 'Mailing list', + metadata_schema: { doc_id: 'string' }, + }, + }, + }); + + const result = await ctx.getNamespacesWithCache(); + expect(listNamespaces).toHaveBeenCalledWith({ mailing: { doc_id: 'string' } }, ['mailing']); + expect(result.data[0]?.description).toBe('Mailing list'); + expect(result.data[0]?.schema_source).toBe('declared'); + }); + it('constructs with suggestionFlowSeed and no config without throwing', () => { expect( () => diff --git a/src/core/server/server-context.ts b/src/core/server/server-context.ts index 9ef3f21..df7c7c6 100644 --- a/src/core/server/server-context.ts +++ b/src/core/server/server-context.ts @@ -12,6 +12,7 @@ import type { RecommendedTool } from './query-suggestion.js'; import type { UrlGenerationResult, UrlGeneratorFn } from './url-registry.js'; import { buildSourceRegistry, type SourceRegistry } from './source-registry.js'; import { fetchNamespacesWithDeclaredConfig, type NamespacesCacheEntry } from './namespace-cache.js'; +import type { NamespaceDeclaration } from './source-config.js'; export type NamespaceInfo = { namespace: string; @@ -64,6 +65,8 @@ export interface ServerContextComposition { urlGenerators?: Iterable; namespaceCacheSeed?: NamespaceCacheSeed; suggestionFlowSeed?: SuggestionFlowSeedEntry[]; + /** Remote or file-loaded namespace declarations for single-key mode. */ + declaredNamespaces?: Record; } type FlowState = { @@ -123,9 +126,11 @@ export class ServerContext< private readonly urlGenerators = new Map(); private readonly suggestionFlow = new Map(); private namespacesCache: NamespacesCacheEntry | null = null; + private readonly declaredNamespaces?: Record; constructor(config?: T, composition?: ServerContextComposition, init?: ServerContextInitOptions) { this.unconfiguredAlliance = init?.unconfiguredAlliance ?? false; + this.declaredNamespaces = composition?.declaredNamespaces; if (composition?.client && composition?.sourceRegistry) { throw new Error('Cannot pass both client and sourceRegistry in ServerContextComposition.'); } @@ -622,7 +627,10 @@ export class ServerContext< } const cfg = this.getConfig(); - const { data, warnings } = await fetchNamespacesWithDeclaredConfig(this.getClient()); + const { data, warnings } = await fetchNamespacesWithDeclaredConfig( + this.getClient(), + this.declaredNamespaces + ); const expiresAt = now + cfg.cacheTtlMs; this.namespacesCache = { data, expiresAt, warnings }; return { diff --git a/src/core/server/source-config.test.ts b/src/core/server/source-config.test.ts index 37c6861..95ae216 100644 --- a/src/core/server/source-config.test.ts +++ b/src/core/server/source-config.test.ts @@ -222,49 +222,6 @@ describe('source-config', () => { expect(sources[0]).not.toHaveProperty('namespaces'); }); - it('resolveSourceDefinitions parses JSON sources-map with description and namespaces', () => { - const inline = JSON.stringify({ - api_key_1: { - indexName: 'index_name_1', - description: 'Staff corpus hint', - namespaces: { - example_ns: { - description: 'Namespace hint', - metadata_schema: { field_a: 'string' }, - }, - }, - }, - }); - const env = { api_key_1: 'api-1', PINECONE_SOURCES: inline }; - const parsed = resolveSourceDefinitions({}, env); - expect(parsed?.defaultSource).toBe('api_key_1'); - expect(parsed?.sources[0]).toMatchObject({ - name: 'api_key_1', - apiKey: 'api-1', - indexName: 'index_name_1', - description: 'Staff corpus hint', - }); - expect(parsed?.sources[0]?.namespaces?.example_ns).toMatchObject({ - description: 'Namespace hint', - metadata_schema: { field_a: 'string' }, - }); - }); - - it('resolveSourceDefinitions parses full JSON file shape inline', () => { - const inline = JSON.stringify({ - defaultSource: 'api_key_2', - sources: { - api_key_1: { apiKey: '${K1}', indexName: 'index_name_1' }, - api_key_2: { apiKey: '${K2}', indexName: 'index_name_2' }, - }, - }); - const env = { K1: 'api-1', K2: 'api-2', PINECONE_SOURCES: inline }; - const parsed = resolveSourceDefinitions({}, env); - expect(parsed?.defaultSource).toBe('api_key_2'); - expect(parsed?.sources).toHaveLength(2); - expect(parsed?.sources.find((s) => s.name === 'api_key_2')?.apiKey).toBe('api-2'); - }); - it('resolveSourceDefinitions colon format still works (regression)', () => { const env = { K1: 'api-1', PINECONE_SOURCES: 'api_key_1:${K1}:index_name_1' }; const parsed = resolveSourceDefinitions({}, env); @@ -276,14 +233,16 @@ describe('source-config', () => { expect(parsed?.sources[0]).not.toHaveProperty('description'); }); - it('resolveSourceDefinitions throws on invalid JSON PINECONE_SOURCES', () => { - const env = { PINECONE_SOURCES: '{not valid json' }; + it('resolveSourceDefinitions throws when PINECONE_SOURCES is inline JSON', () => { + const env = { + PINECONE_SOURCES: JSON.stringify({ api_key_1: { indexName: 'index_name_1' } }), + }; expect(() => resolveSourceDefinitions({}, env)).toThrow( - /Failed to parse PINECONE_SOURCES JSON/ + /PINECONE_SOURCES no longer accepts inline JSON/ ); }); - it('resolveSourceDefinitions prefers config file over JSON PINECONE_SOURCES', () => { + it('resolveSourceDefinitions prefers config file over colon PINECONE_SOURCES', () => { const dir = mkdtempSync(join(tmpdir(), 'pinecone-sources-')); const filePath = join(dir, 'sources.json'); writeFileSync( @@ -296,9 +255,7 @@ describe('source-config', () => { }) ); const env = { - PINECONE_SOURCES: JSON.stringify({ - from_inline: { apiKey: 'inline-key', indexName: 'inline-index' }, - }), + PINECONE_SOURCES: 'from_inline:sk:inline-index', PINECONE_CONFIG_FILE: filePath, }; const parsed = resolveSourceDefinitions({}, env); @@ -306,40 +263,40 @@ describe('source-config', () => { expect(parsed?.sources[0]?.indexName).toBe('file-index'); }); - it('resolveSourceDefinitions throws when defaulted apiKey env is missing', () => { - const inline = JSON.stringify({ - api_key_1: { indexName: 'index_name_1' }, - }); - const env = { PINECONE_SOURCES: inline }; - expect(() => resolveSourceDefinitions({}, env)).toThrow( + it('parseSourcesConfigFile throws when defaulted apiKey env is missing', () => { + const dir = mkdtempSync(join(tmpdir(), 'pinecone-sources-')); + const filePath = join(dir, 'defaulted-apikey.json'); + writeFileSync( + filePath, + JSON.stringify({ + defaultSource: 'api_key_1', + sources: { + api_key_1: { indexName: 'index_name_1' }, + }, + }) + ); + expect(() => parseSourcesConfigFile(filePath, {})).toThrow( /Environment variable api_key_1 is not set/ ); }); - it('resolveSourceDefinitions throws when hyphenated source name uses defaulted apiKey', () => { - const inline = JSON.stringify({ - 'internal-corpus': { indexName: 'index_name_1' }, - }); - const env = { PINECONE_SOURCES: inline }; - expect(() => resolveSourceDefinitions({}, env)).toThrow( + it('parseSourcesConfigFile throws when hyphenated source name uses defaulted apiKey', () => { + const dir = mkdtempSync(join(tmpdir(), 'pinecone-sources-')); + const filePath = join(dir, 'hyphenated.json'); + writeFileSync( + filePath, + JSON.stringify({ + defaultSource: 'internal-corpus', + sources: { + 'internal-corpus': { indexName: 'index_name_1' }, + }, + }) + ); + expect(() => parseSourcesConfigFile(filePath, {})).toThrow( /Invalid environment variable reference "\$\{internal-corpus\}"/ ); }); - it('resolveSourceDefinitions treats bare map with source named sources as sources-map', () => { - const inline = JSON.stringify({ - sources: { indexName: 'rag-hybrid' }, - }); - const env = { sources: 'api-key-for-sources', PINECONE_SOURCES: inline }; - const parsed = resolveSourceDefinitions({}, env); - expect(parsed?.sources).toHaveLength(1); - expect(parsed?.sources[0]).toMatchObject({ - name: 'sources', - apiKey: 'api-key-for-sources', - indexName: 'rag-hybrid', - }); - }); - it('resolveConfig uses PINECONE_SOURCES when set', () => { vi.stubEnv('PINECONE_SOURCES', 'api_key_1:sk-test:my-index'); vi.stubEnv('PINECONE_API_KEY', 'ignored'); diff --git a/src/core/server/source-config.ts b/src/core/server/source-config.ts index 609d121..519e4d5 100644 --- a/src/core/server/source-config.ts +++ b/src/core/server/source-config.ts @@ -1,12 +1,12 @@ /** - * Multi-source Pinecone configuration parsing (inline PINECONE_SOURCES and JSON config file). + * Multi-source Pinecone configuration parsing (colon PINECONE_SOURCES and JSON config file). */ import { readFileSync } from 'node:fs'; import { resolve } from 'node:path'; import { trimOptional } from '../config.js'; -/** Per-namespace declaration loaded from JSON config file or JSON-shaped PINECONE_SOURCES. */ +/** Per-namespace declaration loaded from JSON config file or remote `_mcp_config` manifest. */ export interface NamespaceDeclaration { description?: string; metadata_schema?: Record; @@ -19,9 +19,9 @@ export interface SourceDefinition { indexName: string; sparseIndexName?: string; rerankModel?: string; - /** Optional corpus-level description (JSON config file or JSON-shaped PINECONE_SOURCES only). */ + /** Optional corpus-level description (JSON config file or remote `_mcp_config` manifest). */ description?: string; - /** Optional per-namespace declarations (JSON config file or JSON-shaped PINECONE_SOURCES only). */ + /** Optional per-namespace declarations (JSON config file or remote `_mcp_config` manifest). */ namespaces?: Record; } @@ -64,7 +64,7 @@ function validateSourceName(name: string): void { } } -function validateNamespaces( +export function validateNamespaces( sourceName: string, raw: unknown ): Record | undefined { @@ -233,7 +233,7 @@ function defaultApiKeyForSource(name: string, rawApiKey: unknown): string { return trimmed ?? `\${${name}}`; } -/** Parse a JSON sources object (config file or inline PINECONE_SOURCES JSON). */ +/** Parse a JSON sources object (config file). */ export function parseSourcesConfigObject( parsed: JsonSourceFile, env: NodeJS.ProcessEnv = process.env, @@ -290,39 +290,6 @@ export function parseSourcesConfigObject( return { sources, defaultSource }; } -function looksLikeSourceEntry(v: unknown): boolean { - return !!v && typeof v === 'object' && !Array.isArray(v); -} - -function parseJsonSourcesInline( - inline: string, - env: NodeJS.ProcessEnv, - options?: ParseSourcesOptions -): { sources: SourceDefinition[]; defaultSource: string } { - let parsed: unknown; - try { - parsed = JSON.parse(inline); - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - throw new Error(`Failed to parse PINECONE_SOURCES JSON: ${message}`); - } - if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { - throw new Error('PINECONE_SOURCES JSON must be a JSON object.'); - } - const obj = parsed as Record; - const sourcesCandidate = obj['sources']; - const hasFileShapeSourcesField = - sourcesCandidate != null && - typeof sourcesCandidate === 'object' && - !Array.isArray(sourcesCandidate) && - Object.values(sourcesCandidate as Record).every(looksLikeSourceEntry); - const fileShape = - obj['defaultSource'] != null || hasFileShapeSourcesField - ? (parsed as JsonSourceFile) - : ({ sources: parsed } as JsonSourceFile); - return parseSourcesConfigObject(fileShape, env, options, 'PINECONE_SOURCES JSON'); -} - /** Parse JSON config file for multi-source setup. */ export function parseSourcesConfigFile( filePath: string, @@ -356,7 +323,9 @@ export function resolveSourceDefinitions( } if (inline) { if (inline.startsWith('{')) { - return parseJsonSourcesInline(inline, env, options); + throw new Error( + 'PINECONE_SOURCES no longer accepts inline JSON. Use PINECONE_CONFIG_FILE for per-source description/namespaces, or rely on automatic "_mcp_config" schema loading (see docs/CONFIGURATION.md).' + ); } const sources = parseInlineSources(inline, env, options); return { sources, defaultSource: sources[0]!.name }; diff --git a/src/index.ts b/src/index.ts index c231539..875732a 100644 --- a/src/index.ts +++ b/src/index.ts @@ -15,9 +15,15 @@ import { resolveAllianceConfig } from './alliance/config.js'; import { PineconeClient } from './core/pinecone-client.js'; import { createServer } from './core/server/server-context.js'; import { buildSourceRegistry } from './core/server/source-registry.js'; +import { + loadRemoteSchemaForSource, + loadRemoteSchemaForSources, +} from './core/server/remote-schema.js'; +import type { NamespaceDeclaration } from './core/server/source-config.js'; import { setupAllianceServer } from './alliance/setup.js'; import { error as logError, + info as logInfo, redactApiKey, setLogFormat, setLogLevel, @@ -65,14 +71,53 @@ async function main(): Promise { let ctx; if (config.sources && config.sources.length > 0) { + const clients = new Map(); + for (const source of config.sources) { + clients.set( + source.name, + new PineconeClient({ + apiKey: source.apiKey, + indexName: source.indexName, + sparseIndexName: source.sparseIndexName, + rerankModel: source.rerankModel, + defaultTopK: config.defaultTopK, + requestTimeoutMs: config.requestTimeoutMs, + }) + ); + } + + let sources = config.sources; + const sourcesNeedingRemoteSchema = sources.filter( + (s) => !s.namespaces || Object.keys(s.namespaces).length === 0 + ); + if ( + !config.disableRemoteSchema && + !config.checkIndexes && + sourcesNeedingRemoteSchema.length > 0 + ) { + logInfo( + `Loading _mcp_config schema manifest for ${sourcesNeedingRemoteSchema.length} source(s) without local namespace declarations...` + ); + const entries = sources.map((definition) => ({ + definition, + client: clients.get(definition.name)!, + })); + const loaded = await loadRemoteSchemaForSources(entries); + sources = loaded.definitions; + for (const warning of loaded.warnings) { + logWarn(redactApiKey(warning)); + } + } + const sourceRegistry = buildSourceRegistry({ - sources: config.sources, - defaultSource: config.defaultSource ?? config.sources[0]!.name, + sources, + defaultSource: config.defaultSource ?? sources[0]!.name, cacheTtlMs: config.cacheTtlMs, defaultTopK: config.defaultTopK, requestTimeoutMs: config.requestTimeoutMs, + clients, }); - ctx = createServer(config, { sourceRegistry }); + ctx = createServer({ ...config, sources }, { sourceRegistry }); } else { const client = new PineconeClient({ apiKey: config.apiKey, @@ -82,7 +127,24 @@ async function main(): Promise { defaultTopK: config.defaultTopK, requestTimeoutMs: config.requestTimeoutMs, }); - ctx = createServer(config, { client }); + + let declaredNamespaces: Record | undefined; + if (!config.disableRemoteSchema && !config.checkIndexes) { + logInfo('Loading _mcp_config schema manifest...'); + const loaded = await loadRemoteSchemaForSource(client, { + name: 'default', + apiKey: config.apiKey, + indexName: config.indexName, + sparseIndexName: config.sparseIndexName, + ...(config.rerankModel !== undefined ? { rerankModel: config.rerankModel } : {}), + }); + if (loaded.warning) { + logWarn(redactApiKey(loaded.warning)); + } + declaredNamespaces = loaded.definition.namespaces; + } + + ctx = createServer(config, { client, declaredNamespaces }); } if (config.checkIndexes) { From 436ad2d8c22705aed6de95469388df22f604b40e Mon Sep 17 00:00:00 2001 From: zho Date: Wed, 15 Jul 2026 02:24:36 +0800 Subject: [PATCH 5/8] addressed ai reviews --- src/core/pinecone-client.ts | 4 ++- src/core/pinecone/indexes.test.ts | 21 ++++++++++++++++ src/core/pinecone/indexes.ts | 41 +++++++++++++++++++------------ 3 files changed, 49 insertions(+), 17 deletions(-) diff --git a/src/core/pinecone-client.ts b/src/core/pinecone-client.ts index f86ceb6..6c67e40 100644 --- a/src/core/pinecone-client.ts +++ b/src/core/pinecone-client.ts @@ -15,6 +15,7 @@ import type { HybridLegFailed, } from '../types.js'; import { DEFAULT_TOP_K, MAX_TOP_K, COUNT_TOP_K, COUNT_FIELDS } from '../constants.js'; +import { DEFAULT_REQUEST_TIMEOUT_MS } from './config.js'; import { PineconeIndexSession, type NamespacesWithMetadataResult } from './pinecone/indexes.js'; import { countUniqueDocumentsFromHits, @@ -39,7 +40,8 @@ export class PineconeClient { this.indexSession = new PineconeIndexSession( config.apiKey, config.indexName, - config.sparseIndexName + config.sparseIndexName, + config.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS ); const normalizedRerankModel = config.rerankModel?.trim(); this.rerankModel = normalizedRerankModel ? normalizedRerankModel : undefined; diff --git a/src/core/pinecone/indexes.test.ts b/src/core/pinecone/indexes.test.ts index e4d388f..1b77fd4 100644 --- a/src/core/pinecone/indexes.test.ts +++ b/src/core/pinecone/indexes.test.ts @@ -327,5 +327,26 @@ describe('PineconeIndexSession', () => { const fields = await session.fetchRecordFields('_mcp_config', 'schema_manifest'); expect(fields?.chunk_text).toBe('{"from":"metadata"}'); }); + + it('rejects when fetch exceeds requestTimeoutMs', async () => { + class HangingFetchSession extends PineconeIndexSession { + constructor() { + super('test-api-key', 'test-index', undefined, 50); + } + + override ensureClient() { + return { + index: () => ({ + fetch: () => new Promise(() => {}), + }), + } as never; + } + } + + const session = new HangingFetchSession(); + await expect(session.fetchRecordFields('_mcp_config', 'schema_manifest')).rejects.toThrow( + /Timeout after 50ms while waiting for fetchRecordFields/ + ); + }); }); }); diff --git a/src/core/pinecone/indexes.ts b/src/core/pinecone/indexes.ts index f97abed..998f744 100644 --- a/src/core/pinecone/indexes.ts +++ b/src/core/pinecone/indexes.ts @@ -3,6 +3,8 @@ */ import { Pinecone } from '@pinecone-database/pinecone'; +import { DEFAULT_REQUEST_TIMEOUT_MS } from '../config.js'; +import { withTimeout } from '../server/retry.js'; import { error as logError, info as logInfo } from '../../logger.js'; import type { KeywordIndexNamespacesResult, @@ -46,7 +48,8 @@ export class PineconeIndexSession { constructor( private readonly apiKey: string, private readonly indexName: string, - private readonly sparseIndexName?: string + private readonly sparseIndexName?: string, + private readonly requestTimeoutMs: number = DEFAULT_REQUEST_TIMEOUT_MS ) {} /** Sparse index name; defaults to `{indexName}-sparse`. */ @@ -235,21 +238,27 @@ export class PineconeIndexSession { * may appear on the record or inside metadata. */ async fetchRecordFields(namespace: string, id: string): Promise | null> { - const pc = this.ensureClient(); - const response = await pc.index(this.indexName).fetch({ ids: [id], namespace }); - const record = response.records?.[id] as - (Record & { metadata?: Record }) | undefined; - if (!record) { - return null; - } - const metadata = record.metadata ?? {}; - const merged: Record = { ...metadata }; - for (const [k, v] of Object.entries(record)) { - if (k !== 'metadata' && k !== 'values' && k !== 'sparseValues' && !(k in merged)) { - merged[k] = v; - } - } - return merged; + return withTimeout( + async (_signal) => { + const pc = this.ensureClient(); + const response = await pc.index(this.indexName).fetch({ ids: [id], namespace }); + const record = response.records?.[id] as + | (Record & { metadata?: Record }) + | undefined; + if (!record) { + return null; + } + const metadata = record.metadata ?? {}; + const merged: Record = { ...metadata }; + for (const [k, v] of Object.entries(record)) { + if (k !== 'metadata' && k !== 'values' && k !== 'sparseValues' && !(k in merged)) { + merged[k] = v; + } + } + return merged; + }, + { timeoutMs: this.requestTimeoutMs, label: 'fetchRecordFields' } + ); } /** From 3a94e6a966fe1f9489855863a51db583f885bfc5 Mon Sep 17 00:00:00 2001 From: zho Date: Wed, 15 Jul 2026 02:33:58 +0800 Subject: [PATCH 6/8] fixed format error --- src/core/pinecone/indexes.ts | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/core/pinecone/indexes.ts b/src/core/pinecone/indexes.ts index 998f744..65beb0c 100644 --- a/src/core/pinecone/indexes.ts +++ b/src/core/pinecone/indexes.ts @@ -243,8 +243,7 @@ export class PineconeIndexSession { const pc = this.ensureClient(); const response = await pc.index(this.indexName).fetch({ ids: [id], namespace }); const record = response.records?.[id] as - | (Record & { metadata?: Record }) - | undefined; + (Record & { metadata?: Record }) | undefined; if (!record) { return null; } From 49fe850479a3b36e81864932c0151247b830bc4d Mon Sep 17 00:00:00 2001 From: zho Date: Wed, 15 Jul 2026 03:10:49 +0800 Subject: [PATCH 7/8] One-line JSDoc on loadRemoteSchemaForSource and loadRemoteSchemaForSources --- docs/CONFIGURATION.md | 37 ++++++++++++++++++++++----- src/cli.test.ts | 8 ++++++ src/core/pinecone-client.test.ts | 15 +++++++++++ src/core/server/remote-schema.test.ts | 24 +++++++++++++++++ src/core/server/remote-schema.ts | 2 ++ src/core/server/source-config.test.ts | 17 ++++++++++++ 6 files changed, 96 insertions(+), 7 deletions(-) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 9379fee..9a4cf32 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -59,7 +59,7 @@ Need more than one Pinecone project in the same MCP entry? See [Multi-source mod | `cacheTtlMs` | `--cache-ttl-seconds` | `PINECONE_CACHE_TTL_SECONDS` | `1800` seconds → ms | | `requestTimeoutMs` | `--request-timeout-ms` | `PINECONE_REQUEST_TIMEOUT_MS` | `15000` | | `disableSuggestFlow` | `--disable-suggest-flow` | `PINECONE_DISABLE_SUGGEST_FLOW` | **Core:** `true` (gate off). **Alliance:** `false` (gate on). Bool parsing: `true`/`1`/`yes`/`on` | -| `checkIndexes` | `--check-indexes` | `PINECONE_CHECK_INDEXES` | `false` | +| `checkIndexes` | `--check-indexes` | `PINECONE_CHECK_INDEXES` | `false`; when `true`, probes indexes and exits (skips remote `_mcp_config` loading) | | `disableRemoteSchema` | `--disable-remote-schema` | `PINECONE_DISABLE_REMOTE_SCHEMA` | `false` (remote `_mcp_config` loading on) | | `sources` | `--sources` | `PINECONE_SOURCES` | Multi-source only; see below | | `configFile` | `--config-file` | `PINECONE_CONFIG_FILE` | Multi-source only; see below | @@ -105,7 +105,9 @@ Semicolon-separated `name:apiKey:indexName` entries: PINECONE_SOURCES=api_key_1:${PINECONE_API_KEY_1}:index_name_1;api_key_2:${PINECONE_API_KEY_2}:index_name_2 ``` -API keys may contain colons; the parser treats the last `:` segment as `indexName` and everything between `name:` and `:indexName` as the key. This format never carries `description` or `namespaces` — use JSON for those. +API keys may contain colons; the parser treats the last `:` segment as `indexName` and everything between `name:` and `:indexName` as the key. This format never carries `description` or `namespaces` inline — use a [JSON config file](#json-config-file-descriptions--namespaces) or the [remote manifest](#remote-schema-manifest-_mcp_config). + +> **MCP host `env` values must be strings.** Hosts such as Cursor do not JSON-stringify nested objects in `env`. Set `PINECONE_SOURCES` as a single colon-format string (as above). Inline JSON objects or JSON-shaped `PINECONE_SOURCES` strings (values starting with `{`) are **rejected at startup** — use `PINECONE_CONFIG_FILE` or rely on `_mcp_config` instead. ### JSON config file (descriptions + namespaces) @@ -153,9 +155,24 @@ When a Pinecone project has a schema manifest upserted into the reserved `_mcp_c The manifest supplies optional source-level `description` and per-namespace `description` + `metadata_schema` (same effect as a config file). Local declarations always win; remote loading is skipped when local `namespaces` are set for that source. +The manifest body is stored in the record's `chunk_text` field as JSON with this shape: + +```json +{ + "description": "Optional source-level hint for list_sources", + "namespaces": { + "example_ns": { + "description": "Namespace hint shown by list_namespaces", + "metadata_schema": { "field_a": "string" } + } + } +} +``` + - **Opt out:** set `PINECONE_DISABLE_REMOTE_SCHEMA=true` or pass `--disable-remote-schema`. -- **Failures are non-fatal:** missing manifest, network errors, or malformed JSON log a warning and the server falls back to live namespace sampling (same as having no declared schema). -- **Publishing:** Alliance ingestion (`cloud_rag/schema_update.py`) rebuilds and upserts manifests independently of document ingestion. +- **Skipped with `--check-indexes`:** index probe mode exits after connectivity checks; remote manifest loading does not run. +- **Failures are non-fatal:** missing manifest, network errors, timeouts (`requestTimeoutMs`), or malformed JSON log a warning and the server falls back to live namespace sampling (same as having no declared schema). +- **Publishing:** Alliance ingestion rebuilds and upserts manifests into `_mcp_config` independently of document ingestion (see internal `cloud_rag/schema_update.py` when available). Colon-format `PINECONE_SOURCES` plus remote manifest loading is the recommended staff setup — API keys and index names in `mcp.json`, descriptions and schemas in Pinecone. @@ -213,10 +230,16 @@ Prefer `${ENV_VAR}` indirection over embedding raw API keys directly. For intern ## Library embedding 1. Build `ServerConfig` with `resolveConfig({ apiKey: '...', indexName: '...', ... })` or `resolveAllianceConfig(...)` for the full tool surface. -2. `const ctx = createServer(config)` then `ctx.setClient(new PineconeClient({ ... }))` (mirrors `src/index.ts`). -3. `await setupAllianceServer({ context: ctx })` (or `setupCoreServer({ context: ctx })` for generic tools only) then connect an MCP transport. +2. Create a `PineconeClient` and optionally enrich source definitions with remote schema **before** `createServer` (see below). +3. `const ctx = createServer(config, { client })` or pass a pre-built `sourceRegistry` for multi-source mode. +4. `await setupAllianceServer({ context: ctx })` (or `setupCoreServer({ context: ctx })` for generic tools only) then connect an MCP transport. + +**Remote `_mcp_config` loading** runs in the **CLI entry point** (`src/index.ts`), not inside `createServer`. To match CLI behavior in a custom embedder: + +- **Single-key:** call `loadRemoteSchemaForSource(client, definition)` and pass `declaredNamespaces: loaded.definition.namespaces` in `createServer` composition (when `disableRemoteSchema` is false). +- **Multi-source:** call `loadRemoteSchemaForSources(entries)` per source, then pass the enriched `sources` to `buildSourceRegistry` / `createServer`. -Pass `config` at setup only when the context is not yet configured; after `createServer` + `setClient`, pass `{ context: ctx }` only. +Pass `config` at setup only when the context is not yet configured; after `createServer` + client injection, pass `{ context: ctx }` only. See [README deployment model](../README.md#deployment-model), [examples/quickstart/README.md](../examples/quickstart/README.md) (generic), and [examples/alliance/library-embedding-demo.ts](../examples/alliance/library-embedding-demo.ts) (Alliance surface). diff --git a/src/cli.test.ts b/src/cli.test.ts index 21f6855..e9221da 100644 --- a/src/cli.test.ts +++ b/src/cli.test.ts @@ -36,4 +36,12 @@ describe('parseCli', () => { expect(r.overrides.defaultTopK).toBe(25); } }); + + it('parses --disable-remote-schema', () => { + const r = parseCli(['--disable-remote-schema']); + expect(r.kind).toBe('config'); + if (r.kind === 'config') { + expect(r.overrides.disableRemoteSchema).toBe(true); + } + }); }); diff --git a/src/core/pinecone-client.test.ts b/src/core/pinecone-client.test.ts index aec4edd..437a1ca 100644 --- a/src/core/pinecone-client.test.ts +++ b/src/core/pinecone-client.test.ts @@ -69,6 +69,21 @@ describe('PineconeClient', () => { }); }); + describe('fetchRecordFields', () => { + it('delegates to indexSession.fetchRecordFields', async () => { + const fields = { chunk_text: '{"ok":true}' }; + const fetchMock = vi.fn().mockResolvedValue(fields); + Object.assign(client, { + indexSession: { fetchRecordFields: fetchMock }, + }); + + const result = await client.fetchRecordFields('_mcp_config', 'schema_manifest'); + + expect(fetchMock).toHaveBeenCalledWith('_mcp_config', 'schema_manifest'); + expect(result).toEqual(fields); + }); + }); + describe('query', () => { it('should throw error for empty query', async () => { await expect( diff --git a/src/core/server/remote-schema.test.ts b/src/core/server/remote-schema.test.ts index e4a5dc0..74451c1 100644 --- a/src/core/server/remote-schema.test.ts +++ b/src/core/server/remote-schema.test.ts @@ -116,6 +116,30 @@ describe('remote-schema', () => { expect(warning).toMatch(/failed to fetch/); }); + it('returns unchanged when chunk_text is empty or whitespace (no warning)', async () => { + const client = mockClient(async () => ({ chunk_text: ' ' })); + const input = baseDefinition(); + + const { definition, warning } = await loadRemoteSchemaForSource(client, input); + + expect(warning).toBeUndefined(); + expect(definition).toEqual(input); + }); + + it('returns warning when manifest namespaces fail validation', async () => { + const client = mockClient(async () => ({ + chunk_text: JSON.stringify({ + namespaces: { ns1: { metadata_schema: { field_a: 123 } } }, + }), + })); + const input = baseDefinition(); + + const { definition, warning } = await loadRemoteSchemaForSource(client, input); + + expect(definition).toEqual(input); + expect(warning).toMatch(/namespaces invalid/); + }); + it('skips fetch when local namespaces are already set', async () => { const fetchFn = vi.fn(async () => ({ chunk_text: '{}' })); const client = mockClient(fetchFn); diff --git a/src/core/server/remote-schema.ts b/src/core/server/remote-schema.ts index 441ab11..3ea1049 100644 --- a/src/core/server/remote-schema.ts +++ b/src/core/server/remote-schema.ts @@ -48,6 +48,7 @@ function parseManifestChunkText(chunkText: string): ManifestParseResult { }; } +/** Fetch and merge `_mcp_config` manifest into a source when local `namespaces` are unset; failures are non-fatal warnings. */ export async function loadRemoteSchemaForSource( client: PineconeClient, definition: SourceDefinition @@ -110,6 +111,7 @@ export async function loadRemoteSchemaForSource( return { definition: enriched }; } +/** Load remote schema manifests for multiple sources in parallel, collecting per-source warnings. */ export async function loadRemoteSchemaForSources( entries: { definition: SourceDefinition; client: PineconeClient }[] ): Promise<{ definitions: SourceDefinition[]; warnings: string[] }> { diff --git a/src/core/server/source-config.test.ts b/src/core/server/source-config.test.ts index 95ae216..8ce60d0 100644 --- a/src/core/server/source-config.test.ts +++ b/src/core/server/source-config.test.ts @@ -159,6 +159,23 @@ describe('source-config', () => { expect(first?.namespaces).toBeUndefined(); }); + it('parseSourcesConfigFile throws when source entry is not an object', () => { + const dir = mkdtempSync(join(tmpdir(), 'pinecone-sources-')); + const filePath = join(dir, 'bad-source-entry.json'); + writeFileSync( + filePath, + JSON.stringify({ + defaultSource: 'api_key_1', + sources: { + api_key_1: 'not-an-object', + }, + }) + ); + expect(() => parseSourcesConfigFile(filePath, {})).toThrow( + /Source "api_key_1" in config file .* must be an object/ + ); + }); + it('parseSourcesConfigFile throws when source description is not a string', () => { const dir = mkdtempSync(join(tmpdir(), 'pinecone-sources-')); const filePath = join(dir, 'bad-source-description.json'); From 534cb02bdc12ea814ebfdec0aeb1910c86516ec9 Mon Sep 17 00:00:00 2001 From: zho Date: Wed, 15 Jul 2026 03:42:18 +0800 Subject: [PATCH 8/8] fixed CONFIGURATION.md --- docs/CONFIGURATION.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 9a4cf32..13585dc 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -234,9 +234,9 @@ Prefer `${ENV_VAR}` indirection over embedding raw API keys directly. For intern 3. `const ctx = createServer(config, { client })` or pass a pre-built `sourceRegistry` for multi-source mode. 4. `await setupAllianceServer({ context: ctx })` (or `setupCoreServer({ context: ctx })` for generic tools only) then connect an MCP transport. -**Remote `_mcp_config` loading** runs in the **CLI entry point** (`src/index.ts`), not inside `createServer`. To match CLI behavior in a custom embedder: +**Remote `_mcp_config` loading** runs in the **CLI entry point** (`src/index.ts`), not inside `createServer`. To match CLI behavior in a custom embedder, load remote schema only when `disableRemoteSchema` is false **and** `checkIndexes` is false (skip during `--check-indexes` probe mode): -- **Single-key:** call `loadRemoteSchemaForSource(client, definition)` and pass `declaredNamespaces: loaded.definition.namespaces` in `createServer` composition (when `disableRemoteSchema` is false). +- **Single-key:** call `loadRemoteSchemaForSource(client, definition)` and pass `declaredNamespaces: loaded.definition.namespaces` in `createServer` composition. - **Multi-source:** call `loadRemoteSchemaForSources(entries)` per source, then pass the enriched `sources` to `buildSourceRegistry` / `createServer`. Pass `config` at setup only when the context is not yet configured; after `createServer` + client injection, pass `{ context: ctx }` only.