Add multi-source Pinecone MCP support (#187)

* Implemented multi-api server

* update md and tests

* fixed lint error

* fixed format check

* fixed codeQl error

* added exit-on-failure.ts

* remove the words "private", "public"

* addressed ai reviews

* addressed all reviews

---------

Co-authored-by: zho <jornathanm910923@gmail.com>

Author: jonathanMLDev

.env.example

@@ -33,3 +33,7 @@ PINECONE_READ_ONLY_MCP_LOG_FORMAT=text
 
 # Optional: Request timeout for Pinecone calls, in milliseconds (default: 15000)
 # PINECONE_REQUEST_TIMEOUT_MS=15000
+
+# Optional: Multi-source mode (replaces PINECONE_API_KEY + PINECONE_INDEX_NAME when set)
+# PINECONE_SOURCES=api_key_1:${PINECONE_API_KEY_1}:index_name_1;api_key_2:${PINECONE_API_KEY_2}:index_name_2
+# Or: PINECONE_CONFIG_FILE=./pinecone-sources.json

CHANGELOG.md

@@ -8,6 +8,10 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release
 
 ## [Unreleased]
 
+### Added
+
+- **Multi-source mode:** configure multiple Pinecone API keys / indexes in one MCP server via `PINECONE_SOURCES`, `--sources`, or a JSON config file (`PINECONE_CONFIG_FILE` / `--config-file`). New `list_sources` tool (when more than one source is configured). Optional `source` parameter on discovery and query tools; `list_namespaces` aggregates across sources and tags each namespace with `source`. See [CONFIGURATION.md](docs/CONFIGURATION.md#multi-source-mode), [TOOLS.md](docs/TOOLS.md#multi-source-mode), and deployment profiles in [CONFIGURATION.md](docs/CONFIGURATION.md#deployment-profiles).
+
 ### Changed
 
 - **Library:** `resolveConfig()` returns `CoreServerConfig`; `resolveAllianceConfig()` returns `AllianceServerConfig`. `setupCoreServer` / `setupAllianceServer` accept only their respective branded config and context types (`CoreServerContext` / `AllianceServerContext`). `ServerConfig` remains an alias for `ServerConfigBase` on read paths (`ctx.getConfig()`). See [MIGRATION.md](docs/MIGRATION.md#unreleased-branded-serverconfig-types).

README.md

@@ -57,6 +57,7 @@ For successful `query`, `query_documents`, and `guided_query` payloads, **rerank
 - **Hybrid Search**: Combines dense and sparse embeddings for superior recall
 - **Semantic Reranking**: Uses BGE reranker model for improved precision
 - **Dynamic Namespace Discovery**: Automatically discovers available namespaces in your Pinecone index
+- **Multi-Source (optional)**: Connect multiple Pinecone projects in one MCP server via `PINECONE_SOURCES` or a JSON config file; use `list_sources` and per-result `source` tags for routing
 - **Metadata Filtering**: Supports optional metadata filters for refined searches
 - **Fast presets**: Lazy initialization, connection pooling, and efficient result merging; use the `query` tool `preset=fast | detailed | full` to trade latency vs quality (no published benchmarks yet — treat descriptions as qualitative).
 - **Production-oriented defaults**: Input validation, error handling, and configurable logging; upgrading from **0.1.x** — see [MIGRATION.md](docs/MIGRATION.md).
@@ -122,6 +123,8 @@ The codebase is split into two layers:
 
 You need a **Pinecone API key**. **Index** (`PINECONE_INDEX_NAME` or `--index-name`) is required for core/library use; the **published CLI** defaults to `rag-hybrid` when unset (Alliance deployment). Sparse index defaults to `{index}-sparse`. **Rerank:** set `PINECONE_RERANK_MODEL` to enable; the CLI defaults to `bge-reranker-v2-m3` when unset. See [docs/CONFIGURATION.md](docs/CONFIGURATION.md) (core vs Alliance table).
 
+**Multiple projects:** set `PINECONE_SOURCES` or `PINECONE_CONFIG_FILE` instead of a single `PINECONE_API_KEY` — see [Multi-source mode](docs/CONFIGURATION.md#multi-source-mode).
+
 Quick reference (published CLI / Alliance — core embedders require index, no index/rerank defaults):
 
 | Variable                            | Required                | Default (Alliance CLI)            |

benchmarks/latency.ts

@@ -12,6 +12,7 @@ import { performance } from 'node:perf_hooks';
 import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
 import { PineconeClient } from '../src/pinecone-client.js';
 import { setLogLevel } from '../src/logger.js';
+import { exitOnDemoFailure } from '../examples/lib/exit-on-failure.js';
 import { setPineconeClient } from '../src/server/client-context.js';
 import { invalidateNamespacesCache, getNamespacesWithCache } from '../src/server/namespaces-cache.js';
 import { registerGuidedQueryTool } from '../src/server/tools/guided-query-tool.js';
@@ -323,7 +324,4 @@ async function main(): Promise<void> {
   console.log(`Wrote ${baselinePath}`);
 }
 
-main().catch((err) => {
-  console.error(err);
-  process.exit(1);
-});
+main().catch(exitOnDemoFailure('latency benchmark'));

docs/CONFIGURATION.md

@@ -25,8 +25,10 @@ Configuration is built from **CLI flags** (when using the binary), **environment
 | `requestTimeoutMs` | `requestTimeoutMs` / `PINECONE_REQUEST_TIMEOUT_MS` | `15000` |
 | `disableSuggestFlow` | `disableSuggestFlow` / `PINECONE_DISABLE_SUGGEST_FLOW` | **Core `resolveConfig`:** `true` (gate off). **Alliance `resolveAllianceConfig` / CLI:** `false` (gate on). Bool parsing: true/1/yes/on |
 | `checkIndexes` | `checkIndexes` / `PINECONE_CHECK_INDEXES` | `false` |
+| `sources` | `sources` / `PINECONE_SOURCES` or JSON config file | Omitted in single-key mode; see [Multi-source mode](#multi-source-mode) |
+| `defaultSource` | JSON config `defaultSource` only | First source when using inline `PINECONE_SOURCES` |
 
-**Throws** if `apiKey` or `indexName` is missing after trim.
+**Throws** if `apiKey` or `indexName` is missing after trim (single-key mode). In multi-source mode, `PINECONE_API_KEY` is ignored when `PINECONE_SOURCES` or a config file is set; credentials come from each source entry.
 
 For the full Alliance tool surface (including `suggest_query_params`, `guided_query`, and built-in URL generators), import from `@will-cppa/pinecone-read-only-mcp/alliance` and use the three-step instance-first recipe at [Library embedding](#library-embedding) below.
 
@@ -43,6 +45,113 @@ C++ Alliance deployers can copy [examples/alliance/.env.example](../examples/all
 
 ---
 
+## Multi-source mode
+
+Use **one MCP server entry** with multiple Pinecone API keys / projects when `PINECONE_SOURCES`, `--sources`, or a JSON config file (`PINECONE_CONFIG_FILE` / `--config-file`) is set.
+
+### Inline format (`PINECONE_SOURCES` / `--sources`)
+
+Semicolon-separated entries: `name:apiKey:indexName`
+
+```bash
+PINECONE_SOURCES=api_key_1:${PINECONE_API_KEY_1}:index_name_1;api_key_2:${PINECONE_API_KEY_2}:index_name_2
+```
+
+API keys may contain colons; the parser treats the last `:` segment as `indexName` and everything between `name:` and `:indexName` as the key.
+
+### JSON config file
+
+Set `PINECONE_CONFIG_FILE` (or `--config-file`) to a path such as [examples/multi-source/pinecone-sources.json.example](../examples/multi-source/pinecone-sources.json.example):
+
+```json
+{
+  "defaultSource": "api_key_1",
+  "sources": {
+    "api_key_1": { "apiKey": "${PINECONE_API_KEY_1}", "indexName": "index_name_1" },
+    "api_key_2": { "apiKey": "${PINECONE_API_KEY_2}", "indexName": "index_name_2" }
+  }
+}
+```
+
+Values support `${ENV_VAR}` indirection (resolved at startup). Per-source `sparseIndexName` and `rerankModel` are optional; Alliance defaults apply when omitted.
+
+### MCP tools and routing
+
+| Tool | `source` parameter |
+| ---- | ------------------ |
+| `list_sources` | Registered only when more than one source is configured |
+| `list_namespaces`, `namespace_router` | Omit to aggregate all sources; results include `source` when tagged |
+| `query`, `count`, `query_documents`, `keyword_search`, `generate_urls`, `suggest_query_params`, `guided_query` | Omit when the namespace uniquely identifies one source; required when the same namespace exists on multiple sources |
+
+Discovery responses may include `source_errors` when one project fails but others succeed. Suggest-flow state uses compound keys `source:namespace` in multi-source mode.
+
+Single-key deployments (`PINECONE_API_KEY` + `PINECONE_INDEX_NAME` only) are unchanged — no `source` field on responses and no `list_sources` tool.
+
+### Deployment profiles
+
+Multi-source mode supports two operational profiles. **Never** ship a merged internal config through the same channel used for external partners.
+
+| Profile | Who | Config | Risk if mis-shared |
+| ------- | --- | ------ | ------------------ |
+| **External (public-only)** | External companies, public MCP distribution | `PINECONE_API_KEY` + `PINECONE_INDEX_NAME`, or `PINECONE_SOURCES` with **one** entry | Low — single public key only |
+| **Internal (merged)** | Staff machines with access to private data | `PINECONE_SOURCES` or JSON config with **two+** entries | **High** — private API key and private namespace names exposed |
+
+**External MCP config (unchanged):**
+
+```json
+{
+  "mcpServers": {
+    "pinecone-search": {
+      "command": "npx",
+      "args": ["-y", "@will-cppa/pinecone-read-only-mcp"],
+      "env": {
+        "PINECONE_API_KEY": "your-public-key",
+        "PINECONE_INDEX_NAME": "rag-hybrid"
+      }
+    }
+  }
+}
+```
+
+**Internal MCP config (merged sources):**
+
+```json
+{
+  "mcpServers": {
+    "pinecone-search": {
+      "command": "npx",
+      "args": ["-y", "@will-cppa/pinecone-read-only-mcp"],
+      "env": {
+        "PINECONE_SOURCES": "api_key_1:${PINECONE_API_KEY_1}:index_name_1;api_key_2:${PINECONE_API_KEY_2}:index_name_2",
+        "PINECONE_API_KEY_1": "pcsk_...",
+        "PINECONE_API_KEY_2": "pcsk_..."
+      }
+    }
+  }
+}
+```
+
+Prefer `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over inline API keys in `PINECONE_SOURCES`. See [SECURITY.md](./SECURITY.md).
+
+### Architecture decision
+
+**Chosen:** Option A — multi-source server with optional `source` parameter on tools (`SourceRegistry` + `PINECONE_SOURCES` / JSON config).
+
+**Rejected:**
+
+- **Option B (Cursor routing rules only):** Does not fix the UX problem when users forget which MCP entry to use; no code changes.
+- **Option C (thin proxy MCP):** Extra package and latency; duplicates routing logic already in `ServerContext`.
+
+**Rationale:** Pinecone SDK v8 supports multiple `Pinecone({ apiKey })` instances per process; MCP has no barrier to aggregating backends. Security is enforced by **deployment profiles** (public-only vs merged config), not per-query MCP authorization. All multi-source results include `source` for LLM provenance.
+
+**Execution semantics:** Discovery tools aggregate all sources when `source` is omitted. Execution tools (`query`, `count`, etc.) call `resolveSource`: infer source when the namespace exists on exactly one project; return `VALIDATION` when ambiguous. They do **not** fan out one query to all sources. `guided_query` without `namespace` routes via the aggregated namespace list and sets `selected_source` in `decision_trace`.
+
+**Audit logging:** When a tool resolves a specific source, stderr logs `toolname [source=name]` at INFO (execution tools and discovery tools that pass an explicit `source` filter). Aggregated discovery without `source` does not log per-source lines.
+
+See also [TOOLS.md § Multi-source mode](./TOOLS.md#multi-source-mode).
+
+---
+
 ## CLI flags (`parseCli` / `src/cli.ts`)
 
 | Flag | Maps to |
@@ -58,6 +167,8 @@ C++ Alliance deployers can copy [examples/alliance/.env.example](../examples/all
 | `--request-timeout-ms` | `requestTimeoutMs` |
 | `--disable-suggest-flow` | `disableSuggestFlow: true` |
 | `--check-indexes` | `checkIndexes: true` |
+| `--sources` | `sources` (inline multi-source string) |
+| `--config-file` | `configFile` / `PINECONE_CONFIG_FILE` |
 | `--help` / `-h` | Print help and exit |
 | `--version` / `-v` | Print version and exit |
 

docs/MIGRATION.md

@@ -165,6 +165,26 @@ await setupCoreServer({ context: ctx });
 
 See [deprecation-policy.md § Active deprecations](./deprecation-policy.md#active-deprecations-legacy-module-facades) for the full inventory.
 
+### Multi-source mode and legacy facades
+
+When `PINECONE_SOURCES` or a config file is active, legacy module facades (`getNamespacesWithCache`, `registerUrlGenerator`, `markSuggested`, `requireSuggested`, etc.) operate on the **default source only** (first inline source, or `defaultSource` from JSON). They do not aggregate across projects. Prefer an explicit `ServerContext` built with `sourceRegistry` from `buildSourceRegistry()` for multi-tenant or multi-project embedding.
+
+## Unreleased: Multi-source Pinecone projects
+
+**Who is affected:** Operators running separate MCP entries per Pinecone project, or library embedders needing multiple indexes.
+
+**After (single MCP server):**
+
+```bash
+PINECONE_SOURCES=api_key_1:${PINECONE_API_KEY_1}:index_name_1;api_key_2:${PINECONE_API_KEY_2}:index_name_2
+```
+
+Or point `PINECONE_CONFIG_FILE` at a JSON file (see [examples/multi-source/pinecone-sources.json.example](../examples/multi-source/pinecone-sources.json.example)).
+
+**Client flow:** `list_sources` → `list_namespaces` (note `source` on each row) → pass `source` on `query` / `count` / etc. when a namespace name is ambiguous.
+
+Single-key configs are unchanged; no migration required when using one API key.
+
 ## Unreleased: trimmed library exports
 
 **Who is affected:** Library embedders that imported `buildQueryExperimental` or `buildGuidedQueryExperimental` from `@will-cppa/pinecone-read-only-mcp` or `/alliance`.

docs/README.md

@@ -4,8 +4,8 @@ Guides for operators, MCP client authors, and library embedders.
 
 | Guide | Description |
 | ----- | ----------- |
-| [TOOLS.md](./TOOLS.md) | All MCP tools, parameters, success/error shapes, suggest-flow |
-| [CONFIGURATION.md](./CONFIGURATION.md) | Environment variables, CLI flags, `resolveConfig`, precedence |
+| [TOOLS.md](./TOOLS.md) | All MCP tools, parameters, success/error shapes, suggest-flow, multi-source |
+| [CONFIGURATION.md](./CONFIGURATION.md) | Environment variables, CLI flags, `resolveConfig`, multi-source mode, deployment profiles |
 | [SECURITY.md](./SECURITY.md) | API keys, log redaction, Docker hardening, reporting issues |
 | [CONTRIBUTING.md](../CONTRIBUTING.md) | Local dev, tests, lint/format, PR expectations |
 | [CI_CD.md](./CI_CD.md) | GitHub Actions, CodeQL, SBOM, Codecov, releases |

docs/SECURITY.md

@@ -2,8 +2,9 @@
 
 ## API keys
 
-- **Never** commit real Pinecone API keys. Use environment variables (`PINECONE_API_KEY`) or secret managers in CI.
-- The CLI and `resolveConfig` read keys only from argv/env/overrides — logs must not echo raw keys.
+- **Never** commit real Pinecone API keys. Use environment variables (`PINECONE_API_KEY`, or per-source keys referenced from `PINECONE_SOURCES` / a JSON config file) or secret managers in CI.
+- The CLI and `resolveConfig` read keys only from argv/env/overrides — logs must not echo raw keys. In multi-source mode, each source may use a different API key; all are redacted in logs and MCP responses.
+- Use **separate deployment profiles** for external (public-only) vs internal (merged) MCP configs — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles).
 
 ## Log redaction