Skip to content

Commit aaf2513

Browse files
author
zho
committed
update md and tests
1 parent d5f3cae commit aaf2513

13 files changed

Lines changed: 444 additions & 77 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Tagged releases are published to npm from GitHub Actions when a **GitHub Release
1010

1111
### Added
1212

13-
- **Multi-source mode:** configure multiple Pinecone API keys / indexes in one MCP server via `PINECONE_SOURCES`, `--sources`, or a JSON config file (`PINECONE_CONFIG_FILE` / `--config-file`). New `list_sources` tool (when more than one source is configured). Optional `source` parameter on discovery and query tools; `list_namespaces` aggregates across sources and tags each namespace with `source`. See [CONFIGURATION.md](docs/CONFIGURATION.md#multi-source-mode).
13+
- **Multi-source mode:** configure multiple Pinecone API keys / indexes in one MCP server via `PINECONE_SOURCES`, `--sources`, or a JSON config file (`PINECONE_CONFIG_FILE` / `--config-file`). New `list_sources` tool (when more than one source is configured). Optional `source` parameter on discovery and query tools; `list_namespaces` aggregates across sources and tags each namespace with `source`. See [CONFIGURATION.md](docs/CONFIGURATION.md#multi-source-mode), [TOOLS.md](docs/TOOLS.md#multi-source-mode), and deployment profiles in [CONFIGURATION.md](docs/CONFIGURATION.md#deployment-profiles).
1414

1515
### Changed
1616

‎docs/CONFIGURATION.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,67 @@ Discovery responses may include `source_errors` when one project fails but other
8787

8888
Single-key deployments (`PINECONE_API_KEY` + `PINECONE_INDEX_NAME` only) are unchanged — no `source` field on responses and no `list_sources` tool.
8989

90+
### Deployment profiles
91+
92+
Multi-source mode supports two operational profiles. **Never** ship a merged internal config through the same channel used for external partners.
93+
94+
| Profile | Who | Config | Risk if mis-shared |
95+
| ------- | --- | ------ | ------------------ |
96+
| **External (public-only)** | External companies, public MCP distribution | `PINECONE_API_KEY` + `PINECONE_INDEX_NAME`, or `PINECONE_SOURCES` with **one** entry | Low — single public key only |
97+
| **Internal (merged)** | Staff machines with access to private data | `PINECONE_SOURCES` or JSON config with **two+** entries | **High** — private API key and private namespace names exposed |
98+
99+
**External MCP config (unchanged):**
100+
101+
```json
102+
{
103+
"mcpServers": {
104+
"pinecone-search": {
105+
"command": "npx",
106+
"args": ["-y", "@will-cppa/pinecone-read-only-mcp"],
107+
"env": {
108+
"PINECONE_API_KEY": "your-public-key",
109+
"PINECONE_INDEX_NAME": "rag-hybrid"
110+
}
111+
}
112+
}
113+
}
114+
```
115+
116+
**Internal MCP config (merged public + private):**
117+
118+
```json
119+
{
120+
"mcpServers": {
121+
"pinecone-search": {
122+
"command": "npx",
123+
"args": ["-y", "@will-cppa/pinecone-read-only-mcp"],
124+
"env": {
125+
"PINECONE_SOURCES": "public:${PINECONE_PUBLIC_API_KEY}:rag-hybrid;private:${PINECONE_PRIVATE_API_KEY}:rag-private"
126+
}
127+
}
128+
}
129+
}
130+
```
131+
132+
Prefer `PINECONE_CONFIG_FILE` with `${ENV_VAR}` indirection over inline API keys in `PINECONE_SOURCES`. See [SECURITY.md](./SECURITY.md).
133+
134+
### Architecture decision
135+
136+
**Chosen:** Option A — multi-source server with optional `source` parameter on tools (`SourceRegistry` + `PINECONE_SOURCES` / JSON config).
137+
138+
**Rejected:**
139+
140+
- **Option B (Cursor routing rules only):** Does not fix the UX problem when users forget which MCP entry to use; no code changes.
141+
- **Option C (thin proxy MCP):** Extra package and latency; duplicates routing logic already in `ServerContext`.
142+
143+
**Rationale:** Pinecone SDK v8 supports multiple `Pinecone({ apiKey })` instances per process; MCP has no barrier to aggregating backends. Security is enforced by **deployment profiles** (public-only vs merged config), not per-query MCP authorization. All multi-source results include `source` for LLM provenance.
144+
145+
**Execution semantics:** Discovery tools aggregate all sources when `source` is omitted. Execution tools (`query`, `count`, etc.) call `resolveSource`: infer source when the namespace exists on exactly one project; return `VALIDATION` when ambiguous. They do **not** fan out one query to all sources. `guided_query` without `namespace` routes via the aggregated namespace list and sets `selected_source` in `decision_trace`.
146+
147+
**Audit logging:** When a tool resolves a specific source, stderr logs `toolname [source=name]` at INFO (execution tools and discovery tools that pass an explicit `source` filter). Aggregated discovery without `source` does not log per-source lines.
148+
149+
See also [TOOLS.md § Multi-source mode](./TOOLS.md#multi-source-mode).
150+
90151
---
91152

92153
## CLI flags (`parseCli` / `src/cli.ts`)

‎docs/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@ Guides for operators, MCP client authors, and library embedders.
44

55
| Guide | Description |
66
| ----- | ----------- |
7-
| [TOOLS.md](./TOOLS.md) | All MCP tools, parameters, success/error shapes, suggest-flow |
8-
| [CONFIGURATION.md](./CONFIGURATION.md) | Environment variables, CLI flags, `resolveConfig`, precedence |
7+
| [TOOLS.md](./TOOLS.md) | All MCP tools, parameters, success/error shapes, suggest-flow, multi-source |
8+
| [CONFIGURATION.md](./CONFIGURATION.md) | Environment variables, CLI flags, `resolveConfig`, multi-source mode, deployment profiles |
99
| [SECURITY.md](./SECURITY.md) | API keys, log redaction, Docker hardening, reporting issues |
1010
| [CONTRIBUTING.md](../CONTRIBUTING.md) | Local dev, tests, lint/format, PR expectations |
1111
| [CI_CD.md](./CI_CD.md) | GitHub Actions, CodeQL, SBOM, Codecov, releases |

‎docs/SECURITY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
- **Never** commit real Pinecone API keys. Use environment variables (`PINECONE_API_KEY`, or per-source keys referenced from `PINECONE_SOURCES` / a JSON config file) or secret managers in CI.
66
- The CLI and `resolveConfig` read keys only from argv/env/overrides — logs must not echo raw keys. In multi-source mode, each source may use a different API key; all are redacted in logs and MCP responses.
7+
- Use **separate deployment profiles** for external (public-only) vs internal (merged) MCP configs — see [CONFIGURATION.md § Deployment profiles](./CONFIGURATION.md#deployment-profiles).
78

89
## Log redaction
910

‎docs/TOOLS.md‎

Lines changed: 84 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -8,24 +8,27 @@ Success payloads separate **stable** fields (safe across minor bumps after `1.0.
88

99
| Tool | Stable | Experimental |
1010
| ---- | ------ | ------------ |
11-
| `list_namespaces` | `status`, `cache_hit`, `cache_ttl_seconds`, `expires_at_iso`, `count`, `namespaces` | _(none)_ |
12-
| `namespace_router` | `status`, `cache_hit`, `user_query`, `suggestions`, `recommended_namespace` | _(none)_ |
13-
| `suggest_query_params` | `status`, `cache_hit`, `suggested_fields`, `recommended_tool`, `use_count_tool`, `explanation`, `namespace_found` | _(none)_ |
14-
| `count` | `status`, `count`, `truncated`, `namespace`, `metadata_filter` | _(none)_ |
15-
| `query` | `status`, `mode`, `query`, `namespace`, `metadata_filter`, `result_count`, `results`, `fields` | `experimental.degraded`, `experimental.degradation_reason`, `experimental.hybrid_leg_failed`, `experimental.rerank_skipped_reason` |
16-
| `keyword_search` | `status`, `query`, `namespace`, `index`, `metadata_filter`, `result_count`, `results`, `fields` | _(none)_ |
17-
| `query_documents` | `status`, `query`, `namespace`, `metadata_filter`, `result_count`, `documents` | Same experimental degradation fields as `query` |
18-
| `guided_query` | `status`, `result` (count or query-shaped stable fields) | `experimental.decision_trace`; query-path `result.experimental` degradation fields |
19-
| `generate_urls` | `status`, `namespace`, `count`, `results` | _(none)_ |
11+
| `list_sources` | `status`, `sources`, `default` | _(none)_ |
12+
| `list_namespaces` | `status`, `cache_hit`, `cache_ttl_seconds`, `expires_at_iso`, `count`, `namespaces`, optional `source_errors` | _(none)_ |
13+
| `namespace_router` | `status`, `cache_hit`, `user_query`, `suggestions`, `recommended_namespace`, optional `recommended_source` | _(none)_ |
14+
| `suggest_query_params` | `status`, `cache_hit`, `suggested_fields`, `recommended_tool`, `use_count_tool`, `explanation`, `namespace_found`, optional `source` | _(none)_ |
15+
| `count` | `status`, `count`, `truncated`, `namespace`, `metadata_filter`, optional `source` | _(none)_ |
16+
| `query` | `status`, `mode`, `query`, `namespace`, `metadata_filter`, `result_count`, `results`, `fields`, optional `source` on payload and rows | `experimental.degraded`, `experimental.degradation_reason`, `experimental.hybrid_leg_failed`, `experimental.rerank_skipped_reason` |
17+
| `keyword_search` | `status`, `query`, `namespace`, `index`, `metadata_filter`, `result_count`, `results`, `fields`, optional `source` | _(none)_ |
18+
| `query_documents` | `status`, `query`, `namespace`, `metadata_filter`, `result_count`, `documents`, optional `source` | Same experimental degradation fields as `query` |
19+
| `guided_query` | `status`, `result` (count or query-shaped stable fields) | `experimental.decision_trace` (includes optional `selected_source`); query-path `result.experimental` degradation fields |
20+
| `generate_urls` | `status`, `namespace`, `count`, `results`, optional `source` | _(none)_ |
21+
22+
Multi-source-only stable fields (`source` on namespace rows, `source_errors`, `recommended_source`, `selected_source`) are **omitted** in single-key deployments.
2023

2124
Promotion process: [deprecation-policy.md § Stable vs experimental](./deprecation-policy.md#stable-vs-experimental-mcp-response-fields).
2225

2326
## Core vs Alliance tool surface
2427

2528
| Setup | Tools | MCP instructions |
2629
| ----- | ----- | ------------------ |
27-
| `setupCoreServer` (package root) | **8:** `list_namespaces`, `namespace_router`, `count`, `query`, `keyword_search`, `query_documents`, `generate_urls`, `guided_query` | `CORE_SERVER_INSTRUCTIONS` — includes `guided_query`; no `suggest_query_params` |
28-
| `setupAllianceServer` / published CLI | **9:** core tools plus `suggest_query_params` | `ALLIANCE_SERVER_INSTRUCTIONS` — includes suggest-flow quickstart |
30+
| `setupCoreServer` (package root) | **8** (single-key) or **9** (multi-source adds `list_sources`): `list_namespaces`, `namespace_router`, `count`, `query`, `keyword_search`, `query_documents`, `generate_urls`, `guided_query` | `CORE_SERVER_INSTRUCTIONS` — includes `guided_query`; no `suggest_query_params` |
31+
| `setupAllianceServer` / published CLI | **9** (single-key) or **10** (multi-source): core tools plus `suggest_query_params` | `ALLIANCE_SERVER_INSTRUCTIONS` — includes suggest-flow quickstart |
2932

3033
## Suggest-flow gate
3134

@@ -37,6 +40,48 @@ When **`disableSuggestFlow`** is **`true`** (core default via `resolveConfig`),
3740

3841
**Core:** gate off by default; set `PINECONE_DISABLE_SUGGEST_FLOW=false` or `disableSuggestFlow: false` to enable the gate. **Alliance:** gate on by default; set `PINECONE_DISABLE_SUGGEST_FLOW=true` or CLI `--disable-suggest-flow` to bypass (not recommended for production).
3942

43+
## Multi-source mode
44+
45+
Activated when `PINECONE_SOURCES`, `--sources`, or a JSON config file (`PINECONE_CONFIG_FILE` / `--config-file`) is set. See [CONFIGURATION.md § Multi-source mode](./CONFIGURATION.md#multi-source-mode) and [Deployment profiles](./CONFIGURATION.md#deployment-profiles).
46+
47+
**Shared `source` parameter** (optional on most tools):
48+
49+
> Pinecone source name (from `list_sources`). Omit on discovery tools to search all sources. On query tools, omit only when the namespace uniquely identifies one source.
50+
51+
| Category | Tools | When `source` is omitted |
52+
| -------- | ----- | ------------------------ |
53+
| **Discovery** | `list_sources`, `list_namespaces`, `namespace_router` | Aggregate or list across **all** configured sources; rows include `source` |
54+
| **Orchestrator** | `guided_query` (no `namespace`) | Route using aggregated namespace lists; `experimental.decision_trace.selected_source` |
55+
| **Execution** | `suggest_query_params`, `query`, `count`, `query_documents`, `keyword_search`, `generate_urls`, `guided_query` (with `namespace`) | `resolveSource`: infer source when namespace is unique; `VALIDATION` (`field: source`) when ambiguous |
56+
57+
**Typical multi-source flow:**
58+
59+
```text
60+
list_sources → list_namespaces → (optional) namespace_router → suggest_query_params → query | count | query_documents
61+
```
62+
63+
Or single-shot: `guided_query` (routes across sources when `namespace` is omitted).
64+
65+
**Suggest-flow in multi-source mode:** gate state uses compound keys `source:namespace`. Pass the same `source` + `namespace` pair for `suggest_query_params` and gated tools when the namespace exists on multiple sources.
66+
67+
---
68+
69+
## `list_sources` (multi-source only)
70+
71+
Registered only when more than one Pinecone source is configured.
72+
73+
| | |
74+
| --- | --- |
75+
| **Input** | _(empty object)_ |
76+
| **Success** | `{ status: 'success', sources: string[], default: string }` |
77+
| **Errors** | `LIFECYCLE` when not in multi-source mode |
78+
79+
**Example:**
80+
81+
```json
82+
{}
83+
```
84+
4085
---
4186

4287
## 1. `list_namespaces`
@@ -45,16 +90,22 @@ When **`disableSuggestFlow`** is **`true`** (core default via `resolveConfig`),
4590

4691
| | |
4792
| --- | --- |
48-
| **Input** | _(empty object)_ |
49-
| **Success** | `{ status: 'success', cache_hit, cache_ttl_seconds, expires_at_iso, count, namespaces: [{ name, record_count, metadata_fields }] }` |
93+
| **Input** | Optional `source` — filter to one configured project |
94+
| **Success** | `{ status: 'success', cache_hit, cache_ttl_seconds, expires_at_iso, count, namespaces: [{ name, record_count, metadata_fields, source? }], source_errors? }` |
5095
| **Errors** | `PINECONE_ERROR`, `TIMEOUT`, etc. |
5196

52-
**Example (conceptual MCP params):**
97+
**Example (multi-source, all projects):**
5398

5499
```json
55100
{}
56101
```
57102

103+
**Example (multi-source, one project):**
104+
105+
```json
106+
{ "source": "public" }
107+
```
108+
58109
---
59110

60111
## 2. `namespace_router`
@@ -65,8 +116,9 @@ When **`disableSuggestFlow`** is **`true`** (core default via `resolveConfig`),
65116
| ----- | ---- | -------- | ----------- |
66117
| `user_query` | string | yes | User question / intent |
67118
| `top_n` | int | no (default 3) | Max suggestions, 1–5 |
119+
| `source` | string | no | Restrict ranking to one configured source (multi-source) |
68120

69-
**Success:** `{ status: 'success', cache_hit, user_query, suggestions, recommended_namespace }`.
121+
**Success:** `{ status: 'success', cache_hit, user_query, suggestions: [{ namespace, score, record_count, reasons, source? }], recommended_namespace, recommended_source? }`.
70122

71123
**Example:**
72124

@@ -84,8 +136,9 @@ When **`disableSuggestFlow`** is **`true`** (core default via `resolveConfig`),
84136
| ----- | ---- | -------- | ----------- |
85137
| `namespace` | string | yes | Target namespace (must exist in cached `list_namespaces`) |
86138
| `user_query` | string | yes | Natural-language task |
139+
| `source` | string | no | Pinecone source (multi-source; required when namespace is ambiguous) |
87140

88-
**Success:** `{ status: 'success', cache_hit, ...suggestQueryParams fields including suggested_fields, recommended_tool, use_count_tool, explanation, namespace_found }`.
141+
**Success:** `{ status: 'success', cache_hit, ...suggestQueryParams fields including suggested_fields, recommended_tool, use_count_tool, explanation, namespace_found, source? }`.
89142

90143
**Example:**
91144

@@ -107,8 +160,9 @@ When **`disableSuggestFlow`** is **`true`** (core default via `resolveConfig`),
107160
| `namespace` | string | yes | Namespace |
108161
| `query_text` | string | yes | Query text (use broad text like `"document"` for metadata-only counts) |
109162
| `metadata_filter` | object | no | Pinecone metadata filter |
163+
| `source` | string | no | Pinecone source (multi-source) |
110164

111-
**Success:** `{ status: 'success', count, truncated, namespace, metadata_filter? }`.
165+
**Success:** `{ status: 'success', count, truncated, namespace, metadata_filter?, source? }`.
112166

113167
---
114168

@@ -125,8 +179,9 @@ When **`disableSuggestFlow`** is **`true`** (core default via `resolveConfig`),
125179
| `use_reranking` | boolean | no | When preset allows reranking |
126180
| `metadata_filter` | object | no | Metadata filter |
127181
| `fields` | string[] | no | Pinecone fields to return |
182+
| `source` | string | no | Pinecone source (multi-source) |
128183

129-
**Success (`QueryResponse`):** `{ status: 'success', mode?, query, namespace, metadata_filter?, result_count, results[], fields?, experimental?: { degraded?, degradation_reason?, hybrid_leg_failed?, rerank_skipped_reason? } }`.
184+
**Success (`QueryResponse`):** `{ status: 'success', mode?, query, namespace, metadata_filter?, result_count, results[], fields?, source?, experimental?: { ... } }`.
130185

131186
Each row: `document_id`, `paper_number` (deprecated alias), `title`, `author`, `url`, `content`, `score`, `reranked`, optional `metadata`.
132187

@@ -168,8 +223,9 @@ Treat **`experimental.degraded: true`** as lower confidence even when `status` i
168223
| `top_k` | int | no | 1–100 |
169224
| `metadata_filter` | object | no | Filter |
170225
| `fields` | string[] | no | Returned fields |
226+
| `source` | string | no | Pinecone source (multi-source) |
171227

172-
**Success:** Similar row shape to `query` (`KeywordSearchResponse`).
228+
**Success:** Similar row shape to `query` (`KeywordSearchResponse`); optional `source` on payload and rows.
173229

174230
---
175231

@@ -184,8 +240,9 @@ Treat **`experimental.degraded: true`** as lower confidence even when `status` i
184240
| `top_k` | int | no | Documents to return (see constants, default 5, max 20) |
185241
| `metadata_filter` | object | no | Filter |
186242
| `max_chunks_per_document` | int | no | Cap merged chunks per doc (default 200, max 500) |
243+
| `source` | string | no | Pinecone source (multi-source) |
187244

188-
**Success:** `{ status: 'success', query, namespace, metadata_filter?, result_count, documents[], experimental?: { degraded?, degradation_reason?, hybrid_leg_failed?, rerank_skipped_reason? } }`.
245+
**Success:** `{ status: 'success', query, namespace, metadata_filter?, result_count, documents[], source?, experimental?: { ... } }`.
189246

190247
---
191248

@@ -201,10 +258,11 @@ Treat **`experimental.degraded: true`** as lower confidence even when `status` i
201258
| `top_k` | int | no | For query paths |
202259
| `preferred_tool` | `auto` \| `count` \| `fast` \| `detailed` \| `full` | no | Override automated tool choice |
203260
| `enrich_urls` | boolean | no (default true) | Run URL generator when metadata lacks `url` |
261+
| `source` | string | no | Pinecone source (multi-source; pin routing when namespace is ambiguous) |
204262

205263
**Success:** `{ status: 'success', experimental: { decision_trace }, result }` where `result` is either a count payload or a `QueryResponse`-shaped query payload.
206264

207-
**`experimental.decision_trace` fields (non-exhaustive):** `cache_hit`, `input_namespace`, `routed_namespace`, `selected_namespace`, `ranked_namespaces`, `suggested_fields`, `suggested_tool`, `selected_tool`, `explanation`, `enrich_urls`, `rerank_status` (`success` \| `skipped` \| `skipped_no_model` \| `failed`).
265+
**`experimental.decision_trace` fields (non-exhaustive):** `cache_hit`, `input_namespace`, `routed_namespace`, `selected_namespace`, `selected_source?`, `ranked_namespaces`, `suggested_fields`, `suggested_tool`, `selected_tool`, `explanation`, `enrich_urls`, `rerank_status` (`success` \| `skipped` \| `skipped_no_model` \| `failed`).
208266

209267
When the inner query path runs, `result.experimental` includes the same degradation fields as `query` (see [Rerank and hybrid degradation](#rerank-and-hybrid-degradation)).
210268

@@ -227,8 +285,9 @@ When the inner query path runs, `result.experimental` includes the same degradat
227285
| ----- | ---- | -------- | ----------- |
228286
| `namespace` | string | yes | Namespace |
229287
| `records` | object[] | yes | Up to 500 records (metadata object or `{ metadata: {...} }`) |
288+
| `source` | string | no | Pinecone source (multi-source) |
230289

231-
**Success:** `{ status: 'success', namespace, count, results: [{ index, url, method, reason, metadata }] }`.
290+
**Success:** `{ status: 'success', namespace, count, results: [...], source? }`.
232291

233292
---
234293

@@ -238,6 +297,9 @@ When the inner query path runs, `result.experimental` includes the same degradat
238297
Typical manual flow:
239298
list_namespaces → (optional) namespace_router → suggest_query_params → query | count | query_documents
240299
300+
Multi-source manual flow:
301+
list_sources → list_namespaces → (optional) namespace_router → suggest_query_params → query | count | query_documents
302+
241303
Keyword-only:
242304
list_namespaces → keyword_search # no suggest gate
243305

0 commit comments

Comments
 (0)