Skip to content

Commit 61df165

Browse files
hallelx2hallelx2
committed
docs: add Stores design + roadmap (Org -> Store -> Documents)
Captures the store concept and the full cross-layer change surface before building. A store is a named collection (Pinecone-index-like) you upload into and query against; it's also the natural home for a domain Profile ("research-papers store"). - docs/STORES.md — design: the model, store as a CP entity vs. an opaque store_id on engine documents, stores-carry-the-profile unification, the auto-created default store (backward compat), the header scoping flow (X-Vectorless-Store/-Profile), and a concrete change surface table per layer (engine, server, control-plane, dashboard, SDKs x3, MCP) sourced from a full investigation. - docs/roadmaps/STORES.md — 4-phase rollout (CP+engine core → dashboard → SDKs+MCP → profile-on-store + store-bound keys + usage). - Indexed both. Investigation only; no code yet. Mirrors the org_id scoping already shipped, one header deeper.
1 parent 657ee08 commit 61df165

4 files changed

Lines changed: 343 additions & 0 deletions

File tree

‎docs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ specific subsystem you're working on.
2323
| [ARCHITECTURE.md](./ARCHITECTURE.md) | The whole stack, layer by layer. Read this first. |
2424
| [REPOS.md](./REPOS.md) | Which repositories exist, which are public vs private, when to split. |
2525
| [ENGINE.md](./ENGINE.md) | The core retrieval engine — library + daemon. |
26+
| [STORES.md](./STORES.md) | Stores — named collections (Org → Store → Documents). |
2627
| [PROFILES.md](./PROFILES.md) | Domain-aware structuring — typed, navigable maps per domain. |
2728
| [SERVER.md](./SERVER.md) | The HTTP + gRPC service that fronts the engine. |
2829
| [LLMGATE.md](./LLMGATE.md) | The "LiteLLM for Go" gateway layer. |

‎docs/STORES.md‎

Lines changed: 243 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,243 @@
1+
# Stores
2+
3+
> A **store** is a named collection within an org — the unit you
4+
> upload documents into and query against. Think Pinecone index,
5+
> Qdrant collection, S3 bucket: the addressable container for a body
6+
> of knowledge.
7+
8+
## Purpose
9+
10+
Today the tenancy model is two levels:
11+
12+
```
13+
Org ─▶ Documents ─▶ Sections
14+
```
15+
16+
Every document in an org lives in one undifferentiated pool. That's
17+
fine for a demo, wrong for real use: a team has *multiple* bodies of
18+
knowledge — a research library, a compliance corpus, a product-docs
19+
set — and wants to query within one, not across all of them.
20+
21+
Stores add the missing middle:
22+
23+
```
24+
Org ─▶ Store ─▶ Documents ─▶ Sections
25+
```
26+
27+
A store is "where stuff goes." You create a store, upload into it,
28+
and queries run against it. It's also the natural home for a
29+
**domain Profile** (see [PROFILES.md](./PROFILES.md)): a store *is* a
30+
specialized collection — "my research-papers store", "my clinical-
31+
guidelines store" — so the profile is a property of the store, not a
32+
per-upload flag.
33+
34+
## What a store is (and where it lives)
35+
36+
A store is a **control-plane entity**. It has a name, a slug, an owning
37+
org, a profile, timestamps, and (later) per-store quota + billing
38+
rollups. The control plane owns its lifecycle.
39+
40+
The **engine** does not know stores as entities. It only sees an opaque
41+
`store_id` scoping column on `documents` — exactly how it treats
42+
`org_id` today. The engine never lists or creates stores; it just
43+
filters by `store_id` when told to. This keeps the engine a clean,
44+
single-responsibility retrieval core.
45+
46+
```
47+
control plane engine
48+
───────────── ──────
49+
stores table documents.store_id (opaque scope column)
50+
id, org_id, + every read filters by it
51+
name, slug, (same mechanism as org_id today)
52+
profile, ...
53+
```
54+
55+
## Stores carry the Profile
56+
57+
This is the key unification. The earlier profile-selection rule —
58+
*declared → auto-detect → generic* — resolves cleanly through the
59+
store:
60+
61+
1. **Store's profile** — if the store declares one (`research-paper`),
62+
every document uploaded into it gets that structuring. This is the
63+
"declared" path, set once per store instead of per upload.
64+
2. **Auto-detect** — if the store's profile is `auto` (or unset), the
65+
engine detects per-document.
66+
3. **`generic`** — fallback.
67+
68+
So "specialize a collection for a domain" = "set the store's profile".
69+
The control plane reads the store's profile and passes it to the
70+
engine as `X-Vectorless-Profile` alongside `X-Vectorless-Store`.
71+
72+
## The default store
73+
74+
Every org gets a **`default` store auto-created** with the org (the
75+
same way we auto-create the org on signup). This keeps everything
76+
backward-compatible and zero-friction:
77+
78+
- Existing single-pool behavior = one store called "Default".
79+
- A new user can upload immediately without creating a store first.
80+
- Uploads with no store specified land in `default`.
81+
- The dashboard's "active store" defaults to `default`.
82+
83+
Stores are additive: nothing breaks for callers that never mention one.
84+
85+
## How scoping flows
86+
87+
Identical shape to the existing org scoping, one header deeper.
88+
89+
```
90+
SDK / dashboard
91+
│ Authorization: Bearer <api-key> (or session cookie)
92+
│ X-Vectorless-Store: <store_id> (optional; default store if omitted)
93+
▼
94+
control plane
95+
• resolve key/session → org
96+
• resolve store: explicit header, else key's bound store, else org default
97+
• verify store belongs to org
98+
• look up store.profile
99+
▼ proxy to engine, injecting:
100+
│ X-Vectorless-Org: <org_id>
101+
│ X-Vectorless-Store: <store_id>
102+
│ X-Vectorless-Profile: <profile> (from the store)
103+
▼
104+
engine
105+
• documents.store_id filters every read/write
106+
• org_id still enforced too (defense in depth + tenant ops)
107+
• profile drives structuring at ingest
108+
```
109+
110+
API keys may be **bound to a store** (null = org-wide). A store-bound
111+
key needs no `X-Vectorless-Store` header — the binding implies it.
112+
This is how an integrator pins one credential to one collection.
113+
114+
## Change surface by layer
115+
116+
### Engine (`vectorless-engine`)
117+
118+
Smallest change — `store_id` is a sibling of `org_id`.
119+
120+
| Area | Change |
121+
|---|---|
122+
| Migration | Add `documents.store_id TEXT NOT NULL DEFAULT '…default…'` + index `(org_id, store_id, created_at)` |
123+
| `db.Document` | Add `StoreID` field |
124+
| `db` CRUD | `GetDocument` / `ListDocuments` / `DeleteDocument` / `CountSections` / `LoadTree` take + filter `storeID` (alongside `orgID`); `*ForWorker` variants unaffected |
125+
| `NewDocument` | Persist `store_id` |
126+
127+
### Server (`vectorless-server`)
128+
129+
| Area | Change |
130+
|---|---|
131+
| `requireOrgID` | Add `requireStoreID` (reads `X-Vectorless-Store`); thread into every documents/query handler |
132+
| Connect handlers | `orgIDFromConnect` → also pull store header |
133+
| Profile (later) | Read `X-Vectorless-Profile`, pass to ingest |
134+
135+
### Control plane (`vectorless-control-plane`)
136+
137+
The biggest net-new surface — stores are a CP entity.
138+
139+
| Area | Change |
140+
|---|---|
141+
| Migration | New `stores` table (id, org_id, name, slug, description, profile, timestamps; `UNIQUE(org_id, slug)`); add `api_keys.store_id` (nullable FK); add `store_id` to `usage_events` / `usage_daily` / `usage_monthly` |
142+
| Model | `model/store.go` (Store + Create/Update requests); add `StoreID` to `APIKey` + `Principal` |
143+
| Store layer | `store/stores.go` — Create / Get / GetByOrgSlug / ListByOrg / Update / Delete; `EnsureDefaultStore(orgID)` |
144+
| Handlers | `handler/store.go` — CRUD at `/admin/v1/orgs/{orgId}/stores[/{storeId}]` |
145+
| API keys | `HandleCreateAPIKey` accepts optional `store_id`; auth resolves key → store |
146+
| Proxy | Inject `X-Vectorless-Store` (+ `X-Vectorless-Profile`) in `proxy.Forward` |
147+
| Auth middleware | `APIKeyAuth`: resolve `apiKey.StoreID`; `SessionAuth`: read + validate `X-Vectorless-Store` against the org |
148+
| Org create | `EnsureDefaultStore` whenever an org is created (signup/login/me, like auto-org) |
149+
150+
### Dashboard (`vectorless-dashboard`)
151+
152+
| Area | Change |
153+
|---|---|
154+
| `lib/cp-proxy.ts` | Add `getActiveStore()` + `storeId` option; inject `X-Vectorless-Store` next to org |
155+
| State | New `StoreProvider` (parallel to `SessionProvider`) holding the active store; persist last choice to localStorage |
156+
| Nav | `StoreSwitcher` dropdown in the sidebar header; new `/dashboard/stores` list+create page |
157+
| Scoped views | documents list, upload, playground, analytics read the active store |
158+
| API keys | optional store selector on key creation; show scope in the table; `apiKeys.store_id` column in the local DB |
159+
160+
### SDKs (`vectorless-sdk/{typescript,go,python}`) — all three exist
161+
162+
| Area | Change |
163+
|---|---|
164+
| Config | optional `storeId` / `StoreID` / `store_id` (default store for the client) |
165+
| New methods | `createStore`, `listStores`, `getStore`, `deleteStore` |
166+
| Existing methods | `ingestDocument` / `query` / `listDocuments` accept a per-call `storeId` override |
167+
| Transport | send `X-Vectorless-Store` when set; precedence: per-call > client config > none |
168+
169+
### MCP (`vectorless-mcp`) — exists as a Next.js app
170+
171+
| Area | Change |
172+
|---|---|
173+
| Schema | `mcp_store` table; add `store_id` to `mcp_document` |
174+
| Tools | add `vectorless_create_store`, `vectorless_list_stores`, `vectorless_get_store`, `vectorless_delete_store` |
175+
| Handlers | `ingest`/`list`/`query` accept + validate `store_id`, pass to SDK |
176+
177+
## API shape
178+
179+
REST (control-plane management API):
180+
181+
```
182+
POST /admin/v1/orgs/{orgId}/stores create
183+
GET /admin/v1/orgs/{orgId}/stores list
184+
GET /admin/v1/orgs/{orgId}/stores/{storeId} get
185+
PATCH /admin/v1/orgs/{orgId}/stores/{storeId} update (name, profile)
186+
DELETE /admin/v1/orgs/{orgId}/stores/{storeId} delete (cascades documents)
187+
```
188+
189+
Data-plane (`/v1/*`) is unchanged in shape — scoping rides the
190+
`X-Vectorless-Store` header, so no path churn for the engine API or
191+
the SDKs' existing method signatures.
192+
193+
SDK:
194+
195+
```ts
196+
const store = await client.createStore({ name: "Research", profile: "research-paper" });
197+
await client.ingestDocument(pdf, { storeId: store.id, filename: "paper.pdf" });
198+
await client.query(docId, "what's the contribution?", { storeId: store.id });
199+
// or pin the whole client to a store:
200+
const research = new VectorlessClient({ apiKey, storeId: store.id });
201+
```
202+
203+
## Decisions to confirm
204+
205+
- **Default store auto-created per org?** (Proposed: yes — backward
206+
compat + zero friction.)
207+
- **Profile lives on the store?** (Proposed: yes — unifies stores +
208+
profiles; profile set once per collection.)
209+
- **Store-bound API keys?** (Proposed: optional; null = org-wide.)
210+
- **Engine scoping: store_id in addition to org_id?** (Proposed: both
211+
— org for tenant isolation, store for collection scope.)
212+
213+
## Open questions
214+
215+
- **Cross-store query** — ever allow a query spanning multiple stores
216+
in an org? (Default: no; one store per query. Multi-doc query already
217+
exists *within* a scope.)
218+
- **Moving documents between stores** — supported, or immutable
219+
membership? (Lean: immutable for v1; re-upload to move.)
220+
- **Per-store quota / billing** — when do usage rollups need
221+
`store_id` granularity vs. org-level? (Schema carries it from day
222+
one; enforcement later.)
223+
- **Slug vs id in URLs** — dashboard routes by slug (`/stores/research`)
224+
or id?
225+
226+
## Phasing
227+
228+
1. **Phase 1 — CP + engine core.** `stores` table + CRUD; `store_id`
229+
on engine documents; default store auto-create; proxy injects the
230+
header; everything lands in `default` transparently. No UI yet.
231+
2. **Phase 2 — dashboard.** StoreProvider + switcher + stores page;
232+
scope documents/upload/playground/analytics to the active store.
233+
3. **Phase 3 — SDKs + MCP.** Store methods + per-call override across
234+
TS/Go/Python; MCP store tools.
235+
4. **Phase 4 — profile-on-store + store-bound keys + per-store usage.**
236+
237+
## Related docs
238+
239+
- [PROFILES.md](./PROFILES.md) — domains; a store carries a profile.
240+
- [ENGINE.md](./ENGINE.md) — the `org_id` scoping `store_id` mirrors.
241+
- [CONTROL-PLANE.md](./CONTROL-PLANE.md) — where stores live.
242+
- [DATA.md](./DATA.md) — the schema stores extend.
243+
- [roadmaps/STORES.md](./roadmaps/STORES.md) — the delivery checklist.

‎docs/roadmaps/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ progress, and what's next.
1616
| Roadmap | Subsystem | Design doc |
1717
|---|---|---|
1818
| [ENGINE.md](./ENGINE.md) | Core engine | [../ENGINE.md](../ENGINE.md) |
19+
| [STORES.md](./STORES.md) | Stores (Org→Store→Docs) | [../STORES.md](../STORES.md) |
1920
| [PROFILES.md](./PROFILES.md) | Domain-aware structuring | [../PROFILES.md](../PROFILES.md) |
2021
| [SERVER.md](./SERVER.md) | HTTP/gRPC transport | [../SERVER.md](../SERVER.md) |
2122
| [LLMGATE.md](./LLMGATE.md) | LLM gateway library | [../LLMGATE.md](../LLMGATE.md) |

‎docs/roadmaps/STORES.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Stores roadmap
2+
3+
> Design doc: [../STORES.md](../STORES.md)
4+
5+
Adding the Org → **Store** → Documents layer. A store is the named
6+
collection you upload into and query against; it also carries the
7+
domain [profile](../PROFILES.md).
8+
9+
## Current status (summary)
10+
11+
| Phase | Status |
12+
|---|---|
13+
| Phase 0 — Org → Documents (no stores) | Shipped (today) |
14+
| Phase 1 — CP + engine core (`stores`, `store_id`, default store) | Not started |
15+
| Phase 2 — dashboard (switcher, stores page, scoped views) | Not started |
16+
| Phase 3 — SDKs + MCP store support | Not started |
17+
| Phase 4 — profile-on-store, store-bound keys, per-store usage | Not started |
18+
19+
Everything is additive: a `default` store per org preserves today's
20+
single-pool behavior with zero breakage.
21+
22+
## Phase 1 — control plane + engine core
23+
24+
The foundation. After this, every document silently belongs to a
25+
store; existing flows keep working via the default store.
26+
27+
- [ ] **CP migration**: `stores` table (id, org_id, name, slug,
28+
description, profile, timestamps; `UNIQUE(org_id, slug)`).
29+
`api_keys.store_id` nullable FK. `store_id` on usage tables.
30+
- [ ] **CP model**: `model/store.go`; `StoreID` on `APIKey` +
31+
`Principal`.
32+
- [ ] **CP store layer**: Create / Get / GetByOrgSlug / ListByOrg /
33+
Update / Delete; `EnsureDefaultStore(orgID)`.
34+
- [ ] **CP handlers + routes**: CRUD under
35+
`/admin/v1/orgs/{orgId}/stores`.
36+
- [ ] **CP auto-default**: call `EnsureDefaultStore` wherever orgs are
37+
ensured (signup / login / me), mirroring auto-org.
38+
- [ ] **CP auth**: `APIKeyAuth` resolves `apiKey.StoreID`; `SessionAuth`
39+
reads + validates `X-Vectorless-Store` against the org.
40+
- [ ] **CP proxy**: inject `X-Vectorless-Store` in `proxy.Forward`
41+
(resolution: header → key binding → org default).
42+
- [ ] **Engine migration**: `documents.store_id` + `(org_id, store_id,
43+
created_at)` index.
44+
- [ ] **Engine db**: `Document.StoreID`; `GetDocument` /
45+
`ListDocuments` / `DeleteDocument` / `CountSections` / `LoadTree`
46+
filter by `store_id`; `NewDocument` persists it.
47+
- [ ] **Server handlers**: `requireStoreID` + thread into documents /
48+
query / connect handlers.
49+
- [ ] **Validate**: upload with + without a store header; confirm
50+
default-store fallback; confirm cross-store isolation (store A
51+
can't see store B's docs).
52+
53+
## Phase 2 — dashboard
54+
55+
- [ ] **`lib/cp-proxy.ts`**: `getActiveStore()` + `storeId` option;
56+
inject `X-Vectorless-Store`.
57+
- [ ] **`StoreProvider`** (parallel to SessionProvider) + localStorage
58+
persistence of last-selected store.
59+
- [ ] **`StoreSwitcher`** dropdown in the sidebar header.
60+
- [ ] **`/dashboard/stores`** list + create + delete page; CP-proxy
61+
route(s) for store CRUD.
62+
- [ ] **Scope existing views** to the active store: documents list,
63+
upload, playground, analytics.
64+
65+
## Phase 3 — SDKs + MCP
66+
67+
- [ ] **TS/Go/Python**: `storeId` client config; `createStore` /
68+
`listStores` / `getStore` / `deleteStore`; per-call `storeId`
69+
override on ingest / query / list; transport sends the header
70+
(precedence: per-call > config > none).
71+
- [ ] **MCP**: `mcp_store` table + `store_id` on `mcp_document`;
72+
`vectorless_create_store` / `_list_stores` / `_get_store` /
73+
`_delete_store` tools; ingest/list/query accept + validate
74+
`store_id`.
75+
- [ ] **Docs**: refresh SDKS.md + MCP.md examples with stores.
76+
77+
## Phase 4 — specialization + accounting
78+
79+
- [ ] **Profile on store**: store declares its profile; CP passes
80+
`X-Vectorless-Profile`; ingest applies it. (Depends on the
81+
Profiles Phase 1 scaffold.)
82+
- [ ] **Store-bound API keys**: key creation UI + enforcement;
83+
store-bound key needs no header.
84+
- [ ] **Per-store usage**: surface usage rollups by store in the
85+
dashboard analytics.
86+
87+
## Open questions (from design doc)
88+
89+
- [ ] Cross-store query (default: no).
90+
- [ ] Move documents between stores (lean: immutable v1).
91+
- [ ] Slug vs id in dashboard URLs.
92+
- [ ] When per-store quota enforcement (vs. org-level) kicks in.
93+
94+
## Related
95+
96+
- [../STORES.md](../STORES.md) — the design doc.
97+
- [../PROFILES.md](../PROFILES.md) — domains; carried on the store.
98+
- [../../ROADMAP.md](../../ROADMAP.md) — root checkbox document.

0 commit comments

Comments
 (0)