Skip to content

Commit cea8708

Browse files
neuromechanistneuromechanist
authored
Add documentation for ephemeral database mirrors (#11)
- New page: docs/osa/mirrors.md with full guide (architecture, CLI commands, API endpoints, quick start, resource limits) - Update CLI reference: add --mirror flag and osa mirror subcommand - Update API reference: add X-Mirror-ID header and mirror endpoints - Add Database Mirrors to mkdocs navigation
1 parent 4c7ddb2 commit cea8708

4 files changed

Lines changed: 286 additions & 0 deletions

File tree

‎docs/osa/api-reference.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -193,6 +193,20 @@ data: {"event": "content", "content": " annotate"}
193193
data: {"event": "done", "session_id": "abc123"}
194194
```
195195

196+
### Mirrors
197+
198+
Ephemeral database mirror management. See [Database Mirrors](mirrors.md) for full documentation.
199+
200+
| Method | Endpoint | Description |
201+
|--------|----------|-------------|
202+
| `POST` | `/mirrors` | Create a mirror |
203+
| `GET` | `/mirrors` | List active mirrors |
204+
| `GET` | `/mirrors/{id}` | Get mirror metadata |
205+
| `DELETE` | `/mirrors/{id}` | Delete a mirror |
206+
| `POST` | `/mirrors/{id}/refresh` | Re-copy from production |
207+
| `POST` | `/mirrors/{id}/sync` | Run sync pipeline |
208+
| `GET` | `/mirrors/{id}/download/{community}` | Download SQLite file |
209+
196210
## Request Parameters
197211

198212
### Ask Request
@@ -219,6 +233,7 @@ data: {"event": "done", "session_id": "abc123"}
219233
| `X-OpenRouter-Key` | OpenRouter API key (BYOK) | Yes (or X-API-Key) |
220234
| `X-API-Key` | Server admin API key | Yes (or BYOK) |
221235
| `X-User-ID` | User ID for cache optimization | No |
236+
| `X-Mirror-ID` | Route to an ephemeral database mirror (see [Database Mirrors](mirrors.md)) | No |
222237
| `Content-Type` | Must be `application/json` | Yes |
223238

224239
## Error Responses

‎docs/osa/cli-reference.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ osa ask -a hed "What is HED?" --no-stream
5959
Options:
6060

6161
- `--assistant, -a`: Community assistant ID (hed, bids, eeglab). Default: hed
62+
- `--mirror, -m`: Mirror ID for ephemeral database routing (see [Database Mirrors](mirrors.md))
6263
- `--api-key, -k`: OpenRouter API key (overrides saved config)
6364
- `--api-url`: Override API URL
6465
- `--output, -o`: Output format: rich, json, plain. Default: rich
@@ -82,6 +83,7 @@ osa chat -a eeglab --no-stream
8283
Options:
8384

8485
- `--assistant, -a`: Community assistant ID (hed, bids, eeglab). Default: hed
86+
- `--mirror, -m`: Mirror ID for ephemeral database routing (see [Database Mirrors](mirrors.md))
8587
- `--api-key, -k`: OpenRouter API key (overrides saved config)
8688
- `--api-url`: Override API URL
8789
- `--no-stream`: Disable streaming
@@ -199,6 +201,33 @@ Options:
199201
- `--port, -p`: Port to bind to. Default: 38528
200202
- `--reload, -r`: Enable auto-reload
201203

204+
### `osa mirror`
205+
206+
Manage ephemeral database mirrors for development. See [Database Mirrors](mirrors.md) for full documentation.
207+
208+
```bash
209+
# Create a mirror
210+
osa mirror create -c hed -c bids --label "my-test"
211+
212+
# List active mirrors
213+
osa mirror list
214+
215+
# Show mirror details
216+
osa mirror info abc123def456
217+
218+
# Delete a mirror
219+
osa mirror delete abc123def456
220+
221+
# Re-copy production data into mirror
222+
osa mirror refresh abc123def456
223+
224+
# Run sync pipeline against mirror
225+
osa mirror sync abc123def456 --type github
226+
227+
# Download mirror databases locally
228+
osa mirror pull abc123def456
229+
```
230+
202231
### `osa sync`
203232

204233
Sync knowledge sources for community assistants. Requires server dependencies. All subcommands accept `--community/-c` to specify the target community.

‎docs/osa/mirrors.md‎

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
# Database Mirrors
2+
3+
Ephemeral database mirrors let developers work on isolated copies of community knowledge databases without running a local server. Instead of full ephemeral backends (Docker containers, port allocation, proxies), mirrors provide lightweight SQLite copies that can be created and destroyed in seconds.
4+
5+
## How it Works
6+
7+
```
8+
Developer's CLI OSA Backend
9+
┌──────────────┐ ┌────────────────────────┐
10+
│ osa ask │ X-Mirror-ID │ │
11+
│ --mirror │ ──────────────> │ Middleware sets │
12+
│ abc123 │ │ ContextVar │
13+
│ │ │ │ │
14+
│ │ │ v │
15+
│ │ │ get_db_path("hed") │
16+
│ │ │ returns: │
17+
│ │ │ mirrors/abc123/hed.db │
18+
│ │ │ (instead of │
19+
│ │ │ knowledge/hed.db) │
20+
└──────────────┘ └────────────────────────┘
21+
```
22+
23+
One shared backend, many lightweight database copies. All search, sync, and tool code works unmodified because every database access flows through a single function (`get_db_path`) that checks a `ContextVar`.
24+
25+
## Quick Start
26+
27+
```bash
28+
# Create a mirror of the HED and BIDS databases
29+
osa mirror create -c hed -c bids --label "testing-new-prompt"
30+
31+
# Use with: osa ask "question" -a hed --mirror abc123def456
32+
# Mirror created: abc123def456
33+
34+
# Ask questions against your mirror
35+
osa ask -a hed "How do I annotate a button press?" --mirror abc123def456
36+
37+
# Interactive chat with your mirror
38+
osa chat -a hed --mirror abc123def456
39+
40+
# Re-sync from public sources into your mirror
41+
osa mirror sync abc123def456 --type github
42+
43+
# When done, clean up
44+
osa mirror delete abc123def456
45+
```
46+
47+
## What Works vs. What Doesn't
48+
49+
### Works with mirrors (no local server needed)
50+
51+
- Querying existing community databases with different data
52+
- Re-syncing from public sources (GitHub, papers, etc.) into mirror
53+
- Using BYOK with any LLM provider
54+
- Multiple developers working independently
55+
- Testing retrieval quality with modified data
56+
57+
### Requires a local server
58+
59+
- **Schema changes**: New tables/columns need new Python code
60+
- **New community from scratch**: Server has no routes for unregistered communities
61+
- **Modified tool/search code**: Remote server runs deployed code, not your branch
62+
- **Modified sync pipeline**: Same; remote runs old sync logic
63+
- **Local LLM**: LLM must be reachable from server
64+
65+
For these cases, use `osa mirror pull` to download mirror databases locally, then run `osa serve`:
66+
67+
```bash
68+
# Download mirror databases locally
69+
osa mirror pull abc123def456
70+
71+
# Run local server with the downloaded data
72+
osa serve
73+
```
74+
75+
## CLI Commands
76+
77+
### `osa mirror create`
78+
79+
Create a new ephemeral database mirror.
80+
81+
```bash
82+
osa mirror create -c hed -c bids
83+
osa mirror create -c hed --label "testing-new-prompt" --ttl 24
84+
```
85+
86+
| Option | Description | Default |
87+
|--------|-------------|---------|
88+
| `--community, -c` | Community ID to include (repeatable) | Required |
89+
| `--label, -l` | Human-readable label | None |
90+
| `--ttl` | Hours until mirror expires (1-168) | 48 |
91+
| `--api-key, -k` | OpenRouter API key | From config |
92+
| `--api-url` | Override API URL | From config |
93+
94+
### `osa mirror list`
95+
96+
List all active mirrors.
97+
98+
```bash
99+
osa mirror list
100+
```
101+
102+
### `osa mirror info`
103+
104+
Show detailed information about a mirror.
105+
106+
```bash
107+
osa mirror info abc123def456
108+
```
109+
110+
### `osa mirror delete`
111+
112+
Delete a mirror and its databases.
113+
114+
```bash
115+
osa mirror delete abc123def456
116+
osa mirror delete abc123def456 --yes # skip confirmation
117+
```
118+
119+
### `osa mirror refresh`
120+
121+
Re-copy production databases into an existing mirror (resets to current production state).
122+
123+
```bash
124+
osa mirror refresh abc123def456
125+
osa mirror refresh abc123def456 -c hed # refresh only HED
126+
```
127+
128+
### `osa mirror sync`
129+
130+
Run sync pipeline against a mirror's databases (populates from public sources).
131+
132+
```bash
133+
osa mirror sync abc123def456 # sync all types
134+
osa mirror sync abc123def456 --type github # sync only GitHub
135+
```
136+
137+
Supported sync types: `github`, `papers`, `docstrings`, `mailman`, `faq`, `beps`, `all`.
138+
139+
### `osa mirror pull`
140+
141+
Download mirror databases locally for offline development.
142+
143+
```bash
144+
osa mirror pull abc123def456
145+
osa mirror pull abc123def456 -c hed -o ./data/knowledge
146+
```
147+
148+
## API Endpoints
149+
150+
All mirror endpoints require authentication (BYOK or admin key).
151+
152+
| Method | Endpoint | Description |
153+
|--------|----------|-------------|
154+
| `POST` | `/mirrors` | Create a mirror |
155+
| `GET` | `/mirrors` | List active mirrors |
156+
| `GET` | `/mirrors/{id}` | Get mirror metadata |
157+
| `DELETE` | `/mirrors/{id}` | Delete a mirror |
158+
| `POST` | `/mirrors/{id}/refresh` | Re-copy from production |
159+
| `POST` | `/mirrors/{id}/sync` | Run sync pipeline |
160+
| `GET` | `/mirrors/{id}/download/{community}` | Download SQLite file |
161+
162+
### Create Mirror
163+
164+
```bash
165+
curl -X POST https://api.osc.earth/osa/mirrors \
166+
-H "Content-Type: application/json" \
167+
-H "X-OpenRouter-Key: your-key" \
168+
-d '{"community_ids": ["hed", "bids"], "ttl_hours": 48, "label": "my-test"}'
169+
```
170+
171+
Response:
172+
173+
```json
174+
{
175+
"mirror_id": "abc123def456",
176+
"community_ids": ["hed", "bids"],
177+
"created_at": "2026-03-07T12:00:00+00:00",
178+
"expires_at": "2026-03-09T12:00:00+00:00",
179+
"label": "my-test",
180+
"size_bytes": 1048576,
181+
"expired": false
182+
}
183+
```
184+
185+
### Query with Mirror
186+
187+
Include the `X-Mirror-ID` header with any existing endpoint:
188+
189+
```bash
190+
curl -X POST https://api.osc.earth/osa/hed/ask \
191+
-H "Content-Type: application/json" \
192+
-H "X-OpenRouter-Key: your-key" \
193+
-H "X-Mirror-ID: abc123def456" \
194+
-d '{"question": "What is HED?"}'
195+
```
196+
197+
## Resource Limits
198+
199+
| Limit | Default |
200+
|-------|---------|
201+
| Max mirrors total | 50 |
202+
| Max mirrors per BYOK user | 2 |
203+
| Max TTL | 7 days (168 hours) |
204+
| Default TTL | 48 hours |
205+
206+
Expired mirrors are automatically cleaned up by the server every hour. Admin key holders are only subject to the global mirror cap.
207+
208+
## Architecture
209+
210+
### Storage Layout
211+
212+
```
213+
data/
214+
knowledge/ # Production databases (unchanged)
215+
hed.db
216+
bids.db
217+
mirrors/ # Ephemeral copies
218+
abc123def456/ # mirror_id
219+
_metadata.json # Timestamps, label, communities
220+
hed.db # Copy of production hed.db
221+
bids.db # Copy of production bids.db
222+
```
223+
224+
### ContextVar Routing
225+
226+
All database access flows through `get_db_path(project)` in `src/knowledge/db.py`. When a mirror is active:
227+
228+
1. Request arrives with `X-Mirror-ID: abc123` header
229+
2. Middleware validates the mirror exists and isn't expired
230+
3. Middleware sets a `contextvars.ContextVar` for the request
231+
4. `get_db_path("hed")` returns `data/mirrors/abc123/hed.db` instead of `data/knowledge/hed.db`
232+
5. All search, sync, and tool functions work unmodified
233+
6. Middleware resets the ContextVar when the request completes
234+
235+
### Security
236+
237+
- Mirror IDs are validated against path traversal (alphanumeric, hyphens, underscores only, max 64 chars)
238+
- Community IDs are validated at the API boundary
239+
- Per-user rate limits prevent resource exhaustion
240+
- Expired mirrors return HTTP 410 Gone
241+
- Auto-cleanup runs hourly

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,7 @@ nav:
161161
- Extensions: osa/registry/extensions.md
162162
- CLI Reference: osa/cli-reference.md
163163
- API Reference: osa/api-reference.md
164+
- Database Mirrors: osa/mirrors.md
164165
- Knowledge Sync: osa/knowledge-sync.md
165166
- Sync Status: osa/status.md
166167
- Deployment:

0 commit comments

Comments
 (0)