Skip to content

Commit 554d957

Browse files
committed
docs: document the database primitive — lifecycle, governance, and DDL
Adds docs/databases.md covering the database as NodeDB's top-level container: default database, reserved ID range, lifecycle DDL (CREATE, DROP, RENAME, clone, mirror, promote, materialize), connection binding, database-scoped roles, tenant scoping, quota fields (memory, storage, QPS, connections, cache weight, maintenance CPU), clone and mirror semantics with consistency guarantees, backup/restore, and migration paths. Extends docs/architecture.md with the resource governance section: the three-tier hierarchy (global ceiling → database budget → tenant budget → engine usage), MemoryGovernor reservation flow, the two-phase connection semaphore, WFQ/DRR scheduling in the SPSC bridge, priority- aware WAL group commit, per-database weighted LRU document cache, and the per-database maintenance CPU budget tracker. Extends docs/query-language.md with the full database DDL surface: CREATE/DROP/RENAME DATABASE, ALTER DATABASE SET QUOTA, CLONE DATABASE, MIRROR DATABASE, MOVE TENANT, USE DATABASE, SHOW DATABASES, SHOW DATABASE LINEAGE/MIRROR STATUS/QUOTA/USAGE, KILL SESSION, and audit DML controls. All examples annotate what each variant does and when it fails. Updates docs/README.md to link the new databases guide.
1 parent c87f85e commit 554d957

4 files changed

Lines changed: 650 additions & 0 deletions

File tree

docs/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ Welcome to the NodeDB docs. These guides explain what each engine does, when to
77
- [Getting Started](getting-started.md) — Prerequisites, build, run, first queries
88
- [Architecture](architecture.md) — How the three-plane execution model works (now includes cross-engine identity)
99
- [Query Language](query-language.md) — Full SQL reference (DDL, DML, engine-specific syntax, functions)
10+
- [Databases](databases.md) — Database lifecycle, quotas, clone/mirror, multi-tenancy, isolation
1011
- [Protocols](protocols.md) — Six wire protocols (pgwire, NDB, HTTP, RESP, ILP, Sync)
1112

1213
## Engine Guides

docs/architecture.md

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,168 @@ Single-shard writes bypass the sequencer entirely and go directly through the re
154154

155155
All engines share the same snapshot, transaction context, and memory budget. A query that combines vector similarity, graph traversal, spatial filtering, and document field access executes inside one process — no network hops between engines, no application-level joins.
156156

157+
## Resource Governance
158+
159+
NodeDB enforces multi-tier resource isolation to prevent a single tenant or database from starving others. Resource governance encompasses memory, connections, request rate, storage I/O scheduling, WAL commit priority, document cache allocation, background task CPU budgets, and compute bounds.
160+
161+
### Three-Tier Resource Hierarchy
162+
163+
All request-path resource decisions follow a **global ceiling → database budget → tenant budget → engine internal usage** hierarchy:
164+
165+
- **Global ceiling**: Cluster-wide configuration (e.g., `cluster.max_memory_bytes`). Applies to all databases.
166+
- **Database budget**: Per-database quotas set via `ALTER DATABASE name SET QUOTA (...)`. Applies to all tenants within that database.
167+
- **Tenant budget**: Per-tenant quotas within a database set via `ALTER TENANT name IN DATABASE db SET QUOTA (...)`. Applies within the narrowest scope.
168+
- **Engine internal usage**: Governed by `MemoryGovernor`; descends the hierarchy to reserve bytes.
169+
170+
Admission ordering for requests:
171+
172+
1. Identity & database access (established during authentication)
173+
2. **Tenant** rate/concurrency check → `TENANT_QUOTA_EXCEEDED`
174+
3. **Database** rate/concurrency check → `DATABASE_QUOTA_EXCEEDED`
175+
4. **Global** pressure check → `SERVER_OVERLOAD`
176+
177+
Memory reservation flips the order (largest scope first to fail fast): global → database → tenant → engine. Failure at any layer aborts before the next is consulted.
178+
179+
### Memory Governor
180+
181+
Located in `nodedb-mem/src/governor.rs`, the `MemoryGovernor` enforces hierarchical memory reservations:
182+
183+
```rust
184+
pub struct MemoryGovernor {
185+
global_ceiling: usize,
186+
database_budgets: HashMap<DatabaseId, Budget>,
187+
tenant_budgets: HashMap<(DatabaseId, TenantId), Budget>,
188+
engine_budgets: HashMap<EngineId, Budget>,
189+
}
190+
191+
impl MemoryGovernor {
192+
pub fn try_reserve(
193+
&self,
194+
db: DatabaseId,
195+
tenant: TenantId,
196+
engine: EngineId,
197+
size: usize,
198+
) -> Result<ReservationToken, BudgetError>
199+
// Reserves across all four levels: global → database → tenant → engine
200+
}
201+
```
202+
203+
`ReservationToken` is RAIIdropping it releases against all four levels atomically. This prevents leaks across plane boundaries. Every allocation site walks all four levels before touching memory.
204+
205+
### Connection Semaphore
206+
207+
Defined in `nodedb/src/control/server/listener.rs`, connection admission occurs in two phases:
208+
209+
**Phase 1 (at TLS accept):** Acquire a temporary global permit from `cluster.max_connections`.
210+
211+
**Phase 2 (post-auth):** After SCRAM/Argon2 handshake succeeds and `database_id`/`tenant_id` are known, convert the temporary permit into a database-bucketed + tenant-bucketed permit (3-level ref-counted). Verify both `database.quota.max_connections` and `tenant.quota.max_connections`.
212+
213+
On disconnect, all three ref-counted permits are released atomically. If Phase 2 fails, the temporary permit is dropped and an error is returned.
214+
215+
### SPSC BridgeWeighted-Fair Queue
216+
217+
The lock-free SPSC bridge between Control and Data Planes (`nodedb-bridge/src/wfq.rs`) implements **Weighted Fair Queueing (WFQ)** via **Deficit Round-Robin (DRR)**.
218+
219+
Each Data Plane core's request ring contains a set of virtual sub-queues, one per active `DatabaseId` on that core. The dispatcher schedules these sub-queues fairly by:
220+
221+
- Assigning each database a weight proportional to `database.quota.priority_class` (three tiers: `critical`, `standard`, `bulk`).
222+
- Running deficit round-robin: on each dispatcher tick, the next database with deficit > 0 is chosen; its requests are dequeued until deficit exhausts, then the next database is selected.
223+
- Computing backpressure (85% / 95%) per virtual queue (not globally), so one saturated database never throttles another's enqueue rate.
224+
- Preserving total ring capacity (bounded).
225+
226+
A database saturating its deficit on a core throttles only its own writes; co-resident databases continue flowing.
227+
228+
### WAL Group CommitPriority-Aware
229+
230+
Write-ahead log group commit in `nodedb-wal/src/group_commit.rs` separates commits into three independent priority-based fsync groups:
231+
232+
| Class | Behavior |
233+
| ---------- | -------------------------------------------------- |
234+
| `critical` | Dedicated fsync group, committed first |
235+
| `standard` | Default; batched with other standard-class commits |
236+
| `bulk` | Extended timeout, lower fsync rate |
237+
238+
Three groups maximum to avoid fsync amplification. The `priority_class` field is set per database via `WITH (priority_class='...')` on `CREATE` / `ALTER DATABASE SET QUOTA`. Critical-class databases' writes are flushed to durable storage first; bulk writes extend the timeout window and reduce fsync frequency.
239+
240+
### Document Cache — Per-Database Weighted LRU
241+
242+
The document cache in `nodedb/src/engine/sparse/doc_cache.rs` is a weighted LRU sharded by `DatabaseId`. Each shard's capacity share is proportional to `database.quota.cache_weight` (default 1).
243+
244+
`CacheKey` includes both `database_id` and `tenant_id`. Eviction prefers the database with the highest current-vs-weight overshoot, ensuring hot databases do not evict cold databases below their proportional share.
245+
246+
### Background TasksPer-Database CPU-Seconds Budget
247+
248+
The maintenance scheduler in `nodedb/src/control/maintenance/budget.rs` tracks per-database CPU-seconds spent in background tasks (compaction, index maintenance, cleanup) per minute and enforces a cap of `database.quota.maintenance_cpu_pct` (default 25%) of core time. Databases that exceed the cap defer their maintenance to the next minute window.
249+
250+
The following maintenance tasks acquire `MaintenanceLease` from the budget tracker (RAII handle):
251+
252+
- **Vector**HNSW node removal + neighbour-link remapping (compact operation)
253+
- **Graph**CSR compaction (level-based reorganization) + dangling edge sweep
254+
- **Timeseries**Segment compaction + continuous aggregation refresh
255+
- **Spatial**R-tree rebalancing
256+
- **Array**Tile compaction and version cleanup
257+
- **Full-Text Search**LSM level compaction via `run_fts_compaction()` in `nodedb/src/data/executor/handlers/compact_fts.rs`
258+
259+
All sites using `acquire_maintenance_lease(db, force)` observe the per-database budget.
260+
261+
### Rate Limiter
262+
263+
The rate limiter in `nodedb/src/control/security/ratelimit/limiter.rs` enforces request-rate caps using token buckets. Buckets are consulted in order of specificityfirst-deny wins:
264+
265+
```
266+
user:{id} → org:{id} → tenant:{id} → database:{id}
267+
```
268+
269+
The database bucket capacity is `database.quota.max_qps`. Additionally, pre-auth login rate limits are enforced:
270+
271+
- `login_ip:{addr}` — capacity `cluster.login_attempts_per_ip_per_min` (default 30)
272+
- `login_user:{username}` — capacity `cluster.login_attempts_per_user_per_min` (default 10)
273+
274+
These pre-auth buckets are consulted before SCRAM/Argon2 verification (cheap exit path) and run in constant time with a uniform delay on any denial to prevent timing leaks.
275+
276+
### Per-Tenant Compute Caps
277+
278+
Two per-tenant compute bounds are enforced at planner time via `nodedb/src/control/planner/sql_plan_convert/`:
279+
280+
- **`max_vector_dim`**: Maximum vector dimensionality. Vector inserts / index operations above this bound are rejected with `TENANT_VECTOR_DIM_EXCEEDED`.
281+
- **`max_graph_depth`**: Maximum graph traversal depth. MATCH queries / graph traversals declaring depth > this bound are rejected with `TENANT_GRAPH_DEPTH_EXCEEDED`.
282+
283+
Both are surfaced in `SHOW TENANT QUOTA` output.
284+
285+
### Metrics & Observability
286+
287+
All resource consumption is observable via Prometheus metrics prefixed `nodedb_`:
288+
289+
**Per-Database Metrics** (`nodedb_database_*{database="<name"}`):
290+
291+
- `qps` — Requests per second
292+
- `memory_bytes` — Total memory reserved
293+
- `storage_bytes` — Cumulative storage on-disk
294+
- `connections` — Open connections
295+
- `bridge_queue_depth` — Pending requests in SPSC bridge virtual queue
296+
- `wal_commit_latency_p99` — 99th percentile WAL fsync latency (milliseconds)
297+
- `maintenance_cpu_seconds` — CPU seconds spent in background tasks (per minute, reset window)
298+
- `mirror_lag_ms` — Replication lag to follower replicas (distributed deployments)
299+
300+
**Per-Tenant Metrics** (`nodedb_tenant_*{database="<name>", tenant="<tenant>"}`):
301+
302+
- `qps` — Requests per second
303+
- `memory_bytes` — Total memory reserved
304+
- `storage_bytes` — Cumulative storage on-disk
305+
306+
Counters are incremented at the sites where resources are consumed:
307+
308+
- **QPS**: Request dispatcher (Control Plane)
309+
- **Memory**: `MemoryGovernor::try_reserve()` success path
310+
- **Storage**: WAL append + segment compaction commit
311+
- **Connections**: Connection listener (Accept + Disconnect)
312+
- **Bridge queue depth**: SPSC enqueue / dequeue
313+
- **WAL latency**: Group commit fsync completion
314+
- **Maintenance CPU**: Lease acquisition / release
315+
- **Replication lag**: Raft follower log application
316+
317+
All metrics are dimensionalized by database and tenant to enable per-customer tracking and alerting.
318+
157319
## Cross-Engine Identity
158320
159321
All engines use a unified, distributed identity space called **surrogate identity**. Each row, cell, node, and document has a surrogate ID (u64) that is globally unique within a database. Surrogates enable fused queries that combine filtering across all engines in a single bitmap.

0 commit comments

Comments
 (0)