You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: document the database primitive — lifecycle, governance, and DDL
Adds docs/databases.md covering the database as NodeDB's top-level
container: default database, reserved ID range, lifecycle DDL (CREATE,
DROP, RENAME, clone, mirror, promote, materialize), connection binding,
database-scoped roles, tenant scoping, quota fields (memory, storage,
QPS, connections, cache weight, maintenance CPU), clone and mirror
semantics with consistency guarantees, backup/restore, and migration
paths.
Extends docs/architecture.md with the resource governance section:
the three-tier hierarchy (global ceiling → database budget → tenant
budget → engine usage), MemoryGovernor reservation flow, the two-phase
connection semaphore, WFQ/DRR scheduling in the SPSC bridge, priority-
aware WAL group commit, per-database weighted LRU document cache, and
the per-database maintenance CPU budget tracker.
Extends docs/query-language.md with the full database DDL surface:
CREATE/DROP/RENAME DATABASE, ALTER DATABASE SET QUOTA, CLONE DATABASE,
MIRROR DATABASE, MOVE TENANT, USE DATABASE, SHOW DATABASES, SHOW
DATABASE LINEAGE/MIRROR STATUS/QUOTA/USAGE, KILL SESSION, and audit
DML controls. All examples annotate what each variant does and when it
fails.
Updates docs/README.md to link the new databases guide.
Copy file name to clipboardExpand all lines: docs/architecture.md
+162Lines changed: 162 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -154,6 +154,168 @@ Single-shard writes bypass the sequencer entirely and go directly through the re
154
154
155
155
All engines share the same snapshot, transaction context, and memory budget. A query that combines vector similarity, graph traversal, spatial filtering, and document field access executes inside one process — no network hops between engines, no application-level joins.
156
156
157
+
## Resource Governance
158
+
159
+
NodeDB enforces multi-tier resource isolation to prevent a single tenant or database from starving others. Resource governance encompasses memory, connections, request rate, storage I/O scheduling, WAL commit priority, document cache allocation, background task CPU budgets, and compute bounds.
160
+
161
+
### Three-Tier Resource Hierarchy
162
+
163
+
All request-path resource decisions follow a **global ceiling → database budget → tenant budget → engine internal usage** hierarchy:
164
+
165
+
-**Global ceiling**: Cluster-wide configuration (e.g., `cluster.max_memory_bytes`). Applies to all databases.
166
+
-**Database budget**: Per-database quotas set via `ALTER DATABASE name SET QUOTA (...)`. Applies to all tenants within that database.
167
+
-**Tenant budget**: Per-tenant quotas within a database set via `ALTER TENANT name IN DATABASE db SET QUOTA (...)`. Applies within the narrowest scope.
168
+
-**Engine internal usage**: Governed by `MemoryGovernor`; descends the hierarchy to reserve bytes.
169
+
170
+
Admission ordering for requests:
171
+
172
+
1. Identity & database access (established during authentication)
Memory reservation flips the order (largest scope first to fail fast): global → database → tenant → engine. Failure at any layer aborts before the next is consulted.
178
+
179
+
### Memory Governor
180
+
181
+
Located in `nodedb-mem/src/governor.rs`, the `MemoryGovernor` enforces hierarchical memory reservations:
Threegroupsmaximumtoavoidfsyncamplification.The `priority_class` fieldissetperdatabasevia `WITH (priority_class='...')` on `CREATE` / `ALTERDATABASESETQUOTA`.Critical-classdatabases' writes are flushed to durable storage first; bulk writes extend the timeout window and reduce fsync frequency.
239
+
240
+
### Document Cache — Per-Database Weighted LRU
241
+
242
+
The document cache in `nodedb/src/engine/sparse/doc_cache.rs` is a weighted LRU sharded by `DatabaseId`. Each shard'scapacityshareisproportionalto `database.quota.cache_weight` (default1).
243
+
244
+
`CacheKey` includesboth `database_id` and `tenant_id`.Evictionprefersthedatabasewiththehighestcurrent-vs-weightovershoot, ensuringhotdatabasesdonotevictcolddatabasesbelowtheirproportionalshare.
These pre-auth buckets are consulted before SCRAM/Argon2 verification (cheap exit path) and run in constant time with a uniform delay on any denial to prevent timing leaks.
275
+
276
+
### Per-Tenant Compute Caps
277
+
278
+
Two per-tenant compute bounds are enforced at planner time via `nodedb/src/control/planner/sql_plan_convert/`:
279
+
280
+
- **`max_vector_dim`**: Maximum vector dimensionality. Vector inserts / index operations above this bound are rejected with `TENANT_VECTOR_DIM_EXCEEDED`.
281
+
- **`max_graph_depth`**: Maximum graph traversal depth. MATCH queries / graph traversals declaring depth > this bound are rejected with `TENANT_GRAPH_DEPTH_EXCEEDED`.
282
+
283
+
Both are surfaced in `SHOW TENANT QUOTA` output.
284
+
285
+
### Metrics & Observability
286
+
287
+
All resource consumption is observable via Prometheus metrics prefixed `nodedb_`:
All metrics are dimensionalized by database and tenant to enable per-customer tracking and alerting.
318
+
157
319
## Cross-Engine Identity
158
320
159
321
All engines use a unified, distributed identity space called **surrogate identity**. Each row, cell, node, and document has a surrogate ID (u64) that is globally unique within a database. Surrogates enable fused queries that combine filtering across all engines in a single bitmap.
0 commit comments