Skip to content

Commit d0884d6

Browse files
committed
feat(docs): add Graph Engine, Document Engine, and NodeDB-Lite sections
Introduce three entirely new documentation sections that were absent from the initial scaffold: - engines/graph/: CSR index internals, GraphRAG integration, traversal query reference, and engine overview - engines/document/: JSON functions, secondary indexes, and engine overview - nodedb-lite/: embedded/edge runtime overview, quickstart, Rust and JavaScript SDKs, iOS/Android guide, sync protocol, shape definitions, and offline compensation semantics Also adds query/aggregations.rdx and query/joins.rdx to the SQL query reference, covering GROUP BY / window functions and join strategies respectively.
1 parent 5fed3ac commit d0884d6

17 files changed

Lines changed: 4933 additions & 0 deletions
Lines changed: 358 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,358 @@
1+
---
2+
title: Document Field Functions
3+
description: doc_get, doc_exists, doc_array_contains UDFs for nested access.
4+
---
5+
6+
# Document Field Functions
7+
8+
NodeDB provides user-defined functions (UDFs) to extract and query nested fields within MessagePack documents. These enable flexible querying of semi-structured data.
9+
10+
## Core Functions
11+
12+
### doc_get(document, path) → value
13+
14+
Extract a value from a document at the specified path.
15+
16+
**Parameters**:
17+
- `document`: MessagePack blob column
18+
- `path`: String path to field (dot-notation or array subscript)
19+
20+
**Returns**: Extracted value, or NULL if path doesn't exist
21+
22+
**Examples**:
23+
24+
```sql
25+
-- Simple field
26+
SELECT doc_get(data, 'title') as title FROM documents;
27+
28+
-- Nested field
29+
SELECT doc_get(data, 'metadata.author') as author FROM documents;
30+
31+
-- Array element (0-indexed)
32+
SELECT doc_get(data, 'tags.0') as first_tag FROM documents;
33+
34+
-- Deep nesting
35+
SELECT doc_get(data, 'metadata.settings.notifications.email') as email_notif
36+
FROM documents;
37+
```
38+
39+
**Return type inference**:
40+
41+
```sql
42+
-- Automatically typed based on document value
43+
SELECT
44+
doc_get(data, 'title') as title_string, -- returns STRING
45+
doc_get(data, 'count') as count_int, -- returns INTEGER
46+
doc_get(data, 'score') as score_float, -- returns FLOAT
47+
doc_get(data, 'created') as timestamp, -- returns TIMESTAMP
48+
doc_get(data, 'active') as is_active -- returns BOOLEAN
49+
FROM documents;
50+
```
51+
52+
**Path notation**:
53+
54+
```sql
55+
-- Dot notation (preferred)
56+
doc_get(data, 'user.email') -- user.email field
57+
doc_get(data, 'tags.0') -- first element of tags array
58+
59+
-- Bracket notation (alternative)
60+
doc_get(data, 'user[email]')
61+
doc_get(data, 'tags[0]')
62+
63+
-- Escaped dots (if field contains literal dot)
64+
doc_get(data, 'config\.version') -- config.version field
65+
```
66+
67+
### doc_exists(document, path) → boolean
68+
69+
Check if a field exists in a document without extracting it.
70+
71+
**Parameters**:
72+
- `document`: MessagePack blob
73+
- `path`: String path
74+
75+
**Returns**: TRUE if path exists, FALSE otherwise
76+
77+
**Examples**:
78+
79+
```sql
80+
-- Filter documents that have a specific field
81+
SELECT id FROM documents
82+
WHERE doc_exists(data, 'author');
83+
84+
-- Check nested field
85+
SELECT id FROM documents
86+
WHERE doc_exists(data, 'metadata.email');
87+
88+
-- Combine with value extraction
89+
SELECT
90+
id,
91+
CASE WHEN doc_exists(data, 'verified')
92+
THEN doc_get(data, 'verified')
93+
ELSE FALSE
94+
END as is_verified
95+
FROM documents;
96+
```
97+
98+
**Use cases**:
99+
100+
```sql
101+
-- Find incomplete documents (missing required field)
102+
SELECT id FROM documents
103+
WHERE NOT doc_exists(data, 'required_field');
104+
105+
-- Process only documents with optional fields
106+
SELECT id, doc_get(data, 'optional_field')
107+
FROM documents
108+
WHERE doc_exists(data, 'optional_field');
109+
110+
-- Count documents with a feature
111+
SELECT doc_get(data, 'type'), COUNT(*)
112+
FROM documents
113+
WHERE doc_exists(data, 'premium_features')
114+
GROUP BY doc_get(data, 'type');
115+
```
116+
117+
### doc_array_contains(array_field, value) → boolean
118+
119+
Check if an array field contains a specific value.
120+
121+
**Parameters**:
122+
- `array_field`: Result of doc_get() extracting array, or array literal
123+
- `value`: Value to search for in array
124+
125+
**Returns**: TRUE if value found, FALSE otherwise
126+
127+
**Examples**:
128+
129+
```sql
130+
-- Simple array membership
131+
SELECT id FROM documents
132+
WHERE doc_array_contains(doc_get(data, 'tags'), 'ai');
133+
134+
-- Multiple membership checks
135+
SELECT id FROM documents
136+
WHERE doc_array_contains(doc_get(data, 'tags'), 'ai')
137+
AND doc_array_contains(doc_get(data, 'tags'), 'database');
138+
139+
-- Find documents NOT containing a tag
140+
SELECT id FROM documents
141+
WHERE NOT doc_array_contains(doc_get(data, 'tags'), 'deprecated');
142+
```
143+
144+
**Type matching**:
145+
146+
```sql
147+
-- String array
148+
WHERE doc_array_contains(doc_get(data, 'tags'), 'ai')
149+
150+
-- Integer array
151+
WHERE doc_array_contains(doc_get(data, 'priorities'), 1)
152+
153+
-- Mixed type arrays (searches by value AND type)
154+
SELECT id FROM documents
155+
WHERE doc_array_contains(doc_get(data, 'values'), 3.14)
156+
```
157+
158+
**Performance**: Uses index if array field is indexed; otherwise O(n) scan.
159+
160+
## Advanced Patterns
161+
162+
### Exploding Arrays
163+
164+
Convert array field into multiple rows:
165+
166+
```sql
167+
-- One row per tag
168+
SELECT id, UNNEST(doc_get(data, 'tags')) as tag
169+
FROM documents;
170+
171+
-- Result:
172+
-- id | tag
173+
-- doc:1 | ai
174+
-- doc:1 | database
175+
-- doc:2 | vector
176+
```
177+
178+
### Array Aggregation
179+
180+
Combine array values across documents:
181+
182+
```sql
183+
-- All unique tags across all documents
184+
SELECT DISTINCT tag
185+
FROM (
186+
SELECT UNNEST(doc_get(data, 'tags')) as tag
187+
FROM documents
188+
) t
189+
ORDER BY tag;
190+
191+
-- Most common tag
192+
SELECT tag, COUNT(*) as count
193+
FROM (
194+
SELECT UNNEST(doc_get(data, 'tags')) as tag
195+
FROM documents
196+
) t
197+
GROUP BY tag
198+
ORDER BY count DESC
199+
LIMIT 1;
200+
```
201+
202+
### Type Casting
203+
204+
Convert extracted fields to specific types:
205+
206+
```sql
207+
-- String to integer
208+
SELECT
209+
id,
210+
CAST(doc_get(data, 'status_code') AS INTEGER) as code
211+
FROM documents;
212+
213+
-- Timestamp parsing
214+
SELECT
215+
id,
216+
CAST(doc_get(data, 'created') AS TIMESTAMP) as created_at
217+
FROM documents
218+
ORDER BY created_at DESC;
219+
220+
-- JSON to type-specific value
221+
SELECT
222+
id,
223+
doc_get(data, 'score')::FLOAT as score
224+
FROM documents
225+
WHERE CAST(doc_get(data, 'score') AS FLOAT) > 0.8;
226+
```
227+
228+
### Combining with Full-Text Search
229+
230+
Extract text for BM25 ranking:
231+
232+
```sql
233+
-- Search full text, extract metadata
234+
SELECT
235+
id,
236+
doc_get(data, 'title') as title,
237+
bm25_score(doc_get(data, 'content'), 'neural networks') as relevance
238+
FROM documents
239+
WHERE bm25_score(doc_get(data, 'content'), 'neural networks') > 0.0
240+
ORDER BY relevance DESC
241+
LIMIT 20;
242+
```
243+
244+
### Conditional Extraction
245+
246+
Handle missing or optional fields:
247+
248+
```sql
249+
-- Use COALESCE for defaults
250+
SELECT
251+
id,
252+
COALESCE(doc_get(data, 'name'), 'Unknown') as name,
253+
COALESCE(doc_get(data, 'priority'), 0) as priority
254+
FROM documents;
255+
256+
-- Conditional logic
257+
SELECT
258+
id,
259+
CASE
260+
WHEN doc_exists(data, 'premium') THEN doc_get(data, 'premium')
261+
ELSE FALSE
262+
END as has_premium
263+
FROM documents;
264+
```
265+
266+
## Performance Considerations
267+
268+
### Path Depth
269+
270+
Extraction cost increases with nesting depth:
271+
272+
```sql
273+
-- Fast (1 level)
274+
doc_get(data, 'title') -- ~100ns
275+
276+
-- Medium (3 levels)
277+
doc_get(data, 'metadata.author.name') -- ~200ns
278+
279+
-- Slow (5+ levels)
280+
doc_get(data, 'a.b.c.d.e') -- ~500ns
281+
```
282+
283+
**Optimization**: Flatten deeply nested structures or precompute derived fields.
284+
285+
### Indexing for Performance
286+
287+
Index frequently extracted fields:
288+
289+
```sql
290+
-- This query is slow (no index):
291+
SELECT id FROM documents
292+
WHERE doc_get(data, 'status') = 'active'
293+
ORDER BY doc_get(data, 'created');
294+
-- Full scan: 100ms
295+
296+
-- Add index:
297+
CREATE INDEX idx_status ON documents (doc_get(data, 'status'));
298+
-- Now: 5-10ms (index scan)
299+
```
300+
301+
### Array Operations
302+
303+
Array membership is O(n) unless indexed:
304+
305+
```sql
306+
-- Without index: scans all elements
307+
SELECT id FROM documents
308+
WHERE doc_array_contains(doc_get(data, 'tags'), 'ai');
309+
-- ~50ms for 1M documents
310+
311+
-- With index:
312+
CREATE INDEX idx_tags ON documents USING UNNEST(doc_get(data, 'tags'));
313+
-- Now: 1-5ms
314+
```
315+
316+
## Examples: Real-World Queries
317+
318+
**User profile queries**:
319+
320+
```sql
321+
SELECT
322+
doc_get(data, 'username') as username,
323+
doc_get(data, 'email') as email,
324+
doc_array_contains(doc_get(data, 'roles'), 'admin') as is_admin
325+
FROM documents
326+
WHERE doc_get(data, 'type') = 'user'
327+
AND doc_exists(data, 'verified_at');
328+
```
329+
330+
**Log filtering and analysis**:
331+
332+
```sql
333+
SELECT
334+
doc_get(data, 'timestamp') as ts,
335+
doc_get(data, 'level') as level,
336+
doc_get(data, 'message') as msg
337+
FROM documents
338+
WHERE doc_get(data, 'type') = 'log'
339+
AND doc_get(data, 'level') IN ('ERROR', 'WARN')
340+
AND doc_get(data, 'timestamp') >= NOW() - INTERVAL '1 hour'
341+
ORDER BY ts DESC;
342+
```
343+
344+
**Product catalog with tags**:
345+
346+
```sql
347+
SELECT
348+
id,
349+
doc_get(data, 'name') as name,
350+
doc_get(data, 'price') as price
351+
FROM documents
352+
WHERE doc_get(data, 'type') = 'product'
353+
AND doc_array_contains(doc_get(data, 'categories'), 'electronics')
354+
AND CAST(doc_get(data, 'price') AS FLOAT) < 100
355+
ORDER BY doc_get(data, 'rating') DESC;
356+
```
357+
358+
See [Document Engine Overview](/engines/document/overview) for storage and [Secondary Indexes](/engines/document/secondary-indexes) for optimization.

0 commit comments

Comments
 (0)