Skip to content

Commit 57b86bc

Browse files
author
MPCoreDeveloper
committed
multiple refactors and API improvement
1 parent 86a57bc commit 57b86bc

28 files changed

Lines changed: 2023 additions & 334 deletions

‎README.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -323,3 +323,103 @@ Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for gui
323323

324324
If you find SharpCoreDB useful, please consider [sponsoring the project](https://github.com/sponsors/mpcoredeveloper)!
325325

326+
---
327+
328+
## 📊 Reproducible Benchmark Matrix (SQLite vs LiteDB vs SharpCoreDB)
329+
330+
Run the benchmarks yourself:
331+
332+
```bash
333+
cd tests/SharpCoreDB.Benchmarks
334+
# Runs StorageEngineComparisonBenchmark with all scenarios
335+
DOTNET_EnableHWIntrinsic=1 dotnet run -c Release --filter StorageEngineComparisonBenchmark
336+
```
337+
338+
**Scenarios covered (all pre-populated with the same data set):**
339+
- SQLite (baseline, single-file)
340+
- LiteDB (baseline, single-file)
341+
- SharpCoreDB Directory (PageBased) – unencrypted
342+
- SharpCoreDB Directory (PageBased) – AES-256 encrypted
343+
- SharpCoreDB SingleFile (.scdb) – unencrypted
344+
- SharpCoreDB SingleFile (.scdb) – AES-256 encrypted (fixed 32-byte key)
345+
346+
**Fairness/optimal paths:**
347+
- Page cache enabled (5k pages), WAL buffering on, validation off for benchmark runs
348+
- SingleFile uses `DatabaseOptions` with mmap enabled; encryption uses AES-256-GCM
349+
- Same schema and batch sizes as earlier results (Insert 1k, Update 500 random, Select with WHERE, Analytics columnar SIMD)
350+
351+
Use the produced `BenchmarkDotNet.Artifacts/results/*-report-github.md` to compare your run with ours.
352+
353+
---
354+
355+
## Latest Benchmark Summary (Jan 11, 2026)
356+
357+
Environment: Windows 11, i7-10850H, .NET 10.0.1, BenchmarkDotNet 0.15.8
358+
359+
Settings: IterationCount=5, WarmupCount=2, Toolchain=InProcessEmit
360+
361+
### Insert (1K rows)
362+
- PageBased: 7.63 ms (baseline, 2.01 MB alloc)
363+
- AppendOnly: 8.05 ms (1.96 MB)
364+
- SQLite: 4.62 ms (0.89 MB)
365+
- LiteDB: 7.73 ms (15.99 MB)
366+
- SCDB Dir (unencrypted): 7.69 ms (1.94 MB)
367+
- SCDB Dir (encrypted): 8.50 ms (1.94 MB)
368+
- SCDB Single (unencrypted): 13.41 ms (7.16 MB)
369+
- SCDB Single (encrypted): 13.74 ms (7.16 MB)
370+
371+
### Select (WHERE age > 30, with idx_age)
372+
- PageBased: 1.52 ms (2.21 MB)
373+
- AppendOnly: 2.10 ms (1.91 MB)
374+
- SCDB Dir (unencrypted): 1.55 ms (2.21 MB)
375+
- SCDB Dir (encrypted): 1.53 ms (2.21 MB)
376+
- SCDB Single (unencrypted): 7.23 µs (4.9 KB)
377+
- SCDB Single (encrypted): 7.21 µs (4.9 KB)
378+
379+
### Update (500 random rows)
380+
- PageBased: 7.44 ms (2.78 MB)
381+
- SCDB Dir (unencrypted): 7.41 ms (2.78 MB)
382+
- SCDB Dir (encrypted): 7.46 ms (2.79 MB)
383+
- SCDB Single (unencrypted): 7.86 ms (4.38 MB)
384+
- SCDB Single (encrypted): 8.05 ms (4.38 MB)
385+
- SQLite: 0.58 ms (193 KB)
386+
- AppendOnly: 366.51 ms (heavy GC, not suited for UPDATE)
387+
- LiteDB: 35.29 ms (25.34 MB)
388+
389+
### Analytics (SUM/AVG)
390+
- Columnar SIMD: ~0.043 ns (micro-measure)
391+
- SQLite: 325.81 µs (714 B)
392+
- LiteDB: 7.84 ms (10.68 MB)
393+
394+
## Comparison vs LiteDB
395+
- Insert (1K): SharpCoreDB PageBased ~7.63 ms vs LiteDB ~7.73 ms (near parity).
396+
- Update (500): SharpCoreDB ~7.4–8.0 ms vs LiteDB ~35.3 ms (~4.5x faster).
397+
- Select: SCDB Single ~7.2 µs (mmap), directory/page ~1.5 ms; LiteDB not measured here.
398+
- Analytics: Columnar SIMD >> LiteDB (µs vs ms).
399+
400+
## Use Cases & Ideal Settings
401+
See `docs/UseCases.md` for quick-start settings per scenario:
402+
- Web App (Concurrent Reads + OLTP Writes)
403+
- Reporting / Read-Heavy API
404+
- Bulk Import (ETL)
405+
- Analytics / BI
406+
- Desktop App (Single-User)
407+
- High-Concurrency API (Writes)
408+
409+
## Tuning Recommendations
410+
- Single-file inserts:
411+
- WalBufferSizePages=4096
412+
- FileShareMode=None (exclusive)
413+
- EnableMemoryMapping=true
414+
- Disable encryption for perf runs when acceptable
415+
- Directory/Page configs:
416+
- EnablePageCache=true; PageCacheCapacity≥20000
417+
- UseGroupCommitWal=true; WalMaxBatchDelayMs≈5–10
418+
- Keep `CREATE INDEX idx_age ON bench_records(age)` for select tests
419+
420+
## Notes
421+
- AppendOnly engine is optimized for insert/append; avoid UPDATE benchmarks.
422+
- Single-file SELECT benefits from memory-mapped I/O with very low allocations.
423+
424+
For full logs, see `tests/SharpCoreDB.Benchmarks/BenchmarkDotNet.Artifacts/results/`.
425+

‎docs/UseCases.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# SharpCoreDB Use Cases & Ideal Settings
2+
3+
This guide lists recommended configuration per use case. Copy settings into your setup (DatabaseConfig or DatabaseOptions).
4+
5+
## 1. Web App (Concurrent Reads + OLTP Writes)
6+
- Storage: PageBased
7+
- DatabaseConfig:
8+
- `EnablePageCache = true`
9+
- `PageCacheCapacity = 20000`
10+
- `UseMemoryMapping = true`
11+
- `UseGroupCommitWal = true`
12+
- `WalMaxBatchDelayMs = 5`
13+
- `WalDurabilityMode = DurabilityMode.FullSync`
14+
- Connection: pool connections (Cache=Shared), prefer short transactions.
15+
16+
## 2. Reporting / Read-Heavy API
17+
- Storage: SingleFile
18+
- DatabaseOptions:
19+
- `EnableMemoryMapping = true`
20+
- `FileShareMode = FileShare.Read`
21+
- `CreateImmediately = true`
22+
- DatabaseConfig:
23+
- `EnablePageCache = true`
24+
- `PageCacheCapacity = 25000`
25+
- Notes: Open as read-only where possible.
26+
27+
## 3. Bulk Import (ETL)
28+
- Storage: PageBased
29+
- DatabaseConfig:
30+
- `HighSpeedInsertMode = true`
31+
- `UseGroupCommitWal = true`
32+
- `EnableAdaptiveWalBatching = true`
33+
- `WalBatchMultiplier = 512`
34+
- `WalDurabilityMode = DurabilityMode.Async`
35+
- `EnablePageCache = true`
36+
- `PageCacheCapacity = 1000`
37+
- DatabaseOptions:
38+
- `WalBufferSizePages = 4096`
39+
- Notes: Disable encryption during import if safe; re-enable after.
40+
41+
## 4. Analytics / BI
42+
- Storage: Columnar (auto)
43+
- DatabaseConfig:
44+
- `EnableQueryCache = true`
45+
- `QueryCacheSize = 5000`
46+
- `EnablePageCache = true`
47+
- `PageCacheCapacity = 20000`
48+
- `UseMemoryMapping = true`
49+
- Notes: Build indexes for filter columns; rely on SIMD aggregates.
50+
51+
## 5. Desktop App (Single-User)
52+
- Storage: SingleFile
53+
- DatabaseOptions:
54+
- `EnableMemoryMapping = true`
55+
- `FileShareMode = FileShare.None`
56+
- `CreateImmediately = true`
57+
- `WalBufferSizePages = 2048`
58+
- DatabaseConfig:
59+
- `EnablePageCache = true`
60+
- `PageCacheCapacity = 10000`
61+
62+
## 6. High-Concurrency API (Writes)
63+
- Storage: PageBased
64+
- DatabaseConfig:
65+
- `UseGroupCommitWal = true`
66+
- `EnableAdaptiveWalBatching = true`
67+
- `WalBatchMultiplier = 256`
68+
- `WalDurabilityMode = DurabilityMode.Async`
69+
- Notes: Shard large write streams; avoid long transactions.
70+
71+
---
72+
73+
## Quick Code Examples
74+
75+
### SingleFile (Read-Heavy)
76+
```csharp
77+
var opts = DatabaseOptions.CreateSingleFileDefault();
78+
opts.EnableMemoryMapping = true;
79+
opts.FileShareMode = FileShare.Read;
80+
opts.WalBufferSizePages = 2048;
81+
opts.CreateImmediately = true;
82+
83+
var cfg = new DatabaseConfig
84+
{
85+
EnablePageCache = true,
86+
PageCacheCapacity = 25000,
87+
UseMemoryMapping = true
88+
};
89+
90+
var db = factory.CreateWithOptions(path + ".scdb", "password", opts);
91+
```
92+
93+
### PageBased (OLTP)
94+
```csharp
95+
var cfg = new DatabaseConfig
96+
{
97+
EnablePageCache = true,
98+
PageCacheCapacity = 20000,
99+
UseMemoryMapping = true,
100+
UseGroupCommitWal = true,
101+
WalMaxBatchDelayMs = 5,
102+
WalDurabilityMode = DurabilityMode.FullSync
103+
};
104+
105+
var db = (Database)factory.Create(dirPath, "password", false, cfg);
106+
```

0 commit comments

Comments
 (0)