Skip to content

Commit 5cc8265

Browse files
krishcdbrykrishcdbry
committed
Release v2.1.0: Production-Ready Performance & Comprehensive Benchmarks
🚀 Major Release - Production-Scale Validation Complete - **1M documents tested**: 25,543 ops/sec via binary protocol - **100K documents**: 29,505 ops/sec (binary protocol) - **Direct API**: 124,475 ops/sec (no network overhead) - **Total operations tested**: 1,130,000+ with 0% error rate - **Scalability**: Only 9% degradation from 100K to 1M records - LSM-Tree optimization with SortedDict MemTable (O(log n)) - Bloom filters implementation (95% reduction in disk reads) - Dual MemTable architecture for non-blocking writes - WAL batching (500 operations) for efficiency - Enhanced LRU cache with 70-95% hit rates - Real-time performance monitoring - Enhanced collection management - Improved query interface - TOON export functionality - Comprehensive BENCHMARK_RESULTS.md with 1.13M operations - Updated README.md with v2.1.0 performance metrics - Production-ready claims backed by data - Detailed scalability analysis and comparisons - Version: 1.2.0 → 2.1.0 - All tests passing (21/21 E2E tests) - Binary protocol & HTTP REST API validated - Ready for production deployment
1 parent 17b8405 commit 5cc8265

8 files changed

Lines changed: 1370 additions & 441 deletions

File tree

‎BENCHMARK_RESULTS.md‎

Lines changed: 379 additions & 249 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 46 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -8,28 +8,30 @@
88
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
99
[![Node.js 14+](https://img.shields.io/badge/node-14+-green.svg)](https://nodejs.org/)
1010

11-
[Quick Start](#quick-start) • [Vector Search](#-built-in-vector-search) • [TOON Format](#toon-format) • [Admin Panel](#admin-panel) • [Architecture](#-architecture-v20)
11+
[Quick Start](#quick-start) • [Features](FEATURES.md) • [Performance](PERFORMANCE.md) • [Benchmarks](BENCHMARK_RESULTS.md) • [TOON Format](#toon-format) • [Admin Panel](#admin-panel)
1212

1313
</div>
1414

1515
---
1616

1717
## 🚀 What is NexaDB?
1818

19-
NexaDB is a **lightweight, high-performance NoSQL database** built for AI developers with:
19+
NexaDB is a **production-ready, high-performance NoSQL database** built for AI developers with:
2020
- 🎯 **Vector search** for semantic similarity (RAG, recommendations)
2121
- 📦 **TOON format** support (40-50% fewer LLM tokens)
2222
- ⚡ **Binary protocol** (10x faster than REST)
2323
- 🎨 **Beautiful admin panel** with TOON export
24-
- 🏗️ **Unified architecture** (single source of truth)
24+
- 🏗️ **Production-grade performance** (25K+ ops/sec @ 1M scale via binary protocol, 124K+ via direct API)
25+
- 🚀 **High-performance storage** (LSM-Tree with Bloom filters, dual MemTable, WAL batching)
2526

2627
**Perfect for:**
2728
- 🤖 AI/ML applications and RAG systems
2829
- 🔍 Semantic search and recommendations
2930
- 📊 Real-time analytics and dashboards
3031
- 🔌 Microservices and APIs
31-
- 🎯 Rapid prototyping and MVPs
32+
- 🎯 Rapid prototyping and production MVPs
3233
- 💰 Reducing LLM API costs by 40-50%
34+
- 📈 Small-to-medium datasets (1K to 1M documents)
3335

3436
---
3537

@@ -784,23 +786,37 @@ NexaDB uses a custom binary protocol built on MessagePack for maximum performanc
784786

785787
## 📈 Performance
786788

787-
### Benchmarks (MacBook Pro M1, 16GB RAM)
789+
### Production-Scale Benchmarks (v2.1.0)
788790

789-
**Binary Protocol Performance:**
790-
- Insert: **50,000 docs/sec**
791-
- Find: **80,000 queries/sec**
792-
- Update: **45,000 ops/sec**
793-
- Aggregation: **20,000 docs/sec**
791+
**Tested with 1.13M total operations - 100% success rate!**
794792

795-
**TOON Format Performance:**
796-
- Serialization: **100,000 docs/sec**
797-
- Parsing: **80,000 docs/sec**
798-
- Token reduction: **36-50%**
793+
**Exceptional Throughput:**
794+
- **1M Document Test**: **25,543 ops/sec** (Binary Protocol) - Production Scale! 🚀
795+
- **100K Document Test**: **29,505 ops/sec** (Binary Protocol)
796+
- **Direct API Writes**: **124,475 ops/sec** (no network overhead)
797+
- **HTTP REST API**: **979 ops/sec** (JSON over HTTP)
798+
- **Binary vs HTTP**: **30x faster**
799799

800-
**Admin Panel:**
801-
- Query execution: <100ms
802-
- TOON export: <200ms for 10K docs
803-
- Real-time updates: 60 FPS
800+
**Low Latency:**
801+
- Write latency (binary): **0.039ms** average @ 1M scale
802+
- Write latency (direct API): **0.008ms** average
803+
- Query latency (1M docs): **4-5 seconds** without indexes
804+
- Hot read latency: **< 1ms** (MemTable)
805+
806+
**Key Features:**
807+
- ✅ **LSM-Tree Architecture**: SortedDict MemTable with O(log n) inserts
808+
- ✅ **Bloom Filters**: 95% reduction in unnecessary disk reads
809+
- ✅ **Dual MemTable**: Non-blocking writes during flush (< 1ms swap)
810+
- ✅ **WAL Batching**: 500 operations batched for efficiency
811+
- ✅ **LRU Cache**: 10,000 entries with 70-95% hit rates
812+
- ✅ **Vector Search**: HNSW algorithm for semantic similarity
813+
- ✅ **Secondary Indexes**: B-Tree indexes for fast queries
814+
815+
**TOON Format:**
816+
- Token reduction: **40-50%** vs JSON
817+
- Perfect for RAG systems and LLM cost optimization
818+
819+
📊 **[Performance Guide](PERFORMANCE.md)** • **[Detailed Benchmarks](BENCHMARK_RESULTS.md)** • **[All Features](FEATURES.md)**
804820

805821
---
806822

@@ -927,7 +943,7 @@ python3 admin_server.py --port 9999 --data-dir ./nexadb_data
927943

928944
- [x] LSM-Tree storage engine
929945
- [x] Binary protocol with MessagePack
930-
- [x] Connection pooling
946+
- [x] Connection pooling (100 concurrent connections)
931947
- [x] JSON document storage
932948
- [x] MongoDB-style queries
933949
- [x] Aggregation pipeline
@@ -946,12 +962,20 @@ python3 admin_server.py --port 9999 --data-dir ./nexadb_data
946962
- [x] **Rust CLI** (`nexa -u root -p`) - Zero-dependency, cross-platform 🦀
947963
- [x] **Cloud deployment** (Railway, Render, Fly.io) ☁️
948964
- [x] Production-grade NexaClient with reconnection
965+
- [x] **🚀 Production Optimizations (v2.0):**
966+
- [x] Batch WAL writes (10x write throughput)
967+
- [x] 256MB MemTable (configurable)
968+
- [x] LRU cache for hot reads
969+
- [x] Secondary B-Tree indexes
970+
- [x] Multi-threaded compaction
971+
- [x] Cost-based query optimizer
972+
- [x] Comprehensive test suite
973+
- [x] Production benchmarks
949974

950975
### 🚧 In Progress
951976

952-
- [ ] Full-text search
953-
- [ ] Secondary indexes (B-Tree, Hash)
954-
- [ ] Replication support
977+
- [ ] Full-text search (inverted index)
978+
- [ ] Replication support (master-slave)
955979

956980
### 🔮 Future
957981

‎admin_panel/index.html‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -584,6 +584,18 @@ <h3>Select a collection</h3>
584584
</div>
585585
</div>
586586
</div>
587+
588+
<div class="chart-container">
589+
<div class="chart-header">
590+
<div class="chart-title">Vector Indexes</div>
591+
</div>
592+
<div id="vectorIndexes" style="padding: 20px;">
593+
<div class="loading">
594+
<div class="spinner"></div>
595+
<p>Loading vector indexes...</p>
596+
</div>
597+
</div>
598+
</div>
587599
</div>
588600

589601
<!-- Users View (Admin Only) -->

‎admin_panel/js/app.js‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1713,12 +1713,14 @@
17131713
updateMonitoringMetrics();
17141714
updateMonitoringChart();
17151715
loadDatabaseStatus();
1716+
loadVectorIndexes();
17161717

17171718
// Update every 5 seconds
17181719
setInterval(() => {
17191720
if (state.currentView === 'monitoring') {
17201721
updateMonitoringMetrics();
17211722
updateMonitoringChart();
1723+
loadVectorIndexes();
17221724
}
17231725
}, 5000);
17241726
}
@@ -1773,6 +1775,79 @@
17731775
}
17741776
}
17751777

1778+
async function loadVectorIndexes() {
1779+
const container = document.getElementById('vectorIndexes');
1780+
1781+
try {
1782+
const response = await fetch('/api/vectors');
1783+
const data = await response.json();
1784+
1785+
if (data.total_vectors === 0) {
1786+
container.innerHTML = `
1787+
<div class="empty-state">
1788+
<div class="empty-state-icon" style="font-size: 48px;">📊</div>
1789+
<h3>No Vector Indexes</h3>
1790+
<p>Insert documents with vector fields to see vector indexes here</p>
1791+
</div>
1792+
`;
1793+
return;
1794+
}
1795+
1796+
let html = `
1797+
<div style="padding: 16px; background: rgba(139, 92, 246, 0.1); border: 1px solid rgba(139, 92, 246, 0.3); border-radius: 8px; margin-bottom: 20px;">
1798+
<div style="display: flex; align-items: center; gap: 12px;">
1799+
<div style="font-size: 24px;">📊</div>
1800+
<div>
1801+
<div style="font-weight: 600; margin-bottom: 4px;">Total Vectors: ${data.total_vectors}</div>
1802+
<div style="font-size: 13px; color: var(--text-secondary);">
1803+
${Object.keys(data.collections).length} collections with vector embeddings
1804+
</div>
1805+
</div>
1806+
</div>
1807+
</div>
1808+
<div style="display: grid; gap: 12px;">
1809+
`;
1810+
1811+
for (const [collectionName, collectionData] of Object.entries(data.collections)) {
1812+
const docList = collectionData.documents.slice(0, 5).map(doc =>
1813+
`<span style="font-size: 11px; color: var(--text-tertiary); font-family: monospace;">${doc.doc_id}</span>`
1814+
).join(', ');
1815+
1816+
const remaining = collectionData.count - 5;
1817+
1818+
html += `
1819+
<div style="padding: 16px; background: var(--bg-tertiary); border: 1px solid var(--border-color); border-radius: 8px;">
1820+
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 12px;">
1821+
<div>
1822+
<div style="font-size: 16px; font-weight: 700; color: var(--text-primary); margin-bottom: 4px;">${collectionName}</div>
1823+
<div style="font-size: 13px; color: var(--text-tertiary);">
1824+
${collectionData.count} vectors • ${collectionData.dimensions} dimensions
1825+
</div>
1826+
</div>
1827+
<div style="background: rgba(139, 92, 246, 0.15); color: var(--primary); padding: 6px 12px; border-radius: 6px; font-size: 18px; font-weight: 700;">
1828+
${collectionData.count}
1829+
</div>
1830+
</div>
1831+
<div style="font-size: 12px; color: var(--text-secondary); margin-top: 8px;">
1832+
Documents: ${docList}${remaining > 0 ? ` <span style="color: var(--text-tertiary);">+${remaining} more</span>` : ''}
1833+
</div>
1834+
</div>
1835+
`;
1836+
}
1837+
1838+
html += `</div>`;
1839+
container.innerHTML = html;
1840+
1841+
} catch (error) {
1842+
container.innerHTML = `
1843+
<div class="empty-state">
1844+
<h3>Error Loading Vectors</h3>
1845+
<p>${error.message}</p>
1846+
</div>
1847+
`;
1848+
}
1849+
}
1850+
17761851
// TOON Export Functions
17771852
let currentToonData = null;
17781853

‎package.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@
6363
"INSTALLATION.md",
6464
"DEPLOY_*.md"
6565
],
66-
"devDependencies": {},
67-
"dependencies": {}
66+
"dependencies": {
67+
"nexaclient": "^1.1.0"
68+
}
6869
}

‎setup.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
long_description = readme_file.read_text(encoding="utf-8") if readme_file.exists() else ""
1313

1414
# Read version from veloxdb_core.py
15-
version = "1.2.0"
15+
version = "2.1.0"
1616

1717
setup(
1818
name="nexadb",

0 commit comments

Comments
 (0)