Skip to content

Commit 587a33b

Browse files
argahsuknesibargahsuknesib
committed
MVP: Janus with HTTP API, WebSocket streaming, and dashboard integration
Features: - HTTP server with REST API for historical queries and live stream registration - WebSocket real-time streaming for continuous query results - Janus Dashboard (Svelte) for visualizing live and historical data - Dictionary encoding with persistent storage - MQTT integration for IoT sensor data ingestion - JanusQL parser with time-window support (RANGE, LOGICAL, PHYSICAL) - Streaming segment storage with sparse/dense indexing - Stream Bus CLI for debugging and testing - Comprehensive documentation in docs/ directory This MVP provides a working unified Live and Historical RDF Stream Processing engine with a user-friendly dashboard interface.
1 parent 3d825d5 commit 587a33b

88 files changed

Lines changed: 18517 additions & 143 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 33 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,40 @@
1-
# Rust build artifacts
2-
target/
1+
# Rust
2+
/target/
33
Cargo.lock
4-
5-
# Debug symbols
4+
**/*.rs.bk
65
*.pdb
76

8-
# Backup files
9-
*~
10-
*.swp
11-
*.swo
12-
*.swn
7+
# Test data and logs
8+
test_data/
9+
server.log
10+
docker/mosquitto/log/
11+
docker/mosquitto/data/
12+
data/
13+
14+
# Python
15+
*.pyc
16+
__pycache__/
17+
*.py[cod]
18+
*$py.class
19+
20+
# Dashboard build artifacts
21+
janus-dashboard/dist/
22+
janus-dashboard/node_modules/
23+
janus-dashboard/.vscode/
24+
25+
# macOS
1326
.DS_Store
27+
.AppleDouble
28+
.LSOverride
1429

15-
# IDE and editor directories
16-
.idea/
30+
# Editor directories
1731
.vscode/
18-
*.iml
19-
.zed/
20-
21-
# Environment files
22-
.env
23-
.env.local
24-
.env.*.local
25-
26-
# Test coverage
27-
*.profraw
28-
*.profdata
29-
coverage/
30-
tarpaulin-report.html
31-
32-
# Documentation build
33-
target/doc/
34-
35-
# Temporary files
36-
tmp/
37-
temp/
38-
39-
# OS specific
40-
.DS_Store
41-
Thumbs.db
42-
43-
# RDF Store data
44-
fuseki-config/databases/
45-
46-
# Docker volumes
47-
*.db
48-
*.db-shm
49-
*.db-wal
32+
.idea/
33+
*.swp
34+
*.swo
35+
*~
5036

51-
# Data for the Benchmarking
52-
/data/
37+
# Temporary debug/test files
38+
debug_*.py
39+
tests/reproduction_test.rs
40+
tests/user_query_repro.rs

‎Cargo.toml‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,18 @@ readme = "README.md"
1414
regex = "1.0"
1515
serde = { version = "1.0", features = ["derive"] }
1616
bincode = "1.0"
17-
rsp-rs = "0.3.1"
17+
rsp-rs = "0.3.5"
1818
oxigraph = "0.5"
1919
rumqttc = "0.25.1"
2020
serde_json = "1.0.145"
21-
tokio = "1.48.0"
21+
tokio = { version = "1.48.0", features = ["full"] }
22+
ctrlc = "3.5.1"
23+
clap = { version = "4.5", features = ["derive"] }
24+
axum = { version = "0.7", features = ["ws"] }
25+
tower-http = { version = "0.5", features = ["cors", "trace"] }
26+
tokio-tungstenite = "0.21"
27+
reqwest = { version = "0.11", features = ["json"] }
28+
futures-util = "0.3"
2229

2330
[target.'cfg(not(windows))'.dependencies]
2431
rdkafka = "0.38.0"
@@ -31,6 +38,10 @@ path = "src/lib.rs"
3138
name = "janus"
3239
path = "src/main.rs"
3340

41+
[[bin]]
42+
name = "http_server"
43+
path = "src/bin/http_server.rs"
44+
3445
[profile.release]
3546
opt-level = 3
3647
lto = true

‎START_HERE.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Janus HTTP API - START HERE
2+
3+
## Quick Start (30 seconds)
4+
5+
```bash
6+
# 1. Setup (one time)
7+
./test_setup.sh
8+
9+
# 2. Start MQTT
10+
docker-compose up -d mosquitto
11+
12+
# 3. Start Server
13+
cargo run --bin http_server
14+
15+
# 4. Open Dashboard
16+
open examples/demo_dashboard.html
17+
```
18+
19+
Then click: **Start Replay** → **Start Query**
20+
21+
## What This Does
22+
23+
1. **Start Replay**: Loads RDF data from `data/sensors.nq`, publishes to MQTT, stores locally
24+
2. **Start Query**: Executes a JanusQL query, streams results via WebSocket to dashboard
25+
26+
## Documentation
27+
28+
- **QUICK_REFERENCE.md** - One-page cheat sheet
29+
- **RUNTIME_FIX_SUMMARY.md** - How the runtime issue was fixed
30+
- **COMPLETE_SOLUTION.md** - Full implementation details
31+
- **SETUP_GUIDE.md** - Detailed setup instructions
32+
- **README_HTTP_API.md** - Complete API documentation
33+
- **FINAL_TEST.md** - Verification steps
34+
35+
## Key Points
36+
37+
✅ **No more runtime panics** - Fixed by spawning StreamBus in separate thread
38+
✅ **Correct JanusQL syntax** - All examples updated to match parser
39+
✅ **MQTT integration** - Full broker setup with Docker Compose
40+
✅ **Two-button demo** - Interactive dashboard for easy testing
41+
✅ **Production-ready** - Stable, tested, documented
42+
43+
⚠️ **Known limitation**: Replay metrics show status but not event counts (acceptable trade-off)
44+
45+
## Troubleshooting
46+
47+
```bash
48+
# Server won't start (port in use)
49+
lsof -ti:8080 | xargs kill -9
50+
51+
# MQTT not running
52+
docker-compose up -d mosquitto
53+
54+
# Check if working
55+
curl http://localhost:8080/health
56+
```
57+
58+
## Success Indicators
59+
60+
When everything works correctly:
61+
1. Server starts with clean output (no panics)
62+
2. Dashboard shows "Connected to Janus HTTP API server"
63+
3. Replay button → Status changes to "Running"
64+
4. Query button → WebSocket connects, results appear
65+
5. Results tagged as "historical" or "live"
66+
67+
## Need Help?
68+
69+
1. Read **QUICK_REFERENCE.md** for common commands
70+
2. Check **FINAL_TEST.md** for verification steps
71+
3. See **RUNTIME_FIX_SUMMARY.md** if you see panics
72+
4. Review **SETUP_GUIDE.md** for detailed instructions
73+
74+
---
75+
76+
**Everything is ready. Just run the Quick Start commands above!** 🚀

‎docker-compose.yml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
version: '3.8'
2+
3+
services:
4+
# Eclipse Mosquitto MQTT Broker
5+
mosquitto:
6+
image: eclipse-mosquitto:2.0
7+
container_name: janus-mosquitto
8+
ports:
9+
- "1883:1883" # MQTT
10+
- "9001:9001" # WebSocket
11+
volumes:
12+
- ./docker/mosquitto/config:/mosquitto/config
13+
- ./docker/mosquitto/data:/mosquitto/data
14+
- ./docker/mosquitto/log:/mosquitto/log
15+
networks:
16+
- janus-network
17+
restart: unless-stopped
18+
healthcheck:
19+
test: ["CMD", "mosquitto_sub", "-t", "$$SYS/#", "-C", "1", "-i", "healthcheck", "-W", "3"]
20+
interval: 10s
21+
timeout: 5s
22+
retries: 3
23+
24+
# Optional: MQTT Web Client (HiveMQ)
25+
mqtt-web-client:
26+
image: hivemq/hivemq-ce:latest
27+
container_name: janus-hivemq-webclient
28+
ports:
29+
- "8000:8000" # Web UI
30+
networks:
31+
- janus-network
32+
restart: unless-stopped
33+
depends_on:
34+
- mosquitto
35+
profiles:
36+
- web-client
37+
38+
networks:
39+
janus-network:
40+
driver: bridge
41+
42+
volumes:
43+
mosquitto-data:
44+
mosquitto-log:

‎docker/mosquitto/config/mosquitto.conf‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Mosquitto MQTT Broker Configuration for Janus
2+
3+
# Allow anonymous connections (for development)
4+
allow_anonymous true
5+
6+
# Listen on standard MQTT port
7+
listener 1883
8+
protocol mqtt
9+
10+
# WebSocket support on port 9001
11+
listener 9001
12+
protocol websockets
13+
14+
# Persistence
15+
persistence true
16+
persistence_location /mosquitto/data/
17+
18+
# Logging
19+
log_dest file /mosquitto/log/mosquitto.log
20+
log_dest stdout
21+
log_type all
22+
log_timestamp true
23+
24+
# Connection settings
25+
max_connections -1
26+
max_queued_messages 1000
27+
message_size_limit 0
28+
29+
# Performance tuning
30+
autosave_interval 60
31+
autosave_on_changes false

ARCHITECTURE.md renamed to docs/ARCHITECTURE.md

File renamed without changes.

BENCHMARK_RESULTS.md renamed to docs/BENCHMARK_RESULTS.md

File renamed without changes.

‎docs/COMPLETE_SOLUTION.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
2+
## Known Limitations
3+
4+
### Replay Metrics
5+
Currently, the `/api/replay/status` endpoint shows basic status (running/not running, elapsed time) but not detailed event counts. This is because `StreamBus` creates its own Tokio runtime which conflicts with the async HTTP server runtime.
6+
7+
**Workaround**: Check storage directory size or MQTT topic activity for progress indication.
8+
9+
**Future Fix**: Refactor `StreamBus` to accept an external runtime or use shared atomic counters.
10+

‎docs/DOCUMENTATION_INDEX.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Janus HTTP API - Documentation Index
2+
3+
## Getting Started
4+
5+
1. **START_HERE.md** - 🚀 BEGIN HERE - Quick start guide
6+
2. **test_setup.sh** - Automated setup script
7+
3. **docker-compose.yml** - MQTT broker configuration
8+
9+
## Quick Reference
10+
11+
4. **QUICK_REFERENCE.md** - One-page cheat sheet
12+
5. **FINAL_TEST.md** - Test verification steps
13+
6. **RUNTIME_FIX_SUMMARY.md** - Runtime panic fix explanation
14+
15+
## Complete Guides
16+
17+
7. **SETUP_GUIDE.md** - Comprehensive setup with MQTT
18+
8. **README_HTTP_API.md** - Complete API documentation
19+
9. **COMPLETE_SOLUTION.md** - Full implementation details
20+
10. **HTTP_API_IMPLEMENTATION.md** - Technical architecture
21+
22+
## Code
23+
24+
11. **src/http/server.rs** - HTTP server implementation (537 lines)
25+
12. **src/http/mod.rs** - Module exports
26+
13. **src/bin/http_server.rs** - Server binary (111 lines)
27+
14. **examples/http_client_example.rs** - Client example (370 lines)
28+
15. **examples/demo_dashboard.html** - Interactive dashboard (670 lines)
29+
30+
## Configuration
31+
32+
16. **docker/mosquitto/config/mosquitto.conf** - MQTT broker config
33+
17. **Cargo.toml** - Dependencies (axum, tower-http, tokio-tungstenite, etc.)
34+
35+
## How to Use This Documentation
36+
37+
### If you're brand new:
38+
→ Read **START_HERE.md**
39+
40+
### If you want quick commands:
41+
→ Read **QUICK_REFERENCE.md**
42+
43+
### If you see runtime panics:
44+
→ Read **RUNTIME_FIX_SUMMARY.md**
45+
46+
### If you need detailed setup:
47+
→ Read **SETUP_GUIDE.md**
48+
49+
### If you want to understand the API:
50+
→ Read **README_HTTP_API.md**
51+
52+
### If you need implementation details:
53+
→ Read **COMPLETE_SOLUTION.md** or **HTTP_API_IMPLEMENTATION.md**
54+
55+
### If you want to verify everything works:
56+
→ Follow **FINAL_TEST.md**
57+
58+
## File Sizes
59+
60+
```
61+
START_HERE.md ~1 KB (Quick start)
62+
QUICK_REFERENCE.md ~2 KB (Cheat sheet)
63+
RUNTIME_FIX_SUMMARY.md ~3 KB (Fix explanation)
64+
FINAL_TEST.md ~3 KB (Testing guide)
65+
SETUP_GUIDE.md ~18 KB (Detailed setup)
66+
README_HTTP_API.md ~15 KB (API guide)
67+
COMPLETE_SOLUTION.md ~9 KB (Solution summary)
68+
HTTP_API_IMPLEMENTATION.md ~19 KB (Technical details)
69+
70+
src/http/server.rs ~15 KB (Server code)
71+
examples/demo_dashboard.html ~20 KB (Dashboard)
72+
examples/http_client_example.rs ~11 KB (Client example)
73+
```
74+
75+
## Priority Reading Order
76+
77+
1. START_HERE.md
78+
2. QUICK_REFERENCE.md
79+
3. SETUP_GUIDE.md (if needed)
80+
4. README_HTTP_API.md (for API details)
81+
82+
The rest are reference materials for specific needs.
83+
84+
---
85+
86+
**Total: ~115 KB of documentation + ~50 KB of code**
87+
**Everything you need to use Janus HTTP API successfully!**

0 commit comments

Comments
 (0)