Skip to content

EMBEDDED_GRAPE_DHT.md

Latest commit

 

History

History
342 lines (262 loc) · 10.5 KB

EMBEDDED_GRAPE_DHT.md

File metadata and controls

342 lines (262 loc) · 10.5 KB

Embedded Grape DHT - True P2P Architecture

🎯 Overview

This document explains the Embedded Grape DHT architecture that makes this exchange a true P2P system. Each node now runs its own Grape DHT server, eliminating the need for external infrastructure.

❌ Previous Architecture (CENTRALIZED)

┌──────────┐     ┌──────────┐     ┌──────────┐
│  Node 1  │     │  Node 2  │     │  Node 3  │
│(Exchange)│     │(Exchange)│     │(Exchange)│
└────┬─────┘     └────┬─────┘     └────┬─────┘
     │                │                │
     └────────────────┼────────────────┘
                      │
         ┌────────────▼────────────┐
         │   EXTERNAL GRAPE DHT    │  ❌ Single Point of Failure
         │   (Manual servers)      │  ❌ External Infrastructure
         │   grape --dp 20001      │  ❌ NOT True P2P
         │   grape --dp 20002      │
         └─────────────────────────┘

Problems:

  • ❌ Required manual Grape server setup
  • ❌ External infrastructure dependency
  • ❌ Single point of failure
  • ❌ NOT true P2P (centralized discovery)

✅ New Architecture (TRUE P2P)

┌─────────────────────┐  ┌─────────────────────┐  ┌─────────────────────┐
│      Node 1         │  │      Node 2         │  │      Node 3         │
│  ┌──────────────┐   │  │  ┌──────────────┐   │  │  ┌──────────────┐   │
│  │   Exchange   │   │  │  │   Exchange   │   │  │  │   Exchange   │   │
│  └──────────────┘   │  │  └──────────────┘   │  │  └──────────────┘   │
│  ┌──────────────┐   │  │  ┌──────────────┐   │  │  ┌──────────────┐   │
│  │Embedded Grape│◄──┼──┼─►│Embedded Grape│◄──┼──┼─►│Embedded Grape│   │
│  │DHT (Kademlia)│   │  │  │DHT (Kademlia)│   │  │  │DHT (Kademlia)│   │
│  └──────────────┘   │  │  └──────────────┘   │  │  └──────────────┘   │
└─────────────────────┘  └─────────────────────┘  └─────────────────────┘
        ↑  ↑                     ↑  ↑                     ↑  ↑
        │  └─────────────────────┘  └─────────────────────┘  │
        └───────────────────────────────────────────────────┘
                    Distributed Kademlia DHT Network

Benefits:

  • ✅ Each node runs its own Grape DHT server
  • ✅ Zero external infrastructure needed
  • ✅ No single point of failure
  • ✅ TRUE P2P distributed system
  • ✅ Kademlia DHT protocol (same as BitTorrent)

🚀 How to Use

Mode 1: True P2P (Embedded Grape) - RECOMMENDED ⭐

Each node runs its own Grape server, forming a distributed DHT network:

# Terminal 1 - Node 1 (Bootstrap node)
GRAPE_DHT_PORT=20001 \
GRAPE_API_PORT=30001 \
P2P_PORT=3001 \
npm start

# Terminal 2 - Node 2 (Connects to Node 1's DHT)
GRAPE_DHT_PORT=20002 \
GRAPE_API_PORT=30002 \
GRAPE_BOOTSTRAP_NODES=127.0.0.1:20001 \
P2P_PORT=3002 \
npm start

# Terminal 3 - Node 3 (Connects to existing DHT)
GRAPE_DHT_PORT=20003 \
GRAPE_API_PORT=30003 \
GRAPE_BOOTSTRAP_NODES=127.0.0.1:20001,127.0.0.1:20002 \
P2P_PORT=3003 \
npm start

Mode 2: Legacy (External Grape) - NOT RECOMMENDED

If you want to use external Grape servers (old way):

# Start external Grape servers
grape --dp 20001 --aph 30001 --bn '127.0.0.1:20002'
grape --dp 20002 --aph 40001 --bn '127.0.0.1:20001'

# Start nodes with embedded Grape disabled
EMBEDDED_GRAPE=false \
GRAPE_URL=http://127.0.0.1:30001 \
npm start

⚙️ Configuration

Environment Variables

Embedded Grape Configuration (True P2P)

# Enable/disable embedded Grape (default: true)
EMBEDDED_GRAPE=true

# Grape DHT port (each node needs different port)
GRAPE_DHT_PORT=20001

# Grape HTTP API port (each node needs different port)
GRAPE_API_PORT=30001

# Bootstrap DHT nodes (comma-separated host:port)
GRAPE_BOOTSTRAP_NODES=127.0.0.1:20001,127.0.0.1:20002

# Grape bind host (default: 127.0.0.1)
GRAPE_HOST=127.0.0.1

Legacy Grenache Configuration

# Only used when EMBEDDED_GRAPE=false
GRAPE_URL=http://127.0.0.1:30001

Port Configuration

Each node needs unique ports when running on the same machine:

Node GRAPE_DHT_PORT GRAPE_API_PORT P2P_PORT
1 20001 30001 3001
2 20002 30002 3002
3 20003 30003 3003
N 2000N 3000N 300N

🔍 How It Works

1. Node Startup Sequence

Node starts
    ↓
Start Embedded Grape DHT server
    ↓
Connect to bootstrap nodes (if any)
    ↓
Join Kademlia DHT network
    ↓
Wait for DHT to bootstrap (2s)
    ↓
Start Exchange Client
    ↓
Announce service in DHT
    ↓
Ready to trade!

2. Peer Discovery Flow

Node 3 starts
    ↓
Connects to bootstrap nodes (Node 1, Node 2)
    ↓
Learns about other DHT participants via Kademlia
    ↓
Builds routing table of nearby nodes
    ↓
Can now discover ANY service in the DHT
    ↓
No central server needed!

3. DHT Network Formation

The Kademlia DHT protocol ensures:

  • Decentralized: No central coordinator
  • Scalable: O(log n) routing complexity
  • Resilient: Nodes can join/leave freely
  • Self-healing: Automatically replaces failed nodes

📊 Comparison

Feature External Grape Embedded Grape
Infrastructure Manual servers Automatic
Setup Complexity High Low
External Dependency Yes ❌ No ✅
Single Point of Failure Yes ❌ No ✅
True P2P No ❌ Yes ✅
Production Ready No ❌ Yes ✅
DHT Protocol Kademlia Kademlia
Scalability Limited Unlimited

🎯 Why This is TRUE P2P

✅ Meets P2P Requirements:

  1. Decentralized Peer Discovery

    • Each node runs its own DHT server
    • Kademlia protocol (same as BitTorrent)
    • No central registry

  2. No External Infrastructure

    • Zero manual setup required
    • Self-contained nodes
    • Autonomous operation

  3. No Single Point of Failure

    • DHT is distributed across all nodes
    • Node failures don't affect network
    • Self-healing network

  4. Direct Peer Communication

    • Nodes connect directly after discovery
    • No proxy or intermediary
    • True peer-to-peer connections

  5. Independent Operation

    • Each node fully autonomous
    • No dependency on external services
    • Can operate in isolation

🧪 Testing

Test 1: Single Node (Bootstrap)

GRAPE_DHT_PORT=20001 GRAPE_API_PORT=30001 P2P_PORT=3001 npm start

Expected output:

🍇 Starting embedded Grape DHT server...
✅ Embedded Grape server started!
   DHT Port: 20001
   API Port: 30001
🍇 Grape Mode: Embedded (True P2P)

Test 2: Three Node Network

# Terminal 1
GRAPE_DHT_PORT=20001 GRAPE_API_PORT=30001 P2P_PORT=3001 npm start

# Terminal 2
GRAPE_DHT_PORT=20002 GRAPE_API_PORT=30002 GRAPE_BOOTSTRAP_NODES=127.0.0.1:20001 P2P_PORT=3002 npm start

# Terminal 3
GRAPE_DHT_PORT=20003 GRAPE_API_PORT=30003 GRAPE_BOOTSTRAP_NODES=127.0.0.1:20001 P2P_PORT=3003 npm start

Expected: All nodes discover each other automatically via DHT.

Test 3: Resilience Test

  1. Start 3 nodes as above
  2. Kill Node 1 (bootstrap)
  3. Start Node 4 using Node 2 as bootstrap
  4. Node 4 should still discover Node 3

Expected: Network continues functioning without Node 1.

🔧 Troubleshooting

Problem: Port conflicts

Error: EADDRINUSE: address already in use

Solution: Each node needs unique ports:

# Node 1
GRAPE_DHT_PORT=20001 GRAPE_API_PORT=30001 ...

# Node 2
GRAPE_DHT_PORT=20002 GRAPE_API_PORT=30002 ...

Problem: Nodes not discovering each other

Solution: Check bootstrap configuration:

# Node 2+ must specify bootstrap nodes
GRAPE_BOOTSTRAP_NODES=127.0.0.1:20001

Problem: "DHT not bootstrapped"

Solution: Wait longer or increase bootstrap time:

// In index.js, increase timeout
await new Promise(resolve => setTimeout(resolve, 5000)); // 5 seconds

🎓 Technical Details

Kademlia DHT

Grenache uses Kademlia, the same DHT protocol as:

  • BitTorrent
  • IPFS
  • Ethereum node discovery

Key Properties:

  • XOR distance metric: Efficient routing
  • k-buckets: Distributed routing tables
  • Parallel lookups: Fast peer discovery
  • Redundancy: Multiple paths to data

Embedded vs External

Aspect Embedded External
Deployment One process Two processes
Lifecycle Automatic Manual
Coupling Tight Loose
Overhead Minimal Higher
Reliability Higher Lower

📚 References

✅ Conclusion

The Embedded Grape architecture transforms this exchange from a pseudo-P2P system (requiring external infrastructure) into a TRUE P2P system where:

  • ✅ Each node is fully autonomous
  • ✅ No external infrastructure required
  • ✅ Distributed Kademlia DHT for peer discovery
  • ✅ No single point of failure
  • ✅ Production-ready and scalable

This is now a REAL P2P exchange! 🎉