Skip to content

Commit 26c9507

Browse files
yourfavDevyourfavDev
authored
Update README.md
1 parent 2232d96 commit 26c9507

1 file changed

Lines changed: 154 additions & 110 deletions

File tree

‎README.md‎

Lines changed: 154 additions & 110 deletions
Original file line numberDiff line numberDiff line change
@@ -1,147 +1,191 @@
1-
# DynaRust: A Distributed Key-Value Store with Dynamic Membership
1+
```markdown
2+
# DynaRust: A Simple Distributed Key-Value Store
23

3-
DynaRust is a distributed key-value store built in Rust that provides reliable data storage with dynamic cluster membership management. The system offers seamless node joining, automatic state synchronization, and persistent storage capabilities.
4+
DynaRust is a distributed key-value store built in Rust. It's designed to be reliable and easy to manage, allowing you to add or remove nodes (servers) dynamically without interrupting service.
45

5-
The system maintains data consistency across nodes through periodic synchronization while providing high availability through its distributed architecture. It features an in-memory store with disk persistence, cluster membership management, and a RESTful API interface for data operations. The implementation leverages Actix Web for HTTP services and supports dynamic cluster topology changes with automatic state merging.
6+
Think of it as a shared dictionary spread across multiple computers. You can store data (key-value pairs), retrieve it, and delete it using a simple web API. DynaRust automatically copies your data across the available nodes for high availability and synchronizes changes, ensuring data consistency over time (eventual consistency). It stores data in memory for speed and saves it to disk (`storage.db`) so your data isn't lost if a node restarts.
67

7-
## Repository Structure
8-
```
9-
.
10-
├── src/ # Source code directory
11-
│ ├── main.rs # Application entry point with HTTP server and cluster initialization
12-
│ ├── logger.rs # Logging utilities with colored output
13-
│ ├── network/ # Network communication components
14-
│ │ ├── broadcaster.rs # Cluster membership synchronization
15-
│ │ └── mod.rs # Network module definition
16-
│ └── storage/ # Storage implementation
17-
│ ├── engine.rs # Core storage engine
18-
│ ├── mod.rs # Storage module definition
19-
│ └── persistance.rs # Disk persistence implementation
20-
├── docs/ # Documentation assets
21-
│ ├── infra.dot # Infrastructure diagram source
22-
│ └── infra.svg # Infrastructure visualization
23-
├── Dockerfile # Multi-stage container build definition
24-
├── Cargo.toml # Rust package manifest
25-
└── Cargo.lock # Dependency lock file
26-
```
8+
**Key Features:**
9+
10+
* **Distributed Storage:** Data is spread across all nodes.
11+
* **High Availability:** If one node fails, others can still serve requests.
12+
* **Dynamic Cluster Membership:** Nodes can join or leave the cluster easily.
13+
* **Automatic State Sync:** New or returning nodes automatically get the latest data.
14+
* **Persistent Storage:** Data is saved to disk for durability.
15+
* **RESTful API:** Simple HTTP interface for interacting with your data.
16+
17+
## Getting Started
18+
19+
Follow these steps to get DynaRust running.
2720

28-
## Usage Instructions
2921
### Prerequisites
30-
- Rust 1.86.0 or later
31-
- OpenSSL development package
32-
- pkg-config
33-
- Build essentials (for compilation)
22+
23+
You'll need the following installed:
24+
25+
* Rust (version 1.86.0 or newer)
26+
* Standard build tools (like `gcc`, `make` - often called `build-essential` on Debian/Ubuntu)
27+
* OpenSSL development libraries (e.g., `libssl-dev` on Debian/Ubuntu, `openssl-devel` on Fedora/CentOS)
28+
* `pkg-config`
3429

3530
### Installation
3631

37-
1. Clone the repository:
38-
```bash
39-
git clone <repository-url>
40-
cd dynarust
41-
```
32+
**Option 1: Build from Source**
4233

43-
2. Build the project:
44-
```bash
45-
cargo build --release
46-
```
34+
1. Clone the repository:
35+
```bash
36+
git clone https://github.com/yourfavDev/DynaRust
37+
cd dynarust
38+
```
39+
2. Build the project (this creates an executable in `target/release/`):
40+
```bash
41+
cargo build --release
42+
```
4743

48-
3. Using Docker:
49-
```bash
50-
docker build -t dynarust .
51-
```
44+
**Option 2: Using Docker**
45+
46+
If you prefer Docker:
47+
48+
1. Build the Docker image:
49+
```bash
50+
docker build -t dynarust .
51+
```
52+
*(See the [Deployment with Docker](#deployment-with-docker) section for running instructions).*
53+
54+
### Running DynaRust
5255

53-
### Quick Start
56+
The DynaRust executable takes one or two arguments:
5457

55-
1. Start a single node:
58+
1. `LISTEN_ADDRESS`: The IP address and port this node should listen on (e.g., `127.0.0.1:6660`).
59+
2. `JOIN_ADDRESS` (Optional): The address of an existing node in the cluster to join (e.g., `127.0.0.1:6660`).
60+
61+
**Example 1: Start the first node**
62+
63+
This node listens on `127.0.0.1` port `6660`.
5664
```bash
5765
./target/release/DynaRust 127.0.0.1:6660
5866
```
5967

60-
2. Start additional nodes and join the cluster:
68+
**Example 2: Start a second node and join the first one**
69+
70+
This node listens on `127.0.0.1` port `6661` and connects to the first node running at `127.0.0.1:6660` to join the cluster.
6171
```bash
6272
./target/release/DynaRust 127.0.0.1:6661 127.0.0.1:6660
6373
```
74+
*You can add more nodes similarly.*
6475

65-
### More Detailed Examples
76+
## Using the API
6677

67-
1. Store a value:
68-
```bash
69-
curl -X PUT http://localhost:6660/key/mykey -d '{"value": "mydata"}'
70-
```
78+
Interact with DynaRust using simple HTTP requests (e.g., with `curl`). Replace `localhost:6660` with the address of any node in your cluster.
7179

72-
2. Retrieve a value:
73-
```bash
74-
curl http://localhost:6660/key/mykey
75-
```
80+
**Note:** Data is sent and received as JSON. For PUT requests, the body should be `{"value": "your-data"}`.
7681

77-
3. Delete a value:
78-
```bash
79-
curl -X DELETE http://localhost:6660/key/mykey
80-
```
82+
1. **Store a value:** (HTTP PUT)
83+
```bash
84+
curl -X PUT http://localhost:6660/key/mykey -H "Content-Type: application/json" -d '{"value": "mydata"}'
85+
# Expected Response: HTTP 200 OK
86+
```
8187

82-
4. View cluster membership:
83-
```bash
84-
curl http://localhost:6660/membership
85-
```
88+
2. **Retrieve a value:** (HTTP GET)
89+
```bash
90+
curl http://localhost:6660/key/mykey
91+
# Expected Response: {"value": "mydata"}
92+
```
93+
94+
3. **Delete a value:** (HTTP DELETE)
95+
```bash
96+
curl -X DELETE http://localhost:6660/key/mykey
97+
# Expected Response: HTTP 200 OK
98+
```
8699

87-
### Troubleshooting
100+
4. **View cluster members:** (HTTP GET)
101+
```bash
102+
curl http://localhost:6660/membership
103+
# Expected Response: A JSON list of node addresses in the cluster, e.g.,
104+
# ["127.0.0.1:6660", "127.0.0.1:6661"]
105+
```
88106

89-
1. Node Join Issues
90-
- Problem: Node fails to join cluster
91-
- Solution:
92-
```bash
93-
# Verify the join node is running
94-
curl http://join-node-address/membership
95-
# Check network connectivity
96-
ping join-node-address
97-
```
107+
## How it Works (Conceptual Overview)
98108

99-
2. Data Synchronization Issues
100-
- Enable debug logging by setting RUST_LOG:
101-
```bash
102-
RUST_LOG=debug ./target/release/DynaRust 127.0.0.1:6660
103-
```
104-
- Check storage.db file permissions if persistence fails
109+
DynaRust aims for eventual consistency across the cluster. Here's the basic flow:
105110
106-
## Data Flow
107-
DynaRust implements a distributed key-value store with eventual consistency. Data flows from client requests through the HTTP API, is processed by the storage engine, and is eventually synchronized across all cluster nodes.
111+
1. **Client Request:** A client sends an HTTP request (GET, PUT, DELETE) to any node's API endpoint.
112+
2. **Local Processing:** The receiving node processes the request, updating its local in-memory store.
113+
3. **Persistence:** Changes are periodically saved to a local file (`storage.db`) for durability.
114+
4. **Cluster Synchronization:**
115+
* Nodes periodically gossip (share) their list of known members with each other.
116+
* When data changes, the update eventually propagates to other nodes during synchronization cycles.
117+
* When a new node joins, it contacts an existing node, gets the current cluster membership, and fetches the existing data state to merge with its own.
108118

109119
```ascii
110-
Client -> HTTP API -> Storage Engine -> Disk Persistence
111-
^ |
112-
| v
113-
+---- Cluster Sync ----+
120+
+--------+ +-------------------+ +-----------------+
121+
| Client | ----> | Node API Endpoint | ----> | In-Memory Store | ----+
122+
+--------+ +-------------------+ +-----------------+ |
123+
^ | |
124+
| v v
125+
| +--------------+ +-----------------+
126+
+-------- Cluster Synchronization ----- | Other Nodes | | Disk Persistence|
127+
+--------------+ +-----------------+
114128
```
115129

116-
Key component interactions:
117-
1. HTTP API receives client requests for data operations
118-
2. Storage engine processes operations on local state
119-
3. Periodic sync task broadcasts membership updates
120-
4. Background task persists state to disk
121-
5. Joining nodes merge remote state with local state
130+
## Deployment with Docker
122131

123-
## Infrastructure
124-
- server (Docker::Container): Main application container running the DynaRust service
125-
- Exposes port 6660 for API access
126-
- Includes OpenSSL runtime dependencies
127-
- Built using multi-stage build for minimal image size
132+
Using Docker simplifies deployment and dependency management.
128133

129-
## Deployment
130134
### Prerequisites
131-
- Docker 20.10 or later
132-
- Network access for container registry
133135

134-
### Deployment Steps
135-
1. Build container:
136-
```bash
137-
docker build -t dynarust:latest .
136+
* Docker (version 20.10 or newer)
137+
* Network connectivity between Docker containers if running a cluster.
138+
139+
### Steps
140+
141+
1. **Build the Image:** (If you haven't already)
142+
```bash
143+
docker build -t dynarust:latest .
144+
```
145+
146+
2. **Run the Container(s):**
147+
148+
* **First Node:**
149+
```bash
150+
# Runs the first node, mapping container port 6660 to host port 6660
151+
docker run -d --name dynarust-node1 -p 6660:6660 dynarust:latest 0.0.0.0:6660
152+
```
153+
* `-d`: Run in detached mode (background).
154+
* `--name`: Give the container a recognizable name.
155+
* `-p 6660:6660`: Map port 6660 on your host machine to port 6660 inside the container.
156+
* `0.0.0.0:6660`: Tells DynaRust inside the container to listen on all network interfaces on port 6660.
157+
158+
* **Joining Nodes:**
159+
You need to tell the new node the address of the first node *as reachable from within the Docker network*. Replace `<first-node-ip-or-hostname>` with the appropriate address. If using Docker's default bridge network, you might use the container name (`dynarust-node1`) or its internal IP.
160+
161+
```bash
162+
# Runs a second node, mapping container port 6660 to host port 6661
163+
# It joins the cluster via the first node (dynarust-node1:6660)
164+
docker run -d --name dynarust-node2 -p 6661:6660 \
165+
dynarust:latest 0.0.0.0:6660 dynarust-node1:6660
166+
```
167+
* `-p 6661:6660`: Maps host port 6661 to the container's port 6660.
168+
* `0.0.0.0:6660`: The address the *new* node listens on inside its container.
169+
* `dynarust-node1:6660`: The address of the *first* node (using its container name) that this new node should contact to join the cluster. Docker's internal DNS usually resolves container names.
170+
171+
**Important:** Ensure your Docker network setup allows containers to reach each other by name or IP address on the specified ports (e.g., 6660 in this case).
172+
173+
## Troubleshooting
174+
175+
1. **Node Fails to Join Cluster:**
176+
* **Verify the target node is running:** `curl http://<join-node-address>/membership` (e.g., `curl http://127.0.0.1:6660/membership`). Does it respond?
177+
* **Check Network:** Can the joining node reach the `JOIN_ADDRESS`? Use `ping <join-node-address>` or check firewall rules. If using Docker, ensure containers are on the same network and can communicate.
178+
* **Check Logs:** Run the node with debug logging enabled (see below).
179+
180+
2. **Data Not Synchronizing / Other Issues:**
181+
* **Enable Debug Logs:** Set the `RUST_LOG` environment variable before running:
182+
```bash
183+
# For native execution
184+
RUST_LOG=debug ./target/release/DynaRust <LISTEN_ADDRESS> [JOIN_ADDRESS]
185+
186+
# For Docker (add -e RUST_LOG=debug to docker run)
187+
docker run -e RUST_LOG=debug [...] dynarust:latest [...]
188+
```
189+
Check the console output for errors or detailed synchronization messages.
190+
* **Check Disk Persistence:** If data isn't saved after restarts, ensure the process has write permissions for the `storage.db` file in its working directory. For Docker, consider using volumes to persist data outside the container.
138191
```
139-
140-
2. Run container:
141-
```bash
142-
# First node
143-
docker run -p 6660:6660 dynarust:latest
144-
145-
# Additional nodes
146-
docker run -p 6661:6660 dynarust:latest 0.0.0.0:6660 first-node:6660
147-
```

0 commit comments

Comments
 (0)