LRU Object Store with Master-Slave Architecture (Vert.x)

Overview

This project is a scalable Java-based Object Store using Vert.x and a master-slave architecture to overcome JVM memory limits. Each process (slave) is restricted to a set memory (e.g., 2 GB). When a slave approaches this limit, a new slave process is spawned, effectively allowing the system to scale across available system RAM.

• Master: Central coordinator that handles all external object PUT and GET requests.
• Slaves: Independent Vert.x verticles running in separate JVM processes. Each manages its own LRU cache.
• Vert.x Event Bus: Handles communication between the master and slaves.
• LRU Cache: Least Recently Used caching strategy per slave (in-memory).

Architecture

+---------+ PUT/GET +---------+ | Client | ---------------> | Master | +---------+ +----+----+ | +-------------+-------------+ | | +------+-----+ +-------+------+ | Slave 1 | ... | Slave N | +------------+ +--------------+ (2GB memory max) (spawns more as needed)

---

Prerequisites

• Java 11+
• Maven
• Python 3 (for demo value generator in the script)

---

Build the Project

mvn clean package

Run the Master

java -cp target/object-store-vertx-1.0-SNAPSHOT.jar com.objstore.MainApp

This launches the master and automatically deploys the first slave. Run a Client (PUT or GET)

java -cp target/object-store-vertx-1.0-SNAPSHOT.jar
io.vertx.core.Launcher run com.objstore.cli.CacheClientVerticle
-Dcmd=put -Dkey=myKey -Dvalue="Some big JSON string"

For GET:

java -cp target/object-store-vertx-1.0-SNAPSHOT.jar
io.vertx.core.Launcher run com.objstore.cli.CacheClientVerticle
-Dcmd=get -Dkey=myKey

Run the Demo Script (auto-put multiple large objects)

chmod +x run-demo.sh ./run-demo.sh

This script:

Builds the project

Launches the master

Sends multiple large objects (~150MB each) to fill the memory of slaves

Triggers automatic slave spawning when memory usage exceeds 1.8GB

How It Works

The master maintains a mapping of keys to slave addresses.

Each slave holds an LRUCache with a fixed number of entries.

If a slave exceeds the set memory threshold, it spawns a new Java process (new slave).

Clients interact only with the master, which routes operations accordingly.

Notes

Each slave runs independently in its own JVM to bypass per-process memory limits.

You can tune memory thresholds and LRU size in SlaveVerticle.java.

Object keys and values are treated as Serializable.

TODOs (Future Enhancements)

Add persistent storage (e.g., spillover to disk or Redis)

Support TTL-based eviction

Health monitoring of slaves

Dockerize for container orchestration