Skip to content

Commit 2012b2f

Browse files
committed
docs(readme): refactor with comparative benchmark + logo
Add Dockermint logo, new badge row (License, Website, Docs, Go MSRV, Release, CI, Release workflow, Container), 3-way comparative benchmark table (LevelDB native / Level2Pebble / Pebblify), restructured install section (Docker/Podman recommended, Binary, Build from source), beta warning on daemon mode, improved intro and CTA. Removed Artifact attestation section and early-stage warning. Closes #55
1 parent b1de7e5 commit 2012b2f

2 files changed

Lines changed: 283 additions & 111 deletions

File tree

README.md

Lines changed: 129 additions & 111 deletions
Original file line numberDiff line numberDiff line change
@@ -1,92 +1,142 @@
1+
<p align="center">
2+
<img src="./assets/logo.svg" alt="Pebblify" width="200" />
3+
</p>
4+
15
# Pebblify
26

37
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
4-
[![Go](https://img.shields.io/badge/Go-1.25+-00ADD8?logo=go&logoColor=white)](https://go.dev/)
5-
![Benchmark](https://img.shields.io/badge/benchmark-217M_keys_in_4m11s-brightgreen)
6-
![Throughput](https://img.shields.io/badge/throughput-865k_keys%2Fs-blue)
7-
8-
**Pebblify** is a high-performance migration tool that converts LevelDB databases to PebbleDB format, specifically designed for Cosmos SDK and CometBFT (formerly Tendermint) blockchain nodes.
9-
10-
PebbleDB offers significant performance improvements over LevelDB, including better write throughput, more efficient compaction, and reduced storage overhead. Pebblify makes it easy to migrate your existing node data without manual intervention.
8+
[![Website](https://img.shields.io/badge/website-dockermint.io-informational)](https://dockermint.io/tools)
9+
[![Documentation](https://img.shields.io/badge/docs-docs.dockermint.io-blue)](https://docs.dockermint.io/pebblify/)
10+
[![Go MSRV](https://img.shields.io/badge/Go-1.25+-00ADD8?logo=go&logoColor=white)](https://go.dev/doc/install)
11+
[![Latest Release](https://img.shields.io/github/v/release/Dockermint/pebblify?label=release)](https://github.com/Dockermint/pebblify/releases)
12+
[![CI](https://github.com/Dockermint/pebblify/actions/workflows/ci.yml/badge.svg)](https://github.com/Dockermint/pebblify/actions/workflows/ci.yml)
13+
[![Release](https://github.com/Dockermint/pebblify/actions/workflows/release.yml/badge.svg)](https://github.com/Dockermint/pebblify/actions/workflows/release.yml)
14+
[![Container](https://img.shields.io/badge/container-ghcr.io-blue?logo=github)](https://github.com/orgs/Dockermint/packages/container/package/pebblify)
1115

12-
📖 [Documentation](https://docs.dockermint.io/pebblify/) · 🌐 [Website](https://dockermint.io/tools)
16+
**Pebblify** is a high-performance migration tool that converts LevelDB databases to PebbleDB format, specifically designed for Cosmos SDK and CometBFT blockchain nodes.
1317

14-
> [!WARNING]
15-
> This tool is still in the early stages of development and may contain bugs or be unstable. If you notice any unusual behavior, please open an issue.
18+
PebbleDB offers significant performance improvements over LevelDB, including better write throughput, more efficient compaction, and reduced storage overhead. Pebblify outperforms existing tools: in a real-world production benchmark it completed a full snapshot conversion in **6m48** versus 7m13 for Level2Pebble, while producing a **20.34 GiB** output compared to 26 GiB, with no data loss. It adds crash recovery, disk space checks, verification, Prometheus metrics, health probes, notifications, and daemon mode that Level2Pebble does not provide.
1619

17-
## Features
20+
**[Install now](#installation)** or read the [full documentation](https://docs.dockermint.io/pebblify/).
1821

19-
- **Fast parallel conversion** — Process multiple databases concurrently with configurable worker count
20-
- **Crash recovery** — Resume interrupted migrations from the last checkpoint
21-
- **Adaptive batching** — Automatically adjusts batch sizes based on memory constraints
22-
- **Real-time progress** — Live progress bar with throughput metrics and ETA
23-
- **Data verification** — Verify converted data integrity with configurable sampling
24-
- **Disk space checks** — Pre-flight validation to ensure sufficient storage
25-
- **Docker support** — Multi-architecture container images (amd64/arm64)
26-
- **Health probes** — HTTP liveness, readiness, and startup endpoints for orchestrators
27-
- **Prometheus metrics** — Opt-in metrics exporter for conversion monitoring
28-
- **Shell completion** — Bash and zsh autocompletion via `pebblify completion`
22+
## Quick Start
2923

30-
## Requirements
24+
### Convert a LevelDB database
3125

32-
- **Go 1.25+** (for building from source)
33-
- **Sufficient disk space** — Approximately 1.5x the source data size during conversion
34-
- **Source database** — Valid LevelDB directory structure (Cosmos/CometBFT `data/` format)
26+
```bash
27+
pebblify level-to-pebble ~/.gaia/data ./gaia-pebble
28+
```
3529

36-
## Installation
30+
### Recover an interrupted conversion
3731

38-
### CLI (all platforms)
32+
```bash
33+
pebblify recover --tmp-dir /var/tmp
34+
```
3935

40-
Cross-platform. Works on `linux/amd64`, `linux/arm64`, `darwin/amd64`, and `darwin/arm64`.
36+
### Verify converted data
4137

4238
```bash
43-
git clone https://github.com/Dockermint/pebblify.git
44-
cd pebblify
45-
make install-cli # build and install pebblify binary to PATH
39+
pebblify verify --sample 10 ~/.gaia/data ./gaia-pebble/data
4640
```
4741

48-
Or download a pre-built binary from [releases](https://github.com/Dockermint/pebblify/releases).
42+
For all available flags and subcommands, see the [CLI reference](https://docs.dockermint.io/pebblify/).
4943

50-
### Systemd daemon — **Linux only**
44+
## Benchmark
5145

52-
Installs the daemon as a system service. Requires root. Not supported on macOS.
46+
Real-world comparison on a production Cosmos node snapshot:
47+
48+
| Metric | LevelDB (native) | [Level2Pebble](https://github.com/notional-labs/level2pebble) | Pebblify |
49+
|---|---|---|---|
50+
| Size | 25 GiB | 26 GiB | **20.34 GiB** |
51+
| Duration | -- | 7m13 | **6m48** |
52+
| DB performance | average | good | good |
53+
| Full snapshot conversion | -- |||
54+
| Disk space checks | -- |||
55+
| Prometheus metrics | -- |||
56+
| Probes | -- |||
57+
| Resume on error | -- |||
58+
| Verification | -- |||
59+
| Daemon mode | -- |||
60+
| Notifications* | -- |||
61+
| Push* | -- |||
62+
63+
\* Daemon mode only.
5364

54-
```bash
55-
sudo make install-systemd-daemon
56-
```
65+
## Features
5766

58-
After installation, fill in `/etc/pebblify/.env`, then:
67+
- **Fast parallel conversion** -- Process multiple databases concurrently with configurable worker count
68+
- **Crash recovery** -- Resume interrupted migrations from the last checkpoint
69+
- **Adaptive batching** -- Automatically adjusts batch sizes based on memory constraints
70+
- **Real-time progress** -- Live progress bar with throughput metrics and ETA
71+
- **Data verification** -- Verify converted data integrity with configurable sampling
72+
- **Disk space checks** -- Pre-flight validation to ensure sufficient storage
73+
- **Docker support** -- Multi-architecture container images (amd64/arm64)
74+
- **Health probes** -- HTTP liveness, readiness, and startup endpoints for orchestrators
75+
- **Prometheus metrics** -- Opt-in metrics exporter for conversion monitoring
76+
- **Shell completion** -- Bash and zsh autocompletion via `pebblify completion`
5977

60-
```bash
61-
sudo systemctl daemon-reload
62-
sudo systemctl enable --now pebblify
63-
```
78+
## Installation
6479

65-
See [daemon quickstart](docs/markdown/daemon-quickstart.md) for the full setup guide.
80+
### Docker / Podman (recommended)
6681

67-
### Podman Quadlet (rootless)
82+
> [!TIP]
83+
> Podman is rootless, giving it a security advantage over Docker.
6884
69-
Deploys the daemon as a rootless Podman container managed by the systemd user session. Linux-native; macOS users need Podman Desktop.
85+
**Prerequisites:** [Docker](https://docs.docker.com/get-docker/) or [Podman](https://podman.io/docs/installation).
7086

7187
```bash
72-
make install-podman
73-
```
88+
# Pull the image
89+
docker pull ghcr.io/dockermint/pebblify:0.4.0
90+
podman pull ghcr.io/dockermint/pebblify:0.4.0
7491

75-
After installation:
92+
# Run a conversion
93+
docker run --rm \
94+
-v /path/to/source:/data/source:ro \
95+
-v /path/to/output:/data/output \
96+
-v /path/to/tmp:/tmp \
97+
ghcr.io/dockermint/pebblify:0.4.0 \
98+
level-to-pebble --health --metrics /data/source /data/output
7699

77-
```bash
78-
systemctl --user daemon-reload
79-
systemctl --user start pebblify
100+
podman run --rm \
101+
-v /path/to/source:/data/source:ro \
102+
-v /path/to/output:/data/output \
103+
-v /path/to/tmp:/tmp \
104+
ghcr.io/dockermint/pebblify:0.4.0 \
105+
level-to-pebble --health --metrics /data/source /data/output
80106
```
81107

82-
### Docker
108+
### Binary
109+
110+
Download a pre-built binary for your platform from the [v0.4.0 release page](https://github.com/Dockermint/pebblify/releases/tag/v0.4.0).
111+
112+
Available architectures: Linux/AMD64, Linux/ARM64, Darwin/ARM64, Darwin/AMD64.
113+
114+
After downloading, verify the checksum against the `checksums.txt` file on the release page, then place the binary on your `PATH`.
115+
116+
### Build from source
117+
118+
> [!NOTE]
119+
> **Prerequisites:** [Go 1.25+](https://go.dev/doc/install), `make`, `git`.
120+
>
121+
> Per-platform Makefile targets:
122+
> - `install-cli` -- build and install the CLI binary to your `PATH` (all platforms)
123+
> - `build-docker` -- build a Docker image for the local platform
124+
> - `install-systemd-daemon` -- install the daemon as a systemd service (Linux only, requires root)
125+
> - `install-podman` -- install the daemon as a rootless Podman Quadlet (Linux only)
83126
84127
```bash
85-
make build-docker
128+
git clone --branch v0.4.0 https://github.com/Dockermint/pebblify.git
129+
cd pebblify
130+
make install-cli
86131
```
87132

133+
For daemon and container installation options, see the [installation guide](https://docs.dockermint.io/pebblify/).
134+
88135
## Daemon mode
89136

137+
> [!WARNING]
138+
> The daemon mode is currently in **beta**. If you encounter a bug, please [open an issue](https://github.com/Dockermint/pebblify/issues/new/choose).
139+
90140
`pebblify daemon` is a long-running HTTP service that accepts snapshot archive URLs, converts them from LevelDB to PebbleDB format, repacks the output, and saves it to one or more storage targets (local directory, SCP, S3). Jobs are submitted via a REST API and processed serially.
91141

92142
**Platform:** The `daemon` subcommand is Linux-only at runtime. On macOS, use Docker Compose or Podman.
@@ -109,92 +159,60 @@ curl -s -X POST http://127.0.0.1:2324/v1/jobs \
109159
110160
Full guide: [docs/markdown/daemon-quickstart.md](docs/markdown/daemon-quickstart.md)
111161

112-
## Quick Start
162+
## Systemd daemon (Linux only)
113163

114-
### Convert
164+
Installs the daemon as a system service. Requires root. Not supported on macOS.
115165

116166
```bash
117-
pebblify level-to-pebble ~/.gaia/data ./gaia-pebble
167+
sudo make install-systemd-daemon
118168
```
119169

120-
### Recover an Interrupted Conversion
170+
After installation, fill in `/etc/pebblify/.env`, then:
121171

122172
```bash
123-
pebblify recover --tmp-dir /var/tmp
173+
sudo systemctl daemon-reload
174+
sudo systemctl enable --now pebblify
124175
```
125176

126-
### Verify
177+
See [daemon quickstart](docs/markdown/daemon-quickstart.md) for the full setup guide.
127178

128-
```bash
129-
pebblify verify --sample 10 ~/.gaia/data ./gaia-pebble/data
130-
```
179+
## Podman Quadlet (rootless)
131180

132-
### Docker
181+
Deploys the daemon as a rootless Podman container managed by the systemd user session. Linux-native; macOS users need Podman Desktop.
133182

134183
```bash
135-
docker pull ghcr.io/dockermint/pebblify:latest
136-
137-
docker run --rm \
138-
-v /path/to/source:/data/source:ro \
139-
-v /path/to/output:/data/output \
140-
-v /path/to/tmp:/tmp \
141-
ghcr.io/dockermint/pebblify:latest \
142-
level-to-pebble --health --metrics /data/source /data/output
184+
make install-podman
143185
```
144186

145-
> For full command reference and all available flags, see the [documentation](https://docs.dockermint.io/pebblify/).
146-
147-
## Artifact attestation
148-
149-
Starting with v0.4.0, every release binary and Docker image is published with SLSA provenance and SBOM attestations. Verify before running:
187+
After installation:
150188

151189
```bash
152-
# Verify a release binary
153-
gh attestation verify pebblify-linux-amd64 \
154-
--repo Dockermint/Pebblify
155-
156-
# Verify the Docker image
157-
gh attestation verify \
158-
oci://ghcr.io/dockermint/pebblify:v0.4.0 \
159-
--repo Dockermint/Pebblify
190+
systemctl --user daemon-reload
191+
systemctl --user start pebblify
160192
```
161193

162-
Available attestation targets: `pebblify-linux-amd64`, `pebblify-linux-arm64`, `pebblify-darwin-amd64`, `pebblify-darwin-arm64`, and the multi-arch Docker image.
163-
164-
Full guide: [docs/markdown/release-automation.md](docs/markdown/release-automation.md)
165-
166-
## Benchmark
167-
168-
Real-world conversion on a production Cosmos node dataset:
169-
170-
| Metric | Value |
171-
|---|---|
172-
| Total keys | 217,895,735 |
173-
| Duration | 4m 11s |
174-
| Throughput | ~865k keys/s · ~160 MB/s |
175-
| Data processed | 39.54 GiB read / 39.54 GiB written |
176-
| Size reduction | -16.3% (LevelDB 23.32 GiB → PebbleDB 19.51 GiB) |
177-
| Data loss | None — 1:1 write/read parity |
194+
## Performance Tips
178195

179-
> [!NOTE]
180-
> Benchmark performed on AMD Ryzen 9 8940HX, 32 GiB DDR5, NVMe (Btrfs). Temp folder on disk, not in RAM.
196+
- **Use SSDs** -- NVMe storage significantly improves conversion speed
197+
- **Increase workers** -- For systems with many CPU cores, increase `-w` for faster parallel processing
198+
- **Adjust batch memory** -- Increase `--batch-memory` if you have RAM to spare
199+
- **Use local temp** -- If `/tmp` is a tmpfs (RAM-based), use `--tmp-dir` to point to disk storage for large datasets
181200

182-
## Performance Tips
201+
## Requirements
183202

184-
- **Use SSDs** — NVMe storage significantly improves conversion speed
185-
- **Increase workers** — For systems with many CPU cores, increase `-w` for faster parallel processing
186-
- **Adjust batch memory** — Increase `--batch-memory` if you have RAM to spare
187-
- **Use local temp** — If `/tmp` is a tmpfs (RAM-based), use `--tmp-dir` to point to disk storage for large datasets
203+
- **Go 1.25+** (for building from source)
204+
- **Sufficient disk space** -- Approximately 1.5x the source data size during conversion
205+
- **Source database** -- Valid LevelDB directory structure (Cosmos/CometBFT `data/` format)
188206

189207
## Contributing
190208

191-
Contributions are welcome! Please feel free to submit issues and pull requests.
209+
Contributions are welcome. Please submit issues and pull requests via [GitHub](https://github.com/Dockermint/pebblify).
192210

193211
## License
194212

195-
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENCE) file for details.
213+
This project is licensed under the Apache License 2.0. See the [LICENSE](LICENCE) file for details.
196214

197215
## Acknowledgments
198216

199-
- [CockroachDB Pebble](https://github.com/cockroachdb/pebble) The high-performance storage engine
200-
- [syndtr/goleveldb](https://github.com/syndtr/goleveldb) LevelDB implementation in Go
217+
- [CockroachDB Pebble](https://github.com/cockroachdb/pebble) -- The high-performance storage engine
218+
- [syndtr/goleveldb](https://github.com/syndtr/goleveldb) -- LevelDB implementation in Go

0 commit comments

Comments
 (0)