Skip to content

Commit d503c2d

Browse files
taariqtaariq
committed
Add Show HN blog post announcing v5.0.0 release
1 parent dd70549 commit d503c2d

1 file changed

Lines changed: 188 additions & 0 deletions

File tree

‎docs/blog-post-show-hn.md‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# Show HN: Database-replicator – Zero-downtime database migrations to PostgreSQL
2+
3+
**TL;DR:** We built an open-source CLI tool that replicates databases from PostgreSQL, MySQL, MongoDB, and SQLite to PostgreSQL with zero downtime. It's written in Rust, supports continuous sync via logical replication, and can optionally run on managed cloud infrastructure. [GitHub](https://github.com/serenorg/database-replicator) | [Crates.io](https://crates.io/crates/database-replicator)
4+
5+
---
6+
7+
## The Problem We Were Solving
8+
9+
At SerenAI, we're building PostgreSQL databases optimized for AI workloads. Our users kept asking the same question: "How do I migrate my existing database to SerenDB without downtime?"
10+
11+
The existing options weren't great:
12+
13+
**pg_dump/pg_restore** works, but requires downtime. For a 100GB database, you're looking at hours of your application being offline. Not acceptable for production systems.
14+
15+
**AWS DMS and similar tools** are powerful but complex. They require provisioning infrastructure, configuring replication instances, and managing IAM roles. Overkill for most migrations.
16+
17+
**Logical replication setup** is the right approach, but it's tedious. You need to configure `wal_level`, create publications, set up subscriptions, handle the initial snapshot, monitor lag, and verify data integrity. Each step has gotchas.
18+
19+
We wanted something simpler: a single command that handles everything.
20+
21+
## What We Built
22+
23+
`database-replicator` is a Rust CLI that automates the entire migration process:
24+
25+
```bash
26+
# Validate both databases meet prerequisites
27+
database-replicator validate --source $SOURCE --target $TARGET
28+
29+
# Initial snapshot + set up continuous sync
30+
database-replicator init --source $SOURCE --target $TARGET --enable-sync
31+
32+
# Monitor replication lag
33+
database-replicator status --source $SOURCE --target $TARGET
34+
35+
# Verify data integrity with checksums
36+
database-replicator verify --source $SOURCE --target $TARGET
37+
```
38+
39+
### How It Works (PostgreSQL to PostgreSQL)
40+
41+
For PostgreSQL sources, we use native logical replication, which is the gold standard for zero-downtime migrations:
42+
43+
1. **Validate** - Check that the source has `wal_level = logical`, user has REPLICATION privilege, and target can create subscriptions.
44+
45+
2. **Initial Snapshot** - Use `pg_dump` in directory format with parallel workers (auto-detected based on CPU cores). We dump globals, schema, and data separately for proper dependency ordering.
46+
47+
3. **Create Publication** - Set up a publication on the source for all tables (or filtered tables if you're doing selective replication).
48+
49+
4. **Create Subscription** - The target subscribes to the source. PostgreSQL handles the rest—every INSERT, UPDATE, and DELETE is streamed in real-time.
50+
51+
5. **Monitor** - Track replication lag in bytes and time. When lag hits zero, you're ready to cut over.
52+
53+
6. **Verify** - Compute MD5 checksums across all tables on both sides. Any discrepancy is flagged immediately.
54+
55+
The entire flow handles edge cases: TCP keepalives for long-running operations behind load balancers, retry with exponential backoff for transient failures, and proper credential handling via `.pgpass` files (never exposed in process arguments).
56+
57+
### Multi-Source Support
58+
59+
Not everyone is migrating from PostgreSQL. We added support for:
60+
61+
- **MySQL/MariaDB → PostgreSQL**: Schema translation, type mapping, one-time snapshot with optional periodic refresh
62+
- **MongoDB → PostgreSQL**: Documents stored as JSONB, preserving the flexible schema while gaining SQL queryability
63+
- **SQLite → PostgreSQL**: Perfect for graduating from a local SQLite database to a production PostgreSQL instance
64+
65+
For non-PostgreSQL sources, we can't use logical replication (it's a PostgreSQL-specific feature), so these are snapshot-based migrations with optional scheduled refreshes.
66+
67+
## Selective Replication
68+
69+
Real migrations are rarely "replicate everything." You might want to:
70+
71+
- Exclude large analytics tables that you'll backfill separately
72+
- Filter old data (only replicate orders from the last 90 days)
73+
- Skip tables with PII for a staging environment
74+
75+
We support this through CLI flags or a TOML config file:
76+
77+
```toml
78+
[databases.mydb]
79+
schema_only = ["large_analytics_table"]
80+
81+
[[databases.mydb.table_filters]]
82+
table = "orders"
83+
where = "created_at > NOW() - INTERVAL '90 days'"
84+
85+
[[databases.mydb.time_filters]]
86+
table = "events"
87+
column = "timestamp"
88+
last = "6 months"
89+
```
90+
91+
The filter predicates are pushed down to `pg_dump` and used in publication WHERE clauses (PostgreSQL 15+), so you're not transferring data you don't need.
92+
93+
## The Cloud Execution Feature
94+
95+
Here's where it gets interesting. Running a migration locally has drawbacks:
96+
97+
- Your laptop needs to stay connected for hours (or days for large databases)
98+
- Network bandwidth is limited by your ISP upload speed
99+
- If your connection drops, you need to restart or resume from checkpoints
100+
101+
For users migrating to SerenDB, we built a remote execution feature. Instead of running locally, the replication job runs on our AWS infrastructure:
102+
103+
```bash
104+
export SEREN_API_KEY="your-api-key" # from console.serendb.com
105+
database-replicator init \
106+
--source "postgresql://user:pass@your-rds.amazonaws.com:5432/db" \
107+
--target "postgresql://user:pass@your-db.serendb.com:5432/db"
108+
```
109+
110+
When the target is a SerenDB database, we automatically provision an EC2 worker, run the migration there, stream progress back to your terminal, and clean up when done. Your laptop can disconnect—the job continues.
111+
112+
The architecture:
113+
- API Gateway + Lambda for job coordination
114+
- SQS for reliable job queuing
115+
- EC2 workers with the replicator binary pre-installed
116+
- KMS encryption for credentials (never stored in plaintext)
117+
- DynamoDB for job state tracking
118+
119+
For non-SerenDB targets, add `--local` and it runs on your machine like any other CLI tool.
120+
121+
## Technical Decisions
122+
123+
**Why Rust?** Performance and reliability. Database migrations are long-running operations where memory safety matters. We also get easy cross-compilation for Linux, macOS Intel, and macOS ARM.
124+
125+
**Why shell out to pg_dump?** We considered implementing dump logic directly, but `pg_dump` is battle-tested across millions of databases. It handles edge cases we'd never think of. We wrap it with proper error handling, progress tracking, and retry logic.
126+
127+
**Why logical replication over CDC tools?** PostgreSQL's built-in logical replication is robust and doesn't require additional infrastructure. It's what PostgreSQL itself uses for read replicas. For PostgreSQL-to-PostgreSQL migrations, there's no reason to add complexity.
128+
129+
**Checkpoint system** - Long migrations need resume support. We write checkpoints to `.seren-replicator/` after each database completes. If interrupted, the next run picks up where it left off. Checkpoints include a fingerprint of the filter configuration—if you change filters, it starts fresh to prevent data inconsistency.
130+
131+
## Limitations (Being Honest)
132+
133+
- **PostgreSQL-only targets**: We replicate *to* PostgreSQL, not from it to other databases
134+
- **Logical replication requires PostgreSQL 10+** on the source (we recommend 12+)
135+
- **Large objects (BLOBs)** aren't replicated via logical replication—use `pg_dump` for those
136+
- **DDL changes** during replication need manual handling (logical replication doesn't capture schema changes)
137+
- **Sequences** need to be manually advanced after cutover
138+
139+
## Getting Started
140+
141+
Install from crates.io:
142+
```bash
143+
cargo install database-replicator
144+
```
145+
146+
Or download binaries from the [GitHub releases](https://github.com/serenorg/database-replicator/releases).
147+
148+
Basic migration:
149+
```bash
150+
# Check prerequisites
151+
database-replicator validate \
152+
--source "postgresql://user:pass@source:5432/mydb" \
153+
--target "postgresql://user:pass@target:5432/mydb"
154+
155+
# Run migration with continuous sync
156+
database-replicator init \
157+
--source "postgresql://user:pass@source:5432/mydb" \
158+
--target "postgresql://user:pass@target:5432/mydb" \
159+
--enable-sync
160+
161+
# Monitor until lag is zero
162+
database-replicator status \
163+
--source "postgresql://user:pass@source:5432/mydb" \
164+
--target "postgresql://user:pass@target:5432/mydb"
165+
166+
# Verify checksums match
167+
database-replicator verify \
168+
--source "postgresql://user:pass@source:5432/mydb" \
169+
--target "postgresql://user:pass@target:5432/mydb"
170+
171+
# Cut over your application to the new database
172+
# Then drop the subscription on target
173+
```
174+
175+
## What's Next
176+
177+
We're working on:
178+
- **CDC for non-PostgreSQL sources** - Real-time sync from MySQL using binlog
179+
- **Progress percentage** - Better estimation of completion time
180+
- **Web dashboard** - Visual monitoring for remote jobs
181+
182+
The tool is Apache 2.0 licensed. Contributions welcome—especially around MySQL CDC and additional source database support.
183+
184+
---
185+
186+
[GitHub](https://github.com/serenorg/database-replicator) | [Documentation](https://github.com/serenorg/database-replicator#readme) | [SerenAI Console](https://console.serendb.com)
187+
188+
Happy to answer questions about the architecture, PostgreSQL logical replication gotchas, or anything else.

0 commit comments

Comments
 (0)