sequin-postgres-cdc

Postgres CDC to Kafka using Sequin

NOTE: docker setup from here

graph LR
    A[Postgres] --> B[Sequin]
    B --> C[Kafka]

Loading

Usage

Prerequisites

• mise
• pre-commit
• just

Setup

just setup

Run

just up # run docker compose up
# go to sequin ui & validate sink config
just run # insert data
just down # run docker compose down

• Sequin UI: http://localhost:7376
• Grafana: http://localhost:3000
• Kafka UI: http://localhost:8088

Export Sequin Config

# install cli
curl -sf https://raw.githubusercontent.com/sequinstream/sequin/main/cli/installer.sh | sh
# export config

Notes

• when you set a BATCH_SIZE - ordering of events in a batch is not guaranteed
• cannot run WITHOUT redis

Todo

• multiple tables with high volume using 1 replication slot. how does partitioning help w load
• ordering guarantee
• what happens if i run sequin without redis
• what happens if i run multiple sequin instances (clustering)
• latency in cloud deployment
• good batch size at 1000 inserts/sec
• idempotency key on backfills
• running in k8s
• nice to have
  • transforms & filters
  • audit logs

Running in Production

• postgres
  • own logical db for sequin to store source & sink configs
• redis
  • need to track replication

graph LR
    A[Postgres-Sequin] <--> B[Sequin]
    C[Redis-Sequin] <--> B
    E[Postgres-Source] --> B
    B --> D[Kafka]

Loading

Performance

• debezium
• sequin
• debezium vs sequin benchmark