Skip to content

README.md

Latest commit

 

History

History
76 lines (63 loc) · 1.78 KB

README.md

File metadata and controls

76 lines (63 loc) · 1.78 KB

Postgres CDC to Kafka using Sequin

NOTE: docker setup from here

graph LR
    A[Postgres] --> B[Sequin]
    B --> C[Kafka]
Loading

Usage

Prerequisites

Setup

just setup

Run

just up # run docker compose up
# go to sequin ui & validate sink config
just run # insert data
just down # run docker compose down

Export Sequin Config

# install cli
curl -sf https://raw.githubusercontent.com/sequinstream/sequin/main/cli/installer.sh | sh
# export config

Notes

  • when you set a BATCH_SIZE - ordering of events in a batch is not guaranteed
  • cannot run WITHOUT redis

Todo

  • multiple tables with high volume using 1 replication slot. how does partitioning help w load
  • ordering guarantee
  • what happens if i run sequin without redis
  • what happens if i run multiple sequin instances (clustering)
  • latency in cloud deployment
  • good batch size at 1000 inserts/sec
  • idempotency key on backfills
  • running in k8s
  • nice to have
    • transforms & filters
    • audit logs

Running in Production

  • postgres
    • own logical db for sequin to store source & sink configs
  • redis
    • need to track replication
graph LR
    A[Postgres-Sequin] <--> B[Sequin]
    C[Redis-Sequin] <--> B
    E[Postgres-Source] --> B
    B --> D[Kafka]
Loading

Performance