Skip to content

crunchy-postgres.md

Latest commit

 

History

History
94 lines (65 loc) · 4.42 KB

crunchy-postgres.md

File metadata and controls

94 lines (65 loc) · 4.42 KB
sidebar_label Crunchy Bridge Postgres
description Set up Crunchy Bridge Postgres as a source for ClickPipes
slug /integrations/clickpipes/postgres/source/crunchy-postgres
title Crunchy Bridge Postgres Source Setup Guide
keywords
crunchy bridge
postgres
clickpipes
logical replication
data ingestion
doc_type guide
integration
support_level
core
category
clickpipes

import firewall_rules_crunchy_bridge from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/crunchy-postgres/firewall_rules_crunchy_bridge.png' import add_firewall_rules_crunchy_bridge from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/crunchy-postgres/add_firewall_rules_crunchy_bridge.png' import Image from '@theme/IdealImage';

Crunchy Bridge Postgres source setup guide

ClickPipes supports Postgres version 12 and later.

Enable logical replication {#enable-logical-replication}

Crunchy Bridge comes with logical replication enabled by default. Ensure that the settings below are configured correctly. If not, adjust them accordingly.

SHOW wal_level; -- should be logical
SHOW max_wal_senders; -- should be 10
SHOW max_replication_slots; -- should be 10

Creating ClickPipes user and granting permissions {#creating-clickpipes-user-and-granting-permissions}

Connect to your Crunchy Bridge Postgres through the postgres user and run the below commands:

  1. Create a dedicated user for ClickPipes:

    CREATE USER clickpipes_user PASSWORD 'some-password';

  2. Grant schema-level, read-only access to the user you created in the previous step. The following example shows permissions for the public schema. Repeat these commands for each schema containing tables you want to replicate:

    GRANT USAGE ON SCHEMA "public" TO clickpipes_user;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user;

  3. Grant replication privileges to the user:

     ALTER USER clickpipes_user WITH REPLICATION;

  4. Create a publication with the tables you want to replicate. We strongly recommend only including the tables you need in the publication to avoid performance overhead.

    :::warning Any table included in the publication must either have a primary key defined or have its replica identity configured to FULL. See the Postgres FAQs for guidance on scoping. :::

    • To create a publication for specific tables:

      CREATE PUBLICATION clickpipes FOR TABLE table_to_replicate, table_to_replicate2;

    • To create a publication for all tables in a specific schema:

      CREATE PUBLICATION clickpipes FOR TABLES IN SCHEMA "public";

    The clickpipes publication will contain the set of change events generated from the specified tables, and will later be used to ingest the replication stream.

    :::warning Avoid using FOR ALL TABLES unless you intend to replicate every table. Including unnecessary tables increases WAL traffic from Postgres to ClickPipes and reduces overall replication efficiency. :::

    :::note ClickPipes can automatically create and manage the publication on your behalf. However, this requires granting the ClickPipes user both table ownership and the CREATE permission on the database. If you prefer read-only access for the ClickPipes user, we recommend creating and managing the publication manually. :::

Safe list ClickPipes IPs {#safe-list-clickpipes-ips}

Safelist ClickPipes IPs by adding the Firewall Rules in Crunchy Bridge.

Where to find Firewall Rules in Crunchy Bridge?

Add the Firewall Rules for ClickPipes

What's next? {#whats-next}

You can now create your ClickPipe and start ingesting data from your Postgres instance into ClickHouse Cloud. Make sure to note down the connection details you used while setting up your Postgres instance as you will need them during the ClickPipe creation process.