Skip to content

patroni-cluster.md

Latest commit

 

History

History
266 lines (209 loc) · 8.81 KB

patroni-cluster.md

File metadata and controls

266 lines (209 loc) · 8.81 KB
Raw
title Backing up a Patroni cluster
description Configure pg_hardstorage against a 3-node Patroni cluster — slot continuity across failover, leader-aware backup routing.
tags
patroni
failover
cluster

Backing up a Patroni cluster

Wires pg_hardstorage to a 3-node Patroni cluster: one primary, two replicas, leader changes happen, the WAL stream continues without gaps. The agent watches Patroni's REST API, follows the leader, and recreates its replication slot on the new primary at the right LSN. About 25 minutes against a Docker-based Patroni stack.

The contract: the agent talks to one Patroni-managed PG cluster as a single logical deployment. WAL streaming routes to the current leader; on a leader change the streamer reconnects against the new primary's slot at the saved LSN, and any cross-timeline gap is recorded as a structured audit event so you can reason about RPO afterwards.

What you need

  • A 3-node Patroni cluster reachable from the operator host. The patroni/patroni Docker examples or any of the published Compose stacks work; for a quick local stand-up, use Patroni's reference stack.
  • The Patroni REST URL — usually http://patroni-leader:8008 for the reference Compose. Read endpoints (/cluster, /leader, /history) need no auth by default.
  • A replication user (CREATE ROLE pgbackup REPLICATION LOGIN ...) with a pg_hba.conf line that admits the operator host on every member. Patroni's bootstrap.pg_hba block applies this fleet-wide.

Steps

1. Confirm Patroni topology

Before any backup config, prove you can talk to Patroni:

# RUNNABLE skip-in-ci="needs Patroni cluster"
pg_hardstorage patroni status \
    --url http://patroni-leader:8008
patroni cluster "patroni-tutorial"
  Leader: patroni-1 (TLI 3)

  NAME       ROLE     STATE      HOST       PORT  TLI  LAG
  patroni-1  Leader   running    10.0.0.11  5432  3    0
  patroni-2  Replica  streaming  10.0.0.12  5432  3    0
  patroni-3  Replica  streaming  10.0.0.13  5432  3    0

patroni history prints the timeline-history events that record every promotion. The agent's leader-follow coordinator consumes those events when it reconnects after a failover.

2. Initialise the repo and the deployment

# RUNNABLE skip-in-ci="needs Patroni cluster"
pg_hardstorage repo init file:///srv/hs-patroni-repo

For Patroni clusters, write the connection string against the Patroni REST URL's leader endpoint rather than a single member — the agent rewrites the host every time the leader moves. The simplest form just points at the leader at config-time and lets the follow-loop take over:

# RUNNABLE skip-in-ci="needs Patroni cluster"
pg_hardstorage init \
    --pg-connection 'postgres://pgbackup@patroni-leader.example.com/postgres' \
    --repo file:///srv/hs-patroni-repo \
    --deployment pg-tutorial \
    --skip-backup \
    --yes

3. Add the Patroni block to the deployment

pg_hardstorage init writes the deployment without Patroni awareness. Either run

pg_hardstorage deployment edit pg-tutorial \
    --patroni-url http://patroni-leader:8008 \
    --patroni-slot pg_hardstorage_pg_tutorial \
    --patroni-interval 5s

…or hand-edit pg_hardstorage.yaml (path printed by pg_hardstorage doctor) and add a patroni: block:

deployments:
  pg-tutorial:
    pg_connection: postgres://pgbackup@patroni-leader.example.com/postgres
    repo: file:///srv/hs-patroni-repo
    patroni:
      url: http://patroni-leader:8008
      slot: pg_hardstorage_pg_tutorial   # explicit; default is auto-derived
      interval: 5s                        # poll cadence; matches Patroni's default leader TTL
      # user: opspg                       # optional HTTP basic auth
      # password: ...

pg_hardstorage deployment list shows the configured patroni-url, slot, and interval so you can verify the block landed.

The patroni.url activates the agent's leader-follow coordinator. With this block in place, every WAL streaming or backup run begins with a Patroni leader probe and routes to the current primary — not the host baked into pg_connection.

4. Start the agent (or run streaming directly)

For a long-lived setup you would run the agent under systemd:

pg_hardstorage agent

The agent spawns one Patroni follower coordinator per Patroni-enabled deployment. A misconfigured Patroni URL on one deployment is non-fatal — the follower for that deployment is dropped, the agent emits patroni.startup_partial, and the rest of the fleet keeps running.

For the tutorial, run the streamer directly so you can see the events:

pg_hardstorage wal stream pg-tutorial \
    --pg-connection 'postgres://pgbackup@patroni-leader.example.com/postgres' \
    --repo file:///srv/hs-patroni-repo

5. Take a backup

# RUNNABLE skip-in-ci="needs Patroni cluster"
pg_hardstorage backup pg-tutorial \
    --pg-connection 'postgres://pgbackup@patroni-leader.example.com/postgres' \
    --repo file:///srv/hs-patroni-repo

The base backup connects to the endpoint you give it. To take it from a replica (to keep primary I/O free), point --pg-connection at a replica's host — there is no automatic replica routing.

6. Force a failover and watch the slot survive

In a Patroni-shell terminal, force a manual failover:

patronictl -c /etc/patroni.yml failover --master patroni-1 --candidate patroni-2

In a third terminal, follow the Patroni leader stream:

pg_hardstorage patroni follow \
    --url http://patroni-leader:8008

patroni follow streams a patroni.leader_change event each time the leader changes (and patroni.poll_error on a transient REST error); each event carries the old and new leader's name and host:port. In text mode it renders through the generic event renderer:

11:03:42 [NOTICE] patroni.leader_change

In the wal stream terminal you will see the follower coordinator react. It emits events under the wal.follower namespace — the ones that matter on a failover:

  • wal.follower.leader_change — the coordinator saw the new leader;
  • wal.follower.slot_reconciled — the physical slot was found (or recreated) on the new leader;
  • wal.follower.wal_gap_detected — critical, emitted only if the reconnect could not bridge the WAL (e.g. the slot was lost and recreated past the last archived LSN).
11:03:42 [NOTICE] wal.follower.leader_change
11:03:43 [NOTICE] wal.follower.slot_reconciled

The agent recreated its replication slot on the new primary at the saved LSN (or the lowest available LSN, whichever is more conservative), and recorded the cross-timeline gap as a structured audit event. You can list gaps later with:

pg_hardstorage wal gaps pg-tutorial \
    --repo file:///srv/hs-patroni-repo

7. Confirm continuity

pg_hardstorage status rolls up the picture:

pg_hardstorage status pg-tutorial
pg-tutorial  PG 17.x  primary @ 10.0.0.12:5432  (Patroni leader)
  Last backup     8m ago  · full · 1.2 GB · ✓ verified
  WAL streaming   active   · lag 4s · slot pg_hardstorage_pg_tutorial
  Patroni         3 members · TLI 4 · last failover 2m ago
  RPO / RTO       8m / ~3m (estimate)
  Health          ✓ all clear

What just happened

You configured one logical deployment against a real 3-node Patroni cluster, exercised a failover, and watched the agent rebind to the new primary without operator intervention. The two non-trivial pieces of the design that earned their keep:

  • Patroni reuse, not duplication. The agent reads Patroni's REST state and writes its own coordination keys under /pg_hardstorage/<deployment>/... in the existing DCS (etcd, Consul, ZooKeeper — whatever Patroni already runs). No second DCS, no coordination tax.
  • The slot is the source of truth. PG retains WAL until the slot ACKs; the agent only ACKs after a segment commits in the repo. Combine that with the leader-follow coordinator and the gap window shrinks to "the seconds between the old primary's last commit and the new primary's promotion" — which is precisely the cross- timeline gap event the audit log records.

Next steps