Skip to content

Commit 3d3ee56

Browse files
meskillmeskill
committed
fix: improve documentation for SHOW TASKS and cutover flow
1 parent c4c92bd commit 3d3ee56

5 files changed

Lines changed: 214 additions & 11 deletions

File tree

‎docs/administration/index.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,13 @@ The admin database name is [configurable](../configuration/pgdog.toml/admin.md).
3232
| `SHOW QUERY_CACHE` | List statements currently in the AST cache used for query routing. |
3333
| [`MAINTENANCE`](maintenance_mode.md) | Pause all queries to synchronize configuration changes across multiple instances of PgDog. |
3434
| [`SHOW REPLICATION`](replication.md) | Show the status of PostgreSQL replication for each database, including replica lag. |
35+
| [`RESHARD`](../features/sharding/resharding/index.md) | Reshard a database online and cut over traffic automatically. Returns a background `task_id`. |
36+
| [`COPY_DATA`](../features/sharding/resharding/move.md) | Copy and reshard data to a destination cluster, then stream changes. Returns a background `task_id`. |
37+
| [`SCHEMA_SYNC`](../features/sharding/resharding/schema.md) | Synchronize schema for a given phase between source and destination. Returns a background `task_id`. |
38+
| `REPLICATE` | Stream changes from source to destination (schema and data must already be synced). Returns a background `task_id`. |
39+
| [`CUTOVER`](../features/sharding/resharding/cutover.md) | Cut traffic over to the destination cluster for a running replication task. |
40+
| [`SHOW TASKS`](tasks.md) | List background tasks (resharding, copy, replication, schema sync) with their status and elapsed time. |
41+
| [`STOP_TASK`](tasks.md#stopping-a-task) | Stop a running background task by its `task_id`. |
3542

3643
## Shutting down PgDog
3744

‎docs/administration/tasks.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
icon: material/format-list-checks
3+
---
4+
5+
# Background tasks
6+
7+
Long-running operations like [resharding](../features/sharding/resharding/index.md) don't block the connection that started them. Instead, `RESHARD`, `COPY_DATA`, `REPLICATE`, and `SCHEMA_SYNC` each return a `task_id` and run in the background. `SHOW TASKS` is how you track those tasks: it reports their lifecycle status and progress, lets you find the `task_id` to pass to [`STOP_TASK`](#stopping-a-task) or [`CUTOVER`](../features/sharding/resharding/cutover.md), and surfaces failures.
8+
9+
Run it on the [admin database](index.md):
10+
11+
```
12+
SHOW TASKS;
13+
```
14+
15+
!!! note "Admin database tasks only"
16+
`SHOW TASKS` lists the background tasks running *inside this PgDog process* — the ones started by the admin database commands above. The equivalent [CLI commands](../features/sharding/resharding/index.md#running-the-steps-manually) (`pgdog data-sync`, `pgdog schema-sync`) run as a separate, one-off `pgdog` process in the foreground: they block until they finish and are stopped with `Ctrl-C`, so they neither return a `task_id` nor appear here.
17+
18+
=== "Output"
19+
```
20+
-[ RECORD 1 ]+----------------------------------
21+
id | 14
22+
scope | root
23+
type | reshard prod -> prod_sharded
24+
status | running
25+
inner_status | syncing data
26+
started_at | 2026-06-30 18:02:11.004 UTC
27+
updated_at | 2026-06-30 18:05:42.119 UTC
28+
elapsed | 00:03:31:115
29+
elapsed_ms | 211115
30+
-[ RECORD 2 ]+----------------------------------
31+
id |
32+
scope | subtask
33+
type | copy_data prod -> prod_sharded
34+
status | running
35+
inner_status |
36+
started_at | 2026-06-30 18:02:30.550 UTC
37+
updated_at | 2026-06-30 18:05:42.119 UTC
38+
elapsed | 00:03:11:569
39+
elapsed_ms | 191569
40+
```
41+
42+
The most recently started task is listed first.
43+
44+
## Columns
45+
46+
| Column | Description |
47+
|-|-|
48+
| `id` | The task's id. This is the handle you pass to [`STOP_TASK`](#stopping-a-task) and [`CUTOVER`](../features/sharding/resharding/cutover.md). Only **root** rows carry an id; subtasks share their root's id and leave this column empty. |
49+
| `scope` | `root` for a top-level task, `subtask` for a step it spawned (e.g. the `copy_data` and `replication` steps of a `reshard`). |
50+
| `type` | What the task is, usually with the source and destination databases — e.g. `reshard`, `copy_data`, `replication`, `replication ... (reverse)`, `schema_sync(pre)`. |
51+
| `status` | The lifecycle status of the task. See [Statuses](#statuses). |
52+
| `inner_status` | Fine-grained progress within the current status. See [Progress](#progress). For a finished or failed task, this keeps the last progress it reported. |
53+
| `started_at` | When the task started. |
54+
| `updated_at` | When the task last changed status or progress. For a terminal task this is when it finished. |
55+
| `elapsed` / `elapsed_ms` | How long the task ran. For a terminal task this is the total run time (measured to `updated_at`), not a clock that keeps ticking. |
56+
57+
## Statuses
58+
59+
The `status` column describes where the task is in its lifecycle:
60+
61+
| Status | Meaning |
62+
|-|-|
63+
| `started` | The task is being set up. |
64+
| `running` | The task is actively working. |
65+
| `cancelling` | A [`STOP_TASK`](#stopping-a-task) was requested and the task is winding down cooperatively. |
66+
| `finished` | The task completed successfully. |
67+
| `cancelled` | The task was stopped before completing. |
68+
| `failed: <error>` | The task errored. The error message is included in the status. |
69+
| `panicked: <error>` | The task hit an unexpected internal error. |
70+
71+
`finished`, `cancelled`, `failed`, and `panicked` are **terminal**. Terminal tasks stay listed with their final status (and last progress) for a while so you can inspect the outcome, then they're pruned automatically.
72+
73+
## Progress
74+
75+
The `inner_status` column shows what the task is doing within its current `status`. The values depend on the task `type`:
76+
77+
| Task | Progress values |
78+
|-|-|
79+
| `reshard` | `syncing schema` → `syncing data` → `finalizing schema` → `replicating` |
80+
| `copy_data` | (no sub-status; track copy progress with [`SHOW TABLE_COPIES`](../features/sharding/resharding/move.md#monitoring-progress)) |
81+
| `replication` | `replicating` → `cutting over` (or `rolling back` for a reverse stream) / `stopping` |
82+
| `schema_sync` | `loading schema` → `syncing tables` → `creating indexes` (or `syncing cutover schema`) |
83+
84+
## Finding the right id
85+
86+
`STOP_TASK` and `CUTOVER` always operate on the **root** `id`:
87+
88+
- A `RESHARD` or `COPY_DATA` shows as a `root` task with its child steps listed as `subtask` rows. Target the root id, not a subtask — subtask rows have an empty `id` on purpose.
89+
- After a cutover, the [reverse replication](../features/sharding/resharding/cutover.md#after-the-cutover) stream appears as its own `root` task of type `replication ... (reverse)`. Use its id to [roll back](../features/sharding/resharding/cutover.md#rolling-back) (`CUTOVER <id>`) or to [finalize the migration](../features/sharding/resharding/cutover.md#finalizing-the-migration) (`STOP_TASK <id>`).
90+
91+
## Spotting issues
92+
93+
| Symptom | What to check |
94+
|-|-|
95+
| `status` is `failed: ...` or `panicked: ...` | Read the error message in the `status` column, then check the PgDog logs for the full context. The task is stopped; address the cause and re-run the command. |
96+
| Task stuck at one `inner_status` for a long time | Cross-reference [`SHOW TABLE_COPIES`](../features/sharding/resharding/move.md#monitoring-progress) (during `syncing data`) and [`SHOW REPLICATION_SLOTS`](../features/sharding/resharding/move.md#streaming-updates) (during `replicating`) to see whether data is still flowing. |
97+
| `status` is `cancelling` and not clearing | The task is draining its replication streams. A cutover that has already started runs to completion before the status settles. |
98+
| Terminal task you expected to still be running | It reached a terminal state (`finished`/`cancelled`/`failed`). Check `inner_status` and `updated_at` to see where and when it stopped. |
99+
100+
## Stopping a task
101+
102+
Stop any running task by its root `id`:
103+
104+
```
105+
STOP_TASK <task_id>;
106+
```
107+
108+
This requests cancellation; the task winds down gracefully and stops appearing as `running` once it has actually stopped. For replication and copy tasks, a graceful stop also drops the replication slot the task created. See [finalizing the migration](../features/sharding/resharding/cutover.md#finalizing-the-migration) for how this is used after a cutover.

‎docs/features/sharding/resharding/cutover.md‎

Lines changed: 45 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,16 @@ Traffic cutover involves moving application traffic (read and write queries) to
1212

1313
## Performing the cutover
1414

15-
The cutover can be executed by executing a command on the [admin database](../../../administration/index.md):
15+
The cutover can be executed by running a command on the [admin database](../../../administration/index.md):
1616

1717
```
18-
CUTOVER;
18+
CUTOVER [<task_id>];
19+
```
20+
21+
Without a `task_id`, PgDog cuts over the first running replication task. To target a specific task, pass its id (as reported by [`SHOW TASKS`](../../../administration/tasks.md)):
22+
23+
```
24+
CUTOVER 12;
1925
```
2026

2127
Under typical conditions, the whole process takes less than a second, so applications shouldn't experience any errors or downtime.
@@ -25,8 +31,12 @@ Under typical conditions, the whole process takes less than a second, so applica
2531
must connect to the database through PgDog. Any applications that connect to the database directly, or through another proxy,
2632
will not receive the cutover signal and will continue to send writes to the source database, causing a split-brain
2733
situation.
28-
29-
If you're using the `RESHARD` command, the cutover step is executed automatically and you don't need to perform any additional steps.
34+
35+
`CUTOVER` is only needed when you run resharding manually (via [`COPY_DATA`](move.md) / [`REPLICATE`](../../../administration/index.md)). If you use the `RESHARD` command, the cutover step is executed automatically and you don't need to perform any additional steps.
36+
37+
`CUTOVER` only works on a migration that was started on **this** PgDog through the admin database (`COPY_DATA`, `REPLICATE`, or `RESHARD`). A migration started with the `pgdog data-sync` CLI runs in a separate process and **cannot** be cut over with `CUTOVER` command automatically.
38+
39+
In order to cut over a CLI migration by hand: once replication has caught up (check the log output of running the CLI), swap the source and destination in `pgdog.toml` and `users.toml`, run [`RELOAD`](../../../administration/index.md) on the serving instance, then stop the CLI. Pause writes ([`MAINTENANCE`](../../../administration/maintenance_mode.md)) while you do this to avoid losing in-flight transactions; there's no automatic rollback with this path. Then stop the CLI replication process and resume writes.
3040

3141
## Step by step
3242

@@ -88,7 +98,7 @@ When enabled, PgDog will backup both configuration files, `pgdog.toml` as `pgdog
8898

8999
!!! note "Multi-node deployments"
90100
If you're running more than one PgDog node, you should consider deploying our [Enterprise Edition](../../../enterprise_edition/index.md), which has support for saving the configuration files on multiple PgDog nodes at the same time.
91-
101+
92102
#### Thresholds
93103

94104
Before swapping the configuration, PgDog waits for the two databases to be completely identical. These thresholds are configurable as follows:
@@ -143,3 +153,33 @@ The reverse replication is created while the queries to both databases are pause
143153
### Resume queries
144154

145155
With the reverse replication set up, it is now safe to move traffic to the destination (now source) database. PgDog does this by turning off [maintenance mode](../../../administration/maintenance_mode.md), and this step concludes the cutover. The entire process takes less than a second, typically, and allows PgDog to reshard Postgres databases without downtime.
156+
157+
## After the cutover
158+
159+
Once the cutover completes, the reverse [replication stream](#reverse-replication) keeps the original cluster up to date with every write that now lands on the new cluster. It runs as a background task — find it in [`SHOW TASKS`](../../../administration/tasks.md) (its `type` is `replication`). While it is running, you can either roll back to the original cluster or, once you're satisfied, finalize the migration.
160+
161+
### Rolling back
162+
163+
To restore traffic to the original cluster, run a `CUTOVER` against the reverse replication task, passing its id from `SHOW TASKS`:
164+
165+
```
166+
CUTOVER <task_id>;
167+
```
168+
169+
This performs the same atomic swap in reverse: the source and destination are swapped back in `pgdog.toml` and `users.toml`, and traffic returns to the original cluster. Because the reverse stream kept the original cluster in sync, no data written to the new cluster is lost. The task's status in `SHOW TASKS` shows `rolling back` while this happens.
170+
171+
!!! note "Restoring the configuration files"
172+
The rollback swaps the configuration back in memory (and on disk when [`cutover_save_config`](#swap-the-configuration) is enabled). If you enabled `cutover_save_config`, the configuration as it was *before* the original cutover is also preserved in the `pgdog.bak.toml` and `users.bak.toml` backups, so you can always restore the previous state by hand.
173+
174+
### Finalizing the migration
175+
176+
When you're confident the new cluster is healthy and no longer need the ability to roll back, stop the reverse replication task:
177+
178+
```
179+
STOP_TASK <task_id>;
180+
```
181+
182+
This winds the reverse stream down gracefully and **drops the replication slot** it created, so Postgres can resume recycling WAL. After this point rollback is no longer possible — the migration is complete.
183+
184+
!!! warning "Don't leave reverse replication running indefinitely"
185+
Until the reverse replication task is stopped, its permanent replication slot prevents the new (now source) cluster from recycling WAL, which accumulates on disk. Issue `STOP_TASK` once you've decided not to roll back.

‎docs/features/sharding/resharding/index.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,25 @@ RESHARD <source> <destination> <publication>;
3535

3636
The `<source>` and `<destination>` parameters accept the name of the source and destination databases respectively. The `<publication>` parameter expects the name of the Postgres [publication](schema.md#publication) for the tables that need to be resharded.
3737

38+
`RESHARD` returns a `task_id` and runs in the background. You can track its progress with [`SHOW TASKS`](../../../administration/tasks.md), and stop it with `STOP_TASK <task_id>`. When `RESHARD` is used, traffic cutover happens automatically as the final step.
39+
40+
### Running the steps manually
41+
42+
Instead of `RESHARD`, you can run the process one step at a time. This gives you control over *when* traffic is cut over. Each step can be run either as an [admin database](../../../administration/index.md) command or as a `pgdog` CLI command:
43+
44+
| Step | Admin database | CLI |
45+
|-|-|-|
46+
| [Schema sync](schema.md) | `SCHEMA_SYNC <phase> <source> <destination> <publication>` | `pgdog schema-sync ...` |
47+
| [Move & reshard data](move.md) | `COPY_DATA <source> <destination> <publication>` | `pgdog data-sync ...` |
48+
| [Cutover traffic](cutover.md) | `CUTOVER [<task_id>]` | _admin database only_ |
49+
50+
The two run differently:
51+
52+
- **Admin database commands** run as background tasks *inside the running PgDog*. They return a `task_id` immediately, are tracked with [`SHOW TASKS`](../../../administration/tasks.md), and are controlled with [`STOP_TASK`](../../../administration/tasks.md#stopping-a-task) and [`CUTOVER`](cutover.md).
53+
- **CLI commands** run as a separate, one-off `pgdog` process in the **foreground**: they block until the operation finishes and are stopped with `Ctrl-C`. They do not appear in `SHOW TASKS`.
54+
55+
Unlike `RESHARD`, the manual path does **not** cut over automatically: the data move copies the data and keeps streaming changes indefinitely, and you switch traffic explicitly with [`CUTOVER`](cutover.md).
56+
3857
!!! note "Traffic cutover"
3958
Traffic cutover requires careful synchronization to avoid data loss and a split-brain situation. The `RESHARD` command supports this for **single node** PgDog deployments only. The [Enterprise Edition](../../../enterprise_edition/index.md) provides a control plane, which supports traffic cutover with multiple PgDog containers.
4059

‎docs/features/sharding/resharding/move.md‎

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,11 +36,18 @@ This will spawn a background task that will copy all tables in the [publication]
3636

3737
To copy data from database `"prod"` to database `"prod_sharded"` and the `"all_tables"` publication, execute the following command:
3838

39-
```
40-
COPY_DATA prod prod_sharded all_tables;
41-
```
39+
=== "Admin command"
40+
```
41+
COPY_DATA prod prod_sharded all_tables;
42+
```
43+
=== "Output"
44+
```
45+
task_id | replication_slot
46+
---------+-----------------------------
47+
12 | __pgdog_repl_a1b2c3d4e5f6...
48+
```
4249

43-
The name of the replication slot will be automatically generated.
50+
`COPY_DATA` returns immediately with the background `task_id` and the auto-generated replication slot name. Track the task with [`SHOW TASKS`](../../../administration/tasks.md) and stop it with `STOP_TASK <task_id>`. The slot name is not provided, so it is generated for you.
4451

4552
### CLI
4653

@@ -63,6 +70,9 @@ Required (*) and optional parameters for this command are as follows:
6370
| `--replication-slot` | Name of the replication slot to use (and create if it doesn't exist) for syncing real-time changes. |
6471
| `--replicate-only` | Don't copy data, just stream changes from the replication slot. |
6572
| `--sync-only` | Perform the initial data sync only and exit. |
73+
| `--skip-schema-sync` | Don't run the pre-data schema sync first (assume the destination schema already exists). |
74+
75+
Unlike the `COPY_DATA` admin command, the CLI runs in the **foreground** as a separate `pgdog` process: it blocks while it copies the data and then keeps streaming changes until you stop it with `Ctrl-C` (which winds the task down gracefully), or until you pass `--sync-only` to copy and exit. It does not return a `task_id` and is not visible in [`SHOW TASKS`](../../../administration/tasks.md).
6676

6777
## How it works
6878

@@ -238,7 +248,26 @@ SELECT
238248
pg_current_wal_lsn()
239249
FROM pg_replication_slots;
240250
```
241-
The replication delay between the two database clusters is measured in bytes. When that number reaches zero, the two databases are byte-for-byte identical, and traffic can be [cut over](cutover.md) to the destination database.
251+
The replication delay between the two database clusters is measured in bytes. When that number reaches zero, the two databases are byte-for-byte identical.
252+
253+
`COPY_DATA` keeps streaming changes indefinitely, keeping the destination in sync with the source. To monitor or stop the background task, see the task-level view below.
254+
255+
### Monitoring the task
256+
257+
When started from the admin database, `COPY_DATA` runs as a background task. List all tasks, their current status, and elapsed time with:
258+
259+
=== "Admin command"
260+
```
261+
SHOW TASKS;
262+
```
263+
=== "Output"
264+
```
265+
id | scope | type | status | inner_status | elapsed
266+
----+-------+------------------------------+---------+--------------+----------
267+
12 | root | reshard prod -> prod_sharded | running | replicating | 00:02:14
268+
```
269+
270+
`COPY_DATA` runs the same pipeline as `RESHARD` but leaves the traffic in place, so it appears with a `reshard` type and progresses through `syncing data` to `replicating`. Stop it with `STOP_TASK <task_id>`. See [Background tasks](../../../administration/tasks.md) for the full column and status reference.
242271

243272
## Troubleshooting
244273

@@ -311,7 +340,7 @@ See [Integer primary keys](#integer-primary-keys) for the other common reason to
311340

312341
### Replication slot not cleaned up after stop or crash
313342

314-
If `COPY_DATA` is stopped via `STOP_TASK` or PgDog exits unexpectedly, the **permanent** replication slot will remain on the source. Postgres will not recycle WAL until it is dropped. Clean it up manually:
343+
Stopping a task gracefully with `STOP_TASK` winds the replication stream down and drops its **permanent** replication slot for you. However, if PgDog exits unexpectedly (crash, `SIGKILL`, lost connection) the slot can remain on the source. Postgres will not recycle WAL until it is dropped, so clean it up manually:
315344

316345
```postgresql
317346
SELECT pg_drop_replication_slot('slot_name');

0 commit comments

Comments
 (0)