Skip to content

Commit 2c32890

Browse files
jensensclaude
andcommitted
docs: lead migration guide with zodb-convert CLI
Now that zodb-convert supports --incremental -w N, make the CLI the primary path in the migration docs. Python API moves to sub-sections for users who need more control. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 1e880fe commit 2c32890

1 file changed

Lines changed: 34 additions & 24 deletions

File tree

docs/sources/how-to/migrate-with-zodbconvert.md

Lines changed: 34 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
# How to migrate with zodbconvert
44

5-
This guide shows you how to migrate an existing ZODB database (FileStorage or RelStorage) to zodb-pgjsonb using `zodbconvert` or the Python API.
5+
This guide shows you how to migrate an existing ZODB database (FileStorage or RelStorage) to zodb-pgjsonb using [zodb-convert](https://github.com/bluedynamics/zodb-convert) or the Python API.
66

77
## Write a zodbconvert configuration file
88

@@ -43,18 +43,33 @@ Create a file named `migrate.cfg`:
4343
</destination>
4444
```
4545

46-
## Run zodbconvert
46+
## Run the migration
4747

4848
```bash
49-
zodbconvert migrate.cfg
49+
zodb-convert migrate.cfg
5050
```
5151

5252
This copies all transactions sequentially, including blobs.
5353
The destination storage creates its schema automatically.
5454

55+
`zodb-convert` is a standalone tool that works with any ZODB-compatible storage.
56+
Install it with `pip install zodb-convert` (or `uv pip install zodb-convert`).
57+
5558
## Run a parallel migration
5659

57-
For large databases, use the Python API with the `workers` parameter for faster migration:
60+
For large databases, use multiple worker threads for faster migration:
61+
62+
```bash
63+
zodb-convert -w 4 migrate.cfg
64+
```
65+
66+
This delegates to the destination storage's `copyTransactionsFrom(source, workers=4)`.
67+
The main thread iterates the source and decodes pickles; worker threads write to PostgreSQL concurrently.
68+
Set `pool-max-size` in your destination config to at least the number of workers plus one.
69+
70+
### Python API
71+
72+
If you need more control, use the Python API directly:
5873

5974
```python
6075
from zodb_pgjsonb.storage import PGJsonbStorage
@@ -71,14 +86,23 @@ dest.close()
7186
source.close()
7287
```
7388

74-
Workers are capped to the destination storage's `pool_max_size`.
75-
The main thread iterates the source and decodes pickles; worker threads write to PostgreSQL concurrently.
76-
Set `pool_max_size` to at least the number of workers you plan to use plus one (for the watermark tracker).
89+
## Resume an interrupted migration
7790

78-
## Resume an interrupted parallel migration
91+
If a migration is interrupted (Ctrl-C, crash, network failure), resume with `--incremental`:
92+
93+
```bash
94+
zodb-convert --incremental -w 4 migrate.cfg
95+
```
96+
97+
During parallel copy, the storage tracks a *watermark* — the highest TID where all
98+
prior TIDs are also committed.
99+
On resume, iteration starts from the watermark (not `lastTransaction()`) to fill any
100+
gaps left by out-of-order worker commits.
101+
Already-committed transactions are skipped automatically.
102+
The watermark table is dropped on successful completion, adding zero overhead to
103+
non-incremental imports.
79104

80-
If a parallel migration is interrupted (Ctrl-C, crash, network failure), it can be
81-
resumed without losing progress or re-copying already-committed transactions:
105+
### Python API
82106

83107
```python
84108
from zodb_pgjsonb.storage import PGJsonbStorage
@@ -99,20 +123,6 @@ dest.close()
99123
source.close()
100124
```
101125

102-
During parallel copy, the storage tracks a *watermark* — the highest TID where all
103-
prior TIDs are also committed.
104-
On resume, iteration starts from the watermark (not `lastTransaction()`) to fill any
105-
gaps left by out-of-order worker commits.
106-
Already-committed transactions are skipped automatically.
107-
The watermark table is dropped on successful completion, adding zero overhead to
108-
non-incremental imports.
109-
110-
With [zodb-convert](https://github.com/bluedynamics/zodb-convert), this is simply:
111-
112-
```bash
113-
zodb-convert --incremental -w 4 migrate.cfg
114-
```
115-
116126
## Migrate blobs
117127

118128
Blobs are migrated automatically by both `zodbconvert` and `copyTransactionsFrom`.

0 commit comments

Comments
 (0)