You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Now that zodb-convert supports --incremental -w N, make the CLI
the primary path in the migration docs. Python API moves to
sub-sections for users who need more control.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/sources/how-to/migrate-with-zodbconvert.md
+34-24Lines changed: 34 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
# How to migrate with zodbconvert
4
4
5
-
This guide shows you how to migrate an existing ZODB database (FileStorage or RelStorage) to zodb-pgjsonb using `zodbconvert` or the Python API.
5
+
This guide shows you how to migrate an existing ZODB database (FileStorage or RelStorage) to zodb-pgjsonb using [zodb-convert](https://github.com/bluedynamics/zodb-convert) or the Python API.
6
6
7
7
## Write a zodbconvert configuration file
8
8
@@ -43,18 +43,33 @@ Create a file named `migrate.cfg`:
43
43
</destination>
44
44
```
45
45
46
-
## Run zodbconvert
46
+
## Run the migration
47
47
48
48
```bash
49
-
zodbconvert migrate.cfg
49
+
zodb-convert migrate.cfg
50
50
```
51
51
52
52
This copies all transactions sequentially, including blobs.
53
53
The destination storage creates its schema automatically.
54
54
55
+
`zodb-convert` is a standalone tool that works with any ZODB-compatible storage.
56
+
Install it with `pip install zodb-convert` (or `uv pip install zodb-convert`).
57
+
55
58
## Run a parallel migration
56
59
57
-
For large databases, use the Python API with the `workers` parameter for faster migration:
60
+
For large databases, use multiple worker threads for faster migration:
61
+
62
+
```bash
63
+
zodb-convert -w 4 migrate.cfg
64
+
```
65
+
66
+
This delegates to the destination storage's `copyTransactionsFrom(source, workers=4)`.
67
+
The main thread iterates the source and decodes pickles; worker threads write to PostgreSQL concurrently.
68
+
Set `pool-max-size` in your destination config to at least the number of workers plus one.
69
+
70
+
### Python API
71
+
72
+
If you need more control, use the Python API directly:
58
73
59
74
```python
60
75
from zodb_pgjsonb.storage import PGJsonbStorage
@@ -71,14 +86,23 @@ dest.close()
71
86
source.close()
72
87
```
73
88
74
-
Workers are capped to the destination storage's `pool_max_size`.
75
-
The main thread iterates the source and decodes pickles; worker threads write to PostgreSQL concurrently.
76
-
Set `pool_max_size` to at least the number of workers you plan to use plus one (for the watermark tracker).
89
+
## Resume an interrupted migration
77
90
78
-
## Resume an interrupted parallel migration
91
+
If a migration is interrupted (Ctrl-C, crash, network failure), resume with `--incremental`:
92
+
93
+
```bash
94
+
zodb-convert --incremental -w 4 migrate.cfg
95
+
```
96
+
97
+
During parallel copy, the storage tracks a *watermark* — the highest TID where all
98
+
prior TIDs are also committed.
99
+
On resume, iteration starts from the watermark (not `lastTransaction()`) to fill any
100
+
gaps left by out-of-order worker commits.
101
+
Already-committed transactions are skipped automatically.
102
+
The watermark table is dropped on successful completion, adding zero overhead to
103
+
non-incremental imports.
79
104
80
-
If a parallel migration is interrupted (Ctrl-C, crash, network failure), it can be
81
-
resumed without losing progress or re-copying already-committed transactions:
105
+
### Python API
82
106
83
107
```python
84
108
from zodb_pgjsonb.storage import PGJsonbStorage
@@ -99,20 +123,6 @@ dest.close()
99
123
source.close()
100
124
```
101
125
102
-
During parallel copy, the storage tracks a *watermark* — the highest TID where all
103
-
prior TIDs are also committed.
104
-
On resume, iteration starts from the watermark (not `lastTransaction()`) to fill any
105
-
gaps left by out-of-order worker commits.
106
-
Already-committed transactions are skipped automatically.
107
-
The watermark table is dropped on successful completion, adding zero overhead to
108
-
non-incremental imports.
109
-
110
-
With [zodb-convert](https://github.com/bluedynamics/zodb-convert), this is simply:
111
-
112
-
```bash
113
-
zodb-convert --incremental -w 4 migrate.cfg
114
-
```
115
-
116
126
## Migrate blobs
117
127
118
128
Blobs are migrated automatically by both `zodbconvert` and `copyTransactionsFrom`.
0 commit comments