Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,12 @@

## 2.4.1 (unreleased)

- No changes yet.
- Feature: Generate generic `convert-import.conf` and `convert-export.conf`
for all non-filestorage backends (RelStorage, PGJsonb, ZEO) using the
storage-agnostic [zodb-convert](https://pypi.org/project/zodb-convert/)
tool. Deprecate RelStorage-specific `zodbconvert` configuration in favor
of `zodb-convert`. Add migration how-to guide.
[@jensens]

## 2.4.0 (2026-02-25)

Expand Down
5 changes: 5 additions & 0 deletions cookiecutter.json
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,11 @@
"db_pgjsonb_blob_cache_dir": "",
"db_pgjsonb_blob_cache_size": "",

"db_convert_import_filestorage_location": "",
"db_convert_import_blobs_location": "",
"db_convert_export_filestorage_location": "",
"db_convert_export_blobs_location": "",

"db_z3blobs_enabled": false,
"db_z3blobs_bucket_name": "",
"db_z3blobs_s3_prefix": "",
Expand Down
11 changes: 9 additions & 2 deletions docs/sources/explanation/storage-backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,5 +161,12 @@ For most teams, the decision comes down to:
5. **Need S3 blob storage with direct, relstorage, or zeo?** Enable `db_z3blobs_enabled`.

All four backends support the same ZODB API, so switching between them is a
configuration change (plus data migration). Your application code does not
need to change.
configuration change plus a one-time data migration. The
[zodb-convert](https://pypi.org/project/zodb-convert/) tool handles this
migration generically -- it can copy data between any two ZODB-compatible
storages. Because `cookiecutter-zope-instance` generates a complete
`zope.conf` for each backend, `zodb-convert` can read the storage
configuration directly from these files. See {doc}`/how-to/migrate-storage`
for a step-by-step guide.

Your application code does not need to change.
2 changes: 2 additions & 0 deletions docs/sources/how-to/configure-pgjsonb.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ used** with PGJsonb. PGJsonb manages blob storage through its own settings

## Next steps

- {doc}`migrate-storage` -- Migrate data from an existing storage backend
into PGJsonb using `zodb-convert`.
- {doc}`/reference/database-pgjsonb` -- Full reference for all PGJsonb
options including connection pool, cache, and S3 settings.
- {doc}`/explanation/storage-backends` -- Comparison of all storage backends.
Expand Down
10 changes: 10 additions & 0 deletions docs/sources/how-to/configure-relstorage.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,18 @@ for RelStorage command-line utilities:
settings are provided. Used with `zodbconvert` to import data from a
filestorage.

:::{tip}
For migrating data between storage backends, the generic
[zodb-convert](https://pypi.org/project/zodb-convert/) tool is now the
recommended approach. It works with all storage backends and can read storage
configuration directly from `zope.conf` files.
See {doc}`migrate-storage` for a step-by-step guide.
:::

## Next steps

- {doc}`migrate-storage` -- Migrate data between any storage backends using
`zodb-convert`.
- {doc}`/reference/database-relstorage` -- Full reference for all RelStorage
options including caching, replication, MySQL, Oracle, and SQLite settings.
- {doc}`/explanation/storage-backends` -- Comparison of all storage backends.
5 changes: 5 additions & 0 deletions docs/sources/how-to/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ Goal-oriented guides for common tasks.
- {doc}`configure-zeo`
- {doc}`configure-z3blobs`

## Storage Migration

- {doc}`migrate-storage`

## Web Layer

- {doc}`configure-cors`
Expand All @@ -31,6 +35,7 @@ configure-relstorage
configure-pgjsonb
configure-zeo
configure-z3blobs
migrate-storage
configure-cors
use-environment-variables
upgrade-v1-to-v2
Expand Down
140 changes: 140 additions & 0 deletions docs/sources/how-to/migrate-storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
# Migrate Data Between Storage Backends

<!-- diataxis: how-to -->

This guide shows how to copy ZODB data from one storage backend to another
using the generic [zodb-convert](https://pypi.org/project/zodb-convert/) tool.

## Prerequisites

- The `zodb-convert` package installed in your virtual environment
- The storage-specific packages for both source and destination installed
(e.g. `ZODB` for filestorage, `RelStorage` for relstorage, `zodb-pgjsonb`
for pgjsonb)

## Approach 1: Generated configuration files

When your instance uses a non-filestorage backend (RelStorage, PGJsonb, or
ZEO), the cookiecutter generates `convert-import.conf` and
`convert-export.conf` files in the `etc/` directory. These are ready-made
configuration files for `zodb-convert` that convert between your configured
backend and a FileStorage.

### Step 1: Set the conversion paths

In your `instance.yaml`, provide the FileStorage paths for the "other side"
of the conversion:

```yaml
default_context:
db_storage: pgjsonb
db_pgjsonb_dsn: "dbname='zodb' user='zodb' host='localhost' port='5433'"

# Conversion settings (FileStorage side)
db_convert_import_filestorage_location: var/import/Data.fs
db_convert_import_blobs_location: var/import/blobs
db_convert_export_filestorage_location: var/export/Data.fs
db_convert_export_blobs_location: var/export/blobs
```

These settings work with all non-filestorage backends.

### Step 2: Import data from FileStorage

Place your `Data.fs` and blobs at the import locations, then run:

```bash
zodb-convert etc/convert-import.conf
```

### Step 3: Export data to FileStorage

To create a portable FileStorage backup:

```bash
zodb-convert etc/convert-export.conf
```

## Approach 2: Using existing zope.conf files

If you have two Zope instances with different storage backends, `zodb-convert`
can read storage configuration directly from their `zope.conf` files. No
extra configuration files are needed.

### Step 1: Generate the destination instance

Create a second instance configured for the new storage backend:

```yaml
# instance-new.yaml
default_context:
db_storage: pgjsonb
db_pgjsonb_dsn: "dbname='zodb' user='zodb' host='localhost' port='5433'"
```

Run cookiecutter to generate:

```bash
cookiecutter -f --no-input --config-file instance-new.yaml \
gh:plone/cookiecutter-zope-instance
```

### Step 2: Run the conversion

```bash
zodb-convert \
--source-zope-conf /old-instance/etc/zope.conf \
--dest-zope-conf /new-instance/etc/zope.conf
```

### Specifying a named database

If your `zope.conf` defines multiple `<zodb_db>` sections:

```bash
zodb-convert \
--source-zope-conf old/etc/zope.conf --source-db main \
--dest-zope-conf new/etc/zope.conf --dest-db main
```

The default database name is `main`.

## Useful options

- `--dry-run` -- preview what would be copied without writing data
- `--incremental` -- resume from the last transaction in the destination
(useful for large databases or after interruptions)
- `-v` -- show INFO-level progress; `-vv` for DEBUG-level detail

## Common migration scenarios

| From | To | Notes |
|---|---|---|
| direct (filestorage) | relstorage | Classic scale-out migration |
| direct (filestorage) | pgjsonb | Move to SQL-queryable storage |
| relstorage | pgjsonb | Switch to JSONB representation |
| zeo | relstorage | Remove ZEO server dependency |
| any | direct (filestorage) | Create a portable backup |

All combinations work -- `zodb-convert` is storage-agnostic.

## Configuration reference

| Setting | Default | Description |
|---|---|---|
| `db_convert_import_filestorage_location` | *(unset)* | Path to the source FileStorage for import |
| `db_convert_import_blobs_location` | *(unset)* | Path to the source blob directory for import |
| `db_convert_export_filestorage_location` | *(unset)* | Path for the destination FileStorage on export |
| `db_convert_export_blobs_location` | *(unset)* | Path for the destination blob directory on export |

The `convert-import.conf` file is generated when both `db_convert_import_*`
settings are provided. The `convert-export.conf` file is generated when both
`db_convert_export_*` settings are provided. Neither file is generated for
the `direct` (filestorage) backend since it *is* the portable format.

## Next steps

- [zodb-convert on PyPI](https://pypi.org/project/zodb-convert/) -- Full
documentation and source code
- {doc}`/explanation/storage-backends` -- Understanding the trade-offs
between backends
14 changes: 14 additions & 0 deletions docs/sources/reference/database-relstorage.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,20 @@ replications. For details read
RelStorage provides helper scripts for packing (`zodbpack`) and
import/export from filestorage (`zodbconvert`).

:::{deprecated} 2.5
The RelStorage-specific `zodbconvert` import/export configuration files
(`relstorage-import.conf`, `relstorage-export.conf`) and their corresponding
`db_relstorage_import_*` / `db_relstorage_export_*` settings are deprecated
in favor of the generic
[zodb-convert](https://pypi.org/project/zodb-convert/) tool, which works
with any ZODB storage backend. Use the new `db_convert_*` settings and the
generated `convert-import.conf` / `convert-export.conf` files instead.
See {doc}`/how-to/migrate-storage` for details.

The existing settings and generated files continue to work but may be
removed in a future major version.
:::

The file `relstorage-pack.conf` for the command line utility `zodbpack` is
always generated for all RelStorage configurations. For usage information
read [Packing Or Reference Checking A ZODB Storage: zodbpack](https://relstorage.readthedocs.io/en/latest/zodbpack.html).
Expand Down
27 changes: 27 additions & 0 deletions hooks/post_gen_project.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,29 @@
if db_relstorage_export_filestorage_location:
filepath = Path(db_relstorage_export_filestorage_location)
filepath.parent.mkdir(parents=True, exist_ok=True)
if "{{ cookiecutter.db_storage }}" != "direct":
# generic convert import directories
db_convert_import_blobs = (
"{{ cookiecutter.db_convert_import_blobs_location | abspath }}"
)
if db_convert_import_blobs:
Path(db_convert_import_blobs).mkdir(parents=True, exist_ok=True)
db_convert_import_fs = (
"{{ cookiecutter.db_convert_import_filestorage_location | abspath }}"
)
if db_convert_import_fs:
Path(db_convert_import_fs).parent.mkdir(parents=True, exist_ok=True)
# generic convert export directories
db_convert_export_blobs = (
"{{ cookiecutter.db_convert_export_blobs_location | abspath }}"
)
if db_convert_export_blobs:
Path(db_convert_export_blobs).mkdir(parents=True, exist_ok=True)
db_convert_export_fs = (
"{{ cookiecutter.db_convert_export_filestorage_location | abspath }}"
)
if db_convert_export_fs:
Path(db_convert_export_fs).parent.mkdir(parents=True, exist_ok=True)
if "{{ cookiecutter.db_storage }}" == "pgjsonb":
blob_temp_dir = "{{ cookiecutter.db_pgjsonb_blob_temp_dir }}"
if blob_temp_dir:
Expand All @@ -85,6 +108,10 @@
z3blobs_cache_dir = "{{ cookiecutter.db_z3blobs_cache_dir }}"
if z3blobs_cache_dir:
Path(z3blobs_cache_dir).mkdir(parents=True, exist_ok=True)
if "{{ cookiecutter.db_storage }}" == "direct":
# cleanup convert files — filestorage doesn't need conversion configs
(etc / "convert-import.conf").unlink()
(etc / "convert-export.conf").unlink()
if "{{ cookiecutter.db_storage }}" != "relstorage":
# cleanup relstorage files if no relstorage is configured
(etc / "relstorage-export.conf").unlink()
Expand Down
14 changes: 8 additions & 6 deletions templates/db.conf
Original file line number Diff line number Diff line change
Expand Up @@ -192,9 +192,10 @@
drop_cache_rather_verify,
client_label,
storage,
wait
wait,
tagmarker=""
)-%}
<zeoclient>
<zeoclient{{ tagmarker }}>
{%- if not z3blobs_active %}
# blobs
shared-blob-dir {{ "true" if blob_mode == "shared" else "false" }}
Expand Down Expand Up @@ -248,7 +249,7 @@
{%- if wait %}
wait {{ wait }}
{%- endif %}
</zeoclient>
</zeoclient{{ tagmarker }}>
{%- endmacro %}

# =============================================================================
Expand All @@ -271,10 +272,11 @@
s3_secret_key,
s3_use_ssl,
blob_cache_dir,
blob_cache_size
blob_cache_size,
tagmarker=""
)-%}
%import zodb_pgjsonb
<pgjsonb>
<pgjsonb{{ tagmarker }}>
dsn {{ dsn }}
{%- if name and name != "pgjsonb" %}
name {{ name }}
Expand Down Expand Up @@ -326,7 +328,7 @@
blob-cache-size {{ blob_cache_size }}
{%- endif %}
{%- endif %}
</pgjsonb>
</pgjsonb{{ tagmarker }}>
{%- endmacro %}

# =============================================================================
Expand Down
Loading