Skip to content

Commit f99c2f4

Browse files
committed
docs(migration): explain mandatory backups in concept guide + lock format
- docs/concepts/index-migrations.md: add a 'Vector backups (mandatory for quantization)' section that states backups are unconditional for any quantization step, documents the default location (./migration_backups), the on-disk layout, what backups enable (crash-safe resume + manual rollback), and the new manual retention rule. Add a short 'Overlapping indexes' section that mirrors the troubleshooting entry in the how-to guide so the concept is reachable from the conceptual docs as well. - tests/unit/test_batch_migration.py: add test_overlap_error_matches_documented_format which calls BatchMigrationPlanner._format_overlap_error directly and asserts the exact substrings that the user-facing docs reproduce, so the docs cannot drift from the implementation without the test failing.
1 parent 6ac3aa1 commit f99c2f4

2 files changed

Lines changed: 86 additions & 0 deletions

File tree

docs/concepts/index-migrations.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,74 @@ Typical reductions:
103103

104104
Quantization time is proportional to document count. Plan for downtime accordingly.
105105

106+
## Vector backups (mandatory for quantization)
107+
108+
Quantization mutates the raw bytes of every vector in place. If the
109+
migration is interrupted partway through, or if the converted bytes turn
110+
out to be unacceptable for your application, there is no way to recover
111+
the original precision from the quantized values. To make these
112+
migrations safe to run, the migrator **always writes a vector backup
113+
before mutating any data** when a quantization step is needed.
114+
115+
There is no opt-out. The previous `--keep-backup` flag and any code path
116+
that allowed quantizing without a backup have been removed.
117+
118+
### Where backups are written
119+
120+
Pass `--backup-dir <dir>` (CLI) or `backup_dir="<dir>"` (Python API) to
121+
choose the location. If you do not supply one, the migrator auto-defaults
122+
to `./migration_backups` and logs the chosen path. Passing an empty
123+
string is treated as an explicit refusal of a backup and raises a
124+
`ValueError` before any data is touched.
125+
126+
Each migrated index produces two files:
127+
128+
```
129+
<backup-dir>/
130+
migration_backup_<index_name>.header # JSON: phase, progress counters, field metadata
131+
migration_backup_<index_name>.data # Binary: length-prefixed batches of original vectors
132+
```
133+
134+
Disk usage is roughly `num_docs × dims × bytes_per_element`. For 1M
135+
documents with 768-dimensional float32 vectors that is approximately
136+
2.9 GB.
137+
138+
### What backups enable
139+
140+
1. **Crash-safe resume.** If the executor dies mid-migration (process
141+
killed, network drop, OOM), re-running the same command with the same
142+
`--backup-dir` reads the header file, detects partial progress, and
143+
resumes from the last completed batch instead of re-quantizing the
144+
keys that already converted successfully.
145+
2. **Manual rollback.** The data file contains the original
146+
pre-quantization vector bytes. After a migration, you can use the
147+
rollback CLI (`rvl migrate rollback`) or the Python API to restore
148+
those bytes if you need to back out the change.
149+
150+
### Retention
151+
152+
Backup files are **retained on disk** after a successful migration.
153+
Cleanup is now a deliberate operator action, performed only after the
154+
new vectors have been verified and rollback is no longer needed. Delete
155+
the backup directory manually when you are done.
156+
157+
## Overlapping indexes
158+
159+
Two RediSearch indexes whose key prefixes overlap (one prefix is a
160+
literal string-prefix of the other, matching `FT.CREATE PREFIX`
161+
semantics) cover the same Redis keyspace. Running a batch quantization
162+
migration over them re-reads vectors that an earlier index in the batch
163+
has already quantized, producing garbage bytes. To prevent this,
164+
`batch-plan` performs an overlap check across every applicable index and
165+
**refuses to write a plan** if any pair conflicts. The error names the
166+
conflicting indexes and the specific prefix pairs that overlap.
167+
168+
The check is plan-time only — no data is mutated when a batch is
169+
refused. Resolve by splitting the indexes into prefix-disjoint groups
170+
and creating one batch plan per group. Indexes that are skipped for
171+
other reasons (e.g. `applicable: false` because a field is missing) do
172+
not participate in the check.
173+
106174
## Why some changes are blocked
107175

108176
### Vector dimension changes

tests/unit/test_batch_migration.py

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1585,3 +1585,21 @@ def test_empty_prefix_overlaps_everything(self, monkeypatch, tmp_path):
15851585
schema_patch_path=str(patch_path),
15861586
redis_client=mock_client,
15871587
)
1588+
1589+
def test_overlap_error_matches_documented_format(self):
1590+
"""Guard against drift between the error string and the docs.
1591+
1592+
The user-facing docs (docs/user_guide/how_to_guides/migrate-indexes.md
1593+
troubleshooting section) reproduce this error verbatim, so changes to
1594+
the format should be intentional.
1595+
"""
1596+
msg = BatchMigrationPlanner._format_overlap_error(
1597+
[("products_main", "products_premium", [("product:", "product:premium:")])]
1598+
)
1599+
assert "Refusing to create batch plan: overlapping indexes detected." in msg
1600+
assert "Conflicts:" in msg
1601+
assert (
1602+
"products_main <-> products_premium: 'product:' <-> 'product:premium:'"
1603+
in msg
1604+
)
1605+
assert "disjoint prefixes" in msg

0 commit comments

Comments
 (0)