Skip to content

Commit ecff7d1

Browse files
committed
docs: add How It Works section explaining wizard/plan/apply and batch flows
1 parent fe0cabe commit ecff7d1

1 file changed

Lines changed: 104 additions & 0 deletions

File tree

docs/user_guide/how_to_guides/migrate-indexes.md

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,110 @@ docker run -d --name redis -p 6379:6379 redis:8.0
4040

4141
**Note:** Redis 8.0+ is required for INT8/UINT8 vector datatypes. SVS-VAMANA algorithm requires Redis 8.2+ and Intel AVX-512 hardware.
4242

43+
44+
## How It Works
45+
46+
Every migration follows the same three-phase flow: **describe what changed** (the patch),
47+
**generate a plan** (diffing the patch against the live schema), and **execute the plan**.
48+
49+
### Single-Index Flow: wizard/plan then apply
50+
51+
```
52+
wizard (interactive) plan (non-interactive)
53+
| |
54+
v v
55+
SchemaPatch YAML <----or----> SchemaPatch YAML
56+
| |
57+
+------ planner.create_plan() -------+
58+
|
59+
v
60+
MigrationPlan YAML
61+
|
62+
v
63+
executor.apply()
64+
|
65+
v
66+
MigrationReport YAML
67+
```
68+
69+
**Phase 1: Build a SchemaPatch.**
70+
A patch is a small YAML file that declares *what you want to change*, not the full target schema.
71+
You can build it interactively with `rvl migrate wizard`, or write it by hand. The patch has
72+
five sections, each optional:
73+
74+
| Patch Section | What it does |
75+
|---|---|
76+
| `add_fields` | Adds new field definitions to the index |
77+
| `remove_fields` | Removes fields from the index (document data is kept, just no longer indexed) |
78+
| `rename_fields` | Renames fields in both the index schema and all documents (HGET old, HSET new, HDEL old) |
79+
| `update_fields` | Modifies field attributes: algorithm, datatype, distance metric, sortable, separator, etc. |
80+
| `index` | Changes the index name or key prefix |
81+
82+
**Phase 2: Generate a MigrationPlan.**
83+
The planner connects to Redis, snapshots the live index schema and stats,
84+
then merges the patch into the source schema to produce a `merged_target_schema`.
85+
It classifies every change as supported or blocked and extracts rename operations.
86+
87+
The plan YAML contains:
88+
- `source`: frozen snapshot of the live index at planning time (schema, stats, key sample, prefixes)
89+
- `requested_changes`: the patch that was applied
90+
- `merged_target_schema`: source + patch = what the index will look like after migration
91+
- `diff_classification`: whether the migration is supported and any blocked reasons
92+
- `rename_operations`: extracted index renames, prefix changes, and field renames
93+
- `warnings`: any important notes (downtime required, lossy quantization, etc.)
94+
95+
The same patch produces different plans per index because each index has a different source schema.
96+
97+
**Phase 3: Apply.**
98+
The executor reads the plan and runs the migration steps:
99+
100+
1. Enumerate keys (SCAN with source prefix)
101+
2. Field renames (pipelined HGET/HSET/HDEL)
102+
3. Dump original vectors to backup file (if quantizing and backup-dir provided)
103+
4. Drop index (FT.DROPINDEX, documents are preserved)
104+
5. Key prefix renames (RENAME or DUMP/RESTORE for cluster)
105+
6. Quantize vectors from backup (pipelined read/convert/write)
106+
7. Create index (FT.CREATE with merged target schema)
107+
8. Wait for re-indexing to complete
108+
9. Validate (doc count, schema match, key sample)
109+
110+
### Batch Flow: wizard/plan then batch-plan then batch-apply
111+
112+
For applying the same change across multiple indexes:
113+
114+
```
115+
SchemaPatch YAML (shared, written once)
116+
|
117+
v
118+
batch_planner.create_batch_plan()
119+
for each index:
120+
snapshot live schema
121+
merge patch into source
122+
if applicable: write per-index MigrationPlan
123+
if not: mark skip_reason
124+
|
125+
v
126+
BatchPlan YAML
127+
shared_patch: { ... }
128+
indexes:
129+
- name: idx_a, applicable: true, plan_path: plans/idx_a.yaml
130+
- name: idx_b, applicable: true, plan_path: plans/idx_b.yaml
131+
- name: idx_c, applicable: false, skip_reason: "field not found"
132+
|
133+
v
134+
batch_executor.apply()
135+
for each applicable index (sequentially):
136+
executor.apply(per_index_plan)
137+
```
138+
139+
The batch planner takes a **single shared patch** and tests it against every target index.
140+
Indexes where the patch doesn't apply (e.g., it references a field that doesn't exist in that
141+
index, or the change is blocked) are marked `applicable: false` with a `skip_reason` and skipped
142+
during apply. Each applicable index gets its own full `MigrationPlan` written to disk.
143+
144+
This means you can review each per-index plan individually before running `batch-apply`.
145+
146+
43147
## Step 1: Discover Available Indexes
44148

45149
```bash

0 commit comments

Comments
 (0)