feat: Index Migrator (#583)

# feat: Index Migrator (Pre-release)

Zero-downtime, crash-safe index migration for RedisVL. Plan, apply, and
rollback schema changes, including vector quantization, field renames,
prefix changes, and algorithm swaps, through a single CLI or
programmatic API.

## Summary

This PR adds a complete index migration system to RedisVL, enabling
users to evolve their index schemas without data loss. The migrator
handles the full lifecycle: **plan → review → apply → validate →
rollback**.

### Key Capabilities

| Category | Operations |
|----------|-----------|
| **Index-only** | Change algorithm (FLAT ↔ HNSW ↔ SVS-VAMANA), distance
metric, HNSW params (M, EF_CONSTRUCTION), make fields sortable |
| **Schema + Data** | Add/remove fields, rename fields, rename index,
change key prefix, change field options (separator, stemming) |
| **Vector Quantization** | float32 → float16, bfloat16, int8, uint8
with automatic re-encoding and crash-safe backup |

### Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    CLI: rvl migrate                          │
│  wizard │ plan │ apply │ rollback │ estimate │ validate     │
├─────────┴──────┴───────┴──────────┴──────────┴──────────────┤
│                  MigrationPlanner                           │
│  Schema diffing, change classification, plan generation     │
├─────────────────────────────────────────────────────────────┤
│            MigrationExecutor (sync + async)                 │
│  enumerate → dump → drop → key-renames → quantize →        │
│  create → index → validate                                  │
├─────────────────────────────────────────────────────────────┤
│  VectorBackup    │  Quantize Pipeline  │  Dtype Helpers     │
│  Crash-safe dump │  Batched R/W/Convert│  Width / detection │
└──────────────────┴─────────────────────┴────────────────────┘
```

## Usage

### CLI: Interactive Wizard

```bash
# List available indexes
rvl index listall --url redis://localhost:6379

# Interactive wizard: walks through changes step by step
rvl migrate wizard --url redis://localhost:6379

# Generate a plan from a target schema YAML
rvl migrate plan --index my_index --target-schema target_schema.yaml --url redis://localhost:6379

# Apply with crash-safe backup and multi-worker quantization
rvl migrate apply --plan migration_plan.yaml \
  --backup-dir /tmp/migration_backup \
  --workers 4 --batch-size 1000 \
  --url redis://localhost:6379

# Estimate disk space (dry-run, no mutations)
rvl migrate estimate --plan migration_plan.yaml --url redis://localhost:6379

# Rollback if needed
rvl migrate rollback --backup-dir /tmp/migration_backup \
  --index my_index --url redis://localhost:6379

# Validate post-migration
rvl migrate validate --plan migration_plan.yaml --url redis://localhost:6379
```

### CLI: Batch Migration (Multiple Indexes)

```bash
# Plan: shared schema-patch applied to indexes matched by pattern,
# explicit --indexes list, or --indexes-file
rvl migrate batch-plan --schema-patch shared_patch.yaml \
  --pattern '*_idx' --url redis://localhost:6379

rvl migrate batch-apply --plan batch_plan.yaml \
  --state batch_state.yaml --url redis://localhost:6379

rvl migrate batch-status --state batch_state.yaml
```

### Programmatic API

```python
from redisvl.migration import MigrationPlanner, MigrationExecutor

# Plan (provide either schema_patch_path or target_schema_path)
planner = MigrationPlanner()
plan = planner.create_plan(
    index_name="my_index",
    target_schema_path="target_schema.yaml",
    redis_url="redis://localhost:6379",
)

# Apply
executor = MigrationExecutor()
report = executor.apply(
    plan,
    redis_url="redis://localhost:6379",
    backup_dir="/tmp/migration_backup",
    num_workers=4,
    batch_size=1000,
)

print(f"Result: {report.result}")  # "succeeded"
print(f"Duration: {report.timings.total_migration_duration_seconds}s")
```

### Async API

```python
from redisvl.migration import AsyncMigrationExecutor

executor = AsyncMigrationExecutor()
report = await executor.apply(
    plan,
    redis_url="redis://localhost:6379",
    backup_dir="/tmp/migration_backup",
    num_workers=4,
)
```

## Crash Safety & Resume

Quantization migrations always write a vector backup to disk before
mutating data. When `--backup-dir` is omitted, the executor
auto-defaults to `./migration_backups`.

1. **Before drop**: Original vectors are dumped to a binary backup file
on disk
2. **On crash**: Re-running the same command detects the backup and
resumes from the last completed batch
3. **Rollback**: `rvl migrate rollback` restores original vectors from
the backup at any time

The backup file tracks phase (dump → ready → active → completed) and
batch progress, so resume skips already-completed work. Backups are
retained on disk after success for audit/rollback; cleanup is a manual
step.

## Performance

- **Pipelined reads/writes**: Batch HGET/HSET operations (configurable
`--batch-size`)
- **Multi-worker quantization**: `--workers N` parallelizes vector
re-encoding via ThreadPoolExecutor (sync) or asyncio.gather (async)
- **Redis Cluster support**: Batched DUMP/RESTORE/DEL for cross-slot key
renames (100 keys/pipeline)
- **Disk space estimation**: `rvl migrate estimate` calculates RDB + AOF
impact before any mutations

## What's Blocked

| Change | Why | Workaround |
|--------|-----|------------|
| Change vector dimensions | Requires re-embedding | Re-embed with new
model, reload data |
| Change storage type (hash ↔ JSON) | Different data format | Export,
transform, reload |
| Add a new vector field | Requires vectors for all docs | Add vectors
first, then migrate |

## New Files

### `redisvl/migration/` (core module)
| File | Description |
|------|-------------|
| `models.py` | Data models: `MigrationPlan`, `MigrationReport`,
`MigrationTimings`, etc. |
| `planner.py` | Schema diffing, change classification, plan generation
|
| `executor.py` | Sync migration executor, full apply lifecycle |
| `async_executor.py` | Async migration executor |
| `async_planner.py` | Async planner |
| `validation.py` / `async_validation.py` | Pre/post-migration
validation |
| `backup.py` | `VectorBackup`: crash-safe binary backup format |
| `quantize.py` | Pipelined vector quantization + multi-worker
orchestration |
| `reliability.py` | Dtype helpers: width comparison, vector-dtype
detection, idempotent-quantize check |
| `wizard.py` | Interactive migration wizard |
| `batch_planner.py` / `batch_executor.py` | Multi-index batch migration
|
| `utils.py` | Shared utilities (disk estimation, key enumeration, etc.)
|

### `redisvl/cli/migrate.py`
CLI with 11 subcommands: `helper`, `wizard`, `plan`, `apply`,
`estimate`, `rollback`, `validate`, `batch-plan`, `batch-apply`,
`batch-resume`, `batch-status`

## Tests

| Suite | Tests | Description |
|-------|-------|-------------|
| `test_migration_planner.py` | 23 | Schema diffing, change
classification |
| `test_migration_wizard.py` | 45 | Interactive wizard, adversarial
inputs |
| `test_vector_backup.py` | 26 | Backup
create/load/resume/rollback/cleanup |
| `test_pipeline_quantize.py` | 16 | Pipelined read/write/convert |
| `test_executor_backup_quantize.py` | 13 | Executor backup integration
|
| `test_multi_worker_quantize.py` | 15 | Multi-worker, resume, dtype
scaling |
| `test_async_migration_executor.py` | 35 | Async executor |
| `test_async_migration_planner.py` | 4 | Async planner |
| `test_batch_migration.py` | 42 | Batch planner/executor, overlap
detection |
| Integration tests | 5 files | Full end-to-end with live Redis |

**Total: 219 unit tests collected, all pre-commit checks clean.**

## Review Notes

- This is a **pre-release**: API surface is stable but may evolve based
on feedback
- 8 rounds of automated code review (nkode-review) have been applied,
addressing correctness, security, performance, and backward
compatibility findings
- The branch includes removal of the MCP module (previously merged
separately). Those deletions are unrelated to the migrator
- Documentation is in `docs/user_guide/how_to_guides/migrate-indexes.md`
and `docs/concepts/index-migrations.md`

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **High Risk**
> Introduces a large new migration subsystem that can drop/recreate
indexes, rename keys/fields, and re-encode vector data; mistakes or edge
cases can cause downtime or irreversible data changes despite
backups/resume safeguards.
>
> **Overview**
> Adds a new **experimental index migrator** exposed via `rvl migrate`
(wizard/plan/apply/estimate/rollback/validate plus
batch-plan/apply/resume/status) and a new `redisvl.migration` package
exporting sync/async planners, executors, validators, batch tooling, and
models.
>
> Implements an async migration executor that supports **drop/recreate
migrations** with key enumeration (FT.AGGREGATE w/ SCAN fallback), field
renames (hash/JSON), prefix renames (standalone and cluster-safe), and
vector datatype changes; **quantization now enforces mandatory on-disk
backups** (auto-defaulting to `./migration_backups`) to enable
crash-safe resume and rollback.
>
> Expands documentation substantially: adds a CLI reference page, new
migration concept/how-to guides, and updates existing docs/notebooks to
include migrate commands; also updates `.gitignore`, `CLAUDE.md`, and
CLI connection option helpers.
>

Author: nkanu17

.gitignore

@@ -237,3 +237,4 @@ tests/data
 
 # Local working directory (personal scripts, docs, tools)
 local/
+local_docs/

CLAUDE.md

@@ -48,6 +48,9 @@ index = SearchIndex(schema, redis_url="redis://localhost:6379")
   token.strip().strip(",").replace(""", "").replace(""", "").lower()
   ```
 
+### Protected Directories
+**CRITICAL**: NEVER delete the `local_docs/` directory or any files within it.
+
 ### Git Operations
 **CRITICAL**: NEVER use `git push` or attempt to push to remote repositories. The user will handle all git push operations.