feat(store): iaf migrate-store + dual-store contract suite (epic #540 phase 3d)

README.md

@@ -84,6 +84,7 @@ This framework is built around the full loop: **create strategies → vector bac
 - 📉 **[Benchmark Comparison](https://coding-kitties.github.io/investing-algorithm-framework/Getting%20Started/backtest-reports)** — Beat-rate analysis vs Buy & Hold, DCA, risk-free & custom benchmarks
 - 📄 **[One-Click HTML Report](https://coding-kitties.github.io/investing-algorithm-framework/Getting%20Started/backtest-reports)** — Self-contained file, no server, dark & light theme, shareable
 - 📦 **[Custom `.iafbt` Backtest Bundle Format](https://coding-kitties.github.io/investing-algorithm-framework/Data/backtest_data)** — An explicit, versioned, compressed, language-portable container (zstd + msgpack with magic-byte header) plus a separate parquet index for fast filtering without loading. ~21× smaller and ~27× fewer files than standard filebased directory layouts, with parallel I/O for fast load/save of large amounts of backtests.
+- 🗄️ **[Tiered Backtest Storage Layer](examples/storage_layer_demo/README.md)** — Manage thousands of `.iafbt` bundles with a Tier-1 SQLite index (sub-100 ms ranks/filters over 10k+ backtests), a swappable `BacktestStore` protocol (`LocalDirStore`, `LocalTieredStore`), content-addressed Tier-3 OHLCV deduplication, and a CLI (`iaf index` / `iaf list` / `iaf rank` / `iaf migrate-store`) that plugs straight into the HTML dashboard.
 - 🌐 **[Load External Data](https://coding-kitties.github.io/investing-algorithm-framework/Data/external-data)** — Fetch CSV, JSON, or Parquet from any URL with caching and auto-refresh
 - � **[Per-Market Deposit Schedules & Portfolio Sync](https://coding-kitties.github.io/investing-algorithm-framework/Advanced%20Concepts/portfolio-sync)** — Declare recurring or one-shot external cash flows on a market with `deposit_schedule=` / `auto_sync=True`. Backtests simulate the deposits; live mode reconciles with the broker — same `context.sync_portfolio()` API in both modes.
 - �📝 **[Record Custom Variables](https://coding-kitties.github.io/investing-algorithm-framework/Advanced%20Concepts/recording-variables)** — Track any indicator or metric during backtests with `context.record()`
@@ -141,10 +142,99 @@ Every backtest produces a **self-contained HTML dashboard** — open it in any b
 - **Built-in MCP server** — let Copilot, Claude, or any MCP-compatible agent query your backtests, rank strategies, and reason over trades through `investing-algorithm-framework mcp`
 - **Notes keeping** — annotate every backtest with hypotheses, observations and conclusions; notes travel with the report so your research is never lost
 
+#### From backtest results to a report
+
+Every backtest API — vector or event-driven — returns the same `Backtest` object, which the `BacktestReport` consumes directly. So whether you're iterating over an in-memory list or a folder of persisted `.iafbt` bundles, the path to the dashboard is the same:
+
+```python
+from investing_algorithm_framework import BacktestReport
+
+# --- Single event-driven backtest ---
+backtest = app.run_backtest(backtest_date_range=date_range)
+BacktestReport(backtests=[backtest]).save("event_report.html")
+
+# --- A sweep of vector backtests (parameter grid / multi-window) ---
+backtests = app.run_vector_backtests(
+    strategies=[StrategyA(), StrategyB(), StrategyC()],
+    backtest_date_ranges=[range_2022, range_2023, range_2024],
+    n_workers=-1,
+    backtest_storage_directory="./my-backtests/",  # persists .iafbt bundles
+    show_progress=True,
+)
+BacktestReport(backtests=backtests).save("sweep_report.html")
+
+# --- Or: load a folder of bundles back later (parallel decode) ---
+report = BacktestReport.open(
+    directory_path="./my-backtests/",
+    workers=-1,
+    show_progress=True,
+)
+report.save("from_disk_report.html")
+```
+
+For sweeps that grow into the thousands, combine this with the [Backtest Storage Layer](examples/storage_layer_demo/README.md) below — rank in SQLite first, then load only the winners into the report.
+
 → [Backtest dashboard docs](https://coding-kitties.github.io/investing-algorithm-framework/Getting%20Started/backtesting) · [MCP server docs](https://coding-kitties.github.io/investing-algorithm-framework/Advanced%20Concepts/mcp-server)
 
 </details>
 
+<details open>
+<summary>
+  <strong>Backtest Storage Layer — scale to thousands of backtests</strong>
+</summary> <br>
+
+Once you start sweeping parameter grids and walk-forward windows, a flat folder of `.iafbt` bundles stops scaling: every comparison re-decodes multi-MB Parquet metric blobs just to read a Sharpe number. The storage layer fixes that with three tiers behind a single `BacktestStore` protocol:
+
+- **Tier-1 — SQLite index (`index.sqlite`)**: one row per bundle with every scalar from `BacktestSummaryMetrics` promoted to its own column. Ranking 10k+ bundles becomes a sub-100 ms SQL query — no `.iafbt` is opened.
+- **Tier-2 — `BacktestStore` adapters**: `LocalDirStore` (flat folder of bundles) or `LocalTieredStore` (hive-partitioned layout). Same handle-based API, swap the implementation without touching call sites.
+- **Tier-3 — content-addressed OHLCV chunks**: SHA-256 deduped per-symbol OHLCV blobs shared across every bundle that references them. `garbage_collect_ohlcv()` reclaims orphans.
+
+A CLI ties it all together: `iaf index` builds/refreshes the Tier-1 SQLite, `iaf list` / `iaf rank` query it, and `iaf migrate-store` moves a whole collection between store kinds in one command.
+
+#### Typical workflow
+
+```python
+from investing_algorithm_framework import BacktestReport
+from investing_algorithm_framework.cli.index_command import (
+    build_index, rank_index,
+)
+from investing_algorithm_framework.services.backtest_store import (
+    LocalDirStore,
+)
+
+# 1. Build (or refresh) the Tier-1 SQLite index over a folder of .iafbt bundles.
+build_index("./my-backtests/")          # equivalent to: iaf index ./my-backtests/
+
+# 2. Pick the top 20 by Sharpe straight from SQLite — no Parquet decoded.
+top = rank_index(
+    "./my-backtests/",
+    by="sharpe_ratio",
+    where="summary_number_of_trades > 50",
+    limit=20,
+)
+
+# 3. Materialise just those 20 bundles through the BacktestStore protocol.
+store = LocalDirStore("./my-backtests/")
+backtests = [store.open(row["bundle_path"]) for row in top]
+
+# 4. Feed them straight into the HTML dashboard.
+BacktestReport(backtests=backtests).save("top20.html")
+```
+
+Or from the shell:
+
+```bash
+iaf index ./my-backtests/
+iaf rank  ./my-backtests/ --by sharpe_ratio --where "summary_number_of_trades > 50" -n 20
+iaf list  ./my-backtests/ --sort calmar_ratio --json
+iaf migrate-store --from local-dir --src ./my-backtests/ \
+                  --to   local-tiered --dst ./tiered/
+```
+
+→ End-to-end runnable example: [`examples/storage_layer_demo/`](examples/storage_layer_demo/README.md)
+
+</details>
+
 <details open>
 <summary>
   <strong>Live Trading</strong>

docusaurus/docs/Getting Started/backtest-reports.md

@@ -6,6 +6,10 @@ sidebar_position: 10
 
 The framework generates self-contained HTML dashboard reports for analyzing backtest results. Reports work for both single and multi-strategy backtests — no external dependencies required.
 
+:::tip Working with hundreds or thousands of backtests?
+A `BacktestReport` inlines every backtest into a single HTML file, which becomes too heavy for a browser past a few dozen backtests. Use the [Backtest Storage Layer](./backtest-storage.md) to filter your collection down (in SQLite, sub-100 ms) and render reports only over the winners.
+:::
+
 ## Quick Start
 
 ```python

docusaurus/docs/Getting Started/backtest-storage.md

@@ -0,0 +1,235 @@
+---
+sidebar_position: 11
+---
+
+# Backtest Storage Layer
+
+Once you start sweeping parameter grids and walk-forward windows, you quickly end up with **hundreds or thousands of backtests on disk**. A flat folder of `.iafbt` bundles works for tens of them, but it stops scaling once you want to compare them all in a single HTML dashboard — every comparison re-decodes multi-MB Parquet metric blobs just to read a Sharpe number, and the resulting `report.html` becomes too heavy for a browser to open.
+
+The **backtest storage layer** is the framework's answer to that. It separates *where bundles live* from *how you query them*, and it gives you the tools to keep your dashboards fast even when the backing collection grows into the thousands.
+
+## Mental model
+
+```
+                ┌─────────────────────────────────────────────┐
+                │  Tier-1: SQLite index   (index.sqlite)      │
+                │   - one row per .iafbt                      │
+                │   - all summary metrics promoted to columns │
+                │   - sub-100 ms ranks / filters over 10k+    │
+                └─────────────────────────────────────────────┘
+                                    │ derived from
+                                    ▼
+                ┌─────────────────────────────────────────────┐
+                │  Tier-2: Parquet sidecars (analytics-ready) │
+                │   - hive-partitioned on run_id              │
+                │   - portfolio_snapshots / trades / orders   │
+                └─────────────────────────────────────────────┘
+                                    │ derived from
+                                    ▼
+                ┌─────────────────────────────────────────────┐
+                │  CANONICAL: .iafbt bundles                  │
+                │   - the single source of truth              │
+                │   - everything else can be rebuilt from it  │
+                └─────────────────────────────────────────────┘
+                                    │ references
+                                    ▼
+                ┌─────────────────────────────────────────────┐
+                │  Tier-3: content-addressed OHLCV chunks     │
+                │   - <sha256>.parquet, deduped across all    │
+                │     bundles that reference the same data    │
+                └─────────────────────────────────────────────┘
+```
+
+The `.iafbt` bundle is **canonical**. The SQLite index, the Tier-2 Parquet sidecars and the Tier-3 OHLCV chunks are all *derived* — they can be rebuilt from the bundles at any time and they're best-effort: a malformed sidecar never blocks a write or read against the bundle.
+
+## The `BacktestStore` protocol
+
+Two concrete implementations ship today, both exposing the same API:
+
+| Store | Layout | Best for |
+|---|---|---|
+| `LocalDirStore` | flat folder of `.iafbt` files (+ `index.sqlite`) | Most users, simple to inspect, fast `ls` |
+| `LocalTieredStore` | full Tier-1/2/3 layout | Large collections, OHLCV dedup, analytics workflows |
+
+Both are drop-in interchangeable — swap the implementation without touching call sites:
+
+```python
+from investing_algorithm_framework.services.backtest_store import (
+    LocalDirStore,
+)
+from investing_algorithm_framework.services.backtest_store.\
+local_tiered_store import LocalTieredStore
+
+store = LocalDirStore("./my-backtests/")
+# store = LocalTieredStore("./my-backtests/")  # same API
+
+len(store)                       # how many bundles?
+"momentum_v1.iafbt" in store     # exists?
+bt = store.open("momentum_v1.iafbt")
+for handle in store.iter_handles():
+    ...
+```
+
+## The normal developer workflow
+
+Below is the canonical loop most users will run. The **same five steps** hold whether you have 10 backtests or 10,000 — you just lean harder on the index as the collection grows.
+
+### 1. Run a sweep, persist the bundles
+
+```python
+backtests = app.run_vector_backtests(
+    strategies=[StrategyA(), StrategyB(), StrategyC()],
+    backtest_date_ranges=[range_2022, range_2023, range_2024],
+    n_workers=-1,
+    backtest_storage_directory="./my-backtests/",   # writes .iafbt here
+    show_progress=True,
+)
+```
+
+After this you have a folder of `.iafbt` bundles on disk. That folder is *the* artifact — everything downstream operates on it.
+
+### 2. Build the Tier-1 index
+
+```bash
+iaf index ./my-backtests/
+```
+
+Or from Python:
+
+```python
+from investing_algorithm_framework.cli.index_command import build_index
+build_index("./my-backtests/")
+```
+
+This walks the folder once, writes `index.sqlite` with every scalar from `BacktestSummaryMetrics` promoted to its own column, and is **idempotent** — re-run it any time after adding new bundles.
+
+### 3. Filter / rank in SQLite (no bundles opened)
+
+The point of the index is that **you never need to decode a Parquet metric blob just to choose which backtests are interesting**. Pick winners with a SQL `WHERE` clause:
+
+```python
+from investing_algorithm_framework.cli.index_command import (
+    list_index, rank_index,
+)
+
+# Top 20 by Sharpe, but only among bundles with > 50 trades.
+top = rank_index(
+    "./my-backtests/",
+    by="sharpe_ratio",
+    where="summary_number_of_trades > 50",
+    limit=20,
+)
+
+for r in top:
+    print(r["algorithm_id"], r["summary_sharpe_ratio"])
+```
+
+Or from the shell:
+
+```bash
+iaf rank ./my-backtests/ \
+    --by sharpe_ratio \
+    --where "summary_number_of_trades > 50" -n 20
+iaf list ./my-backtests/ --sort calmar_ratio --json
+```
+
+This step is **sub-100 ms** even over 10k+ bundles. No Parquet, no decompression, no bundle opens.
+
+### 4. Materialise only the bundles you actually need
+
+```python
+store = LocalDirStore("./my-backtests/")
+backtests = [store.open(row["bundle_path"]) for row in top]
+```
+
+`bundle_path` from the index row is exactly the store handle, so this is a one-liner. **You only pay the bundle-decode cost for the bundles you selected**, not the whole collection.
+
+### 5. Render the report
+
+```python
+from investing_algorithm_framework import BacktestReport
+
+BacktestReport(backtests=backtests).save("top20.html")
+```
+
+That's the whole loop.
+
+## Avoid overloading your `report.html`
+
+The `BacktestReport` produces a **self-contained** HTML file: every backtest's full per-run data (equity curve, drawdown series, trades, positions, monthly returns) is inlined into the document so the dashboard works offline with no server.
+
+The trade-off: file size grows linearly with the number of backtests inlined. Rough orders of magnitude:
+
+| Backtests in report | Approx. HTML size | Browser experience |
+|---|---|---|
+| 1 – 10 | tens of KB to ~1 MB | instant |
+| 10 – 50 | a few MB | smooth |
+| 50 – 200 | 10 – 50 MB | slower load, still usable |
+| 200+ | 100 MB+ | browsers struggle / refuse to open |
+
+The point of the storage layer is that **you don't need to put 200 backtests in one report to compare them**. The Tier-1 index is your comparison surface for the full collection; the HTML report is your deep-dive surface for a small, hand-picked subset.
+
+### Anti-pattern
+
+```python
+# DON'T do this with thousands of bundles.
+report = BacktestReport.open(directory_path="./my-backtests/", workers=-1)
+report.save("everything.html")     # multi-hundred-MB file, browser dies
+```
+
+This decodes every bundle in the folder and inlines all of them. Fine for a few dozen; fatal at scale.
+
+### The right pattern
+
+```python
+# Filter in SQLite first, then render only the winners.
+top = rank_index("./my-backtests/", by="sharpe_ratio", limit=25)
+store = LocalDirStore("./my-backtests/")
+BacktestReport(
+    backtests=[store.open(r["bundle_path"]) for r in top],
+).save("top25_by_sharpe.html")
+```
+
+Same principle applies for slicing by anything else — most-trades, best-Calmar, lowest-drawdown, only-2024-windows, only-momentum-strategies, etc. Compose multiple narrow reports rather than one giant one.
+
+### Rules of thumb
+
+- **Keep any single `report.html` to ≤ 50 backtests.** Past that, render multiple narrower reports (one per strategy family, one per regime, one for the top-N) instead of one mega-report.
+- **Use the index as your comparison plane** for the full collection. CLI: `iaf list` / `iaf rank`. Python: `list_index` / `rank_index`. SQL: `sqlite3 index.sqlite` for anything ad-hoc.
+- **Render for the audience.** A "winners" report (top 10–20) is what you actually send to teammates. A "full deep-dive" report on one strategy is what you keep for yourself.
+- **Don't trust `BacktestReport.open(directory_path=…)` at scale.** It walks and decodes the whole folder; it's a convenience for ≤ 50-bundle directories, not a scaling story.
+
+## When to use which store
+
+- **`LocalDirStore`** — start here. A flat folder of `.iafbt` files is what every other tool understands (you can `ls`, `rsync`, `tar`, `git lfs` it). Tier-1 SQLite gets built next to the bundles. This is the default for `app.run_vector_backtests(backtest_storage_directory=...)`.
+
+- **`LocalTieredStore`** — switch to this when you need any of:
+  - **Cross-bundle analytics** without decoding bundles (DuckDB / Polars over the Tier-2 Parquet sidecars: `read_parquet('store/parquet/trades/**/*.parquet', hive_partitioning=True)`).
+  - **OHLCV deduplication** — every bundle that references the same `BTC/EUR:1h` data shares one `<sha256>.parquet` blob on disk; reclaim orphans with `store.garbage_collect_ohlcv()`.
+  - **Migration target** for archival / production pipelines.
+
+Move a whole collection between store kinds with a single command:
+
+```bash
+iaf migrate-store --from local-dir    --src ./my-backtests/ \
+                  --to   local-tiered --dst ./tiered/
+```
+
+## End-to-end runnable example
+
+A complete worked example (seed bundles → build index → rank → load winners → render dashboard) lives in the repo at [`examples/storage_layer_demo/`](https://github.com/coding-kitties/investing-algorithm-framework/tree/main/examples/storage_layer_demo). Run it from a checkout:
+
+```bash
+source .venv/bin/activate
+python examples/storage_layer_demo/demo.py
+```
+
+It prints each step, leaves the bundles + index + dashboard in a temp directory, and shows you the exact `iaf` CLI commands you could run by hand against the same data.
+
+## Reference
+
+- CLI: `iaf index`, `iaf list`, `iaf rank`, `iaf migrate-store` (see `iaf <cmd> --help`)
+- Python: `investing_algorithm_framework.cli.index_command.{build_index, list_index, rank_index}`
+- Stores: `investing_algorithm_framework.services.backtest_store.{LocalDirStore, LocalTieredStore}`
+- Bundle format: see [Backtest Data](../Data/backtest_data.md)
+- Report API: see [Backtest Reports](./backtest-reports.md)

docusaurus/sidebars.js

@@ -57,6 +57,10 @@ const sidebars = {
                     type: 'doc',
                     id: 'Getting Started/backtest-reports',
                 },
+                {
+                    type: 'doc',
+                    id: 'Getting Started/backtest-storage',
+                },
                 {
                     type: 'doc',
                     id: 'Getting Started/metrics',