You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- 📄 **[One-Click HTML Report](https://coding-kitties.github.io/investing-algorithm-framework/Getting%20Started/backtest-reports)** — Self-contained file, no server, dark & light theme, shareable
86
86
- 📦 **[Custom `.iafbt` Backtest Bundle Format](https://coding-kitties.github.io/investing-algorithm-framework/Data/backtest_data)** — An explicit, versioned, compressed, language-portable container (zstd + msgpack with magic-byte header) plus a separate parquet index for fast filtering without loading. ~21× smaller and ~27× fewer files than standard filebased directory layouts, with parallel I/O for fast load/save of large amounts of backtests.
87
+
- 🗄️ **[Tiered Backtest Storage Layer](examples/storage_layer_demo/README.md)** — Manage thousands of `.iafbt` bundles with a Tier-1 SQLite index (sub-100 ms ranks/filters over 10k+ backtests), a swappable `BacktestStore` protocol (`LocalDirStore`, `LocalTieredStore`), content-addressed Tier-3 OHLCV deduplication, and a CLI (`iaf index` / `iaf list` / `iaf rank` / `iaf migrate-store`) that plugs straight into the HTML dashboard.
87
88
- 🌐 **[Load External Data](https://coding-kitties.github.io/investing-algorithm-framework/Data/external-data)** — Fetch CSV, JSON, or Parquet from any URL with caching and auto-refresh
88
89
- � **[Per-Market Deposit Schedules & Portfolio Sync](https://coding-kitties.github.io/investing-algorithm-framework/Advanced%20Concepts/portfolio-sync)** — Declare recurring or one-shot external cash flows on a market with `deposit_schedule=` / `auto_sync=True`. Backtests simulate the deposits; live mode reconciles with the broker — same `context.sync_portfolio()` API in both modes.
89
90
- �📝 **[Record Custom Variables](https://coding-kitties.github.io/investing-algorithm-framework/Advanced%20Concepts/recording-variables)** — Track any indicator or metric during backtests with `context.record()`
@@ -145,6 +146,63 @@ Every backtest produces a **self-contained HTML dashboard** — open it in any b
145
146
146
147
</details>
147
148
149
+
<detailsopen>
150
+
<summary>
151
+
<strong>Backtest Storage Layer — scale to thousands of backtests</strong>
152
+
</summary> <br>
153
+
154
+
Once you start sweeping parameter grids and walk-forward windows, a flat folder of `.iafbt` bundles stops scaling: every comparison re-decodes multi-MB Parquet metric blobs just to read a Sharpe number. The storage layer fixes that with three tiers behind a single `BacktestStore` protocol:
155
+
156
+
-**Tier-1 — SQLite index (`index.sqlite`)**: one row per bundle with every scalar from `BacktestSummaryMetrics` promoted to its own column. Ranking 10k+ bundles becomes a sub-100 ms SQL query — no `.iafbt` is opened.
157
+
-**Tier-2 — `BacktestStore` adapters**: `LocalDirStore` (flat folder of bundles) or `LocalTieredStore` (hive-partitioned layout). Same handle-based API, swap the implementation without touching call sites.
158
+
-**Tier-3 — content-addressed OHLCV chunks**: SHA-256 deduped per-symbol OHLCV blobs shared across every bundle that references them. `garbage_collect_ohlcv()` reclaims orphans.
159
+
160
+
A CLI ties it all together: `iaf index` builds/refreshes the Tier-1 SQLite, `iaf list` / `iaf rank` query it, and `iaf migrate-store` moves a whole collection between store kinds in one command.
161
+
162
+
#### Typical workflow
163
+
164
+
```python
165
+
from investing_algorithm_framework import BacktestReport
166
+
from investing_algorithm_framework.cli.index_command import (
167
+
build_index, rank_index,
168
+
)
169
+
from investing_algorithm_framework.services.backtest_store import (
170
+
LocalDirStore,
171
+
)
172
+
173
+
# 1. Build (or refresh) the Tier-1 SQLite index over a folder of .iafbt bundles.
174
+
build_index("./my-backtests/") # equivalent to: iaf index ./my-backtests/
175
+
176
+
# 2. Pick the top 20 by Sharpe straight from SQLite — no Parquet decoded.
177
+
top = rank_index(
178
+
"./my-backtests/",
179
+
by="sharpe_ratio",
180
+
where="summary_number_of_trades > 50",
181
+
limit=20,
182
+
)
183
+
184
+
# 3. Materialise just those 20 bundles through the BacktestStore protocol.
185
+
store = LocalDirStore("./my-backtests/")
186
+
backtests = [store.open(row["bundle_path"]) for row in top]
0 commit comments