Skip to content

feat(store): BacktestStore Protocol + LocalDirStore (epic #540 phase 3a)#544

Merged
MDUYN merged 1 commit into
feature/iaf-index-clifrom
feature/iaf-backtest-store
May 12, 2026
Merged

feat(store): BacktestStore Protocol + LocalDirStore (epic #540 phase 3a)#544
MDUYN merged 1 commit into
feature/iaf-index-clifrom
feature/iaf-backtest-store

Conversation

@MDUYN
Copy link
Copy Markdown
Collaborator

@MDUYN MDUYN commented May 11, 2026

Stacked on #543 (feature/iaf-index-cli), which is itself stacked on #537. Merge order: #537#543 → this PR. The diff shown here is just the single Phase 3a commit on top of #543.

First slice of Phase 3 of epic #540 — introduces the storage seam that decouples where a backtest is persisted from the rest of the framework.

What's in this PR

BacktestStore Protocol

A runtime-checkable Protocol with the minimum contract every backtest store must satisfy:

class BacktestStore(Protocol):
    def write(self, backtest, *, handle=None) -> StoreHandle: ...
    def open(self, handle, *, summary_only=False) -> Backtest: ...
    def exists(self, handle) -> bool: ...
    def delete(self, handle) -> None: ...
    def iter_handles(self) -> Iterator[StoreHandle]: ...
    def iter_index_rows(self) -> Iterator[BacktestIndexRow]: ...
    def __len__(self) -> int: ...

Semantics mirror today's Backtest.save_bundle / Backtest.open exactly — LocalDirStore is a 1:1 adapter, so existing consumers (HTML report, iaf list / rank, MCP server) work unchanged.

Optional capability mixins

Capabilities not every store can or should implement are declared as separate runtime-checkable Protocols, so callers feature-test with isinstance(store, SupportsXxx):

  • SupportsCopyFrom (this PR) — for iaf migrate-store (Phase 3d) and finterion push (closed-source). Default fallback is a per-handle write(src.open(h)) loop; tiered stores can override to dedup chunks during ingest.
  • SupportsRelations (Phase 3+, closed-source) — strategy/version/report graph; only FinterionStore will implement it.
  • SupportsContentAddressedChunks (Phase 3c) — Tier-3 dedup.

LocalDirStore

Thin adapter over today's .iafbt directory layout:

  • Handles are bundle paths relative to the store root (e.g. "sweep_a/run_03.iafbt"), so moving the root directory does not invalidate any handle.
  • .iafbt suffix is normalised — callers may omit it.
  • Path-traversal guards reject handles that escape the root (../escape, absolute paths).
  • Sidecar SqliteBacktestIndex (<root>/index.sqlite) is built lazily and incrementally — same machinery as iaf index — so iter_index_rows() does not re-decode bundles. Disable with use_index=False for one-shot scripts.
  • copy_from(src) loops src.iter_handles()write(src.open(h), handle=h).

Tests

19 new tests, all passing:

  • Protocol + SupportsCopyFrom runtime conformance.
  • write / open round-trip; summary_only honoured.
  • default-handle derivation from algorithm_id.
  • iter_handles / __len__ / __contains__.
  • iter_index_rows returns typed BacktestIndexRow with relative paths; sidecar SQLite is created on first call; use_index=False skips sidecar.
  • path-traversal rejection (../escape, absolute paths).
  • copy_from with and without a handle subset.
  • handle normalisation (suffix added).
  • root auto-creation; open of missing handle raises StoreHandleNotFoundError; delete of missing handle is a no-op.

Targeted suite (tests/services/backtest_store/ + tests/services/backtest_index/ + tests/cli/): 86 / 86 passing.

What's next (still in Phase 3)

Slice Scope
3b LocalTieredStore Tier-1 (SQLite) + Tier-2 (per-project Parquet for snapshots / trades / orders / metric_series)
3c Tier-3 content-addressed chunks (ohlcv, code, params, symbols) — where the 64 GB → 20 GB headline lives
3d iaf migrate-store --from local-dir --to local-tiered + byte-identical backtest.export() / Backtest.import_() round-trip + the parameterised test fixture that runs every backtest test against both stores

Each slice will land as a separate stacked PR.

Introduces the storage seam that decouples *where* a backtest is
persisted from the rest of the framework. Phase 3a is intentionally
scoped to the Protocol + a thin adapter over today's .iafbt layout;
LocalTieredStore (Tier-2 Parquet + Tier-3 chunks) and FinterionStore
land in follow-up PRs.

- BacktestStore Protocol: write / open / exists / delete / iter_handles
  / iter_index_rows / __len__ / __contains__. Mirrors today's
  Backtest.save_bundle / Backtest.open semantics so LocalDirStore is
  a 1:1 adapter.
- StoreHandle: opaque str token (relative bundle path for
  LocalDirStore; uuid7 run_id for the upcoming tiered stores).
- StoreError + StoreHandleNotFoundError.
- Optional capability mixin SupportsCopyFrom — declared as a separate
  runtime_checkable Protocol so 'iaf migrate-store' (Phase 3d) and
  'finterion push' (closed-source) can isinstance-test for it. Future
  capabilities (SupportsRelations for the strategy/version/report
  graph, SupportsContentAddressedChunks for Tier-3 dedup) follow the
  same pattern.
- LocalDirStore: handle = bundle path relative to the root, so the
  store stays portable across moves. Sidecar SqliteBacktestIndex
  (built lazily, incrementally — same machinery as 'iaf index') backs
  iter_index_rows so listing does not re-decode bundles. Path-traversal
  guards reject handles that escape the root.
- Tests (19, all passing): Protocol/SupportsCopyFrom conformance,
  round-trip, summary_only, default-handle derivation, sidecar index
  caching, copy_from with and without a handle subset, handle
  normalisation, path-traversal rejection, missing-handle error,
  delete idempotency.

Targeted suite (store + index + cli): 86 / 86 passing.
@MDUYN MDUYN merged commit e07b9e6 into feature/iaf-index-cli May 12, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant