feat(store): BacktestStore Protocol + LocalDirStore (epic #540 phase 3a)#544
Merged
Merged
Conversation
Introduces the storage seam that decouples *where* a backtest is persisted from the rest of the framework. Phase 3a is intentionally scoped to the Protocol + a thin adapter over today's .iafbt layout; LocalTieredStore (Tier-2 Parquet + Tier-3 chunks) and FinterionStore land in follow-up PRs. - BacktestStore Protocol: write / open / exists / delete / iter_handles / iter_index_rows / __len__ / __contains__. Mirrors today's Backtest.save_bundle / Backtest.open semantics so LocalDirStore is a 1:1 adapter. - StoreHandle: opaque str token (relative bundle path for LocalDirStore; uuid7 run_id for the upcoming tiered stores). - StoreError + StoreHandleNotFoundError. - Optional capability mixin SupportsCopyFrom — declared as a separate runtime_checkable Protocol so 'iaf migrate-store' (Phase 3d) and 'finterion push' (closed-source) can isinstance-test for it. Future capabilities (SupportsRelations for the strategy/version/report graph, SupportsContentAddressedChunks for Tier-3 dedup) follow the same pattern. - LocalDirStore: handle = bundle path relative to the root, so the store stays portable across moves. Sidecar SqliteBacktestIndex (built lazily, incrementally — same machinery as 'iaf index') backs iter_index_rows so listing does not re-decode bundles. Path-traversal guards reject handles that escape the root. - Tests (19, all passing): Protocol/SupportsCopyFrom conformance, round-trip, summary_only, default-handle derivation, sidecar index caching, copy_from with and without a handle subset, handle normalisation, path-traversal rejection, missing-handle error, delete idempotency. Targeted suite (store + index + cli): 86 / 86 passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of Phase 3 of epic #540 — introduces the storage seam that decouples where a backtest is persisted from the rest of the framework.
What's in this PR
BacktestStoreProtocolA runtime-checkable Protocol with the minimum contract every backtest store must satisfy:
Semantics mirror today's
Backtest.save_bundle/Backtest.openexactly —LocalDirStoreis a 1:1 adapter, so existing consumers (HTML report,iaf list/rank, MCP server) work unchanged.Optional capability mixins
Capabilities not every store can or should implement are declared as separate runtime-checkable Protocols, so callers feature-test with
isinstance(store, SupportsXxx):SupportsCopyFrom(this PR) — foriaf migrate-store(Phase 3d) andfinterion push(closed-source). Default fallback is a per-handlewrite(src.open(h))loop; tiered stores can override to dedup chunks during ingest.SupportsRelations(Phase 3+, closed-source) — strategy/version/report graph; onlyFinterionStorewill implement it.SupportsContentAddressedChunks(Phase 3c) — Tier-3 dedup.LocalDirStoreThin adapter over today's
.iafbtdirectory layout:"sweep_a/run_03.iafbt"), so moving the root directory does not invalidate any handle..iafbtsuffix is normalised — callers may omit it.../escape, absolute paths).<root>/index.sqlite) is built lazily and incrementally — same machinery asiaf index— soiter_index_rows()does not re-decode bundles. Disable withuse_index=Falsefor one-shot scripts.copy_from(src)loopssrc.iter_handles()→write(src.open(h), handle=h).Tests
19 new tests, all passing:
SupportsCopyFromruntime conformance.summary_onlyhonoured.algorithm_id.iter_handles/__len__/__contains__.iter_index_rowsreturns typedBacktestIndexRowwith relative paths; sidecar SQLite is created on first call;use_index=Falseskips sidecar.../escape, absolute paths).copy_fromwith and without a handle subset.openof missing handle raisesStoreHandleNotFoundError;deleteof missing handle is a no-op.Targeted suite (
tests/services/backtest_store/+tests/services/backtest_index/+tests/cli/): 86 / 86 passing.What's next (still in Phase 3)
LocalTieredStoreTier-1 (SQLite) + Tier-2 (per-project Parquet for snapshots / trades / orders / metric_series)ohlcv,code,params,symbols) — where the 64 GB → 20 GB headline livesiaf migrate-store --from local-dir --to local-tiered+ byte-identicalbacktest.export()/Backtest.import_()round-trip + the parameterised test fixture that runs every backtest test against both storesEach slice will land as a separate stacked PR.