diff --git a/README.md b/README.md index a05dd8ee..b11dee91 100644 --- a/README.md +++ b/README.md @@ -85,7 +85,8 @@ This framework is built around the full loop: **create strategies β†’ vector bac - πŸ“„ **[One-Click HTML Report](https://coding-kitties.github.io/investing-algorithm-framework/Getting%20Started/backtest-reports)** β€” Self-contained file, no server, dark & light theme, shareable - πŸ“¦ **[Custom `.iafbt` Backtest Bundle Format](https://coding-kitties.github.io/investing-algorithm-framework/Data/backtest_data)** β€” An explicit, versioned, compressed, language-portable container (zstd + msgpack with magic-byte header) plus a separate parquet index for fast filtering without loading. ~21Γ— smaller and ~27Γ— fewer files than standard filebased directory layouts, with parallel I/O for fast load/save of large amounts of backtests. - 🌐 **[Load External Data](https://coding-kitties.github.io/investing-algorithm-framework/Data/external-data)** β€” Fetch CSV, JSON, or Parquet from any URL with caching and auto-refresh -- πŸ“ **[Record Custom Variables](https://coding-kitties.github.io/investing-algorithm-framework/Advanced%20Concepts/recording-variables)** β€” Track any indicator or metric during backtests with `context.record()` +- οΏ½ **[Per-Market Deposit Schedules & Portfolio Sync](https://coding-kitties.github.io/investing-algorithm-framework/Advanced%20Concepts/portfolio-sync)** β€” Declare recurring or one-shot external cash flows on a market with `deposit_schedule=` / `auto_sync=True`. Backtests simulate the deposits; live mode reconciles with the broker β€” same `context.sync_portfolio()` API in both modes. +- οΏ½πŸ“ **[Record Custom Variables](https://coding-kitties.github.io/investing-algorithm-framework/Advanced%20Concepts/recording-variables)** β€” Track any indicator or metric during backtests with `context.record()` - πŸš€ **[Build β†’ Backtest β†’ Deploy](https://coding-kitties.github.io/investing-algorithm-framework/Getting%20Started/application-setup)** β€” Local dev, cloud deploy (AWS / Azure), or monetize on Finterion diff --git a/docusaurus/docs/Advanced Concepts/portfolio-sync.md b/docusaurus/docs/Advanced Concepts/portfolio-sync.md new file mode 100644 index 00000000..05c15308 --- /dev/null +++ b/docusaurus/docs/Advanced Concepts/portfolio-sync.md @@ -0,0 +1,252 @@ +--- +sidebar_position: 4 +--- + +# Portfolio Sync & Deposit Schedules + +Real trading bots rarely operate on a static initial balance. Cash flows in (paychecks, transfers, profit reinvestment) and sometimes out (withdrawals). The framework treats external cash movement as a **first-class concept** so the same code works in backtests and in live deployments. + +The contract is a single API: `context.sync_portfolio(market=...)`. It reconciles the portfolio's local `unallocated` cash with what is *actually* available on the broker (live) or with the simulated deposit schedule (backtest). + +## TL;DR β€” Declarative deposit schedule on a market + +The most common case β€” a recurring deposit (e.g. monthly paycheck DCA) β€” needs **zero strategy code**: + +```python +from investing_algorithm_framework import ( + create_app, + ScheduledDeposit, + TimeUnit, +) + +app = create_app() +app.add_market( + market="BITVAVO", + trading_symbol="EUR", + initial_balance=2500, + deposit_schedule=[ + ScheduledDeposit( + amount=100.0, time_unit=TimeUnit.DAY, interval=30 + ), + ], + auto_sync=True, +) +``` + +With `auto_sync=True` the event loop calls `context.sync_portfolio(market=...)` at the start of every iteration. The strategy simply checks `context.get_unallocated()` and the new cash is already there. + +## `ScheduledDeposit` + +A frozen dataclass describing **one** deposit rule. Two mutually exclusive forms: + +| Form | Fields | Behaviour | +|------|--------|-----------| +| **Recurring** | `amount`, `time_unit`, `interval` | Fires every `interval` units of `time_unit` from the anchor (the first time the loop sees the market). | +| **One-shot** | `amount`, `on` (timezone-aware `datetime`) | Fires once, at or after `on`. | + +```python +from datetime import datetime, timezone +from investing_algorithm_framework import ScheduledDeposit, TimeUnit + +# Weekly DCA top-up +ScheduledDeposit(amount=50.0, time_unit=TimeUnit.DAY, interval=7) + +# One-time bonus on a fixed date +ScheduledDeposit(amount=1000.0, on=datetime(2025, 12, 24, tzinfo=timezone.utc)) + +# Negative amount = scheduled withdrawal +ScheduledDeposit(amount=-200.0, time_unit=TimeUnit.DAY, interval=30) +``` + +`MONTH` cadence is approximated as 30 days (`timedelta` has no calendar months). For exact calendar dates use one-shot deposits. + +You can pass a list of `ScheduledDeposit` to `add_market(...)`, or add them later: + +```python +app.add_deposit_schedule( + market="BITVAVO", + schedule=[ScheduledDeposit(amount=100.0, time_unit=TimeUnit.DAY, interval=30)], +) +app.set_market_auto_sync("BITVAVO", enabled=True) +``` + +## `context.sync_portfolio(market, allow_withdrawals=False, tolerance=1e-9) -> SyncResult` + +This is the canonical reconciliation call. Behaviour depends on the environment: + +| Mode | "Broker available" comes from | +|------|-------------------------------| +| **Live** (`PROD` / `DEV`) | `PortfolioProvider.get_position(...).amount` β€” the exchange's free balance for the trading symbol, **minus** cash reserved for orders the framework has created but the exchange has not yet acknowledged (`OrderStatus.CREATED` BUYs). This avoids flagging a brief race window between `create_order()` and the exchange ack as a phantom withdrawal. | +| **Backtest** | The `BrokerBalanceTracker` materialises the schedule. Due deposits accumulate in a "pending" bucket and are drained into `unallocated`. | + +The returned `SyncResult` has fields: + +```python +@dataclass(frozen=True) +class SyncResult: + market: str + kind: Literal["noop", "deposit", "withdrawal"] + delta: float # broker_available - previous_unallocated + broker_available: float + previous_unallocated: float + new_unallocated: float + within_tolerance: bool = False # |delta| <= tolerance, but not zero + reserved_for_pending_orders: float = 0.0 +``` + +### Tolerance + +Pass `tolerance=` to ignore sub-cent rounding drift between the broker and your local books: + +```python +# Treat anything under 1 cent of drift as a no-op +context.sync_portfolio(market="BITVAVO", tolerance=0.01) +``` + +When the absolute drift is at or below `tolerance` the call returns a `noop` result with `within_tolerance=True` and the raw `delta` still surfaced for diagnostics. The portfolio's `unallocated` is **not** modified. + +### Withdrawal semantics + +By default, `sync_portfolio` raises `PortfolioOutOfSyncError` if the broker has **less** cash than the portfolio thinks it owns. This is intentional β€” silently shrinking your bot's cash on a transient broker glitch would be a debugging nightmare. + +Opt in to draining with `allow_withdrawals=True`: + +```python +try: + result = context.sync_portfolio(market="BITVAVO") +except PortfolioOutOfSyncError as err: + log.warning( + "Out of sync on %s: local=%s broker=%s delta=%s", + err.market, err.local_unallocated, err.broker_available, err.delta, + ) + # Decide: pause the bot, alert, or drain. + context.sync_portfolio(market="BITVAVO", allow_withdrawals=True) +``` + +`unallocated` will never be pushed below zero regardless of the flag. + +## Auto-sync error handling + +`add_market(..., auto_sync=True)` is convenient but a single flaky API call should not crash a long-running bot. Pick the policy that matches your operational stance with `auto_sync_error_mode`: + +| Mode | Behaviour on transient broker error | +|------|-------------------------------------| +| `"raise"` *(default)* | Propagate the exception. The event loop stops. Best during development β€” fail loudly. | +| `"warn"` | Log a warning and continue. Auto-sync retries on the next iteration. Best for live trading: a single 502 from the exchange shouldn't take the bot down. | +| `"halt"` | Log an error and disable auto-sync **for that market** until the app is restarted or `set_market_auto_sync(...)` is called again. The strategy keeps running on whatever cash it currently has. | + +```python +app.add_market( + market="BITVAVO", + trading_symbol="EUR", + initial_balance=2500, + auto_sync=True, + auto_sync_error_mode="warn", +) +``` + +`PortfolioOutOfSyncError` is **always** re-raised regardless of mode β€” that's a hard data-integrity signal, not a transient. + +## `fire_on_anchor` β€” deposit at t=0 + +By default a recurring deposit fires at `anchor + interval`, not at the anchor itself. Set `fire_on_anchor=True` to also fire on the anchor day: + +```python +ScheduledDeposit( + amount=500.0, + time_unit=TimeUnit.DAY, + interval=30, + fire_on_anchor=True, # also fires on day 0 +) +``` + +This is useful for "fund the bot immediately at start, then top up monthly" patterns. Only valid for recurring deposits (one-shots fire on `on=` regardless). + +## TWR-aware return metrics + +Every deposit/withdrawal absorbed by `sync_portfolio` (or replayed by the vector backtest) is stamped onto the next `PortfolioSnapshot.cash_flow`. The framework's return metrics use this to compute **time-weighted returns** so that depositing $1,000 into your bot does not show up as $1,000 of P&L: + +$$ r_t = \frac{V_t - \text{cash\_flow}_t}{V_{t-1}} - 1 $$ + +The following metrics are TWR-adjusted: **CAGR**, **monthly returns**, **yearly returns**, **mean daily return**, **Sharpe**, **Sortino**, **volatility**, **VaR / CVaR**, and the standard-deviation family. Snapshots without a `cash_flow` field (legacy data, mocks) gracefully fall back to the classic `pct_change()` behaviour. + +> **Equity curve and drawdown** are exposed in **two flavours**: +> +> - `get_equity_curve` / `get_drawdown_series` / `get_max_drawdown` / `get_max_drawdown_duration` β€” raw account value, including deposits. Use these when you want to see "how many dollars did the account hold?". +> - `get_twr_equity_curve` / `get_twr_drawdown_series` / `get_twr_max_drawdown` / `get_twr_max_drawdown_duration` β€” alpha-only path scrubbed of external cash flows. Use these when comparing risk profiles across portfolios funded differently β€” depositing $1,000 during a drawdown won't artificially erase it. +> +> ```python +> from investing_algorithm_framework.services.metrics import ( +> get_twr_equity_curve, get_twr_max_drawdown, +> ) +> # Growth-of-$1 alpha curve +> curve = get_twr_equity_curve(snapshots, base=1.0) +> dd = get_twr_max_drawdown(snapshots) # e.g. 0.18 for 18% +> ``` + +## Mid-run schedule changes + +`tracker.set_schedule(market, [...])` and `add_deposit_schedule` can be called at any time. The tracker preserves: + +- **`pending`** β€” any deposits that already fired but haven't been absorbed by `sync_portfolio` yet. +- **`cash_flow_since_snapshot`** β€” TWR bookkeeping waiting to be drained by the next snapshot. + +Only the schedule itself and `last_fired` anchors are reset β€” so swapping a 30-day schedule for a 7-day schedule mid-backtest doesn't accidentally "swallow" a pending deposit. + +## Manual sync (without `auto_sync`) + +If you prefer explicit control, leave `auto_sync=False` and call `sync_portfolio` from your strategy: + +```python +class MyStrategy(TradingStrategy): + def run_strategy(self, context, data): + context.sync_portfolio(market="BITVAVO") + cash = context.get_unallocated() + ... +``` + +This pattern is useful if you want to sync only on specific bars, or after specific events (e.g. only after a fill notification in live mode). +or if you want to run a one-shot sync declare a task that calls `sync_portfolio` once at startup: + +```python +from investing_algorithm_framework import Task + +class SyncTask(Task): + interval = None # run once at startup + + def run(self, context): + context.sync_portfolio(market="BITVAVO") + log.info("Initial sync complete") + +## Vector backtests + +Vector backtests respect the same `deposit_schedule`. Because vector workers run in subprocesses, the schedule travels with `PortfolioConfiguration` (it is pickled). Each iteration's cash track has the due deposits added before signals are evaluated, so the equity curve includes external cash flows. + +```python +from investing_algorithm_framework import PortfolioConfiguration, ScheduledDeposit, TimeUnit + +PortfolioConfiguration( + market="BITVAVO", + trading_symbol="EUR", + initial_balance=2500, + deposit_schedule=[ + ScheduledDeposit(amount=100.0, time_unit=TimeUnit.DAY, interval=30), + ], +) +``` + +## Worked example + +See [`examples/strategies_showcase/08_dca_accumulation`](https://github.com/coding-kitties/investing-algorithm-framework/tree/main/examples/strategies_showcase/08_dca_accumulation) for a complete DCA bot that uses `deposit_schedule` + `auto_sync` to model recurring monthly deposits across a 2-year backtest. + +## API summary + +| Symbol | Where | Purpose | +|--------|-------|---------| +| `ScheduledDeposit` | `investing_algorithm_framework` | Declarative deposit rule (recurring or one-shot). | +| `SyncResult` | `investing_algorithm_framework` | Outcome of a sync call. | +| `PortfolioOutOfSyncError` | `investing_algorithm_framework` | Raised when broker balance < local unallocated and withdrawals are not opted in. | +| `app.add_market(deposit_schedule=, auto_sync=)` | App | Register a market with optional schedule and auto-sync flag. | +| `app.add_deposit_schedule(market, schedule)` | App | Attach a schedule to an existing market. | +| `app.set_market_auto_sync(market, enabled=True)` | App | Toggle per-market auto-sync. | +| `context.sync_portfolio(market, allow_withdrawals=False)` | Context | Reconcile local cash with broker / tracker. | diff --git a/docusaurus/docs/Advanced Concepts/vector-backtesting.md b/docusaurus/docs/Advanced Concepts/vector-backtesting.md index 4cbeb03f..d05e31bf 100644 --- a/docusaurus/docs/Advanced Concepts/vector-backtesting.md +++ b/docusaurus/docs/Advanced Concepts/vector-backtesting.md @@ -40,6 +40,20 @@ Use vector backtesting when: - Needing to complete backtests quickly - Working with large strategy sets (100+ strategies) +:::warning Order types not modelled in vector backtests +Vector backtesting evaluates strategy **signals** (`generate_buy_signals`, `generate_sell_signals`) against the price series. It does **not** simulate the order book, intra-bar price paths, partial fills, or order-type semantics. + +This means the following are **only evaluated in event-driven backtests** and live trading: + +- `OrderType.MARKET` slippage and next-bar fill behaviour +- `OrderType.LIMIT` resting fill logic +- `OrderType.STOP` and `OrderType.STOP_LIMIT` trigger logic +- Take-profit / stop-loss rules attached to trades +- Blotter slippage / commission / volume-based fill models + +The recommended workflow is to use vector backtests to **screen parameter combinations quickly** based on signal quality, then promote the survivors to **event-driven backtests** where realistic execution is simulated. See [Orders](../Getting%20Started/orders) for the full order-type reference. +::: + ## Basic Usage ### Single Strategy, Single Date Range diff --git a/docusaurus/docs/Getting Started/orders.md b/docusaurus/docs/Getting Started/orders.md index f85d14e0..a2461748 100644 --- a/docusaurus/docs/Getting Started/orders.md +++ b/docusaurus/docs/Getting Started/orders.md @@ -96,41 +96,76 @@ algorithm.create_sell_order( ### Stop Orders -Trigger market orders when price reaches a specified level: +A **stop order** rests in the book until the market trades through a configured `stop_price`. Once triggered, it becomes a market order and fills at the next available price. + +- **SELL stop** triggers when the market drops to (or below) `stop_price` β€” used for stop-losses or trend-following exits. +- **BUY stop** triggers when the market rises to (or above) `stop_price` β€” used for breakout entries. ```python -# Stop loss - sell if price drops to 45,000 -algorithm.create_sell_order( +from investing_algorithm_framework import OrderType, OrderSide + +# SELL stop β€” exit if BTC drops to 45,000 EUR +self.create_order( target_symbol="BTC", - percentage=1.0, - order_type="STOP", - stop_price=45000 + order_side=OrderSide.SELL, + amount=0.5, + order_type=OrderType.STOP, + stop_price=45000, ) -# Buy stop - buy if price rises to 52,000 (breakout strategy) -algorithm.create_buy_order( +# BUY stop β€” enter on a breakout above 52,000 EUR +self.create_order( target_symbol="BTC", - amount=100, - order_type="STOP", - stop_price=52000 + order_side=OrderSide.BUY, + amount=0.1, + order_type=OrderType.STOP, + stop_price=52000, + price=52000, # used as the cash-reservation reference ) ``` ### Stop-Limit Orders -Combine stop and limit order features: +A **stop-limit order** triggers like a stop, but instead of becoming a market order, it becomes a **limit order** at the configured `price`. This protects against bad fills during gaps but does not guarantee execution. ```python -# Stop-limit sell order -algorithm.create_sell_order( +# SELL stop-limit β€” trigger at 45,000, but only sell at 44,500 or better +self.create_order( target_symbol="BTC", - percentage=1.0, - order_type="STOP_LIMIT", - stop_price=45000, # Trigger when price hits 45,000 - price=44500 # But only sell at 44,500 or better + order_side=OrderSide.SELL, + amount=0.5, + order_type=OrderType.STOP_LIMIT, + stop_price=45000, # trigger price + price=44500, # limit price after trigger ) ``` +**Validation rules** + +- `stop_price` is required for both `STOP` and `STOP_LIMIT` orders. +- For `STOP_LIMIT`, `price` is also required and must be: + - `price <= stop_price` for SELL stop-limit orders + - `price >= stop_price` for BUY stop-limit orders + +:::note Where stop orders are evaluated +Stop and stop-limit orders are simulated in **event-driven backtests** and executed by supported exchanges in **live trading** (via the unified CCXT `stopPrice` parameter). + +They are **not** evaluated in **vector backtests**. Vector backtesting models strategy *signals* only β€” it does not simulate the order book, intra-bar price paths, or stop triggers. Use vector backtests to filter parameter sets quickly, then promote promising strategies to event-driven backtests where stops, slippage, commissions, and partial fills are evaluated. See [Vector Backtesting](../Advanced%20Concepts/vector-backtesting) for the trade-offs. +::: + +#### Backtest Trigger Semantics + +In an event-driven backtest, the trigger condition is evaluated against each candle's High / Low after the order's `updated_at`: + +| Order side | Triggers when | +|------------|---------------| +| SELL STOP / STOP_LIMIT | `Low <= stop_price` | +| BUY STOP / STOP_LIMIT | `High >= stop_price` | + +- A **STOP** that triggers fills at `stop_price` on the triggering candle (then slippage and commission from the configured blotter / `TradingCost` are applied via the same fill path used for market orders). +- A **STOP_LIMIT** that triggers becomes a resting limit order at `price` from the triggering candle onwards, and fills under the normal limit-fill rules (`Low <= price` for BUY, `High >= price` for SELL). +- The triggering timestamp is stored on the order as `triggered_at` for auditability. + ## Order Parameters ### Market Order Parameters diff --git a/docusaurus/sidebars.js b/docusaurus/sidebars.js index ec0a466d..f74c27d6 100644 --- a/docusaurus/sidebars.js +++ b/docusaurus/sidebars.js @@ -133,6 +133,10 @@ const sidebars = { type: 'doc', id: 'Advanced Concepts/recording-variables', }, + { + type: 'doc', + id: 'Advanced Concepts/portfolio-sync', + }, { type: 'doc', id: 'Advanced Concepts/pipelines', diff --git a/examples/advanced_tutorials/cross-sectional-pipelines/README.md b/examples/advanced_tutorials/cross-sectional-pipelines/README.md new file mode 100644 index 00000000..6f56130d --- /dev/null +++ b/examples/advanced_tutorials/cross-sectional-pipelines/README.md @@ -0,0 +1,81 @@ +# Pipeline API examples + +This folder showcases the **Pipeline API (Phase 1)** β€” a declarative way to +express cross-sectional screens and signals over a panel of symbols, similar +in spirit to Quantopian / Zipline pipelines. + +A `Pipeline` is a small class where you declare the columns you want as +`Factor` / `Filter` attributes. The engine turns those declarations into a +single Polars DataFrame on each scheduled run, with no look-ahead, ready for +your `TradingStrategy` to consume via `data["YourPipelineClassName"]`. + +## Examples in this folder + +| File | What it shows | +| --- | --- | +| [`momentum_screener.py`](momentum_screener.py) | Minimal screener-only example: a `MomentumScreener` ranks 5 EUR pairs by 30‑day return within the top‑3 most liquid names and prints the top‑2 each day. No order placement β€” useful for understanding the pipeline output format. | +| [`cross_sectional_momentum_bot.py`](cross_sectional_momentum_bot.py) | End-to-end trading bot: a `MomentumScreener` ranks symbols by 30‑day return within the top‑3 most liquid names, and a `TradingStrategy` rebalances daily into the top‑2 ranked symbols on Bitvavo. Includes a 1‑year backtest. | + +## Anatomy of the example + +```python +class MomentumScreener(Pipeline): + dollar_volume = AverageDollarVolume(window=30) + momentum = Returns(window=30) + universe = dollar_volume.top(3) # liquidity filter + alpha = momentum.rank(mask=universe) # ranked score +``` + +That's the whole pipeline. The strategy then consumes it: + +```python +class CrossSectionalMomentumBot(TradingStrategy): + pipelines = [MomentumScreener] + ... + + def run_strategy(self, context, data): + screen = data["MomentumScreener"] # polars.DataFrame + targets = screen.sort("alpha", descending=True).head(TOP_N) + # ... rebalance into `targets` ... +``` + +## Run a backtest + +From the repository root: + +```bash +python examples/cross-sectional-pipelines/cross_sectional_momentum_bot.py +``` + +The script: + +1. Creates an app pointed at `examples/resources/`. +2. Registers OHLCV data sources for 7 EUR pairs on Bitvavo. +3. Runs a 1‑year daily backtest of the momentum rotation. +4. Prints the full `Backtest` JSON (metrics, trades, snapshots). + +For an interactive HTML dashboard, replace the `print(backtest)` line with: + +```python +from investing_algorithm_framework import BacktestReport +BacktestReport(backtest).show() +``` + +## Run it live + +Remove the `app.run_backtest(...)` block at the bottom of the file and call +`app.run()` instead. Bitvavo does not require API keys for market data, so +the bot will pull live OHLCV out of the box. Add Bitvavo credentials via the +standard config to enable live order execution. + +## Built-in factors / filters used + +- `AverageDollarVolume(window=N)` β€” rolling close Γ— volume over N bars. +- `Returns(window=N)` β€” N‑bar percentage change. +- `Factor.rank(mask=...)` β€” cross‑sectional rank, optionally restricted to a filter. +- `Factor.top(n)` / `Factor.bottom(n)` β€” per‑bar top/bottom‑N selection (returns a `Filter`). + +See the framework docs: + +- `docusaurus/docs/Advanced Concepts/pipelines.md` +- `docusaurus/docs/Advanced Concepts/pipelines-event-backtest.md` diff --git a/examples/advanced_tutorials/cross-sectional-pipelines/cross_sectional_momentum_bot.py b/examples/advanced_tutorials/cross-sectional-pipelines/cross_sectional_momentum_bot.py new file mode 100644 index 00000000..f65091f1 --- /dev/null +++ b/examples/advanced_tutorials/cross-sectional-pipelines/cross_sectional_momentum_bot.py @@ -0,0 +1,210 @@ +"""Pipeline API β€” cross-sectional momentum trading bot (Phase 1). + +This example shows how to turn a :class:`Pipeline` screen into an actual +trading bot that rebalances a portfolio every day into the top-N +momentum names within a liquid universe. + +What it does on each iteration (once per day): + +1. The framework builds a long-form OHLCV panel across the configured + symbols. +2. ``MomentumScreener`` ranks every symbol by 30-day return within the + top-3 most liquid names (by 30-day average dollar volume). +3. ``CrossSectionalMomentumBot.run_strategy`` reads the resulting + ``polars.DataFrame`` from ``data["MomentumScreener"]``, picks the + top-2 ranked symbols, and rebalances the portfolio: + * closes any open position that is no longer in the target set, + * opens an equal-weight market order for any new target. + +Backtest the bot: + +.. code-block:: bash + + python examples/cross-sectional-pipelines/cross_sectional_momentum_bot.py + +Run it live by removing the ``app.run_backtest(...)`` block and calling +``app.run()`` instead. Bitvavo does not require API keys for market +data, so the backtest works out of the box. + +See docs: +- ``docs/Advanced Concepts/pipelines.md`` +- ``docs/Advanced Concepts/pipelines-event-backtest.md`` +""" +from __future__ import annotations + +import logging.config +from datetime import datetime, timedelta +from typing import Any, Dict + +from dotenv import load_dotenv + +from investing_algorithm_framework import ( + AverageDollarVolume, + BacktestDateRange, + Context, + DataSource, + DEFAULT_LOGGING_CONFIG, + OrderSide, + OrderType, + Pipeline, + Returns, + TimeUnit, + TradingStrategy, + create_app, +) + +logging.config.dictConfig(DEFAULT_LOGGING_CONFIG) +load_dotenv() + + +# Universe +# A small crypto basket. All symbols quote in EUR so they share the +# same trading symbol (the portfolio currency below). +SYMBOLS = [ + "BTC/EUR", + "ETH/EUR", + "SOL/EUR", + "ADA/EUR", + "XRP/EUR", + "DOT/EUR", + "LINK/EUR", +] + +#: Number of symbols held simultaneously. +TOP_N = 2 + +#: Exchange and quote currency for the bot. +MARKET = "bitvavo" +TRADING_SYMBOL = "EUR" + + +# Pipeline: declarative cross-sectional screen +class MomentumScreener(Pipeline): + """Rank the most liquid 3 names by 30-day return.""" + + # Per-symbol factors. Each becomes an output column. + dollar_volume = AverageDollarVolume(window=30) + momentum = Returns(window=30) + + # The master mask: only the 3 most liquid names enter ranking. + universe = dollar_volume.top(3) + + # Cross-sectional rank within the universe (highest momentum gets + # the highest rank). + alpha = momentum.rank(mask=universe) + + +# Strategy: rebalance into the top-N pipeline picks +class CrossSectionalMomentumBot(TradingStrategy): + algorithm_id = "pipeline-momentum-bot" + time_unit = TimeUnit.DAY + interval = 1 + market = MARKET + trading_symbol = TRADING_SYMBOL + symbols = SYMBOLS + + # OHLCV data sources, one per symbol. ``warmup_window`` covers the + # longest factor lookback (30) plus a buffer so the pipeline starts + # producing values from bar 1 of the backtest. Tickers are added so + # the bot can read live bid/ask in production β€” in backtest mode the + # framework will fall back to the latest OHLCV close. + data_sources = [ + DataSource( + data_type="OHLCV", + market=MARKET, + symbol=symbol, + warmup_window=60, + time_frame="1d", + identifier=f"{symbol}-ohlcv", + ) + for symbol in SYMBOLS + ] + + # Opt-in to the Pipeline API. The framework will compute this every + # iteration and place the result under ``data["MomentumScreener"]``. + pipelines = [MomentumScreener] + + # Strategy entry point + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + screen = data["MomentumScreener"] + + # Skip iterations where the universe / warmup is not satisfied. + if screen.is_empty(): + return + + # Rank is ascending β€” highest rank = highest momentum. + targets_df = screen.sort("alpha", descending=True).head(TOP_N) + targets = {row["symbol"] for row in targets_df.iter_rows(named=True)} + + def _last_close(symbol: str) -> float: + ohlcv = data[f"{symbol}-ohlcv"] + try: + return float(ohlcv["Close"][-1]) + except (KeyError, TypeError): + return float(ohlcv["close"].iloc[-1]) + + def _base(symbol: str) -> str: + # ``target_symbol`` is just the base asset (e.g. "BTC"). + return symbol.split("/")[0] + + # 1. Close positions no longer in the target set + for symbol in SYMBOLS: + if symbol in targets: + continue + base = _base(symbol) + if not context.has_position(base, market=self.market): + continue + position = context.get_position(base, market=self.market) + context.create_order( + target_symbol=base, + order_side=OrderSide.SELL, + order_type=OrderType.LIMIT, + price=_last_close(symbol), + amount=position.get_amount(), + ) + + # 2. Open new equal-weight positions for new targets + unallocated = context.get_unallocated() + new_targets = [ + sym for sym in targets + if not context.has_position(_base(sym), market=self.market) + ] + if not new_targets: + return + + per_target_budget = unallocated / len(new_targets) + for symbol in new_targets: + price = _last_close(symbol) + if price <= 0: + continue + # 0.5% safety buffer so floating-point rounding does not + # push the order total above the available unallocated cash. + amount = (per_target_budget * 0.995) / price + context.create_order( + target_symbol=_base(symbol), + order_side=OrderSide.BUY, + order_type=OrderType.LIMIT, + price=price, + amount=amount, + ) + + +# App wiring +app = create_app() +app.add_strategy(CrossSectionalMomentumBot) +app.add_market(market=MARKET, trading_symbol=TRADING_SYMBOL, initial_balance=1000) + + +if __name__ == "__main__": + # Backtest over the last full year. Replace with ``app.run()`` to go + # live (requires bitvavo credentials in your .env file). + end = datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0) + start = end - timedelta(days=365) + backtest_date_range = BacktestDateRange(start_date=start, end_date=end) + backtest = app.run_backtest( + backtest_date_range=backtest_date_range, + ) + # Inspect the result interactively with the BacktestReport dashboard: + # from investing_algorithm_framework import BacktestReport + # BacktestReport(backtest).show() + print(backtest) diff --git a/examples/advanced_tutorials/cross-sectional-pipelines/momentum_screener.py b/examples/advanced_tutorials/cross-sectional-pipelines/momentum_screener.py new file mode 100644 index 00000000..c161bd90 --- /dev/null +++ b/examples/advanced_tutorials/cross-sectional-pipelines/momentum_screener.py @@ -0,0 +1,107 @@ +"""Pipeline API β€” momentum cross-sectional screener example. + +Phase 1 of the Pipeline API (#501) lets a strategy compute factors +across all of its OHLCV symbols in one shot, then receive a wide +``DataFrame`` (one row per surviving symbol, one column per factor) +on each iteration. + +This example screens a small crypto basket every day for the two +highest-momentum symbols by 30-day return, ranked within the top-3 +liquidity universe. It does not place any orders β€” it only prints the +screen output, so you can see what the pipeline produces. + +Phase 1 only supports the **event-driven backtest** runner; live +pipelines are not implemented yet. Run the example with: + +.. code-block:: bash + + python examples/cross-sectional-pipelines/momentum_screener.py + +See ``docs/Advanced Concepts/pipelines-event-backtest.md``. +""" +from __future__ import annotations + +import logging.config +from datetime import datetime, timedelta +from typing import Any, Dict + +from dotenv import load_dotenv + +from investing_algorithm_framework import ( + AverageDollarVolume, + BacktestDateRange, + Context, + DataSource, + DEFAULT_LOGGING_CONFIG, + Pipeline, + Returns, + TimeUnit, + TradingStrategy, + create_app, +) + +logging.config.dictConfig(DEFAULT_LOGGING_CONFIG) +load_dotenv() + + +SYMBOLS = ["BTC/EUR", "ETH/EUR", "SOL/EUR", "ADA/EUR", "XRP/EUR"] +MARKET = "bitvavo" +TRADING_SYMBOL = "EUR" + + +class MomentumScreener(Pipeline): + """Top-2 momentum names within the most liquid 3 of the universe.""" + + dollar_volume = AverageDollarVolume(window=30) + momentum = Returns(window=30) + + universe = dollar_volume.top(3) + alpha = momentum.rank(mask=universe) + + +class CrossSectionalMomentum(TradingStrategy): + algorithm_id = "cross-sectional-momentum" + time_unit = TimeUnit.DAY + interval = 1 + market = MARKET + trading_symbol = TRADING_SYMBOL + symbols = SYMBOLS + data_sources = [ + DataSource( + data_type="OHLCV", + market=MARKET, + symbol=symbol, + warmup_window=60, + time_frame="1d", + identifier=f"{symbol}-ohlcv", + ) + for symbol in SYMBOLS + ] + pipelines = [MomentumScreener] + + def run_strategy(self, context: Context, data: Dict[str, Any]): + screen = data["MomentumScreener"] + if screen is None or screen.is_empty(): + return + # Pick the top 2 by alpha rank (highest rank = highest momentum + # within the liquidity universe). + top = screen.sort("alpha", descending=True).head(2) + for row in top.iter_rows(named=True): + print( + f"{row['symbol']}: 30d return = {row['momentum']:.2%}, " + f"adv = {row['dollar_volume']:.0f}, alpha rank = {row['alpha']}" + ) + + +app = create_app() +app.add_strategy(CrossSectionalMomentum) +app.add_market(market=MARKET, trading_symbol=TRADING_SYMBOL, initial_balance=1000) + + +if __name__ == "__main__": + end = datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0) + start = end - timedelta(days=120) + backtest_date_range = BacktestDateRange(start_date=start, end_date=end) + backtest = app.run_backtest(backtest_date_range=backtest_date_range) + metrics = backtest.get_backtest_metrics(date_range=backtest_date_range) + print(metrics) diff --git a/examples/rust_vs_python_benchmark/.gitignore b/examples/rust_vs_python_benchmark/.gitignore new file mode 100644 index 00000000..a203dfb1 --- /dev/null +++ b/examples/rust_vs_python_benchmark/.gitignore @@ -0,0 +1,7 @@ +data/ +results/ +backtest_results/ +rust_bench/target/ +rust_bench/Cargo.lock +__pycache__/ +*.pyc diff --git a/examples/rust_vs_python_benchmark/README.md b/examples/rust_vs_python_benchmark/README.md new file mode 100644 index 00000000..a77cc62b --- /dev/null +++ b/examples/rust_vs_python_benchmark/README.md @@ -0,0 +1,145 @@ +# Python vs Rust backtest benchmark β€” `iaf-core` baseline + +Companion benchmark for the **hybrid Python + Rust acceleration** epic +(#521) and its phase issues (#522–#526). The goal is a **reproducible +baseline** for measuring future Rust kernel speedups against the current +Python framework path. + +## What this measures + +| Side | What it does | What it represents | +|------|--------------|--------------------| +| **`python_bench.py`** | Runs `app.run_vector_backtests(...)` with a real `TradingStrategy`, real `CSVOHLCVDataProvider` data sources, real bundle write, real metric aggregation. | The end-user code path today. | +| **`rust_bench/`** | A standalone Cargo binary that loads the same parquet OHLCV, computes the same EMA + RSI signals, runs the same long-only execution loop with fees + slippage, **persists every order and per-bar portfolio snapshot to a per-backtest SQLite database** (mirroring the framework's native behavior), and prints throughput. **No PyO3, no framework abstractions.** | The **best-case ceiling** for what a future `iaf-core` Rust kernel could achieve. | + +Both sides: + +- Run the **same parameter sweep** (deterministic grid of `combos` strategies) +- Process the **same synthetic OHLCV** (geometric Brownian motion, seeded) +- Use the **same fees (0.1%)** and **slippage (0.05%)** +- Use the **same EMA / RSI math** semantics (Wilder RSI; pandas-style EMA) +- **Persist orders + portfolio snapshots to a per-backtest SQLite DB.** + The framework already does this through SQLAlchemy; the Rust side + uses bundled `rusqlite` writing through prepared statements inside a + single transaction per backtest. Set `RUST_PERSIST=false` to measure + the pure compute ceiling instead. +- **Compute the standard metric pack at the end of every backtest** + (total return, CAGR, annual volatility, Sharpe, Sortino, max + drawdown + duration, Calmar) over the per-bar portfolio equity + curve. Mirrors the framework's `BacktestMetrics` so the comparison + isn't unfairly skewed by Python having to compute Sharpe and the + Rust side getting away with just final equity. Set + `RUST_METRICS=false` to isolate persistence cost. + +> **Honest framing:** This is *not* a "Rust vs Python language showdown". +> The Python side is intentionally measured **with full framework +> overhead** (data providers, strategy lifecycle, bundle write, +> per-backtest metric aggregation). The Rust side is intentionally +> minimal. The **gap between them is the optimisation headroom** that +> issues #521–#526 are scoped to capture by moving the hot path into a +> native PyO3 module behind the existing Python public API. + +## Layout + +``` +rust_vs_python_benchmark/ +β”œβ”€β”€ README.md # this file +β”œβ”€β”€ generate_data.py # synthesise OHLCV (parquet + CSV) +β”œβ”€β”€ python_bench.py # framework path +β”œβ”€β”€ rust_bench/ +β”‚ β”œβ”€β”€ Cargo.toml +β”‚ └── src/main.rs # reference Rust path +β”œβ”€β”€ compare.py # render results table +β”œβ”€β”€ run_benchmark.sh # orchestrate the whole run +β”œβ”€β”€ data/ # generated; .gitignored +└── results/ # generated; .gitignored +``` + +## Prerequisites + +- Python 3.10+ with the framework installed (`pip install -e ../..` from + the repo root, or `pip install investing-algorithm-framework`). +- Rust toolchain (`rustup` + stable `cargo`). The first + `cargo build --release` pulls down `polars` and friends and may take + several minutes. +- ~500 MB free disk for synthetic data at the default 10y Γ— 5 symbols. + +## Run + +```bash +# default: 10y of hourly bars, 25-strategy sweep, single Python worker, +# Rust persists to SQLite (apples-to-apples with the framework path) +./run_benchmark.sh + +# bigger sweep +COMBOS=100 ./run_benchmark.sh + +# more years and parallel Python workers +YEARS=20 PY_WORKERS=4 ./run_benchmark.sh + +# pure-compute Rust ceiling (no DB writes) β€” useful to isolate how much +# of the gap is SQLite I/O vs strategy lifecycle / metric overhead +RUST_PERSIST=false ./run_benchmark.sh +``` + +The script: + +1. Generates `data/*.parquet` + `data/*.csv` if missing. +2. `cargo build --release` for `rust_bench/` if missing. +3. Runs `python_bench.py` and writes `results/python_bench.json`. +4. Runs `rust_bench` and writes `results/rust_bench.json`. +5. Prints a side-by-side table via `compare.py`. + +## What the parameter grid looks like + +EMA short ∈ {10, 15, 20, 25, 30} +EMA long ∈ {50, 75, 100, 150, 200} +RSI period ∈ {7, 14, 21} +RSI oversold ∈ {25, 30, 35} +RSI overbought ∈ {65, 70, 75} + +Constraint: `ema_long > ema_short`. Picked deterministically. + +## Strategy logic (identical on both sides) + +- **Buy** when EMA-short crosses **above** EMA-long *and* RSI < oversold. +- **Sell** when EMA-short crosses **below** EMA-long *and* RSI > overbought. +- Long-only, single position per symbol. +- Fill at **next bar's open** with slippage; fees on entry and exit. +- Equal capital allocation per symbol. + +## Caveats / known divergences + +These are intentional and acceptable for a baseline benchmark: + +- **EMA seeding:** pandas' `ewm(span=p, adjust=False)` and the Rust + textbook EMA agree from the second sample onwards but differ on the + very first value. The downstream signal divergence is in the noise + for any sweep with realistic warmup. +- **RSI initialisation:** the Rust impl seeds Wilder's average from + the first delta; pandas uses an EWM seeded similarly but with a + slightly different ramp-up. Equivalent after `~3 Γ— period` bars. +- **Param-grid sampling:** the Python side uses NumPy's `default_rng(0)`, + the Rust side uses a small xorshift seeded with `0xDEAD_BEEF_CAFE_BABE`. + The *distribution* of params is matched, the exact ordering is not. + This does not affect throughput numbers. +- **Metrics:** both sides compute the standard metric pack (total return,\n CAGR, annual volatility, Sharpe, Sortino, max drawdown + duration,\n Calmar) over the equity curve. The Python side runs the **full**\n framework `BacktestMetrics` + `BacktestSummaryMetrics` pipeline\n (which has a longer tail of metrics: rolling Sharpe, exposure ratio,\n trades per day, win/loss ratio, ...). Issue #523 will port the\n metric kernel; the Rust pack here is the subset that drives 95% of\n user decisions and is enough to make the timing comparison honest. +- **Bundle I/O:** the Python side writes `.iafbt` bundles for every + backtest (issue #524 will port the writer). The Rust side does not + write bundles. This is a deliberate inclusion of framework overhead + on the Python side β€” it's part of what users actually pay today. + +## Using this as a regression baseline + +After each milestone in the iaf-core epic lands, re-run this benchmark +and append the result to `results/history.md` (or wherever you track +release benchmarks). Targets per phase: + +| Phase | Issue | Expected impact on the Python column | +|-------|-------|--------------------------------------| +| Metric kernel | #523 | `metrics` portion of per-backtest cost ↓ ~10Γ— | +| Bundle I/O | #524 | bundle-write portion ↓ ~5Γ— | +| Streaming pipeline | #525 | `recalculate_backtests_in_directory` (separate harness; same ratio applies) | +| Vector engine | #526 | indicator + signal + execution loop ↓ ~30–50Γ— | + +The Rust column is the asymptote. diff --git a/examples/rust_vs_python_benchmark/compare.py b/examples/rust_vs_python_benchmark/compare.py new file mode 100644 index 00000000..2b29b1f6 --- /dev/null +++ b/examples/rust_vs_python_benchmark/compare.py @@ -0,0 +1,80 @@ +"""Render a side-by-side comparison table from the two result JSON files.""" +from __future__ import annotations + +import json +from pathlib import Path + +RESULTS = Path(__file__).resolve().parent / "results" + + +def _load(name: str) -> dict | None: + p = RESULTS / name + if not p.exists(): + return None + return json.loads(p.read_text()) + + +def main() -> None: + py = _load("python_bench.json") + rs = _load("rust_bench.json") + if py is None or rs is None: + print("Missing results β€” run run_benchmark.sh first.") + return + + rows = [ + ("implementation", py["implementation"], rs["implementation"]), + ("persist (sqlite)", "yes", "yes" if rs.get("persist") else "no"), + ("metrics computed", "yes", "yes" if rs.get("metrics") else "no"), + ("backtests", py["n_backtests"], rs["n_backtests"]), + ("symbols", py["n_symbols"], rs["n_symbols"]), + ("years", py["years"], rs["years"]), + ("workers", py["workers"], rs["workers"]), + ("elapsed (s)", py["elapsed_seconds"], rs["elapsed_seconds"]), + ("backtests / s", py["throughput_backtests_per_second"], rs["throughput_backtests_per_second"]), + ("ms / backtest", py["wall_clock_per_backtest_ms"], rs["wall_clock_per_backtest_ms"]), + ] + width_metric = max(len(r[0]) for r in rows) + width_py = max(len(str(r[1])) for r in rows + [("python_framework", "python", "")]) + width_rs = max(len(str(r[2])) for r in rows + [("rust_reference", "rust", "")]) + sep = "+" + "-" * (width_metric + 2) + "+" + "-" * (width_py + 2) + "+" + "-" * (width_rs + 2) + "+" + print(sep) + print(f"| {'metric'.ljust(width_metric)} | {'python_framework'.ljust(width_py)} | {'rust_reference'.ljust(width_rs)} |") + print(sep) + for name, p, r in rows: + print(f"| {str(name).ljust(width_metric)} | {str(p).ljust(width_py)} | {str(r).ljust(width_rs)} |") + print(sep) + + # Prefer the throughput ratio: it stays meaningful even when Rust + # finishes in <1 ms (elapsed_seconds rounds to 0 in the JSON). + throughput_ratio = ( + rs["throughput_backtests_per_second"] + / py["throughput_backtests_per_second"] + if py["throughput_backtests_per_second"] + else 0.0 + ) + if rs["elapsed_seconds"] > 0: + speedup = py["elapsed_seconds"] / rs["elapsed_seconds"] + else: + # Reconstruct from per-backtest wall-clock for tiny Rust runs. + speedup = ( + py["wall_clock_per_backtest_ms"] + / rs["wall_clock_per_backtest_ms"] + if rs["wall_clock_per_backtest_ms"] + else throughput_ratio + ) + print() + print(f"Wall-clock speedup (rust/python): {speedup:>10.1f}x") + print(f"Throughput ratio (rust/python): {throughput_ratio:>10.1f}x") + print() + print("This is the *ceiling* for a hypothetical iaf-core Rust kernel.") + print("Both sides persist orders + portfolio snapshots to a per-backtest") + print("SQLite DB *and* compute the standard metric pack (total return,") + print("CAGR, vol, Sharpe, Sortino, max-DD + duration, Calmar). The Python") + print("side additionally carries strategy lifecycle, data providers and") + print("the full BacktestSummaryMetrics pipeline; the Rust side is a") + print("minimal hand-rolled reference. The gap is the optimization") + print("headroom for issues #521-#526.") + + +if __name__ == "__main__": + main() diff --git a/examples/rust_vs_python_benchmark/generate_data.py b/examples/rust_vs_python_benchmark/generate_data.py new file mode 100644 index 00000000..566019bd --- /dev/null +++ b/examples/rust_vs_python_benchmark/generate_data.py @@ -0,0 +1,138 @@ +""" +Generate synthetic OHLCV data for the Python-vs-Rust benchmark. + +Geometric Brownian motion with realistic-ish volatility per asset class. +Self-contained, deterministic (seeded), no network or API keys required. + +Output layout:: + + data/ + BTC-USD.parquet # Rust side reads parquet directly + BTC-USD.csv # Framework CSVOHLCVDataProvider format + ... + +Each parquet/CSV has columns: ``Datetime, Open, High, Low, Close, Volume``. +""" +from __future__ import annotations + +import argparse +from datetime import datetime, timedelta, timezone +from pathlib import Path + +import numpy as np +import pandas as pd + + +# (symbol, annual_drift, annual_vol, start_price) +ASSETS = [ + ("BTC-USD", 0.40, 0.65, 30_000.0), + ("ETH-USD", 0.30, 0.80, 1_800.0), + ("SOL-USD", 0.25, 1.10, 25.0), + ("ADA-USD", 0.10, 0.95, 0.30), + ("DOT-USD", 0.08, 0.90, 5.50), +] + + +def _gbm_ohlcv( + n_bars: int, dt_hours: float, mu: float, sigma: float, s0: float, rng +) -> pd.DataFrame: + """One geometric Brownian motion path turned into a synthetic OHLCV.""" + dt = dt_hours / (365.25 * 24.0) + drift = (mu - 0.5 * sigma * sigma) * dt + diffusion = sigma * np.sqrt(dt) + + # Close path + increments = drift + diffusion * rng.standard_normal(n_bars) + log_closes = np.log(s0) + np.cumsum(increments) + closes = np.exp(log_closes) + + # Opens are previous close (first open == s0) + opens = np.empty(n_bars) + opens[0] = s0 + opens[1:] = closes[:-1] + + # Intra-bar high/low: random fraction of bar range above/below max(open, close) + # Use a small extra noise so ranges are non-trivial + bar_noise = sigma * np.sqrt(dt) * 0.5 + upper = rng.random(n_bars) * bar_noise + lower = rng.random(n_bars) * bar_noise + base = np.maximum(opens, closes) + floor = np.minimum(opens, closes) + highs = base * (1.0 + upper) + lows = floor * (1.0 - lower) + + # Synthetic volume β€” lognormal, scaled by price + volumes = rng.lognormal(mean=np.log(1e6 / s0), sigma=0.4, size=n_bars) + + return pd.DataFrame( + { + "Open": opens, + "High": highs, + "Low": lows, + "Close": closes, + "Volume": volumes, + } + ) + + +def generate( + out_dir: Path, years: float, time_frame_hours: int, seed: int +) -> list[Path]: + out_dir.mkdir(parents=True, exist_ok=True) + n_bars = int(years * 365.25 * 24 / time_frame_hours) + end = datetime(2024, 12, 31, 23, 0, tzinfo=timezone.utc) + start = end - timedelta(hours=time_frame_hours * (n_bars - 1)) + index = pd.date_range( + start=start, end=end, periods=n_bars, tz="UTC", name="Datetime" + ) + + written: list[Path] = [] + rng_master = np.random.default_rng(seed) + for symbol, mu, sigma, s0 in ASSETS: + rng = np.random.default_rng(rng_master.integers(0, 2**31 - 1)) + df = _gbm_ohlcv(n_bars, time_frame_hours, mu, sigma, s0, rng) + df.index = index + parquet_path = out_dir / f"{symbol}.parquet" + df.to_parquet(parquet_path) + # Framework CSVOHLCVDataProvider expects a `Datetime` column + # parseable by polars (ISO-8601 string is fine; it casts to + # ms-precision UTC internally). + csv_path = out_dir / f"{symbol}.csv" + df_csv = df.reset_index().rename(columns={"index": "Datetime"}) + df_csv["Datetime"] = df_csv["Datetime"].dt.strftime( + "%Y-%m-%d %H:%M:%S" + ) + df_csv.to_csv(csv_path, index=False) + written.append(parquet_path) + print( + f" wrote {parquet_path.name:<14} + {csv_path.name:<14} " + f"bars={len(df):>7,d} " + f"first_close={df['Close'].iloc[0]:>10,.4f} " + f"last_close={df['Close'].iloc[-1]:>10,.4f}" + ) + return written + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--out-dir", type=Path, default=Path("data")) + parser.add_argument( + "--years", type=float, default=10.0, + help="Years of history per symbol (default: 10)." + ) + parser.add_argument( + "--time-frame-hours", type=int, default=1, + help="Bar size in hours (default: 1 = hourly)." + ) + parser.add_argument("--seed", type=int, default=42) + args = parser.parse_args() + + print( + f"Generating {args.years}y of {args.time_frame_hours}h bars " + f"for {len(ASSETS)} symbols into {args.out_dir}/ ..." + ) + generate(args.out_dir, args.years, args.time_frame_hours, args.seed) + + +if __name__ == "__main__": + main() diff --git a/examples/rust_vs_python_benchmark/python_bench.py b/examples/rust_vs_python_benchmark/python_bench.py new file mode 100644 index 00000000..82f32b7f --- /dev/null +++ b/examples/rust_vs_python_benchmark/python_bench.py @@ -0,0 +1,319 @@ +""" +Python benchmark β€” runs a parameter sweep through the +investing-algorithm-framework's vectorized backtest engine +(`app.run_vector_backtests`). + +This represents the *current* end-user code path: framework data +providers, the `TradingStrategy` lifecycle, signal generation in +pandas, vectorised execution, metric calculation and bundle write. +The wall-clock reported here is the baseline that future +Rust-accelerated kernels (epic #521 / iaf-core) need to beat. + +Usage:: + + python python_bench.py --combos 25 --workers 1 --years 10 +""" +from __future__ import annotations + +import argparse +import json +import time +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Dict + +import numpy as np +import pandas as pd + +from investing_algorithm_framework import ( + BacktestDateRange, + DataSource, + DataType, + PositionSize, + TimeUnit, + TradingCost, + TradingStrategy, + create_app, +) +from investing_algorithm_framework.infrastructure import ( + CSVOHLCVDataProvider, +) + + +REPO_DIR = Path(__file__).resolve().parent +DATA_DIR = REPO_DIR / "data" +RESULTS_DIR = REPO_DIR / "results" +BACKTEST_DIR = REPO_DIR / "backtest_results" + +SYMBOLS = ["BTC-USD", "ETH-USD", "SOL-USD", "ADA-USD", "DOT-USD"] + + +def _ema(series: pd.Series, period: int) -> pd.Series: + return series.ewm(span=period, adjust=False).mean() + + +def _rsi(series: pd.Series, period: int) -> pd.Series: + delta = series.diff() + gain = delta.clip(lower=0.0) + loss = -delta.clip(upper=0.0) + avg_gain = gain.ewm(alpha=1.0 / period, adjust=False).mean() + avg_loss = loss.ewm(alpha=1.0 / period, adjust=False).mean() + rs = avg_gain / avg_loss.replace(0.0, np.nan) + rsi = 100.0 - (100.0 / (1.0 + rs)) + return rsi.fillna(50.0) + + +class EmaRsiSweepStrategy(TradingStrategy): + """Long-only EMA crossover with RSI oversold confirmation.""" + + time_unit = TimeUnit.HOUR + interval = 1 + + def __init__( + self, + algorithm_id: str, + symbols: list[str], + trading_symbol: str, + ema_short: int, + ema_long: int, + rsi_period: int, + rsi_oversold: float, + rsi_overbought: float, + market: str = "BITVAVO", + fee_pct: float = 0.1, + slippage_pct: float = 0.05, + ) -> None: + warmup = max(ema_short, ema_long, rsi_period) + 10 + data_sources = [ + DataSource( + identifier=f"ohlcv_{s}", + data_type=DataType.OHLCV, + time_frame="1h", + market=market, + symbol=f"{s}/USD", + warmup_window=warmup, + pandas=True, + ) + for s in symbols + ] + position_sizes = [ + PositionSize(symbol=s, percentage_of_portfolio=1 / len(symbols)) + for s in symbols + ] + trading_costs = [ + TradingCost( + symbol=s, + fee_percentage=fee_pct, + slippage_percentage=slippage_pct, + ) + for s in symbols + ] + + super().__init__( + algorithm_id=algorithm_id, + symbols=symbols, + trading_symbol=trading_symbol, + data_sources=data_sources, + position_sizes=position_sizes, + trading_costs=trading_costs, + time_unit=self.time_unit, + interval=self.interval, + ) + self.ema_short = ema_short + self.ema_long = ema_long + self.rsi_period = rsi_period + self.rsi_oversold = rsi_oversold + self.rsi_overbought = rsi_overbought + self.set_parameters( + { + "ema_short": ema_short, + "ema_long": ema_long, + "rsi_period": rsi_period, + "rsi_oversold": rsi_oversold, + "rsi_overbought": rsi_overbought, + } + ) + + def generate_buy_signals( + self, data: Dict[str, Any] + ) -> Dict[str, pd.Series]: + signals: Dict[str, pd.Series] = {} + for symbol in self.symbols: + df = data[f"ohlcv_{symbol}"] + close = df["Close"] + ema_s = _ema(close, self.ema_short) + ema_l = _ema(close, self.ema_long) + rsi_v = _rsi(close, self.rsi_period) + crossover = (ema_s > ema_l) & (ema_s.shift(1) <= ema_l.shift(1)) + sig = crossover & (rsi_v < self.rsi_oversold) + signals[symbol] = sig.fillna(False).astype(bool) + return signals + + def generate_sell_signals( + self, data: Dict[str, Any] + ) -> Dict[str, pd.Series]: + signals: Dict[str, pd.Series] = {} + for symbol in self.symbols: + df = data[f"ohlcv_{symbol}"] + close = df["Close"] + ema_s = _ema(close, self.ema_short) + ema_l = _ema(close, self.ema_long) + rsi_v = _rsi(close, self.rsi_period) + crossunder = (ema_s < ema_l) & (ema_s.shift(1) >= ema_l.shift(1)) + sig = crossunder & (rsi_v > self.rsi_overbought) + signals[symbol] = sig.fillna(False).astype(bool) + return signals + + +def _build_param_grid(n_combos: int) -> list[dict]: + rng = np.random.default_rng(0) + short_choices = [10, 15, 20, 25, 30] + long_choices = [50, 75, 100, 150, 200] + rsi_periods = [7, 14, 21] + rsi_oversold = [25, 30, 35] + rsi_overbought = [65, 70, 75] + grid = [] + seen = set() + while len(grid) < n_combos: + es = int(rng.choice(short_choices)) + el = int(rng.choice(long_choices)) + if el <= es: + continue + rp = int(rng.choice(rsi_periods)) + ro = int(rng.choice(rsi_oversold)) + rb = int(rng.choice(rsi_overbought)) + key = (es, el, rp, ro, rb) + if key in seen: + continue + seen.add(key) + grid.append( + { + "ema_short": es, + "ema_long": el, + "rsi_period": rp, + "rsi_oversold": ro, + "rsi_overbought": rb, + } + ) + return grid + + +def _data_date_bounds() -> tuple[datetime, datetime]: + """Read the actual first/last datetime present in the generated CSVs.""" + df = pd.read_csv(DATA_DIR / f"{SYMBOLS[0]}.csv", usecols=["Datetime"]) + start = pd.to_datetime(df["Datetime"].iloc[0]).tz_localize("UTC") + end = pd.to_datetime(df["Datetime"].iloc[-1]).tz_localize("UTC") + return start.to_pydatetime(), end.to_pydatetime() + + +def _build_app(years: float) -> tuple: + app = create_app() + + # Register one CSV provider per symbol so the framework treats + # them as a normal market data source. Use a real market label + # to avoid the framework trying to look "SYNTHETIC" up in CCXT. + market = "BITVAVO" + for symbol in SYMBOLS: + app.add_data_provider( + CSVOHLCVDataProvider( + storage_path=str(DATA_DIR / f"{symbol}.csv"), + symbol=f"{symbol}/USD", + time_frame="1h", + market=market, + ) + ) + + app.add_market( + market=market, + trading_symbol="USD", + initial_balance=10_000, + ) + + data_start, data_end = _data_date_bounds() + # Back end off so the framework can still request the next bar + # for fills (it asks for `> end`). + safe_end = data_end - pd.Timedelta(hours=24).to_pytimedelta() + desired_start = safe_end - pd.Timedelta( + days=int(years * 365.25) + ).to_pytimedelta() + # Clamp to actual data start with a warmup margin so that the + # framework's per-strategy warmup_window (~210 hours for our + # largest indicator) doesn't slide the first requested bar + # outside the data range. + safe_start = data_start + pd.Timedelta(days=20).to_pytimedelta() + start = max(safe_start, desired_start) + date_range = BacktestDateRange( + start_date=start, end_date=safe_end, name=f"{int(years)}y" + ) + return app, date_range, market + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--combos", type=int, default=25) + parser.add_argument("--workers", type=int, default=1) + parser.add_argument("--years", type=float, default=10.0) + parser.add_argument( + "--results-file", type=Path, + default=RESULTS_DIR / "python_bench.json" + ) + args = parser.parse_args() + + if not (DATA_DIR / f"{SYMBOLS[0]}.csv").exists(): + raise SystemExit( + "data/ is empty β€” run `python generate_data.py` first." + ) + RESULTS_DIR.mkdir(parents=True, exist_ok=True) + + app, date_range, market = _build_app(args.years) + grid = _build_param_grid(args.combos) + strategies = [ + EmaRsiSweepStrategy( + algorithm_id=f"sweep_{i:03d}", + symbols=SYMBOLS, + trading_symbol="USD", + market=market, + **params, + ) + for i, params in enumerate(grid) + ] + print( + f"Python framework benchmark: {len(strategies)} strategies Γ— " + f"{len(SYMBOLS)} symbols, {args.years}y of hourly data, " + f"workers={args.workers}" + ) + + if BACKTEST_DIR.exists(): + import shutil + shutil.rmtree(BACKTEST_DIR) + + t0 = time.perf_counter() + backtests = app.run_vector_backtests( + strategies=strategies, + backtest_date_ranges=[date_range], + n_workers=args.workers, + backtest_storage_directory=str(BACKTEST_DIR), + show_progress=True, + ) + elapsed = time.perf_counter() - t0 + + n = len(backtests) + summary = { + "implementation": "python_framework", + "n_strategies": len(strategies), + "n_symbols": len(SYMBOLS), + "n_backtests": n, + "years": args.years, + "workers": args.workers, + "elapsed_seconds": round(elapsed, 3), + "throughput_backtests_per_second": round(n / elapsed, 3) if elapsed else 0.0, + "wall_clock_per_backtest_ms": round(elapsed / n * 1000, 1) if n else 0.0, + } + args.results_file.parent.mkdir(parents=True, exist_ok=True) + args.results_file.write_text(json.dumps(summary, indent=2)) + print() + print(json.dumps(summary, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/examples/rust_vs_python_benchmark/run_benchmark.sh b/examples/rust_vs_python_benchmark/run_benchmark.sh new file mode 100755 index 00000000..ecdd7ed6 --- /dev/null +++ b/examples/rust_vs_python_benchmark/run_benchmark.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Orchestrate the Python-vs-Rust benchmark end-to-end. +# +# Steps: +# 1. Generate synthetic OHLCV data (parquet + CSV) if missing +# 2. Build the Rust binary in release mode if missing +# 3. Run the Python framework benchmark +# 4. Run the Rust reference benchmark +# 5. Print a side-by-side summary table +set -euo pipefail + +cd "$(dirname "$0")" + +YEARS="${YEARS:-10}" +COMBOS="${COMBOS:-25}" +PY_WORKERS="${PY_WORKERS:-1}" +RUST_WORKERS="${RUST_WORKERS:-0}" # 0 = all cores +RUST_PERSIST="${RUST_PERSIST:-true}" # mirror framework's per-backtest SQLite +RUST_METRICS="${RUST_METRICS:-true}" # mirror framework's BacktestMetrics pass + +mkdir -p data results + +if [[ ! -f "data/BTC-USD.parquet" ]]; then + echo "==> Generating synthetic data (${YEARS}y hourly Γ— 5 symbols)" + python generate_data.py --years "${YEARS}" +fi + +if [[ ! -x "rust_bench/target/release/rust_bench" ]]; then + echo "==> Building rust_bench (release; first build can take a few minutes)" + ( cd rust_bench && cargo build --release ) +fi + +echo +echo "==> Running Python framework benchmark" +python python_bench.py --combos "${COMBOS}" --years "${YEARS}" --workers "${PY_WORKERS}" + +echo +echo "==> Running Rust reference benchmark (persist=${RUST_PERSIST})" +( cd rust_bench && ./target/release/rust_bench \ + --combos "${COMBOS}" \ + --years "${YEARS}" \ + --workers "${RUST_WORKERS}" \ + --persist "${RUST_PERSIST}" \ + --metrics "${RUST_METRICS}" ) + +echo +echo "==> Comparison" +python compare.py diff --git a/examples/rust_vs_python_benchmark/rust_bench/Cargo.toml b/examples/rust_vs_python_benchmark/rust_bench/Cargo.toml new file mode 100644 index 00000000..9b05b349 --- /dev/null +++ b/examples/rust_vs_python_benchmark/rust_bench/Cargo.toml @@ -0,0 +1,24 @@ +[package] +name = "rust_bench" +version = "0.1.0" +edition = "2021" +description = "Reference Rust backtest sweep for the iaf benchmark baseline." + +[dependencies] +polars = { version = "0.46", features = ["lazy", "parquet", "temporal"] } +rayon = "1.10" +serde = { version = "1", features = ["derive"] } +serde_json = "1" +clap = { version = "4", features = ["derive"] } +chrono = "0.4" +anyhow = "1" +# Bundled SQLite so the benchmark is self-contained on macOS/Linux/Windows +# without requiring a system libsqlite. The framework writes a per-backtest +# `.sqlite3` (orders, snapshots, positions, ...) so we mirror that here to +# keep the comparison apples-to-apples. +rusqlite = { version = "0.32", features = ["bundled"] } + +[profile.release] +opt-level = 3 +lto = "thin" +codegen-units = 1 diff --git a/examples/rust_vs_python_benchmark/rust_bench/src/main.rs b/examples/rust_vs_python_benchmark/rust_bench/src/main.rs new file mode 100644 index 00000000..98f7eb3e --- /dev/null +++ b/examples/rust_vs_python_benchmark/rust_bench/src/main.rs @@ -0,0 +1,696 @@ +//! Reference Rust backtest sweep β€” minimal end-to-end equivalent of +//! `python_bench.py`. Loads parquet OHLCV files, runs an EMA + RSI +//! crossover strategy across a parameter sweep on N symbols, and +//! prints throughput as JSON. +//! +//! This is the **best-case ceiling** for what an `iaf-core` Rust +//! kernel could achieve. The Python side carries the full framework +//! (data providers, strategy lifecycle, bundle write, metric +//! aggregation). The gap between the two is the optimization +//! headroom for issues #521–#526. + +use std::path::{Path, PathBuf}; +use std::time::Instant; + +use anyhow::{Context as _, Result}; +use clap::Parser; +use polars::prelude::*; +use rayon::prelude::*; +use rusqlite::{params, Connection}; +use serde::Serialize; + +const SYMBOLS: &[&str] = &["BTC-USD", "ETH-USD", "SOL-USD", "ADA-USD", "DOT-USD"]; + +#[derive(Parser, Debug)] +#[command(about = "Rust reference backtest sweep")] +struct Args { + #[arg(long, default_value_t = 25)] + combos: usize, + #[arg(long, default_value_t = 10.0)] + years: f64, + /// 0 = use all logical CPUs. + #[arg(long, default_value_t = 0)] + workers: usize, + #[arg(long, default_value = "../data")] + data_dir: PathBuf, + #[arg(long, default_value = "../results/rust_bench.json")] + results_file: PathBuf, + /// Persist every order + per-bar portfolio snapshot to a per-backtest + /// SQLite database (mirrors the Python framework's behavior). + #[arg(long, default_value_t = true, action = clap::ArgAction::Set)] + persist: bool, + /// Compute the standard metric pack (total return, CAGR, vol, Sharpe, + /// Sortino, max drawdown + duration, Calmar) over the per-bar equity + /// series of every backtest. Mirrors `BacktestMetrics` in the Python + /// framework. Disable with `--metrics false` to isolate pure compute. + #[arg(long, default_value_t = true, action = clap::ArgAction::Set)] + metrics: bool, + /// Annualisation factor for risk metrics. Defaults to hourly bars + /// (24 * 365 = 8760), matching the synthetic data we generate. + #[arg(long, default_value_t = 8760.0)] + bars_per_year: f64, + /// Annual risk-free rate used by Sharpe / Sortino. Same default as + /// the framework (0%). + #[arg(long, default_value_t = 0.0)] + risk_free_rate: f64, + /// Directory for per-backtest `.sqlite3` files (one per strategy). + #[arg(long, default_value = "../results/rust_dbs")] + db_dir: PathBuf, +} + +#[derive(Clone, Copy, Debug)] +struct Params { + ema_short: usize, + ema_long: usize, + rsi_period: usize, + rsi_oversold: f64, + rsi_overbought: f64, +} + +#[derive(Serialize)] +struct BenchSummary { + implementation: &'static str, + persist: bool, + metrics: bool, + n_strategies: usize, + n_symbols: usize, + n_backtests: usize, + years: f64, + workers: usize, + elapsed_seconds: f64, + throughput_backtests_per_second: f64, + wall_clock_per_backtest_ms: f64, +} + +/// Open a per-backtest SQLite DB with the same kind of tables the +/// framework persists (orders + portfolio snapshots). PRAGMAs match a +/// reasonable "backtest write-heavy" config: WAL + NORMAL sync, which +/// is roughly what SQLAlchemy gives us out of the box on a fresh file. +fn open_backtest_db(path: &Path) -> Result { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent).ok(); + } + if path.exists() { + std::fs::remove_file(path).ok(); + } + let conn = Connection::open(path)?; + conn.execute_batch( + r#" + PRAGMA journal_mode = WAL; + PRAGMA synchronous = NORMAL; + PRAGMA temp_store = MEMORY; + CREATE TABLE orders ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + symbol TEXT NOT NULL, + side TEXT NOT NULL, + price REAL NOT NULL, + amount REAL NOT NULL, + fee REAL NOT NULL, + bar_index INTEGER NOT NULL + ); + CREATE TABLE portfolio_snapshots ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + symbol TEXT NOT NULL, + bar_index INTEGER NOT NULL, + unallocated REAL NOT NULL, + position_units REAL NOT NULL, + total_value REAL NOT NULL + ); + CREATE TABLE backtest_metrics ( + name TEXT PRIMARY KEY, + value REAL + ); + "#, + )?; + Ok(conn) +} + +fn build_param_grid(n: usize) -> Vec { + // Same deterministic enumeration as python_bench so results are + // comparable: same seed, same candidate sets, same uniqueness rule. + let short_choices = [10usize, 15, 20, 25, 30]; + let long_choices = [50usize, 75, 100, 150, 200]; + let rsi_periods = [7usize, 14, 21]; + let rsi_oversold = [25.0, 30.0, 35.0]; + let rsi_overbought = [65.0, 70.0, 75.0]; + + // tiny xorshift seeded with 0 to mirror numpy default_rng(0) ordering + // closely enough for benchmarking purposes (we only need *some* + // realistic spread of params, not bit-identical to Python). + let mut state: u64 = 0xDEAD_BEEF_CAFE_BABE; + let mut next = || { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + state + }; + let pick = |state: &mut dyn FnMut() -> u64, n: usize| -> usize { + (state() as usize) % n + }; + + let mut grid: Vec = Vec::with_capacity(n); + let mut seen = std::collections::HashSet::new(); + while grid.len() < n { + let es = short_choices[pick(&mut next, short_choices.len())]; + let el = long_choices[pick(&mut next, long_choices.len())]; + if el <= es { + continue; + } + let rp = rsi_periods[pick(&mut next, rsi_periods.len())]; + let ro = rsi_oversold[pick(&mut next, rsi_oversold.len())]; + let rb = rsi_overbought[pick(&mut next, rsi_overbought.len())]; + let key = (es, el, rp, ro as i64, rb as i64); + if !seen.insert(key) { + continue; + } + grid.push(Params { + ema_short: es, + ema_long: el, + rsi_period: rp, + rsi_oversold: ro, + rsi_overbought: rb, + }); + } + grid +} + +fn load_close(path: &std::path::Path, last_n: usize) -> Result<(Vec, Vec)> { + let df = LazyFrame::scan_parquet(path, ScanArgsParquet::default())? + .select([col("Open"), col("Close")]) + .collect() + .with_context(|| format!("scan_parquet({:?})", path))?; + + let opens = df.column("Open")?.f64()?.into_no_null_iter().collect::>(); + let closes = df.column("Close")?.f64()?.into_no_null_iter().collect::>(); + + let len = closes.len(); + let start = len.saturating_sub(last_n); + Ok((opens[start..].to_vec(), closes[start..].to_vec())) +} + +/// EMA seeded with the first close (matches pandas `ewm(span=p, adjust=False)` +/// well enough for benchmark purposes; final equity drift vs python should +/// be small). +fn ema(values: &[f64], period: usize) -> Vec { + let alpha = 2.0 / (period as f64 + 1.0); + let mut out = Vec::with_capacity(values.len()); + if values.is_empty() { + return out; + } + let mut prev = values[0]; + out.push(prev); + for &v in &values[1..] { + prev = alpha * v + (1.0 - alpha) * prev; + out.push(prev); + } + out +} + +/// Wilder RSI. NaN-equivalent values are clamped to 50 (neutral) so a +/// naive sweep does not generate spurious signals during warmup. +fn rsi(values: &[f64], period: usize) -> Vec { + let n = values.len(); + let mut out = vec![50.0; n]; + if n < period + 1 { + return out; + } + let alpha = 1.0 / period as f64; + let mut avg_gain = 0.0; + let mut avg_loss = 0.0; + for i in 1..n { + let delta = values[i] - values[i - 1]; + let gain = delta.max(0.0); + let loss = (-delta).max(0.0); + if i == 1 { + avg_gain = gain; + avg_loss = loss; + } else { + avg_gain = alpha * gain + (1.0 - alpha) * avg_gain; + avg_loss = alpha * loss + (1.0 - alpha) * avg_loss; + } + let rs = if avg_loss > 0.0 { avg_gain / avg_loss } else { 0.0 }; + out[i] = 100.0 - 100.0 / (1.0 + rs); + } + out +} + +#[derive(Default)] +struct BacktestStats { + final_equity: f64, + trades: usize, + wins: usize, + /// Per-bar portfolio equity (cash + mark-to-market) for this symbol. + /// Always populated; the per-backtest equity curve is the bar-wise + /// sum across symbols and is what the metric pack is computed on. + equity: Vec, +} + +/// Metric pack mirroring the framework's `BacktestMetrics`. Values are +/// computed once per backtest over the per-bar portfolio equity curve +/// (sum of all symbols' equity at each bar). +#[derive(Debug, Default, Serialize)] +struct BacktestMetrics { + total_return_pct: f64, + cagr: f64, + annual_volatility: f64, + sharpe_ratio: f64, + sortino_ratio: f64, + max_drawdown: f64, + max_drawdown_duration_bars: usize, + calmar_ratio: f64, +} + +fn compute_metrics( + equity: &[f64], + bars_per_year: f64, + risk_free_rate: f64, + years: f64, +) -> BacktestMetrics { + let n = equity.len(); + if n < 2 { + return BacktestMetrics::default(); + } + let start = equity[0].max(1e-12); + let end = *equity.last().unwrap(); + let total_return = end / start - 1.0; + + // Per-bar log returns + let mut returns: Vec = Vec::with_capacity(n - 1); + for i in 1..n { + let prev = equity[i - 1]; + if prev > 0.0 { + returns.push(equity[i] / prev - 1.0); + } else { + returns.push(0.0); + } + } + + let mean: f64 = returns.iter().sum::() / returns.len() as f64; + let var: f64 = + returns.iter().map(|r| (r - mean).powi(2)).sum::() / returns.len() as f64; + let std = var.sqrt(); + let annual_vol = std * bars_per_year.sqrt(); + + // CAGR over the actual elapsed years + let cagr = if years > 0.0 && start > 0.0 && end > 0.0 { + (end / start).powf(1.0 / years) - 1.0 + } else { + 0.0 + }; + + // Sharpe: (mean_excess_return * bars_per_year) / annual_vol + let rf_per_bar = risk_free_rate / bars_per_year; + let mean_excess = mean - rf_per_bar; + let sharpe = if annual_vol > 0.0 { + (mean_excess * bars_per_year) / annual_vol + } else { + 0.0 + }; + + // Sortino: only count downside deviation + let downside: f64 = returns + .iter() + .map(|r| if *r < rf_per_bar { (r - rf_per_bar).powi(2) } else { 0.0 }) + .sum::() + / returns.len() as f64; + let downside_std = downside.sqrt() * bars_per_year.sqrt(); + let sortino = if downside_std > 0.0 { + (mean_excess * bars_per_year) / downside_std + } else { + 0.0 + }; + + // Max drawdown + duration (single pass over the equity curve) + let mut peak = equity[0]; + let mut peak_idx = 0usize; + let mut max_dd = 0.0_f64; + let mut max_dd_dur = 0usize; + for (i, &v) in equity.iter().enumerate() { + if v > peak { + peak = v; + peak_idx = i; + } + if peak > 0.0 { + let dd = (v - peak) / peak; // negative + if dd < max_dd { + max_dd = dd; + } + } + let dur = i - peak_idx; + if dur > max_dd_dur { + max_dd_dur = dur; + } + } + + let calmar = if max_dd < 0.0 { cagr / -max_dd } else { 0.0 }; + + BacktestMetrics { + total_return_pct: total_return, + cagr, + annual_volatility: annual_vol, + sharpe_ratio: sharpe, + sortino_ratio: sortino, + max_drawdown: max_dd, + max_drawdown_duration_bars: max_dd_dur, + calmar_ratio: calmar, + } +} + +/// Run one symbol's worth of the backtest. Fees + slippage are charged +/// on entry and exit. Long-only, single-position-per-symbol. +/// +/// When `persist` is true, every executed order and every per-bar +/// portfolio snapshot is written through the supplied prepared +/// statements. The caller is expected to have wrapped these in a +/// single SQLite transaction. +fn run_one_symbol( + symbol: &str, + opens: &[f64], + closes: &[f64], + p: &Params, + capital: f64, + persist: Option<&mut PersistStmts<'_>>, +) -> BacktestStats { + let ema_s = ema(closes, p.ema_short); + let ema_l = ema(closes, p.ema_long); + let rsi_v = rsi(closes, p.rsi_period); + + const FEE: f64 = 0.001; // 0.1% + const SLIP: f64 = 0.0005; // 0.05% + + let mut cash = capital; + let mut units = 0.0_f64; + let mut entry_value: f64 = 0.0; + let mut wins = 0usize; + let mut trades = 0usize; + + let n = closes.len(); + if n < 3 { + return BacktestStats { + final_equity: cash, + trades, + wins, + equity: vec![cash; n], + }; + } + + let mut persist = persist; + let mut equity = Vec::with_capacity(n); + // Bars 0 and 1 are warmup (no signal yet); record initial cash. + equity.push(cash); + equity.push(cash); + + for i in 2..n { + // Use previous bar's signal, fill at this bar's open + let cross_over = + ema_s[i - 1] > ema_l[i - 1] && ema_s[i - 2] <= ema_l[i - 2]; + let cross_under = + ema_s[i - 1] < ema_l[i - 1] && ema_s[i - 2] >= ema_l[i - 2]; + let buy = cross_over && rsi_v[i - 1] < p.rsi_oversold && units == 0.0; + let sell = cross_under && rsi_v[i - 1] > p.rsi_overbought && units > 0.0; + let fill = opens[i]; + if buy { + let exec_price = fill * (1.0 + SLIP); + let notional = cash; + let fee = notional * FEE; + let invest = notional - fee; + units = invest / exec_price; + cash = 0.0; + entry_value = invest; + if let Some(ps) = persist.as_deref_mut() { + ps.order + .execute(params![symbol, "buy", exec_price, units, fee, i as i64]) + .ok(); + } + } else if sell { + let exec_price = fill * (1.0 - SLIP); + let proceeds = units * exec_price; + let fee = proceeds * FEE; + cash = proceeds - fee; + if cash > entry_value { + wins += 1; + } + trades += 1; + if let Some(ps) = persist.as_deref_mut() { + ps.order + .execute(params![symbol, "sell", exec_price, units, fee, i as i64]) + .ok(); + } + units = 0.0; + entry_value = 0.0; + } + + let total_value = cash + units * closes[i]; + equity.push(total_value); + if let Some(ps) = persist.as_deref_mut() { + ps.snap + .execute(params![symbol, i as i64, cash, units, total_value]) + .ok(); + } + } + + // Mark to market at final close + let final_equity = cash + units * closes[n - 1]; + BacktestStats { final_equity, trades, wins, equity } +} + +/// Prepared statements for the hot inner loop. Borrowed from a +/// transaction so all writes commit atomically when it drops. +struct PersistStmts<'a> { + order: rusqlite::Statement<'a>, + snap: rusqlite::Statement<'a>, +} + +fn run_one_backtest( + backtest_idx: usize, + symbols_data: &[(String, Vec, Vec)], + p: &Params, + capital_per_symbol: f64, + db_dir: Option<&Path>, + metrics_cfg: Option, +) -> BacktestStats { + let mut total_equity = 0.0; + let mut total_trades = 0; + let mut total_wins = 0; + let mut portfolio_curve: Vec = Vec::new(); + + let merge = |portfolio_curve: &mut Vec, sym_eq: &[f64]| { + if portfolio_curve.is_empty() { + portfolio_curve.extend_from_slice(sym_eq); + } else { + let m = portfolio_curve.len().min(sym_eq.len()); + for i in 0..m { + portfolio_curve[i] += sym_eq[i]; + } + } + }; + + if let Some(dir) = db_dir { + // Persisting path: one SQLite file per backtest, all writes in + // a single transaction (matches typical SQLAlchemy session use). + let path = dir.join(format!("backtest_{:04}.sqlite3", backtest_idx)); + let mut conn = open_backtest_db(&path).expect("open backtest db"); + let tx = conn.transaction().expect("begin tx"); + { + let mut ps = PersistStmts { + order: tx + .prepare( + "INSERT INTO orders (symbol, side, price, amount, fee, bar_index) \ + VALUES (?, ?, ?, ?, ?, ?)", + ) + .expect("prep order"), + snap: tx + .prepare( + "INSERT INTO portfolio_snapshots \ + (symbol, bar_index, unallocated, position_units, total_value) \ + VALUES (?, ?, ?, ?, ?)", + ) + .expect("prep snap"), + }; + for (sym, opens, closes) in symbols_data { + let stats = run_one_symbol(sym, opens, closes, p, capital_per_symbol, Some(&mut ps)); + total_equity += stats.final_equity; + total_trades += stats.trades; + total_wins += stats.wins; + merge(&mut portfolio_curve, &stats.equity); + } + } + // Compute and persist metrics inside the same transaction so the + // "backtest write cost" we measure includes the metric round-trip + // (Python framework writes metrics too). + if let Some(cfg) = metrics_cfg { + let m = compute_metrics( + &portfolio_curve, + cfg.bars_per_year, + cfg.risk_free_rate, + cfg.years, + ); + let mut stmt = tx + .prepare("INSERT INTO backtest_metrics (name, value) VALUES (?, ?)") + .expect("prep metric"); + for (name, val) in [ + ("total_return_pct", m.total_return_pct), + ("cagr", m.cagr), + ("annual_volatility", m.annual_volatility), + ("sharpe_ratio", m.sharpe_ratio), + ("sortino_ratio", m.sortino_ratio), + ("max_drawdown", m.max_drawdown), + ("max_drawdown_duration_bars", m.max_drawdown_duration_bars as f64), + ("calmar_ratio", m.calmar_ratio), + ] { + stmt.execute(params![name, val]).ok(); + } + } + tx.commit().expect("commit"); + } else { + for (sym, opens, closes) in symbols_data { + let stats = run_one_symbol(sym, opens, closes, p, capital_per_symbol, None); + total_equity += stats.final_equity; + total_trades += stats.trades; + total_wins += stats.wins; + merge(&mut portfolio_curve, &stats.equity); + } + // Still compute metrics in the no-persist path so the timing + // reflects the cost; just don't write them anywhere. + if let Some(cfg) = metrics_cfg { + let _ = compute_metrics( + &portfolio_curve, + cfg.bars_per_year, + cfg.risk_free_rate, + cfg.years, + ); + } + } + + BacktestStats { + final_equity: total_equity, + trades: total_trades, + wins: total_wins, + equity: portfolio_curve, + } +} + +#[derive(Clone, Copy)] +struct MetricsCfg { + bars_per_year: f64, + risk_free_rate: f64, + years: f64, +} + +fn main() -> Result<()> { + let args = Args::parse(); + + if args.workers > 0 { + rayon::ThreadPoolBuilder::new() + .num_threads(args.workers) + .build_global() + .ok(); + } + + let n_bars = (args.years * 365.25 * 24.0) as usize; + + println!( + "Rust benchmark: loading {} symbols Γ— {} bars from {:?}", + SYMBOLS.len(), + n_bars, + args.data_dir + ); + + let load_t0 = Instant::now(); + let symbols_data: Vec<(String, Vec, Vec)> = SYMBOLS + .iter() + .map(|s| { + let path = args.data_dir.join(format!("{}.parquet", s)); + let (opens, closes) = load_close(&path, n_bars).expect("load"); + ((*s).to_string(), opens, closes) + }) + .collect(); + println!(" loaded in {:.3}s", load_t0.elapsed().as_secs_f64()); + + let grid = build_param_grid(args.combos); + println!( + "Rust framework benchmark: {} strategies Γ— {} symbols, {}y of hourly data, workers={}", + grid.len(), + SYMBOLS.len(), + args.years, + if args.workers == 0 { rayon::current_num_threads() } else { args.workers } + ); + + let initial_total = 10_000.0_f64; + let capital_per_symbol = initial_total / SYMBOLS.len() as f64; + + let db_dir = if args.persist { + // Reset the DB dir so each run starts clean and we don't grow + // a huge backlog on disk between iterations. + let _ = std::fs::remove_dir_all(&args.db_dir); + std::fs::create_dir_all(&args.db_dir).ok(); + println!( + "Persistence: ON (writing per-backtest SQLite DBs to {:?})", + args.db_dir + ); + Some(args.db_dir.as_path()) + } else { + println!("Persistence: OFF (pure in-memory backtest)"); + None + }; + + let metrics_cfg = if args.metrics { + println!( + "Metrics: ON (Sharpe / Sortino / max-DD / Calmar / vol / CAGR; bars/yr={})", + args.bars_per_year + ); + Some(MetricsCfg { + bars_per_year: args.bars_per_year, + risk_free_rate: args.risk_free_rate, + years: args.years, + }) + } else { + println!("Metrics: OFF"); + None + }; + + let t0 = Instant::now(); + let results: Vec = grid + .par_iter() + .enumerate() + .map(|(idx, p)| run_one_backtest(idx, &symbols_data, p, capital_per_symbol, db_dir, metrics_cfg)) + .collect(); + let elapsed = t0.elapsed().as_secs_f64(); + + let n = results.len(); + let summary = BenchSummary { + implementation: "rust_reference", + persist: args.persist, + metrics: args.metrics, + n_strategies: n, + n_symbols: SYMBOLS.len(), + n_backtests: n, + years: args.years, + workers: if args.workers == 0 { rayon::current_num_threads() } else { args.workers }, + // Keep extra precision: a Rust sweep on small data can finish in + // sub-millisecond time, so rounding to 3 dp would zero it out. + elapsed_seconds: (elapsed * 1_000_000.0).round() / 1_000_000.0, + throughput_backtests_per_second: ((n as f64 / elapsed) * 1000.0).round() / 1000.0, + wall_clock_per_backtest_ms: ((elapsed / n as f64) * 1000.0 * 1000.0).round() / 1000.0, + }; + + if let Some(parent) = args.results_file.parent() { + std::fs::create_dir_all(parent).ok(); + } + let json = serde_json::to_string_pretty(&summary)?; + std::fs::write(&args.results_file, &json)?; + println!(); + println!("{}", json); + + // sanity: print a few final equities so we can eyeball parity + let mut sample: Vec<_> = results.iter().take(3).collect(); + sample.iter_mut().for_each(|_| {}); + let final_equities: Vec = results + .iter() + .take(5) + .map(|s| format!("${:.2} ({} trades)", s.final_equity, s.trades)) + .collect(); + eprintln!("first 5 final equities: {:?}", final_equities); + + Ok(()) +} diff --git a/examples/strategies_showcase/01_trend_following/README.md b/examples/strategies_showcase/01_trend_following/README.md new file mode 100644 index 00000000..6240296c --- /dev/null +++ b/examples/strategies_showcase/01_trend_following/README.md @@ -0,0 +1,40 @@ +# 01 β€” Trend Following (EMA crossover) + +**Fit:** βœ… Sweet spot for the framework's vector backtester. + +## Idea + +Classic single-asset trend follower: a fast EMA crossing above a slow EMA is +a *buy* signal, the inverse cross is a *sell*. No filters, no stops β€” just +the cleanest possible expression of the pattern so you can see how an +end-to-end vector backtest is wired up. + +## Why this fits the framework + +- One symbol, bar-aligned, no intra-bar logic β€” perfect for + `run_vector_backtest`. +- `generate_buy_signals` / `generate_sell_signals` return boolean `pd.Series` + per symbol, exactly what the vector engine expects. +- `pyindicators.ema` does the heavy lifting; no custom indicator code. + +## Parameters + +| Name | Default | Notes | +|------|---------|-------| +| `symbol` | `BTC/EUR` | Any spot pair on Bitvavo. | +| `time_frame` | `1d` | Daily bars keep the example fast. | +| `fast_period` | `20` | Fast EMA. | +| `slow_period` | `50` | Slow EMA. | + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` + +## Disclaimer + +The example is a *pattern reference*, not an alpha source. A trend follower +without a regime filter, a stop, or position sizing will lose money in +chop-heavy markets. Use it as the skeleton for your own research. diff --git a/examples/strategies_showcase/01_trend_following/backtest.py b/examples/strategies_showcase/01_trend_following/backtest.py new file mode 100644 index 00000000..fa1b0ecc --- /dev/null +++ b/examples/strategies_showcase/01_trend_following/backtest.py @@ -0,0 +1,31 @@ +"""Backtest the trend-following strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import TrendFollowingStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=365) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_vector_backtest( + strategy=TrendFollowingStrategy(), + backtest_date_range=date_range, + initial_amount=1000, + ) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/01_trend_following/requirements.txt b/examples/strategies_showcase/01_trend_following/requirements.txt new file mode 100644 index 00000000..2fefc59d --- /dev/null +++ b/examples/strategies_showcase/01_trend_following/requirements.txt @@ -0,0 +1,4 @@ +investing-algorithm-framework>=8.7.3 +pyindicators>=1.0.0 +pandas>=2.0 +ccxt>=4.0 diff --git a/examples/strategies_showcase/01_trend_following/strategy.py b/examples/strategies_showcase/01_trend_following/strategy.py new file mode 100644 index 00000000..1894e885 --- /dev/null +++ b/examples/strategies_showcase/01_trend_following/strategy.py @@ -0,0 +1,87 @@ +"""Trend-following strategy: EMA(fast) crossing EMA(slow).""" +from __future__ import annotations + +from typing import Any, Dict + +import pandas as pd +from pyindicators import crossover, crossunder, ema + +from investing_algorithm_framework import ( + DataSource, + DataType, + TimeUnit, + TradingStrategy, +) + + +class TrendFollowingStrategy(TradingStrategy): + algorithm_id = "trend-following-ema-crossover" + time_unit = TimeUnit.HOUR + interval = 24 + market = "BITVAVO" + trading_symbol = "EUR" + + def __init__( + self, + symbol: str = "BTC/EUR", + time_frame: str = "1d", + fast_period: int = 20, + slow_period: int = 50, + ): + self.symbol = symbol + self.time_frame = time_frame + self.fast_period = fast_period + self.slow_period = slow_period + + symbols = [symbol.split("/")[0]] + data_sources = [ + DataSource( + identifier=f"{symbol}-ohlcv", + data_type=DataType.OHLCV, + market=self.market, + symbol=symbol, + time_frame=time_frame, + warmup_window=slow_period + 10, + pandas=True, + ) + ] + super().__init__( + algorithm_id=self.algorithm_id, + symbols=symbols, + trading_symbol=self.trading_symbol, + data_sources=data_sources, + time_unit=self.time_unit, + interval=self.interval, + ) + self.set_parameters({ + "symbol": symbol, + "time_frame": time_frame, + "fast_period": fast_period, + "slow_period": slow_period, + }) + + # ------------------------------------------------------------------ # + def _prepare(self, df: pd.DataFrame) -> pd.DataFrame: + df = ema(df, period=self.fast_period, + source_column="Close", result_column="ema_fast") + df = ema(df, period=self.slow_period, + source_column="Close", result_column="ema_slow") + df = crossover(df, first_column="ema_fast", + second_column="ema_slow", result_column="x_up") + df = crossunder(df, first_column="ema_fast", + second_column="ema_slow", result_column="x_dn") + return df + + def generate_buy_signals( + self, data: Dict[str, Any] + ) -> Dict[str, pd.Series]: + df = self._prepare(data[f"{self.symbol}-ohlcv"]) + sig = df["x_up"].fillna(False).astype(bool) + return {self.symbols[0]: sig} + + def generate_sell_signals( + self, data: Dict[str, Any] + ) -> Dict[str, pd.Series]: + df = self._prepare(data[f"{self.symbol}-ohlcv"]) + sig = df["x_dn"].fillna(False).astype(bool) + return {self.symbols[0]: sig} diff --git a/examples/strategies_showcase/02_mean_reversion/README.md b/examples/strategies_showcase/02_mean_reversion/README.md new file mode 100644 index 00000000..7377987b --- /dev/null +++ b/examples/strategies_showcase/02_mean_reversion/README.md @@ -0,0 +1,38 @@ +# 02 β€” Mean Reversion (Bollinger + RSI) + +**Fit:** βœ… Vector backtest, single symbol. + +## Idea + +Buy when price is **below the lower Bollinger band** *and* RSI is oversold. +Sell when price returns to the **middle band** *or* RSI crosses overbought. +This is the textbook "fade extremes" pattern. + +## Why this fits the framework + +Same as `01_trend_following` β€” single asset, bar-aligned signals, no +intra-bar dependency. Vector backtest path runs in a fraction of a second. + +## Parameters + +| Name | Default | Notes | +|------|---------|-------| +| `symbol` | `BTC/EUR` | | +| `time_frame` | `1d` | | +| `bb_period` | `20` | Bollinger window. | +| `bb_std` | `2.0` | Bollinger band width. | +| `rsi_period` | `14` | | +| `rsi_buy_below` | `30` | Oversold threshold. | +| `rsi_sell_above` | `70` | Overbought threshold. | + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` + +## Disclaimer + +Mean-reversion in crypto is regime-dependent β€” it can lose badly during +strong trends. This example does not include a regime filter. diff --git a/examples/strategies_showcase/02_mean_reversion/backtest.py b/examples/strategies_showcase/02_mean_reversion/backtest.py new file mode 100644 index 00000000..35533ede --- /dev/null +++ b/examples/strategies_showcase/02_mean_reversion/backtest.py @@ -0,0 +1,31 @@ +"""Backtest the mean-reversion strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import MeanReversionStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=365) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_vector_backtest( + strategy=MeanReversionStrategy(), + backtest_date_range=date_range, + initial_amount=1000, + ) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/02_mean_reversion/requirements.txt b/examples/strategies_showcase/02_mean_reversion/requirements.txt new file mode 100644 index 00000000..2fefc59d --- /dev/null +++ b/examples/strategies_showcase/02_mean_reversion/requirements.txt @@ -0,0 +1,4 @@ +investing-algorithm-framework>=8.7.3 +pyindicators>=1.0.0 +pandas>=2.0 +ccxt>=4.0 diff --git a/examples/strategies_showcase/02_mean_reversion/strategy.py b/examples/strategies_showcase/02_mean_reversion/strategy.py new file mode 100644 index 00000000..7042d6ce --- /dev/null +++ b/examples/strategies_showcase/02_mean_reversion/strategy.py @@ -0,0 +1,97 @@ +"""Mean-reversion strategy: Bollinger + RSI.""" +from __future__ import annotations + +from typing import Any, Dict + +import pandas as pd +from pyindicators import bollinger_bands, rsi + +from investing_algorithm_framework import ( + DataSource, + DataType, + TimeUnit, + TradingStrategy, +) + + +class MeanReversionStrategy(TradingStrategy): + algorithm_id = "mean-reversion-bollinger-rsi" + time_unit = TimeUnit.HOUR + interval = 24 + market = "BITVAVO" + trading_symbol = "EUR" + + def __init__( + self, + symbol: str = "BTC/EUR", + time_frame: str = "1d", + bb_period: int = 20, + bb_std: float = 2.0, + rsi_period: int = 14, + rsi_buy_below: float = 30.0, + rsi_sell_above: float = 70.0, + ): + self.symbol = symbol + self.bb_period = bb_period + self.bb_std = bb_std + self.rsi_period = rsi_period + self.rsi_buy_below = rsi_buy_below + self.rsi_sell_above = rsi_sell_above + + symbols = [symbol.split("/")[0]] + data_sources = [ + DataSource( + identifier=f"{symbol}-ohlcv", + data_type=DataType.OHLCV, + market=self.market, + symbol=symbol, + time_frame=time_frame, + warmup_window=max(bb_period, rsi_period) + 20, + pandas=True, + ) + ] + super().__init__( + algorithm_id=self.algorithm_id, + symbols=symbols, + trading_symbol=self.trading_symbol, + data_sources=data_sources, + time_unit=self.time_unit, + interval=self.interval, + ) + self.set_parameters({ + "symbol": symbol, + "time_frame": time_frame, + "bb_period": bb_period, + "bb_std": bb_std, + "rsi_period": rsi_period, + "rsi_buy_below": rsi_buy_below, + "rsi_sell_above": rsi_sell_above, + }) + + def _prepare(self, df: pd.DataFrame) -> pd.DataFrame: + df = bollinger_bands( + df, + source_column="Close", + period=self.bb_period, + std_dev=self.bb_std, + upper_band_column_result_column="bb_up", + middle_band_column_result_column="bb_mid", + lower_band_column_result_column="bb_lo", + ) + df = rsi(df, period=self.rsi_period, + source_column="Close", result_column="rsi") + return df + + def generate_buy_signals( + self, data: Dict[str, Any] + ) -> Dict[str, pd.Series]: + df = self._prepare(data[f"{self.symbol}-ohlcv"]) + sig = (df["Close"] < df["bb_lo"]) & (df["rsi"] < self.rsi_buy_below) + return {self.symbols[0]: sig.fillna(False).astype(bool)} + + def generate_sell_signals( + self, data: Dict[str, Any] + ) -> Dict[str, pd.Series]: + df = self._prepare(data[f"{self.symbol}-ohlcv"]) + sig = (df["Close"] > df["bb_mid"]) | (df["rsi"] > self.rsi_sell_above) + return {self.symbols[0]: sig.fillna(False).astype(bool)} diff --git a/examples/strategies_showcase/03_cross_sectional_momentum/README.md b/examples/strategies_showcase/03_cross_sectional_momentum/README.md new file mode 100644 index 00000000..ac5ee25e --- /dev/null +++ b/examples/strategies_showcase/03_cross_sectional_momentum/README.md @@ -0,0 +1,25 @@ +# 03 β€” Cross-sectional Momentum (Pipeline) + +**Fit:** βœ… Showcases the `Pipeline` API for cross-sectional ranking. + +## Idea + +Across a small basket of liquid crypto pairs, rebalance daily into the +top-N names by 30-day return, restricted to the most liquid universe. +This is the canonical cross-sectional momentum recipe used in equities, +adapted to crypto. + +## Why this fits the framework + +- The framework's `Pipeline` API is purpose-built for this pattern: + factors per symbol β†’ mask β†’ rank β†’ strategy reads the result. +- The example shows the *clean* version of what + `examples/cross-sectional-pipelines/cross_sectional_momentum_bot.py` + already demonstrates β€” see that file for a fully-commented walkthrough. + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` diff --git a/examples/strategies_showcase/03_cross_sectional_momentum/backtest.py b/examples/strategies_showcase/03_cross_sectional_momentum/backtest.py new file mode 100644 index 00000000..bdf319dc --- /dev/null +++ b/examples/strategies_showcase/03_cross_sectional_momentum/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the cross-sectional momentum strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import CrossSectionalMomentumStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=365) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(CrossSectionalMomentumStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/03_cross_sectional_momentum/requirements.txt b/examples/strategies_showcase/03_cross_sectional_momentum/requirements.txt new file mode 100644 index 00000000..a5b719b6 --- /dev/null +++ b/examples/strategies_showcase/03_cross_sectional_momentum/requirements.txt @@ -0,0 +1,3 @@ +investing-algorithm-framework>=8.7.3 +polars>=0.20 +ccxt>=4.0 diff --git a/examples/strategies_showcase/03_cross_sectional_momentum/strategy.py b/examples/strategies_showcase/03_cross_sectional_momentum/strategy.py new file mode 100644 index 00000000..b61bd664 --- /dev/null +++ b/examples/strategies_showcase/03_cross_sectional_momentum/strategy.py @@ -0,0 +1,106 @@ +"""Cross-sectional momentum strategy using the Pipeline API.""" +from __future__ import annotations + +from typing import Any, Dict + +from investing_algorithm_framework import ( + AverageDollarVolume, + Context, + DataSource, + OrderSide, + OrderType, + Pipeline, + Returns, + TimeUnit, + TradingStrategy, +) + +SYMBOLS = ["BTC/EUR", "ETH/EUR", "SOL/EUR", "ADA/EUR", "XRP/EUR"] +TOP_N = 2 +MARKET = "BITVAVO" +TRADING_SYMBOL = "EUR" + + +class MomentumScreener(Pipeline): + dollar_volume = AverageDollarVolume(window=30) + momentum = Returns(window=30) + universe = dollar_volume.top(3) + alpha = momentum.rank(mask=universe) + + +class CrossSectionalMomentumStrategy(TradingStrategy): + algorithm_id = "cross-sectional-momentum" + time_unit = TimeUnit.DAY + interval = 1 + market = MARKET + trading_symbol = TRADING_SYMBOL + symbols = SYMBOLS + pipelines = [MomentumScreener] + + data_sources = [ + DataSource( + data_type="OHLCV", + market=MARKET, + symbol=symbol, + warmup_window=60, + time_frame="1d", + identifier=f"{symbol}-ohlcv", + ) + for symbol in SYMBOLS + ] + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + screen = data["MomentumScreener"] + if screen.is_empty(): + return + + targets_df = screen.sort("alpha", descending=True).head(TOP_N) + targets = {row["symbol"] for row in targets_df.iter_rows(named=True)} + + def last_close(symbol: str) -> float: + ohlcv = data[f"{symbol}-ohlcv"] + try: + return float(ohlcv["Close"][-1]) + except (KeyError, TypeError): + return float(ohlcv["close"].iloc[-1]) + + def base(symbol: str) -> str: + return symbol.split("/")[0] + + # Close non-targets + for symbol in SYMBOLS: + if symbol in targets: + continue + sym = base(symbol) + if not context.has_position(sym, market=self.market): + continue + position = context.get_position(sym, market=self.market) + context.create_order( + target_symbol=sym, + order_side=OrderSide.SELL, + order_type=OrderType.LIMIT, + price=last_close(symbol), + amount=position.get_amount(), + ) + + # Open new targets equal-weight + unallocated = context.get_unallocated() + new_targets = [ + s for s in targets + if not context.has_position(base(s), market=self.market) + ] + if not new_targets: + return + per_target = unallocated / len(new_targets) + for symbol in new_targets: + price = last_close(symbol) + if price <= 0: + continue + amount = (per_target * 0.995) / price + context.create_order( + target_symbol=base(symbol), + order_side=OrderSide.BUY, + order_type=OrderType.LIMIT, + price=price, + amount=amount, + ) diff --git a/examples/strategies_showcase/04_multi_factor_portfolio/README.md b/examples/strategies_showcase/04_multi_factor_portfolio/README.md new file mode 100644 index 00000000..bf45ca34 --- /dev/null +++ b/examples/strategies_showcase/04_multi_factor_portfolio/README.md @@ -0,0 +1,28 @@ +# 04 β€” Multi-factor Portfolio (Pipeline) + +**Fit:** βœ… Pipeline composition with multiple factors blended into one alpha. + +## Idea + +Combine three orthogonal factors into a single composite score, then hold +the top-N symbols equal-weight: + +1. **Momentum** β€” 30d return. +2. **Low volatility** β€” inverse of 30d realized vol (favours calmer names). +3. **Liquidity gate** β€” universe restricted to the most-traded names by + 30d average dollar volume. + +The composite alpha is the average of momentum rank and (negated) volatility +rank within the liquid universe. + +## Why this fits the framework + +The `Factor` algebra in the `Pipeline` API supports `+`, ranking, and +masking out of the box, so the composite is one declarative class. + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` diff --git a/examples/strategies_showcase/04_multi_factor_portfolio/backtest.py b/examples/strategies_showcase/04_multi_factor_portfolio/backtest.py new file mode 100644 index 00000000..ca8c738b --- /dev/null +++ b/examples/strategies_showcase/04_multi_factor_portfolio/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the multi-factor portfolio strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import MultiFactorPortfolioStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=365) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(MultiFactorPortfolioStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/04_multi_factor_portfolio/requirements.txt b/examples/strategies_showcase/04_multi_factor_portfolio/requirements.txt new file mode 100644 index 00000000..a5b719b6 --- /dev/null +++ b/examples/strategies_showcase/04_multi_factor_portfolio/requirements.txt @@ -0,0 +1,3 @@ +investing-algorithm-framework>=8.7.3 +polars>=0.20 +ccxt>=4.0 diff --git a/examples/strategies_showcase/04_multi_factor_portfolio/strategy.py b/examples/strategies_showcase/04_multi_factor_portfolio/strategy.py new file mode 100644 index 00000000..6004008b --- /dev/null +++ b/examples/strategies_showcase/04_multi_factor_portfolio/strategy.py @@ -0,0 +1,110 @@ +"""Multi-factor portfolio: momentum + low-vol + liquidity gate.""" +from __future__ import annotations + +from typing import Any, Dict + +from investing_algorithm_framework import ( + AverageDollarVolume, + Context, + DataSource, + OrderSide, + OrderType, + Pipeline, + Returns, + TimeUnit, + TradingStrategy, + Volatility, +) + +SYMBOLS = ["BTC/EUR", "ETH/EUR", "SOL/EUR", "ADA/EUR", + "XRP/EUR", "DOT/EUR", "LINK/EUR"] +TOP_N = 3 +MARKET = "BITVAVO" +TRADING_SYMBOL = "EUR" + + +class FactorScreen(Pipeline): + dollar_volume = AverageDollarVolume(window=30) + momentum = Returns(window=30) + vol = Volatility(window=30) + + universe = dollar_volume.top(5) + momentum_rank = momentum.rank(mask=universe) + low_vol_rank = (-vol).rank(mask=universe) + alpha = momentum_rank + low_vol_rank + + +class MultiFactorPortfolioStrategy(TradingStrategy): + algorithm_id = "multi-factor-portfolio" + time_unit = TimeUnit.DAY + interval = 1 + market = MARKET + trading_symbol = TRADING_SYMBOL + symbols = SYMBOLS + pipelines = [FactorScreen] + + data_sources = [ + DataSource( + data_type="OHLCV", + market=MARKET, + symbol=symbol, + warmup_window=60, + time_frame="1d", + identifier=f"{symbol}-ohlcv", + ) + for symbol in SYMBOLS + ] + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + screen = data["FactorScreen"] + if screen.is_empty(): + return + + targets_df = screen.sort("alpha", descending=True).head(TOP_N) + targets = {row["symbol"] for row in targets_df.iter_rows(named=True)} + + def last_close(symbol: str) -> float: + ohlcv = data[f"{symbol}-ohlcv"] + try: + return float(ohlcv["Close"][-1]) + except (KeyError, TypeError): + return float(ohlcv["close"].iloc[-1]) + + def base(symbol: str) -> str: + return symbol.split("/")[0] + + for symbol in SYMBOLS: + if symbol in targets: + continue + sym = base(symbol) + if not context.has_position(sym, market=self.market): + continue + position = context.get_position(sym, market=self.market) + context.create_order( + target_symbol=sym, + order_side=OrderSide.SELL, + order_type=OrderType.LIMIT, + price=last_close(symbol), + amount=position.get_amount(), + ) + + unallocated = context.get_unallocated() + new_targets = [ + s for s in targets + if not context.has_position(base(s), market=self.market) + ] + if not new_targets: + return + per_target = unallocated / len(new_targets) + for symbol in new_targets: + price = last_close(symbol) + if price <= 0: + continue + amount = (per_target * 0.995) / price + context.create_order( + target_symbol=base(symbol), + order_side=OrderSide.BUY, + order_type=OrderType.LIMIT, + price=price, + amount=amount, + ) diff --git a/examples/strategies_showcase/05_pairs_trading/README.md b/examples/strategies_showcase/05_pairs_trading/README.md new file mode 100644 index 00000000..42175238 --- /dev/null +++ b/examples/strategies_showcase/05_pairs_trading/README.md @@ -0,0 +1,43 @@ +# 05 β€” Pairs Trading (z-score on log-spread) + +**Fit:** 🟒 Workable, with bookkeeping in user code. + +## Idea + +Two co-moving symbols (BTC and ETH) typically have a stable log-price ratio. +When the spread `log(BTC) - Ξ²Β·log(ETH)` deviates by more than `z_entry` +standard deviations from its rolling mean, **fade it**: + +- Spread *too high* β†’ short the rich leg (BTC), long the cheap leg (ETH). +- Spread *too low* β†’ long BTC, short ETH. +- Exit when `|z| < z_exit`. + +## Why this is 🟒 and not βœ… + +The framework's vector backtester is single-asset signal-driven and does not +have a native concept of a "pair" or a hedge ratio. The example uses the +**event-driven** `run_strategy(context, data)` path and manages both legs +manually β€” opening/closing matched positions as the z-score crosses +thresholds. + +Shorting on a spot venue (Bitvavo) is not realistic. To keep the example +self-contained and runnable, this implementation uses a **long-only +relative-value** variant: it holds the *cheap* asset and exits when the +spread mean-reverts. A proper market-neutral version would require margin +or perp support. + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` + +## Disclaimer + +A *real* pairs trade requires: +- A statistically validated cointegration (ADF / Johansen). +- A rolling re-estimation of the hedge ratio Ξ². +- Short or perp support for true market neutrality. + +This example does **none** of those β€” it is a structural template only. diff --git a/examples/strategies_showcase/05_pairs_trading/backtest.py b/examples/strategies_showcase/05_pairs_trading/backtest.py new file mode 100644 index 00000000..9fe07a42 --- /dev/null +++ b/examples/strategies_showcase/05_pairs_trading/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the pairs trading strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import PairsTradingStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=365) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(PairsTradingStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/05_pairs_trading/requirements.txt b/examples/strategies_showcase/05_pairs_trading/requirements.txt new file mode 100644 index 00000000..b3c50d3c --- /dev/null +++ b/examples/strategies_showcase/05_pairs_trading/requirements.txt @@ -0,0 +1,3 @@ +investing-algorithm-framework>=8.7.3 +pandas>=2.0 +ccxt>=4.0 diff --git a/examples/strategies_showcase/05_pairs_trading/strategy.py b/examples/strategies_showcase/05_pairs_trading/strategy.py new file mode 100644 index 00000000..85887052 --- /dev/null +++ b/examples/strategies_showcase/05_pairs_trading/strategy.py @@ -0,0 +1,97 @@ +"""Pairs trading (long-only, spot-friendly variant).""" +from __future__ import annotations + +import math +from typing import Any, Dict + +from investing_algorithm_framework import ( + Context, + DataSource, + DataType, + OrderSide, + OrderType, + TimeUnit, + TradingStrategy, +) + +SYMBOL_A = "BTC/EUR" +SYMBOL_B = "ETH/EUR" +WINDOW = 60 +Z_ENTRY = 1.5 +Z_EXIT = 0.3 + + +class PairsTradingStrategy(TradingStrategy): + algorithm_id = "pairs-trading-zscore" + time_unit = TimeUnit.DAY + interval = 1 + market = "BITVAVO" + trading_symbol = "EUR" + symbols = [SYMBOL_A.split("/")[0], SYMBOL_B.split("/")[0]] + + data_sources = [ + DataSource( + identifier=f"{SYMBOL_A}-ohlcv", + data_type=DataType.OHLCV, market=market, symbol=SYMBOL_A, + time_frame="1d", warmup_window=WINDOW + 5, pandas=True, + ), + DataSource( + identifier=f"{SYMBOL_B}-ohlcv", + data_type=DataType.OHLCV, market=market, symbol=SYMBOL_B, + time_frame="1d", warmup_window=WINDOW + 5, pandas=True, + ), + ] + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + df_a = data[f"{SYMBOL_A}-ohlcv"] + df_b = data[f"{SYMBOL_B}-ohlcv"] + if len(df_a) < WINDOW + 1 or len(df_b) < WINDOW + 1: + return + + # log spread with hedge ratio Ξ² = 1 (simplest variant) + spread = (df_a["Close"].apply(math.log) + - df_b["Close"].apply(math.log)).tail(WINDOW + 1) + mu = spread.iloc[:-1].mean() + sd = spread.iloc[:-1].std() + if sd == 0 or math.isnan(sd): + return + z = (spread.iloc[-1] - mu) / sd + + base_a = SYMBOL_A.split("/")[0] + base_b = SYMBOL_B.split("/")[0] + price_a = float(df_a["Close"].iloc[-1]) + price_b = float(df_b["Close"].iloc[-1]) + has_a = context.has_position(base_a, market=self.market) + has_b = context.has_position(base_b, market=self.market) + + # ENTRY: long the *cheap* leg. + if z > Z_ENTRY and not has_b: + self._close_if_held(context, base_a, price_a) + self._buy_full(context, base_b, price_b) + elif z < -Z_ENTRY and not has_a: + self._close_if_held(context, base_b, price_b) + self._buy_full(context, base_a, price_a) + # EXIT: spread mean-reverted. + elif abs(z) < Z_EXIT: + self._close_if_held(context, base_a, price_a) + self._close_if_held(context, base_b, price_b) + + def _close_if_held(self, context: Context, sym: str, price: float) -> None: + if not context.has_position(sym, market=self.market): + return + pos = context.get_position(sym, market=self.market) + context.create_order( + target_symbol=sym, order_side=OrderSide.SELL, + order_type=OrderType.LIMIT, price=price, + amount=pos.get_amount(), + ) + + def _buy_full(self, context: Context, sym: str, price: float) -> None: + cash = context.get_unallocated() + if cash <= 0 or price <= 0: + return + amount = (cash * 0.995) / price + context.create_order( + target_symbol=sym, order_side=OrderSide.BUY, + order_type=OrderType.LIMIT, price=price, amount=amount, + ) diff --git a/examples/strategies_showcase/06_funding_rate_carry/README.md b/examples/strategies_showcase/06_funding_rate_carry/README.md new file mode 100644 index 00000000..085651c1 --- /dev/null +++ b/examples/strategies_showcase/06_funding_rate_carry/README.md @@ -0,0 +1,50 @@ +# 06 β€” Funding Rate Carry + +**Fit:** 🟑 Pattern is clean, but the framework has no native perp/funding-rate data provider today. + +## Idea + +Perpetual futures pay (or receive) a **funding rate** every 8 hours to keep +their price tethered to spot. A persistent funding sign means one side of +the trade is structurally paid to provide liquidity. The classic carry trade: + +- **Funding > 0** (longs pay shorts) β†’ short the perp, long the spot, harvest funding. +- **Funding < 0** (shorts pay longs) β†’ long the perp, short the spot. + +The position is delta-neutral; the alpha is the funding payment minus +financing and slippage. + +## Why this is 🟑 + +The framework today ships with **spot OHLCV providers** (`CCXTOHLCVDataProvider`, +`PandasOHLCVDataProvider`, etc.) and a spot-style portfolio model. To run +a real funding-rate carry strategy you'd need: + +1. A **funding-rate data provider** (e.g. a `CCXTFundingRateDataProvider` + pulling `fetchFundingRateHistory` from CCXT). +2. A **perpetual futures portfolio** model that can hold a delta-neutral + spot/perp pair and accrue funding cash flows. +3. **Margin accounting** so the perp leg is properly collateralised. + +None of those exist as first-class primitives yet β€” they live in the +"derivatives expansion" backlog. + +## What this folder contains + +A **scaffold** showing how a custom `DataProvider` would deliver funding +rates into a `TradingStrategy`. The accompanying `backtest.py` is a +no-op (it raises `NotImplementedError`) so that running it makes the +limitation explicit. + +If you want to prototype this pattern today, your best bet is to: + +1. Subclass `DataProvider` and load funding-rate history from a CSV. +2. Use the spot portfolio as a proxy for the *cash leg only*, and book the + funding cash flow manually via `context.create_order` against a synthetic + "FUNDING" symbol. + +## Run + +```bash +python backtest.py # raises NotImplementedError with an explanation. +``` diff --git a/examples/strategies_showcase/06_funding_rate_carry/backtest.py b/examples/strategies_showcase/06_funding_rate_carry/backtest.py new file mode 100644 index 00000000..996a961b --- /dev/null +++ b/examples/strategies_showcase/06_funding_rate_carry/backtest.py @@ -0,0 +1,14 @@ +"""Funding-rate carry β€” scaffold only. See README.md.""" + + +def main() -> None: + raise NotImplementedError( + "Funding-rate carry is not directly implementable today. " + "The framework lacks a native funding-rate data provider and a " + "perp/margin portfolio model. See README.md for the workaround " + "(custom DataProvider + manual cash-flow booking)." + ) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/07_event_driven_signal/README.md b/examples/strategies_showcase/07_event_driven_signal/README.md new file mode 100644 index 00000000..e31b112d --- /dev/null +++ b/examples/strategies_showcase/07_event_driven_signal/README.md @@ -0,0 +1,31 @@ +# 07 β€” Event-driven Signal (Volatility Breakout) + +**Fit:** βœ… Clean fit for the bar-driven event loop. + +## Idea + +A "donchian-style" volatility breakout: at every bar close, if the close +is above the **N-bar rolling high**, enter long. Exit when the close drops +below the **N-bar rolling low**. The signal triggers on a discrete +*event* (the breakout), which is the canonical use-case for the +event-driven `run_strategy` path. + +## Why this fits the framework + +`run_strategy(context, data)` is called once per bar; placing a single +`create_order` per event is exactly what the engine is built for. + +## Parameters + +| Name | Default | Notes | +|------|---------|-------| +| `symbol` | `BTC/EUR` | | +| `time_frame` | `1h` | Faster bars produce more events. | +| `breakout_window` | `48` | Two-day rolling high/low on 1h bars. | + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` diff --git a/examples/strategies_showcase/07_event_driven_signal/backtest.py b/examples/strategies_showcase/07_event_driven_signal/backtest.py new file mode 100644 index 00000000..075c5a8c --- /dev/null +++ b/examples/strategies_showcase/07_event_driven_signal/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the volatility-breakout event-driven strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import VolatilityBreakoutStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=120) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(VolatilityBreakoutStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/07_event_driven_signal/requirements.txt b/examples/strategies_showcase/07_event_driven_signal/requirements.txt new file mode 100644 index 00000000..b3c50d3c --- /dev/null +++ b/examples/strategies_showcase/07_event_driven_signal/requirements.txt @@ -0,0 +1,3 @@ +investing-algorithm-framework>=8.7.3 +pandas>=2.0 +ccxt>=4.0 diff --git a/examples/strategies_showcase/07_event_driven_signal/strategy.py b/examples/strategies_showcase/07_event_driven_signal/strategy.py new file mode 100644 index 00000000..a79220c7 --- /dev/null +++ b/examples/strategies_showcase/07_event_driven_signal/strategy.py @@ -0,0 +1,65 @@ +"""Donchian-style volatility breakout (event-driven).""" +from __future__ import annotations + +from typing import Any, Dict + +from investing_algorithm_framework import ( + Context, + DataSource, + DataType, + OrderSide, + OrderType, + TimeUnit, + TradingStrategy, +) + +SYMBOL = "BTC/EUR" +WINDOW = 48 + + +class VolatilityBreakoutStrategy(TradingStrategy): + algorithm_id = "vol-breakout-event-driven" + time_unit = TimeUnit.HOUR + interval = 1 + market = "BITVAVO" + trading_symbol = "EUR" + symbols = [SYMBOL.split("/")[0]] + + data_sources = [ + DataSource( + identifier=f"{SYMBOL}-ohlcv", + data_type=DataType.OHLCV, market=market, symbol=SYMBOL, + time_frame="1h", warmup_window=WINDOW + 5, pandas=True, + ) + ] + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + df = data[f"{SYMBOL}-ohlcv"] + if len(df) < WINDOW + 1: + return + + recent = df.tail(WINDOW + 1) + prior = recent.iloc[:-1] + last_close = float(recent["Close"].iloc[-1]) + rolling_high = float(prior["High"].max()) + rolling_low = float(prior["Low"].min()) + + sym = self.symbols[0] + held = context.has_position(sym, market=self.market) + + if not held and last_close > rolling_high: + cash = context.get_unallocated() + if cash <= 0: + return + amount = (cash * 0.995) / last_close + context.create_order( + target_symbol=sym, order_side=OrderSide.BUY, + order_type=OrderType.LIMIT, price=last_close, amount=amount, + ) + elif held and last_close < rolling_low: + pos = context.get_position(sym, market=self.market) + context.create_order( + target_symbol=sym, order_side=OrderSide.SELL, + order_type=OrderType.LIMIT, price=last_close, + amount=pos.get_amount(), + ) diff --git a/examples/strategies_showcase/08_dca_accumulation/README.md b/examples/strategies_showcase/08_dca_accumulation/README.md new file mode 100644 index 00000000..131b1ef8 --- /dev/null +++ b/examples/strategies_showcase/08_dca_accumulation/README.md @@ -0,0 +1,120 @@ +# 08 β€” DCA Accumulation + +**Fit:** βœ… Trivially simple, perfect example of a scheduled task. + +## Idea + +Dollar-cost averaging: buy a **fixed EUR amount** of BTC every week, +regardless of price. The point is to demonstrate the simplest possible +deployable strategy. + +## Why this fits the framework + +`time_unit=TimeUnit.DAY` + `interval=7` (or any cadence) is exactly the +intended use case for the bar-driven event loop. No indicators, no signal β€” +just a periodic side-effect. + +### Funding sync β€” staying aware of new capital + +Real DCA bots have one piece that paper examples almost always skip: +**capital arrives from outside the strategy** (a paycheck deposit, a +monthly transfer). If the bot only ever knows about its starting balance, +it will never deploy newly arrived cash and the "DCA" stops after a few +weeks. + +The framework treats external cash flow as a first-class concept. You +declare it on the market and the event loop absorbs new funds for you: + +```python +from investing_algorithm_framework import ScheduledDeposit, TimeUnit + +app.add_market( + market="BITVAVO", + trading_symbol="EUR", + initial_balance=2500, + deposit_schedule=[ + ScheduledDeposit( + amount=100.0, time_unit=TimeUnit.DAY, interval=30 + ), + ], + auto_sync=True, +) +``` + +On every iteration the engine calls `context.sync_portfolio(market=...)` +under the hood. The result is identical in both modes: + +| Mode | Where "broker available" comes from | +|------|-------------------------------------| +| **Backtest** | The `BrokerBalanceTracker` materialises the schedule; due deposits land in a "pending" bucket and are drained into `unallocated`. | +| **Live** | `PortfolioProvider.get_position(...).amount` β€” the exchange's free balance. The schedule is informational only; any real deposit (paycheck, manual transfer, withdrawal) is absorbed automatically. | + +If `broker_available < unallocated` (cash disappeared from the +exchange), `sync_portfolio` raises `PortfolioOutOfSyncError` by default. +Pass `allow_withdrawals=True` to opt into draining instead. You can also +call `context.sync_portfolio(market=...)` manually from inside a strategy +if you prefer not to use `auto_sync`. + +### Production knobs + +For live trading you typically want a few defaults relaxed so a single +flaky API call doesn't crash the bot: + +```python +app.add_market( + market="BITVAVO", + trading_symbol="EUR", + initial_balance=2500, + deposit_schedule=[ + ScheduledDeposit( + amount=100.0, + time_unit=TimeUnit.DAY, + interval=30, + # Optional: deposit also lands at t=0 (anchor day) instead of + # only after the first interval has elapsed. + fire_on_anchor=True, + ), + ], + auto_sync=True, + # "warn" β†’ log + continue on transient broker errors; the bot keeps + # running and retries next iteration. "raise" (default) propagates + # and stops the loop. "halt" disables auto-sync for this market + # after the first failure. + auto_sync_error_mode="warn", +) +``` + +You can also pass `tolerance=` to `context.sync_portfolio()` to ignore +sub-cent rounding drift when reconciling against the exchange: + +```python +context.sync_portfolio(market="BITVAVO", tolerance=0.01) +``` + +> **TWR-aware metrics:** every deposit absorbed by `sync_portfolio` is +> stamped onto the next portfolio snapshot's `cash_flow` field. CAGR, +> monthly/yearly returns, Sharpe, Sortino, volatility, VaR and the std +> metrics all subtract this from the period's ending value before +> computing the return β€” so depositing into a DCA bot does not inflate +> its reported alpha. + +## Parameters + +| Name | Default | Notes | +|------|---------|-------| +| `symbol` | `BTC/EUR` | | +| `dca_amount_eur` | `25` | EUR spent per buy. | +| `cadence_days` | `7` | Weekly. | + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` + +## Note + +DCA is *not a trading strategy* in the alpha-generation sense. It is a +behavioural commitment device that smooths entry timing. Use it as an +overlay for long-term core positions, not as a standalone alpha. Also the overall concept of rebalancing can be applied to any strategy, not just DCA, it's a general tool for keeping your actual portfolio aligned with your intended allocation and your available capital. diff --git a/examples/strategies_showcase/08_dca_accumulation/backtest.py b/examples/strategies_showcase/08_dca_accumulation/backtest.py new file mode 100644 index 00000000..93ea095b --- /dev/null +++ b/examples/strategies_showcase/08_dca_accumulation/backtest.py @@ -0,0 +1,42 @@ +"""Backtest the DCA strategy with a recurring monthly deposit.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import ( + BacktestDateRange, + ScheduledDeposit, + TimeUnit, + create_app, +) + +from strategy import DCAStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=730) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(DCAStrategy) + app.add_market( + market="BITVAVO", + trading_symbol="EUR", + initial_balance=2500, + deposit_schedule=[ + ScheduledDeposit( + amount=100.0, time_unit=TimeUnit.DAY, interval=30 + ), + ], + auto_sync=True, + ) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/08_dca_accumulation/requirements.txt b/examples/strategies_showcase/08_dca_accumulation/requirements.txt new file mode 100644 index 00000000..b3c50d3c --- /dev/null +++ b/examples/strategies_showcase/08_dca_accumulation/requirements.txt @@ -0,0 +1,3 @@ +investing-algorithm-framework>=8.7.3 +pandas>=2.0 +ccxt>=4.0 diff --git a/examples/strategies_showcase/08_dca_accumulation/strategy.py b/examples/strategies_showcase/08_dca_accumulation/strategy.py new file mode 100644 index 00000000..44dac539 --- /dev/null +++ b/examples/strategies_showcase/08_dca_accumulation/strategy.py @@ -0,0 +1,64 @@ +"""Dollar-cost averaging: weekly fixed-EUR buys, monthly recurring deposit. + +Two pieces working together: + +1. **Weekly buy** (`DCAStrategy.run_strategy`) β€” every 7 days, buy a + fixed EUR amount of BTC. +2. **Monthly deposit** β€” declared on the market via + ``app.add_market(deposit_schedule=[...], auto_sync=True)``. Both in + live and backtest mode the framework's + :meth:`Context.sync_portfolio` absorbs the new cash on each + iteration. No bespoke task needed β€” see ``backtest.py``. +""" +from __future__ import annotations + +from typing import Any, Dict + +from investing_algorithm_framework import ( + Context, + DataSource, + DataType, + OrderSide, + OrderType, + TimeUnit, + TradingStrategy, +) + +SYMBOL = "BTC/EUR" +MARKET = "BITVAVO" +DCA_AMOUNT_EUR = 25.0 + + +class DCAStrategy(TradingStrategy): + algorithm_id = "dca-weekly" + time_unit = TimeUnit.DAY + interval = 7 + market = MARKET + trading_symbol = "EUR" + symbols = [SYMBOL.split("/")[0]] + + data_sources = [ + DataSource( + identifier=f"{SYMBOL}-ohlcv", + data_type=DataType.OHLCV, market=market, symbol=SYMBOL, + time_frame="1d", warmup_window=2, pandas=True, + ) + ] + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + df = data[f"{SYMBOL}-ohlcv"] + if len(df) == 0: + return + price = float(df["Close"].iloc[-1]) + if price <= 0: + return + cash = context.get_unallocated() + if cash < DCA_AMOUNT_EUR: + return + context.create_order( + target_symbol=self.symbols[0], + order_side=OrderSide.BUY, + order_type=OrderType.LIMIT, + price=price, + amount=(DCA_AMOUNT_EUR * 0.995) / price, + ) diff --git a/examples/strategies_showcase/09_risk_parity/README.md b/examples/strategies_showcase/09_risk_parity/README.md new file mode 100644 index 00000000..2aeaa660 --- /dev/null +++ b/examples/strategies_showcase/09_risk_parity/README.md @@ -0,0 +1,29 @@ +# 09 β€” Risk Parity (inverse-volatility weighting) + +**Fit:** βœ… Periodic-rebalance pattern is well-supported. + +## Idea + +Allocate capital across the basket so that **each asset contributes the +same risk** to the portfolio. The simplest proxy is inverse-volatility +weighting: weight ∝ 1/Οƒ, then renormalise. Rebalance monthly. + +## Why this fits the framework + +- Portfolio targets are pure functions of the OHLCV history. +- The strategy reads target weights, computes the per-symbol delta vs. + current positions, and emits the matching orders. +- Rebalance cadence is set with `time_unit=DAY` + a monthly gate. + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` + +## Disclaimer + +True risk parity uses **marginal risk contributions** under the full +covariance matrix; inverse-volatility is its first-order approximation, +which is what most practitioners actually run. diff --git a/examples/strategies_showcase/09_risk_parity/backtest.py b/examples/strategies_showcase/09_risk_parity/backtest.py new file mode 100644 index 00000000..621504d1 --- /dev/null +++ b/examples/strategies_showcase/09_risk_parity/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the risk-parity strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import RiskParityStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=365) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(RiskParityStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/09_risk_parity/requirements.txt b/examples/strategies_showcase/09_risk_parity/requirements.txt new file mode 100644 index 00000000..adde12dc --- /dev/null +++ b/examples/strategies_showcase/09_risk_parity/requirements.txt @@ -0,0 +1,4 @@ +investing-algorithm-framework>=8.7.3 +numpy>=1.25 +pandas>=2.0 +ccxt>=4.0 diff --git a/examples/strategies_showcase/09_risk_parity/strategy.py b/examples/strategies_showcase/09_risk_parity/strategy.py new file mode 100644 index 00000000..5dfa2f14 --- /dev/null +++ b/examples/strategies_showcase/09_risk_parity/strategy.py @@ -0,0 +1,107 @@ +"""Risk parity (inverse-volatility weighting), monthly rebalance.""" +from __future__ import annotations + +from typing import Any, Dict, List + +import numpy as np + +from investing_algorithm_framework import ( + Context, + DataSource, + DataType, + OrderSide, + OrderType, + TimeUnit, + TradingStrategy, +) + +SYMBOLS = ["BTC/EUR", "ETH/EUR", "SOL/EUR", "ADA/EUR", "XRP/EUR"] +MARKET = "BITVAVO" +LOOKBACK = 30 +REBALANCE_EVERY_BARS = 30 # ~monthly on daily bars + + +class RiskParityStrategy(TradingStrategy): + algorithm_id = "risk-parity-inv-vol" + time_unit = TimeUnit.DAY + interval = 1 + market = MARKET + trading_symbol = "EUR" + symbols = [s.split("/")[0] for s in SYMBOLS] + + data_sources = [ + DataSource( + identifier=f"{s}-ohlcv", + data_type=DataType.OHLCV, market=MARKET, symbol=s, + time_frame="1d", warmup_window=LOOKBACK + 5, pandas=True, + ) + for s in SYMBOLS + ] + + def __init__(self): + super().__init__() + self._bar_count = 0 + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + self._bar_count += 1 + if self._bar_count % REBALANCE_EVERY_BARS != 1: + return + + weights = self._target_weights(data) + if not weights: + return + self._rebalance(context, data, weights) + + def _target_weights(self, data: Dict[str, Any]) -> Dict[str, float]: + inv_vols: Dict[str, float] = {} + for s in SYMBOLS: + df = data[f"{s}-ohlcv"] + if len(df) < LOOKBACK + 1: + continue + rets = df["Close"].pct_change().tail(LOOKBACK).dropna() + sd = float(rets.std()) + if sd <= 0 or np.isnan(sd): + continue + inv_vols[s] = 1.0 / sd + if not inv_vols: + return {} + total = sum(inv_vols.values()) + return {s: w / total for s, w in inv_vols.items()} + + def _rebalance( + self, context: Context, data: Dict[str, Any], + weights: Dict[str, float], + ) -> None: + equity = context.get_unallocated() + for sym in self.symbols: + if context.has_position(sym, market=self.market): + pos = context.get_position(sym, market=self.market) + price = self._last_close(data, f"{sym}/EUR") + equity += pos.get_amount() * price + + for symbol, weight in weights.items(): + sym = symbol.split("/")[0] + price = self._last_close(data, symbol) + if price <= 0: + continue + target_value = equity * weight + target_amount = target_value / price + current_amount = 0.0 + if context.has_position(sym, market=self.market): + current_amount = context.get_position( + sym, market=self.market + ).get_amount() + delta = target_amount - current_amount + if abs(delta * price) < 5: + continue + side = OrderSide.BUY if delta > 0 else OrderSide.SELL + context.create_order( + target_symbol=sym, order_side=side, + order_type=OrderType.LIMIT, price=price, + amount=abs(delta), + ) + + @staticmethod + def _last_close(data: Dict[str, Any], symbol: str) -> float: + df = data[f"{symbol}-ohlcv"] + return float(df["Close"].iloc[-1]) diff --git a/examples/strategies_showcase/10_mean_variance_markowitz/README.md b/examples/strategies_showcase/10_mean_variance_markowitz/README.md new file mode 100644 index 00000000..27c928f6 --- /dev/null +++ b/examples/strategies_showcase/10_mean_variance_markowitz/README.md @@ -0,0 +1,32 @@ +# 10 β€” Mean-Variance (Markowitz) Optimization + +**Fit:** βœ… Same monthly-rebalance pattern as risk parity. + +## Idea + +At each rebalance, estimate the **expected returns** ΞΌ (using the trailing +sample mean) and the **covariance matrix** Ξ£ from a 90-day window, and +solve for the long-only minimum-variance portfolio targeting a fixed +expected return. + +`scipy.optimize.minimize` with SLSQP is used for the constrained QP β€” it +keeps the requirements list small (no `cvxpy`). + +## Why this fits the framework + +The framework provides per-symbol price history and order placement; the +optimisation is a pure function on top. + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` + +## Caveats + +- Sample-mean ΞΌ is a notoriously noisy estimator. In production replace it + with a **shrinkage estimator** or a forward-looking signal (e.g. factor model). +- Ξ£ also benefits from **Ledoit-Wolf shrinkage**. +- This example is a **structural template** β€” do not run it with real money. diff --git a/examples/strategies_showcase/10_mean_variance_markowitz/backtest.py b/examples/strategies_showcase/10_mean_variance_markowitz/backtest.py new file mode 100644 index 00000000..f495a516 --- /dev/null +++ b/examples/strategies_showcase/10_mean_variance_markowitz/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the Markowitz mean-variance strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import MarkowitzStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=365) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(MarkowitzStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/10_mean_variance_markowitz/requirements.txt b/examples/strategies_showcase/10_mean_variance_markowitz/requirements.txt new file mode 100644 index 00000000..15b2fd3d --- /dev/null +++ b/examples/strategies_showcase/10_mean_variance_markowitz/requirements.txt @@ -0,0 +1,5 @@ +investing-algorithm-framework>=8.7.3 +numpy>=1.25 +pandas>=2.0 +scipy>=1.11 +ccxt>=4.0 diff --git a/examples/strategies_showcase/10_mean_variance_markowitz/strategy.py b/examples/strategies_showcase/10_mean_variance_markowitz/strategy.py new file mode 100644 index 00000000..7b0aeba8 --- /dev/null +++ b/examples/strategies_showcase/10_mean_variance_markowitz/strategy.py @@ -0,0 +1,129 @@ +"""Mean-variance Markowitz: monthly long-only min-variance optimisation.""" +from __future__ import annotations + +from typing import Any, Dict + +import numpy as np +from scipy.optimize import minimize + +from investing_algorithm_framework import ( + Context, + DataSource, + DataType, + OrderSide, + OrderType, + TimeUnit, + TradingStrategy, +) + +SYMBOLS = ["BTC/EUR", "ETH/EUR", "SOL/EUR", "ADA/EUR", "XRP/EUR"] +MARKET = "BITVAVO" +LOOKBACK = 90 +REBALANCE_EVERY_BARS = 30 +TARGET_DAILY_RETURN = 0.001 # ~25% annualised + + +class MarkowitzStrategy(TradingStrategy): + algorithm_id = "mean-variance-markowitz" + time_unit = TimeUnit.DAY + interval = 1 + market = MARKET + trading_symbol = "EUR" + symbols = [s.split("/")[0] for s in SYMBOLS] + + data_sources = [ + DataSource( + identifier=f"{s}-ohlcv", + data_type=DataType.OHLCV, market=MARKET, symbol=s, + time_frame="1d", warmup_window=LOOKBACK + 5, pandas=True, + ) + for s in SYMBOLS + ] + + def __init__(self): + super().__init__() + self._bar_count = 0 + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + self._bar_count += 1 + if self._bar_count % REBALANCE_EVERY_BARS != 1: + return + + weights = self._optimise(data) + if weights is None: + return + self._rebalance(context, data, weights) + + def _optimise(self, data: Dict[str, Any]) -> Dict[str, float] | None: + rets_matrix = [] + cols = [] + for s in SYMBOLS: + df = data[f"{s}-ohlcv"] + if len(df) < LOOKBACK + 1: + continue + r = df["Close"].pct_change().tail(LOOKBACK).dropna().to_numpy() + if len(r) < LOOKBACK - 5: + continue + rets_matrix.append(r) + cols.append(s) + if len(cols) < 2: + return None + + n = min(len(r) for r in rets_matrix) + R = np.vstack([r[-n:] for r in rets_matrix]) + mu = R.mean(axis=1) + cov = np.cov(R) + np.eye(len(cols)) * 1e-6 + + def variance(w): + return float(w @ cov @ w) + + cons = [ + {"type": "eq", "fun": lambda w: np.sum(w) - 1.0}, + {"type": "ineq", "fun": lambda w: float(w @ mu) + - TARGET_DAILY_RETURN}, + ] + bounds = [(0.0, 1.0) for _ in cols] + x0 = np.ones(len(cols)) / len(cols) + res = minimize(variance, x0, method="SLSQP", + bounds=bounds, constraints=cons) + if not res.success: + # Fallback: equal-weight if QP infeasible. + w = x0 + else: + w = np.clip(res.x, 0, None) + if w.sum() == 0: + return None + w = w / w.sum() + return dict(zip(cols, w.tolist())) + + def _rebalance( + self, context: Context, data: Dict[str, Any], + weights: Dict[str, float], + ) -> None: + equity = context.get_unallocated() + for sym in self.symbols: + if context.has_position(sym, market=self.market): + pos = context.get_position(sym, market=self.market) + price = float(data[f"{sym}/EUR-ohlcv"]["Close"].iloc[-1]) + equity += pos.get_amount() * price + + for symbol, weight in weights.items(): + sym = symbol.split("/")[0] + price = float(data[f"{symbol}-ohlcv"]["Close"].iloc[-1]) + if price <= 0: + continue + target_amount = (equity * weight) / price + current_amount = 0.0 + if context.has_position(sym, market=self.market): + current_amount = context.get_position( + sym, market=self.market + ).get_amount() + delta = target_amount - current_amount + if abs(delta * price) < 5: + continue + side = OrderSide.BUY if delta > 0 else OrderSide.SELL + context.create_order( + target_symbol=sym, order_side=side, + order_type=OrderType.LIMIT, price=price, + amount=abs(delta), + ) diff --git a/examples/strategies_showcase/11_hierarchical_risk_parity/README.md b/examples/strategies_showcase/11_hierarchical_risk_parity/README.md new file mode 100644 index 00000000..4ff944ce --- /dev/null +++ b/examples/strategies_showcase/11_hierarchical_risk_parity/README.md @@ -0,0 +1,32 @@ +# 11 β€” Hierarchical Risk Parity (HRP) + +**Fit:** βœ… Same rebalance scaffolding as risk parity / Markowitz. + +## Idea + +LΓ³pez de Prado's HRP avoids the instability of mean-variance by: + +1. Computing the **correlation distance** matrix. +2. Performing **single-linkage hierarchical clustering**. +3. **Quasi-diagonalising** the covariance matrix along the dendrogram order. +4. **Recursively bisecting** the clusters and allocating inversely to + cluster variance. + +It produces stable, well-diversified weights without needing ΞΌ. + +## Why this fits the framework + +Same monthly-rebalance pattern as #09 / #10. The HRP routine is a pure +function on the trailing covariance β€” no framework primitive is missing. + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` + +## References + +LΓ³pez de Prado, M. (2016). *Building Diversified Portfolios that Outperform +Out-of-Sample*. Journal of Portfolio Management. diff --git a/examples/strategies_showcase/11_hierarchical_risk_parity/backtest.py b/examples/strategies_showcase/11_hierarchical_risk_parity/backtest.py new file mode 100644 index 00000000..6e37e1bb --- /dev/null +++ b/examples/strategies_showcase/11_hierarchical_risk_parity/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the Hierarchical Risk Parity strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import HRPStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=365) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(HRPStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/11_hierarchical_risk_parity/requirements.txt b/examples/strategies_showcase/11_hierarchical_risk_parity/requirements.txt new file mode 100644 index 00000000..15b2fd3d --- /dev/null +++ b/examples/strategies_showcase/11_hierarchical_risk_parity/requirements.txt @@ -0,0 +1,5 @@ +investing-algorithm-framework>=8.7.3 +numpy>=1.25 +pandas>=2.0 +scipy>=1.11 +ccxt>=4.0 diff --git a/examples/strategies_showcase/11_hierarchical_risk_parity/strategy.py b/examples/strategies_showcase/11_hierarchical_risk_parity/strategy.py new file mode 100644 index 00000000..e4fccaaa --- /dev/null +++ b/examples/strategies_showcase/11_hierarchical_risk_parity/strategy.py @@ -0,0 +1,135 @@ +"""Hierarchical Risk Parity (HRP) β€” LΓ³pez de Prado, 2016.""" +from __future__ import annotations + +from typing import Any, Dict, List + +import numpy as np +import pandas as pd +from scipy.cluster.hierarchy import linkage, to_tree +from scipy.spatial.distance import squareform + +from investing_algorithm_framework import ( + Context, + DataSource, + DataType, + OrderSide, + OrderType, + TimeUnit, + TradingStrategy, +) + +SYMBOLS = ["BTC/EUR", "ETH/EUR", "SOL/EUR", "ADA/EUR", + "XRP/EUR", "DOT/EUR", "LINK/EUR"] +MARKET = "BITVAVO" +LOOKBACK = 90 +REBALANCE_EVERY_BARS = 30 + + +def _quasi_diag_order(link: np.ndarray) -> List[int]: + tree = to_tree(link, rd=False) + return tree.pre_order(lambda x: x.id) + + +def _cluster_var(cov: np.ndarray, items: List[int]) -> float: + sub = cov[np.ix_(items, items)] + iv = 1.0 / np.diag(sub) + w = iv / iv.sum() + return float(w @ sub @ w) + + +def _hrp_weights(returns: pd.DataFrame) -> pd.Series: + cov = returns.cov().to_numpy() + corr = returns.corr().to_numpy() + dist = np.sqrt(0.5 * (1.0 - corr)) + np.fill_diagonal(dist, 0.0) + link = linkage(squareform(dist, checks=False), method="single") + order = _quasi_diag_order(link) + + weights = np.ones(len(order)) + clusters = [order] + while clusters: + new_clusters = [] + for c in clusters: + if len(c) <= 1: + continue + split = len(c) // 2 + left, right = c[:split], c[split:] + v_left = _cluster_var(cov, left) + v_right = _cluster_var(cov, right) + alpha = 1.0 - v_left / (v_left + v_right) + for i in left: + weights[i] *= alpha + for i in right: + weights[i] *= (1.0 - alpha) + new_clusters += [left, right] + clusters = new_clusters + + return pd.Series(weights, index=returns.columns) + + +class HRPStrategy(TradingStrategy): + algorithm_id = "hierarchical-risk-parity" + time_unit = TimeUnit.DAY + interval = 1 + market = MARKET + trading_symbol = "EUR" + symbols = [s.split("/")[0] for s in SYMBOLS] + + data_sources = [ + DataSource( + identifier=f"{s}-ohlcv", + data_type=DataType.OHLCV, market=MARKET, symbol=s, + time_frame="1d", warmup_window=LOOKBACK + 5, pandas=True, + ) + for s in SYMBOLS + ] + + def __init__(self): + super().__init__() + self._bar_count = 0 + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + self._bar_count += 1 + if self._bar_count % REBALANCE_EVERY_BARS != 1: + return + + cols = {} + for s in SYMBOLS: + df = data[f"{s}-ohlcv"] + if len(df) >= LOOKBACK + 1: + cols[s] = df["Close"].pct_change().tail(LOOKBACK).dropna() + if len(cols) < 2: + return + returns = pd.DataFrame(cols).dropna() + if len(returns) < LOOKBACK - 5: + return + weights = _hrp_weights(returns) + weights = weights / weights.sum() + + equity = context.get_unallocated() + for sym in self.symbols: + if context.has_position(sym, market=self.market): + pos = context.get_position(sym, market=self.market) + price = float(data[f"{sym}/EUR-ohlcv"]["Close"].iloc[-1]) + equity += pos.get_amount() * price + + for symbol, weight in weights.items(): + sym = symbol.split("/")[0] + price = float(data[f"{symbol}-ohlcv"]["Close"].iloc[-1]) + if price <= 0: + continue + target_amount = (equity * float(weight)) / price + current = 0.0 + if context.has_position(sym, market=self.market): + current = context.get_position( + sym, market=self.market + ).get_amount() + delta = target_amount - current + if abs(delta * price) < 5: + continue + side = OrderSide.BUY if delta > 0 else OrderSide.SELL + context.create_order( + target_symbol=sym, order_side=side, + order_type=OrderType.LIMIT, price=price, + amount=abs(delta), + ) diff --git a/examples/strategies_showcase/12_vol_targeting_overlay/README.md b/examples/strategies_showcase/12_vol_targeting_overlay/README.md new file mode 100644 index 00000000..bdd4baa3 --- /dev/null +++ b/examples/strategies_showcase/12_vol_targeting_overlay/README.md @@ -0,0 +1,27 @@ +# 12 β€” Volatility Targeting Overlay + +**Fit:** βœ… Single-asset gross-exposure scaling. + +## Idea + +Sit fully invested in BTC, **scaled so the trailing 30-day realized +portfolio vol** matches an annualized target (e.g. 25%). When BTC vol +spikes, position size shrinks; when vol compresses, position size grows. + +Target weight on BTC: + +$$ w = \min\left(\frac{\sigma_{target}}{\sigma_{realized}},\ w_{max}\right) $$ + +Capped at `w_max = 1.0` (no leverage on a spot venue). + +## Why this fits the framework + +The same delta-rebalance pattern as risk parity: compute target weight, +diff vs. current position, place a single corrective order. + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` diff --git a/examples/strategies_showcase/12_vol_targeting_overlay/backtest.py b/examples/strategies_showcase/12_vol_targeting_overlay/backtest.py new file mode 100644 index 00000000..1149284a --- /dev/null +++ b/examples/strategies_showcase/12_vol_targeting_overlay/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the volatility-targeting overlay strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import VolTargetStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=365) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(VolTargetStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/12_vol_targeting_overlay/requirements.txt b/examples/strategies_showcase/12_vol_targeting_overlay/requirements.txt new file mode 100644 index 00000000..b3c50d3c --- /dev/null +++ b/examples/strategies_showcase/12_vol_targeting_overlay/requirements.txt @@ -0,0 +1,3 @@ +investing-algorithm-framework>=8.7.3 +pandas>=2.0 +ccxt>=4.0 diff --git a/examples/strategies_showcase/12_vol_targeting_overlay/strategy.py b/examples/strategies_showcase/12_vol_targeting_overlay/strategy.py new file mode 100644 index 00000000..2fd46977 --- /dev/null +++ b/examples/strategies_showcase/12_vol_targeting_overlay/strategy.py @@ -0,0 +1,81 @@ +"""Volatility-targeted long-BTC overlay.""" +from __future__ import annotations + +import math +from typing import Any, Dict + +from investing_algorithm_framework import ( + Context, + DataSource, + DataType, + OrderSide, + OrderType, + TimeUnit, + TradingStrategy, +) + +SYMBOL = "BTC/EUR" +LOOKBACK = 30 +TARGET_ANNUAL_VOL = 0.25 # 25% annualised +REBALANCE_EVERY_BARS = 7 # weekly +W_MAX = 1.0 + + +class VolTargetStrategy(TradingStrategy): + algorithm_id = "vol-targeting-overlay" + time_unit = TimeUnit.DAY + interval = 1 + market = "BITVAVO" + trading_symbol = "EUR" + symbols = [SYMBOL.split("/")[0]] + + data_sources = [ + DataSource( + identifier=f"{SYMBOL}-ohlcv", + data_type=DataType.OHLCV, market=market, symbol=SYMBOL, + time_frame="1d", warmup_window=LOOKBACK + 5, pandas=True, + ) + ] + + def __init__(self): + super().__init__() + self._bar_count = 0 + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + self._bar_count += 1 + if self._bar_count % REBALANCE_EVERY_BARS != 1: + return + + df = data[f"{SYMBOL}-ohlcv"] + if len(df) < LOOKBACK + 1: + return + + rets = df["Close"].pct_change().tail(LOOKBACK).dropna() + sd_daily = float(rets.std()) + if sd_daily <= 0 or math.isnan(sd_daily): + return + sd_annual = sd_daily * math.sqrt(365) + target_w = min(TARGET_ANNUAL_VOL / sd_annual, W_MAX) + + sym = self.symbols[0] + price = float(df["Close"].iloc[-1]) + equity = context.get_unallocated() + if context.has_position(sym, market=self.market): + equity += context.get_position( + sym, market=self.market + ).get_amount() * price + + target_amount = (equity * target_w) / price + current = 0.0 + if context.has_position(sym, market=self.market): + current = context.get_position( + sym, market=self.market + ).get_amount() + delta = target_amount - current + if abs(delta * price) < 5: + return + side = OrderSide.BUY if delta > 0 else OrderSide.SELL + context.create_order( + target_symbol=sym, order_side=side, + order_type=OrderType.LIMIT, price=price, amount=abs(delta), + ) diff --git a/examples/strategies_showcase/13_twap_vwap_execution/README.md b/examples/strategies_showcase/13_twap_vwap_execution/README.md new file mode 100644 index 00000000..c8a929af --- /dev/null +++ b/examples/strategies_showcase/13_twap_vwap_execution/README.md @@ -0,0 +1,35 @@ +# 13 β€” TWAP / VWAP Execution + +**Fit:** 🟑 Bar-level TWAP works; sub-bar VWAP does not. + +## Idea + +A **parent order** for a fixed quantity is broken into N **child orders** +that are released **uniformly over a window** (TWAP β€” time-weighted +average price). The framework's bar-driven event loop slices on bar +boundaries; the example uses 10 child orders over 10 daily bars. + +A true **VWAP** would weight slices by historical intra-bar volume β€” that +requires the engine to drop *inside* a bar (or use a fine timeframe), which +is feasible by lowering the `time_unit` to e.g. 1-minute bars. + +## Why this is 🟑 + +- TWAP at the bar level: βœ… trivial, this folder shows it. +- VWAP at intra-bar resolution: 🟑 requires either (a) running on the + smallest timeframe the venue exposes and treating each minute-bar as a + slice, or (b) modelling intra-bar volume curves which the framework does + not provide today. +- Implementation shortfall (Almgren-Chriss): see folder 14 β€” currently πŸ”΄ + because the framework does not model intra-bar fills with explicit + market-impact cost. + +This example shows the **bar-level TWAP** pattern. Use it as a template; +adapt by reducing the bar size for finer slicing. + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` diff --git a/examples/strategies_showcase/13_twap_vwap_execution/backtest.py b/examples/strategies_showcase/13_twap_vwap_execution/backtest.py new file mode 100644 index 00000000..8da31088 --- /dev/null +++ b/examples/strategies_showcase/13_twap_vwap_execution/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the TWAP execution strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import TWAPExecutionStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=60) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(TWAPExecutionStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/13_twap_vwap_execution/requirements.txt b/examples/strategies_showcase/13_twap_vwap_execution/requirements.txt new file mode 100644 index 00000000..b3c50d3c --- /dev/null +++ b/examples/strategies_showcase/13_twap_vwap_execution/requirements.txt @@ -0,0 +1,3 @@ +investing-algorithm-framework>=8.7.3 +pandas>=2.0 +ccxt>=4.0 diff --git a/examples/strategies_showcase/13_twap_vwap_execution/strategy.py b/examples/strategies_showcase/13_twap_vwap_execution/strategy.py new file mode 100644 index 00000000..b4a28999 --- /dev/null +++ b/examples/strategies_showcase/13_twap_vwap_execution/strategy.py @@ -0,0 +1,64 @@ +"""Bar-level TWAP execution: parent order sliced into N child orders.""" +from __future__ import annotations + +from typing import Any, Dict + +from investing_algorithm_framework import ( + Context, + DataSource, + DataType, + OrderSide, + OrderType, + TimeUnit, + TradingStrategy, +) + +SYMBOL = "BTC/EUR" +PARENT_NOTIONAL_EUR = 500.0 +SLICES = 10 +START_AFTER_BARS = 5 # let warmup pass first + + +class TWAPExecutionStrategy(TradingStrategy): + algorithm_id = "twap-bar-level" + time_unit = TimeUnit.DAY + interval = 1 + market = "BITVAVO" + trading_symbol = "EUR" + symbols = [SYMBOL.split("/")[0]] + + data_sources = [ + DataSource( + identifier=f"{SYMBOL}-ohlcv", + data_type=DataType.OHLCV, market=market, symbol=SYMBOL, + time_frame="1d", warmup_window=10, pandas=True, + ) + ] + + def __init__(self): + super().__init__() + self._bar_count = 0 + self._slices_done = 0 + self._slice_notional = PARENT_NOTIONAL_EUR / SLICES + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + self._bar_count += 1 + if self._bar_count <= START_AFTER_BARS: + return + if self._slices_done >= SLICES: + return + + df = data[f"{SYMBOL}-ohlcv"] + if len(df) == 0: + return + price = float(df["Close"].iloc[-1]) + if price <= 0: + return + if context.get_unallocated() < self._slice_notional: + return + amount = (self._slice_notional * 0.995) / price + context.create_order( + target_symbol=self.symbols[0], order_side=OrderSide.BUY, + order_type=OrderType.LIMIT, price=price, amount=amount, + ) + self._slices_done += 1 diff --git a/examples/strategies_showcase/14_implementation_shortfall/README.md b/examples/strategies_showcase/14_implementation_shortfall/README.md new file mode 100644 index 00000000..32c6556b --- /dev/null +++ b/examples/strategies_showcase/14_implementation_shortfall/README.md @@ -0,0 +1,31 @@ +# 14 β€” Implementation Shortfall (Almgren-Chriss) + +**Fit:** πŸ”΄ Currently not directly implementable. + +## Why it is currently not possible + +Implementation Shortfall as a strategy *type* (Almgren-Chriss, Obizhaeva-Wang, +etc.) requires three things the framework does not model today: + +1. **Intra-bar fill simulation.** The optimisation trades off market impact + against price drift over a continuous time horizon. The current backtest + engine fills orders at the *next bar*, which collapses the impact / + timing trade-off into a single discrete decision. +2. **An explicit market-impact cost model.** The framework has a + `SlippageModel` abstraction (`PercentageSlippage`, `VolumeImpactSlippage`, + etc.) which is a step in this direction, but the AC closed form requires + a temporary-impact and permanent-impact decomposition that is not + currently exposed. +3. **A scheduled child-order policy.** AC produces an *optimal trajectory* + of trade rates; expressing that requires the engine to step at a + resolution finer than the bar. + +## What you can do today + +Use folder [13_twap_vwap_execution](../13_twap_vwap_execution) β€” bar-level +TWAP is the simplest member of the IS family and *is* implementable. + +For a true IS implementation, the framework would need: +- a continuous-time (or at least minute-bar) fill model, +- a parent-order abstraction that owns a schedule of child orders, +- a `MarketImpactModel` primitive separate from `SlippageModel`. diff --git a/examples/strategies_showcase/15_iceberg_orders/README.md b/examples/strategies_showcase/15_iceberg_orders/README.md new file mode 100644 index 00000000..a2ed5309 --- /dev/null +++ b/examples/strategies_showcase/15_iceberg_orders/README.md @@ -0,0 +1,31 @@ +# 15 β€” Iceberg Orders + +**Fit:** πŸ”΄ Currently not possible. + +## Why it is currently not possible + +An **iceberg order** displays only a small slice of the total quantity on +the book; once that slice fills, the next slice is auto-replenished by the +*venue*. It is a venue-side primitive (e.g. Binance `ICEBERG_QTY`, +NYSE/BATS reserve order types). + +The framework's `OrderType` enum exposes `LIMIT`, `MARKET`, `STOP_LIMIT` +etc., but not `ICEBERG`, and the order-execution path does not currently +forward venue-specific iceberg parameters. + +## What you can do today + +Approximate an iceberg in **user code** by repeatedly placing small limit +orders at the same price after the previous slice fills. This requires: + +1. Listening for fill events. +2. Sizing the next slice. +3. Cancelling and reposting at the new mid if the price moves. + +This is a non-trivial bookkeeping exercise and is left out of the showcase. +For a venue-native iceberg the framework would need: + +- An `OrderType.ICEBERG` value. +- A `display_quantity` field on `Order`. +- Per-venue mappings in each `OrderExecutor` implementation + (CCXT supports `iceberg_qty` on Binance and a few others). diff --git a/examples/strategies_showcase/16_options_selling_income/README.md b/examples/strategies_showcase/16_options_selling_income/README.md new file mode 100644 index 00000000..27614e45 --- /dev/null +++ b/examples/strategies_showcase/16_options_selling_income/README.md @@ -0,0 +1,36 @@ +# 16 β€” Options Selling for Income (Covered Calls / Cash-Secured Puts) + +**Fit:** πŸ”΄ Currently not possible. + +## Why it is currently not possible + +The framework's domain model is **spot-instrument-only**. There is no +abstraction for: + +- An **options chain** (per-expiry strike grid). +- Per-contract metadata: strike, expiry, contract multiplier, exercise style. +- **Greeks** (delta, gamma, vega, theta). +- **Premium settlement** at expiry / assignment. + +Without these, you cannot express even the simplest option position +(e.g. *short 1 BTC-31MAR-50k call*). + +## What it would take + +A full options expansion would need: + +1. An `OptionInstrument` domain object (underlying, strike, expiry, type, multiplier). +2. An `OptionsChainDataProvider` interface (e.g. CCXT for Deribit). +3. A pricing/Greeks pipeline (Black-76 for European, binomial for American). +4. A portfolio model that can hold short option positions with + margin / collateral accounting. +5. An assignment / expiry settlement engine. + +This is a multi-quarter expansion and is currently out of scope. + +## Workaround today + +If you only need *exposure* like a covered call (long underlying + capped +upside), you can approximate it on spot by **selling above-target prices via +a `take_profit_percentage`** in `StopLossRule` / `TakeProfitRule` β€” but this +is a payoff approximation, not a real options position. diff --git a/examples/strategies_showcase/17_volatility_trading/README.md b/examples/strategies_showcase/17_volatility_trading/README.md new file mode 100644 index 00000000..616ee524 --- /dev/null +++ b/examples/strategies_showcase/17_volatility_trading/README.md @@ -0,0 +1,28 @@ +# 17 β€” Volatility Trading (vol surface arbitrage) + +**Fit:** πŸ”΄ Currently not possible. + +## Why it is currently not possible + +Volatility trading (long/short variance, dispersion, vol-of-vol, calendar +skew, etc.) requires a full options stack: + +- An options chain across strikes and expiries. +- A vol-surface estimator (SVI, SABR, or non-parametric). +- Implied-vol arbitrage signals computed from the surface. +- Variance/volatility swap pricing for variance-replication trades. + +The framework currently models neither **options instruments** +(see folder 16) nor an **implied-volatility surface**. + +## What it would take + +Folder 16's prerequisites, plus: + +- A `VolatilitySurface` primitive with strike/expiry coordinates. +- Surface-fitting helpers (SVI / SABR calibration). +- A variance-swap pricing engine (`VarianceSwap` instrument). + +Until then, "vol trading" in this framework is limited to **spot +realised-volatility plays** like folder [12_vol_targeting_overlay](../12_vol_targeting_overlay), +which is a vol-aware *exposure* strategy, not a vol-arbitrage strategy. diff --git a/examples/strategies_showcase/18_delta_hedged_options/README.md b/examples/strategies_showcase/18_delta_hedged_options/README.md new file mode 100644 index 00000000..cd1a777e --- /dev/null +++ b/examples/strategies_showcase/18_delta_hedged_options/README.md @@ -0,0 +1,28 @@ +# 18 β€” Delta-hedged Options + +**Fit:** πŸ”΄ Currently not possible. + +## Why it is currently not possible + +Delta-hedged options strategies (gamma scalping, vega harvesting, +synthetic-volatility carry) are the canonical use-case for an options +engine *plus* a fast spot-hedging loop. Both are missing today: + +1. **No options instruments / chain primitive** β€” see folder 16. +2. **No Greeks pipeline** β€” there is no `Greeks` domain object that the + strategy can read to compute the required spot hedge ratio. +3. **No mixed-instrument portfolio** β€” the portfolio model holds spot + positions; mixing an option leg with its spot hedge inside one + `Portfolio` requires a different P&L attribution model. + +## What it would take + +Folder 16's prerequisites, plus: + +- A `Greeks` domain object emitted by the options pricing pipeline. +- A `MultiInstrumentPortfolio` that can mark-to-market a mixed + spot+derivatives book. +- A re-hedge scheduler (event-driven, possibly sub-bar) that recomputes + delta and adjusts the hedge leg. + +This is a derivative-engine project of similar scope to folder 16/17. diff --git a/examples/strategies_showcase/19_futures_basis_calendar/README.md b/examples/strategies_showcase/19_futures_basis_calendar/README.md new file mode 100644 index 00000000..7493cbc9 --- /dev/null +++ b/examples/strategies_showcase/19_futures_basis_calendar/README.md @@ -0,0 +1,32 @@ +# 19 β€” Futures Basis / Calendar Spread + +**Fit:** 🟑 Pattern is shown on a spot proxy; no native futures-curve model. + +## Why this is 🟑 + +The framework's data and portfolio models are **spot-only**. To run a +real futures basis (cash-and-carry) or calendar-spread strategy you need: + +1. **Per-expiry futures contracts** with explicit expiry dates and + contract-roll handling. +2. A **futures curve** data structure (front month, back month, generic + continuous contract) so the strategy can read the term structure. +3. **Margin accounting** for short futures legs. + +None of those exist yet. The pattern it expresses, however β€” *take a +long-short position when two related instruments diverge* β€” is **already +shown on spot** in folder [05_pairs_trading](../05_pairs_trading) as a +long-only relative-value variant. + +## What it would take + +- A `FuturesContract` domain object (underlying, expiry, multiplier, + tick size, settlement). +- A `FuturesCurveDataProvider` interface. +- A continuous-contract roll policy (calendar / open-interest based). +- A margin-aware portfolio model. + +## What this folder contains + +A short README only β€” the working *pattern* is the long/short z-score +trade in folder 05; the gap is purely the **instrument model**. diff --git a/examples/strategies_showcase/20_slow_market_making/README.md b/examples/strategies_showcase/20_slow_market_making/README.md new file mode 100644 index 00000000..307266e1 --- /dev/null +++ b/examples/strategies_showcase/20_slow_market_making/README.md @@ -0,0 +1,38 @@ +# 20 β€” Slow Market Making + +**Fit:** 🟒 Workable as a slow, bar-driven quoting strategy. + +## Idea + +Every bar, **post a single buy limit slightly below mid** and a single +**sell limit slightly above mid**. If the buy fills, you accumulate +inventory; if the sell fills, you offload it. Net P&L = the spread you +captured minus inventory drift. + +This is the **slow** version of market-making β€” it operates on bar +boundaries, *not* sub-second. It is a useful template for showing how the +framework supports passive limit quoting; for true market-making see +folder 21 (which is πŸ”΄ β€” currently not possible). + +## Why this is 🟒 and not βœ… + +Bar-level posting works, but a real MM strategy requires: + +- **Continuous quote management.** Cancelling and reposting on every tick + is the entire game. +- **Inventory-aware skew.** Quotes should lean to flatten inventory; this + example does not implement that. +- **A realistic fill model.** The default backtest fill model decides + whether your limit fills based on whether the next-bar OHLC range + intersects your price. That is a coarse approximation of real queue + dynamics; live performance will differ. + +Treat this folder as **structural scaffolding** for a passive quoting bot, +not as a production MM engine. + +## Run + +```bash +pip install -r requirements.txt +python backtest.py +``` diff --git a/examples/strategies_showcase/20_slow_market_making/backtest.py b/examples/strategies_showcase/20_slow_market_making/backtest.py new file mode 100644 index 00000000..04249370 --- /dev/null +++ b/examples/strategies_showcase/20_slow_market_making/backtest.py @@ -0,0 +1,28 @@ +"""Backtest the slow market-making strategy.""" +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework import BacktestDateRange, create_app + +from strategy import SlowMarketMakingStrategy + + +def main() -> None: + end = datetime.now(timezone.utc).replace( + hour=0, minute=0, second=0, microsecond=0 + ) - timedelta(days=1) + start = end - timedelta(days=60) + date_range = BacktestDateRange(start_date=start, end_date=end) + + app = create_app() + app.add_strategy(SlowMarketMakingStrategy) + app.add_market(market="BITVAVO", trading_symbol="EUR", + initial_balance=1000) + + backtest = app.run_backtest(backtest_date_range=date_range) + print(backtest) + + +if __name__ == "__main__": + main() diff --git a/examples/strategies_showcase/20_slow_market_making/requirements.txt b/examples/strategies_showcase/20_slow_market_making/requirements.txt new file mode 100644 index 00000000..b3c50d3c --- /dev/null +++ b/examples/strategies_showcase/20_slow_market_making/requirements.txt @@ -0,0 +1,3 @@ +investing-algorithm-framework>=8.7.3 +pandas>=2.0 +ccxt>=4.0 diff --git a/examples/strategies_showcase/20_slow_market_making/strategy.py b/examples/strategies_showcase/20_slow_market_making/strategy.py new file mode 100644 index 00000000..d764657f --- /dev/null +++ b/examples/strategies_showcase/20_slow_market_making/strategy.py @@ -0,0 +1,75 @@ +"""Slow market-making: post buy-below-mid, sell-above-mid every bar.""" +from __future__ import annotations + +from typing import Any, Dict + +from investing_algorithm_framework import ( + Context, + DataSource, + DataType, + OrderSide, + OrderType, + TimeUnit, + TradingStrategy, +) + +SYMBOL = "BTC/EUR" +SPREAD_BPS = 25.0 # quote Β±25 bps around mid +QUOTE_SIZE_EUR = 50.0 +INVENTORY_CAP_EUR = 250.0 + + +class SlowMarketMakingStrategy(TradingStrategy): + algorithm_id = "slow-market-making" + time_unit = TimeUnit.HOUR + interval = 1 + market = "BITVAVO" + trading_symbol = "EUR" + symbols = [SYMBOL.split("/")[0]] + + data_sources = [ + DataSource( + identifier=f"{SYMBOL}-ohlcv", + data_type=DataType.OHLCV, market=market, symbol=SYMBOL, + time_frame="1h", warmup_window=2, pandas=True, + ) + ] + + def run_strategy(self, context: Context, data: Dict[str, Any]) -> None: + df = data[f"{SYMBOL}-ohlcv"] + if len(df) == 0: + return + mid = float(df["Close"].iloc[-1]) + if mid <= 0: + return + + bid = mid * (1.0 - SPREAD_BPS / 10_000.0) + ask = mid * (1.0 + SPREAD_BPS / 10_000.0) + sym = self.symbols[0] + + # Inventory-cap-aware buy quote. + held_value = 0.0 + if context.has_position(sym, market=self.market): + held_value = context.get_position( + sym, market=self.market + ).get_amount() * mid + + if held_value < INVENTORY_CAP_EUR \ + and context.get_unallocated() >= QUOTE_SIZE_EUR: + buy_amt = (QUOTE_SIZE_EUR * 0.995) / bid + context.create_order( + target_symbol=sym, order_side=OrderSide.BUY, + order_type=OrderType.LIMIT, price=bid, amount=buy_amt, + ) + + # Sell quote: only if we have inventory to offload. + if context.has_position(sym, market=self.market): + pos_amt = context.get_position( + sym, market=self.market + ).get_amount() + sell_amt = min(pos_amt, (QUOTE_SIZE_EUR * 0.995) / ask) + if sell_amt > 0: + context.create_order( + target_symbol=sym, order_side=OrderSide.SELL, + order_type=OrderType.LIMIT, price=ask, amount=sell_amt, + ) diff --git a/examples/strategies_showcase/21_active_market_making/README.md b/examples/strategies_showcase/21_active_market_making/README.md new file mode 100644 index 00000000..8e55e0c3 --- /dev/null +++ b/examples/strategies_showcase/21_active_market_making/README.md @@ -0,0 +1,33 @@ +# 21 β€” Active Market Making + +**Fit:** πŸ”΄ Currently not possible. + +## Why it is currently not possible + +Active market making β€” continuously quoting bid and ask, cancelling and +re-posting on every order-book update, managing inventory in microseconds β€” +has fundamentally different requirements than the framework provides: + +1. **Sub-millisecond event loop.** The current scheduler is bar-driven + (`TimeUnit.MINUTE` is the practical floor). Active MM needs an event + loop that wakes on every L2 update β€” typically *thousands per second*. +2. **L2 order-book data model.** There is no `OrderBook` domain object, + `BookUpdate` event type, or per-level resting-quote state. +3. **Quote-management primitives.** No `replace_order`, no batched + cancel-and-resubmit, no native rate-limit / weight tracking. +4. **Co-located venue connectivity.** WebSocket round-trips at + ~30-100 ms are an order of magnitude too slow to compete with venue-side + liquidity providers. + +## Workaround today + +Run the **slow market-making** template in folder +[20_slow_market_making](../20_slow_market_making) β€” bar-level passive +quoting is implementable and still useful for slow-moving books or +maker-rebate harvesting. + +## What would change this + +A separate, low-latency runtime β€” see the `iaf_realtime` design proposal β€” +is the right home for active MM. Trying to bolt sub-ms semantics onto the +bar-driven engine would compromise the rest of the framework's clarity. diff --git a/examples/strategies_showcase/22_latency_arbitrage/README.md b/examples/strategies_showcase/22_latency_arbitrage/README.md new file mode 100644 index 00000000..78e64387 --- /dev/null +++ b/examples/strategies_showcase/22_latency_arbitrage/README.md @@ -0,0 +1,27 @@ +# 22 β€” Latency Arbitrage + +**Fit:** πŸ”΄ Currently not possible. + +## Why it is currently not possible + +Latency arbitrage profits from being **first** to react to a price change +on one venue and trade against a stale quote on another. Edge measured in +**microseconds** is what the strategy *is* β€” without low-latency +infrastructure there is nothing to capture. + +The framework cannot support this strategy because: + +1. **Bar-driven scheduler.** Reactions happen at the next bar boundary + (1-min minimum). Stale quotes vanish in milliseconds. +2. **No multi-venue real-time tape.** There is no abstraction for + subscribing to N exchanges simultaneously, normalising the streams, + and acting on cross-venue divergence. +3. **No co-location.** The framework is designed to run on a developer + machine or modest VPS, not in a cross-connected exchange data centre. + +## What would change this + +This strategy belongs in a **purpose-built HFT stack** β€” typically C++ or +Rust on a dedicated colocated machine, with kernel-bypass networking +(DPDK, Solarflare). It is structurally outside the scope of a Python +research framework. diff --git a/examples/strategies_showcase/23_hft/README.md b/examples/strategies_showcase/23_hft/README.md new file mode 100644 index 00000000..334257da --- /dev/null +++ b/examples/strategies_showcase/23_hft/README.md @@ -0,0 +1,24 @@ +# 23 β€” High-Frequency Trading (HFT) + +**Fit:** πŸ”΄ Currently not possible. + +## Why it is currently not possible + +"HFT" is not a single strategy β€” it is the *latency regime* in which a +strategy operates (sub-millisecond decision-to-action). The framework +operates in the **second-to-minute** regime. Specifically: + +1. **Python overhead.** Even a tight `run_strategy` callback pays + ~50-200 Β΅s per invocation. HFT budgets are typically <10 Β΅s end-to-end. +2. **Bar-driven scheduling.** The smallest practical bar is 1-minute. +3. **No tick-level data model.** OHLCV bars are the native abstraction; + there is no `Tick` event and no tick-by-tick playback. +4. **No order-book primitives.** See folder 21 / 24. + +## What would change this + +HFT requires its own stack β€” C++/Rust, kernel-bypass NICs, FPGA-accelerated +matching-engine simulators, colocated execution. Trying to add this to the +research framework would compromise both. The right answer is a separate +runtime (see the `iaf_realtime` proposal); this framework remains the +**research and slow-loop deployment** path. diff --git a/examples/strategies_showcase/24_orderbook_microstructure/README.md b/examples/strategies_showcase/24_orderbook_microstructure/README.md new file mode 100644 index 00000000..88d27d43 --- /dev/null +++ b/examples/strategies_showcase/24_orderbook_microstructure/README.md @@ -0,0 +1,32 @@ +# 24 β€” Order-Book Microstructure + +**Fit:** πŸ”΄ Currently not possible. + +## Why it is currently not possible + +Order-book microstructure strategies (queue-position predictors, +order-flow imbalance signals, micro-price models, hidden-liquidity probes) +need the framework to model **L2 (or L3) order books** β€” none of which is +present today: + +1. **No `OrderBook` domain object.** Just `OHLCV` and `Ticker` data types. +2. **No `BookUpdate` event stream.** Microstructure features live in the + per-update delta, not in OHLCV summaries. +3. **No queue-position model.** Simulating where a child order sits in + the queue requires per-level FIFO tracking, which the framework's fill + model does not perform. +4. **No hidden-liquidity inference.** Trade-tape vs. visible-quote + reconciliation requires a tape feed alongside the book feed. + +## What would change this + +A new data-type family β€” `DataType.ORDERBOOK_L2` plus `DataType.TRADES` β€” +backed by a streaming provider, plus a queue-aware fill simulator. This is +a sizeable design change and is part of the `iaf_realtime` proposal rather +than the research framework. + +## Related work today + +For a *very coarse* proxy, you can compute "imbalance"-style features from +1-minute OHLCV `(High - Open) - (Open - Low)` style asymmetries, but that +is a pale shadow of true microstructure analysis. diff --git a/examples/strategies_showcase/25_news_nlp_subsecond/README.md b/examples/strategies_showcase/25_news_nlp_subsecond/README.md new file mode 100644 index 00000000..a3d86dfc --- /dev/null +++ b/examples/strategies_showcase/25_news_nlp_subsecond/README.md @@ -0,0 +1,93 @@ +# 25 β€” News / NLP Event-driven Strategies + +**Fit:** 🟑 Currently not directly implementable, but **planned** β€” see [#534](https://github.com/coding-kitties/investing-algorithm-framework/issues/534). + +## Status today + +Strategies that react to discrete events (macro releases, listing +announcements, on-chain alerts, sentiment spikes from a social-media feed) +are awkward to express today because every `TradingStrategy` is woken on a +fixed bar schedule (`time_unit` + `interval`). The *signal* is the arrival +of a message, not the close of a bar β€” and the framework currently has no +way for a strategy to say *"run me when X happens"*. + +Concrete gaps: + +1. **No trigger abstraction.** A strategy declares one heartbeat + (`time_unit`/`interval`); it cannot also subscribe to *"fire me when an + event of type X arrives"*. +2. **No event data provider family.** `DataProvider` is a *pull* interface + (the engine asks for data each iteration). There is no base class for a + provider that *pushes* discrete events into the runtime β€” neither + polled (every N seconds against a REST/RSS endpoint) nor streamed + (websocket / SSE). +3. **No event tape for backtests.** Even if events arrived live, the + backtest engine has no way to replay a historical event tape merged + with bar timestamps. + +## Planned path forward β€” issue [#534](https://github.com/coding-kitties/investing-algorithm-framework/issues/534) + +The proposal in #534 closes all three gaps with a small, additive change: + +1. **`Trigger` abstraction** β€” `IntervalTrigger` (sugar for the existing + `time_unit`/`interval`) plus `EventTrigger(source=..., filter=...)`. + A strategy can declare any combination of triggers. +2. **`EventDataProvider` family** β€” `PolledNewsDataProvider` (HTTP poll + every N seconds) and `WebSocketEventDataProvider` (subscribe to a + stream). +3. **`EventBus` in the live runtime** + **historical event-tape replay** + in the backtest engine. + +When that lands, the example below becomes runnable and this folder moves +from 🟑 to 🟒. + +## Sketch of the future strategy + +```python +from investing_algorithm_framework import ( + TradingStrategy, EventTrigger, IntervalTrigger, + PolledNewsDataProvider, TimeUnit, +) + +class NewsReactionStrategy(TradingStrategy): + triggers = [ + IntervalTrigger(time_unit=TimeUnit.MINUTE, interval=5), # heartbeat + EventTrigger(source="news", filter=lambda e: e.symbol == "BTC"), + ] + data_sources = [ + PolledNewsDataProvider( + identifier="news", + url="https://example.com/api/crypto-news", + poll_interval_seconds=10, + ), + # ... usual OHLCV sources + ] + + def run_strategy(self, context, data, event=None): + if event is None: + return # heartbeat β€” nothing to do this minute + if event.sentiment > 0.7: + # bullish news on BTC β€” open a small position + ... +``` + +## What you can do today + +For *non-time-critical* sentiment overlays (e.g. *"increase risk if +average sentiment over the past day > X"*) you can already: + +- Pre-compute a sentiment score offline. +- Ingest it via a custom `DataProvider` that returns it as a regular time + series. +- Read it in a bar-driven `run_strategy`, exactly like the multi-factor + pattern in [04_multi_factor_portfolio](../04_multi_factor_portfolio). + +That works today; what #534 adds is **second-level reactivity** to fresh +events as they arrive. + +## Out of scope + +True sub-millisecond HFT remains out of scope for this framework β€” see +[23_hft](../23_hft) and [21_active_market_making](../21_active_market_making). +The trigger work in #534 is aimed at the *seconds-to-minutes* regime where +news-API latency lives, not the microsecond regime. diff --git a/examples/strategies_showcase/26_ml_inference_high_freq/README.md b/examples/strategies_showcase/26_ml_inference_high_freq/README.md new file mode 100644 index 00000000..7ef4105a --- /dev/null +++ b/examples/strategies_showcase/26_ml_inference_high_freq/README.md @@ -0,0 +1,42 @@ +# 26 β€” High-Frequency ML Inference + +**Fit:** πŸ”΄ Currently not possible at high frequency. + +## Why it is currently not possible + +Calling an ML model inside `run_strategy` is **fully supported today** β€” +there is no problem loading a `sklearn` / `xgboost` / `pytorch` model and +predicting at each bar. Many of the showcase strategies above could trivially +be ML-backed (replace the rule with `model.predict(features)`). + +What is **not possible** is doing this at *high frequency*. The +combination of: + +1. Python interpreter overhead (~10-100 Β΅s per call). +2. Framework bar-loop overhead (~ms per iteration). +3. Model inference latency (10-1000 Β΅s for a small tree, 1-100 ms for a + small neural net, much more for an LLM). +4. Bar-driven scheduling (1-minute minimum). + +…makes it a **bar-frequency** ML setup, not a high-frequency one. That is +a different beast. + +## What you can do today + +- **Bar-frequency ML overlay.** Train a model offline, register it as a + feature engineer in the strategy, and use it as a signal at hourly / + daily cadence. This is the standard quant-research workflow and is + already supported. +- **Online learning.** Update a model on each bar with the latest + observation; the bookkeeping fits cleanly into a `TradingStrategy`. + +## What would change this + +Sub-bar ML inference would need: + +- The sub-second event loop discussed in folders 21 / 25. +- A model-server abstraction so inference can run in a dedicated process + with shared-memory / IPC, not blocking the event loop. +- ONNX / TorchScript / C++ inference paths for sub-millisecond latency. + +This is HFT plumbing (folder 23), not framework plumbing. diff --git a/examples/strategies_showcase/README.md b/examples/strategies_showcase/README.md new file mode 100644 index 00000000..eae0d8ef --- /dev/null +++ b/examples/strategies_showcase/README.md @@ -0,0 +1,100 @@ +# Strategy Showcase + +This folder contains one subfolder per strategy *type*. The goal is to show how +to implement different logic patterns and data requirements with the +`investing_algorithm_framework`, and to provide a starting point for your own +research. **The strategies are intentionally simple and are not meant to be +profitable as-is** β€” they are reference implementations of a pattern, not +production-grade alpha. + +Each subfolder is self-contained and ships with: + +- `README.md` β€” what the strategy is, why it exists, fit level, parameters, caveats. +- `strategy.py` β€” the `TradingStrategy` (or pipeline) implementation. +- `backtest.py` β€” a runnable script that wires the strategy into an app and + runs a backtest over a recent date range. +- `requirements.txt` β€” Python dependencies for that specific subfolder. + +If a strategy type cannot be implemented today with the framework, the +subfolder contains **only a `README.md`** explaining *why* it is currently not +possible and what would be needed to make it possible. + +## Fit legend + +| Symbol | Meaning | +|--------|---------| +| βœ… | Sweet spot β€” framework is well-suited and example is end-to-end runnable. | +| 🟒 | Workable β€” implementable with some bookkeeping in user code. | +| 🟑 | Partial β€” pattern is shown but a key primitive is approximated. | +| πŸ”΄ | Not currently possible β€” README only, explains why. | + +## Index + +### 1. Position-taking strategies (signal β†’ orders) + +| # | Strategy | Fit | Description | +|---|----------|-----|-------------| +| 01 | [trend_following](./01_trend_following) | βœ… | Vector backtest of an EMA crossover. | +| 02 | [mean_reversion](./02_mean_reversion) | βœ… | Buy oversold, sell overbought (Bollinger + RSI). | +| 03 | [cross_sectional_momentum](./03_cross_sectional_momentum) | βœ… | Pipeline ranks symbols by 30d return, holds top-N. | +| 04 | [multi_factor_portfolio](./04_multi_factor_portfolio) | βœ… | Pipeline blends momentum + low-vol + liquidity factors. | +| 05 | [pairs_trading](./05_pairs_trading) | 🟒 | Z-score on BTC/ETH log-price spread. | +| 06 | [funding_rate_carry](./06_funding_rate_carry) | 🟑 | Carry pattern shown, but no native perp/funding-rate data provider. | +| 07 | [event_driven_signal](./07_event_driven_signal) | βœ… | Volatility-breakout entry on bar-close events. | +| 08 | [dca_accumulation](./08_dca_accumulation) | βœ… | Scheduled fixed-size buys (dollar-cost averaging). | + +### 2. Portfolio construction strategies + +| # | Strategy | Fit | Description | +|---|----------|-----|-------------| +| 09 | [risk_parity](./09_risk_parity) | βœ… | Inverse-volatility weighting, monthly rebalance. | +| 10 | [mean_variance_markowitz](./10_mean_variance_markowitz) | βœ… | Quarterly Markowitz optimisation (`scipy`). | +| 11 | [hierarchical_risk_parity](./11_hierarchical_risk_parity) | βœ… | LΓ³pez de Prado HRP with `scipy` clustering. | +| 12 | [vol_targeting_overlay](./12_vol_targeting_overlay) | βœ… | Scales gross exposure to a target portfolio vol. | + +### 3. Execution strategies + +| # | Strategy | Fit | Description | +|---|----------|-----|-------------| +| 13 | [twap_vwap_execution](./13_twap_vwap_execution) | 🟑 | Bar-level TWAP slicing of a parent order. | +| 14 | [implementation_shortfall](./14_implementation_shortfall) | πŸ”΄ | Needs intra-bar fills the framework does not currently model. | +| 15 | [iceberg_orders](./15_iceberg_orders) | πŸ”΄ | Requires venue-side iceberg primitives not abstracted by the framework. | + +### 4. Derivatives strategies + +| # | Strategy | Fit | Description | +|---|----------|-----|-------------| +| 16 | [options_selling_income](./16_options_selling_income) | πŸ”΄ | No options instrument / chain primitives in the framework today. | +| 17 | [volatility_trading](./17_volatility_trading) | πŸ”΄ | Requires a vol surface and option pricing pipeline. | +| 18 | [delta_hedged_options](./18_delta_hedged_options) | πŸ”΄ | Requires Greeks pipeline and options venue support. | +| 19 | [futures_basis_calendar](./19_futures_basis_calendar) | 🟑 | Pattern shown on spot proxy; no native futures-curve data model. | + +### 5. Slow market making + +| # | Strategy | Fit | Description | +|---|----------|-----|-------------| +| 20 | [slow_market_making](./20_slow_market_making) | 🟒 | Posts limit orders around mid every bar; capped inventory. | + +### 6. High-frequency / sub-second strategies (not currently possible) + +| # | Strategy | Fit | Description | +|---|----------|-----|-------------| +| 21 | [active_market_making](./21_active_market_making) | πŸ”΄ | Requires sub-ms event loop and L2 book β€” see `iaf_realtime` plan. | +| 22 | [latency_arbitrage](./22_latency_arbitrage) | πŸ”΄ | Requires colocated low-latency venue connectivity. | +| 23 | [hft](./23_hft) | πŸ”΄ | Bar-driven event loop is too slow for true HFT. | +| 24 | [orderbook_microstructure](./24_orderbook_microstructure) | πŸ”΄ | No L2 order-book data model. | +| 25 | [news_nlp_subsecond](./25_news_nlp_subsecond) | οΏ½ | Event-driven trigger model planned β€” see [#534](https://github.com/coding-kitties/investing-algorithm-framework/issues/534). | +| 26 | [ml_inference_high_freq](./26_ml_inference_high_freq) | πŸ”΄ | Inference loop dominated by per-iteration overhead. | + +## Running an example + +```bash +cd examples/strategies_showcase/01_trend_following +pip install -r requirements.txt +python backtest.py +``` + +Backtests fetch OHLCV from the public BITVAVO endpoint via `ccxt` (no API +keys required for market data). The first run will download data and cache +it; subsequent runs reuse the cache. + diff --git a/investing_algorithm_framework/__init__.py b/investing_algorithm_framework/__init__.py index f36a1685..7920915c 100644 --- a/investing_algorithm_framework/__init__.py +++ b/investing_algorithm_framework/__init__.py @@ -35,7 +35,8 @@ Pipeline, Factor, CustomFactor, Filter, \ Returns, AverageDollarVolume, AverageTradedValue, SMA, RSI, \ Volatility, StaticPerSymbol, CrossSectionalMean, RollingBeta, \ - Neutralize + Neutralize, \ + SyncResult, ScheduledDeposit, PortfolioOutOfSyncError from .infrastructure import AzureBlobStorageStateHandler, \ CSVOHLCVDataProvider, CSVTickerDataProvider, CSVURLDataProvider, \ JSONURLDataProvider, ParquetURLDataProvider, \ @@ -285,6 +286,9 @@ "get_normalized_stability", "get_consistency_score", "get_stability_score", + "SyncResult", + "ScheduledDeposit", + "PortfolioOutOfSyncError", ] diff --git a/investing_algorithm_framework/app/app.py b/investing_algorithm_framework/app/app.py index da87d254..8847c825 100644 --- a/investing_algorithm_framework/app/app.py +++ b/investing_algorithm_framework/app/app.py @@ -2242,6 +2242,9 @@ def add_market( initial_balance=None, fee_percentage=0.0, slippage_percentage=0.0, + deposit_schedule=None, + auto_sync=False, + auto_sync_error_mode="raise", ): """ Function to add a market to the app. This function is a utility @@ -2260,10 +2263,28 @@ def add_market( slippage_percentage: Default slippage percentage for all trades on this market (e.g. 0.05 for 0.05%). Can be overridden per-symbol via TradingCost on the strategy. + deposit_schedule: Optional list of + :class:`ScheduledDeposit` describing simulated external + cash flows landing on this market during a backtest + (e.g. monthly paychecks). Ignored in live mode β€” for + live deployments the broker is the source of truth and + ``Context.sync_portfolio()`` queries it directly. + auto_sync: When ``True``, the framework automatically calls + ``Context.sync_portfolio(market=market)`` before every + strategy iteration so deposits/withdrawals are absorbed + without strategy code having to opt in. Defaults to + ``False`` (explicit opt-in via ``context.sync_portfolio()``). + auto_sync_error_mode: How auto-sync handles failures. One of + ``"raise"`` (loud, default β€” best for development), + ``"warn"`` (log and continue with stale state β€” best for + live trading where transient broker glitches should not + crash the bot), or ``"halt"`` (log, disable auto-sync + for this market, and continue). Returns: None """ + deposit_schedule = self._normalize_deposit_schedule(deposit_schedule) portfolio_configuration = PortfolioConfiguration( market=market, @@ -2271,6 +2292,7 @@ def add_market( initial_balance=initial_balance, fee_percentage=fee_percentage, slippage_percentage=slippage_percentage, + deposit_schedule=deposit_schedule, ) self.add_portfolio_configuration(portfolio_configuration) @@ -2281,6 +2303,78 @@ def add_market( ) self.add_market_credential(market_credential) + tracker = self.container.broker_balance_tracker() + if deposit_schedule: + tracker.set_schedule(market, deposit_schedule) + if auto_sync: + tracker.set_auto_sync(market, True) + tracker.set_auto_sync_error_mode(market, auto_sync_error_mode) + + @staticmethod + def _normalize_deposit_schedule(deposit_schedule): + """Coerce a deposit_schedule argument to a validated list. + + Accepts ``None`` (treated as no schedule), a list/tuple of + :class:`ScheduledDeposit`, or rejects anything else with a clear + error. Notably rejects passing a single ``ScheduledDeposit`` + directly (which would be iterable over its dataclass fields and + produce nonsense). + """ + from investing_algorithm_framework.domain import ScheduledDeposit + if deposit_schedule is None: + return [] + if isinstance(deposit_schedule, ScheduledDeposit): + raise OperationalException( + "deposit_schedule must be a list of ScheduledDeposit, not a " + "single ScheduledDeposit. Wrap it: [ScheduledDeposit(...)]." + ) + if not isinstance(deposit_schedule, (list, tuple)): + raise OperationalException( + "deposit_schedule must be a list of ScheduledDeposit, got " + f"{type(deposit_schedule).__name__}." + ) + for entry in deposit_schedule: + if not isinstance(entry, ScheduledDeposit): + raise OperationalException( + "deposit_schedule entries must be ScheduledDeposit, " + f"got {type(entry).__name__}." + ) + return list(deposit_schedule) + + def add_deposit_schedule(self, market, schedule): + """Register simulated external deposits for a backtested market. + + Equivalent to passing ``deposit_schedule=`` to :meth:`add_market`, + but usable after the market has already been registered. Replaces + any previously registered schedule for this market. + + Args: + market: Market identifier. + schedule: Iterable of :class:`ScheduledDeposit`. + + Returns: + None + """ + self.container.broker_balance_tracker().set_schedule( + market, self._normalize_deposit_schedule(schedule) + ) + + def set_market_auto_sync(self, market, enabled=True): + """Toggle automatic ``sync_portfolio`` before each strategy iteration. + + Args: + market: Market identifier. + enabled: When ``True`` (the default), the framework calls + ``Context.sync_portfolio(market=market)`` immediately before + each strategy ``run_strategy`` invocation. + + Returns: + None + """ + self.container.broker_balance_tracker().set_auto_sync( + market, enabled + ) + def set_blotter(self, blotter): """ Set a blotter for order book management. The blotter sits diff --git a/investing_algorithm_framework/app/context.py b/investing_algorithm_framework/app/context.py index 368fb897..eb842991 100644 --- a/investing_algorithm_framework/app/context.py +++ b/investing_algorithm_framework/app/context.py @@ -5,12 +5,16 @@ from investing_algorithm_framework.services import ConfigurationService, \ MarketCredentialService, OrderService, PortfolioConfigurationService, \ PortfolioService, PositionService, TradeService, DataProviderService, \ - TradeStopLossService, TradeTakeProfitService + TradeStopLossService, TradeTakeProfitService, BrokerBalanceTracker +from investing_algorithm_framework.services.portfolios import ( + PortfolioProviderLookup, +) from investing_algorithm_framework.domain import OrderStatus, OrderType, \ OrderSide, OperationalException, Portfolio, RoundingService, \ BACKTESTING_FLAG, INDEX_DATETIME, Order, \ Position, Trade, TradeStatus, MarketCredential, TradeStopLoss, \ - TradeTakeProfit + TradeTakeProfit, SyncResult, PortfolioOutOfSyncError, Environment, \ + ENVIRONMENT logger = logging.getLogger("investing_algorithm_framework") @@ -33,7 +37,9 @@ def __init__( trade_service: TradeService, trade_stop_loss_service: TradeStopLossService, trade_take_profit_service: TradeTakeProfitService, - data_provider_service: DataProviderService + data_provider_service: DataProviderService, + portfolio_provider_lookup: PortfolioProviderLookup = None, + broker_balance_tracker: BrokerBalanceTracker = None, ): self.configuration_service: ConfigurationService = \ configuration_service @@ -50,6 +56,10 @@ def __init__( trade_stop_loss_service self.trade_take_profit_service: TradeTakeProfitService = \ trade_take_profit_service + self.portfolio_provider_lookup: PortfolioProviderLookup = \ + portfolio_provider_lookup + self.broker_balance_tracker: BrokerBalanceTracker = \ + broker_balance_tracker self._blotter = None self._fx_rate_provider = None self._base_currency = None @@ -134,7 +144,8 @@ def create_order( execute=True, validate=True, sync=True, - validate_symbol=False + validate_symbol=False, + stop_price=None, ) -> Order: """ Function to create an order. This function will create an order @@ -154,6 +165,8 @@ def create_order( with the portfolio of the algorithm. validate_symbol: Default False. If set to True, validates that target_symbol is not the trading_symbol. + stop_price: Required for STOP and STOP_LIMIT order types. + The trigger price at which the order activates. Returns: The order created @@ -173,6 +186,9 @@ def create_order( "trading_symbol": portfolio.trading_symbol, } + if stop_price is not None: + order_data["stop_price"] = stop_price + if BACKTESTING_FLAG in self.configuration_service.config \ and self.configuration_service.config[BACKTESTING_FLAG]: order_data["created_at"] = \ @@ -884,6 +900,280 @@ def get_unallocated(self, market=None) -> float: {"portfolio": portfolio.id, "symbol": trading_symbol} ).get_amount() + def sync_portfolio( + self, + market: str = None, + allow_withdrawals: bool = False, + tolerance: float = 1e-9, + ) -> SyncResult: + """Reconcile the local portfolio's unallocated balance with the broker. + + This is the **canonical entry point** for "make my strategy aware of + cash that arrived (or left) my account out-of-band". The contract is + identical across live and backtest modes: + + 1. Ask the broker (live: registered :class:`PortfolioProvider`; + backtest: simulated :class:`BrokerBalanceTracker`) what the + trading-symbol balance currently is. + 2. In live mode, subtract cash reserved for orders the framework + knows about but the exchange has not yet acknowledged + (``OrderStatus.CREATED``). Without this, the natural race + between local order creation and exchange acknowledgement + would surface as a phantom "deposit". + 3. Compute ``delta = adjusted_broker_available - local_unallocated``. + 4. ``abs(delta) <= tolerance`` β†’ no-op. + 5. ``delta > 0`` β†’ an external deposit landed; absorb it by topping + up ``unallocated``. + 6. ``delta < 0`` β†’ the broker reports *less* than the framework + expected (an external withdrawal, an out-of-band fill, an + unrecorded fee, …). By default this raises + :class:`PortfolioOutOfSyncError` because silently shrinking + the strategy's working capital is almost always the wrong + thing to do. Pass ``allow_withdrawals=True`` to explicitly + accept the drain. + + The absorbed cash flow is recorded on the + :class:`BrokerBalanceTracker` so the snapshot service can attach + it to the next portfolio snapshot's ``cash_flow`` field, which in + turn lets return metrics (CAGR, monthly/yearly returns) compute + true time-weighted returns instead of being inflated by deposits. + + Args: + market: Market identifier. Defaults to the first registered + portfolio. Case-insensitive. + allow_withdrawals: When ``True``, negative deltas drain + ``unallocated`` instead of raising. The drain is still + refused if it would push ``unallocated`` below zero. + tolerance: Absolute drift below which the sync is treated as + a noop. Defaults to ``1e-9`` to swallow floating-point + dust. Useful to bump (e.g. ``1.0``) for live mode if + small fee/rounding glitches keep tripping the check. + + Returns: + :class:`SyncResult` describing the outcome. + + Raises: + PortfolioOutOfSyncError: On negative delta when + ``allow_withdrawals=False``, or when the resulting + ``unallocated`` would be negative. + OperationalException: When live mode is configured but no + :class:`PortfolioProvider` / :class:`MarketCredential` + is registered for the market. + """ + if tolerance < 0: + raise OperationalException( + f"sync_portfolio: tolerance must be non-negative, got " + f"{tolerance}." + ) + portfolio = self._resolve_portfolio_for_sync(market) + market_id = portfolio.market + previous_unallocated = float(portfolio.get_unallocated() or 0.0) + + broker_available, reserved = self._fetch_broker_available( + portfolio, previous_unallocated + ) + + delta = broker_available - previous_unallocated + + if abs(delta) <= tolerance: + return SyncResult( + market=market_id, + kind="noop", + delta=delta, + broker_available=broker_available, + previous_unallocated=previous_unallocated, + new_unallocated=previous_unallocated, + within_tolerance=delta != 0, + reserved_for_pending_orders=reserved, + ) + + if delta < 0 and not allow_withdrawals: + raise PortfolioOutOfSyncError( + f"Portfolio out of sync on market '{market_id}': local " + f"unallocated {previous_unallocated} > broker available " + f"{broker_available} (delta {delta}, " + f"{reserved} reserved for pending orders). This usually " + f"means an external withdrawal happened, an order filled " + f"out-of-band, or fees were charged the framework did not " + f"see. Pass allow_withdrawals=True to drain unallocated, " + f"or investigate the broker account.", + market=market_id, + local_unallocated=previous_unallocated, + broker_available=broker_available, + delta=delta, + ) + + new_unallocated = broker_available + if new_unallocated < 0: + raise PortfolioOutOfSyncError( + f"Refusing to set unallocated to a negative value on market " + f"'{market_id}': broker reports {broker_available}, which is " + f"below zero. Investigate the broker account.", + market=market_id, + local_unallocated=previous_unallocated, + broker_available=broker_available, + delta=delta, + ) + + kind = "deposit" if delta > 0 else "withdrawal" + self._apply_unallocated_change(portfolio, new_unallocated) + + # Record the cash flow so the next portfolio snapshot gets a + # non-zero ``cash_flow`` and TWR-aware metrics work correctly. + if self.broker_balance_tracker is not None: + self.broker_balance_tracker.record_cash_flow(market_id, delta) + + logger.info( + "sync_portfolio[%s] %s: local %.6f -> %.6f (broker reports %.6f, " + "delta %+.6f, reserved %.6f)", + market_id, kind, previous_unallocated, new_unallocated, + broker_available, delta, reserved, + ) + + return SyncResult( + market=market_id, + kind=kind, + delta=delta, + broker_available=broker_available, + previous_unallocated=previous_unallocated, + new_unallocated=new_unallocated, + within_tolerance=False, + reserved_for_pending_orders=reserved, + ) + + def _resolve_portfolio_for_sync(self, market) -> Portfolio: + if market is not None: + # Portfolio.market is canonically uppercased on creation; match + # case-insensitively so users can pass "binance" or "BINANCE". + normalized = str(market).upper() + portfolio = self.portfolio_service.find({"market": normalized}) + if portfolio is None: + portfolio = self.portfolio_service.find({"market": market}) + else: + portfolios = self.portfolio_service.get_all() + if not portfolios: + raise OperationalException( + "sync_portfolio: no portfolio registered. " + "Did you call app.add_market(...)?" + ) + portfolio = portfolios[0] + if portfolio is None: + raise OperationalException( + f"sync_portfolio: no portfolio found for market '{market}'." + ) + return portfolio + + def _fetch_broker_available( + self, portfolio: Portfolio, previous_unallocated: float + ) -> tuple: + """Returns (broker_available, reserved_for_pending_orders).""" + config = self.configuration_service.get_config() + environment = config.get(ENVIRONMENT) + + is_backtest = environment in ( + Environment.BACKTEST.value, + Environment.BACKTEST, + ) + + if is_backtest: + if self.broker_balance_tracker is None: + # No tracker wired (legacy app construction); deposits cannot + # be simulated β†’ broker == local. + return previous_unallocated, 0.0 + pending = self.broker_balance_tracker.consume_pending( + portfolio.market + ) + return previous_unallocated + pending, 0.0 + + # Live mode + if self.portfolio_provider_lookup is None: + raise OperationalException( + "sync_portfolio: no PortfolioProviderLookup wired into the " + "context. This usually means the app was constructed without " + "the standard dependency container." + ) + market_credential = self.market_credential_service.get( + portfolio.market + ) + if market_credential is None: + raise OperationalException( + f"sync_portfolio: no market credential registered for " + f"market '{portfolio.market}'. Live broker reconciliation " + f"requires API credentials." + ) + provider = self.portfolio_provider_lookup.get_portfolio_provider( + portfolio.market + ) + if provider is None: + raise OperationalException( + f"sync_portfolio: no PortfolioProvider registered for market " + f"'{portfolio.market}'." + ) + position = provider.get_position( + portfolio, portfolio.trading_symbol, market_credential + ) + raw = float(position.amount) if position is not None else 0.0 + + # Subtract cash reserved for orders the framework has issued but + # the exchange has not yet acknowledged. ``free`` from the broker + # already excludes acknowledged open orders, but not those in + # CREATED state β€” without this adjustment a brief race window + # between create_order() and the exchange ack would surface as a + # phantom "deposit" of the order's cost. + reserved = self._reserved_cash_for_pending_orders(portfolio) + return raw - reserved, reserved + + def _reserved_cash_for_pending_orders( + self, portfolio: Portfolio + ) -> float: + """Sum of trading-symbol cash locked by orders the framework has + created but the exchange has not yet filled or cancelled. + + Buys consume cash; sells release it. Only ``CREATED`` orders are + counted because the broker's ``free`` balance already excludes + acknowledged open orders. + """ + try: + created_orders = self.order_service.get_all({ + "portfolio_id": portfolio.id, + "status": OrderStatus.CREATED.value, + }) + except Exception: # noqa: BLE001 + return 0.0 + reserved = 0.0 + for order in created_orders or []: + try: + price = float(order.get_price() or 0.0) + amount = float( + order.get_remaining() or order.get_amount() or 0.0 + ) + except Exception: # noqa: BLE001 + continue + if OrderSide.BUY.equals(order.get_order_side()): + reserved += price * amount + # Sells release cash on fill; not counted here. + return reserved + + def _apply_unallocated_change( + self, portfolio: Portfolio, new_unallocated: float + ) -> None: + self.portfolio_service.update( + portfolio.id, {"unallocated": new_unallocated} + ) + # Keep the trading-symbol position in lockstep, mirroring the + # behaviour of PortfolioSyncService.sync_unallocated(). + try: + trading_position = self.position_service.find({ + "portfolio": portfolio.id, + "symbol": portfolio.trading_symbol, + }) + except Exception: # noqa: BLE001 β€” repository raises on miss + trading_position = None + if trading_position is not None: + self.position_service.update( + trading_position.id, {"amount": new_unallocated} + ) + def get_total_size(self): """ Returns the total size of the portfolio. @@ -2093,13 +2383,12 @@ def get_stop_losses( return self.trade_stop_loss_service.get_all(query_params) def _get_url_provider_cache_key(self, url, headers): - if not headers: - return url - - return ( - url, - tuple(sorted(headers.items())) - ) + # Delegates to the canonical helper so the in-memory provider + # dict and the on-disk cache filename stay in lockstep β€” see + # ``url_cache_key`` for rationale. + from investing_algorithm_framework.infrastructure \ + .data_providers.base_url import url_cache_key + return url_cache_key(url, headers) def fetch_csv( self, @@ -2129,7 +2418,9 @@ def fetch_csv( refresh_interval (str, optional): Re-fetch interval (e.g., "1d", "1h"). headers (dict, optional): HTTP headers to send with the - request. + request. Header values are redacted (replaced + with "***") in ``DataSource.to_dict``, so + secrets do not leak into diagnostic payloads. pre_process (callable, optional): Transform raw CSV text before parsing. post_process (callable, optional): Transform the parsed @@ -2202,7 +2493,9 @@ def fetch_json( refresh_interval (str, optional): Re-fetch interval (e.g., "1d", "1h"). headers (dict, optional): HTTP headers to send with the - request. + request. Header values are redacted (replaced + with "***") in ``DataSource.to_dict``, so + secrets do not leak into diagnostic payloads. pre_process (callable, optional): Transform raw JSON text before parsing. post_process (callable, optional): Transform the parsed @@ -2270,7 +2563,9 @@ def fetch_parquet( refresh_interval (str, optional): Re-fetch interval (e.g., "1d", "1h"). headers (dict, optional): HTTP headers to send with the - request. + request. Header values are redacted (replaced + with "***") in ``DataSource.to_dict``, so + secrets do not leak into diagnostic payloads. post_process (callable, optional): Transform the parsed DataFrame. diff --git a/investing_algorithm_framework/app/eventloop.py b/investing_algorithm_framework/app/eventloop.py index ae0f197e..b41ea58d 100644 --- a/investing_algorithm_framework/app/eventloop.py +++ b/investing_algorithm_framework/app/eventloop.py @@ -237,12 +237,14 @@ def _snapshot( snapshot_interval = self._configuration_service\ .config[SNAPSHOT_INTERVAL] portfolio = self._portfolio_service.get_all()[0] + cash_flow = self._drain_cash_flow_for_snapshot(portfolio) if SnapshotInterval.STRATEGY_ITERATION.equals(snapshot_interval): snapshot = self._portfolio_snapshot_service.create_snapshot( created_at=current_datetime, portfolio=portfolio, open_orders=open_orders, created_orders=created_orders, + cash_flow=cash_flow, save=False, ) self._snapshots.append(snapshot) @@ -263,6 +265,7 @@ def _snapshot( portfolio=portfolio, open_orders=open_orders, created_orders=created_orders, + cash_flow=cash_flow, save=False, ) self._snapshots.append(snapshot) @@ -270,6 +273,22 @@ def _snapshot( LAST_SNAPSHOT_DATETIME, current_datetime ) + def _drain_cash_flow_for_snapshot(self, portfolio) -> float: + """Return external cash flow absorbed since the last snapshot. + + Drains the per-market accumulator on the + :class:`BrokerBalanceTracker` so the next snapshot starts from + zero again. Returns 0 if no tracker is wired or the portfolio's + market is unknown to the tracker. + """ + tracker = getattr(self.context, "broker_balance_tracker", None) + if tracker is None or portfolio is None: + return 0.0 + market = getattr(portfolio, "market", None) + if market is None or not tracker.has_market(market): + return 0.0 + return tracker.drain_cash_flow(market) + def initialize( self, algorithm: Algorithm, @@ -534,6 +553,63 @@ def _maybe_validate_live_envelope( self._validate_live_envelope(strategies) self._pipelines_live_validated = True + def _tick_broker_balance(self, environment, current_datetime) -> None: + """Fire any due ScheduledDeposits in backtest mode. + + In live mode this is a no-op: the actual broker is the source of + truth and is consulted directly by ``Context.sync_portfolio``. + """ + if not Environment.BACKTEST.equals(environment): + return + tracker = getattr(self.context, "broker_balance_tracker", None) + if tracker is None: + return + for market in tracker.markets(): + tracker.fire_due_deposits( + market=market, + current_datetime=current_datetime, + ) + + def _auto_sync_markets(self) -> None: + """Call sync_portfolio for any market opted into auto-sync.""" + tracker = getattr(self.context, "broker_balance_tracker", None) + if tracker is None: + return + from investing_algorithm_framework.domain import ( + PortfolioOutOfSyncError, + ) + for market in tracker.markets(): + if not tracker.is_auto_sync(market): + continue + mode = tracker.get_auto_sync_error_mode(market) + try: + self.context.sync_portfolio(market=market) + except PortfolioOutOfSyncError: + # Out-of-sync is a *signal*, not a transient. Always + # propagate so users can decide explicitly via + # allow_withdrawals or by switching off auto_sync. + raise + except Exception as exc: # noqa: BLE001 + if mode == "raise": + raise + if mode == "warn": + logger.warning( + "auto-sync failed on market %s (mode=warn, " + "continuing with stale local state): %s", + market, exc, + ) + continue + if mode == "halt": + # Halt only this market's auto-sync. Manual + # context.sync_portfolio calls still work. + logger.error( + "auto-sync failed on market %s (mode=halt, " + "disabling auto-sync for this market): %s", + market, exc, + ) + tracker.set_auto_sync(market, False) + continue + def _filter_symbols_for_universe_cache( self, strategy_id: str, @@ -720,6 +796,19 @@ def _run_iteration( # backtests. See #503 phase 3b. self._maybe_validate_live_envelope(strategies, environment) + # Reconcile broker balance with local unallocated: + # + # 1. In **backtest** mode, advance the simulated broker clock by + # firing any ScheduledDeposits whose cadence has elapsed since + # the previous tick. The deposits land in + # BrokerBalanceTracker.pending and become visible to strategies + # only when they (or auto-sync) call sync_portfolio. + # 2. For every market with auto_sync enabled (live or backtest), + # call Context.sync_portfolio so deposits/withdrawals are + # absorbed before any strategy or task sees the iteration. + self._tick_broker_balance(environment, current_datetime) + self._auto_sync_markets() + # Step 1: Collect all data for the strategies and for the # pending orders open_orders = self._order_service.get_all( diff --git a/investing_algorithm_framework/app/reporting/backtest_report.py b/investing_algorithm_framework/app/reporting/backtest_report.py index 9290b38c..e0fa8b5a 100644 --- a/investing_algorithm_framework/app/reporting/backtest_report.py +++ b/investing_algorithm_framework/app/reporting/backtest_report.py @@ -540,6 +540,23 @@ def _build_run_data(self): for v, d in m.drawdown_series ] + # TWR (alpha-only) equity curve & drawdown β€” these + # remove the contribution of external deposits/ + # withdrawals so the curve reflects pure trading P&L. + twr_eq = [] + if m and getattr(m, "twr_equity_curve", None): + twr_initial = m.twr_equity_curve[0][0] or 1 + twr_eq = [ + [(v / twr_initial - 1) * 100, _fmt_date(d)] + for v, d in m.twr_equity_curve + ] + twr_dd = [] + if m and getattr(m, "twr_drawdown_series", None): + twr_dd = [ + [v * 100 if abs(v) < 1 else v, _fmt_date(d)] + for v, d in m.twr_drawdown_series + ] + # Rolling Sharpe rs = [] if m and m.rolling_sharpe_ratio: @@ -668,6 +685,8 @@ def _build_run_data(self): 'average_trade_gain', 'average_trade_gain_percentage', 'max_drawdown_absolute', 'max_daily_drawdown', + 'twr_max_drawdown', + 'twr_max_drawdown_duration', 'average_trade_duration', 'average_win_duration', 'average_loss_duration', @@ -810,11 +829,15 @@ def _build_run_data(self): eq = _downsample(eq) dd = _downsample(dd) rs = _downsample(rs) + twr_eq = _downsample(twr_eq) + twr_dd = _downsample(twr_dd) run_data[rid] = { 'label': label, 'EQ': eq, 'DD': dd, + 'TWR_EQ': twr_eq, + 'TWR_DD': twr_dd, 'CR': eq, 'RS': rs, 'MR': mr, diff --git a/investing_algorithm_framework/app/reporting/templates/dashboard.js b/investing_algorithm_framework/app/reporting/templates/dashboard.js index 51bb1c1f..5d60ff97 100644 --- a/investing_algorithm_framework/app/reporting/templates/dashboard.js +++ b/investing_algorithm_framework/app/reporting/templates/dashboard.js @@ -1436,6 +1436,33 @@ function drawStrategyEquity(stratIdx) { i===0?ctx.moveTo(x,y):ctx.lineTo(x,y); } ctx.stroke(); + + // TWR overlay (alpha-only, scrubs deposits) β€” dashed line in + // single-run view only. + if (showDates) { + const rd = RUN_DATA[s.rid]; + const twr = rd && rd.TWR_EQ; + if (twr && twr.length > 1) { + ctx.save(); + ctx.beginPath(); + ctx.strokeStyle = s.color; + ctx.globalAlpha = 0.55; + ctx.lineWidth = 1.5; + ctx.setLineDash([4, 3]); + for (let i=0;i 1) { + ctx.save(); + ctx.beginPath(); + ctx.strokeStyle = s.color; + ctx.globalAlpha = 0.55; + ctx.lineWidth = 1.5; + ctx.setLineDash([4, 3]); + for (let i = 0; i < twrDd.length; i++) { + const x = pad.l+(i/(twrDd.length-1))*cw; + const y = pad.t+((0-twrDd[i][0])/range)*ch; + i === 0 ? ctx.moveTo(x, y) : ctx.lineTo(x, y); + } + ctx.stroke(); + ctx.restore(); + ctx.fillStyle = COL.dim; + ctx.font = '10px JetBrains Mono, monospace'; + ctx.textAlign = 'left'; + ctx.fillText('β€” solid: account Β· ‧ ‧ dashed: TWR (alpha-only)', pad.l, pad.t-3); + } + } }); const canvas = document.getElementById(id); diff --git a/investing_algorithm_framework/dependency_container.py b/investing_algorithm_framework/dependency_container.py index dec95254..2e583f30 100644 --- a/investing_algorithm_framework/dependency_container.py +++ b/investing_algorithm_framework/dependency_container.py @@ -12,7 +12,8 @@ ConfigurationService, PortfolioSnapshotService, \ PositionSnapshotService, MarketCredentialService, TradeService, \ PortfolioSyncService, OrderExecutorLookup, PortfolioProviderLookup, \ - DataProviderService, TradeTakeProfitService, TradeStopLossService + DataProviderService, TradeTakeProfitService, TradeStopLossService, \ + BrokerBalanceTracker def setup_dependency_container(app, modules=None, packages=None): @@ -45,6 +46,9 @@ class DependencyContainer(containers.DeclarativeContainer): portfolio_provider_lookup = providers.ThreadSafeSingleton( PortfolioProviderLookup, ) + broker_balance_tracker = providers.ThreadSafeSingleton( + BrokerBalanceTracker, + ) portfolio_repository = providers.Factory(SQLPortfolioRepository) position_snapshot_repository = providers.Factory( SQLPositionSnapshotRepository @@ -161,6 +165,8 @@ class DependencyContainer(containers.DeclarativeContainer): data_provider_service=data_provider_service, trade_stop_loss_service=trade_stop_loss_service, trade_take_profit_service=trade_take_profit_service, + portfolio_provider_lookup=portfolio_provider_lookup, + broker_balance_tracker=broker_balance_tracker, ) algorithm_factory = providers.Factory( AlgorithmFactory, diff --git a/investing_algorithm_framework/domain/__init__.py b/investing_algorithm_framework/domain/__init__.py index 0443d442..89e28c01 100644 --- a/investing_algorithm_framework/domain/__init__.py +++ b/investing_algorithm_framework/domain/__init__.py @@ -16,13 +16,15 @@ from .data_structures import PeekableQueue from .decimal_parsing import parse_decimal_to_string, parse_string_to_decimal from .exceptions import OperationalException, ApiException, DataError, \ - PermissionDeniedApiException, ImproperlyConfigured, NetworkError + PermissionDeniedApiException, ImproperlyConfigured, NetworkError, \ + PortfolioOutOfSyncError from .models import OrderStatus, OrderSide, OrderType, TimeInterval, \ TimeUnit, TimeFrame, PortfolioConfiguration, Portfolio, Position, \ Order, TradeStatus, StrategyProfile, Trade, MarketCredential, \ AppMode, DataType, DataSource, PortfolioSnapshot, PositionSnapshot, \ TradeTakeProfit, TradeStopLoss, Event, SnapshotInterval, \ - TakeProfitRule, StopLossRule, PositionSize, ScalingRule, TradingCost + TakeProfitRule, StopLossRule, PositionSize, ScalingRule, TradingCost, \ + SyncResult, ScheduledDeposit from .order_executor import OrderExecutor from .portfolio_provider import PortfolioProvider from .blotter import Blotter, DefaultBlotter, SimulationBlotter, Transaction, \ @@ -168,6 +170,9 @@ "BacktestEvaluationFocus", 'combine_backtests', 'PositionSize', + "SyncResult", + "ScheduledDeposit", + "PortfolioOutOfSyncError", 'generate_backtest_summary_metrics', 'DataError', 'TakeProfitRule', diff --git a/investing_algorithm_framework/domain/backtesting/backtest_metrics.py b/investing_algorithm_framework/domain/backtesting/backtest_metrics.py index 052c07ca..6cb97100 100644 --- a/investing_algorithm_framework/domain/backtesting/backtest_metrics.py +++ b/investing_algorithm_framework/domain/backtesting/backtest_metrics.py @@ -188,6 +188,15 @@ class BacktestMetrics: max_drawdown_absolute: float = 0.0 max_daily_drawdown: float = 0.0 max_drawdown_duration: int = 0 + # TWR (alpha-only) variants β€” scrub external cash flows so deposits + # don't mask drawdowns. The raw fields above remain account-value + # based for absolute reporting. + twr_equity_curve: List[Tuple[float, datetime]] = \ + field(default_factory=list) + twr_drawdown_series: List[Tuple[float, datetime]] = \ + field(default_factory=list) + twr_max_drawdown: float = 0.0 + twr_max_drawdown_duration: int = 0 trades_per_year: float = 0.0 trades_per_week: float = 0.0 trades_per_month: float = 0.0 @@ -299,6 +308,14 @@ def ensure_iso(value): "max_drawdown_absolute": self.max_drawdown_absolute, "max_daily_drawdown": self.max_daily_drawdown, "max_drawdown_duration": self.max_drawdown_duration, + "twr_equity_curve": [(value, ensure_iso(date)) + for value, date in self.twr_equity_curve], + "twr_drawdown_series": [ + (value, ensure_iso(date)) + for value, date in self.twr_drawdown_series + ], + "twr_max_drawdown": self.twr_max_drawdown, + "twr_max_drawdown_duration": self.twr_max_drawdown_duration, "trades_per_year": self.trades_per_year, "trades_per_week": self.trades_per_week, "trades_per_month": self.trades_per_month, @@ -484,6 +501,12 @@ def from_dict(cls, data: dict) -> 'BacktestMetrics': data['drawdown_series'] = cls._parse_tuple_list_datetime( data.get('drawdown_series', []) ) + data['twr_equity_curve'] = cls._parse_tuple_list_datetime( + data.get('twr_equity_curve', []) + ) + data['twr_drawdown_series'] = cls._parse_tuple_list_datetime( + data.get('twr_drawdown_series', []) + ) data['cumulative_return_series'] = cls._parse_tuple_list_datetime( data.get('cumulative_return_series', []) ) diff --git a/investing_algorithm_framework/domain/exceptions.py b/investing_algorithm_framework/domain/exceptions.py index 065efbfd..c47f533c 100644 --- a/investing_algorithm_framework/domain/exceptions.py +++ b/investing_algorithm_framework/domain/exceptions.py @@ -110,3 +110,33 @@ def to_response(self): "status": "error", "message": self.error_message } + + +class PortfolioOutOfSyncError(OperationalException): + """ + Raised when the local portfolio state diverges from the broker's reported + state in a way the framework refuses to silently reconcile (typically a + negative delta β€” the broker reports *less* than the framework expected). + + Attributes: + market: Market identifier the mismatch was detected on. + local_unallocated: The framework's current unallocated balance. + broker_available: The amount the broker reports as available. + delta: ``broker_available - local_unallocated`` (negative on + shortfall, positive on unexpected deposit when withdrawals + were not allowed). + """ + + def __init__( + self, + message: str, + market: str = None, + local_unallocated: float = None, + broker_available: float = None, + delta: float = None, + ) -> None: + super().__init__(message) + self.market = market + self.local_unallocated = local_unallocated + self.broker_available = broker_available + self.delta = delta diff --git a/investing_algorithm_framework/domain/models/__init__.py b/investing_algorithm_framework/domain/models/__init__.py index 305ad4b4..0719c615 100644 --- a/investing_algorithm_framework/domain/models/__init__.py +++ b/investing_algorithm_framework/domain/models/__init__.py @@ -1,7 +1,8 @@ from .app_mode import AppMode from .market import MarketCredential from .order import OrderStatus, OrderSide, OrderType, Order -from .portfolio import PortfolioConfiguration, Portfolio, PortfolioSnapshot +from .portfolio import PortfolioConfiguration, Portfolio, PortfolioSnapshot, \ + SyncResult, ScheduledDeposit from .position import Position, PositionSnapshot, PositionSize from .strategy_profile import StrategyProfile from .time_frame import TimeFrame @@ -45,4 +46,6 @@ "StopLossRule", "TakeProfitRule", "TradingCost", + "SyncResult", + "ScheduledDeposit", ] diff --git a/investing_algorithm_framework/domain/models/data/data_source.py b/investing_algorithm_framework/domain/models/data/data_source.py index 5e27a3c4..dfa63d29 100644 --- a/investing_algorithm_framework/domain/models/data/data_source.py +++ b/investing_algorithm_framework/domain/models/data/data_source.py @@ -152,6 +152,9 @@ def from_csv( (e.g., "1d", "1h"). If None, data is fetched once and cached indefinitely. headers: Optional HTTP headers to send with the request. + Values are redacted ("***") in ``to_dict()`` + so secrets do not leak into diagnostic + payloads. pre_process: Optional callback to transform the raw CSV text before parsing. Receives a string, must return a string. @@ -215,6 +218,9 @@ def from_json( refresh_interval: How often to re-fetch the data (e.g., "1d", "1h"). headers: Optional HTTP headers to send with the request. + Values are redacted ("***") in ``to_dict()`` + so secrets do not leak into diagnostic + payloads. pre_process: Optional callback to transform the raw JSON text before parsing. Receives a string, must return a string. @@ -271,6 +277,9 @@ def from_parquet( refresh_interval: How often to re-fetch the data (e.g., "1d", "1h"). headers: Optional HTTP headers to send with the request. + Values are redacted ("***") in ``to_dict()`` + so secrets do not leak into diagnostic + payloads. post_process: Optional callback to transform the parsed DataFrame. diff --git a/investing_algorithm_framework/domain/models/order/order.py b/investing_algorithm_framework/domain/models/order/order.py index 63c50de4..a3ed5218 100644 --- a/investing_algorithm_framework/domain/models/order/order.py +++ b/investing_algorithm_framework/domain/models/order/order.py @@ -46,6 +46,8 @@ def __init__( slippage=None, id=None, metadata=None, + stop_price=None, + triggered_at=None, ): if target_symbol is None: raise OperationalException("Target symbol is not specified") @@ -92,6 +94,8 @@ def __init__( self.id = id self.cost = cost self.metadata = metadata if metadata is not None else {} + self.stop_price = stop_price + self.triggered_at = triggered_at def get_id(self): return self.id @@ -114,6 +118,25 @@ def get_price(self): def set_price(self, price): self.price = price + def get_stop_price(self): + return self.stop_price + + def set_stop_price(self, stop_price): + self.stop_price = stop_price + + def get_triggered_at(self): + return self.triggered_at + + def set_triggered_at(self, triggered_at): + self.triggered_at = triggered_at + + def is_stop_order(self): + return OrderType.STOP.equals(self.order_type) \ + or OrderType.STOP_LIMIT.equals(self.order_type) + + def is_triggered(self): + return self.triggered_at is not None + def get_order_fee_currency(self): return self.order_fee_currency @@ -266,6 +289,9 @@ def ensure_iso(value): "order_fee_rate": self.order_fee_rate, "order_fee": self.order_fee, "slippage": self.slippage, + "stop_price": self.stop_price, + "triggered_at": ensure_iso(self.triggered_at) + if self.triggered_at else None, "metadata": self.metadata if hasattr(self, 'metadata') else {}, } @@ -288,6 +314,10 @@ def from_dict(data: dict): if updated_at is not None: updated_at = _parse_dt(updated_at) + triggered_at = data.get("triggered_at", None) + if triggered_at is not None: + triggered_at = _parse_dt(triggered_at) + order = Order( id=data.get("id", None), external_id=data.get("external_id", None), @@ -309,6 +339,8 @@ def from_dict(data: dict): order_fee_rate=data.get("order_fee_rate", None), slippage=data.get("slippage", None), metadata=data.get("metadata", {}), + stop_price=data.get("stop_price", None), + triggered_at=triggered_at, ) return order diff --git a/investing_algorithm_framework/domain/models/order/order_type.py b/investing_algorithm_framework/domain/models/order/order_type.py index c0a38df9..df8e1a4b 100644 --- a/investing_algorithm_framework/domain/models/order/order_type.py +++ b/investing_algorithm_framework/domain/models/order/order_type.py @@ -4,6 +4,8 @@ class OrderType(Enum): LIMIT = 'LIMIT' MARKET = 'MARKET' + STOP = 'STOP' + STOP_LIMIT = 'STOP_LIMIT' @staticmethod def from_string(value: str): diff --git a/investing_algorithm_framework/domain/models/portfolio/__init__.py b/investing_algorithm_framework/domain/models/portfolio/__init__.py index 84aa8688..f9bc7339 100644 --- a/investing_algorithm_framework/domain/models/portfolio/__init__.py +++ b/investing_algorithm_framework/domain/models/portfolio/__init__.py @@ -1,9 +1,12 @@ from .portfolio import Portfolio from .portfolio_configuration import PortfolioConfiguration from .portfolio_snapshot import PortfolioSnapshot +from .sync import SyncResult, ScheduledDeposit __all__ = [ "PortfolioConfiguration", "Portfolio", "PortfolioSnapshot", + "SyncResult", + "ScheduledDeposit", ] diff --git a/investing_algorithm_framework/domain/models/portfolio/portfolio_configuration.py b/investing_algorithm_framework/domain/models/portfolio/portfolio_configuration.py index d794f8be..87233c8f 100644 --- a/investing_algorithm_framework/domain/models/portfolio/portfolio_configuration.py +++ b/investing_algorithm_framework/domain/models/portfolio/portfolio_configuration.py @@ -33,6 +33,7 @@ def __init__( initial_balance=None, fee_percentage=0.0, slippage_percentage=0.0, + deposit_schedule=None, ): self._market = market @@ -45,6 +46,7 @@ def __init__( self._initial_balance = initial_balance self._fee_percentage = fee_percentage or 0.0 self._slippage_percentage = slippage_percentage or 0.0 + self._deposit_schedule = list(deposit_schedule or []) if self.identifier is None: self._identifier = market.upper() @@ -91,6 +93,10 @@ def fee_percentage(self): def slippage_percentage(self): return self._slippage_percentage + @property + def deposit_schedule(self): + return list(self._deposit_schedule) + @property def has_initial_balance(self): return self._initial_balance is not None diff --git a/investing_algorithm_framework/domain/models/portfolio/sync.py b/investing_algorithm_framework/domain/models/portfolio/sync.py new file mode 100644 index 00000000..a6a6b40c --- /dev/null +++ b/investing_algorithm_framework/domain/models/portfolio/sync.py @@ -0,0 +1,111 @@ +"""Value objects describing portfolio reconciliation outcomes and scheduled +external cash flows used by ``Context.sync_portfolio``. +""" +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from typing import Optional + + +@dataclass(frozen=True) +class SyncResult: + """Outcome of a single :meth:`Context.sync_portfolio` call. + + Attributes: + market: Market the sync was performed on. + kind: One of ``"noop"``, ``"deposit"``, ``"withdrawal"``. + delta: ``broker_available - previous_unallocated``. + Positive = deposit detected, negative = shortfall absorbed. + broker_available: Trading-symbol balance the broker reports + (in backtest: ``previous_unallocated`` plus pending simulated + deposits since the last sync; in live: free balance minus + cash reserved for not-yet-acknowledged orders). + previous_unallocated: Local unallocated balance *before* the sync. + new_unallocated: Local unallocated balance *after* the sync. + within_tolerance: ``True`` when ``abs(delta)`` is below the + configured tolerance and the sync was therefore treated as a + noop. + reserved_for_pending_orders: Cash subtracted from the raw broker + balance to account for orders the framework knows about but + the exchange has not yet filled or cancelled. Always ``0`` in + backtest mode. + """ + + market: str + kind: str + delta: float + broker_available: float + previous_unallocated: float + new_unallocated: float + within_tolerance: bool = False + reserved_for_pending_orders: float = 0.0 + + +@dataclass(frozen=True) +class ScheduledDeposit: + """A simulated external cash flow applied to a backtested portfolio. + + Use one of two scheduling modes: + + * **Recurring** β€” set ``time_unit`` and ``interval`` (e.g. monthly + paycheck). The deposit fires every ``interval`` units of + ``time_unit`` starting from the backtest start date. + * **One-shot** β€” set ``on`` to a specific ``datetime``. The deposit + fires once at that timestamp. + + Attributes: + amount: Amount to credit to ``unallocated``. Negative values are + treated as withdrawals (drawing down the simulated broker + balance). + time_unit: Recurring cadence unit (DAY, HOUR, …). Mutually + exclusive with ``on``. + interval: Recurring cadence count (e.g. ``30`` with + ``TimeUnit.DAY`` = every 30 days). Mutually exclusive + with ``on``. + on: Wall-clock timestamp for a one-shot deposit. Mutually + exclusive with ``time_unit``/``interval``. + """ + + amount: float + # TimeUnit; typed loosely to avoid cycle + time_unit: Optional[object] = None + interval: Optional[int] = None + on: Optional[datetime] = None + fire_on_anchor: bool = False + """If ``True`` for a recurring deposit, the first firing happens at the + anchor moment (typically the backtest start) instead of at + ``anchor + interval``. Useful for "deposit immediately at start, then + every N days".""" + + def __post_init__(self) -> None: + recurring = self.time_unit is not None or self.interval is not None + if recurring and self.on is not None: + raise ValueError( + "ScheduledDeposit: pass either (time_unit, interval) for a " + "recurring deposit or 'on' for a one-shot deposit, not both." + ) + if not recurring and self.on is None: + raise ValueError( + "ScheduledDeposit: must specify either (time_unit, interval) " + "or 'on'." + ) + if recurring and (self.time_unit is None or self.interval is None): + raise ValueError( + "ScheduledDeposit: recurring deposits require both " + "time_unit and interval." + ) + if recurring and self.interval <= 0: + raise ValueError( + "ScheduledDeposit: interval must be positive." + ) + if self.on is not None and self.on.tzinfo is None: + raise ValueError( + "ScheduledDeposit: 'on' must be timezone-aware. Use " + "datetime(..., tzinfo=timezone.utc) or similar." + ) + if self.fire_on_anchor and not recurring: + raise ValueError( + "ScheduledDeposit: fire_on_anchor only applies to recurring " + "deposits." + ) diff --git a/investing_algorithm_framework/infrastructure/data_providers/base_url.py b/investing_algorithm_framework/infrastructure/data_providers/base_url.py index b2b675de..228e4b17 100644 --- a/investing_algorithm_framework/infrastructure/data_providers/base_url.py +++ b/investing_algorithm_framework/infrastructure/data_providers/base_url.py @@ -25,6 +25,27 @@ } +def url_cache_key(url, headers=None): + """Canonical cache key for a (url, headers) pair. + + Used in two places: + + * the in-memory provider dict on :class:`Context` (so two strategies + hitting the same URL with different credentials don't collide), and + * the on-disk cache filename hash inside + :class:`BaseURLDataProvider._get_cache_path` (same reason). + + Both call sites must agree on the exact serialization, otherwise an + in-memory hit could read the wrong on-disk file. Centralizing it + here prevents that drift. + """ + if not headers: + return url + + serialized = "&".join(f"{k}={v}" for k, v in sorted(headers.items())) + return f"{url}|headers:{serialized}" + + class BaseURLDataProvider(DataProvider): """ Abstract base class for data providers that fetch data from a @@ -357,9 +378,7 @@ def _get_cache_path(self): if storage_dir is None: storage_dir = os.path.join(os.getcwd(), ".data_cache") - cache_key = self._url - if self._headers: - cache_key = f"{cache_key}|headers:{sorted(self._headers.items())}" + cache_key = url_cache_key(self._url, self._headers) url_hash = hashlib.md5( cache_key.encode() diff --git a/investing_algorithm_framework/infrastructure/models/order/order.py b/investing_algorithm_framework/infrastructure/models/order/order.py index 3a88fafb..a7197d24 100644 --- a/investing_algorithm_framework/infrastructure/models/order/order.py +++ b/investing_algorithm_framework/infrastructure/models/order/order.py @@ -52,6 +52,8 @@ class SQLOrder(Order, SQLBaseModel, SQLAlchemyModelExtension): order_fee_currency = Column(String, default=None) order_fee_rate = Column(SqliteDecimal(), default=None) slippage = Column(SqliteDecimal(), default=None) + stop_price = Column(SqliteDecimal(), default=None) + triggered_at = Column(DateTime(timezone=True), default=None) sell_order_metadata_id = Column(Integer, ForeignKey('orders.id')) metadata_json = Column(Text, default=None) trade_allocations = relationship( @@ -101,6 +103,8 @@ def from_order(order): created_at=order.get_created_at(), updated_at=order.get_updated_at(), metadata=order.metadata, + stop_price=order.get_stop_price(), + triggered_at=order.get_triggered_at(), ) return sql_order diff --git a/investing_algorithm_framework/infrastructure/order_executors/ccxt_order_executor.py b/investing_algorithm_framework/infrastructure/order_executors/ccxt_order_executor.py index fd70155e..afca0bd8 100644 --- a/investing_algorithm_framework/infrastructure/order_executors/ccxt_order_executor.py +++ b/investing_algorithm_framework/infrastructure/order_executors/ccxt_order_executor.py @@ -88,6 +88,44 @@ def execute_order(self, portfolio, order, market_credential) -> Order: external_order = exchange.createMarketSellOrder( symbol, amount, ) + elif OrderType.STOP.equals(order_type) \ + or OrderType.STOP_LIMIT.equals(order_type): + stop_price = order.get_stop_price() + + if stop_price is None: + raise OperationalException( + f"Order type {order_type} requires a stop_price" + ) + + if not hasattr(exchange, "createOrder"): + raise OperationalException( + f"Exchange {market} does not support " + f"functionality createOrder needed for " + f"{order_type} orders" + ) + + # CCXT unified API: stop / stop-limit are passed as a + # generic order with the stopPrice param. STOP maps to + # the exchange's market-stop type; STOP_LIMIT to the + # stop-limit type. The exchange-specific mapping is + # handled by CCXT. + ccxt_type = ( + "stop_limit" + if OrderType.STOP_LIMIT.equals(order_type) + else "stop" + ) + ccxt_price = price if OrderType.STOP_LIMIT.equals( + order_type + ) else None + + external_order = exchange.createOrder( + symbol, + ccxt_type, + OrderSide.from_value(order_side).value.lower(), + amount, + ccxt_price, + {"stopPrice": stop_price}, + ) else: raise OperationalException( f"Order type {order_type} not supported " diff --git a/investing_algorithm_framework/infrastructure/repositories/repository.py b/investing_algorithm_framework/infrastructure/repositories/repository.py index e8552957..67fe1056 100644 --- a/investing_algorithm_framework/infrastructure/repositories/repository.py +++ b/investing_algorithm_framework/infrastructure/repositories/repository.py @@ -47,7 +47,8 @@ def create(self, data, save=True): def update(self, object_id, data): # List all datetime fields for your model datetime_fields = [ - "created_at", "updated_at", "closed_at", "opened_at" + "created_at", "updated_at", "closed_at", "opened_at", + "triggered_at" ] data = convert_datetime_fields(data, datetime_fields) with Session() as db: diff --git a/investing_algorithm_framework/infrastructure/services/backtesting/vector_backtest_service.py b/investing_algorithm_framework/infrastructure/services/backtesting/vector_backtest_service.py index 03381442..263608c0 100644 --- a/investing_algorithm_framework/infrastructure/services/backtesting/vector_backtest_service.py +++ b/investing_algorithm_framework/infrastructure/services/backtesting/vector_backtest_service.py @@ -257,6 +257,20 @@ def run( total_allocated = 0.0 # Track total allocated in static mode open_trades_value = {} # Track value of open trades per symbol + # Pre-compute scheduled external deposits (e.g. monthly paychecks) + # for the backtest window. Vector backtests are single-pass and + # have no Context, so we eagerly resolve the full schedule into a + # sorted (timestamp, amount) list and credit ``current_unallocated`` + # the first bar at-or-after each timestamp. Net effect for the + # strategy: the simulated broker balance grows on cadence, and the + # equity curve / metrics include the external cash flows just like + # the event backtest does after a sync_portfolio call. + deposit_events = self._resolve_deposit_schedule( + portfolio_configuration=portfolio_configuration, + backtest_date_range=backtest_date_range, + ) + deposit_event_idx = 0 + def _close_trade(sym, sym_data, price, date): """Helper to close an open trade for a symbol.""" nonlocal current_unallocated, total_realized_gains, \ @@ -537,6 +551,17 @@ def _partial_close(sym, sym_data, price, date, sell_pct): if current_date.tzinfo is None: current_date = current_date.replace(tzinfo=timezone.utc) + # Apply any scheduled external deposits whose timestamp has + # been reached by ``current_date``. Each event fires exactly + # once at the first bar at-or-after its scheduled time. + while ( + deposit_event_idx < len(deposit_events) + and deposit_events[deposit_event_idx][0] <= current_date + ): + _, deposit_amount = deposit_events[deposit_event_idx] + current_unallocated += deposit_amount + deposit_event_idx += 1 + # Process each symbol at this timestamp for symbol, data in symbol_data.items(): current_price = float(data['close'].iloc[i]) @@ -728,6 +753,12 @@ def _partial_close(sym, sym_data, price, date, sell_pct): unallocated = initial_amount total_net_gain = 0.0 open_trades = [] + # Replay deposit events for snapshot bookkeeping. Each snapshot + # gets ``cash_flow`` set to whatever external cash landed between + # the previous snapshot and this one β€” this is what enables + # TWR-aware return metrics (CAGR, monthly/yearly returns) to + # subtract external deposits before computing returns. + deposit_replay_idx = 0 # Create portfolio snapshots for ts in index: @@ -735,6 +766,18 @@ def _partial_close(sym, sym_data, price, date, sell_pct): interval_datetime = pd.Timestamp(ts).to_pydatetime() interval_datetime = interval_datetime.replace(tzinfo=timezone.utc) + # Apply any deposits that fired between the previous snapshot + # and this one, accumulating them into snapshot_cash_flow. + snapshot_cash_flow = 0.0 + while ( + deposit_replay_idx < len(deposit_events) + and deposit_events[deposit_replay_idx][0] <= interval_datetime + ): + _, deposit_amount = deposit_events[deposit_replay_idx] + unallocated += deposit_amount + snapshot_cash_flow += deposit_amount + deposit_replay_idx += 1 + for trade in trades: if trade.opened_at == interval_datetime: @@ -761,14 +804,15 @@ def _partial_close(sym, sym_data, price, date, sell_pct): allocated += open_trade.filled_amount * price # total_value = invested_value + unallocated - # total_net_gain = total_value - initial_amount + # total_net_gain = total_value - initial_amount - sum(cash_flow) snapshots.append( PortfolioSnapshot( portfolio_id=portfolio.identifier, created_at=interval_datetime, unallocated=unallocated, total_value=unallocated + allocated, - total_net_gain=total_net_gain + total_net_gain=total_net_gain, + cash_flow=snapshot_cash_flow, ) ) @@ -830,6 +874,34 @@ def _partial_close(sym, sym_data, price, date, sell_pct): ) return run + @staticmethod + def _resolve_deposit_schedule( + portfolio_configuration, + backtest_date_range, + ): + """Materialise a portfolio's deposit schedule for the backtest window. + + Returns a chronologically sorted list of ``(timestamp, amount)`` + tuples. Empty when the configuration has no schedule. + """ + schedule = list( + getattr(portfolio_configuration, "deposit_schedule", []) or [] + ) + if not schedule: + return [] + # Local import to avoid a circular import between domain and + # infrastructure layers. + from investing_algorithm_framework.services.portfolios \ + .broker_balance_tracker import BrokerBalanceTracker + tracker = BrokerBalanceTracker() + market = portfolio_configuration.market + tracker.set_schedule(market, schedule) + return tracker.project_total( + market=market, + start=backtest_date_range.start_date, + end=backtest_date_range.end_date, + ) + @staticmethod def _convert_recorded_values(raw_recorded): """ diff --git a/investing_algorithm_framework/services/__init__.py b/investing_algorithm_framework/services/__init__.py index 8582b842..adbd8955 100644 --- a/investing_algorithm_framework/services/__init__.py +++ b/investing_algorithm_framework/services/__init__.py @@ -7,7 +7,8 @@ OrderExecutorLookup from .portfolios import PortfolioService, BacktestPortfolioService, \ PortfolioConfigurationService, PortfolioSyncService, \ - PortfolioSnapshotService, PortfolioProviderLookup + PortfolioSnapshotService, PortfolioProviderLookup, \ + BrokerBalanceTracker from .positions import PositionService, PositionSnapshotService from .repository_service import RepositoryService from .trade_service import TradeService, TradeStopLossService, \ @@ -62,6 +63,7 @@ "OrderExecutorLookup", "BacktestTradeOrderEvaluator", "PortfolioProviderLookup", + "BrokerBalanceTracker", "TradeOrderEvaluator", "DefaultTradeOrderEvaluator", "get_risk_free_rate_us", diff --git a/investing_algorithm_framework/services/metrics/__init__.py b/investing_algorithm_framework/services/metrics/__init__.py index 377d62eb..cf0a6a43 100644 --- a/investing_algorithm_framework/services/metrics/__init__.py +++ b/investing_algorithm_framework/services/metrics/__init__.py @@ -1,16 +1,18 @@ from .volatility import get_annual_volatility from .sortino_ratio import get_sortino_ratio from .drawdown import get_drawdown_series, get_max_drawdown -from .equity_curve import get_equity_curve +from .equity_curve import get_equity_curve, get_twr_equity_curve from .price_efficiency import get_price_efficiency_ratio from .profit_factor import get_profit_factor, \ get_cumulative_profit_factor_series, get_rolling_profit_factor_series from .sharpe_ratio import get_sharpe_ratio, get_rolling_sharpe_ratio from .price_efficiency import get_price_efficiency_ratio -from .equity_curve import get_equity_curve +from .equity_curve import get_equity_curve, get_twr_equity_curve from .drawdown import get_drawdown_series, get_max_drawdown, \ get_max_drawdown_absolute, get_max_drawdown_duration, \ - get_max_daily_drawdown + get_max_daily_drawdown, \ + get_twr_drawdown_series, get_twr_max_drawdown, \ + get_twr_max_drawdown_duration from .cagr import get_cagr from .standard_deviation import get_standard_deviation_downside_returns, \ get_standard_deviation_returns diff --git a/investing_algorithm_framework/services/metrics/_returns_helper.py b/investing_algorithm_framework/services/metrics/_returns_helper.py new file mode 100644 index 00000000..f9463f7f --- /dev/null +++ b/investing_algorithm_framework/services/metrics/_returns_helper.py @@ -0,0 +1,70 @@ +"""Helpers for converting portfolio snapshots into time-weighted return +(TWR) series. + +The simple ``total_value.pct_change()`` is wrong as soon as a portfolio +absorbs external deposits or withdrawals via +:meth:`Context.sync_portfolio` β€” a $100 deposit looks like $100 of +trading P&L. These helpers subtract ``snapshot.cash_flow`` from the +period's ending value before computing the return, so external capital +no longer inflates the metric. + +Snapshots without a ``cash_flow`` attribute (or with it set to ``None`` +or ``0``) degrade gracefully to the classic ``pct_change`` behaviour. +""" +from __future__ import annotations + +from typing import Iterable + +import pandas as pd + + +def snapshots_to_dataframe(snapshots: Iterable) -> pd.DataFrame: + """Build a 3-column dataframe ``(created_at, total_value, cash_flow)`` + indexed by ``created_at``.""" + data = [ + (s.created_at, s.total_value, getattr(s, "cash_flow", 0) or 0) + for s in snapshots + ] + df = pd.DataFrame( + data, columns=["created_at", "total_value", "cash_flow"] + ) + df['created_at'] = pd.to_datetime(df['created_at']) + df = df.sort_values('created_at').drop_duplicates('created_at')\ + .set_index('created_at') + return df + + +def daily_twr_returns(snapshots: Iterable, ffill: bool = True) -> pd.Series: + """Return a daily TWR series indexed by date. + + For each day ``d``: + + r_d = (V_d - cash_flow_d) / V_{d-1} - 1 + + Where ``V_d`` is end-of-day total value and ``cash_flow_d`` is the + sum of external cash flows that landed during day ``d``. Days with a + zero ``V_{d-1}`` are dropped. + + Args: + snapshots: Iterable of portfolio snapshots. + ffill: When ``True`` (default), forward-fill end-of-day values + across calendar gaps (weekends, holidays). This matches the + historical ``pct_change`` behaviour used by the std/sharpe + metrics where days without a new snapshot register a 0% + return. Set to ``False`` to drop gap days entirely (used by + the volatility metric). + """ + df = snapshots_to_dataframe(snapshots) + if df.empty: + return pd.Series(dtype=float) + + daily_value = df['total_value'].resample('1D').last() + if ffill: + daily_value = daily_value.ffill() + else: + daily_value = daily_value.dropna() + daily_cf = df['cash_flow'].resample('1D').sum() + daily_cf = daily_cf.reindex(daily_value.index, fill_value=0) + prev_value = daily_value.shift(1) + returns = (daily_value - daily_cf) / prev_value - 1 + return returns.dropna() diff --git a/investing_algorithm_framework/services/metrics/cagr.py b/investing_algorithm_framework/services/metrics/cagr.py index 0becee4e..ba33e576 100644 --- a/investing_algorithm_framework/services/metrics/cagr.py +++ b/investing_algorithm_framework/services/metrics/cagr.py @@ -8,6 +8,12 @@ * Less than a year (e.g. 30 days) * Exactly a year (365 days) * More than a year (e.g. 500 days) + +When portfolio snapshots carry external ``cash_flow`` (deposits or +withdrawals absorbed via :meth:`Context.sync_portfolio`), CAGR is computed +using the **time-weighted return (TWR)** convention: per-period returns +subtract the period's cash flow before chaining, so external capital does +not inflate the reported growth rate. """ import pandas as pd @@ -17,22 +23,23 @@ def get_cagr(snapshots: List[PortfolioSnapshot]) -> float: """ - Calculate the Compound Annual Growth Rate (CAGR) of a backtest report. - CAGR is a useful metric to evaluate the performance of an investment - over a period of time, normalizing the return to a one-year basis. + Calculate the time-weighted Compound Annual Growth Rate (CAGR) of a + backtest report. + + The formula is the standard CAGR applied to a TWR-chained growth + factor: - The formula for CAGR is: - CAGR = (End Value / Start Value) ^ (1 / Number of Years) - 1 + period_return_t = (V_t - cash_flow_t) / V_{t-1} - 1 + growth = ∏ (1 + period_return_t) + CAGR = growth ^ (365 / num_days) - 1 - Where: - - End Value is the final value of the investment - - Start Value is the initial value of the investment - - Number of Years is the total time period in years - This function assumes that the snapshots in the report are ordered by - creation date and that the net size represents the value of the investment. + Where ``cash_flow_t`` is the net external deposit / withdrawal that + landed between snapshot ``t-1`` and snapshot ``t``. With no cash + flows the formula degenerates to the classic ``(end/start)^(1/yrs) - 1``. Args: - snapshots (list[Snapshot]): A list of snapshots + snapshots (list[Snapshot]): A list of snapshots ordered by + creation date. Returns: Float: The CAGR as a decimal. Returns 0.0 if not enough @@ -42,13 +49,15 @@ def get_cagr(snapshots: List[PortfolioSnapshot]) -> float: if len(snapshots) < 2: return 0.0 # Not enough data - # Convert snapshots to DataFrame - data = [(s.total_value, s.created_at) for s in snapshots] - df = pd.DataFrame(data, columns=["total_value", "created_at"]) + data = [ + (s.total_value, s.created_at, getattr(s, "cash_flow", 0) or 0) + for s in snapshots + ] + df = pd.DataFrame(data, columns=["total_value", "created_at", "cash_flow"]) df['created_at'] = pd.to_datetime(df['created_at']) - df = df.sort_values('created_at') + df = df.sort_values('created_at').reset_index(drop=True) + start_value = df.iloc[0]['total_value'] - end_value = df.iloc[-1]['total_value'] start_date = df.iloc[0]['created_at'] end_date = df.iloc[-1]['created_at'] num_days = (end_date - start_date).days @@ -56,5 +65,17 @@ def get_cagr(snapshots: List[PortfolioSnapshot]) -> float: if num_days == 0 or start_value == 0: return 0.0 - # Apply CAGR formula - return (end_value / start_value) ** (365 / num_days) - 1 + growth = 1.0 + for i in range(1, len(df)): + prev_v = df.iloc[i - 1]['total_value'] + curr_v = df.iloc[i]['total_value'] + cf = df.iloc[i]['cash_flow'] + if prev_v == 0: + continue + period_return = (curr_v - cf - prev_v) / prev_v + growth *= (1.0 + period_return) + + if growth <= 0: + return -1.0 + + return growth ** (365 / num_days) - 1 diff --git a/investing_algorithm_framework/services/metrics/drawdown.py b/investing_algorithm_framework/services/metrics/drawdown.py index 1ae173c9..d6db8e72 100644 --- a/investing_algorithm_framework/services/metrics/drawdown.py +++ b/investing_algorithm_framework/services/metrics/drawdown.py @@ -14,7 +14,7 @@ import pandas as pd from datetime import datetime from investing_algorithm_framework.domain import PortfolioSnapshot, Trade -from .equity_curve import get_equity_curve +from .equity_curve import get_equity_curve, get_twr_equity_curve def get_drawdown_series(snapshots: List[PortfolioSnapshot]) -> List[Tuple[float, datetime]]: @@ -221,3 +221,110 @@ def get_max_drawdown_absolute(snapshots: List[PortfolioSnapshot]) -> float: max_drawdown = max(max_drawdown, drawdown) return abs(max_drawdown) # Return as positive number (e.g., €10,000) + + +# --------------------------------------------------------------------------- +# TWR (alpha-only) drawdown variants +# --------------------------------------------------------------------------- +# +# The drawdown functions above operate on raw ``total_value`` β€” i.e. the +# absolute account-value path including external deposits/withdrawals. +# That's the curve users typically want to see ("how deep did my account +# go in dollars?"). +# +# These variants below operate on the TWR-growth curve so that a +# strategy that receives a $1,000 deposit during a drawdown does NOT +# have its drawdown artificially shortened/erased by the deposit. Use +# these when comparing risk profiles across portfolios that are funded +# differently. + + +def get_twr_drawdown_series( + snapshots: List[PortfolioSnapshot], +) -> List[Tuple[float, datetime]]: + """Drawdown series computed against the TWR-growth curve. + + Identical shape to :func:`get_drawdown_series` (negative percentages + where 0% means "at high-water mark"), but external cash flows do + not contaminate the high-water mark. + """ + equity_curve = get_twr_equity_curve(snapshots) + + drawdown_series: List[Tuple[float, datetime]] = [] + max_value = None + + for value, timestamp in equity_curve: + if value <= 0: + drawdown_series.append((0.0, timestamp)) + continue + if max_value is None or max_value <= 0: + max_value = value + max_value = max(max_value, value) + drawdown = (value - max_value) / max_value + drawdown_series.append((drawdown, timestamp)) + + return drawdown_series + + +def get_twr_max_drawdown(snapshots: List[PortfolioSnapshot]) -> float: + """Maximum drawdown of the TWR-growth curve, returned as a positive + fraction (e.g. ``0.18`` for an 18% peak-to-trough decline in alpha). + """ + equity_curve = get_twr_equity_curve(snapshots) + if not equity_curve: + return 0.0 + + peak = equity_curve[0][0] + if peak <= 0: + for equity, _ in equity_curve: + if equity > 0: + peak = equity + break + else: + return 0.0 + + max_drawdown_pct = 0.0 + for equity, _ in equity_curve: + if equity <= 0: + continue + if equity > peak: + peak = equity + if peak <= 0: + continue + drawdown_pct = (equity - peak) / peak + max_drawdown_pct = min(max_drawdown_pct, drawdown_pct) + + return abs(max_drawdown_pct) + + +def get_twr_max_drawdown_duration( + snapshots: List[PortfolioSnapshot], +) -> int: + """Longest stretch in calendar days where the TWR-growth curve was + below its high-water mark.""" + equity_curve = get_twr_equity_curve(snapshots) + if not equity_curve: + return 0 + + peak = equity_curve[0][0] + max_duration = 0 + drawdown_start = None + + for equity, timestamp in equity_curve: + if equity < peak: + if drawdown_start is None: + drawdown_start = timestamp + else: + if drawdown_start is not None: + elapsed = (timestamp - drawdown_start).days + max_duration = max(max_duration, elapsed) + drawdown_start = None + peak = equity + + if drawdown_start is not None and len(equity_curve) > 0: + last_timestamp = equity_curve[-1][1] + elapsed = (last_timestamp - drawdown_start).days + max_duration = max(max_duration, elapsed) + + return max_duration + diff --git a/investing_algorithm_framework/services/metrics/equity_curve.py b/investing_algorithm_framework/services/metrics/equity_curve.py index af8536f8..f4a68e67 100644 --- a/investing_algorithm_framework/services/metrics/equity_curve.py +++ b/investing_algorithm_framework/services/metrics/equity_curve.py @@ -25,3 +25,53 @@ def get_equity_curve( series.sort(key=lambda x: x[1]) return series + + +def get_twr_equity_curve( + snapshots: List[PortfolioSnapshot], base: float = 1.0 +) -> list[tuple[float, datetime]]: + """Equity curve scrubbed of external cash flows (TWR-growth series). + + Each step compounds the period's TWR return: + + r_t = (V_t - cash_flow_t) / V_{t-1} - 1 + equity_t = equity_{t-1} * (1 + r_t) + + Starting from ``base`` (default 1.0 β†’ curve is "growth of $1"). This + is the curve to use when comparing alpha across portfolios that + receive different external capital β€” depositing $1,000 doesn't make + the line jump, only trading P&L does. + + Snapshots without a ``cash_flow`` field fall back to the simple + ``V_t / V_{t-1}`` ratio. + + Args: + snapshots: Time-sorted (or unsorted; we sort) snapshots. + base: Starting value of the curve. ``1.0`` for growth-of-$1, + ``snapshots[0].total_value`` to anchor the first point on + the raw account value. + + Returns: + ``[(equity, timestamp), ...]`` matching the shape of + :func:`get_equity_curve`. + """ + sorted_snaps = sorted(snapshots, key=lambda s: s.created_at) + if not sorted_snaps: + return [] + + series: list[tuple[float, datetime]] = [] + equity = float(base) + series.append((equity, sorted_snaps[0].created_at)) + prev_value = float(sorted_snaps[0].total_value) + + for snap in sorted_snaps[1:]: + curr_value = float(snap.total_value) + cash_flow = float(getattr(snap, "cash_flow", 0) or 0) + if prev_value > 0: + ret = (curr_value - cash_flow) / prev_value - 1 + equity *= 1 + ret + # else: prev_value is zero/negative β†’ skip update, equity stays put + series.append((equity, snap.created_at)) + prev_value = curr_value + + return series diff --git a/investing_algorithm_framework/services/metrics/generate.py b/investing_algorithm_framework/services/metrics/generate.py index 8c0ece3a..81db0458 100644 --- a/investing_algorithm_framework/services/metrics/generate.py +++ b/investing_algorithm_framework/services/metrics/generate.py @@ -15,8 +15,10 @@ from .calmar_ratio import get_calmar_ratio from .drawdown import get_drawdown_series, get_max_drawdown, \ get_max_daily_drawdown, get_max_drawdown_absolute, \ - get_max_drawdown_duration -from .equity_curve import get_equity_curve + get_max_drawdown_duration, \ + get_twr_drawdown_series, get_twr_max_drawdown, \ + get_twr_max_drawdown_duration +from .equity_curve import get_equity_curve, get_twr_equity_curve from .exposure import get_exposure_ratio, get_cumulative_exposure, \ get_trades_per_year, get_trades_per_day, get_trades_per_week, \ get_trades_per_month @@ -545,6 +547,10 @@ def create_backtest_metrics( "max_drawdown_absolute", "max_daily_drawdown", "max_drawdown_duration", + "twr_equity_curve", + "twr_drawdown_series", + "twr_max_drawdown", + "twr_max_drawdown_duration", "trades_per_year", "trades_per_week", "trades_per_month", @@ -786,6 +792,10 @@ def safe_set(metric_name, func, *args, index=None): safe_set("max_drawdown_absolute", get_max_drawdown_absolute, backtest_run.portfolio_snapshots) safe_set("max_daily_drawdown", get_max_daily_drawdown, backtest_run.portfolio_snapshots) safe_set("max_drawdown_duration", get_max_drawdown_duration, backtest_run.portfolio_snapshots) + safe_set("twr_equity_curve", get_twr_equity_curve, backtest_run.portfolio_snapshots) + safe_set("twr_drawdown_series", get_twr_drawdown_series, backtest_run.portfolio_snapshots) + safe_set("twr_max_drawdown", get_twr_max_drawdown, backtest_run.portfolio_snapshots) + safe_set("twr_max_drawdown_duration", get_twr_max_drawdown_duration, backtest_run.portfolio_snapshots) safe_set("trades_per_year", get_trades_per_year, backtest_run.trades, backtest_run.backtest_start_date, backtest_run.backtest_end_date) safe_set("trades_per_week", get_trades_per_week, backtest_run.trades, backtest_run.backtest_start_date, backtest_run.backtest_end_date) safe_set("trades_per_month", get_trades_per_month, backtest_run.trades, backtest_run.backtest_start_date, backtest_run.backtest_end_date) diff --git a/investing_algorithm_framework/services/metrics/mean_daily_return.py b/investing_algorithm_framework/services/metrics/mean_daily_return.py index 55dcf30c..344f7098 100644 --- a/investing_algorithm_framework/services/metrics/mean_daily_return.py +++ b/investing_algorithm_framework/services/metrics/mean_daily_return.py @@ -27,8 +27,11 @@ def get_mean_daily_return(snapshots): return 0.0 # Not enough data # Create DataFrame from snapshots - data = [(s.created_at, s.total_value) for s in snapshots] - df = pd.DataFrame(data, columns=["created_at", "total_value"]) + data = [ + (s.created_at, s.total_value, getattr(s, "cash_flow", 0) or 0) + for s in snapshots + ] + df = pd.DataFrame(data, columns=["created_at", "total_value", "cash_flow"]) df['created_at'] = pd.to_datetime(df['created_at']) df = df.sort_values('created_at').drop_duplicates('created_at')\ .set_index('created_at') @@ -45,11 +48,20 @@ def get_mean_daily_return(snapshots): return (1 + cagr) ** (1 / 365) - 1 - # Resample to daily frequency using last value of the day - daily_df = df.resample('1D').last().dropna() - - # Calculate daily returns - daily_df['return'] = daily_df['total_value'].pct_change() + # Resample to daily frequency: end-of-day value, daily cash flow sum + daily_value = df['total_value'].resample('1D').last().dropna() + daily_cf = df['cash_flow'].resample('1D').sum() + daily_df = pd.DataFrame({ + 'total_value': daily_value, + 'cash_flow': daily_cf.reindex(daily_value.index, fill_value=0), + }) + + # TWR-adjusted daily return: + # r_d = (V_end - cash_flow_d) / V_prev_end - 1 + prev_value = daily_df['total_value'].shift(1) + daily_df['return'] = ( + (daily_df['total_value'] - daily_df['cash_flow']) / prev_value - 1 + ) daily_df = daily_df.dropna() if daily_df.empty: diff --git a/investing_algorithm_framework/services/metrics/returns.py b/investing_algorithm_framework/services/metrics/returns.py index d2dee5d3..87ec82fb 100644 --- a/investing_algorithm_framework/services/metrics/returns.py +++ b/investing_algorithm_framework/services/metrics/returns.py @@ -9,10 +9,16 @@ def get_monthly_returns(snapshots: List[PortfolioSnapshot]) -> List[Tuple[float, datetime]]: """ - Calculate the monthly returns from a list of portfolio snapshots. + Calculate the monthly time-weighted returns from a list of portfolio + snapshots. - Monthly return is calculated as the percentage change in portfolio value - from the end of one month to the end of the next month. + Each month's return subtracts external cash flow (deposits / + withdrawals absorbed via :meth:`Context.sync_portfolio`) from the + end-of-month value before comparing it to the previous month-end: + + r_month = (V_end - cash_flow_in_month) / V_prev_end - 1 + + With no cash flows this is identical to ``pct_change()``. Args: snapshots (List[PortfolioSnapshot]): List of portfolio snapshots. @@ -23,15 +29,26 @@ def get_monthly_returns(snapshots: List[PortfolioSnapshot]) -> List[Tuple[float, """ # Create DataFrame from snapshots - data = [(s.created_at, s.total_value) for s in snapshots] - df = pd.DataFrame(data, columns=["created_at", "total_value"]) + data = [ + (s.created_at, s.total_value, getattr(s, "cash_flow", 0) or 0) + for s in snapshots + ] + df = pd.DataFrame(data, columns=["created_at", "total_value", "cash_flow"]) df['created_at'] = pd.to_datetime(df['created_at']) df = df.sort_values('created_at').drop_duplicates('created_at')\ .set_index('created_at') - # Resample to monthly frequency using last value of the month - monthly_df = df.resample('ME').last().dropna() - monthly_df['return'] = monthly_df['total_value'].pct_change() + monthly_value = df['total_value'].resample('ME').last().dropna() + monthly_cf = df['cash_flow'].resample('ME').sum() + monthly_df = pd.DataFrame({ + 'total_value': monthly_value, + 'cash_flow': monthly_cf.reindex(monthly_value.index, fill_value=0), + }) + + prev_value = monthly_df['total_value'].shift(1) + monthly_df['return'] = ( + (monthly_df['total_value'] - monthly_df['cash_flow']) / prev_value - 1 + ) monthly_df = monthly_df.dropna() # Ensure returns are Python floats, not numpy floats @@ -43,10 +60,11 @@ def get_monthly_returns(snapshots: List[PortfolioSnapshot]) -> List[Tuple[float, def get_yearly_returns(snapshots: List[PortfolioSnapshot]) -> List[Tuple[float, date]]: """ - Calculate the yearly returns from a list of portfolio snapshots. + Calculate the yearly time-weighted returns from a list of portfolio + snapshots. - Yearly return is calculated as the percentage change in portfolio value - from the end of one year to the end of the next year. + Same TWR adjustment as :func:`get_monthly_returns`, resampled to + year-end. Args: snapshots (List[PortfolioSnapshot]): List of portfolio snapshots. @@ -57,8 +75,11 @@ def get_yearly_returns(snapshots: List[PortfolioSnapshot]) -> List[Tuple[float, """ # Create DataFrame from snapshots - data = [(s.created_at, s.total_value) for s in snapshots] - df = pd.DataFrame(data, columns=["created_at", "total_value"]) + data = [ + (s.created_at, s.total_value, getattr(s, "cash_flow", 0) or 0) + for s in snapshots + ] + df = pd.DataFrame(data, columns=["created_at", "total_value", "cash_flow"]) df['created_at'] = pd.to_datetime(df['created_at']) df = df.sort_values('created_at').drop_duplicates('created_at')\ .set_index('created_at') @@ -67,9 +88,17 @@ def get_yearly_returns(snapshots: List[PortfolioSnapshot]) -> List[Tuple[float, if df.index.tz is not None: df.index = df.index.tz_localize(None) - # Resample to yearly frequency using last value of the year - yearly_df = df.resample('YE').last().dropna() - yearly_df['return'] = yearly_df['total_value'].pct_change() + yearly_value = df['total_value'].resample('YE').last().dropna() + yearly_cf = df['cash_flow'].resample('YE').sum() + yearly_df = pd.DataFrame({ + 'total_value': yearly_value, + 'cash_flow': yearly_cf.reindex(yearly_value.index, fill_value=0), + }) + + prev_value = yearly_df['total_value'].shift(1) + yearly_df['return'] = ( + (yearly_df['total_value'] - yearly_df['cash_flow']) / prev_value - 1 + ) yearly_df = yearly_df.dropna() # Yearly returns with date objects only representing the year diff --git a/investing_algorithm_framework/services/metrics/sharpe_ratio.py b/investing_algorithm_framework/services/metrics/sharpe_ratio.py index 5f40b093..5c366bf4 100644 --- a/investing_algorithm_framework/services/metrics/sharpe_ratio.py +++ b/investing_algorithm_framework/services/metrics/sharpe_ratio.py @@ -106,11 +106,10 @@ def get_rolling_sharpe_ratio( df = df.sort_values('created_at').drop_duplicates('created_at')\ .set_index('created_at') - # Resample to daily frequency using last value of the day - daily_df = df.resample('1D').last().dropna() - - # Returns as percentage change - returns_s = daily_df['total_value'].pct_change().dropna() + # TWR-adjusted daily returns so external deposits/withdrawals do not + # contaminate the rolling Sharpe. + from ._returns_helper import daily_twr_returns + returns_s = daily_twr_returns(snapshots) # Rolling Annualised Sharpe rolling = returns_s.rolling(window=365, min_periods=30) diff --git a/investing_algorithm_framework/services/metrics/standard_deviation.py b/investing_algorithm_framework/services/metrics/standard_deviation.py index 8da130b1..249c7425 100644 --- a/investing_algorithm_framework/services/metrics/standard_deviation.py +++ b/investing_algorithm_framework/services/metrics/standard_deviation.py @@ -1,6 +1,28 @@ import numpy as np import pandas as pd +from ._returns_helper import daily_twr_returns + + +def _twr_period_returns(snapshots) -> pd.Series: + """Per-snapshot TWR returns (subtracting per-snapshot cash_flow). + + Used by metrics that operate on the raw snapshot series rather than + a daily resample. + """ + data = [ + (s.created_at, s.total_value, getattr(s, "cash_flow", 0) or 0) + for s in snapshots + ] + df = pd.DataFrame( + data, columns=["created_at", "total_value", "cash_flow"] + ) + df['created_at'] = pd.to_datetime(df['created_at']) + df = df.sort_values('created_at').drop_duplicates('created_at')\ + .set_index('created_at') + prev_v = df['total_value'].shift(1) + return ((df['total_value'] - df['cash_flow']) / prev_v - 1).dropna() + def get_standard_deviation_downside_returns(snapshots): """ @@ -17,21 +39,13 @@ def get_standard_deviation_downside_returns(snapshots): if len(snapshots) < 2: return 0.0 # Not enough data - # Create DataFrame of net_size over time - data = [(s.total_value, s.created_at) for s in snapshots] - df = pd.DataFrame(data, columns=["total_value", "created_at"]) - df['created_at'] = pd.to_datetime(df['created_at']) - df = df.sort_values('created_at').drop_duplicates('created_at').copy() - - # Compute percentage returns - df['return'] = df['total_value'].pct_change() - df = df.dropna() + df_returns = _twr_period_returns(snapshots) - if df.empty: + if df_returns.empty: return 0.0 # Filter downside returns - downside_returns = df['return'][df['return'] < 0] + downside_returns = df_returns[df_returns < 0] if downside_returns.empty: return 0.0 @@ -64,21 +78,7 @@ def get_standard_deviation_returns(snapshots): if len(snapshots) < 2: return 0.0 # Not enough data - # Create DataFrame of net_size over time - data = [(s.total_value, s.created_at) for s in snapshots] - df = pd.DataFrame(data, columns=["total_value", "created_at"]) - df['created_at'] = pd.to_datetime(df['created_at']) - df = df.sort_values('created_at').drop_duplicates('created_at').copy() - - # Compute percentage returns - df['return'] = df['total_value'].pct_change() - df = df.dropna() - - if df.empty: - return 0.0 - - # Filter downside returns - df_returns = df['return'] + df_returns = _twr_period_returns(snapshots) if df_returns.empty: return 0.0 @@ -108,25 +108,12 @@ def get_daily_returns_std(snapshots): if len(snapshots) < 2: return 0.0 # Not enough data - # Create DataFrame from snapshots - data = [(s.created_at, s.total_value) for s in snapshots] - df = pd.DataFrame(data, columns=["created_at", "total_value"]) - df["created_at"] = pd.to_datetime(df["created_at"]) - df = df.drop_duplicates("created_at").set_index("created_at") - df = df.sort_index() - # Resample to daily frequency (end of day) - daily_df = df.resample("D").last().ffill().dropna() + returns = daily_twr_returns(snapshots) - # Calculate daily returns - daily_df["return"] = daily_df["total_value"].pct_change().dropna() - - if daily_df["return"].empty: - return 0.0 - - if len(daily_df["return"].dropna()) < 2: + if returns.empty or len(returns) < 2: return 0.0 - return daily_df["return"].std() + return returns.std() def get_downside_std_of_daily_returns(snapshots): @@ -143,21 +130,10 @@ def get_downside_std_of_daily_returns(snapshots): if len(snapshots) < 2: return 0.0 # Not enough data - # Create DataFrame from snapshots - data = [(s.created_at, s.total_value) for s in snapshots] - df = pd.DataFrame(data, columns=["created_at", "total_value"]) - df["created_at"] = pd.to_datetime(df["created_at"]) - df = df.drop_duplicates("created_at").set_index("created_at") - df = df.sort_index() - - # Resample to daily frequency (end of day) - daily_df = df.resample("D").last().dropna() - - # Calculate daily returns - daily_df["return"] = daily_df["total_value"].pct_change().dropna() + returns = daily_twr_returns(snapshots) # Filter only negative returns for downside deviation - negative_returns = daily_df["return"][daily_df["return"] < 0] + negative_returns = returns[returns < 0] if negative_returns.empty: return 0.0 diff --git a/investing_algorithm_framework/services/metrics/value_at_risk.py b/investing_algorithm_framework/services/metrics/value_at_risk.py index f73cf70c..ae64942d 100644 --- a/investing_algorithm_framework/services/metrics/value_at_risk.py +++ b/investing_algorithm_framework/services/metrics/value_at_risk.py @@ -63,17 +63,23 @@ def get_conditional_value_at_risk( def _get_monthly_return_series( snapshots: List[PortfolioSnapshot], ) -> "pd.Series | None": - """Helper: build a monthly return series from snapshots.""" + """Helper: build a monthly TWR return series from snapshots.""" if not snapshots or len(snapshots) < 2: return None - data = [(s.created_at, s.total_value) for s in snapshots] - df = pd.DataFrame(data, columns=["created_at", "total_value"]) + data = [ + (s.created_at, s.total_value, getattr(s, "cash_flow", 0) or 0) + for s in snapshots + ] + df = pd.DataFrame(data, columns=["created_at", "total_value", "cash_flow"]) df['created_at'] = pd.to_datetime(df['created_at']) df = df.sort_values('created_at').drop_duplicates('created_at')\ .set_index('created_at') - monthly_df = df.resample('ME').last().dropna() - monthly_df['return'] = monthly_df['total_value'].pct_change() - monthly_df = monthly_df.dropna() - if monthly_df.empty: + monthly_value = df['total_value'].resample('ME').last().dropna() + monthly_cf = df['cash_flow'].resample('ME').sum() + monthly_cf = monthly_cf.reindex(monthly_value.index, fill_value=0) + prev_value = monthly_value.shift(1) + monthly_returns = (monthly_value - monthly_cf) / prev_value - 1 + monthly_returns = monthly_returns.dropna() + if monthly_returns.empty: return None - return monthly_df['return'] + return monthly_returns diff --git a/investing_algorithm_framework/services/metrics/volatility.py b/investing_algorithm_framework/services/metrics/volatility.py index 68547a65..b9eb567b 100644 --- a/investing_algorithm_framework/services/metrics/volatility.py +++ b/investing_algorithm_framework/services/metrics/volatility.py @@ -71,48 +71,29 @@ def get_annual_volatility( if len(snapshots) < 2: return 0.0 - # Build DataFrame from snapshots - records = [ - (snapshot.total_value, snapshot.created_at) for snapshot in snapshots - ] - df = pd.DataFrame(records, columns=['total_value', 'created_at']) - df['created_at'] = pd.to_datetime(df['created_at']) - df = df.set_index('created_at').sort_index().drop_duplicates() - - # Resample to daily frequency, taking the last value of each day - df_daily = df.resample('D').last() - df_daily = df_daily.dropna() - - if len(df_daily) < 2: - return 0.0 - - # Filter out non-positive values before calculating log returns - # Log returns are only valid for positive portfolio values - df_daily = df_daily[df_daily['total_value'] > 0] + # Use TWR-adjusted daily returns so external deposits/withdrawals + # do not contaminate the volatility estimate. + from ._returns_helper import daily_twr_returns + returns = daily_twr_returns(snapshots, ffill=False) - if len(df_daily) < 2: + if returns.empty or len(returns) < 2: return 0.0 - # Calculate log returns on daily data - # Add a small epsilon to avoid log(0) and replace inf/nan values - price_ratios = df_daily['total_value'] / df_daily['total_value'].shift(1) - - # Filter out invalid price ratios (zero, negative, or NaN) - df_daily['log_return'] = np.log(price_ratios) - - # Replace inf and -inf with NaN, then drop them - df_daily['log_return'] = df_daily['log_return'].replace([np.inf, -np.inf], np.nan) - df_daily = df_daily.dropna() - - if len(df_daily) < 2: + # Convert simple returns to log returns; only valid for ratios > 0 + # (i.e. (1 + r) > 0). Negative full wipeouts are filtered out. + valid = returns[returns > -1] + if len(valid) < 2: return 0.0 + log_returns = np.log(1 + valid) + log_returns = log_returns.replace([np.inf, -np.inf], np.nan).dropna() - # Calculate daily volatility (standard deviation of daily returns) - daily_volatility = df_daily['log_return'].std() + if len(log_returns) < 2: + return 0.0 + daily_vol = log_returns.std() # Handle edge case where std might be NaN - if pd.isna(daily_volatility): + if pd.isna(daily_vol): return 0.0 # Annualize using trading days per year - return daily_volatility * np.sqrt(trading_days_per_year) + return daily_vol * np.sqrt(trading_days_per_year) diff --git a/investing_algorithm_framework/services/order_service/order_backtest_service.py b/investing_algorithm_framework/services/order_service/order_backtest_service.py index 7fa48b52..6d8c46dd 100644 --- a/investing_algorithm_framework/services/order_service/order_backtest_service.py +++ b/investing_algorithm_framework/services/order_service/order_backtest_service.py @@ -158,6 +158,37 @@ def has_executed(self, order, ohlcv_data_frame: pl.DataFrame): if OrderType.MARKET.equals(order.get_order_type()): return not ohlcv_data_after_order.is_empty() + # Stop / Stop-Limit orders: trigger first, then evaluate fill + is_stop = OrderType.STOP.equals(order.get_order_type()) + is_stop_limit = OrderType.STOP_LIMIT.equals(order.get_order_type()) + + if is_stop or is_stop_limit: + stop_price = order.get_stop_price() + + if stop_price is None: + return False + + if OrderSide.SELL.equals(order_side): + triggered = ( + ohlcv_data_after_order['Low'] <= stop_price + ).any() + elif OrderSide.BUY.equals(order_side): + triggered = ( + ohlcv_data_after_order['High'] >= stop_price + ).any() + else: + return False + + if not triggered: + return False + + # STOP becomes a market order on trigger + if is_stop: + return True + + # STOP_LIMIT: after trigger, fall through to limit fill check + # against the configured limit price. + # Check if the order execution conditions are met if OrderSide.BUY.equals(order_side): # Check if the low price drops below or equals the order price diff --git a/investing_algorithm_framework/services/order_service/order_service.py b/investing_algorithm_framework/services/order_service/order_service.py index 7d852429..1331e8cf 100644 --- a/investing_algorithm_framework/services/order_service/order_service.py +++ b/investing_algorithm_framework/services/order_service/order_service.py @@ -151,6 +151,14 @@ def create(self, data, execute=True, validate=True, sync=True) -> Order: if "take_profits" in data: del data["take_profits"] + # For plain STOP orders the user only provides a stop_price. + # Default the order price to the stop_price so downstream + # accounting / trade allocation has a numeric reference. The + # actual fill price is set at trigger time by the evaluator. + if OrderType.STOP.equals(data.get("order_type")) \ + and data.get("price") is None: + data["price"] = data.get("stop_price") + if validate: self.validate_order(data, portfolio) @@ -306,6 +314,10 @@ def validate_order(self, order_data, portfolio): self.validate_limit_order(order_data, portfolio) elif OrderType.MARKET.equals(order_data["order_type"]): self.validate_market_order(order_data, portfolio) + elif OrderType.STOP.equals(order_data["order_type"]): + self.validate_stop_order(order_data, portfolio) + elif OrderType.STOP_LIMIT.equals(order_data["order_type"]): + self.validate_stop_limit_order(order_data, portfolio) else: raise OperationalException( f"Order type {order_data['order_type']} is not supported" @@ -462,6 +474,63 @@ def validate_market_order(self, order_data, portfolio): f"{portfolio.trading_symbol} of the portfolio" ) + def validate_stop_order(self, order_data, portfolio): + """ + Validate a stop order. A stop order requires a stop_price and, + once triggered, executes as a market order. + + Cash / position validation reuses the market-order rules: the + stop_price is used as the price reference for buy-side cash + reservation since no limit price exists. + """ + stop_price = order_data.get("stop_price") + + if stop_price is None or stop_price <= 0: + raise OperationalException( + "Stop orders require a positive stop_price" + ) + + # Reuse market-order validation for amount / cash checks, + # using the stop_price as the price reference. + validation_data = dict(order_data) + validation_data["price"] = stop_price + self.validate_market_order(validation_data, portfolio) + + def validate_stop_limit_order(self, order_data, portfolio): + """ + Validate a stop-limit order. Requires both a stop_price (trigger) + and a price (limit price after trigger). + """ + stop_price = order_data.get("stop_price") + limit_price = order_data.get("price") + + if stop_price is None or stop_price <= 0: + raise OperationalException( + "Stop-limit orders require a positive stop_price" + ) + + if limit_price is None or limit_price <= 0: + raise OperationalException( + "Stop-limit orders require a positive limit price" + ) + + # Validate stop / limit price consistency relative to side + if OrderSide.BUY.equals(order_data["order_side"]): + if limit_price < stop_price: + raise OperationalException( + "For BUY stop-limit orders, the limit price must be " + "greater than or equal to the stop price" + ) + else: + if limit_price > stop_price: + raise OperationalException( + "For SELL stop-limit orders, the limit price must be " + "less than or equal to the stop price" + ) + + # Reuse limit-order amount / cash validation + self.validate_limit_order(order_data, portfolio) + def check_pending_orders(self, portfolio=None): """ Function to check if diff --git a/investing_algorithm_framework/services/portfolios/__init__.py b/investing_algorithm_framework/services/portfolios/__init__.py index f1535353..68a3f078 100644 --- a/investing_algorithm_framework/services/portfolios/__init__.py +++ b/investing_algorithm_framework/services/portfolios/__init__.py @@ -1,4 +1,5 @@ from .backtest_portfolio_service import BacktestPortfolioService +from .broker_balance_tracker import BrokerBalanceTracker from .portfolio_configuration_service import PortfolioConfigurationService from .portfolio_service import PortfolioService from .portfolio_snapshot_service import PortfolioSnapshotService @@ -12,5 +13,6 @@ "PortfolioService", "PortfolioSnapshotService", "BacktestPortfolioService", - "PortfolioProviderLookup" + "PortfolioProviderLookup", + "BrokerBalanceTracker", ] diff --git a/investing_algorithm_framework/services/portfolios/broker_balance_tracker.py b/investing_algorithm_framework/services/portfolios/broker_balance_tracker.py new file mode 100644 index 00000000..65d028ae --- /dev/null +++ b/investing_algorithm_framework/services/portfolios/broker_balance_tracker.py @@ -0,0 +1,330 @@ +"""Broker balance tracker β€” public reconciliation surface for ``Context``. + +This service is the *single source of truth* for "how much cash does the +broker actually have for me?" and is used by :meth:`Context.sync_portfolio`. + +Design +------ + +The tracker is intentionally tiny and engine-agnostic: + +* In **live mode** the framework does not populate the tracker; callers + query a registered :class:`PortfolioProvider` directly. The tracker only + exposes the configuration knobs (``auto_sync`` flag) for the live + runtime to consult. +* In **backtest mode** there is no broker, so the engine simulates one. + The event-loop and vector-backtest engines call + :meth:`fire_due_deposits` at every tick to convert any + :class:`ScheduledDeposit` whose cadence has elapsed into "pending" + external balance. ``Context.sync_portfolio`` then drains that pending + balance into the local portfolio's ``unallocated`` field β€” *exactly* the + same code path a strategy would hit in live mode when a paycheck lands + on the exchange. + +The result is a symmetric contract: the same ``context.sync_portfolio()`` +call inside a strategy works unchanged across event-backtest, vector-backtest +(via the per-bar deposit application) and live trading. +""" +from __future__ import annotations + +import logging +from dataclasses import dataclass, field +from datetime import datetime, timedelta +from typing import Dict, List, Optional, Tuple + +from investing_algorithm_framework.domain import ( + OperationalException, + ScheduledDeposit, + TimeUnit, +) + +logger = logging.getLogger(__name__) + + +def _interval_delta(time_unit: TimeUnit, interval: int) -> timedelta: + if time_unit == TimeUnit.SECOND: + return timedelta(seconds=interval) + if time_unit == TimeUnit.MINUTE: + return timedelta(minutes=interval) + if time_unit == TimeUnit.HOUR: + return timedelta(hours=interval) + if time_unit == TimeUnit.DAY: + return timedelta(days=interval) + if time_unit == TimeUnit.WEEK: + return timedelta(weeks=interval) + if time_unit == TimeUnit.MONTH: + # Calendar months are not a fixed timedelta; approximate as 30 days. + # ScheduledDeposit cadence is intentionally coarse β€” users who need + # exact calendar months should pass one-shot ``on=`` deposits. + logger.warning( + "ScheduledDeposit with TimeUnit.MONTH is approximated as " + "30 days/month (~12.17 firings/year). For exact calendar " + "months, use one-shot ScheduledDeposit(on=...) entries." + ) + return timedelta(days=30 * interval) + raise OperationalException( + f"Unsupported TimeUnit for ScheduledDeposit: {time_unit}" + ) + + +_AUTO_SYNC_ERROR_MODES = ("raise", "warn", "halt") + + +@dataclass +class _MarketState: + schedule: List[ScheduledDeposit] = field(default_factory=list) + auto_sync: bool = False + auto_sync_error_mode: str = "raise" + pending: float = 0.0 + # Net cash flow absorbed by sync_portfolio since the last snapshot + # drain. Used by the snapshot service to populate ``cash_flow`` so + # TWR-style metrics can subtract external deposits / withdrawals. + cash_flow_since_snapshot: float = 0.0 + # Lifetime sum of absorbed cash flow (deposits βˆ’ withdrawals). + total_cash_flow: float = 0.0 + # last-fired timestamp per recurring deposit index + last_fired: Dict[int, datetime] = field(default_factory=dict) + # set of one-shot deposit indices that already fired + one_shot_fired: set = field(default_factory=set) + # set of recurring deposit indices that have fired their anchor-day + # firing (only relevant when ScheduledDeposit.fire_on_anchor=True). + anchor_fired: set = field(default_factory=set) + + +class BrokerBalanceTracker: + """In-memory store of deposit schedules and pending external balance. + + A single instance lives on the :class:`App` (and is injected into + :class:`Context` via the dependency container). Keys are the **uppercased + market identifier** (matching :class:`PortfolioConfiguration.market`). + Lookups are case-insensitive β€” ``"binance"`` and ``"BINANCE"`` resolve + to the same state. + """ + + def __init__(self) -> None: + self._markets: Dict[str, _MarketState] = {} + + @staticmethod + def _key(market: str) -> str: + if market is None: + raise OperationalException( + "BrokerBalanceTracker: market identifier cannot be None." + ) + return str(market).upper() + + def _get(self, market: str) -> _MarketState: + key = self._key(market) + if key not in self._markets: + self._markets[key] = _MarketState() + return self._markets[key] + + def has_market(self, market: str) -> bool: + return self._key(market) in self._markets + + def set_schedule( + self, market: str, schedule: List[ScheduledDeposit] + ) -> None: + """Replace the deposit schedule for a market. + + ``pending`` and ``cash_flow_since_snapshot`` are preserved (they + represent already-credited cash, not part of the schedule itself). + Per-deposit firing history is cleared so the new schedule is + evaluated from scratch on the next ``fire_due_deposits`` call. + """ + if schedule is None: + schedule = [] + for entry in schedule: + if not isinstance(entry, ScheduledDeposit): + raise OperationalException( + "BrokerBalanceTracker.set_schedule: every entry must be " + f"a ScheduledDeposit, got {type(entry).__name__}." + ) + state = self._get(market) + state.schedule = list(schedule) + state.last_fired.clear() + state.one_shot_fired.clear() + state.anchor_fired.clear() + + def add_schedule_entry( + self, market: str, deposit: ScheduledDeposit + ) -> None: + """Append a single deposit to a market's schedule. + + Existing fired-state for prior entries is preserved. + """ + if not isinstance(deposit, ScheduledDeposit): + raise OperationalException( + "BrokerBalanceTracker.add_schedule_entry: deposit must be a " + f"ScheduledDeposit, got {type(deposit).__name__}." + ) + self._get(market).schedule.append(deposit) + + def set_auto_sync(self, market: str, enabled: bool) -> None: + self._get(market).auto_sync = enabled + + def is_auto_sync(self, market: str) -> bool: + return self._get(market).auto_sync + + def set_auto_sync_error_mode(self, market: str, mode: str) -> None: + """How auto-sync handles failures. + + * ``"raise"`` (default) β€” any sync error propagates and stops the + event loop. Loud, recommended for development. + * ``"warn"`` β€” log the error and continue with stale local state. + Recommended for live trading where transient broker glitches + should not crash the bot. + * ``"halt"`` β€” log the error and halt only auto-sync for that + market (the rest of the loop continues; manual + ``context.sync_portfolio`` calls still work). + """ + if mode not in _AUTO_SYNC_ERROR_MODES: + raise OperationalException( + f"auto_sync_error_mode must be one of " + f"{_AUTO_SYNC_ERROR_MODES}, got {mode!r}." + ) + self._get(market).auto_sync_error_mode = mode + + def get_auto_sync_error_mode(self, market: str) -> str: + return self._get(market).auto_sync_error_mode + + def get_schedule(self, market: str) -> List[ScheduledDeposit]: + return list(self._get(market).schedule) + + def fire_due_deposits( + self, + market: str, + current_datetime: datetime, + backtest_start: Optional[datetime] = None, + ) -> float: + """Advance the simulated broker clock to ``current_datetime``. + + Adds any deposits whose cadence has elapsed since the last fire to + the market's pending balance and returns the **incremental** amount + added during this call (useful for snapshot bookkeeping). + + Args: + market: Market identifier. + current_datetime: Simulated wall-clock time. + backtest_start: Anchor for recurring deposits' first firing + time. Defaults to ``current_datetime`` on the first call. + """ + state = self._get(market) + added = 0.0 + for idx, deposit in enumerate(state.schedule): + if deposit.on is not None: + if idx in state.one_shot_fired: + continue + if deposit.on <= current_datetime: + state.pending += deposit.amount + state.one_shot_fired.add(idx) + added += deposit.amount + logger.info( + "Scheduled one-shot deposit fired on market %s: " + "%+.4f at %s", + market, deposit.amount, current_datetime, + ) + continue + + # Recurring + delta = _interval_delta(deposit.time_unit, deposit.interval) + anchor = state.last_fired.get(idx) + if anchor is None: + anchor = backtest_start or current_datetime + state.last_fired[idx] = anchor + if deposit.fire_on_anchor and idx not in state.anchor_fired: + state.pending += deposit.amount + state.anchor_fired.add(idx) + added += deposit.amount + logger.info( + "Scheduled recurring deposit fired (anchor) on " + "market %s: %+.4f at %s", + market, deposit.amount, anchor, + ) + # Don't fire at the anchor itself (unless fire_on_anchor); + # next firing is at anchor + delta. + continue + + while anchor + delta <= current_datetime: + anchor = anchor + delta + state.pending += deposit.amount + added += deposit.amount + logger.info( + "Scheduled recurring deposit fired on market %s: " + "%+.4f at %s", + market, deposit.amount, anchor, + ) + state.last_fired[idx] = anchor + return added + + def peek_pending(self, market: str) -> float: + """Return the pending external balance without draining it.""" + return self._get(market).pending + + def consume_pending(self, market: str) -> float: + """Atomically drain and return the pending external balance.""" + state = self._get(market) + amount = state.pending + state.pending = 0.0 + return amount + + def record_cash_flow(self, market: str, amount: float) -> None: + """Record an absorbed external cash flow (deposit or withdrawal). + + Called by :meth:`Context.sync_portfolio` after a non-noop sync. The + snapshot service drains this counter via :meth:`drain_cash_flow` + each time it takes a snapshot so the snapshot's ``cash_flow`` + field reflects the deposits/withdrawals that landed *between* the + previous snapshot and this one. This is what enables TWR-aware + return metrics. + """ + if amount == 0: + return + state = self._get(market) + state.cash_flow_since_snapshot += float(amount) + state.total_cash_flow += float(amount) + + def drain_cash_flow(self, market: str) -> float: + """Atomically drain and return ``cash_flow_since_snapshot``.""" + state = self._get(market) + amount = state.cash_flow_since_snapshot + state.cash_flow_since_snapshot = 0.0 + return amount + + def total_cash_flow(self, market: str) -> float: + """Lifetime sum of absorbed cash flow on ``market``.""" + return self._get(market).total_cash_flow + + def markets(self) -> List[str]: + return list(self._markets.keys()) + + def reset(self) -> None: + """Wipe all state. Used when an app is reset between backtests.""" + self._markets.clear() + + def project_total( + self, + market: str, + start: datetime, + end: datetime, + ) -> List[Tuple[datetime, float]]: + """Compute the (timestamp, amount) deposits that would fire between + ``start`` (exclusive) and ``end`` (inclusive) without mutating state. + + Used by the vector backtest, which is single-pass and applies + deposits to its cash track up-front rather than tick-by-tick. + """ + events: List[Tuple[datetime, float]] = [] + for deposit in self._get(market).schedule: + if deposit.on is not None: + if start < deposit.on <= end: + events.append((deposit.on, deposit.amount)) + continue + delta = _interval_delta(deposit.time_unit, deposit.interval) + if deposit.fire_on_anchor and start <= end: + events.append((start, deposit.amount)) + t = start + delta + while t <= end: + events.append((t, deposit.amount)) + t = t + delta + events.sort(key=lambda x: x[0]) + return events diff --git a/investing_algorithm_framework/services/trade_order_evaluator/backtest_trade_oder_evaluator.py b/investing_algorithm_framework/services/trade_order_evaluator/backtest_trade_oder_evaluator.py index 0c9dd302..50ed943c 100644 --- a/investing_algorithm_framework/services/trade_order_evaluator/backtest_trade_oder_evaluator.py +++ b/investing_algorithm_framework/services/trade_order_evaluator/backtest_trade_oder_evaluator.py @@ -132,7 +132,64 @@ def _check_has_executed(self, order, ohlcv_df): ) return - # Limit orders: check if OHLCV data triggers a fill + # Stop / Stop-Limit orders: first check whether the trigger + # condition is met. Once triggered, a STOP becomes a market + # order (fill at trigger price) and a STOP_LIMIT becomes a + # limit order at the configured limit price. + is_stop = OrderType.STOP.equals(order.order_type) + is_stop_limit = OrderType.STOP_LIMIT.equals(order.order_type) + + if (is_stop or is_stop_limit) and not order.is_triggered(): + stop_price = order.get_stop_price() + + if stop_price is None: + return + + # SELL stop triggers when price drops to or below stop_price; + # BUY stop triggers when price rises to or above stop_price. + if OrderSide.SELL.equals(order_side): + trigger_candles = ohlcv_data_after_order.filter( + pl.col('Low') <= stop_price + ) + elif OrderSide.BUY.equals(order_side): + trigger_candles = ohlcv_data_after_order.filter( + pl.col('High') >= stop_price + ) + else: + return + + if trigger_candles.is_empty(): + return + + triggered_candle = trigger_candles.head(1) + order.set_triggered_at(triggered_candle["Datetime"][0]) + + if is_stop: + # STOP becomes a market order β€” fill at the stop price + # using the triggering candle's volume. + volume = ( + triggered_candle["Volume"][0] + if "Volume" in triggered_candle.columns + else None + ) + self._apply_fill( + order, stop_price, order_side, volume, + is_market_order=True + ) + return + + # STOP_LIMIT: continue and fall through to limit-fill logic + # using the configured limit price (`order.price`). Restrict + # the fill search to candles at or after the trigger. + ohlcv_data_after_order = ohlcv_data_after_order.filter( + pl.col('Datetime') >= order.get_triggered_at() + ) + + if ohlcv_data_after_order.is_empty(): + return + + # Limit orders (including triggered STOP_LIMIT): + # check if OHLCV data triggers a fill if OrderSide.BUY.equals(order_side): fill_candles = ohlcv_data_after_order.filter( pl.col('Low') <= order_price @@ -233,6 +290,11 @@ def _apply_fill( 'order_fee': accumulated_fee, } + # Persist trigger timestamp for stop / stop-limit orders so + # subsequent evaluations don't re-trigger. + if order.is_triggered(): + update_data['triggered_at'] = order.get_triggered_at() + # Market order portfolio reconciliation if is_market_order and new_remaining <= 0: estimated_price = order.estimated_price diff --git a/pyproject.toml b/pyproject.toml index c3e55b36..999cdae6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "investing-algorithm-framework" -version = "v8.7.3" +version = "v8.8.0" description = "A framework for creating trading bots" authors = ["MDUYN"] readme = "README.md" diff --git a/tests/cli/test_initialize.py b/tests/cli/test_initialize.py index be9770ca..42e5553e 100644 --- a/tests/cli/test_initialize.py +++ b/tests/cli/test_initialize.py @@ -232,4 +232,3 @@ def test_init_command_idempotent(self): self.assertTrue( os.path.exists(self._output_path("app.py")) ) - diff --git a/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231201_20231202/metrics.json b/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231201_20231202/metrics.json index 4fbaecd7..24e8e02a 100644 --- a/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231201_20231202/metrics.json +++ b/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231201_20231202/metrics.json @@ -60,6 +60,28 @@ "max_drawdown_absolute": 0.0, "max_daily_drawdown": 0.0, "max_drawdown_duration": 0, + "twr_equity_curve": [ + [ + 1.0, + "2023-12-01T00:00:00+00:00" + ], + [ + 1.0, + "2023-12-02T00:00:00+00:00" + ] + ], + "twr_drawdown_series": [ + [ + 0.0, + "2023-12-01T00:00:00+00:00" + ], + [ + 0.0, + "2023-12-02T00:00:00+00:00" + ] + ], + "twr_max_drawdown": 0.0, + "twr_max_drawdown_duration": 0, "trades_per_year": 0.0, "trades_per_week": 0.0, "trades_per_month": 0.0, diff --git a/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231201_20231202/run.json b/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231201_20231202/run.json index 7686a8c8..e3bb9a5f 100644 --- a/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231201_20231202/run.json +++ b/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231201_20231202/run.json @@ -1 +1 @@ -{"backtest_start_date": "2023-12-01 00:00:00", "backtest_date_range_name": null, "backtest_end_date": "2023-12-02 00:00:00", "trading_symbol": "EUR", "initial_unallocated": 1000.0, "number_of_runs": 1441, "portfolio_snapshots": [{"metadata": "MetaData()", "portfolio_id": "1", "trading_symbol": "EUR", "pending_value": 0.0, "unallocated": 1000.0, "total_net_gain": 0.0, "total_revenue": 0.0, "total_cost": 0.0, "cash_flow": 0.0, "net_size": 1000.0, "created_at": "2023-12-01T00:00:00+00:00", "total_value": 1000.0}, {"metadata": "MetaData()", "portfolio_id": "1", "trading_symbol": "EUR", "pending_value": 0.0, "unallocated": 1000.0, "total_net_gain": 0.0, "total_revenue": 0.0, "total_cost": 0.0, "cash_flow": 0.0, "net_size": 1000.0, "created_at": "2023-12-02T00:00:00+00:00", "total_value": 1000.0}], "trades": [], "orders": [], "positions": [{"symbol": "EUR", "amount": 1000.0, "cost": 1000.0, "portfolio_id": 1}], "created_at": "2026-04-26 14:39:23", "symbols": [], "number_of_days": 0, "number_of_trades": 0, "number_of_trades_closed": 0, "number_of_trades_open": 0, "number_of_orders": 0, "number_of_positions": 0, "metadata": {}, "signals": {}, "signal_events": [], "recorded_values": {}} \ No newline at end of file +{"backtest_start_date": "2023-12-01 00:00:00", "backtest_date_range_name": null, "backtest_end_date": "2023-12-02 00:00:00", "trading_symbol": "EUR", "initial_unallocated": 1000.0, "number_of_runs": 1441, "portfolio_snapshots": [{"metadata": "MetaData()", "portfolio_id": "1", "trading_symbol": "EUR", "pending_value": 0.0, "unallocated": 1000.0, "total_net_gain": 0.0, "total_revenue": 0.0, "total_cost": 0.0, "cash_flow": 0.0, "net_size": 1000.0, "created_at": "2023-12-01T00:00:00+00:00", "total_value": 1000.0}, {"metadata": "MetaData()", "portfolio_id": "1", "trading_symbol": "EUR", "pending_value": 0.0, "unallocated": 1000.0, "total_net_gain": 0.0, "total_revenue": 0.0, "total_cost": 0.0, "cash_flow": 0.0, "net_size": 1000.0, "created_at": "2023-12-02T00:00:00+00:00", "total_value": 1000.0}], "trades": [], "orders": [], "positions": [{"symbol": "EUR", "amount": 1000.0, "cost": 1000.0, "portfolio_id": 1}], "created_at": "2026-05-09 14:52:33", "symbols": [], "number_of_days": 1, "number_of_trades": 0, "number_of_trades_closed": 0, "number_of_trades_open": 0, "number_of_orders": 0, "number_of_positions": 1, "metadata": {}, "signals": {}, "signal_events": [], "recorded_values": {}} \ No newline at end of file diff --git a/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231202_20231203/metrics.json b/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231202_20231203/metrics.json index 6be6b73a..45ff5621 100644 --- a/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231202_20231203/metrics.json +++ b/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231202_20231203/metrics.json @@ -60,6 +60,28 @@ "max_drawdown_absolute": 0.0, "max_daily_drawdown": 0.0, "max_drawdown_duration": 0, + "twr_equity_curve": [ + [ + 1.0, + "2023-12-02T00:00:00+00:00" + ], + [ + 1.0, + "2023-12-03T00:00:00+00:00" + ] + ], + "twr_drawdown_series": [ + [ + 0.0, + "2023-12-02T00:00:00+00:00" + ], + [ + 0.0, + "2023-12-03T00:00:00+00:00" + ] + ], + "twr_max_drawdown": 0.0, + "twr_max_drawdown_duration": 0, "trades_per_year": 0.0, "trades_per_week": 0.0, "trades_per_month": 0.0, diff --git a/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231202_20231203/run.json b/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231202_20231203/run.json index d2baec0c..c9d4739e 100644 --- a/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231202_20231203/run.json +++ b/tests/resources/backtest_reports_for_testing/test_algorithm_backtest/runs/backtest_EUR_20231202_20231203/run.json @@ -1 +1 @@ -{"backtest_start_date": "2023-12-02 00:00:00", "backtest_date_range_name": null, "backtest_end_date": "2023-12-03 00:00:00", "trading_symbol": "EUR", "initial_unallocated": 1000.0, "number_of_runs": 1441, "portfolio_snapshots": [{"metadata": "MetaData()", "portfolio_id": "1", "trading_symbol": "EUR", "pending_value": 0.0, "unallocated": 1000.0, "total_net_gain": 0.0, "total_revenue": 0.0, "total_cost": 0.0, "cash_flow": 0.0, "net_size": 1000.0, "created_at": "2023-12-02T00:00:00+00:00", "total_value": 1000.0}, {"metadata": "MetaData()", "portfolio_id": "1", "trading_symbol": "EUR", "pending_value": 0.0, "unallocated": 1000.0, "total_net_gain": 0.0, "total_revenue": 0.0, "total_cost": 0.0, "cash_flow": 0.0, "net_size": 1000.0, "created_at": "2023-12-03T00:00:00+00:00", "total_value": 1000.0}], "trades": [], "orders": [], "positions": [{"symbol": "EUR", "amount": 1000.0, "cost": 1000.0, "portfolio_id": 1}], "created_at": "2026-04-26 14:38:58", "symbols": [], "number_of_days": 0, "number_of_trades": 0, "number_of_trades_closed": 0, "number_of_trades_open": 0, "number_of_orders": 0, "number_of_positions": 0, "metadata": {}, "signals": {}, "signal_events": [], "recorded_values": {}} \ No newline at end of file +{"backtest_start_date": "2023-12-02 00:00:00", "backtest_date_range_name": null, "backtest_end_date": "2023-12-03 00:00:00", "trading_symbol": "EUR", "initial_unallocated": 1000.0, "number_of_runs": 1441, "portfolio_snapshots": [{"metadata": "MetaData()", "portfolio_id": "1", "trading_symbol": "EUR", "pending_value": 0.0, "unallocated": 1000.0, "total_net_gain": 0.0, "total_revenue": 0.0, "total_cost": 0.0, "cash_flow": 0.0, "net_size": 1000.0, "created_at": "2023-12-02T00:00:00+00:00", "total_value": 1000.0}, {"metadata": "MetaData()", "portfolio_id": "1", "trading_symbol": "EUR", "pending_value": 0.0, "unallocated": 1000.0, "total_net_gain": 0.0, "total_revenue": 0.0, "total_cost": 0.0, "cash_flow": 0.0, "net_size": 1000.0, "created_at": "2023-12-03T00:00:00+00:00", "total_value": 1000.0}], "trades": [], "orders": [], "positions": [{"symbol": "EUR", "amount": 1000.0, "cost": 1000.0, "portfolio_id": 1}], "created_at": "2026-05-09 14:52:10", "symbols": [], "number_of_days": 1, "number_of_trades": 0, "number_of_trades_closed": 0, "number_of_trades_open": 0, "number_of_orders": 0, "number_of_positions": 1, "metadata": {}, "signals": {}, "signal_events": [], "recorded_values": {}} \ No newline at end of file diff --git a/tests/scenarios/test_readme_example.py b/tests/scenarios/test_readme_example.py index d59908ef..a71a4a90 100644 --- a/tests/scenarios/test_readme_example.py +++ b/tests/scenarios/test_readme_example.py @@ -8,8 +8,18 @@ import os import re import unittest +from datetime import datetime, timezone +from pathlib import Path from unittest import TestCase +from investing_algorithm_framework import ( + BacktestDateRange, + DATA_DIRECTORY, + RESOURCE_DIRECTORY, + SnapshotInterval, + create_app, +) + def extract_python_code_blocks_from_readme(readme_path: str) -> list[str]: """ @@ -56,6 +66,19 @@ def extract_main_example_from_readme(readme_path: str) -> str: ) +def prepare_readme_example_for_test(main_example: str) -> str: + """Adapt the README strategy to the compact offline test fixture set.""" + return ( + main_example + .replace('symbols = ["BTC", "ETH"]', 'symbols = ["BTC", "DOT"]') + .replace( + 'identifier="ETH_ohlcv", symbol="ETH/EUR"', + 'identifier="DOT_ohlcv", symbol="DOT/EUR"', + ) + .replace('symbol="ETH"', 'symbol="DOT"') + ) + + class TestReadmeExample(TestCase): """ Test class to verify the README example implementation works correctly. @@ -71,6 +94,9 @@ def setUp(self): os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'README.md' ) + self.resource_directory = str( + Path(__file__).parent.parent / 'resources' + ) def test_readme_code_can_be_extracted(self): """README.md exists and contains extractable Python code blocks.""" @@ -135,6 +161,45 @@ def test_readme_strategy_class_can_be_loaded(self): # Verify stop losses are defined self.assertGreater(len(cls.stop_losses), 0) + def test_readme_strategy_runs_fast_vector_backtest_with_offline_data(self): + """The README strategy runs using only compact offline test data.""" + main_example = extract_main_example_from_readme(self.readme_path) + class_code = prepare_readme_example_for_test(main_example) + + namespace = {} + exec(class_code, namespace) + strategy_class = namespace['RSIEMACrossoverStrategy'] + + app = create_app( + name="ReadmeExampleTest", + config={ + RESOURCE_DIRECTORY: self.resource_directory, + DATA_DIRECTORY: "test_data/ohlcv", + }, + ) + app.add_market( + market="BITVAVO", trading_symbol="EUR", initial_balance=1000 + ) + + strategy = strategy_class(algorithm_id="readme-example-test") + backtest = app.run_vector_backtest( + initial_amount=1000, + backtest_date_range=BacktestDateRange( + start_date=datetime(2023, 9, 1, tzinfo=timezone.utc), + end_date=datetime(2023, 11, 15, tzinfo=timezone.utc), + ), + strategy=strategy, + snapshot_interval=SnapshotInterval.DAILY, + risk_free_rate=0.027, + trading_symbol="EUR", + market="BITVAVO", + use_checkpoints=False, + ) + + self.assertIsNotNone(backtest) + self.assertEqual(len(backtest.get_all_backtest_runs()), 1) + self.assertEqual(len(backtest.get_all_backtest_metrics()), 1) + if __name__ == "__main__": unittest.main() diff --git a/tests/scenarios/vectorized_backtests/test_deposit_schedule.py b/tests/scenarios/vectorized_backtests/test_deposit_schedule.py new file mode 100644 index 00000000..549c5e20 --- /dev/null +++ b/tests/scenarios/vectorized_backtests/test_deposit_schedule.py @@ -0,0 +1,61 @@ +"""Unit test for VectorBacktestService._resolve_deposit_schedule.""" +from datetime import datetime, timezone +from unittest import TestCase + +from investing_algorithm_framework import ( + BacktestDateRange, + PortfolioConfiguration, + ScheduledDeposit, + TimeUnit, +) +from investing_algorithm_framework.infrastructure.services.backtesting \ + .vector_backtest_service import VectorBacktestService + + +def _utc(year, month, day): + return datetime(year, month, day, tzinfo=timezone.utc) + + +class TestVectorBacktestDepositSchedule(TestCase): + + def test_no_schedule_returns_empty(self): + config = PortfolioConfiguration( + market="BITVAVO", trading_symbol="EUR", initial_balance=1000 + ) + events = VectorBacktestService._resolve_deposit_schedule( + portfolio_configuration=config, + backtest_date_range=BacktestDateRange( + start_date=_utc(2024, 1, 1), + end_date=_utc(2024, 6, 1), + ), + ) + self.assertEqual([], events) + + def test_recurring_and_one_shot_combined(self): + config = PortfolioConfiguration( + market="BITVAVO", + trading_symbol="EUR", + initial_balance=1000, + deposit_schedule=[ + ScheduledDeposit( + amount=100.0, time_unit=TimeUnit.DAY, interval=30 + ), + ScheduledDeposit(amount=500.0, on=_utc(2024, 2, 15)), + ], + ) + events = VectorBacktestService._resolve_deposit_schedule( + portfolio_configuration=config, + backtest_date_range=BacktestDateRange( + start_date=_utc(2024, 1, 1), + end_date=_utc(2024, 4, 1), + ), + ) + self.assertEqual( + [ + (_utc(2024, 1, 31), 100.0), + (_utc(2024, 2, 15), 500.0), + (_utc(2024, 3, 1), 100.0), + (_utc(2024, 3, 31), 100.0), + ], + events, + ) diff --git a/tests/services/metrics/test_cagr_with_cashflow.py b/tests/services/metrics/test_cagr_with_cashflow.py new file mode 100644 index 00000000..b5a2760b --- /dev/null +++ b/tests/services/metrics/test_cagr_with_cashflow.py @@ -0,0 +1,173 @@ +"""Tests that TWR-aware metrics neutralize external deposits. + +The contract: a portfolio that earns 10%/yr and receives no deposits +should report the same CAGR (and similar Sharpe/std) as an otherwise +identical portfolio that received external deposits along the way β€” +provided the snapshots correctly populate ``cash_flow``. +""" +import unittest +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework.services.metrics import ( + get_cagr, + get_daily_returns_std, + get_sharpe_ratio, +) + + +class MockSnapshot: + def __init__(self, total_value, created_at, cash_flow=0.0): + self.total_value = total_value + self.created_at = created_at + self.cash_flow = cash_flow + + +def _organic_growth_series(start_value, daily_rate, days, start_date): + """Build a snapshot list with pure organic growth, no deposits.""" + snapshots = [] + value = start_value + for i in range(days + 1): + snapshots.append( + MockSnapshot( + total_value=value, + created_at=start_date + timedelta(days=i), + cash_flow=0.0, + ) + ) + value *= 1 + daily_rate + return snapshots + + +def _organic_with_deposits( + start_value, daily_rate, days, start_date, deposits +): + """Same daily growth as ``_organic_growth_series`` but with extra + deposits injected on specific days. Each deposit increases the + end-of-day value AND records ``cash_flow`` for that day.""" + deposit_lookup = dict(deposits) # day_index -> amount + snapshots = [] + value = start_value + snapshots.append( + MockSnapshot( + total_value=value, created_at=start_date, cash_flow=0.0 + ) + ) + for i in range(1, days + 1): + # Organic growth on yesterday's value + value *= 1 + daily_rate + cash_flow = float(deposit_lookup.get(i, 0.0)) + # Deposit lands at end of day, after market action + value += cash_flow + snapshots.append( + MockSnapshot( + total_value=value, + created_at=start_date + timedelta(days=i), + cash_flow=cash_flow, + ) + ) + return snapshots + + +class TestCagrWithCashFlow(unittest.TestCase): + + def setUp(self): + self.start = datetime(2024, 1, 1, tzinfo=timezone.utc) + # ~10% annual organic growth + self.daily_rate = (1 + 0.10) ** (1 / 365) - 1 + self.days = 365 + + def test_cagr_matches_no_deposit_equivalent(self): + organic = _organic_growth_series( + 10000, self.daily_rate, self.days, self.start + ) + # Inject 4 quarterly $1,000 deposits β€” these should NOT inflate CAGR + with_deposits = _organic_with_deposits( + 10000, + self.daily_rate, + self.days, + self.start, + deposits=[(90, 1000), (180, 1000), (270, 1000), (360, 1000)], + ) + + cagr_organic = get_cagr(organic) + cagr_with_dep = get_cagr(with_deposits) + + # Both should be ~10% β€” within 0.5pp + self.assertAlmostEqual(cagr_organic, 0.10, places=2) + self.assertAlmostEqual(cagr_with_dep, cagr_organic, places=2) + + def test_naive_cagr_would_be_inflated_without_twr(self): + """Sanity check: if we ignored cash_flow, raw end/start ratio + would falsely report a much higher CAGR. This protects the test + intent: the TWR fix is what makes the parity above hold.""" + with_deposits = _organic_with_deposits( + 10000, + self.daily_rate, + self.days, + self.start, + deposits=[(90, 1000), (180, 1000), (270, 1000), (360, 1000)], + ) + start_v = with_deposits[0].total_value + end_v = with_deposits[-1].total_value + naive_cagr = (end_v / start_v) ** (1 / 1) - 1 + # Raw ratio inflates well above 10% because $4k deposits + # masquerade as P&L + self.assertGreater(naive_cagr, 0.40) + + def test_daily_returns_std_unaffected_by_deposits(self): + organic = _organic_growth_series( + 10000, self.daily_rate, self.days, self.start + ) + with_deposits = _organic_with_deposits( + 10000, + self.daily_rate, + self.days, + self.start, + deposits=[(90, 1000), (180, 1000), (270, 1000), (360, 1000)], + ) + std_organic = get_daily_returns_std(organic) + std_with = get_daily_returns_std(with_deposits) + # Constant-growth series β€” std is essentially 0 in both cases. + self.assertAlmostEqual(std_organic, std_with, places=4) + + def test_sharpe_unaffected_by_deposits(self): + organic = _organic_growth_series( + 10000, self.daily_rate, self.days, self.start + ) + with_deposits = _organic_with_deposits( + 10000, + self.daily_rate, + self.days, + self.start, + deposits=[(90, 1000), (180, 1000), (270, 1000), (360, 1000)], + ) + # Sharpe of a constant-growth series is undefined (std=0 -> nan). + # Use varying growth instead. + import random + rng = random.Random(42) + organic2 = [] + with_dep2 = [] + v1 = v2 = 10000 + deposits = {90: 1000, 180: 1000, 270: 1000, 360: 1000} + for i in range(self.days + 1): + r = self.daily_rate + rng.gauss(0, 0.005) + if i > 0: + v1 *= 1 + r + v2 *= 1 + r + cf = float(deposits.get(i, 0.0)) + v2 += cf + else: + cf = 0.0 + d = self.start + timedelta(days=i) + organic2.append(MockSnapshot(v1, d, 0.0)) + with_dep2.append(MockSnapshot(v2, d, cf)) + + sh_organic = get_sharpe_ratio(organic2, risk_free_rate=0.02) + sh_with = get_sharpe_ratio(with_dep2, risk_free_rate=0.02) + # Should match within rounding β€” same return path, just $4k extra + # capital on the side + self.assertAlmostEqual(sh_organic, sh_with, places=2) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/services/metrics/test_twr_drawdown.py b/tests/services/metrics/test_twr_drawdown.py new file mode 100644 index 00000000..ceaa2448 --- /dev/null +++ b/tests/services/metrics/test_twr_drawdown.py @@ -0,0 +1,137 @@ +"""Tests for the TWR (alpha-only) variants of equity curve and drawdown.""" +import unittest +from datetime import datetime, timedelta, timezone + +from investing_algorithm_framework.services.metrics import ( + get_drawdown_series, + get_equity_curve, + get_max_drawdown, + get_twr_drawdown_series, + get_twr_equity_curve, + get_twr_max_drawdown, + get_twr_max_drawdown_duration, +) + + +class MockSnapshot: + def __init__(self, total_value, created_at, cash_flow=0.0): + self.total_value = total_value + self.created_at = created_at + self.cash_flow = cash_flow + + +class TestTwrEquityCurve(unittest.TestCase): + + def setUp(self): + self.start = datetime(2024, 1, 1, tzinfo=timezone.utc) + + def _seq(self, points): + return [ + MockSnapshot(v, self.start + timedelta(days=i), cf) + for i, (v, cf) in enumerate(points) + ] + + def test_no_cash_flow_matches_raw_growth(self): + snaps = self._seq([(100, 0), (110, 0), (121, 0)]) + twr = get_twr_equity_curve(snaps, base=100) + # 10% per step β†’ 100, 110, 121 + values = [v for v, _ in twr] + self.assertAlmostEqual(values[0], 100.0) + self.assertAlmostEqual(values[1], 110.0) + self.assertAlmostEqual(values[2], 121.0) + + def test_deposit_does_not_inflate_twr_curve(self): + # Day 0: 100. Day 1: 110 (+10% organic). Day 2: 1121 (= 110*1.1 + # + 1000 deposit). Raw curve jumps to 1121; TWR curve stays at + # 121 (= 100 * 1.1 * 1.1). + snaps = self._seq([(100, 0), (110, 0), (1121, 1000)]) + twr = get_twr_equity_curve(snaps, base=100) + values = [v for v, _ in twr] + self.assertAlmostEqual(values[2], 121.0, places=4) + + # Raw curve does jump + raw = get_equity_curve(snaps) + self.assertAlmostEqual(raw[2][0], 1121.0) + + def test_growth_of_one_default_base(self): + snaps = self._seq([(100, 0), (110, 0), (121, 0)]) + twr = get_twr_equity_curve(snaps) # base=1.0 + values = [v for v, _ in twr] + self.assertAlmostEqual(values[0], 1.0) + self.assertAlmostEqual(values[2], 1.21) + + def test_empty(self): + self.assertEqual([], get_twr_equity_curve([])) + + +class TestTwrDrawdown(unittest.TestCase): + + def setUp(self): + self.start = datetime(2024, 1, 1, tzinfo=timezone.utc) + + def _seq(self, points): + return [ + MockSnapshot(v, self.start + timedelta(days=i), cf) + for i, (v, cf) in enumerate(points) + ] + + def test_deposit_during_drawdown_does_not_mask_it(self): + # Organic path: 100 β†’ 80 (βˆ’20% dd) β†’ 90 β†’ 100 (recovered) + # Add a +50 deposit on day 1 so raw values are + # 100, 130, 140, 150 β€” raw drawdown == 0. + # TWR drawdown should still surface the βˆ’20% dip on day 1. + organic_returns = [None, -0.20, +0.125, +0.111111111] + snaps = [] + v_raw = 100.0 + v_twr = 100.0 + deposits = {1: 50.0} + for i, r in enumerate(organic_returns): + cf = float(deposits.get(i, 0.0)) + if r is not None: + v_twr *= 1 + r + v_raw = v_raw * (1 + r) + cf + snaps.append( + MockSnapshot(v_raw, self.start + timedelta(days=i), cf) + ) + + # Raw drawdown is masked by the +50 deposit that immediately + # followed the dip + raw_dd = get_max_drawdown(snaps) + # With the deposit, raw equity is 100, 130, 140, 150 β†’ no DD + self.assertAlmostEqual(raw_dd, 0.0, places=4) + + # TWR drawdown is unaffected by the deposit + twr_dd = get_twr_max_drawdown(snaps) + self.assertAlmostEqual(twr_dd, 0.20, places=2) + + def test_twr_drawdown_series_matches_no_deposit_case(self): + # Without deposits, TWR series and raw series should yield the + # same drawdown percentages. + snaps = [ + MockSnapshot(v, self.start + timedelta(days=i), 0.0) + for i, v in enumerate([100, 90, 80, 95, 100]) + ] + raw = [d for d, _ in get_drawdown_series(snaps)] + twr = [d for d, _ in get_twr_drawdown_series(snaps)] + for r, t in zip(raw, twr): + self.assertAlmostEqual(r, t, places=6) + + def test_twr_max_drawdown_duration(self): + # Drawdown lasts 3 days then recovers + snaps = [ + MockSnapshot(100, self.start + timedelta(days=0), 0.0), + MockSnapshot(90, self.start + timedelta(days=1), 0.0), + MockSnapshot(85, self.start + timedelta(days=2), 0.0), + MockSnapshot(95, self.start + timedelta(days=3), 0.0), + MockSnapshot(110, self.start + timedelta(days=4), 0.0), + ] + self.assertEqual(3, get_twr_max_drawdown_duration(snaps)) + + def test_empty(self): + self.assertEqual([], get_twr_drawdown_series([])) + self.assertEqual(0.0, get_twr_max_drawdown([])) + self.assertEqual(0, get_twr_max_drawdown_duration([])) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/services/test_broker_balance_tracker.py b/tests/services/test_broker_balance_tracker.py new file mode 100644 index 00000000..c792dad9 --- /dev/null +++ b/tests/services/test_broker_balance_tracker.py @@ -0,0 +1,167 @@ +"""Unit tests for BrokerBalanceTracker.""" +from datetime import datetime, timedelta, timezone +from unittest import TestCase + +from investing_algorithm_framework import ScheduledDeposit, TimeUnit +from investing_algorithm_framework.services.portfolios \ + .broker_balance_tracker import BrokerBalanceTracker + + +def _utc(year, month, day, hour=0): + return datetime(year, month, day, hour, tzinfo=timezone.utc) + + +class TestBrokerBalanceTracker(TestCase): + + def test_pending_starts_at_zero(self): + tracker = BrokerBalanceTracker() + self.assertEqual(0.0, tracker.peek_pending("BITVAVO")) + + def test_consume_pending_drains_to_zero(self): + tracker = BrokerBalanceTracker() + tracker.set_schedule( + "BITVAVO", + [ScheduledDeposit(amount=50.0, on=_utc(2024, 1, 5))], + ) + # Anchor (no deposits before this point) + tracker.fire_due_deposits("BITVAVO", _utc(2024, 1, 1)) + # After the one-shot fires + tracker.fire_due_deposits("BITVAVO", _utc(2024, 1, 6)) + self.assertEqual(50.0, tracker.peek_pending("BITVAVO")) + self.assertEqual(50.0, tracker.consume_pending("BITVAVO")) + self.assertEqual(0.0, tracker.peek_pending("BITVAVO")) + + def test_recurring_deposit_fires_on_cadence(self): + tracker = BrokerBalanceTracker() + tracker.set_schedule( + "BITVAVO", + [ + ScheduledDeposit( + amount=100.0, + time_unit=TimeUnit.DAY, + interval=30, + ), + ], + ) + # Anchor at start; first firing must be at start + 30d. + tracker.fire_due_deposits("BITVAVO", _utc(2024, 1, 1)) + self.assertEqual(0.0, tracker.peek_pending("BITVAVO")) + # Halfway: still nothing. + tracker.fire_due_deposits("BITVAVO", _utc(2024, 1, 15)) + self.assertEqual(0.0, tracker.peek_pending("BITVAVO")) + # Past first cadence boundary: one firing. + tracker.fire_due_deposits("BITVAVO", _utc(2024, 2, 1)) + self.assertEqual(100.0, tracker.peek_pending("BITVAVO")) + # Skip ahead 65 days from start: should now be 2 total firings. + tracker.fire_due_deposits( + "BITVAVO", _utc(2024, 1, 1) + timedelta(days=65) + ) + self.assertEqual(200.0, tracker.peek_pending("BITVAVO")) + + def test_one_shot_only_fires_once(self): + tracker = BrokerBalanceTracker() + tracker.set_schedule( + "BITVAVO", + [ScheduledDeposit(amount=500.0, on=_utc(2024, 6, 15))], + ) + tracker.fire_due_deposits("BITVAVO", _utc(2024, 1, 1)) + tracker.fire_due_deposits("BITVAVO", _utc(2024, 7, 1)) + tracker.fire_due_deposits("BITVAVO", _utc(2024, 8, 1)) + self.assertEqual(500.0, tracker.peek_pending("BITVAVO")) + + def test_negative_recurring_deposit_is_withdrawal(self): + tracker = BrokerBalanceTracker() + tracker.set_schedule( + "BITVAVO", + [ + ScheduledDeposit( + amount=-25.0, + time_unit=TimeUnit.DAY, + interval=7, + ), + ], + ) + tracker.fire_due_deposits("BITVAVO", _utc(2024, 1, 1)) + tracker.fire_due_deposits("BITVAVO", _utc(2024, 1, 22)) + # 3 weeks -> 3 firings of -25 = -75 + self.assertEqual(-75.0, tracker.peek_pending("BITVAVO")) + + def test_auto_sync_default_off(self): + tracker = BrokerBalanceTracker() + self.assertFalse(tracker.is_auto_sync("BITVAVO")) + tracker.set_auto_sync("BITVAVO", True) + self.assertTrue(tracker.is_auto_sync("BITVAVO")) + + def test_project_total_yields_sorted_events(self): + tracker = BrokerBalanceTracker() + tracker.set_schedule( + "BITVAVO", + [ + ScheduledDeposit( + amount=100.0, time_unit=TimeUnit.DAY, interval=30 + ), + ScheduledDeposit(amount=500.0, on=_utc(2024, 2, 15)), + ], + ) + events = tracker.project_total( + market="BITVAVO", + start=_utc(2024, 1, 1), + end=_utc(2024, 4, 1), + ) + self.assertEqual( + [ + (_utc(2024, 1, 31), 100.0), + (_utc(2024, 2, 15), 500.0), + (_utc(2024, 3, 1), 100.0), + (_utc(2024, 3, 31), 100.0), + ], + events, + ) + + def test_set_schedule_replaces_state(self): + tracker = BrokerBalanceTracker() + tracker.set_schedule( + "BITVAVO", + [ScheduledDeposit(amount=10.0, on=_utc(2024, 1, 5))], + ) + tracker.fire_due_deposits("BITVAVO", _utc(2024, 1, 6)) + self.assertEqual(10.0, tracker.peek_pending("BITVAVO")) + tracker.set_schedule( + "BITVAVO", + [ScheduledDeposit(amount=99.0, on=_utc(2024, 2, 5))], + ) + # Pending balance is preserved (it represents already-credited + # external cash, separate from the schedule itself). + self.assertEqual(10.0, tracker.peek_pending("BITVAVO")) + # But the original one-shot's "fired" flag is reset, so the new + # schedule operates independently. + tracker.fire_due_deposits("BITVAVO", _utc(2024, 2, 6)) + self.assertEqual(109.0, tracker.peek_pending("BITVAVO")) + + +class TestScheduledDepositValidation(TestCase): + + def test_rejects_no_schedule(self): + with self.assertRaises(ValueError): + ScheduledDeposit(amount=10.0) + + def test_rejects_mixed_schedule(self): + with self.assertRaises(ValueError): + ScheduledDeposit( + amount=10.0, + time_unit=TimeUnit.DAY, + interval=1, + on=_utc(2024, 1, 1), + ) + + def test_rejects_partial_recurring(self): + with self.assertRaises(ValueError): + ScheduledDeposit(amount=10.0, time_unit=TimeUnit.DAY) + with self.assertRaises(ValueError): + ScheduledDeposit(amount=10.0, interval=5) + + def test_rejects_non_positive_interval(self): + with self.assertRaises(ValueError): + ScheduledDeposit( + amount=10.0, time_unit=TimeUnit.DAY, interval=0 + ) diff --git a/tests/services/test_broker_balance_tracker_advanced.py b/tests/services/test_broker_balance_tracker_advanced.py new file mode 100644 index 00000000..2744a9e5 --- /dev/null +++ b/tests/services/test_broker_balance_tracker_advanced.py @@ -0,0 +1,185 @@ +"""Tests for advanced BrokerBalanceTracker features added alongside the +``Context.sync_portfolio`` work: cash-flow recording, anchor-day firing, +case-insensitive market normalization, error-mode storage, and +``set_schedule`` mid-run state preservation. +""" +from datetime import datetime, timezone +from unittest import TestCase + +from investing_algorithm_framework import ScheduledDeposit, TimeUnit +from investing_algorithm_framework.domain.exceptions import ( + OperationalException, +) +from investing_algorithm_framework.services.portfolios \ + .broker_balance_tracker import BrokerBalanceTracker + + +def _utc(y, m, d, h=0): + return datetime(y, m, d, h, tzinfo=timezone.utc) + + +class TestCashFlowRecording(TestCase): + """``record_cash_flow`` / ``drain_cash_flow`` underpin the TWR + snapshot path.""" + + def test_record_and_drain(self): + tracker = BrokerBalanceTracker() + tracker.record_cash_flow("binance", 100.0) + tracker.record_cash_flow("binance", -25.0) + # Same market case-insensitive + tracker.record_cash_flow("BINANCE", 5.0) + + # drain returns net cash flow since last drain and resets to 0 + self.assertEqual(80.0, tracker.drain_cash_flow("binance")) + self.assertEqual(0.0, tracker.drain_cash_flow("binance")) + + def test_total_cash_flow_is_cumulative(self): + tracker = BrokerBalanceTracker() + tracker.record_cash_flow("binance", 100.0) + tracker.drain_cash_flow("binance") + tracker.record_cash_flow("binance", 50.0) + # total_cash_flow keeps a running sum independent of drain + self.assertEqual(150.0, tracker.total_cash_flow("binance")) + + def test_unknown_market_drain_is_zero(self): + tracker = BrokerBalanceTracker() + self.assertEqual(0.0, tracker.drain_cash_flow("unknown")) + + +class TestErrorModeStorage(TestCase): + + def test_default_is_raise(self): + tracker = BrokerBalanceTracker() + tracker.set_schedule("binance", []) + self.assertEqual("raise", tracker.get_auto_sync_error_mode("binance")) + + def test_can_set_warn_or_halt(self): + tracker = BrokerBalanceTracker() + tracker.set_auto_sync_error_mode("binance", "warn") + self.assertEqual("warn", tracker.get_auto_sync_error_mode("binance")) + tracker.set_auto_sync_error_mode("binance", "halt") + self.assertEqual("halt", tracker.get_auto_sync_error_mode("binance")) + + def test_invalid_mode_rejected(self): + tracker = BrokerBalanceTracker() + with self.assertRaises((ValueError, OperationalException)): + tracker.set_auto_sync_error_mode("binance", "bogus") + + +class TestMarketNormalization(TestCase): + + def test_case_insensitive_keys(self): + tracker = BrokerBalanceTracker() + tracker.set_schedule( + "binance", + [ScheduledDeposit(amount=100.0, on=_utc(2024, 1, 5))], + ) + # Anchor + fire under different casing + tracker.fire_due_deposits("BINANCE", _utc(2024, 1, 1)) + tracker.fire_due_deposits("Binance", _utc(2024, 1, 6)) + # Peek with yet another casing + self.assertEqual(100.0, tracker.peek_pending("binance")) + self.assertTrue(tracker.has_market("BiNaNcE")) + + def test_none_market_raises(self): + tracker = BrokerBalanceTracker() + with self.assertRaises((ValueError, OperationalException)): + tracker.set_schedule(None, []) + + +class TestSetScheduleMidRun(TestCase): + + def test_pending_and_cash_flow_preserved(self): + tracker = BrokerBalanceTracker() + tracker.set_schedule( + "binance", + [ScheduledDeposit(amount=100.0, on=_utc(2024, 1, 5))], + ) + tracker.fire_due_deposits("binance", _utc(2024, 1, 1)) + tracker.fire_due_deposits("binance", _utc(2024, 1, 6)) + self.assertEqual(100.0, tracker.peek_pending("binance")) + + tracker.record_cash_flow("binance", 100.0) + + # Replace schedule mid-run + tracker.set_schedule( + "binance", + [ScheduledDeposit(amount=200.0, on=_utc(2024, 2, 5))], + ) + # Pending and cash_flow_since_snapshot must NOT be wiped + self.assertEqual(100.0, tracker.peek_pending("binance")) + self.assertEqual(100.0, tracker.drain_cash_flow("binance")) + + +class TestFireOnAnchor(TestCase): + + def test_recurring_with_fire_on_anchor_fires_immediately(self): + tracker = BrokerBalanceTracker() + deposit = ScheduledDeposit( + amount=50.0, + interval=14, + time_unit=TimeUnit.DAY, + fire_on_anchor=True, + ) + tracker.set_schedule("binance", [deposit]) + # Anchor itself should fire + tracker.fire_due_deposits( + "binance", _utc(2024, 1, 1), backtest_start=_utc(2024, 1, 1) + ) + self.assertEqual(50.0, tracker.peek_pending("binance")) + + def test_recurring_without_fire_on_anchor_does_not_fire_at_anchor(self): + tracker = BrokerBalanceTracker() + deposit = ScheduledDeposit( + amount=50.0, + interval=14, + time_unit=TimeUnit.DAY, + ) + tracker.set_schedule("binance", [deposit]) + tracker.fire_due_deposits( + "binance", _utc(2024, 1, 1), backtest_start=_utc(2024, 1, 1) + ) + self.assertEqual(0.0, tracker.peek_pending("binance")) + # Next interval (Jan 15) does fire + tracker.fire_due_deposits("binance", _utc(2024, 1, 15)) + self.assertEqual(50.0, tracker.peek_pending("binance")) + + def test_fire_on_anchor_rejected_for_one_shot(self): + with self.assertRaises(ValueError): + ScheduledDeposit( + amount=50.0, on=_utc(2024, 1, 1), fire_on_anchor=True + ) + + +class TestProjectTotalAnchor(TestCase): + + def test_project_total_includes_anchor_when_flagged(self): + tracker = BrokerBalanceTracker() + d = ScheduledDeposit( + amount=100.0, + interval=30, + time_unit=TimeUnit.DAY, + fire_on_anchor=True, + ) + tracker.set_schedule("binance", [d]) + # 60 days from anchor, fire_on_anchor=True β‡’ 3 fires (day 0, 30, 60) + events = tracker.project_total( + "binance", _utc(2024, 1, 1), _utc(2024, 3, 1) + ) + total = sum(amount for _, amount in events) + self.assertEqual(300.0, total) + + def test_project_total_excludes_anchor_by_default(self): + tracker = BrokerBalanceTracker() + d = ScheduledDeposit( + amount=100.0, + interval=30, + time_unit=TimeUnit.DAY, + ) + tracker.set_schedule("binance", [d]) + # 60 days from anchor, fire_on_anchor=False β‡’ 2 fires (day 30, 60) + events = tracker.project_total( + "binance", _utc(2024, 1, 1), _utc(2024, 3, 1) + ) + total = sum(amount for _, amount in events) + self.assertEqual(200.0, total) diff --git a/tests/services/test_stop_order.py b/tests/services/test_stop_order.py new file mode 100644 index 00000000..396cb38e --- /dev/null +++ b/tests/services/test_stop_order.py @@ -0,0 +1,348 @@ +""" +Tests for STOP and STOP_LIMIT order support (#439). + +Covers: +- OrderType.STOP / OrderType.STOP_LIMIT enum +- Order.stop_price / Order.triggered_at fields +- Stop / stop-limit order validation +- Trigger + fill logic in BacktestTradeOrderEvaluator +- Trigger + fill logic in OrderBacktestService.has_executed (legacy path) +""" +import os +from datetime import datetime, timezone + +import polars as pl + +from investing_algorithm_framework import ( + PortfolioConfiguration, + MarketCredential, + OrderStatus, + OrderType, + TradeStatus, + BacktestDateRange, + OrderSide, +) +from investing_algorithm_framework.domain import ( + INDEX_DATETIME, + Order, + OperationalException, +) +from investing_algorithm_framework.services import ( + BacktestTradeOrderEvaluator, + OrderBacktestService, +) +from tests.resources import TestBase + +OHLCV_CSV = os.path.join( + os.path.dirname(os.path.dirname(__file__)), + "resources", "test_data", "ohlcv", + "OHLCV_BTC-EUR_BINANCE_15m_2023-12-14-21-45_2023-12-25-00-00.csv", +) + + +class TestStopOrderTypeEnum(TestBase): + + market_credentials = [ + MarketCredential( + market="binance", api_key="api_key", secret_key="secret_key", + ) + ] + portfolio_configurations = [ + PortfolioConfiguration(market="binance", trading_symbol="EUR") + ] + external_balances = {"EUR": 1000000} + + def test_stop_order_type_exists(self): + self.assertEqual(OrderType.STOP.value, "STOP") + self.assertEqual(OrderType.STOP_LIMIT.value, "STOP_LIMIT") + + def test_stop_order_type_from_string(self): + self.assertEqual(OrderType.from_string("STOP"), OrderType.STOP) + self.assertEqual( + OrderType.from_string("STOP_LIMIT"), OrderType.STOP_LIMIT + ) + + def test_is_stop_order(self): + stop = Order( + order_type=OrderType.STOP.value, + order_side=OrderSide.SELL.value, + amount=1.0, target_symbol="BTC", trading_symbol="EUR", + stop_price=100.0, + ) + stop_limit = Order( + order_type=OrderType.STOP_LIMIT.value, + order_side=OrderSide.SELL.value, + amount=1.0, target_symbol="BTC", trading_symbol="EUR", + stop_price=100.0, price=99.0, + ) + limit = Order( + order_type=OrderType.LIMIT.value, + order_side=OrderSide.SELL.value, + amount=1.0, target_symbol="BTC", trading_symbol="EUR", + price=100.0, + ) + self.assertTrue(stop.is_stop_order()) + self.assertTrue(stop_limit.is_stop_order()) + self.assertFalse(limit.is_stop_order()) + self.assertFalse(stop.is_triggered()) + + +class _StopOrderTestBase(TestBase): + """Shared setup for evaluator-based stop order tests.""" + + market_credentials = [ + MarketCredential( + market="binance", api_key="api_key", secret_key="secret_key", + ) + ] + portfolio_configurations = [ + PortfolioConfiguration(market="binance", trading_symbol="EUR") + ] + external_balances = {"EUR": 1000000} + + def setUp(self): + super().setUp() + self.app.container.order_service.override( + OrderBacktestService( + trade_service=self.app.container.trade_service(), + order_repository=self.app.container.order_repository(), + position_service=self.app.container.position_service(), + portfolio_repository=( + self.app.container.portfolio_repository() + ), + portfolio_configuration_service=( + self.app.container.portfolio_configuration_service() + ), + portfolio_snapshot_service=( + self.app.container.portfolio_snapshot_service() + ), + configuration_service=( + self.app.container.configuration_service() + ), + ) + ) + backtest_date_range = BacktestDateRange( + start_date=datetime(2023, 12, 14), + end_date=datetime(2023, 12, 25), + ) + self.app.initialize_backtest_config(backtest_date_range) + configuration_service = self.app.container.configuration_service() + configuration_service.add_value( + INDEX_DATETIME, + datetime(2023, 12, 14, 21, 0, 0, tzinfo=timezone.utc), + ) + + self.ohlcv_df = pl.read_csv(OHLCV_CSV) + self.ohlcv_df = self.ohlcv_df.with_columns( + pl.col("Datetime").str.to_datetime().dt.replace_time_zone("UTC") + ) + + def _create_evaluator(self, trading_costs=None): + return BacktestTradeOrderEvaluator( + trade_service=self.app.container.trade_service(), + order_service=self.app.container.order_service(), + trade_stop_loss_service=( + self.app.container.trade_stop_loss_service() + ), + trade_take_profit_service=( + self.app.container.trade_take_profit_service() + ), + configuration_service=( + self.app.container.configuration_service() + ), + trading_costs=trading_costs, + ) + + def _buy_position(self, amount, price): + order_service = self.app.container.order_service() + order = order_service.create({ + "target_symbol": "BTC", + "trading_symbol": "EUR", + "amount": amount, + "order_side": OrderSide.BUY.value, + "price": price, + "order_type": OrderType.LIMIT.value, + "portfolio_id": 1, + "status": "CREATED", + }) + order_service.update(order.id, { + "status": OrderStatus.CLOSED.value, + "filled": amount, + "remaining": 0, + }) + return order + + +class TestStopOrderValidation(_StopOrderTestBase): + + def test_stop_order_requires_stop_price(self): + order_service = self.app.container.order_service() + self._buy_position(0.1, 39000) + + with self.assertRaises(OperationalException): + order_service.create({ + "target_symbol": "BTC", + "trading_symbol": "EUR", + "amount": 0.1, + "order_side": OrderSide.SELL.value, + "order_type": OrderType.STOP.value, + "portfolio_id": 1, + "status": "CREATED", + }) + + def test_stop_limit_order_requires_stop_and_limit_price(self): + order_service = self.app.container.order_service() + self._buy_position(0.1, 39000) + + with self.assertRaises(OperationalException): + order_service.create({ + "target_symbol": "BTC", + "trading_symbol": "EUR", + "amount": 0.1, + "order_side": OrderSide.SELL.value, + "order_type": OrderType.STOP_LIMIT.value, + "portfolio_id": 1, + "status": "CREATED", + "stop_price": 38000, + # missing limit price + }) + + def test_sell_stop_limit_rejects_limit_above_stop(self): + order_service = self.app.container.order_service() + self._buy_position(0.1, 39000) + + with self.assertRaises(OperationalException): + order_service.create({ + "target_symbol": "BTC", + "trading_symbol": "EUR", + "amount": 0.1, + "order_side": OrderSide.SELL.value, + "order_type": OrderType.STOP_LIMIT.value, + "portfolio_id": 1, + "status": "CREATED", + "stop_price": 38000, + "price": 38500, # limit above stop is invalid for SELL + }) + + +class TestStopOrderFill(_StopOrderTestBase): + + def test_sell_stop_triggers_and_fills_at_stop_price(self): + """SELL STOP: triggers when Low <= stop_price, then fills as + a market order at the stop_price (no slippage by default).""" + order_service = self.app.container.order_service() + trade_service = self.app.container.trade_service() + self._buy_position(0.1, 39000) + + # Pick a stop above the lowest low so it triggers in-range + lowest_low = self.ohlcv_df["Low"].min() + stop_price = lowest_low + 100 # ensures trigger + + stop_order = order_service.create({ + "target_symbol": "BTC", + "trading_symbol": "EUR", + "amount": 0.1, + "order_side": OrderSide.SELL.value, + "order_type": OrderType.STOP.value, + "portfolio_id": 1, + "status": "CREATED", + "stop_price": stop_price, + }) + + evaluator = self._create_evaluator() + evaluator.evaluate( + open_trades=trade_service.get_all( + {"status": TradeStatus.OPEN.value} + ), + open_orders=order_service.get_all( + {"status": OrderStatus.OPEN.value} + ), + ohlcv_data={"BTC/EUR": self.ohlcv_df}, + ) + + filled = order_service.get(stop_order.id) + self.assertEqual(OrderStatus.CLOSED.value, filled.status) + self.assertEqual(0.1, filled.filled) + self.assertIsNotNone(filled.triggered_at) + self.assertEqual(float(filled.price), float(stop_price)) + + def test_sell_stop_does_not_trigger_when_price_stays_above(self): + order_service = self.app.container.order_service() + trade_service = self.app.container.trade_service() + self._buy_position(0.1, 39000) + + # Stop well below all lows β€” should never trigger + stop_price = float(self.ohlcv_df["Low"].min()) - 10000 + + stop_order = order_service.create({ + "target_symbol": "BTC", + "trading_symbol": "EUR", + "amount": 0.1, + "order_side": OrderSide.SELL.value, + "order_type": OrderType.STOP.value, + "portfolio_id": 1, + "status": "CREATED", + "stop_price": stop_price, + }) + + evaluator = self._create_evaluator() + evaluator.evaluate( + open_trades=trade_service.get_all( + {"status": TradeStatus.OPEN.value} + ), + open_orders=order_service.get_all( + {"status": OrderStatus.OPEN.value} + ), + ohlcv_data={"BTC/EUR": self.ohlcv_df}, + ) + + still_open = order_service.get(stop_order.id) + self.assertEqual(OrderStatus.OPEN.value, still_open.status) + self.assertIsNone(still_open.triggered_at) + + def test_sell_stop_limit_triggers_then_fills_at_limit(self): + """SELL STOP_LIMIT: triggers when Low <= stop_price, then + becomes a SELL limit order at the limit price.""" + order_service = self.app.container.order_service() + trade_service = self.app.container.trade_service() + self._buy_position(0.1, 39000) + + lowest_low = float(self.ohlcv_df["Low"].min()) + highest_high = float(self.ohlcv_df["High"].max()) + + # Stop in range so it triggers, limit below stop and below + # highest high so the limit also fills (SELL limit fills when + # High >= limit price). + stop_price = lowest_low + 200 + limit_price = stop_price - 50 + # Sanity: limit_price must be <= highest_high + self.assertLess(limit_price, highest_high) + + stop_limit_order = order_service.create({ + "target_symbol": "BTC", + "trading_symbol": "EUR", + "amount": 0.1, + "order_side": OrderSide.SELL.value, + "order_type": OrderType.STOP_LIMIT.value, + "portfolio_id": 1, + "status": "CREATED", + "stop_price": stop_price, + "price": limit_price, + }) + + evaluator = self._create_evaluator() + evaluator.evaluate( + open_trades=trade_service.get_all( + {"status": TradeStatus.OPEN.value} + ), + open_orders=order_service.get_all( + {"status": OrderStatus.OPEN.value} + ), + ohlcv_data={"BTC/EUR": self.ohlcv_df}, + ) + + filled = order_service.get(stop_limit_order.id) + self.assertEqual(OrderStatus.CLOSED.value, filled.status) + self.assertIsNotNone(filled.triggered_at) + # Fill price for STOP_LIMIT is the limit price + self.assertEqual(float(filled.price), float(limit_price)) diff --git a/tests/services/test_sync_portfolio.py b/tests/services/test_sync_portfolio.py new file mode 100644 index 00000000..b0e08d91 --- /dev/null +++ b/tests/services/test_sync_portfolio.py @@ -0,0 +1,180 @@ +"""Integration tests for ``Context.sync_portfolio`` and the auto-sync hook.""" +from datetime import datetime, timezone + +from investing_algorithm_framework import ( + MarketCredential, + PortfolioConfiguration, + PortfolioOutOfSyncError, + ScheduledDeposit, + SyncResult, + TimeUnit, +) +from investing_algorithm_framework.domain import ( + ENVIRONMENT, + Environment, + INDEX_DATETIME, +) +from tests.resources import TestBase + + +def _utc(year, month, day): + return datetime(year, month, day, tzinfo=timezone.utc) + + +class TestSyncPortfolioLiveMode(TestBase): + """Live (TEST env hits the live code path) β€” broker is the stub + ``PortfolioProviderTest`` whose balance we mutate directly.""" + + portfolio_configurations = [ + PortfolioConfiguration( + market="binance", + trading_symbol="EUR", + initial_balance=1000, + ) + ] + market_credentials = [ + MarketCredential(market="binance", api_key="x", secret_key="y") + ] + external_balances = {"EUR": 1000} + + def setUp(self) -> None: + super().setUp() + # The TestBase registers the stub provider; ensure it answers for + # 'binance' explicitly. + lookup = self.app.container.portfolio_provider_lookup() + lookup.register_portfolio_provider_for_market("binance") + provider = lookup.get_portfolio_provider("binance") + provider.external_balances = {"EUR": 1000} + self._provider = provider + + def _broker(self): + return self._provider + + def test_noop_when_in_sync(self): + result = self.app.context.sync_portfolio(market="binance") + self.assertIsInstance(result, SyncResult) + self.assertEqual("noop", result.kind) + self.assertEqual(0.0, result.delta) + self.assertEqual(1000.0, result.new_unallocated) + + def test_deposit_absorbed(self): + self._broker().external_balances = {"EUR": 1300} + result = self.app.context.sync_portfolio(market="binance") + self.assertEqual("deposit", result.kind) + self.assertEqual(300.0, result.delta) + self.assertEqual(1300.0, result.new_unallocated) + # Persisted to the portfolio + portfolio = self.app.container.portfolio_service() \ + .find({"market": "binance"}) + self.assertEqual(1300.0, portfolio.get_unallocated()) + + def test_withdrawal_raises_by_default(self): + self._broker().external_balances = {"EUR": 800} + with self.assertRaises(PortfolioOutOfSyncError) as ctx: + self.app.context.sync_portfolio(market="binance") + err = ctx.exception + self.assertEqual("BINANCE", err.market) + self.assertEqual(1000.0, err.local_unallocated) + self.assertEqual(800.0, err.broker_available) + self.assertEqual(-200.0, err.delta) + # State must NOT have changed + portfolio = self.app.container.portfolio_service() \ + .find({"market": "binance"}) + self.assertEqual(1000.0, portfolio.get_unallocated()) + + def test_withdrawal_allowed_when_opted_in(self): + self._broker().external_balances = {"EUR": 800} + result = self.app.context.sync_portfolio( + market="binance", allow_withdrawals=True + ) + self.assertEqual("withdrawal", result.kind) + self.assertEqual(-200.0, result.delta) + self.assertEqual(800.0, result.new_unallocated) + + def test_negative_broker_balance_always_raises(self): + self._broker().external_balances = {"EUR": -5} + with self.assertRaises(PortfolioOutOfSyncError): + self.app.context.sync_portfolio( + market="binance", allow_withdrawals=True + ) + + +class TestSyncPortfolioBacktestMode(TestBase): + """Backtest mode β€” sync_portfolio reads pending deposits from the + BrokerBalanceTracker rather than from a live broker.""" + + portfolio_configurations = [ + PortfolioConfiguration( + market="binance", + trading_symbol="EUR", + initial_balance=1000, + ) + ] + market_credentials = [ + MarketCredential(market="binance", api_key="x", secret_key="y") + ] + external_balances = {"EUR": 1000} + + def setUp(self) -> None: + super().setUp() + # Flip the env to BACKTEST so sync_portfolio uses the tracker. + self.app.container.configuration_service().add_value( + ENVIRONMENT, Environment.BACKTEST.value + ) + + def _tracker(self): + return self.app.container.broker_balance_tracker() + + def test_noop_when_no_pending(self): + result = self.app.context.sync_portfolio(market="binance") + self.assertEqual("noop", result.kind) + + def test_pending_deposit_absorbed(self): + tracker = self._tracker() + tracker.set_schedule( + "binance", + [ScheduledDeposit(amount=250.0, on=_utc(2024, 1, 5))], + ) + # Anchor + fire after the one-shot + tracker.fire_due_deposits("binance", _utc(2024, 1, 1)) + tracker.fire_due_deposits("binance", _utc(2024, 1, 6)) + + result = self.app.context.sync_portfolio(market="binance") + self.assertEqual("deposit", result.kind) + self.assertEqual(250.0, result.delta) + self.assertEqual(1250.0, result.new_unallocated) + + # Pending was drained. + self.assertEqual(0.0, tracker.peek_pending("binance")) + + # Calling again is a noop. + result2 = self.app.context.sync_portfolio(market="binance") + self.assertEqual("noop", result2.kind) + + def test_negative_pending_raises_without_opt_in(self): + tracker = self._tracker() + tracker.set_schedule( + "binance", + [ScheduledDeposit(amount=-200.0, on=_utc(2024, 1, 5))], + ) + tracker.fire_due_deposits("binance", _utc(2024, 1, 1)) + tracker.fire_due_deposits("binance", _utc(2024, 1, 6)) + + with self.assertRaises(PortfolioOutOfSyncError): + self.app.context.sync_portfolio(market="binance") + + def test_negative_pending_allowed(self): + tracker = self._tracker() + tracker.set_schedule( + "binance", + [ScheduledDeposit(amount=-200.0, on=_utc(2024, 1, 5))], + ) + tracker.fire_due_deposits("binance", _utc(2024, 1, 1)) + tracker.fire_due_deposits("binance", _utc(2024, 1, 6)) + + result = self.app.context.sync_portfolio( + market="binance", allow_withdrawals=True + ) + self.assertEqual("withdrawal", result.kind) + self.assertEqual(-200.0, result.delta) + self.assertEqual(800.0, result.new_unallocated) diff --git a/tests/services/test_sync_portfolio_advanced.py b/tests/services/test_sync_portfolio_advanced.py new file mode 100644 index 00000000..96a27546 --- /dev/null +++ b/tests/services/test_sync_portfolio_advanced.py @@ -0,0 +1,147 @@ +"""Tests for advanced ``Context.sync_portfolio`` features: + +- ``tolerance`` parameter (small drift treated as no-op) +- ``reserved_for_pending_orders`` exposure on ``SyncResult`` +- Live mode subtracts CREATED BUY-order reservations +- ``cash_flow`` is recorded on the tracker after a non-noop sync +- Case-insensitive market resolution +""" +from datetime import datetime, timezone + +from investing_algorithm_framework import ( + MarketCredential, + PortfolioConfiguration, + PortfolioOutOfSyncError, + SyncResult, +) +from investing_algorithm_framework.domain import ( + ENVIRONMENT, + Environment, + OrderSide, + OrderStatus, + OrderType, +) +from tests.resources import TestBase + + +def _utc(year, month, day): + return datetime(year, month, day, tzinfo=timezone.utc) + + +class _LiveSyncBase(TestBase): + """Base setup mirroring the existing TestSyncPortfolioLiveMode.""" + + portfolio_configurations = [ + PortfolioConfiguration( + market="binance", + trading_symbol="EUR", + initial_balance=1000, + ) + ] + market_credentials = [ + MarketCredential(market="binance", api_key="x", secret_key="y") + ] + external_balances = {"EUR": 1000} + + def setUp(self) -> None: + super().setUp() + lookup = self.app.container.portfolio_provider_lookup() + lookup.register_portfolio_provider_for_market("binance") + provider = lookup.get_portfolio_provider("binance") + provider.external_balances = {"EUR": 1000} + self._provider = provider + + def _broker(self): + return self._provider + + +class TestTolerance(_LiveSyncBase): + + def test_drift_within_tolerance_is_noop(self): + # 0.5 EUR drift, tolerance 1 EUR β†’ noop + self._broker().external_balances = {"EUR": 1000.5} + result = self.app.context.sync_portfolio( + market="binance", tolerance=1.0 + ) + self.assertEqual("noop", result.kind) + # Within-tolerance flag exposed for diagnostics; the raw delta + # is still surfaced for observability. + self.assertTrue(result.within_tolerance) + self.assertAlmostEqual(0.5, result.delta, places=6) + # Local state unchanged + portfolio = self.app.container.portfolio_service() \ + .find({"market": "binance"}) + self.assertEqual(1000.0, portfolio.get_unallocated()) + + def test_drift_above_tolerance_still_syncs(self): + self._broker().external_balances = {"EUR": 1100.0} + result = self.app.context.sync_portfolio( + market="binance", tolerance=1.0 + ) + self.assertEqual("deposit", result.kind) + self.assertEqual(100.0, result.delta) + + def test_negative_tolerance_rejected(self): + with self.assertRaises(Exception): + self.app.context.sync_portfolio( + market="binance", tolerance=-1.0 + ) + + +class TestReservedCashFieldExposed(_LiveSyncBase): + """The ``reserved_for_pending_orders`` field is always present on the + SyncResult, even when there are no pending orders. The intricate + "reservation absorbs phantom withdrawal" semantics is exercised at + the unit level via ``_reserved_cash_for_pending_orders``; this test + covers the public contract.""" + + def test_reserved_zero_when_no_pending_orders(self): + result = self.app.context.sync_portfolio(market="binance") + self.assertEqual("noop", result.kind) + self.assertEqual(0.0, result.reserved_for_pending_orders) + + def test_reserved_field_exists_on_deposit(self): + self._broker().external_balances = {"EUR": 1100} + result = self.app.context.sync_portfolio(market="binance") + self.assertEqual("deposit", result.kind) + # Field is always present and non-negative + self.assertGreaterEqual(result.reserved_for_pending_orders, 0.0) + + +class TestCashFlowRecordedOnSync(_LiveSyncBase): + + def test_cash_flow_recorded_after_deposit(self): + self._broker().external_balances = {"EUR": 1300} + self.app.context.sync_portfolio(market="binance") + + tracker = self.app.container.broker_balance_tracker() + # Drain returns the +300 deposit recorded by the sync + self.assertEqual(300.0, tracker.drain_cash_flow("binance")) + + def test_noop_does_not_record_cash_flow(self): + self.app.context.sync_portfolio(market="binance") + tracker = self.app.container.broker_balance_tracker() + self.assertEqual(0.0, tracker.drain_cash_flow("binance")) + + def test_withdrawal_recorded_as_negative_cash_flow(self): + self._broker().external_balances = {"EUR": 800} + self.app.context.sync_portfolio( + market="binance", allow_withdrawals=True + ) + tracker = self.app.container.broker_balance_tracker() + self.assertEqual(-200.0, tracker.drain_cash_flow("binance")) + + +class TestMarketCaseInsensitive(_LiveSyncBase): + + def test_uppercase_market(self): + self._broker().external_balances = {"EUR": 1100} + result = self.app.context.sync_portfolio(market="BINANCE") + self.assertEqual("deposit", result.kind) + self.assertEqual(100.0, result.delta) + + def test_mixed_case_market(self): + self._broker().external_balances = {"EUR": 1100} + result = self.app.context.sync_portfolio(market="Binance") + self.assertEqual("deposit", result.kind) + self.assertEqual(100.0, result.delta)