Skip to content

Commit df28b50

Browse files
cloudQuantcloudQuant
committed
feat: migrate functional indicators and shared data loaders
1 parent 4f05beb commit df28b50

1,335 files changed

Lines changed: 29051 additions & 79560 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG_v1.2.0.md‎

Lines changed: 310 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,310 @@
1+
# Changelog — Version 1.2.0
2+
3+
**Base branch**: `dev` (merged into `development`)
4+
**Commit range**: `5a886266` … `4f05beb0` (276 commits)
5+
**Period**: 2026-03-08 — 2026-06-01
6+
7+
---
8+
9+
## Highlights
10+
11+
| Metric | Value |
12+
|--------|-------|
13+
| Commits | 276 |
14+
| Files changed | ~3,078 |
15+
| Lines added | +3,346,623 |
16+
| Lines removed | −233,658 |
17+
| Strategy regression tests | 1,271 (all passing) |
18+
| Regression suite speedup | **−46%** (7m18s → 3m56s) |
19+
20+
---
21+
22+
## Breaking Changes
23+
24+
> The following legacy live-trading integrations have been removed. If you rely on any of these, **do not upgrade** and stay on the `development` branch.
25+
26+
| Removed Module | Replacement |
27+
|----------------|------------|
28+
| `backtrader/ccxt/` (full module) | Use `backtrader/stores/btapistore.py` + `bt_api_py` |
29+
| `backtrader/stores/ccxtstore.py` | Use `btapistore` |
30+
| `backtrader/stores/cryptostore.py` | Use `btapistore` |
31+
| `backtrader/stores/ctpstore.py` | Use `btapistore` (SimNow support) |
32+
| `backtrader/stores/futustore.py` | Use `btapistore` (Futu not yet migrated) |
33+
| `backtrader/stores/ibstore.py` | Use `btapistore` (IB not yet migrated) |
34+
| `backtrader/stores/oandastore.py` | Use `btapistore` (OANDA not yet migrated) |
35+
| `backtrader/stores/vcstore.py` | Use `btapistore` (VC not yet migrated) |
36+
| `backtrader/brokers/ccxtbroker.py` | Use `btapibroker` |
37+
| `backtrader/brokers/cryptobroker.py` | Use `btapibroker` |
38+
| `backtrader/brokers/ctpbroker.py` | Use `btapibroker` |
39+
| `backtrader/brokers/futubroker.py` | Use `btapibroker` |
40+
| `backtrader/brokers/ibbroker.py` | Use `btapibroker` |
41+
| `backtrader/brokers/oandabroker.py` | Use `btapibroker` |
42+
| `backtrader/brokers/obbroker.py` | Use `btapibroker` |
43+
| `backtrader/brokers/vcbroker.py` | Use `btapibroker` |
44+
| `backtrader/feeds/ccxt_live_tick.py` | Use `btapifeed` |
45+
| `backtrader/feeds/ccxtfeed.py` | Use `btapifeed` |
46+
| `backtrader/feeds/cryptofeed.py` | Use `btapifeed` |
47+
| `backtrader/feeds/ctpdata.py` | Use `btapifeed` |
48+
| `backtrader/feeds/futufeed.py` | Use `btapifeed` |
49+
| `backtrader/feeds/ibdata.py` | Use `btapifeed` |
50+
| `backtrader/feeds/oanda.py` | Use `btapifeed` |
51+
| `backtrader/feeds/vcdata.py` | Use `btapifeed` |
52+
| `backtrader/commissions/dc_commission.py` | (dead code) |
53+
54+
**Static data feeds** (CSV, Pandas, Yahoo Finance) are **unaffected**.
55+
56+
---
57+
58+
## New Features
59+
60+
### `bt_api_py` Unified Live-Trading API
61+
62+
A new unified store architecture via `bt_api_py` replaces the fragmented per-broker implementations.
63+
64+
- **`backtrader/stores/btapistore.py`** — Central store implementing the `bt_api_py` protocol
65+
- **`backtrader/feeds/btapifeed.py`** — Generic data feed for `bt_api_py` sources
66+
- **`backtrader/brokers/btapibroker.py`** — Generic broker for `bt_api_py` sources
67+
68+
### HFT (High-Frequency Trading) Framework
69+
70+
New `backtrader/brokers/hft/` package for tick-level backtesting and live trading:
71+
72+
| File | Purpose |
73+
|------|---------|
74+
| `binance_bbo.py` | Binance Best Bid-Offer capture |
75+
| `binance_bbo_compare.py` | BBO cross-exchange comparison |
76+
| `matching_core.py` | Order matching engine |
77+
| `latency.py` | Latency modeling |
78+
| `recorder.py` | Trade recording |
79+
| `queue.py` | Priority queue for order book |
80+
| `state.py` | Session state management |
81+
| `exchange.py` | Exchange interface |
82+
| `examples.py` | Usage examples |
83+
84+
### New Indicators
85+
86+
- **`backtrader/indicators/mt5atr.py`** — MT5-style ATR implementation
87+
88+
### Position Modes
89+
90+
- **`backtrader/position_modes.py`** — Explicit position mode management
91+
92+
### Trade Profiles
93+
94+
- **`backtrader/profiles.py`** — Trade profile/account configuration
95+
96+
---
97+
98+
## Performance Improvements
99+
100+
### Regression Suite Speedup (−46%)
101+
102+
The full 1,271-strategy regression suite now completes in **3m56s** vs 7m18s previously (macOS, Python 3.11, 8 parallel workers).
103+
104+
| Strategy Category | Speedup |
105+
|-----------------|---------|
106+
| Simple MA Cross | ~40-45% |
107+
| Multi-Indicator | ~45-50% |
108+
| Multi-Data | ~42-48% |
109+
| Complex Strategies | ~38-42% |
110+
111+
### Core Optimizations
112+
113+
| Module | What was changed |
114+
|--------|-----------------|
115+
| `linebuffer.py` | Hot-path scalar reads, NaN-check optimization |
116+
| `broker.py` | Parameter cache, hot-path method inlining |
117+
| Indicators `once()` | Math-function and constant caching |
118+
| `cerebro.py` | Phase extraction, complexity reduction |
119+
120+
### Multi-Data Clock Fix
121+
122+
Fixed secondary-feed indicator clock advancement in `runonce` mode. Indicators like `SMA((h1.high + h1.low)/2.0)` or `EMA(EMA(h4.close))` inside an M15 strategy now correctly advance on the **secondary feed's clock**, not the strategy's primary feed.
123+
124+
---
125+
126+
## Code Quality Improvements
127+
128+
### S1–S8 Iterations Summary
129+
130+
The dev branch underwent aggressive code-quality remediation across 8 sprint iterations:
131+
132+
| Category | Work Done |
133+
|---------|---------|
134+
| **Exception visibility** | All `except Exception: pass` converted to `except Exception as e` with debug logging |
135+
| **Analyzer hardening** | 49 rounds of sanitization — NaN/Inf handling, divide-by-zero guards, boundary checks |
136+
| **Sharpe Ratio** | Complete rewrite of input validation, trial-count bounds, annualization |
137+
| **Drawdown analyzers** | Invalid value sanitization end-to-end |
138+
| **Mypy compliance** | All core modules cleared (200+ errors → 0) |
139+
| **Ruff/Black** | All files formatted to line-length 100 |
140+
| **Dead code** | ~680 lines of commented-out code removed |
141+
| **Silent except** | Every silent catch replaced with logging/raising |
142+
143+
### Complexity Reductions (selected)
144+
145+
| File | Cyclomatic Complexity | Before → After |
146+
|------|---------------------|----------------|
147+
| `BackBroker._get_value` | Split into 3 helpers | 36 → 11 |
148+
| `SignalStrategy._next_signal` | Phase extraction | 84 → 28 |
149+
| `cerebro.run` flag setup | Extracted to helper | 41 → 32 |
150+
| `TradeAnalyzer.closed_trade_stats` | Extracted to helper | 32 → 3 |
151+
| `SharpeRatio.stop` | Split | 40 → 2 |
152+
| `PandasData` column mapping | Extracted resolver | 22 → 12 |
153+
| `get_pnl_metrics` trade block | Extracted | 26 → 12 |
154+
| `btrun` CLI | Refactored config tabs | reduced |
155+
156+
---
157+
158+
## Testing Improvements
159+
160+
### Strategy Regression Suite
161+
162+
- **1,271 inline regression tests** — All strategy tests migrated from source files into self-contained inline test files
163+
- Test duration tracking — each strategy test is timed; slowest 65% auto-tagged `slow`
164+
- `test-fast` (fastest 35%) completes in ~3.5 min
165+
- `test-slow` (slowest 65%) runs separately
166+
- `test-strategies` (all 1,271) completes in ~4 min on `dev`
167+
168+
### New Unit Test Coverage
169+
170+
- `tests/unit/observers/test_trade_logger_edge_cases.py` (368 lines)
171+
- `tests/unit/observers/test_trade_logger_internal_errors.py` (56 lines)
172+
- `tests/unit/observers/test_trade_logger_monitoring.py` (105 lines)
173+
- `tests/unit/reports/test_performance_calculator_edge_cases.py` (565 lines)
174+
- `tests/unit/reports/test_performance_edge_cases.py` (491 lines)
175+
- `tests/unit/stores/test_btapistore.py` (1,890 lines)
176+
- `tests/unit/stores/test_btapistore_edge_cases.py` (244 lines)
177+
- `tests/unit/stores/test_btapistore_notifications.py` (291 lines)
178+
- `tests/unit/stores/test_credential_safety.py` (85 lines)
179+
- `tests/unit/test_live_profile.py` (577 lines)
180+
- `tests/unit/test_live_validator.py` (45 lines)
181+
- Plus: broker observers, benchmark, buysell, drawdown, logreturns, trades, Sizer/CommInfo zero-price edge cases, and more
182+
183+
---
184+
185+
## Bug Fixes
186+
187+
### Batch 1–57 (Non-Finite Value Sanitization)
188+
189+
57 rounds of fixes for NaN/Inf propagation in:
190+
191+
- `linebuffer.py` — hot-path scalar reads, extend, write entrypoints, binary/unary operations
192+
- `lineseries.py` — index reads, current values, stage2 arithmetic
193+
- `report` charts — equity curve, drawdown, trade metrics, benchmark
194+
- `bbroker.py` — fundstartval=0 division-by-zero, orderbook floats
195+
- `tradingcal.py` — `nextday_week` return value
196+
197+
### Runonce Fixes
198+
199+
- Secondary-feed indicator clock advancement (WMA fsum parity)
200+
- Line action consistency and clocks
201+
- Indicator warmup parity
202+
- Child indicator registration in `bt.If` and `Logic` subclasses
203+
- `LinesOperation` as indicator data source in runonce mode
204+
- `bt.If` self-referencing patterns
205+
206+
### Order/Broker Fixes
207+
208+
- Stale tick prices for stacked bars
209+
- Submitted order cash projection
210+
- Order cancellation handling
211+
- Order `__ne__` None crash
212+
- `orderstatus int.status` bug
213+
- `BackBroker._get_value` divide-by-zero
214+
- `ComminfoDC` credit interest `.seconds` bug
215+
216+
### Data Feed Fixes
217+
218+
- `calendars.fill_price` None TypeError
219+
- Renko autosize divide-by-zero
220+
- Resample data clone
221+
- `PandasData` column mapping resolution (regression)
222+
223+
### Indicator Fixes
224+
225+
- Minperiod propagation for line-bound indicators
226+
- Pre-allocate `dst` array in `Logic` subclass `once()` methods
227+
228+
### TradeLogger Fixes
229+
230+
- Include `datetime` in text logs
231+
- Broker value/cash in bar and position logs
232+
233+
---
234+
235+
## Security Hardening
236+
237+
- **Diagnostic logging** — All `TradeLogger` diagnostic output routed through `get_logger` (no `print`)
238+
- **Credential masking** — API keys/secrets masked in logs for `btapistore` and live-trading paths
239+
- **Network timeout** — Explicit timeout configuration on all network operations
240+
- **Exception visibility** — No more silent failures; all catch blocks log or raise
241+
242+
---
243+
244+
## Documentation
245+
246+
- **ARCHITECTURE.md** — New architecture reference document
247+
- **CLAUDE.md** — Fully rewritten with verified current state
248+
- **README.md** — Refreshed with tiered test commands, performance benchmarks, and backtrader source-switch documentation
249+
- **CONTRIBUTING.md** — Updated with sprint documentation
250+
- Module docstrings added to all 11 modules previously missing them
251+
252+
---
253+
254+
## Test Infrastructure
255+
256+
- **Duration-based test splitting** — `conftest.py` auto-tags slow strategy tests
257+
- `tests/datas/mt5_1d_data/` — MT5 daily CSV fixtures for all regression symbols (XAUUSD, XAGUSD, IVV, IEF, GLD, IWM, etc.)
258+
- `BACKTRADER_USE_INSTALLED` env var and `--use-installed-backtrader` CLI flag to test installed vs local copy
259+
- Scripts: `scripts/refresh_strategy_durations.py`, `scripts/run_strategy_branch_compare.py`
260+
261+
---
262+
263+
## Deprecations
264+
265+
| Deprecated | Status |
266+
|------------|--------|
267+
| `backtrader/commissions/dc_commission.py` | **Removed** (was dead code) |
268+
| All legacy store/broker modules | **Removed** — use `btapistore` + `btapibroker` |
269+
| CCXT-based feeds and stores | **Removed** — use `btapifeed` + `btapistore` |
270+
271+
---
272+
273+
## Migration Guide
274+
275+
### From Any Version < 1.2.0 Using Legacy Live Trading
276+
277+
If you use IB, OANDA, CCXT, CTP, Futu, or VC live trading:
278+
279+
1. Install `bt_api_py`: `pip install bt_api_py`
280+
2. Replace store instantiation:
281+
282+
```python
283+
# Before (removed)
284+
import backtrader as bt
285+
data = bt.feeds.OandaData(...)
286+
broker = bt.brokers.OandaBroker(...)
287+
288+
# After
289+
import backtrader as bt
290+
from backtrader.stores.btapistore import btapistore
291+
store = btapistore(token="your_oanda_token")
292+
data = bt.feeds.BTAPIFeed(store=store, ...)
293+
broker = bt.brokers.BTAPIBroker(store=store)
294+
```
295+
296+
> **Note**: `bt_api_py` support varies by broker. Check `backtrader/stores/btapistore.py` for current broker coverage.
297+
298+
### Static Data Users (CSV, Pandas, Yahoo)
299+
300+
**No action required.** These are completely unaffected.
301+
302+
---
303+
304+
## Version Reference
305+
306+
| Version | Branch | Status |
307+
|---------|--------|--------|
308+
| 1.1.0 | `master` | Stable |
309+
| 1.2.0 | `dev` → `development` | **Release candidate** |
310+

‎backtrader/brokers/hft/binance_bbo.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -356,7 +356,7 @@ def _extract_symbol_and_date(zip_path: Path) -> tuple[str, str]:
356356
def _default_bt_symbol(symbol: str) -> str:
357357
for quote in ("USDT", "USDC", "BUSD", "FDUSD", "BTC", "ETH", "BNB"):
358358
if symbol.endswith(quote) and len(symbol) > len(quote):
359-
return f"{symbol[:-len(quote)]}/{quote}"
359+
return f"{symbol[: -len(quote)]}/{quote}"
360360
return symbol
361361

362362

‎backtrader/brokers/hft/binance_bbo_compare.py‎

Lines changed: 34 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -745,18 +745,21 @@ def _iter_exchange_market_events(market_data_path, symbol: str) -> Iterator[tupl
745745
timestamp = float(int(row["exch_ts"]) / 1_000_000_000.0)
746746
if ev & _TRADE_EVENT:
747747
event_seq += 1
748-
yield "tick", SimpleNamespace(
749-
timestamp=timestamp,
750-
timestamp_ns=int(row["exch_ts"]),
751-
event_seq=event_seq,
752-
symbol=symbol,
753-
price=float(row["px"]),
754-
volume=float(row["qty"]),
755-
direction="buy" if (ev & _BUY_EVENT) else "sell",
756-
bid_price=bid_price,
757-
ask_price=ask_price,
758-
bid_volume=bid_qty,
759-
ask_volume=ask_qty,
748+
yield (
749+
"tick",
750+
SimpleNamespace(
751+
timestamp=timestamp,
752+
timestamp_ns=int(row["exch_ts"]),
753+
event_seq=event_seq,
754+
symbol=symbol,
755+
price=float(row["px"]),
756+
volume=float(row["qty"]),
757+
direction="buy" if (ev & _BUY_EVENT) else "sell",
758+
bid_price=bid_price,
759+
ask_price=ask_price,
760+
bid_volume=bid_qty,
761+
ask_volume=ask_qty,
762+
),
760763
)
761764
if ev & _DEPTH_EVENT:
762765
previous_bid_price = bid_price
@@ -772,23 +775,26 @@ def _iter_exchange_market_events(market_data_path, symbol: str) -> Iterator[tupl
772775
if bid_price is None or ask_price is None:
773776
continue
774777
event_seq += 1
775-
yield "orderbook", SimpleNamespace(
776-
timestamp=timestamp,
777-
timestamp_ns=int(row["exch_ts"]),
778-
event_seq=event_seq,
779-
symbol=symbol,
780-
previous_bids=(
781-
[(previous_bid_price, previous_bid_qty)]
782-
if previous_bid_price is not None
783-
else []
784-
),
785-
previous_asks=(
786-
[(previous_ask_price, previous_ask_qty)]
787-
if previous_ask_price is not None
788-
else []
778+
yield (
779+
"orderbook",
780+
SimpleNamespace(
781+
timestamp=timestamp,
782+
timestamp_ns=int(row["exch_ts"]),
783+
event_seq=event_seq,
784+
symbol=symbol,
785+
previous_bids=(
786+
[(previous_bid_price, previous_bid_qty)]
787+
if previous_bid_price is not None
788+
else []
789+
),
790+
previous_asks=(
791+
[(previous_ask_price, previous_ask_qty)]
792+
if previous_ask_price is not None
793+
else []
794+
),
795+
bids=[(bid_price, bid_qty)],
796+
asks=[(ask_price, ask_qty)],
789797
),
790-
bids=[(bid_price, bid_qty)],
791-
asks=[(ask_price, ask_qty)],
792798
)
793799

794800

0 commit comments

Comments
 (0)