Skip to content

Commit 0b84795

Browse files
luiseimanclaude
luiseiman
and
claude
committed
feat: v3.4.1 — backtesting ADR gate rule for trading stack
New rule stacks/trading/rules/backtesting-adr-gate.md captured from real ADR retrospective in tradingview repo: a "Dual Momentum SPY/QQQ/BIL 12m" baseline ADR was declared based on Sharpe delta +0.04 over QQQ B&H. After fixing a look-ahead bug, the delta shrank to +0.02. Computing PSR(QQQ B&H) per Bailey & López de Prado (2012) showed NONE of 9 strategies passed the 0.95 gate — the "best" gave 70% probability of beating B&H, 30% of being worse. Rule: - PSR(benchmark) > 0.95 required to claim "baseline"/"winner"/"supersedes" - DSR (Deflated Sharpe Ratio) required when testing > 5 strategies in same project (multiple-testing correction) - Below threshold: document as alternative, don't use the strong words - Implementation ~50 lines stdlib-only via statistics.NormalDist Generalization: when ranking N options by noisy metric, compute Pr(top genuinely better than alternatives). Below threshold the ranking is decoration. Inbox processing: - accept: psr-gate-baseline-adrs → active/ incorporated_in ['3.4.1'] - reject: tradingview-session-changes (auto-stub) - defer: agent-memory-underused (low-priority, needs more data) stacks/trading/plugin.json bumped to v2.1.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 6671fd2 commit 0b84795

6 files changed

Lines changed: 178 additions & 4 deletions

File tree

‎VERSION‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
3.4.0
1+
3.4.1

‎docs/changelog.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,33 @@
44
>
55
> Historial de versiones. Las entradas usan español/inglés mixto según la evolución del proyecto. Los términos técnicos son universales.
66
7+
## v3.4.1 (2026-04-27)
8+
9+
### New rule — `stacks/trading/rules/backtesting-adr-gate.md`
10+
11+
Captured from a real ADR retrospective in the `tradingview` repo: a "Dual Momentum SPY/QQQ/BIL 12m" strategy was declared the official baseline of the passive-US sleeve based on walk-forward OOS Sharpe 1.08 vs QQQ B&H 1.04 (delta = +0.04) and Calmar 2.78 vs 1.66. After fixing a look-ahead bug in the rebalancer, the OOS metrics deflated to 1.06 vs 1.04. Computing **PSR(QQQ B&H)** per Bailey & López de Prado (2012) for all 9 strategies tested in the repo showed **none passed the 0.95 threshold** — the "best" strategy gave 70% probability of beating B&H, i.e. 30% probability of being worse.
12+
13+
The new rule encodes:
14+
15+
- **PSR(benchmark) > 0.95** required to claim "baseline", "winner", or "supersedes" in any ADR
16+
- **DSR (Deflated Sharpe Ratio)** required when testing > 5 strategies in the same project (multiple-testing correction)
17+
- Below threshold: ADR may document the strategy as alternative, but must not use the strong words
18+
- Implementation: ~50 lines stdlib-only via `statistics.NormalDist`; no scipy needed
19+
20+
Generalization beyond trading: when ranking N options by a noisy metric, compute Pr(top option genuinely better than alternatives). Below threshold, the ranking is decoration — don't anchor decisions on it.
21+
22+
#### Changed
23+
24+
- `stacks/trading/plugin.json`: bumped to v2.1.0, components.rules now lists both rules.
25+
- `practices/active/2026-04-27-psr-gate-baseline-adrs.md`: incorporated_in `['3.4.1']`.
26+
- `metrics.yml`: monitoring entry, error_type=logic.
27+
28+
#### Inbox processing
29+
30+
- 1 accepted (psr-gate-baseline-adrs → above)
31+
- 1 rejected (tradingview-session-changes — auto-stub, summary-only)
32+
- 1 deferred (agent-memory-underused — low-priority, needs more usage data to evaluate)
33+
734
## v3.4.0 (2026-04-26)
835

936
### `/forge watch` sync from CHANGELOG v2.1.92 → v2.1.119

‎practices/active/2026-04-27-psr-gate-baseline-adrs.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
---
2+
id: psr-gate-baseline-adrs
3+
source: tradingview-session
4+
status: active
5+
captured: 2026-04-27
6+
tags: [backtesting, statistics, adr, multiple-testing, deflated-sharpe]
7+
tested_in: [tradingview]
8+
incorporated_in: ['3.4.1']
9+
---
10+
11+
# Aplicar PSR(benchmark) > 0.95 como gate antes de declarar baseline en cualquier ADR
12+
13+
## Observation
14+
15+
In the `tradingview` repo, an ADR (2026-04-23) declared "Dual Momentum
16+
SPY/QQQ/BIL 12m" as the official baseline of the passive-US sleeve based on
17+
walk-forward OOS Sharpe 1.08 vs QQQ B&H 1.04 (delta = +0.04) and Calmar 2.78
18+
vs 1.66.
19+
20+
A look-ahead bug was later found in `app/swing.py::run_dual_momentum_pair`.
21+
After fixing it, the OOS metrics deflated to Sharpe 1.06 vs 1.04. The
22+
backtest in-sample (22y) was unaffected.
23+
24+
We then implemented the **Probabilistic Sharpe Ratio** (Bailey & López de
25+
Prado 2012) and computed `PSR(benchmark = QQQ B&H)` for all 9 strategies
26+
tested in the repo:
27+
28+
| Strategy | SR_y | PSR(QQQ) |
29+
|---------------------|-------|----------|
30+
| Dual Mom 12m (best) | 0.89 | **0.70** |
31+
| RMA top-3 | 0.75 | 0.48 |
32+
| Vol target QQQ | 0.72 | 0.42 |
33+
| QQQ + SMA200 | 0.70 | 0.38 |
34+
35+
**Not a single strategy** in the repo passed `PSR(QQQ) > 0.95`. The "best"
36+
strategy gave us a 70% probability of having a true Sharpe > QQQ — meaning
37+
30% probability of being worse than B&H. That's not a baseline; that's a
38+
coin-flip with extra steps.
39+
40+
## Lesson
41+
42+
Sharpe-ratio deltas of +0.02 to +0.04 (typical "the strategy beats benchmark"
43+
deltas in retail backtests) are **statistically indistinguishable from noise**
44+
when n is in the order of thousands of daily observations. The naked eye sees
45+
a ranking; the math sees a wash.
46+
47+
A baseline ADR is a strong claim — it commits the project to operate the
48+
strategy with discipline, write rebalancing code, monitor it. Making that
49+
commitment based on a 70% confidence is bad epistemics.
50+
51+
## Suggested action
52+
53+
Add to the project rule (template-level, applicable to any backtesting
54+
project):
55+
56+
> Before any ADR declares a strategy "baseline" or "winner" over another
57+
> strategy, compute `PSR(benchmark)` per Bailey & López de Prado 2012:
58+
>
59+
> 1. PSR(benchmark) = Pr(true Sharpe > Sharpe of benchmark)
60+
> 2. Threshold: PSR(benchmark) > 0.95 to claim significance
61+
> 3. If PSR < 0.95, the ADR may still document the strategy as an
62+
> alternative, but cannot call it a "baseline" or claim it "supersedes"
63+
> another option
64+
>
65+
> When testing > 5 strategies in the same project, also report DSR (Deflated
66+
> Sharpe Ratio) using the empirical variance across the trial Sharpe ratios.
67+
> DSR > 0.95 is the harder gate when multiple-testing is a concern.
68+
>
69+
> Implementation reference: `app/metrics.py` (stdlib-only, no scipy needed
70+
> via `statistics.NormalDist`).
71+
72+
## Implementation cost
73+
74+
- Math is ~50 lines (PSR + expected_max_sharpe + DSR), stdlib-only.
75+
- Tests: ~150 lines covering reference values, monotonicity, edge cases.
76+
- One report script that takes the strategy equity curves and produces a
77+
table.
78+
79+
Total: ~3 hours for any new repo. Lifetime savings: avoiding multi-month
80+
commitments to baselines that were never significant to begin with.
81+
82+
## Generalization
83+
84+
This is a research-quality rule applicable to any quant trading repo, but
85+
the deeper principle generalizes:
86+
87+
> When ranking N options based on a noisy metric, compute the probability
88+
> that the top-ranked option is genuinely better than alternatives. If that
89+
> probability < threshold, the ranking is decoration — don't anchor decisions
90+
> on it.
91+
92+
References:
93+
- Bailey, D. H. & López de Prado, M. (2012). *The Sharpe Ratio Efficient Frontier.*
94+
- Bailey, D. H. & López de Prado, M. (2014). *The Deflated Sharpe Ratio:
95+
Correcting for Selection Bias, Backtest Overfitting and Non-Normality.*
96+
- Lo, A. (2002). *The Statistics of Sharpe Ratios.*

‎practices/metrics.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -461,3 +461,14 @@ practices:
461461
status: monitoring
462462
projects_applied: 1
463463
last_checked: 2026-04-26
464+
465+
- id: psr-gate-baseline-adrs
466+
error_targeted: "ADR declares baseline strategy based on Sharpe delta indistinguishable from noise"
467+
error_type: logic
468+
activated: 2026-04-27
469+
recurrence_checks: 0
470+
recurrence_target: 5
471+
recurred: false
472+
status: monitoring
473+
projects_applied: 0
474+
last_checked: 2026-04-27

‎stacks/trading/plugin.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
{
22
"name": "dotforge-stack-trading",
3-
"version": "2.0.0",
4-
"description": "Trading & investment analysis — thesis tracking, earnings watch, screening, catalyst calendar. AR + US markets.",
3+
"version": "2.1.0",
4+
"description": "Trading & investment analysis — thesis tracking, earnings watch, screening, catalyst calendar, backtesting ADR gate. AR + US markets.",
55
"parent": "dotforge",
66
"components": {
7-
"rules": ["rules/trading.md"],
7+
"rules": ["rules/trading.md", "rules/backtesting-adr-gate.md"],
88
"settings": "settings.json.partial",
99
"skills": [
1010
"skills/thesis-tracker/SKILL.md",

‎stacks/trading/rules/backtesting-adr-gate.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
globs: "**/backtest*,**/strategy*,**/*adr*,**/decisions/**,**/metrics.py,**/sharpe*"
3+
domain: trading
4+
last_verified: 2026-04-27
5+
---
6+
7+
# Backtesting ADR Gate — Statistical Significance Before Baseline Claims
8+
9+
## Rule
10+
11+
Before any ADR declares a strategy a "baseline", "winner", or "supersedes" another option, compute `PSR(benchmark)` per Bailey & López de Prado (2012) and gate the claim:
12+
13+
1. **PSR(benchmark) = Pr(true Sharpe > Sharpe of benchmark)**
14+
2. **Threshold: PSR(benchmark) > 0.95** to claim significance
15+
3. If **PSR < 0.95**, the ADR may document the strategy as an alternative — but cannot use the words *baseline*, *winner*, or *supersedes*
16+
17+
When testing **> 5 strategies** in the same project, also report **DSR (Deflated Sharpe Ratio)** using the empirical variance across the trial Sharpe ratios. DSR > 0.95 is the harder gate when multiple-testing is a concern.
18+
19+
## Why
20+
21+
Sharpe-ratio deltas of +0.02 to +0.04 (typical "the strategy beats benchmark" deltas in retail backtests with daily data) are statistically indistinguishable from noise when n is in the order of thousands of observations. The eye sees a ranking; the math sees a wash.
22+
23+
A baseline ADR is a strong claim — it commits the project to operate the strategy with discipline. Making that commitment based on 70% confidence is bad epistemics.
24+
25+
## Implementation
26+
27+
- Math: ~50 lines (PSR + expected_max_sharpe + DSR), stdlib-only via `statistics.NormalDist`. No scipy needed.
28+
- Tests: ~150 lines covering reference values, monotonicity, edge cases.
29+
- One report script that takes equity curves and produces a PSR/DSR table.
30+
- Reference implementation: `app/metrics.py` in the originating repo.
31+
32+
## Generalization
33+
34+
The deeper principle applies beyond trading: when ranking N options based on a noisy metric, compute the probability that the top-ranked option is genuinely better than the alternatives. If that probability < threshold, the ranking is decoration — don't anchor decisions on it.
35+
36+
## References
37+
38+
- Bailey, D. H. & López de Prado, M. (2012). *The Sharpe Ratio Efficient Frontier.*
39+
- Bailey, D. H. & López de Prado, M. (2014). *The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality.*
40+
- Lo, A. (2002). *The Statistics of Sharpe Ratios.*

0 commit comments

Comments
 (0)