Evidence correction, July 30, 2026: the historical horizon comparison and other estimates built from the pre-correction repaired price panel are invalid. A subsequent framework audit also superseded the intermediate corrected-price replay by separating tree early stopping from inner evaluation and retuning all four models on common folds. The 5/10/20-session live comparison remains active, but 20 sessions is no longer considered historically established. See the framework-clean comparison.
Status date: 2026-07-30
The original matched zero-cost forecast-horizon comparison was invalidated by
the July 30 price-repair audit. The project therefore retains 5, 10, and 20
sessions as prospective comparison arms without claiming a historical winner.
The one-session monitor remains a legacy diagnostic because it tests a
different economic horizon and creates excessive turnover. The paper-live
runner is implemented
as python -m src.cli run-live-forward-20d; it creates daily overlapping
forecast cohorts and separately tracked horizon-matched strategies. Five-day
arms refit and rebalance every 5 sessions, ten-day arms every 10, and
twenty-day arms every 20. The first refreshed v1 run was frozen on 2026-07-28
under the earlier shared-20-session clock after live market and news refresh,
strict FinBERT processing, and independent purged Optuna searches. It remains
an immutable operational record and has no matured outcomes. The corrected v2
history begins in reports/live_forward_horizon_matched_v2/. See
experiments/2026-07-28-live-horizon-decision.md
and
design/LIVE_FORWARD_20_SESSION_DRAFT.md.
The first-run audit is
experiments/2026-07-28-live-forward-20d-first-run.md.
A six-session July 20–28 ablation now audits the assumption that every
rebalance should fully replace the top 40. Freezing the July 20 names improved
Ridge by 3.93 percentage points net, while daily XGBoost replacement improved
by only 0.53 point net after costs. Daily turnover was 615.6% for Ridge and
661.6% for XGBoost including initial purchases, compared with approximately
100% for frozen buy-and-hold portfolios. This supports eliminating daily full
replacement but does not validate a permanent freeze or a buffer from only six
sessions. The horizon-matched v2 full-refresh policy is therefore explicitly
under audit; the next preregistered comparison is full refresh versus frozen
holdings versus model-specific rank- and score-margin buffers. See
experiments/2026-07-29-live-rebalancing-ablation.md.
The controlled follow-up protocol is documented in
design/REBALANCE_BUFFER_EXPERIMENT.md.
A corrected preliminary precision and regime study did not support changing the live policy.
The fixed precision gate lowered positive precision and lost money, while the regime policy
reduced risk mainly by moving almost entirely to cash in weak regimes. A lookahead diagnostic
shows that severe-loss avoidance has substantial economic value, but the simple tail model's
recall collapsed across later folds. See
experiments/2026-07-30-precision-regime-policy-preliminary.md.
A separate corrected false-positive audit confirms that the error rate was not caused by the
price-data defect. Ridge still selected negative-return stocks 46.35% of the time and XGBoost
48.41%. Realized broad-market direction explains much of the variation, while the causal SPY
regime mainly identifies higher severe-loss risk. Individual stocks dominate the worst weeks but
reverse only about 7% to 8% of losing weeks. See
experiments/2026-07-30-corrected-false-positive-audit.md.
The completed timeline shows that sign uncertainty persists throughout the corrected backtest,
while severe losses cluster in stressed periods. Ridge's 2020 H1 and 2022 H1 observations contain
70.43% of its severe events and 75.79% of severe-loss drag. A nested, fully invested A/B treatment
then reduced direct severe-loss contribution by about 8.1% relative, but did not establish CVaR
improvement or return non-inferiority. The active treatment also required a 30% veto budget and
rejected many future winners. The result is inconclusive and remains disabled. See
experiments/2026-07-30-backtest-tail-risk-timeline.md
and
experiments/2026-07-30-severe-loss-risk-control-ab.md.
The framework-clean July 30 comparison supports a narrower project direction. Ridge is the strongest current historical model: mean rank IC is 0.02058, sector-capped top-40 excess is 0.2227% per weekly observation, and its cost-adjusted portfolio produces 8.22% CAGR, 0.443 Sharpe, and -34.29% maximum drawdown. The zero-cost sensitivity produces 14.99% CAGR and 0.693 Sharpe. Full-investment SPY produces 10.73% CAGR, 0.583 Sharpe, and -31.62% maximum drawdown on the same next-open intervals. This is development evidence, not confirmed alpha.
V4 has now solved the main execution defect in development. The frozen 40-name equal-weight hold-zone reduced average weekly turnover to 4.94%, average orders to 2.87, and annual modeled cost to approximately 0.33%. Development CAGR was 12.19%, Sharpe 0.71, and maximum drawdown -27.40%. The final Ridge model, parameters, 40 features, holding rule, and execution policy are frozen before confirmation.
The original full 2024-2025 contract remains blocked because the public FinMultiTime archive ends on 2025-03-28. A shortened 2024 through 2025 Q1 public confirmation was frozen and run once. Inputs covered 505 of 527 point-in-time stocks and 94.8% of business-day news dates; unavailable symbols were excluded before scoring without substitution. The frozen portfolio returned 11.34% CAGR versus 11.05% for 70% SPY plus 30% cash, but failed its permutation and bootstrap gates. The later strict weekly NLP and pair experiments also failed their all-gates-required contracts. Slow sentiment and topics improved development point estimates but were not distinguishable from timestamp-shuffled signals (p=0.366). Pair timing beat circular shifts, but the pair sleeve's confidence interval crossed zero and its random-pair p-value was 0.218. Both remain disabled, and the 2026-H1 lockbox remains sealed.
The project has progressed from a hindsight-selected ten-stock CVaR backtest to a leakage-aware, point-in-time ranking experiment. The current development path ranks up to 250 liquid historical S&P 500 members, predicts five-session sector-neutral residual-return ranks, performs nested purged model and feature selection, and holds a nested-selected 20–50-stock portfolio.
Sparse selection is encouraging in development: net Sharpe was 0.498 versus a median 0.063 from 100 shuffled selections (p = 0.0099). The portfolio produced 10.98% CAGR from 2020 through 2023 H1, but volatility was 29.06%, maximum drawdown was −38.89%, and modeled execution consumed about 6% annually. Small bounded tilts did not improve materially on equal weight among model-selected names. Selection appears useful; sizing and turnover remain weak.
Slow sentiment also produced a meaningful development uplift. The sparse pipeline earned 5.47% CAGR without NLP features and 10.98% with them. Paired annualized incremental return was +6.14%, with a positive 90% block-bootstrap interval but a 95% interval that still included zero. A small exact-time tactical subset was tested separately and failed its development purchase gate.
No market-beating or production-alpha claim is justified. The strongest final outcome is a modular, leakage-aware research platform with corrected execution, permanent placebos, explicit activation gates, and documented negative results. The frozen 40-name portfolio remains a research reference; every unconfirmed incremental sleeve has zero default weight.
For GitHub cleanup, the detailed experiment trail belongs on archive-experiments. The main
branch should keep the reusable pipeline, current rank-model diagnostics, concise documentation,
and final-facing limitations.
- Incremental yfinance caches reuse history and fetch only missing tails.
- Real FNSPID, RSS, NewsAPI, and Alpha Vantage records are normalized and deduplicated.
- Reddit and X are optional and degrade gracefully when access is unavailable.
- FinBERT and lexical fallback paths build eight slow ticker-day NLP features.
- The target matches the weekly holding horizon: next open through five complete sessions.
- Point-in-time membership and lagged liquidity prevent present-constituent substitution.
- Nested purged Ridge/XGBoost selection persists every outer-OOS score and fold decision.
- Portfolio breadth is chosen inside inner validation from 20, 30, 40, and 50 names.
- Permanent shuffled-selection tests use identical costs and execution.
- Fixed fees, spread, slippage, turnover, and order counts are attributed separately.
- Correlation diagnostics are separated from cointegration and pair tradability.
- Tactical sentiment fails closed when exact publication times are absent.
- Public ZIP-range imports retrieve only requested FinMultiTime members and resume from cache.
- FinBERT post scoring is deduplicated, sharded, resumable, and fails closed when the requested model is unavailable.
Model rows are net of the declared costs. SPY is a full-investment index benchmark without strategy costs.
| Portfolio | Cumulative return | CAGR | Sharpe | Max drawdown |
|---|---|---|---|---|
| Ridge | 31.83% | 8.22% | 0.443 | -34.29% |
| Ordinary LightGBM | 2.36% | 0.67% | 0.134 | -34.56% |
| XGBoost | 7.41% | 2.06% | 0.202 | -31.24% |
| LambdaMART | 24.11% | 6.37% | 0.370 | -38.80% |
| SPY index | 42.89% | 10.73% | 0.583 | -31.62% |
Ridge is the strongest corrected historical model, but it trails SPY after costs and has no confirmed active-return advantage. In the zero-cost sensitivity, Ridge reaches 63.04% cumulative return, 14.99% CAGR, 0.693 Sharpe, and -33.80% maximum drawdown.
Model-level diagnostics are more conservative than portfolio returns:
| Diagnostic | Value |
|---|---|
| Ridge rank IC | 0.02058 |
| Ridge positive-IC rate | 57.14% |
| Ridge unconstrained top-40 excess | 0.1539% |
| Ridge sector-capped top-40 excess | 0.2227% |
| Ridge positive-return precision | 53.68% |
| Ridge realized top-40 precision | 17.95% |
The ML winner-selection idea remains promising, but the prediction edge is not confirmed and
requires a future untouched confirmation period. The current evidence record is
in experiments/2026-07-30-framework-clean-rank-challengers.md.
| Portfolio | CAGR | Sharpe | Max drawdown |
|---|---|---|---|
| 250-name residual-rank tilts, 70% invested | 7.50% | 0.53 | −24.84% |
| 70% SPY + 30% cash | 6.76% | 0.49 | −26.02% |
| Fully invested SPY | 8.96% | 0.49 | −35.42% |
V2 reduced drawdown through diversification and cash, but failed its shuffled-forecast test (p = 0.327). Its 250 positions also made the 25 bp trade threshold almost as large as a base position, motivating sparse v3.
| Portfolio | CAGR | Sharpe | Volatility | Max drawdown |
|---|---|---|---|---|
| Model selection + bounded tilts | 10.98% | 0.50 | 29.06% | −38.89% |
| Same selected names, equal weight | 10.95% | 0.50 | 28.92% | −38.90% |
| Same selected names, inverse volatility | 16.38% | 0.73 | 25.01% | −33.57% |
Folds selected [20, 20, 50, 20, 20, 40, 50]. Model Sharpe exceeded every one of the 100
shuffled paths, producing the minimum attainable one-sided p-value of 0.0099. However, the inverse-volatility row still inherits
ML-selected names and cannot be treated as an independent baseline.
The portfolio averaged 96.9% weekly turnover and 47.2 orders per week. Annual variable cost was 4.03% and annual fixed-fee drag was 1.97%. A strategy needing roughly 6% gross annual return just to fund execution is not production ready.
| Arm | CAGR | Sharpe | Max drawdown |
|---|---|---|---|
No nlp_ features |
5.47% | 0.34 | −34.30% |
| Slow NLP features allowed | 10.98% | 0.50 | −38.89% |
- Annualized incremental return: +6.14%.
- Incremental information ratio: 0.88.
- 90% moving-block interval: +0.65% to +12.82%.
- 95% moving-block interval: −0.23% to +14.15%.
- Positive bootstrap draws: 97.1% of 5,000.
Seven NLP variables appeared in selected models. The result is suggestive, consistent with the earlier one-star experiment, but it measures the total pipeline effect: the two arms may choose different models, features, holdings, and K. Only 27 ticker identifiers have complete NLP feature histories.
The tactical overlay was not run. The earlier A/B input had only 167 non-midnight timestamps among 47,198 FNSPID posts; after the expanded import, only 2,589 of 235,993 posts have apparent non-midnight timestamps. Date-only records cannot establish availability before a close decision.
A development-only matched event study audited those 2,589 records and excluded 93 corrupt 2023 batch timestamps. The remaining January-June 2020 sample produced 935 matched direct-ticker FinBERT signals across 62 entry dates. At the primary five-session horizon, net event return was -43.72 basis points and matched incremental return was -44.62 basis points, with a date-clustered 95% interval of [-92.55, +5.68] basis points. Timing p=0.978 and sign p=0.949. All four data-purchase gates failed; this exact next-open strategy does not justify a large data purchase.
The follow-up weekly slow-NLP test removes the main confound in the earlier one-star result. It reuses the same 45,375 numeric-only OOS rows, numeric score, 40-name portfolio, and cost model in every arm; only a 10% bounded NLP rank contribution changes. The numeric baseline earned 7.22% CAGR and 0.46 Sharpe. Sentiment plus point-in-time NMF topics earned 9.10% CAGR and 0.55 Sharpe, an annualized increment of +1.71%, but its 95% block-bootstrap interval was [-1.89%, +5.31%]. The within-ticker timestamp-shuffle p-value was 0.366 and combined row coverage was 12.71%. The stricter result therefore fails the robustness gate and does not confirm that NLP caused the uplift.
The legacy empty pairs.json came from the old ten-stock scan. It did not describe v2. For the
exact 250-name eligible universe on the final 2023-06-30 OOS decision date, the trailing
252-session audit contained all 31,125 pairwise relationships:
- 900 pairs had
|correlation| < 0.10; - 126 were negatively correlated;
- 455 had
|correlation| >= 0.70; - median correlation was 0.357.
A broader archive diagnostic across 563 stock and sector-ETF series found 9,307 near-zero, 1,774 negative, and 622 high-absolute-correlation pairs among 84,666 finite relationships.
That established candidate diversity, not tradable pairs. Point-in-time cointegration, rolling stability, liquidity, and execution costs were still untested at that stage.
The preregistered v5 development experiment has now tested the first five items using 42 monthly formations, effective-dated membership, a top-250 liquidity screen, same-sector cointegration, subwindow stability, next-open execution, and full placebo portfolios. It selected 2.81 pairs per formation on average. The sleeve earned +0.038% annualized with a 95% interval of [-0.186%, +0.270%]. Timing passed its circular-shift test (p=0.0099), but the random-pair test failed (p=0.218). Core CAGR moved from 12.19% to only 12.23%. Five of seven gates passed; because all were required, the pair sleeve remains disabled. Structural-event pausing remains untested due to missing reliable point-in-time event scope labels.
| Portfolio | CAGR | Sharpe | Volatility | Max drawdown |
|---|---|---|---|---|
| Frozen 40-name v4 | 11.34% | 1.11 | 10.19% | -5.73% |
| 70% SPY + 30% cash | 11.05% | 1.14 | 9.63% | -6.26% |
Rank IC was +0.016 and annualized arithmetic active return was +0.31%. The 95% moving-block interval was -12.79% to +8.93%, and the within-date shuffled-score p-value was 0.198. Positive point estimates passed two gates, but the permutation and confidence-interval gates failed. This period is now observed and cannot be reused as untouched confirmation after tuning.
The data layer now caches the official daily market, SMB, HML, and risk-free research factors. Trailing stock betas use only information available through each weekly decision date. A frozen fixed-Ridge comparison then changed only the target on 77,569 common rows and 182 OOS weeks.
| Target arm | CAGR | Sharpe | Max drawdown | Shuffle p-value |
|---|---|---|---|---|
| Current sector-residual rank | 5.34% | 0.38 | -27.68% | 0.881 |
| FF3 plus sector-residual rank | 8.06% | 0.54 | -25.96% | 0.366 |
The FF3 arm's annualized increment was +2.38%, but its 95% interval was -1.26% to +6.54%. Annualized FF3 alpha was +1.18% with p=0.768. Three of five predeclared replacement gates failed, so the existing target remains frozen. Factor attribution is retained because it reveals whether raw portfolio performance is explained by broad market, size, and value exposures. See the v6 experiment log.
- Post-development design: sparse breadth and the 25 bp threshold were introduced after inspecting earlier results.
- Failed shortened confirmation: positive point estimates did not beat shuffled selections convincingly, and the active-return confidence interval crossed zero widely.
- Unstable K: folds chose 20, 40, and 50 holdings.
- Execution burden: roughly 6% annual modeled drag and near-100% weekly turnover.
- Weak sizing layer: bounded tilts add effectively no return over equal weight on selected names.
- Historical coverage: FNSPID covers 552 of 637 members; 85 are excluded.
- Sector proxies: monthly ETF correlations are not official point-in-time GICS labels.
- Sentiment breadth: public confirmation NLP features cover 247 tickers, not the entire point-in-time universe; historical development coverage is narrower.
- Event timing: date-only news cannot support a tactical or intraday claim.
- Regime evidence: current recession/crisis analysis belongs to a superseded pipeline.
- Pair sleeve: stability and timing tests passed, but its confidence bound and random-pair placebo failed; the economically small sleeve remains disabled.
- Execution realism: static costs omit participation, dynamic spreads, capacity, taxes, borrow, and partial fills.
- Compute and caching: FinBERT post scores and nested fold predictions are not fully cached.
- Research lockbox: 2026-H1 remains sealed and must not become another tuning sample.
- Factor-data vintage: the public FF3 research file is current and revisable rather than a historically archived vintage; the alternative target also failed its replacement gates.
- Legacy sequence models: LSTM and Transformer windows are built from consecutive panel rows without enforcing ticker continuity. They are excluded from validated results, and the current project will not expand into a sequence-pipeline reconstruction.
- Weekly smart-universe mismatch: the first transparent 120-name selector had -0.0123 rank IC against the five-session target, changed 16.6 names weekly, and failed four of five gates. It is implemented for audit but inactive.
- Tail-risk guardrail: the first fully invested nested A/B treatment reduced direct severe-loss contribution, but CVaR improvement remained uncertain and the return non-inferiority interval failed. It remains inactive.
The final reference system uses a staged, conservative process:
- screen the point-in-time S&P 500 for liquidity and data quality;
- narrow it with transparent residual-momentum, quality, stability, and cost-aware ranks;
- retain FinBERT, point-in-time topics, and structured LLM schemas as research adapters with zero portfolio authority until they pass new frozen tests;
- use regularized ML only to combine and shrink structured signals;
- construct a diversified 30-40-name hold-zone portfolio;
- keep pair trading at zero after its failed confidence-bound and random-pair gates.
This disposition is summarized in the
final system proposal. The current NLP pipeline has
embeddings for semantic relevance, fixed event categories, and point-in-time NMF and LDA topic
adapters. The
cached FNSPID archive was re-imported for all 552 available point-in-time members: 235,993 stored
posts cover 449 source tickers. BERTopic, dynamic embedding topics, and an LLM stock selector
remain future work. The earlier roadmap remains available as
design/NEXT_RESEARCH_DIRECTION.md.
The expanded NMF baseline completed 71 monthly prior-only refits and produced 97,513 active
ticker-date rows. After the required lag and eligible-universe join, 52,588 rows and 313 tickers
are covered (13.95%). This is far better than the original 1.18%, but it fails the predeclared 50%
eligible-row gate. NMF median rolling alignment is 0.905 and median topic-word diversity is 0.91.
Batch and bounded online LDA both exceeded the fixed 15-minute full-run ceiling. Topic-count
tuning, BERTopic, the topic/no-topic ML A/B, and the shuffled placebo were stopped at the coverage
gate; no forecasting or return uplift is claimed. See
experiments/2026-07-16-topic-features-v5.md.
- Mark 2024 through 2025 Q1 as observed and do not tune v4 against it.
- Keep the pair sleeve at zero because the declared activation gate failed.
- Make any model or sentiment redesign on a separate development sample.
- Freeze a new contract before acquiring or waiting for a genuinely untouched future period.
- Obtain second-source prices for unavailable historical symbols and genuine publication times. University WRDS/CRSP access is unavailable in the current project phase, so CRSP-grade membership, permanent identifiers, and delisting returns are explicitly deferred. Public-data sensitivity results must not be described as a complete survivorship-bias correction.
- Validate effective-dated S&P 500 membership and official effective-dated sectors.
- Freeze one model, feature list, K or K rule, execution policy, and sentiment policy.
- Do not rerun the observed period as confirmation after changing the system.
- Expand historical NLP coverage across the full point-in-time universe.
- Cache post-level entity, FinBERT, event, and model-version outputs.
- Add source-coverage and missingness features separately from neutral sentiment.
- Acquire genuine publication timestamps before enabling the tactical overlay.
- With valid timestamps, compare no sentiment, slow weekly state, timestamp-shuffled state, and next-open tactical triggers under identical costs.
- Add point-in-time topic discovery with novelty, acceleration, dissemination, persistence, and macro/sector/company scope.
- Compare FinBERT, NMF, LDA, embedding topics, structured LLM events, and time-shuffled controls under identical holdings and costs.
- Add overnight-gap, recent-drawdown, liquidity-deterioration, earnings-proximity, SPY-stress, and timestamp-safe event features to the severe-loss candidate panel.
- Compare smaller veto budgets and optimize tail precision, recall, rejected-winner cost, CVaR, drawdown, turnover, and return non-inferiority under the same fully invested Ridge control.
- Compare transparent rules, class weighting, focal-style loss, conditional quantiles, and calibrated ensembles using purged temporal selection and separate calibration blocks.
- Require stable stressed-period and ordinary-period results before any prospective shadow A/B activation.
- Evaluate inverse-volatility and factor-shrinkage sizing independently from ML selection.
- Add a volatility target and crisis exposure rule fixed before evaluation.
- Do not retune the failed pair contract on this observed development sample; any redesign needs a separate period and historical security-level spread and borrow inputs.
- Acquire reliable event scope labels before testing pair pauses around high-severity company events rather than assuming immediate mean reversion.
- Cache fold-level feature rankings, fitted predictions, and completed permutation paths.
- Add artifact hashes, data coverage, date ranges, environment versions, and resolved manifests.
- Make the one-call pipeline and documented CLI emit identical artifacts.
- Add progress reporting and resume support for long nested experiments.
Before opening the lockbox, require all of the following on confirmation data:
- positive rank IC;
- positive net active return under frozen costs;
- shuffled-selection p-value no greater than 0.10;
- active-return confidence interval with a non-negative lower bound;
- stable performance across reasonable cost and turnover assumptions;
- acceptable drawdown and turnover relative to SPY and the selected-name controls;
- no parameter, K, feature, or execution changes after observing confirmation results.
- Main workflow:
../README.md - Methodology equations:
METHODOLOGY_EQUATIONS.md - Sparse v3:
experiments/2026-07-14-sparse-rank-v3.md - Residual-rank v2:
experiments/2026-07-14-residual-rank-v2.md - Sentiment event design:
design/SENTIMENT_EVENT_OVERLAY.md - Final system proposal:
design/FINAL_SYSTEM_PROPOSAL.md - Historical research roadmap:
design/NEXT_RESEARCH_DIRECTION.md - Topic-features v5 baseline:
experiments/2026-07-16-topic-features-v5.md - Fama-French v6 study:
experiments/2026-07-16-fama-french-v6.md - Smart-universe v7:
experiments/2026-07-16-smart-universe-v7.md - Legacy revealing controls:
experiments/2026-07-14-revealing-controls.md