A leakage-aware research platform for cross-sectional stock selection, financial-news NLP, realistic execution, and paper-live monitoring.
The project asks a narrow investment question:
Can machine learning rank future relative winners more reliably than it can predict calibrated stock returns, and does timestamp-valid financial sentiment add useful information?
The current evidence supports continuing that research direction. It does not establish production alpha.
Status date: July 31, 2026
Next experiment: matched ten-year forecast-horizon selection
Universe: point-in-time S&P 500 membership, liquid top 250, sector-capped top 40
Models: Ridge baseline and independently tuned XGBoost challenger
This project did not begin as the current ranking system. It became one by repeatedly challenging attractive backtests and keeping the failed tests.
- We started broad. The original system combined direct return forecasting, CVaR optimization, pair trading, technical features, and financial-news sentiment on a small present-day stock universe. It looked complete, but the architecture was more mature than the evidence. See the original full architecture and experiment trail
- The first audit changed the standard. Fill timing, transaction-cost accounting, label maturity, and optimizer safeguards required correction. That invalidated an early headline result and established the project's fail-closed research discipline. Read the first canonical rerun and corrections
- The question shifted from "what return?" to "which stocks?" Weak return calibration but better cross-sectional ordering led to sector-neutral rank targets, point-in-time membership, nested purged validation, and top-k winner selection. Read the historical rank-40 experiment, later invalidated by the price audit
- Promising additions had to earn activation. Sentiment improved several development point estimates, while smart-universe filtering, survival ML, pair trading, topics, and ensembles either failed gates or remained inconclusive. Negative results were retained rather than blended into an ever-larger strategy. Explore the dated experiment archive
- The live horizon was redesigned, then challenged over a decade. The one-session monitor accumulated observations quickly but tested the wrong economic horizon and created extreme turnover. A corrected 20-session study then used 520 weekly OOS decisions and exact overlapping sleeves. It found no active portfolio that beat SPY, so 20 sessions remains a candidate rather than an established optimum. Read the ten-year 20-session result
- The project now separates horizon selection from model and risk-policy selection. The next experiment freezes Ridge and its continuous rank target while comparing 5, 10, 20, 40, and 60 sessions on common corrected observations. The paper-live 5/10/20 monitor continues collecting prospective evidence, while the low-precision tail veto remains disabled. Review the horizon-selection plan
The long research history remains available through the links above; the rest of this README describes only the final active system and its current evidence.
The July 20 and July 27 tables were invalidated by the price-repair audit. A second audit found that the shared tuning path also reused an inner evaluation fold for tree early stopping. The comparison below replaces those results. All four models use the corrected data, the same seven outer folds, fold-local feature selection, and, for tree models, a separate purged early-stopping split inside each training fold.
The evaluation covers 182 weekly observations from 2020-01-10 through 2023-06-30. Each model arm uses a 70%-invested, sector-capped top-40 portfolio with 8 basis points of one-way variable cost and a $10 fixed fee per trade, assuming $1 million of starting capital. SPY is a full-investment, dividend-adjusted index benchmark on the same next-open holding intervals and does not inherit the strategies' transaction costs.
| Model and portfolio | Model construction | Mean rank IC | Cumulative return | CAGR | Sharpe (net for models) | Maximum drawdown |
|---|---|---|---|---|---|---|
| Ridge | Linear rank model | 0.02058 | 31.83% | 8.22% | 0.443 | -34.29% |
| Ordinary LightGBM | Tree regressor on rank labels | 0.00648 | 2.36% | 0.67% | 0.134 | -34.56% |
| XGBoost ranker | Nonlinear pairwise rank model | 0.01520 | 7.41% | 2.06% | 0.202 | -31.24% |
| LambdaMART | Date-grouped learning-to-rank model | 0.00241 | 24.11% | 6.37% | 0.370 | -38.80% |
| SPY index | Full-investment market benchmark | N/A | 42.89% | 10.73% | 0.583 | -31.62% |
Zero-cost sensitivity is shown separately as an optimistic upper bound. It removes both proportional and fixed execution costs but does not change the holdings or rebalance schedule.
| Portfolio | Zero-cost cumulative return | Zero-cost CAGR | Zero-cost Sharpe | Zero-cost maximum drawdown |
|---|---|---|---|---|
| Ridge | 63.04% | 14.99% | 0.693 | -33.80% |
| Ordinary LightGBM | 31.90% | 8.23% | 0.489 | -33.59% |
| XGBoost ranker | 30.50% | 7.90% | 0.481 | -30.39% |
| LambdaMART | 41.85% | 10.50% | 0.520 | -38.18% |
| SPY index | 42.89% | 10.73% | 0.583 | -31.62% |
Ridge remains the strongest corrected historical model, but it does not beat SPY after the declared costs. Ridge does beat SPY in the zero-cost sensitivity, but that is a research upper bound rather than an implementation claim. LambdaMART turns over less and identifies more eventual top-40 names, but it has lower positive precision, more severe losses, worse CVaR, and a deeper drawdown. None of these development results is a live-return claim, and a future untouched confirmation period is still required.
The Ridge Sharpe of 0.443 is the cost-adjusted value. The same weekly
portfolio returns have a gross Sharpe of 0.693; modeled execution costs
average 0.1168% per week: 0.0462% fixed and 0.0706% proportional. More
capital makes the fixed component smaller, but it does not remove the
proportional component and may increase market impact. Removing only the fixed
fee gives Ridge a 10.85% CAGR and 0.541 Sharpe before any market-impact
allowance. Sharpe values near 1 in older records came from either the
invalidated pre-price-repair panel or a shorter 64-week confirmation window.
The earlier 0.61 corrected-price replay was also superseded by this fully
retuned, common-fold comparison.
Read the framework-clean comparison
The next audit extended the corrected framework to 520 weekly OOS decisions from December 2013 through November 2023 and reconstructed daily returns from exact overlapping 20-session sleeves. The table below reports the zero-cost sensitivity so model behavior is visible without the fixed execution-cost assumption.
| Portfolio | Mean rank IC | Gross cumulative return | Gross CAGR | Gross Sharpe | Gross maximum drawdown |
|---|---|---|---|---|---|
| Ridge return | 0.0018 | 213.39% | 12.09% | 0.619 | -46.44% |
| Ridge rank | 0.0123 | 195.28% | 11.42% | 0.590 | -49.58% |
| XGBoost rank | 0.0128 | 191.46% | 11.28% | 0.596 | -44.28% |
| LambdaMART, five grades | -0.0041 | 160.17% | 10.02% | 0.481 | -50.18% |
| SPY index | N/A | 219.13% | 12.29% | 0.774 | -32.05% |
Ranking labels improve Ridge's mean rank IC, but not its portfolio return or risk. Five-grade LambdaMART is rejected in its current form. None of the active portfolios beats SPY even before transaction costs. Because every arm above uses a 20-session target, this experiment cannot determine whether moving from 5 to 20 sessions caused earlier improvements. That unresolved attribution is why matched horizon selection is now the first priority.
Read the ten-year ranking and tail-control experiment
The historical system contained many experimental sleeves. Its full architecture is preserved on the archive-experiments branch. The active system keeps only the modules needed to test the current winner-selection and sentiment hypotheses:
flowchart TD
subgraph INGEST["Market, membership, and news ingestion"]
MARKET["Adjusted OHLCV<br/>SPY and sector ETFs<br/>incremental local cache"]
MEMBERSHIP["Effective-dated S&P 500 membership<br/>sector history and corporate events"]
NEWS["Timestamped financial news<br/>persistent deduplication<br/>source and freshness audit"]
end
subgraph PANEL["Leakage-safe research panel"]
UNIVERSE["Lagged 63-session liquidity<br/>point-in-time top 250"]
NLP["Entity linking and FinBERT<br/>event tags and recency<br/>15:45 ET availability clock"]
FEATURES["Technical, residual, risk,<br/>sector-relative, sentiment,<br/>event and recency features"]
TARGETS["Sector-neutral rank targets<br/>5, 10, and 20 sessions<br/>next-open to open"]
end
subgraph LEARN["Horizon-matched model development"]
VALIDATION["Rolling temporal folds<br/>mature labels and horizon purge"]
SELECTION["Fold-local feature ranking<br/>feature-count and Optuna search<br/>XGBoost early stopping"]
MODELS["Independent models<br/>Ridge baseline<br/>XGBoost challenger"]
end
subgraph PORTFOLIO["Portfolio construction and execution"]
SCORES["Score the eligible<br/>cross-section"]
TOP40["Sector-capped top 40<br/>100% research exposure<br/>equal weight"]
CLOCKS["Matched refit and rebalance<br/>5 to 5 / 10 to 10 / 20 to 20"]
TACTICAL["Shadow tactical sentiment event<br/>next-open routing only<br/>same close disabled by default"]
BUFFER["Rebalance buffer<br/>next experiment, not active"]
end
subgraph EVIDENCE["Backtesting, evidence, and live monitoring"]
BACKTEST["Point-in-time walk-forward backtest<br/>costs, turnover, benchmark,<br/>return and drawdown"]
COHORTS["Daily immutable forecasts<br/>partial marks and matched maturity"]
TACTICALLOG["Separate tactical ledger<br/>no credit to strategic returns"]
DIAGNOSTICS["IC, quantiles, distributions,<br/>SHAP and shuffled controls"]
DASHBOARD["Two-page live dashboard<br/>stock-level monitor<br/>horizon comparison"]
end
MARKET --> UNIVERSE
MEMBERSHIP --> UNIVERSE
NEWS --> NLP
MARKET --> FEATURES
UNIVERSE --> FEATURES
NLP --> FEATURES
FEATURES --> TARGETS --> VALIDATION --> SELECTION --> MODELS
FEATURES --> SCORES
MODELS --> SCORES --> TOP40 --> CLOCKS
NLP --> TACTICAL --> TACTICALLOG
BUFFER -. "under audit" .-> TOP40
CLOCKS --> BACKTEST
CLOCKS --> COHORTS
BACKTEST --> DIAGNOSTICS
TACTICALLOG --> DIAGNOSTICS
COHORTS --> DIAGNOSTICS --> DASHBOARD
| Module | Active code | Responsibility |
|---|---|---|
| Market, membership, and news ingestion | src/data, src/nlp |
Refresh adjusted market history and financial news; retain effective-dated membership and sectors; deduplicate records; preserve source, coverage, timestamp, and freshness audits |
| Leakage-safe research panel | src/features, src/targets |
Form the lagged liquid universe; run strict local FinBERT and event tagging; build technical, cross-sectional, residual, risk, sentiment, and recency features; create horizon-matched rank targets |
| Horizon-matched model development | src/models/rank_models.py, src/experiments/live_forward_test.py |
Purge overlapping labels; rank features inside temporal folds; tune feature count and hyperparameters; apply early stopping; keep Ridge and XGBoost selection independent |
| Portfolio construction and execution | src/portfolio/sparse_rank.py, src/experiments/live_forward_20d.py |
Score the eligible top 250; enforce the sector cap; freeze equal-weight top 40; operate matched refit/rebalance clocks; route qualifying tactical events to the next open; keep same-close routing disabled unless quote and broker-cutoff controls are supplied |
| Backtesting, evidence, and live monitoring | src/backtest, src/experiments |
Simulate next-open execution and costs; mature immutable cohorts; keep tactical attribution separate from strategic returns; report return, IC, turnover, drawdown, distributions, SHAP, and shuffled controls; publish the two-page dashboard |
| Finding | Current evidence | Boundary |
|---|---|---|
| Winner selection is more promising than raw-return prediction | Framework-clean Ridge rank IC 0.02058 and sector-capped top-40 excess 0.2227% per weekly observation |
Average signal is thin, unstable by year, and still requires a future untouched confirmation period |
| Sentiment appears useful | Technical-plus-sentiment point estimates exceeded technical-only at 5, 10, and 20 sessions | The comparison does not yet prove a causal NLP increment; prior robustness gates were inconclusive |
| The best horizon is unresolved | The corrected ten-year study validates 20-session accounting but contains no competing horizon; the earlier sweep favored 20 sessions only by statistically uncertain point estimates | Compare 5, 10, 20, 40, and 60 sessions with Ridge, continuous rank labels, common OOS rows, equal tuning budgets, and exact sleeves |
| Rebalancing behavior is model-specific | Over six July 20-28 sessions, freezing improved Ridge net return by 3.93 points; daily replacement improved XGBoost by only 0.53 points net |
Six sessions cannot determine permanent policy; a preregistered buffer comparison is required |
| Negative selected stocks persist, while severe losses cluster in stressed markets | Ridge sign false positives occur in every half-year; 2020 H1 and 2022 H1 contain 70.43% of severe observations and 75.79% of severe-loss drag |
The realized market state is explanatory, not available at decision time; the causal SPY regime does not reliably identify losing weeks |
| A bounded tail veto reduces direct severe-loss drag | In the ten-year Ridge-rank A/B, severe-loss frequency fell from 8.03% to 6.90%, CVaR improved, and risk improved in 16 of 17 active-policy folds |
Veto precision was only 13.84%; 53.37% of rejected observations were winners, and return non-inferiority did not generalize across models, so the overlay remains disabled |
| Live operation is feasible | Current market/news refresh, strict local FinBERT, purged tuning, frozen forecasts, and immutable manifests completed | The redesigned live forecasts have not matured; there is no live-return claim yet |
Detailed decision records:
- Next experiment: matched ten-year horizon selection
- Ten-year 20-session ranking and tail-control result
- Framework-clean four-model comparison
- Live forecast-horizon evidence
- Backtest tail-risk timeline
- Severe-loss risk-control A/B experiment
This development backtest covers the corrected ten-year 20-session experiment. The growth panel reports Ridge return, Ridge rank, XGBoost rank, LambdaMART, and full-investment SPY after the declared model implementation costs. The lower panels show drawdown, daily return distributions, and turnover reconstructed from exact sleeve ledgers. SPY remains visible in both growth and drawdown. None of the active portfolios beats SPY.
Source: Ten-year 20-session experiment.
The score report uses the corrected ten-year Ridge-rank OOS scores and separates prediction quality from portfolio accounting:
- mean 20-session return by score quintile;
- same-date Spearman information coefficient;
- positive-IC frequency;
- top-40 versus universe return distribution; and
- average top-40 excess return.
The earlier chart compounded overlapping 20-session outcomes as if they were independent weekly returns. That presentation has been removed. The replacement shows non-compounded quintile means, while exact cumulative performance appears only in the sleeve-accounted portfolio chart. A positive average rank IC still does not imply benchmark-beating economics.
Source: Ten-year 20-session experiment.
The fully invested treatment reduces severe-loss frequency, volatility, drawdown, and CVaR in the primary Ridge-rank test. The chart includes SPY and shows gross zero-cost sensitivity. The treatment's higher compounded endpoint is not proof of return improvement: paired return inference remains uncertain, and the Ridge-return and XGBoost-rank replications lose CAGR. The overlay is not active.
Source: Ten-year 20-session experiment.
The chart uses XGBoost's exact tree contributions for the frozen July 30 20-session model across its 249-name eligible cross-section. It is a local model explanation, not causal importance. Features selected for training can still have zero SHAP contribution when the fitted trees never split on them.
Regenerate all README figures from the local research artifacts:
python -m src.experiments.readme_figuresThe SVGs are tracked; raw datasets, models, and generated report directories remain excluded from Git.
The active experiment issues new forecasts every trading day while matching each strategy's model-refit and portfolio-rebalance clock to its horizon:
| Arm | Model refit | Portfolio rebalance | Label purge | Authority |
|---|---|---|---|---|
| Ridge / XGBoost 5-session | Every 5 sessions | Every 5 sessions | 5 sessions | Short-horizon diagnostic |
| Ridge / XGBoost 10-session | Every 10 sessions | Every 10 sessions | 10 sessions | Required challenger |
| Ridge / XGBoost 20-session | Every 20 sessions | Every 20 sessions | 20 sessions | Provisional longer-horizon arm |
Every scheduled refit performs:
- matured-label filtering;
- rolling temporal validation;
- horizon-matched purging;
- fold-local feature ranking and feature-count selection;
- independent Ridge/XGBoost hyperparameter search;
- final refit on eligible historical labels; and
- immutable model, feature, score, and top-40 artifacts.
Daily forecasts accumulate observations without forcing daily portfolio replacement. After the initial horizon delay, approximately one forecast per horizon matures each trading day, but overlapping outcomes require HAC or block-bootstrap inference.
Run one completed-market-day cycle:
python -m src.cli run-live-forward-20d --as-of YYYY-MM-DDThe default v2 output is reports/live_forward_horizon_matched_v2/. The first
shared-clock implementation remains frozen under reports/live_forward_20d/
and is not mixed into the corrected history.
See the 20-session live-forward design for execution clocks, tactical sentiment gates, evidence thresholds, and artifacts.
The legacy one-session monitor provides a small but useful out-of-sample ablation. It compares the actual daily-changing top 40 with the names frozen from July 20:
| Model | Daily-changing top 40 | Keep July 20 stocks | Better choice |
|---|---|---|---|
| Ridge gross | -1.46% | +1.86% | Freeze by 3.32 points |
| Ridge net | -2.19% | +1.74% | Freeze by 3.93 points |
| XGBoost gross | -0.58% | -1.78% | Daily by 1.20 points |
| XGBoost net | -1.37% | -1.90% | Daily by 0.53 points |
| SPY | -0.17% | -0.17% | N/A |
"Net" uses $1 million starting capital, 8 bps one-way execution cost, and $10 per trade. Daily selection generated approximately 616% Ridge turnover and 662% XGBoost turnover including initial purchases; frozen-name portfolios were near 100%. Frozen equal-weight and buy-and-hold controls were very similar, so membership changes, rather than small weight drift, drove most of the difference.
This result supports removing daily full replacement, but it does not prove that every scheduled rebalance should be a complete top-40 refresh. The current 5/10/20-session design therefore remains under audit. The next controlled experiment will compare, separately for Ridge and XGBoost:
- full horizon-matched refresh;
- frozen holdings;
- rank-buffer replacement; and
- score-margin replacement.
Buffer thresholds must be selected on historical temporal validation and frozen before forward use. The six observed sessions will not choose the threshold.
Read the complete rebalancing ablation
Review the rebalance-buffer experiment draft
The legacy one-session experiment is now a frozen-cohort monitoring track rather than a source of new daily portfolios. Its daily command refreshes prices, extends the existing July 20-27 Ridge/XGBoost cohorts, rebuilds the local dashboard, and regenerates the sanitized GitHub Pages bundle:
python -m src.cli run-live-forward-daily --as-of YYYY-MM-DDThe public monitor also refreshes automatically at 22:30 UTC on weekdays. The scheduled job rebuilds the same frozen-cohort paths from sanitized selections, fails closed on missing prices, validates the bundle, and deploys the refreshed site. The default Portfolio vs Benchmark chart now follows this monitoring path, so its latest date advances even though stock selection remains frozen.
The Holding-period laboratory can keep any dated list unchanged or compare custom 1-60-session replacement horizons, including 5/10/20-session presets. Ridge and XGBoost can be toggled independently while SPY remains permanently visible from the shared decision-close baseline. It reports net return, SPY-relative return, replacement-only turnover, and rebalance counts under identical execution assumptions.
The GitHub Pages workflow publishes two linked pages with one shared visual
shell from main:
The original Stock-level view remains canonical rather than being duplicated on the horizon page. The two-page contract and evidence boundaries are recorded in the dashboard merge design.
This monitoring track does not replace the primary horizon-matched 5/10/20
experiment described above. The former one-session fitting workflow remains
available only through the explicit --retrain flag and is not part of the
normal monitoring run.
The main pipeline is designed to fail closed:
- Historical membership and liquidity are reconstructed point in time.
- Sector classifications are joined backward by effective date.
- Features must exist before the declared decision timestamp.
- Labels must mature before training and are purged from validation boundaries.
- Feature selection and tuning occur inside temporal training folds.
- Ridge remains the hard-to-fake baseline.
- Historical sentiment comparisons require a technical-only paired arm. The current six-arm paper-live monitor includes sentiment in every model, so it cannot by itself attribute future performance specifically to sentiment.
- Same-close news execution requires timestamped quotes and a verified broker cutoff; otherwise execution moves to the next open.
- Missing real market data, stale news, failed FinBERT inference, constant model scores, or incomplete artifacts stop the run.
- Gross selection evidence, cost-adjusted performance, and tactical attribution are reported separately.
python -m venv .venv
# Windows
.venv\Scripts\activate
pip install -r requirements.txt
pip install -e ".[tuning]"
pytest -qUseful research commands:
# Compare matched 5/10/20-session horizons
python -m src.cli run-forecast-horizon-sweep
# Run one paper-live cycle after a completed US market close
python -m src.cli run-live-forward-20d --as-of YYYY-MM-DDRun python -m src.cli --help for the complete command surface.
The main branch keeps the active research system and its current evidence
contract concise. Superseded designs, failed gates, and the full experiment
trail remain valuable but should not be mistaken for simultaneously active
strategies.
- Current documentation index
- Project status and evidence map
- Methodology equations
- Current live-forward design
- Next experiment: ten-year horizon selection
- Ten-year 20-session result
- Rebalance-buffer experiment draft
- Live rebalancing ablation
- Historical experiments and superseded designs
- Archived branch README
Failed experiments are retained as evidence. They are not removed merely because they weaken the preferred narrative.
The next experiment identifies the forecast horizon before adding another model or risk overlay. It will:
- compare 5, 10, 20, 40, and 60 sessions on common corrected OOS dates;
- freeze Ridge and the continuous cross-sectional rank label so only horizon changes in the primary comparison;
- give every horizon the same 600-trial Optuna budget and horizon-matched purge;
- reconstruct exact overlapping sleeves and report both zero-cost and implementation-cost results with SPY always visible;
- use paired block inference and multiple-comparison control rather than selecting the highest observed CAGR;
- choose the shorter horizon when candidates are statistically and economically indistinguishable; and
- retain the paper-live 5/10/20 arms as prospective confirmation rather than mixing them into historical selection.
Only after horizon selection will tail-control research resume. That follow-up will use smaller one-, two-, and four-name veto budgets with minimum precision, rejected-winner, and return-noninferiority constraints. The current treatment remains disabled.
The horizon contract is maintained in the ten-year horizon-selection plan. The tail workflow and failure history remain in the risk-control A/B plan and false-positive research record.
- No redesigned 20-session paper-live forecast has matured yet.
- The corrected 20-session development study now contains 520 weekly OOS decisions and 2,523 daily portfolio observations from December 2013 through December 2023. This longer history improves regime coverage but uses provisional public data and does not make the already-inspected period an untouched confirmation. Long-window backtests still require regime analysis, prospective evidence, and correction for the number of strategies tried. See CFA Institute's backtesting guidance and the deflated-Sharpe discussion of sample length and multiple testing.
- The strongest portfolio results remain development evidence, not a genuine untouched confirmation.
- Public data does not fully solve delisted-security coverage or historical sector classification.
- The average ranking edge is small and varies materially by year and regime.
- Sentiment is promising, but coverage, timestamp quality, and causal attribution remain incomplete. The current six live model/horizon arms do not include a technical-only prospective control.
- The high-recall survival model has not passed its activation gates and remains a shadow watchlist.
- Same-close tactical execution is unvalidated without intraday quotes and actual paper or broker fills.
- Scheduled full top-40 replacement remains under audit; the six-session ablation is too short to activate a freeze or buffer.
- Static cost assumptions do not establish capacity, taxes, borrow availability, market impact, or partial-fill behavior.
- The ten-year fully invested severe-loss treatment reduced severe-loss rate, CVaR, volatility, and drawdown, but veto precision was only 13.84% and return non-inferiority did not generalize across models. It remains inactive.
- The corrected ten-year run evaluates only a 20-session horizon. It cannot prove that 20 sessions is better than 5, 10, 40, or 60 sessions.
This repository is an educational and research system. Backtests, model scores, paper portfolios, and live-forward monitors are not investment advice and are not evidence of guaranteed future performance.