Skip to content

README.md

Latest commit

 

History

History
79 lines (65 loc) · 4.25 KB

README.md

File metadata and controls

79 lines (65 loc) · 4.25 KB
Raw

Forecasting and ranking models

Goal

Estimate cross-sectional stock order without pretending that raw model scores are calibrated expected returns. Model choice, regularization, feature count, and validation all occur inside nested, purged walk-forward training.

How it works

  • rank_models.py supplies the regularized Ridge rank baseline, XGBoost pairwise ranker, and a Qlib-style LightGBM regressor for cross-sectional rank labels.
  • residual_rank_v2.py performs inner model/feature selection and persists outer-OOS scores.
  • Training labels are purged from validation boundaries to avoid overlapping five-session returns.
  • Forecast quality is measured primarily with same-date Spearman rank IC.
  • Shuffled same-date scores are a permanent falsification control.
  • Legacy directional XGBoost, random forest, LSTM, Transformer, and ensemble models remain for comparison, but they are not the frozen v4 confirmation model.
  • double_ensemble_v10.py independently tunes Ridge, LightGBM, and XGBoost with Optuna, model-specific shuffle-ablation selection, early stopping, and inner-OOS ensemble weights.
  • The legacy LSTM/Transformer sequence builder operates on consecutive panel rows without enforcing ticker boundaries. Those models are excluded from all validated experiments unless the sequence pipeline is reconstructed and tested; no reconstruction is in the current scope.

Current achievement and evidence

  • Frozen model: Ridge with alpha=10, 40 stable features, trained through 2023.
  • V2 development rank IC: +0.0077; shuffled-forecast p-value: 0.327.
  • Sparse-v3 development model Sharpe exceeded all 100 shuffled paths, but that design was created after inspecting v2 and therefore was not confirmation evidence.
  • Public v4 confirmation rank IC: +0.016.
  • Public confirmation shuffled-score p-value: 0.198, so randomized rankings were not beaten convincingly.
  • A fixed Ridge FF3-residual target A/B raised CAGR from 5.34% to 8.06%, but its score-shuffle p-value was 0.366 and its factor-alpha p-value was 0.768. Target de-biasing did not establish forecast skill.
  • The corrected v10 study rejected the ensemble: Ridge, LightGBM, XGBoost, and ensemble Rank ICs were 0.0056, -0.0061, 0.0022, and 0.0014. XGBoost's top-40 excess point estimate improved to 0.2121% weekly, but the paired confidence interval included zero.

Limitations

  • The confirmation sample contains only 64 weekly observations.
  • Model selection can still overfit when many feature/count combinations are explored.
  • Positive rank IC does not automatically survive portfolio constraints and transaction costs.
  • The public confirmation result failed two statistical gates and cannot be tuned into a pass.
  • Existing LSTM and Transformer outputs are not valid evidence because a sequence can cross from one ticker into another and the initial prediction padding is not ticker-specific.

Improvement backlog

  • Keep Ridge as a hard-to-fake baseline and require complex models to beat it out of sample.
  • Cache every fold's feature ranking, parameters, predictions, and data hashes.
  • Evaluate feature and coefficient stability across economic regimes.
  • Calibrate score strength only from prior OOS predictions and shrink it toward zero.
  • Develop changes on a separate sample, then freeze before a genuinely untouched period.
  • Keep factor residualization distinct from feature selection: the former defines what counts as stock-specific return; the latter searches for predictors after that definition is fixed.

Main commands

python -m src.cli train-model --help
python -m src.cli train-residual-rank --help
python -m src.cli evaluate-residual-rank-oos --help
python -m src.cli freeze-final-rank-model --help
python -m src.cli run-fama-french-v6 --help
python -m src.cli run-qlib-alpha158-comparison --help
python -m src.cli run-double-ensemble-v10 --help
python -m src.cli evaluate-persisted-double-ensemble-v10 --help

Related documentation