Skip to content

README.md

Latest commit

 

History

History
128 lines (108 loc) · 7.98 KB

README.md

File metadata and controls

128 lines (108 loc) · 7.98 KB
Raw

Experiment governance and evaluation module

Goal

Keep research decisions auditable as the project expands. Every experiment should state what was known beforehand, what may be tuned, what period is untouched, which controls are mandatory, and what result would count as failure.

Experiment lifecycle

  1. Write a dated hypothesis and manifest.
  2. Declare universe, target, features, model candidates, portfolio policy, costs, benchmarks, statistics, and success gates.
  3. Tune only inside nested, purged development data.
  4. Freeze the model artifact, feature list, portfolio rule, and code/data hashes.
  5. Acquire and audit confirmation inputs without synthetic substitution.
  6. Run confirmation once.
  7. Report a failed gate as failure; never tune the same period into a pass.
  8. Keep the 2026-H1 lockbox sealed until a new contract is ready.

Research progression

Version Main question Outcome Evidence status
Core weekly v1 Does the original ten-stock optimizer work after accounting fixes? Accounting and timing defects corrected Superseded
Residual rank v2 Can ML rank five-session sector-neutral returns? Small positive IC; shuffle p=0.327 Development
Sparse rank v3 Can ML choose 20-50 names? Stronger development shuffle result, but 96.9% turnover Post-hoc development
Turnover v4 Can a frozen hold zone control costs? Turnover reduced to about 5% weekly Development success
Public confirmation v4 Does frozen v4 beat matched SPY out of sample? Positive point estimate; 2 of 4 gates failed Failed confirmation
Topic features v5 Can point-in-time topics become rank-model inputs? Expanded NMF covers 313 eligible names but only 13.95% of rows; 50% gate failed Gated development baseline
Timestamped news v5 Does exact-time FinBERT news earn next-open residual returns? Five-session incremental return -44.62 bps; both placebos fail Failed development purchase gate
Weekly slow NLP v5 Do sentiment and topics add to one fixed numeric forecast? Combined +1.71% annualized, but 95% CI crosses zero and shuffle p=0.366 Failed robustness gate
Pair trading v5 Can a point-in-time pair sleeve add net return? +0.038% annualized; timing p=0.0099 but random-pair p=0.218 and CI crosses zero Failed activation gate
Fama-French v6 Does an FF3-neutral label improve fixed Ridge selection, and is return factor alpha? Better point estimate, but shuffle p=0.366, incremental CI crosses zero, and alpha p=0.768 Target replacement rejected
Final reference v1 Can all frozen evidence be packaged under one fail-closed contract? Reproduced 12.19% development CAGR and 11.34% observed-confirmation CAGR without opening lockbox Built research reference
Smart universe v7 Can transparent rules narrow 250 names to 120 before Ridge? Beat random subsets, but lost 6.57% annualized versus full equal weight and failed 4/5 gates Activation rejected
Repaired rank-40 v8 Does repaired price coverage support weekly ML winner selection? Full weekly rank-40 earned 37.22% CAGR; rank IC 0.0059; smart-universe filter did not improve it Promising development direction
Survival filter v8 Can a rare-event model avoid likely S&P 500 exits before rank-40 selection? Top 10% risk bucket caught only 17.07% of future-exit rows; shuffled placebo beat the real filter Activation rejected
Ensemble v10 Do independently tuned Ridge, LightGBM, and XGBoost combine into stronger weekly rankings? XGBoost top-40 point estimate improved, but CI crossed zero; LightGBM and ensemble failed OOS Ensemble rejected

Frozen v4 gates and result

Gate Rule Result Pass?
Rank IC Greater than 0 +0.016 Yes
Annualized net active return Greater than 0 +0.31% Yes
Shuffled-score p-value At most 0.10 0.198 No
95% active-return lower bound At least 0 -12.79% No

The pair sleeve remains disabled after failing its own confidence-bound and random-pair gates. The 2024 through 2025 Q1 period is now observed and cannot be reused as untouched confirmation after changing the model.

Reproducibility artifacts

  • Resolved YAML experiment manifests.
  • Coverage and readiness audits.
  • Frozen model and metadata hashes.
  • Persisted OOS forecasts and selected holdings.
  • Weekly net returns and execution attribution.
  • Shuffled permutation distributions and bootstrap intervals.
  • Dated Markdown experiment logs under docs/experiments.

Main commands

python -m src.cli run-public-confirmation-v4 --help
python -m src.cli run-lockbox-test --help
python -m src.cli run-revealing-controls --help
python -m src.cli build-topic-features --help
python -m src.cli run-timestamped-news-event-study --help
python -m src.cli run-weekly-nlp-incremental-v5 --help
python -m src.cli run-pair-trading-v5 --help
python -m src.cli download-fama-french --help
python -m src.cli run-fama-french-v6 --help
python -m src.cli build-final-version --help
python -m src.cli run-smart-universe-v7 --help
python -m src.cli run-slow-smart-universe-v8 --help
python -m src.cli evaluate-rank-model --help
python -m src.cli run-survival-filter-v8 --help
python -m src.cli run-double-ensemble-v10 --help
python -m src.cli evaluate-persisted-double-ensemble-v10 --help
python -m src.cli run-live-forward-daily --help
python -m src.cli report-live-intraday --help
python -m src.cli build-live-dashboard --help
python -m src.cli publish-live-dashboard --help
python -m src.cli validate-live-dashboard-site --help

Live-forward monitoring

The daily live-forward workflow retrains Ridge and XGBoost independently using only information available before the decision timestamp, freezes each top-40 selection, evaluates previously frozen portfolios, records deterministic model-overlap and day-over-day stability diagnostics, and rebuilds the dashboard. See the operating guide for timing rules, artifacts, formulas, and interpretation boundaries.

The daily command also regenerates the sanitized site/ bundle used by GitHub Pages. The public publisher exposes only allowlisted display fields and validates the bundle before it can be deployed. See the deployment guide.

Improvement backlog

  • Create one resolved manifest consumed by every pipeline stage.
  • Persist environment and package versions with every experiment.
  • Add resume support for permutations and nested folds.
  • Add automated documentation checks that compare reported metrics with JSON artifacts.
  • Follow the staged v5 proposal: transparent selector bake-off, point-in-time topic test, structured LLM incremental test, independent pair test, and only then a combined contract.

Related documentation