diff --git a/README.md b/README.md index 27f4f35..c3186a5 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ ![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-2563eb) ![Research status](https://img.shields.io/badge/status-paper--live_research-d97706) -![Tests](https://img.shields.io/badge/tests-218_passing-0f766e) +![Tests](https://img.shields.io/badge/tests-221_passing-0f766e) ![License](https://img.shields.io/badge/license-MIT-64748b) The project asks a narrow investment question: @@ -227,7 +227,14 @@ visible from the shared decision-close baseline. It reports net return, SPY-relative return, replacement-only turnover, and rebalance counts under identical execution assumptions. -[Open the deployed live-forward dashboard →](https://hansun103.github.io/AI-Driven-Portfolio-Optimization-Backtesting-System/) +The deployed monitor now has two linked pages with one shared visual shell: + +- [Live Forward Monitor — default view →](https://hansun103.github.io/AI-Driven-Portfolio-Optimization-Backtesting-System/) +- [Horizon Comparison — 5/10/20-session research →](https://hansun103.github.io/AI-Driven-Portfolio-Optimization-Backtesting-System/horizons.html) + +The original Stock-level view remains canonical rather than being duplicated +on the horizon page. The two-page contract and evidence boundaries are recorded +in the [dashboard merge design](docs/design/DASHBOARD_TWO_PAGE_MERGE.md). This monitoring track does not replace the primary horizon-matched 5/10/20 experiment described above. The former one-session fitting workflow remains @@ -378,20 +385,29 @@ because they weaken the preferred narrative. ## Known limitations 1. No redesigned 20-session paper-live forecast has matured yet. -2. The strongest portfolio results remain development evidence, not a genuine +2. The strongest repaired historical test covers roughly 2018–2023, not a + full decade. It therefore spans too few independent market regimes for a + durable claim. Extending point-in-time testing toward at least ten years is + a research target, not a guarantee of adequacy: long-window backtests still + require regime analysis, untouched confirmation, and correction for the + number of strategies tried. See + [CFA Institute’s backtesting guidance](https://www.cfainstitute.org/insights/professional-learning/refresher-readings/2026/backtesting-and-simulation) + and the + [deflated-Sharpe discussion of sample length and multiple testing](https://academic.oup.com/jrssig/article/18/6/22/7038278). +3. The strongest portfolio results remain development evidence, not a genuine untouched confirmation. -3. Public data does not fully solve delisted-security coverage or historical +4. Public data does not fully solve delisted-security coverage or historical sector classification. -4. The average ranking edge is small and varies materially by year and regime. -5. Sentiment is promising, but coverage, timestamp quality, and causal +5. The average ranking edge is small and varies materially by year and regime. +6. Sentiment is promising, but coverage, timestamp quality, and causal attribution remain incomplete. -6. The high-recall survival model has not passed its activation gates and +7. The high-recall survival model has not passed its activation gates and remains a shadow watchlist. -7. Same-close tactical execution is unvalidated without intraday quotes and +8. Same-close tactical execution is unvalidated without intraday quotes and actual paper or broker fills. -8. Scheduled full top-40 replacement remains under audit; the six-session +9. Scheduled full top-40 replacement remains under audit; the six-session ablation is too short to activate a freeze or buffer. -9. Static cost assumptions do not establish capacity, taxes, borrow +10. Static cost assumptions do not establish capacity, taxes, borrow availability, market impact, or partial-fill behavior. ## Research disclaimer diff --git a/docs/design/DASHBOARD_TWO_PAGE_MERGE.md b/docs/design/DASHBOARD_TWO_PAGE_MERGE.md new file mode 100644 index 0000000..280180b --- /dev/null +++ b/docs/design/DASHBOARD_TWO_PAGE_MERGE.md @@ -0,0 +1,88 @@ +# Two-page dashboard merge design + +Status: local proposal awaiting approval. Do not merge or deploy yet. + +## Decision + +Publish one dashboard product with two real static pages: + +| Page | Proposed URL | Responsibility | +| --- | --- | --- | +| Live Forward Monitor | `dashboard.html` | Daily operational evidence, benchmark comparison, stability, the canonical stock-level view, and experiment history | +| Horizon Comparison | `horizons.html` | Horizon-matched 5/10/20-session progress, validation context, model specifications, SHAP explanations, and pipeline health | + +Both headers use the same two-option page switcher. The selected page is visually +active. Ordinary links are preferable to a hidden single-page application because +browser back/forward works, each page can fail independently, and GitHub Pages can +serve the files without a runtime. + +## Shared stock-level view + +The Horizon Comparison page does not implement another portfolio table. + +The existing Stock-level view remains the canonical component. Its model selector +will be extended to understand both: + +- legacy daily live-forward snapshots; and +- v2 Ridge/XGBoost arms for the 5-, 10-, and 20-session horizons. + +An `Inspect stocks` action on a horizon model will link to a URL such as: + +```text +dashboard.html?decision=2026-07-29&family=xgb_ranker&horizon=20#stock-level +``` + +The original page reads those parameters, selects the requested frozen portfolio, +and scrolls to the shared Stock-level view. This avoids divergent filtering, +sorting, return formatting, and holding-level evaluation logic. + +## Partial and mature evidence + +The horizon page reports three different evidence states without mixing them: + +1. **Awaiting entry** — the portfolio is frozen but its next-open entry has not + occurred. +2. **Interim mark** — one or more open-to-open observations exist. The page shows + session progress and the compounded partial return immediately. +3. **Matured label** — the complete 5-, 10-, or 20-session evaluation is available + and may enter the formal prospective comparison. + +The page never displays validation return as live return. A zero-session portfolio +is shown as `0 / horizon` and `Awaiting entry`, not as a zero realized return. + +## Explainability + +Every newly frozen v2 forecast writes `shap_summary.csv`. + +- Ridge uses linear SHAP contributions around the current eligible cohort as the + background. +- XGBoost uses exact TreeSHAP contributions from the fitted booster. +- The dashboard publishes mean absolute SHAP, signed mean SHAP, positive + contribution share, method, and explanation-row count. + +Each model card contains a collapsed `Show SHAP feature importance` control. +Opening it displays a horizontal importance chart. Old immutable forecasts that +predate this artifact state that SHAP was not captured; coefficients or XGBoost +gain are not relabelled as SHAP. + +## Daily refresh contract + +The daily pipeline should: + +1. refresh data and generate the current snapshots; +2. evaluate all available interim open-to-open observations; +3. evaluate newly matured forecasts; +4. generate SHAP summaries for new horizon snapshots; +5. rebuild `dashboard.html` and `horizons.html`; +6. run public-payload and browser smoke checks; and +7. publish both pages together through the existing GitHub Pages action. + +Publishing must be atomic: a cycle should never expose one page with a newer data +contract than the other. + +## Approval boundary + +The current local Horizon Comparison preview demonstrates the second page. After +approval, the merge work will add the page switcher and v2 arm support to the +original dashboard, generate the final `horizons.html`, update the refresh and +deployment workflows, and then request approval before pushing. diff --git a/site/assets/dashboard.css b/site/assets/dashboard.css index 325f1ca..7a2b8b5 100644 --- a/site/assets/dashboard.css +++ b/site/assets/dashboard.css @@ -1,11 +1,12 @@ :root{--ink:#102033;--muted:#637083;--line:#dce3ec;--surface:#fff;--canvas:#f4f7fb;--blue:#3156d3;--teal:#0f9f8f;--amber:#d89614;--red:#c94c5b;--shadow:0 14px 35px rgba(24,39,75,.08)} *{box-sizing:border-box}body{margin:0;color:var(--ink);background:var(--canvas);font-family:Inter,ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif} -header{position:relative;overflow:hidden;padding:38px max(28px,calc((100vw - 1440px)/2));color:#fff;background:linear-gradient(120deg,#081525,#12294b 70%,#173c57)} -header:after{content:"";position:absolute;right:-90px;top:-160px;width:420px;height:420px;border:1px solid #ffffff21;border-radius:50%;box-shadow:0 0 0 55px #ffffff0a,0 0 0 115px #ffffff08} -.eyebrow{color:#7dd3c7;font-size:11px;font-weight:800;letter-spacing:.16em;text-transform:uppercase}h1{margin:7px 0 8px;font-size:clamp(28px,3vw,42px);letter-spacing:-.035em}header p{max-width:760px;margin:0;color:#bdc9da;line-height:1.55} -main{width:min(1440px,calc(100% - 40px));margin:auto;padding:26px 0 54px}.panel{background:var(--surface);border:1px solid #d8e0eb;border-radius:18px;box-shadow:var(--shadow)} -.toolbar{display:flex;justify-content:space-between;align-items:end;gap:20px;padding:18px 20px;margin-top:-50px;position:relative;z-index:2}.toolbar-group,.segmented{display:flex;align-items:end;gap:10px;flex-wrap:wrap} +header{position:relative;overflow:hidden;padding:42px max(28px,calc((100vw - 1440px)/2)) 82px;color:#fff;background:linear-gradient(120deg,#081525,#12294b 68%,#173c57)} +header:after{content:"";position:absolute;right:-90px;top:-170px;width:440px;height:440px;border:1px solid #ffffff21;border-radius:50%;box-shadow:0 0 0 55px #ffffff0a,0 0 0 115px #ffffff08} +.hero-row{display:flex;justify-content:space-between;align-items:end;gap:28px;position:relative;z-index:1}.eyebrow{color:#7dd3c7;font-size:11px;font-weight:850;letter-spacing:.16em;text-transform:uppercase}h1{margin:8px 0 9px;font-size:clamp(30px,3.2vw,46px);letter-spacing:-.04em}header p{max-width:780px;margin:0;color:#bdc9da;line-height:1.55}.hero-badge{display:grid;gap:4px;min-width:190px;padding:14px 16px;border:1px solid #ffffff24;border-radius:14px;background:#ffffff0d;backdrop-filter:blur(6px)}.hero-badge span{color:#9fb0c8;font-size:10px;text-transform:uppercase;letter-spacing:.08em}.hero-badge strong{font-size:15px} +.page-switcher{display:flex;gap:9px;margin-top:22px;position:relative;z-index:2}.page-link{display:grid;gap:3px;min-width:210px;padding:12px 15px;border:1px solid #ffffff28;border-radius:12px;color:#c5d2e3;background:#ffffff0a;text-decoration:none;transition:.18s ease}.page-link:hover{transform:translateY(-1px);border-color:#7dd3c7;background:#ffffff12}.page-link.active{color:#fff;border-color:#7dd3c7;background:#7dd3c719;box-shadow:inset 0 -3px 0 #7dd3c7}.page-link strong{font-size:12px}.page-link small{color:#95a8c0;font-size:9px}.page-link.active small{color:#b9d8d3} +main{width:min(1440px,calc(100% - 40px));margin:-52px auto 0;padding-bottom:58px;position:relative}.panel{background:var(--surface);border:1px solid #d8e0eb;border-radius:18px;box-shadow:var(--shadow)} +.toolbar{display:flex;justify-content:space-between;align-items:end;gap:18px;padding:18px 20px;position:relative;z-index:2}.toolbar-group,.segmented{display:flex;align-items:end;gap:10px;flex-wrap:wrap} label{display:grid;gap:6px;color:var(--muted);font-size:11px;font-weight:750;letter-spacing:.04em;text-transform:uppercase}input,select{min-height:39px;padding:8px 11px;color:var(--ink);background:#fbfcfe;border:1px solid #cfd8e5;border-radius:9px;font:inherit;text-transform:none;letter-spacing:0} button{min-height:39px;padding:8px 15px;border:1px solid #cfd8e5;border-radius:9px;color:#34445a;background:#fff;cursor:pointer;font-weight:700}button:hover,button.active{color:#fff;border-color:var(--blue);background:var(--blue)} .status{display:flex;align-items:center;gap:8px;color:var(--muted);font-size:12px;white-space:nowrap}.status:before{content:"";width:8px;height:8px;border-radius:50%;background:#24b47e;box-shadow:0 0 0 4px #24b47e1f} @@ -19,4 +20,4 @@ button{min-height:39px;padding:8px 15px;border:1px solid #cfd8e5;border-radius:9 .stability-grid{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:10px}.stability-card{padding:14px;border:1px solid var(--line);border-radius:12px;background:#fbfcfe}.stability-card span{display:block;color:var(--muted);font-size:10px}.stability-card strong{display:block;margin-top:6px;font-size:20px}.stability-card small{display:block;margin-top:5px;color:var(--muted)}.stock-summary{margin-bottom:12px}.consensus{margin-top:13px;padding:12px 14px;border-radius:11px;background:#edf9f7;color:#245b54;font-size:12px;line-height:1.6}.consensus b{color:#087a6d} .stability-details{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:12px;margin-top:12px}.stability-detail{padding:15px;border:1px solid var(--line);border-radius:12px}.stability-detail h3{margin:0 0 11px;font-size:14px}.stat-row{display:grid;grid-template-columns:repeat(3,1fr);gap:7px}.stat{padding:8px;border-radius:8px;background:#f4f7fa}.stat span{display:block;color:var(--muted);font-size:9px}.stat b{display:block;margin-top:4px;font-size:12px}.ticker-row{margin-top:10px;color:var(--muted);font-size:10px;line-height:1.55}.ticker-row b{color:var(--ink)}details{margin-top:8px;color:var(--muted);font-size:10px}summary{cursor:pointer;color:#425268;font-weight:700}.rule-insight{margin-top:12px;padding:12px 14px;border-left:4px solid var(--blue);border-radius:8px;background:#f2f5ff;color:#34445a;font-size:11px;line-height:1.6} .table-wrap{overflow-x:auto;border:1px solid var(--line);border-radius:12px}table{width:100%;border-collapse:collapse;background:#fff}th,td{padding:11px 13px;border-bottom:1px solid #edf1f5;font-size:12px;text-align:right;white-space:nowrap}th:first-child,td:first-child{text-align:left}th{color:#526075;background:#f7f9fc;font-size:10px;letter-spacing:.045em;text-transform:uppercase}tr:last-child td{border-bottom:0}.pos{color:#078071;font-weight:700}.neg{color:var(--red);font-weight:700}.muted,.empty{color:var(--muted)}.empty{padding:42px;text-align:center}.history-controls{display:flex;align-items:end;justify-content:flex-end;gap:10px;flex-wrap:wrap}.sort-button{min-height:0;padding:0;border:0;border-radius:0;color:inherit;background:transparent;font-size:inherit;font-weight:800;letter-spacing:inherit;text-transform:inherit}.sort-button:hover,.sort-button.active{color:var(--blue);background:transparent}.sort-arrow{display:inline-block;min-width:11px;margin-left:4px}.evaluation-note{margin:0 0 12px;padding:10px 12px;border-radius:9px;color:#536176;background:#f5f7fa;font-size:11px}.evaluation-note strong{color:var(--ink)} -@media(max-width:980px){.kpi-grid,.stability-grid{grid-template-columns:repeat(2,1fr)}.model-grid,.stability-details{grid-template-columns:1fr}.toolbar,.pending-panel{align-items:stretch;flex-direction:column}}@media(max-width:620px){main{width:calc(100% - 20px)}header{padding-left:20px}.kpi-grid,.metric-grid,.parameter-grid,.stability-grid,.stat-row{grid-template-columns:1fr}} +@media(max-width:980px){.kpi-grid,.stability-grid{grid-template-columns:repeat(2,1fr)}.model-grid,.stability-details{grid-template-columns:1fr}.toolbar,.pending-panel{align-items:stretch;flex-direction:column}.hero-row{align-items:start;flex-direction:column}.hero-badge{min-width:0}}@media(max-width:620px){main{width:calc(100% - 20px)}header{padding-left:20px;padding-right:20px}.kpi-grid,.metric-grid,.parameter-grid,.stability-grid,.stat-row{grid-template-columns:1fr}.page-switcher{display:grid}.page-link{min-width:0}.hero-badge{display:none}} diff --git a/site/data/manifest.json b/site/data/manifest.json index b95bbc8..28b9456 100644 --- a/site/data/manifest.json +++ b/site/data/manifest.json @@ -1,10 +1,13 @@ { - "schema_version": 2, - "generated_at_utc": "2026-07-29T16:55:17.967215+00:00", + "schema_version": 3, + "generated_at_utc": "2026-07-29T22:29:54.012984+00:00", "latest_decision_date": "2026-07-28", "snapshot_count": 17, "evaluation_count": 14, "dashboard_sha256": "9d15420d6e5370038763505be97432e381f7a0932cc1aeac953399c030b2185c", "dashboard_bytes": 546502, + "horizon_dashboard_sha256": "20d6bdaa4d619ecca77441801a4f2e63b93c665822979b278cb490999bd4695a", + "horizon_dashboard_bytes": 66861, + "horizon_forecast_arms": 6, "source_policy": "sanitized_live_forward_outputs_only" } \ No newline at end of file diff --git a/site/horizons.html b/site/horizons.html new file mode 100644 index 0000000..afff2dd --- /dev/null +++ b/site/horizons.html @@ -0,0 +1,65 @@ + + +Horizon-Matched Live Forward Monitor + +
AI Portfolio Research

Horizon Comparison

Six independently tuned portfolios tracked on their natural 5, 10, and 20-session clocks. Interim marks, matured outcomes, and validation evidence remain explicitly separated.

Evidence statusPaper-live · collecting
+
+
Frozen evidence loaded
+
+
+

Prospective strategy evidence

Partial mark-to-market outcomes appear as soon as an entered portfolio has a new observable open. They remain interim until the complete 5, 10, or 20-session window matures.

The decision session is not a holding session. A July 29 close signal enters at the July 30 open, so July 29 must remain 0/5 rather than borrowing a pre-entry return. Interim marks answer “what has happened so far?” and are never mixed with validation estimates.
+

Horizon comparison

Paired Ridge and XGBoost development evidence with each arm’s own purge, feature subset, refit interval, and maturity date.

+

Selected model specifications

The winning development configuration for each visible arm. Open a model’s explainability panel to inspect genuine SHAP importance when that frozen vintage captured it.

+

Pipeline and risk controls

Freshness, universe, NLP, tactical routing, survival guardrails, and artifact integrity for the selected decision date.

+

Matured forecast history

Prospective outcomes appear only after their full label interval matures. Pending forecasts are not scored as zero.

+
+ + \ No newline at end of file diff --git a/site/index.html b/site/index.html index fe49769..6375b63 100644 --- a/site/index.html +++ b/site/index.html @@ -2,7 +2,7 @@ Live Forward Research Monitor -
AI Portfolio Research

Live Forward Monitor

Out-of-sample model portfolios tracked against SPY. Every session uses the portfolio frozen before its evaluation window.

+
AI Portfolio Research

Live Forward Monitor

Out-of-sample model portfolios tracked against SPY. Every session uses the portfolio frozen before its evaluation window.

Default viewDaily evidenceFrozen-cohort monitor
Frozen evidence loaded
@@ -10,7 +10,7 @@

Portfolio vs Benchmark

Compounded return across selected live sessions. Models may begin on different dates; use the controls to show or hide individual portfolios.

Holding-period laboratory

Compare both models while keeping a dated top-40 list or replaying a chosen rebalance interval. SPY is always shown from the same decision-close baseline.

Selection stability

Daily model agreement, retained holdings, rank consistency, and implied equal-weight replacement.

-

Stock-level view

Inspect a frozen portfolio for one decision date. Realized returns appear after a holding-level evaluation is available.

+

Stock-level view

Inspect a frozen portfolio for one decision date. Realized returns appear after a holding-level evaluation is available.

Selected models

Only the winning specification refit for each live candidate is shown; candidate trials are omitted.

Experiment history

Newest evaluated evidence is shown first. Filter the archive and sort any column as it grows.

\ No newline at end of file diff --git a/src/experiments/live_dashboard.py b/src/experiments/live_dashboard.py index 1cf5bd8..6d9d907 100644 --- a/src/experiments/live_dashboard.py +++ b/src/experiments/live_dashboard.py @@ -71,11 +71,12 @@ def dashboard_site_assets() -> dict[str, str]: -
AI Portfolio Research

Live Forward Monitor

Out-of-sample model portfolios tracked against SPY. Every session uses the portfolio frozen before its evaluation window.

+
AI Portfolio Research

Live Forward Monitor

Out-of-sample model portfolios tracked against SPY. Every session uses the portfolio frozen before its evaluation window.

Default viewDaily evidenceFrozen-cohort monitor
Frozen evidence loaded
@@ -99,7 +100,7 @@ def dashboard_site_assets() -> dict[str, str]:

Portfolio vs Benchmark

Compounded return across selected live sessions. Models may begin on different dates; use the controls to show or hide individual portfolios.

Holding-period laboratory

Compare both models while keeping a dated top-40 list or replaying a chosen rebalance interval. SPY is always shown from the same decision-close baseline.

Selection stability

Daily model agreement, retained holdings, rank consistency, and implied equal-weight replacement.

-

Stock-level view

Inspect a frozen portfolio for one decision date. Realized returns appear after a holding-level evaluation is available.

+

Stock-level view

Inspect a frozen portfolio for one decision date. Realized returns appear after a holding-level evaluation is available.

Selected models

Only the winning specification refit for each live candidate is shown; candidate trials are omitted.

Experiment history

Newest evaluated evidence is shown first. Filter the archive and sort any column as it grows.

+""" diff --git a/src/experiments/public_dashboard_validation.py b/src/experiments/public_dashboard_validation.py index fe9264d..1f08834 100644 --- a/src/experiments/public_dashboard_validation.py +++ b/src/experiments/public_dashboard_validation.py @@ -18,6 +18,7 @@ def validate_public_dashboard_bundle(output_dir: str | Path) -> dict[str, Any]: root = Path(output_dir) required = [ root / "index.html", + root / "horizons.html", root / "assets/dashboard.css", root / "assets/dashboard.js", root / "assets/loader.js", @@ -86,6 +87,11 @@ def validate_public_dashboard_bundle(output_dir: str | Path) -> dict[str, Any]: manifest = json.loads((root / "data/manifest.json").read_text(encoding="utf-8")) if manifest.get("dashboard_sha256") != dashboard_sha256(dashboard_bytes): raise ValueError("Public dashboard hash does not match its deployment manifest") + horizon_bytes = (root / "horizons.html").read_bytes() + if manifest.get("horizon_dashboard_sha256") != dashboard_sha256(horizon_bytes): + raise ValueError( + "Public horizon dashboard hash does not match its deployment manifest" + ) return { "output_dir": str(root), "snapshots": len(snapshots), @@ -94,6 +100,10 @@ def validate_public_dashboard_bundle(output_dir: str | Path) -> dict[str, Any]: "policies": len(cohort_monitor.get("policies", [])), "dashboard_bytes": len(dashboard_bytes), "dashboard_sha256": manifest["dashboard_sha256"], + "horizon_forecast_arms": int( + manifest.get("horizon_forecast_arms", 0) + ), + "horizon_dashboard_bytes": len(horizon_bytes), } diff --git a/tests/test_live_dashboard.py b/tests/test_live_dashboard.py index 5c7cdcb..56d2514 100644 --- a/tests/test_live_dashboard.py +++ b/tests/test_live_dashboard.py @@ -35,6 +35,9 @@ def test_dashboard_uses_date_ranges_and_overlays_models() -> None: assert "SPY · always shown" in html assert "Keep one stock list" in html assert "Rebalance horizon" in html + assert 'class="page-link active" href="dashboard.html"' in html + assert "../live_forward_horizon_matched_v2/dashboard.html" in html + assert 'id="stock-level"' in html def test_dashboard_draws_visible_session_anchors_and_selected_model_cards() -> None: diff --git a/tests/test_live_dashboard_publish.py b/tests/test_live_dashboard_publish.py index 357425c..73d3a53 100644 --- a/tests/test_live_dashboard_publish.py +++ b/tests/test_live_dashboard_publish.py @@ -150,10 +150,15 @@ def test_publish_site_splits_assets_and_passes_validation( result = publish_live_dashboard(tmp_path / "private", site) index = (site / "index.html").read_text(encoding="utf-8") + horizons = (site / "horizons.html").read_text(encoding="utf-8") assert '' in index assert '' in index + assert 'href="./horizons.html"' in index + assert 'href="./index.html"' in horizons + assert "Horizon Comparison" in horizons assert "ridge:2026-07-22" not in index assert result["snapshots"] == 1 + assert result["horizon_forecast_arms"] == 0 assert validate_public_dashboard_bundle(site)["evaluations"] == 1 diff --git a/tests/test_live_forward_20d.py b/tests/test_live_forward_20d.py index 37498f5..319c79c 100644 --- a/tests/test_live_forward_20d.py +++ b/tests/test_live_forward_20d.py @@ -12,6 +12,7 @@ _refit_sessions_for_horizon, build_exit_labels_fast, build_liquid_eligibility, + build_shap_feature_summary, build_tactical_sentiment_orders, build_timestamp_safe_nlp_features, choose_high_recall_threshold, @@ -27,6 +28,38 @@ from src.utils.schemas import TextPost +def test_ridge_shap_summary_uses_current_cohort_background() -> None: + dates = pd.DatetimeIndex(["2026-07-29"]) + index = pd.MultiIndex.from_product( + [dates, ["A", "B", "C"]], + names=["date", "ticker"], + ) + current = pd.DataFrame( + { + "momentum": [-1.0, 0.0, 1.0], + "sentiment": [0.5, -0.5, 1.5], + }, + index=index, + ) + target = pd.Series([-0.4, 0.0, 0.7], index=index) + model = RidgeRankModel(alpha=1.0).fit(current, target) + + summary = build_shap_feature_summary( + current, + model, + ["momentum", "sentiment"], + family="ridge", + ) + + assert set(summary["feature"]) == {"momentum", "sentiment"} + assert summary["mean_abs_shap"].gt(0.0).all() + assert summary["method"].eq( + "linear_shap_current_cohort_background" + ).all() + assert summary["explanation_rows"].eq(3).all() + assert summary["importance_rank"].tolist() == [1, 2] + + def test_fast_exit_labels_match_reference_implementation() -> None: dates = pd.date_range("2023-01-02", periods=8, freq="B") index = pd.MultiIndex.from_product( @@ -495,6 +528,10 @@ def test_end_to_end_cycle_freezes_and_matures_forecast( assert first["forecasts"][0]["strategic_rebalance"] assert first["forecasts"][0]["strategy_rebalance"] assert first["forecasts"][0]["refit_interval_sessions"] == 5 + assert first["forecasts"][0]["explainability"]["status"] == "available" + assert ( + first_manifest.parent / "shap_summary.csv" + ).exists() assert replay["forecasts"][0]["reused_existing_snapshot"] assert first_manifest.read_bytes() == frozen_bytes assert second["forecasts"][0]["strategic_rebalance"] @@ -509,4 +546,6 @@ def test_end_to_end_cycle_freezes_and_matures_forecast( assert third["strategy_daily_return_rows"] >= 1 assert (tmp_path / "matured_forecasts.csv").exists() assert (tmp_path / "strategy_daily_returns.csv").exists() + strategy_returns = pd.read_csv(tmp_path / "strategy_daily_returns.csv") + assert strategy_returns["spy_return"].notna().all() assert (tmp_path / "dashboard.html").exists() diff --git a/tests/test_live_forward_20d_dashboard.py b/tests/test_live_forward_20d_dashboard.py new file mode 100644 index 0000000..b864649 --- /dev/null +++ b/tests/test_live_forward_20d_dashboard.py @@ -0,0 +1,175 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pandas as pd + +from src.experiments.live_forward_20d_dashboard import ( + build_horizon_dashboard_payload, + horizon_dashboard_html, +) + + +def _write_json(path: Path, payload: dict[str, object]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload), encoding="utf-8") + + +def test_dashboard_payload_collects_public_arm_evidence(tmp_path: Path) -> None: + root = tmp_path / "reports" / "live_forward_horizon_matched_v2" + forecast_root = root / "forecasts" / "2026-07-29" / "ridge_h5" + _write_json( + forecast_root / "manifest.json", + { + "family": "ridge", + "horizon_sessions": 5, + "decision_date": "2026-07-29", + "model_refit_date": "2026-07-29", + "status": "pending", + "expected_entry_date": "2026-07-30", + "expected_maturity_date": "2026-08-06", + "refit_interval_sessions": 5, + "refitted_today": True, + "strategy_rebalance": True, + "strategic_rebalance": True, + "eligible_names": 250, + "selected_names": 2, + "sentiment_coverage": 1.0, + "feature_count": 2, + "features": ["momentum_63", "nlp_sentiment_3d"], + "explainability": { + "status": "available", + "method": "tree_shap", + "rows": 250, + "artifact": "C:/private/shap_summary.csv", + }, + }, + ) + pd.DataFrame( + [ + { + "ticker": "AAA", + "sector": "Technology", + "rank": 1, + "score": 0.03, + "weight": 0.5, + "private_column": "do-not-publish", + } + ] + ).to_csv(forecast_root / "selected_holdings.csv", index=False) + + selection_path = tmp_path / "selection.csv" + pd.DataFrame( + [ + { + "row_type": "aggregate", + "selection_eligible": True, + "top40_raw_excess": 0.012, + "rank_ic": 0.04, + "positive_ic_rate": 0.6, + "resolved_feature_count": 2, + "trial": 7, + } + ] + ).to_csv(selection_path, index=False) + _write_json( + root + / "model_versions" + / "ridge" + / "h5" + / "2026-07-29" + / "manifest.json", + { + "training_start": "2023-01-01", + "training_end": "2026-07-21", + "training_rows": 1000, + "model_config": {"alpha": 10.0}, + "model_selection": { + "report": str(selection_path), + "selection_metric": "top40_raw_excess", + }, + }, + ) + _write_json( + root / "forecasts" / "2026-07-29" / "run_summary.json", + { + "decision_date": "2026-07-29", + "status": "live_forward_20d_cycle_complete", + "matured_forecasts": 0, + "strategy_daily_return_rows": 0, + "tactical_orders": 0, + "model_failures": [], + }, + ) + + summary = { + "experiment_version": "live_forward_horizon_matched_v2", + "decision_date": "2026-07-29", + "primary_horizon_sessions": 20, + "comparison_design": "horizon_matched", + "invested_weight": 1.0, + "rebalance_policy": {"status": "under_audit", "buffer_enabled": False}, + "refit_sessions_by_horizon": {"5": 5, "10": 10, "20": 20}, + "input_audit": {"market_end": "2026-07-29"}, + "universe": {"final_names": 250}, + "survival_validation": {"activated": False}, + } + payload = build_horizon_dashboard_payload(root, summary, pd.DataFrame()) + + assert payload["schema_version"] == 1 + assert len(payload["arms"]) == 1 + arm = payload["arms"][0] + assert arm["selected"] == [ + { + "ticker": "AAA", + "sector": "Technology", + "rank": 1, + "score": 0.03, + "weight": 0.5, + } + ] + assert arm["validation"]["top40_raw_excess"] == 0.012 + assert arm["validation"]["completed_trials"] == 1 + assert arm["explainability"] == { + "status": "available", + "method": "tree_shap", + "rows": 250, + } + assert "private_column" not in json.dumps(payload) + assert "C:/private" not in json.dumps(payload) + assert "model_versions" not in json.dumps(payload) + + +def test_dashboard_html_exposes_v2_information_architecture() -> None: + html = horizon_dashboard_html( + { + "experiment": {"decision_date": "2026-07-29"}, + "rebalance_policy": {}, + "input_audit": {}, + "universe": {}, + "survival_validation": {}, + "arms": [], + "matured": [], + "strategy_returns": [], + "runs": [], + "tactical_orders": [], + } + ) + + assert "Horizon-Matched Live Forward Monitor" in html + assert 'id="decisionSelect"' in html + assert 'id="horizonGrid"' in html + assert "Live Forward Monitor" in html + assert "../live_forward/dashboard.html" in html + assert 'id="strategyProgress"' in html + assert "The decision session is not a holding session" in html + assert "sessions held" in html + assert 'id="modelGrid"' in html + assert 'id="healthGrid"' in html + assert 'id="maturedTable"' in html + assert "never mixed with validation estimates" in html + assert "Show SHAP feature importance" in html + assert "live-forward-horizon-dashboard-v1" in html + assert "width=device-width,initial-scale=1" in html + assert "__DASHBOARD_PAYLOAD__" not in html