Document ensemble backtest results and fail-closed scoring

Author: HanSun103

docs/LIVE_FORWARD_MONITOR.md

@@ -71,6 +71,10 @@ XGBoost also uses fold-safe early stopping:
 3. the outer temporal validation block remains untouched for trial scoring;
 4. the final all-data refit uses the median effective tree count learned across folds.
 
+A trial is now eligible only when every temporal validation fold produces non-constant scores.
+After the final refit, the live snapshot also fails closed if every security receives the same
+score. This prevents tied scores from silently becoming an alphabetical ticker portfolio.
+
 `model_selection.csv` records the requested estimator cap, best iteration, effective estimator
 count, and stopping-window settings for each fold. The aggregate row records the median and range
 of effective trees. Early stopping reduces unnecessary boosting rounds without using the live
@@ -165,3 +169,8 @@ predeclared ensemble, and SPY over many independent sessions, then report active
 drawdown, and uncertainty. Rank IC remains a component-model diagnostic; it is not available for
 the ensemble until paired historical out-of-sample scores are persisted. Model overlap is
 diagnostically useful, but high consensus is neither necessary nor sufficient for outperformance.
+
+The complete historical portfolio comparison is recorded in the
+[Ridge-XGBoost ensemble experiment](experiments/2026-07-27-ridge-xgb-rank-ensemble.md).
+It includes compounding, identical execution costs, SPY, risk metrics, moving-block confidence
+intervals, a shuffled-score placebo, and a final chronological-fold check.

docs/experiments/2026-07-27-ridge-xgb-rank-ensemble.md

@@ -42,3 +42,34 @@ top-tail signal. The live test must determine whether that trade-off improves re
 No claim should be made from the weight-selection objective on all historical folds. The primary
 historical ensemble estimate is the expanding prior-fold result, and the live portfolio is evaluated
 only after it is frozen.
+
+## Complete portfolio backtest
+
+A subsequent portfolio-level test used the same rules for Ridge, XGBoost, and the ensemble:
+
+- 182 weekly observations from 2020-01-10 through 2023-06-30;
+- fully invested equal-weight top 40 with a 25% sector cap;
+- 8 bps per one-way traded weight, a $10 fixed fee per trade, and $1 million starting NAV;
+- adjusted SPY close-to-close returns with a 0.5% annual management fee;
+- 5,000 moving-block bootstrap draws and 100 shuffled-score permutations.
+
+| Portfolio | Net cumulative | Net CAGR | Sharpe | Max drawdown | Weekly turnover |
+|---|---:|---:|---:|---:|---:|
+| SPY | 39.58% | 10.00% | 0.54 | -34.12% | — |
+| Ridge | 103.88% | 22.57% | 0.77 | -27.37% | 142.05% |
+| XGBoost | 190.28% | 35.59% | 0.97 | -39.44% | 111.01% |
+| Ridge-XGBoost ensemble | 110.05% | 23.62% | 0.78 | -35.92% | 117.93% |
+
+The ensemble's gross CAGR was 31.73%, so modeled costs reduced CAGR by 8.11 percentage points.
+Its annualized active return versus SPY was 14.63%, but the 95% moving-block interval was
+[-5.64%, 37.64%]. Its shuffled-score information-ratio p-value was 0.089. These are encouraging
+development statistics, not decisive evidence.
+
+The final chronological fold covered 26 observations from January through June 2023. The ensemble
+returned 9.95%, versus 10.54% for XGBoost and 15.04% for SPY. Its Sharpe of 1.39 and maximum
+drawdown of -6.58% were marginally better than XGBoost's 1.38 and -7.93%, but SPY remained better
+on return, Sharpe, and drawdown in that period.
+
+The final fold is not called a genuine lockbox: it was untouched by each nested model fit, but the
+two-model ensemble was proposed after the v10 results had already been inspected. Survivorship and
+point-in-time data limitations also remain. A new confirmation period is still required.

site/data/dashboard.json

@@ -1,10 +1,10 @@
 {
   "schema_version": 1,
-  "generated_at_utc": "2026-07-27T23:01:45.416516+00:00",
-  "latest_decision_date": "2026-07-27",
-  "snapshot_count": 14,
-  "evaluation_count": 11,
-  "dashboard_sha256": "b33b26ce655ddef3a9eab8150a79977584de64bd7c47a45c7549a32c4ad1f906",
-  "dashboard_bytes": 243869,
+  "generated_at_utc": "2026-07-28T20:51:43.451436+00:00",
+  "latest_decision_date": "2026-07-28",
+  "snapshot_count": 17,
+  "evaluation_count": 14,
+  "dashboard_sha256": "632ee5ddcc24f98828ca332080b7f054c0fb9e8a87c211d143cc675ced32722e",
+  "dashboard_bytes": 305847,
   "source_policy": "sanitized_live_forward_outputs_only"
 }

site/data/manifest.json

@@ -54,6 +54,9 @@ def run_live_forward_snapshot(
     optuna_timeout_seconds: int | None = None,
     selection_metric: str = "top40_raw_excess",
     model_family: str = "auto",
+    preselected_model_params: dict[str, Any] | None = None,
+    preselected_feature_count: int | None = None,
+    preselected_trial: int | None = None,
 ) -> dict[str, Any]:
     """Retrain a live-forward ranker and save today's paper-trading scores."""
     decision_date = pd.Timestamp(as_of).normalize() if as_of else pd.Timestamp.today().normalize()
@@ -113,7 +116,33 @@ def run_live_forward_snapshot(
             label_horizon_sessions,
         )
     )
-    if tune_hyperparameters:
+    recovery_refit = preselected_model_params is not None
+    if recovery_refit:
+        model_name = model_family
+        if model_name not in {"ridge", "xgb_ranker"}:
+            raise ValueError("A recovery refit requires an explicit model family")
+        exclude_prefixes = rank_cfg.get("features", {}).get("exclude_prefixes", [])
+        features = _top_features(train, preselected_feature_count, exclude_prefixes)
+        model_params = dict(preselected_model_params)
+        model = _make_live_model(model_name, model_params).fit(
+            train[features], train["target"]
+        )
+        selection_path = output / "model_selection.csv"
+        tuning = (
+            pd.read_csv(selection_path)
+            if selection_path.exists() else pd.DataFrame()
+        )
+        if preselected_trial is not None and not tuning.empty:
+            chosen = tuning.loc[
+                tuning["row_type"].eq("aggregate")
+                & tuning["trial"].eq(preselected_trial)
+            ]
+            tuning = pd.concat(
+                [chosen, tuning.drop(index=chosen.index)],
+                ignore_index=True,
+            )
+            tuning.to_csv(selection_path, index=False)
+    elif tune_hyperparameters:
         model, model_name, model_params, features, tuning = _select_and_fit_live_model(
             train,
             rank_cfg,
@@ -148,6 +177,12 @@ def run_live_forward_snapshot(
     )
     live_frame["score"] = model.predict(live_frame.reindex(columns=features))
     live_frame = live_frame.replace([np.inf, -np.inf], np.nan).dropna(subset=["score"])
+    live_score_unique = int(live_frame["score"].nunique())
+    if live_score_unique < 2:
+        raise RuntimeError(
+            f"{model_name} produced constant live scores; refusing to freeze an "
+            "arbitrary ticker-ordered portfolio"
+        )
     selected = sector_diversified_top_k(
         live_frame["score"].droplevel("date"),
         live_frame["sector"].droplevel("date"),
@@ -199,9 +234,10 @@ def run_live_forward_snapshot(
         "split_event_cache": str(split_cache_file),
         "split_features_status": "candidate_only_pending_event_study",
         "live_scored_names": int(ranked["ticker"].nunique()),
+        "live_unique_scores": live_score_unique,
         "holdings": int(holdings),
         "model_selection": {
-            "enabled": bool(tune_hyperparameters),
+            "enabled": bool(tune_hyperparameters or recovery_refit),
             "validation_days": int(validation_days) if tune_hyperparameters else None,
             "tuning_folds": int(tuning_folds) if tune_hyperparameters else None,
             "purge_days": (
@@ -216,6 +252,8 @@ def run_live_forward_snapshot(
             "optuna_timeout_seconds": optuna_timeout_seconds if tune_hyperparameters else None,
             "selection_metric": selection_metric if tune_hyperparameters else None,
             "model_family": model_family,
+            "recovery_refit": recovery_refit,
+            "preselected_trial": preselected_trial,
             "search_profile": rank_cfg.get("live_tuning", {}).get("profile", "broad_default"),
             "search_space": rank_cfg.get("live_tuning", {}),
             "report": str(output / "model_selection.csv") if tune_hyperparameters else None,
@@ -1094,7 +1132,10 @@ def objective(trial: Any) -> float:
                 trial.set_user_attr(key, value)
         if selection_metric not in aggregate:
             raise ValueError(f"Unknown live selection metric: {selection_metric}")
-        return float(aggregate[selection_metric])
+        return (
+            float(aggregate[selection_metric])
+            if aggregate["selection_eligible"] else float("-inf")
+        )
 
     sampler = optuna.samplers.TPESampler(seed=42)
     study = optuna.create_study(direction="maximize", sampler=sampler)
@@ -1111,6 +1152,13 @@ def objective(trial: Any) -> float:
         na_position="last",
     ).reset_index(drop=True)
     aggregate_table = table.loc[table["row_type"].eq("aggregate")].copy()
+    aggregate_table = aggregate_table.loc[
+        aggregate_table["selection_eligible"].fillna(False).astype(bool)
+    ]
+    if aggregate_table.empty:
+        raise RuntimeError(
+            "No model-selection trial produced non-constant scores in every fold"
+        )
     aggregate_table = aggregate_table.sort_values(
         [selection_metric, "rank_ic"],
         ascending=False,
@@ -1295,6 +1343,14 @@ def _make_live_model(
 def _aggregate_trial_rows(rows: list[dict[str, Any]], trial: int) -> dict[str, Any]:
     frame = pd.DataFrame(rows)
     first = rows[0]
+    validation_score_dates = (
+        frame["validation_score_dates"]
+        if "validation_score_dates" in frame else pd.Series(1, index=frame.index)
+    )
+    constant_score_dates = (
+        frame["constant_score_dates"]
+        if "constant_score_dates" in frame else pd.Series(0, index=frame.index)
+    )
     aggregate = {
         "row_type": "aggregate",
         "trial": trial,
@@ -1313,7 +1369,18 @@ def _aggregate_trial_rows(rows: list[dict[str, Any]], trial: int) -> dict[str, A
         "top40_target_excess": float(frame["top40_target_excess"].mean()),
         "validation_start": str(frame["validation_start"].min()),
         "validation_end": str(frame["validation_end"].max()),
+        "fold_count": int(len(frame)),
+        "valid_rank_ic_folds": int(frame["rank_ic"].notna().sum()),
+        "constant_score_dates": int(constant_score_dates.sum()),
+        "score_date_coverage": float(
+            (validation_score_dates - constant_score_dates).sum()
+            / max(1, validation_score_dates.sum())
+        ),
     }
+    aggregate["selection_eligible"] = bool(
+        aggregate["valid_rank_ic_folds"] == aggregate["fold_count"]
+        and aggregate["constant_score_dates"] == 0
+    )
     if first["model"] == "xgb_ranker" and "effective_n_estimators" in frame:
         effective = pd.to_numeric(
             frame["effective_n_estimators"],
@@ -1376,6 +1443,7 @@ def _validation_row(
     holdings: int,
 ) -> dict[str, Any]:
     daily_ic = _daily_rank_ic(validation["target"], scores)
+    daily_unique = scores.groupby(level="date").nunique()
     top_raw = _topk_daily_mean(validation["raw_forward_return"], scores, holdings)
     universe_raw = validation["raw_forward_return"].groupby(level="date").mean().mean()
     top_target = _topk_daily_mean(validation["target"], scores, holdings)
@@ -1387,6 +1455,9 @@ def _validation_row(
         "resolved_feature_count": len(feature_names),
         "rank_ic": mean_rank_ic(validation["target"], scores),
         "positive_ic_rate": float(daily_ic.gt(0.0).mean()) if not daily_ic.empty else np.nan,
+        "validation_score_dates": int(len(daily_unique)),
+        "constant_score_dates": int(daily_unique.le(1).sum()),
+        "score_date_coverage": float(daily_unique.gt(1).mean()),
         "top40_raw_forward_return": float(top_raw),
         "universe_raw_forward_return": float(universe_raw),
         "top40_raw_excess": float(top_raw - universe_raw),