Add fold-safe early stopping for XGBoost ranking

Author: HanSun103

README.md

@@ -271,6 +271,12 @@ model selection; they do not by themselves prove that a model will perform bette
 this expensive profile for feature/schema changes and periodic audits. Ordinary daily runs can use
 the validated focused ranges with smaller trial budgets; every run retains purged temporal folds.
 
+XGBoost trials use fold-safe early stopping. The final 21 dates inside each fold's training block
+form an inner stopping window; the outer validation fold remains untouched for model scoring.
+Training stops after 50 rounds without improvement, and the final all-history refit uses the median
+effective tree count learned across the temporal folds. The tuning CSV records requested trees,
+best iteration, and effective trees so runtime savings and boundary hits remain auditable.
+
 The [live-forward operating guide](docs/LIVE_FORWARD_MONITOR.md) documents the close-data guard,
 frozen-snapshot contract, dashboard timestamps, model-overlap metrics, and generated artifacts.
 The [GitHub Pages deployment guide](docs/GITHUB_PAGES_DEPLOYMENT.md) documents the public-data

configs/experiments/sparse_rank_v3_fnspid_2018_2023.yaml

@@ -46,6 +46,8 @@ residual_rank_v2:
     xgb_reg_lambda_max: 20.0
     xgb_gamma_min: 0.0
     xgb_gamma_max: 1.5
+    xgb_early_stopping_rounds: 50
+    xgb_early_stopping_validation_days: 21
   nested_validation:
     outer_min_train_weeks: 104
     outer_test_weeks: 26

docs/LIVE_FORWARD_MONITOR.md

@@ -61,6 +61,18 @@ may use fewer trials inside the same validated ranges; repeat the five-fold audi
 feature/data change or on a predeclared schedule. Any future warm-start or pruning optimization must
 preserve the same temporal validation and frozen-snapshot rules.
 
+XGBoost also uses fold-safe early stopping:
+
+1. the last 21 dates of each fold's training block are reserved as an inner stopping window;
+2. the ranker stops after 50 rounds without validation-NDCG improvement;
+3. the outer temporal validation block remains untouched for trial scoring;
+4. the final all-data refit uses the median effective tree count learned across folds.
+
+`model_selection.csv` records the requested estimator cap, best iteration, effective estimator
+count, and stopping-window settings for each fold. The aggregate row records the median and range
+of effective trees. Early stopping reduces unnecessary boosting rounds without using the live
+decision date or its future return.
+
 ## Point-in-time and leakage controls
 
 - A portfolio is evaluated only after its holdings have been frozen.

docs/experiments/2026-07-24-sector-split-data-repair.md

@@ -115,3 +115,13 @@ The full five-fold profile is appropriate after a feature/data change and for pe
 is too expensive to repeat unchanged every day. Routine daily runs should use smaller trial budgets
 inside the validated ranges; persisted Optuna studies, pruning, and cached fold matrices are the
 next compute improvements.
+
+## Daily XGBoost early stopping
+
+After the July 27 routine retrain still spent approximately one hour in the nonlinear stage,
+fold-safe early stopping was activated. Each fold now reserves the last 21 training dates for
+stopping only, uses 50-round patience, and keeps the outer validation period untouched for scoring.
+The final model is refit on all eligible training rows with the median best tree count from the
+folds. Requested and effective estimator counts are persisted in the tuning report. The next daily
+run must confirm the realized tree reduction and wall-clock saving; no performance or speed claim
+is made from the implementation alone.

src/experiments/live_forward_test.py

@@ -753,6 +753,13 @@ def _select_and_fit_live_model(
     if not splits:
         raise ValueError("Not enough labeled dates for live model selection")
     exclude_prefixes = rank_cfg.get("features", {}).get("exclude_prefixes", [])
+    live_tuning = rank_cfg.get("live_tuning", {})
+    xgb_early_stopping_rounds = int(
+        live_tuning.get("xgb_early_stopping_rounds", 0)
+    )
+    xgb_early_stopping_validation_days = int(
+        live_tuning.get("xgb_early_stopping_validation_days", 21)
+    )
     feature_choices = _candidate_feature_counts(rank_cfg)
     if None not in feature_choices:
         feature_choices.append(None)
@@ -788,10 +795,28 @@ def objective(trial: Any) -> float:
             cached_features,
         ) in enumerate(fold_inputs):
             features = cached_features[feature_count]
-            model = _make_live_model(model_name, model_params).fit(
-                tuning_train[features],
-                tuning_train["target"],
-            )
+            fit_train = tuning_train
+            stopping_validation: pd.DataFrame | None = None
+            fitted_params = dict(model_params)
+            if model_name == "xgb_ranker" and xgb_early_stopping_rounds > 0:
+                fit_train, stopping_validation = _split_xgb_early_stopping_window(
+                    tuning_train,
+                    validation_days=xgb_early_stopping_validation_days,
+                )
+                fitted_params["early_stopping_rounds"] = xgb_early_stopping_rounds
+            model = _make_live_model(model_name, fitted_params)
+            if stopping_validation is None:
+                model.fit(
+                    fit_train[features],
+                    fit_train["target"],
+                )
+            else:
+                model.fit(
+                    fit_train[features],
+                    fit_train["target"],
+                    stopping_validation[features],
+                    stopping_validation["target"],
+                )
             scores = pd.Series(
                 model.predict(validation[features]),
                 index=validation.index,
@@ -813,6 +838,16 @@ def objective(trial: Any) -> float:
                 "validation_start": validation_dates.min().strftime("%Y-%m-%d"),
                 "validation_end": validation_dates.max().strftime("%Y-%m-%d"),
             })
+            if model_name == "xgb_ranker":
+                row.update({
+                    "requested_n_estimators": int(model_params["n_estimators"]),
+                    "best_iteration": model.best_iteration_,
+                    "effective_n_estimators": model.effective_n_estimators_,
+                    "early_stopping_rounds": xgb_early_stopping_rounds,
+                    "early_stopping_validation_days": (
+                        xgb_early_stopping_validation_days
+                    ),
+                })
             fold_rows.append(row)
         aggregate = _aggregate_trial_rows(fold_rows, trial.number)
         trial_rows.extend(fold_rows)
@@ -851,6 +886,11 @@ def objective(trial: Any) -> float:
     best_feature_count = None if pd.isna(best["feature_count"]) else int(best["feature_count"])
     final_features = _top_features(train, best_feature_count, exclude_prefixes)
     final_params = json.loads(best["model_params_json"])
+    if str(best["model"]) == "xgb_ranker":
+        effective_estimators = best.get("median_effective_n_estimators")
+        if effective_estimators is not None and pd.notna(effective_estimators):
+            final_params["n_estimators"] = max(1, int(round(effective_estimators)))
+        final_params.pop("early_stopping_rounds", None)
     final_model = _make_live_model(str(best["model"]), final_params).fit(
         train[final_features],
         train["target"],
@@ -862,6 +902,31 @@ def objective(trial: Any) -> float:
     return final_model, str(best["model"]), final_params, final_features, table
 
 
+def _split_xgb_early_stopping_window(
+    training: pd.DataFrame,
+    *,
+    validation_days: int,
+    min_fit_days: int = 126,
+) -> tuple[pd.DataFrame, pd.DataFrame]:
+    """Reserve a trailing inner window without touching the outer validation fold."""
+    dates = pd.DatetimeIndex(
+        training.index.get_level_values("date").unique()
+    ).sort_values()
+    stopping_days = max(1, int(validation_days))
+    if len(dates) < min_fit_days + stopping_days:
+        raise ValueError(
+            "Not enough training dates for the XGBoost early-stopping window"
+        )
+    stopping_dates = dates[-stopping_days:]
+    fit_dates = dates[:-stopping_days]
+    date_index = training.index.get_level_values("date")
+    fit = training.loc[date_index.isin(fit_dates)]
+    stopping = training.loc[date_index.isin(stopping_dates)]
+    if fit.empty or stopping.empty:
+        raise ValueError("XGBoost early-stopping split produced an empty frame")
+    return fit, stopping
+
+
 def _candidate_feature_counts(rank_cfg: dict[str, Any]) -> list[int | None]:
     raw = rank_cfg["models"].get("feature_counts", ["all"])
     counts: list[int | None] = []
@@ -1015,6 +1080,27 @@ def _aggregate_trial_rows(rows: list[dict[str, Any]], trial: int) -> dict[str, A
         "validation_start": str(frame["validation_start"].min()),
         "validation_end": str(frame["validation_end"].max()),
     }
+    if first["model"] == "xgb_ranker" and "effective_n_estimators" in frame:
+        effective = pd.to_numeric(
+            frame["effective_n_estimators"],
+            errors="coerce",
+        ).dropna()
+        best_iterations = pd.to_numeric(
+            frame["best_iteration"],
+            errors="coerce",
+        ).dropna()
+        if not effective.empty:
+            aggregate.update({
+                "requested_n_estimators": int(first["requested_n_estimators"]),
+                "median_best_iteration": float(best_iterations.median()),
+                "median_effective_n_estimators": float(effective.median()),
+                "min_effective_n_estimators": int(effective.min()),
+                "max_effective_n_estimators": int(effective.max()),
+                "early_stopping_rounds": int(first["early_stopping_rounds"]),
+                "early_stopping_validation_days": int(
+                    first["early_stopping_validation_days"]
+                ),
+            })
     return aggregate
 
 

src/models/rank_models.py

@@ -63,6 +63,8 @@ def __init__(self, **params: Any) -> None:
         self.params = params
         self.model: Any = None
         self.columns: list[str] = []
+        self.best_iteration_: int | None = None
+        self.effective_n_estimators_: int | None = None
 
     def fit(
         self,
@@ -119,6 +121,17 @@ def fit(
                 "eval_group": [_date_groups(validation.index)],
             })
         self.model.fit(x_numeric, y, **fit_kwargs)
+        configured_estimators = int(defaults.get("n_estimators", 200))
+        best_iteration = getattr(self.model, "best_iteration", None)
+        if best_iteration is None:
+            self.best_iteration_ = configured_estimators - 1
+            self.effective_n_estimators_ = configured_estimators
+        else:
+            self.best_iteration_ = int(best_iteration)
+            self.effective_n_estimators_ = min(
+                configured_estimators,
+                self.best_iteration_ + 1,
+            )
         return self
 
     def predict(self, x: pd.DataFrame) -> np.ndarray:

tests/test_double_ensemble_v10.py

@@ -51,6 +51,13 @@ def test_xgb_ranker_supports_more_than_31_cross_sectional_grades() -> None:
     features = pd.DataFrame({"signal": values}, index=index)
     target = pd.Series(values, index=index)
     model = XGBRankModel(
-        n_estimators=5, max_depth=1, tree_method="hist", n_jobs=1
+        n_estimators=20,
+        max_depth=1,
+        tree_method="hist",
+        n_jobs=1,
+        early_stopping_rounds=3,
     ).fit(features, target, features, target)
     assert np.unique(model.predict(features)).size > 1
+    assert model.best_iteration_ is not None
+    assert model.effective_n_estimators_ is not None
+    assert 1 <= model.effective_n_estimators_ <= 20

tests/test_live_model_selection.py

@@ -0,0 +1,69 @@
+import pandas as pd
+
+from src.experiments.live_forward_test import (
+    _aggregate_trial_rows,
+    _split_xgb_early_stopping_window,
+)
+
+
+def test_xgb_early_stopping_window_precedes_outer_validation() -> None:
+    dates = pd.date_range("2024-01-02", periods=170, freq="B")
+    index = pd.MultiIndex.from_product(
+        [dates, ["A", "B"]],
+        names=["date", "ticker"],
+    )
+    training = pd.DataFrame(
+        {
+            "signal": range(len(index)),
+            "target": range(len(index)),
+        },
+        index=index,
+    )
+
+    fit, stopping = _split_xgb_early_stopping_window(
+        training,
+        validation_days=21,
+    )
+
+    fit_dates = pd.DatetimeIndex(fit.index.get_level_values("date").unique())
+    stopping_dates = pd.DatetimeIndex(
+        stopping.index.get_level_values("date").unique()
+    )
+    assert len(stopping_dates) == 21
+    assert fit_dates.max() < stopping_dates.min()
+    assert len(fit_dates) == 149
+
+
+def test_xgb_trial_aggregation_records_median_effective_tree_count() -> None:
+    rows = []
+    for fold, effective in enumerate([120, 160, 140, 180, 100]):
+        rows.append(
+            {
+                "model": "xgb_ranker",
+                "model_params_json": '{"n_estimators": 700}',
+                "feature_count": 22,
+                "resolved_feature_count": 22,
+                "rank_ic": 0.01,
+                "positive_ic_rate": 0.6,
+                "top40_raw_forward_return": 0.002,
+                "universe_raw_forward_return": 0.001,
+                "top40_raw_excess": 0.001,
+                "top40_target_mean": 0.01,
+                "universe_target_mean": 0.0,
+                "top40_target_excess": 0.01,
+                "validation_start": f"2025-0{fold + 1}-01",
+                "validation_end": f"2025-0{fold + 1}-28",
+                "requested_n_estimators": 700,
+                "best_iteration": effective - 1,
+                "effective_n_estimators": effective,
+                "early_stopping_rounds": 50,
+                "early_stopping_validation_days": 21,
+            }
+        )
+
+    aggregate = _aggregate_trial_rows(rows, trial=3)
+
+    assert aggregate["median_effective_n_estimators"] == 140.0
+    assert aggregate["median_best_iteration"] == 139.0
+    assert aggregate["min_effective_n_estimators"] == 100
+    assert aggregate["max_effective_n_estimators"] == 180