Update project implementation and supporting files

Author: HanSun103

.gitignore

@@ -10,6 +10,7 @@ __pycache__/
 *.pyd
 *.so
 .pytest_cache/
+.pytest_candidates*/
 .mypy_cache/
 .ruff_cache/
 tmp/

README.md

@@ -155,14 +155,38 @@ flowchart LR
 
 See [Qlib and ML winner-selection research notes](docs/research/2026-07-17-qlib-winner-selection-research.md).
 
+The controlled U.S. replication is now implemented with the exact 158-feature contract, a
+Qlib-style LightGBM regressor, raw and sector-neutral residual targets, identical temporal folds,
+and a fixed 40-stock portfolio. Build the cached feature panel and run all four arms with:
+
+```bash
+pip install -e ".[tuning]"
+python -m src.cli build-qlib-alpha158-dataset
+python -m src.cli run-qlib-alpha158-comparison
+python -m src.cli run-alpha158-feature-ablation
+python -m src.cli run-double-ensemble-v10 \
+  --output reports/experiments/double_ensemble_v10_consensus
+```
+
+The Ridge-controlled feature ablation found no verified incremental Alpha158 benefit: adding
+Alpha158 raised mean Rank IC from 0.0102 to 0.0114, but the paired confidence interval included
+zero and top-40 excess return fell from 0.1006% to 0.0646% per week. See the
+[Alpha158 U.S. experiment record](docs/experiments/2026-07-20-qlib-alpha158-us.md).
+
+The model-specific v10 study gave Ridge, LightGBM, and XGBoost independent Optuna searches and
+fold-safe shuffle-ablation selection. XGBoost achieved the strongest top-40 point estimate
+(0.2121% weekly excess) but its paired confidence interval included zero. LightGBM and the
+validation-weighted ensemble failed out of sample, so the ensemble remains disabled. See the
+[v10 experiment record](docs/experiments/2026-07-20-double-ensemble-v10.md).
+
 ## Key Modules
 
 | Module | Role |
 |---|---|
 | `src/data` | Market data, macro data, point-in-time universe, price repair, and caching |
-| `src/features` | Technical, cross-sectional, macro, and NLP feature generation |
+| `src/features` | Technical, Alpha158, cross-sectional, macro, and NLP feature generation |
 | `src/targets` | Weekly residual-return ranking targets |
-| `src/models` | Ridge and XGBoost ranking models |
+| `src/models` | Ridge, XGBoost ranking, and LightGBM cross-sectional models |
 | `src/portfolio` | Sparse ranking, active tilts, risk models, and costs |
 | `src/backtest` | Walk-forward execution, accounting, and metrics |
 | `src/nlp` | FinBERT, topic, event, and text-source pipelines |
@@ -220,6 +244,26 @@ the latest close, and a frozen paper portfolio plus `model_selection.csv` are sa
 next-session returns; otherwise the forward test stops being clean. This daily proxy is designed to
 collect evidence faster than waiting ten calendar weeks for ten weekly rebalance observations.
 
+For the recurrent after-close experiment, use the one-call daily pipeline instead of running the
+monitor, two model families, and dashboard separately:
+
+```bash
+python -m src.cli run-live-forward-daily --as-of 2026-07-22
+```
+
+The [live-forward operating guide](docs/LIVE_FORWARD_MONITOR.md) documents the close-data guard,
+frozen-snapshot contract, dashboard timestamps, model-overlap metrics, and generated artifacts.
+
+The daily command first refreshes the latest prior Ridge and XGBoost portfolios against the
+completed session. Yahoo's final hourly bar begins at 15:30, so the command requires that bar and
+retries five times at 60-second intervals if it is not available yet. It then requires an official
+daily price row for the decision date, independently tunes and refits Ridge and XGBoost, saves both
+new top-40 portfolios, writes `selection_stability_YYYY-MM-DD.json`, and rebuilds the shared
+dashboard. The stability report tracks same-day model overlap, day-over-day retention, top-10
+persistence, rank correlation, entrants/exits, implied replacement, and multi-day consensus names.
+This ordering prevents a successful model retrain from leaving the performance chart stuck at an
+earlier intraday bar.
+
 The July 20, 2026 production-style run requested 25 Optuna trials with a 600-second timeout. Ten
 trials completed, using three 42-session validation folds separated from training by a one-session
 purge. The selected model was Ridge with `alpha=6156.9973`, aggregate validation Rank IC of
@@ -261,9 +305,10 @@ Build the static monitoring dashboard:
 python -m src.cli build-live-dashboard
 ```
 
-The dashboard reads every saved snapshot under `reports/live_forward/`, supports changing the
-snapshot and evaluation horizon, and shows portfolio return, benchmark return, active return,
-stock-level winners/losers versus SPY, model-selection evidence, and experiment history.
+The dashboard reads every saved snapshot under `reports/live_forward/`, overlays Ridge, XGBoost,
+and SPY on one multi-session chart, provides date-range controls, identifies the next frozen
+portfolios whose returns are pending, and shows human-readable model and stock-level evidence. It
+is refreshed by the daily pipeline, snapshot, intraday-report, and evaluation commands.
 
 Build the earlier conservative final-reference package:
 

configs/experiments/double_ensemble_v10.yaml

@@ -0,0 +1,42 @@
+experiment:
+  id: double_ensemble_v10_development
+  status: development_only
+  description: Fully tuned member models with fold-safe shuffle feature selection
+
+residual_rank_v2:
+  seed: 42
+  tuning:
+    trials_per_model: 15
+    timeout_seconds_per_model: 900
+    feature_counts: [20, 40, 60, 80, 120]
+    max_feature_candidates: 120
+    fold_stability_penalty: 0.25
+  ensemble:
+    shrink_to_equal: 0.50
+  nested_validation:
+    outer_min_train_weeks: 104
+    outer_test_weeks: 26
+    inner_folds: 3
+    inner_min_train_weeks: 52
+    inner_validation_weeks: 13
+    purge_weeks: 1
+  portfolio:
+    holding_candidates: [40]
+    max_sector_fraction: 0.25
+    invested_weight: 0.70
+    active_l1_budget: 0.10
+    max_active_weight: 0.01
+    calibration_shrink_observations: 500
+    starting_capital: 1000000
+    one_way_cost_bps: 8.0
+    fixed_fee_per_trade: 10.0
+    minimum_trade_weight: 0.0025
+  statistics:
+    permutation_samples: 100
+    moving_block_weeks: 8
+    bootstrap_samples: 5000
+  diagnostics:
+    top_k: 40
+  lockbox:
+    opened: false
+    policy: development comparison only

configs/experiments/qlib_alpha158_us_development.yaml

@@ -0,0 +1,52 @@
+experiment:
+  id: qlib_alpha158_us_development
+  manifest_version: 1
+  status: development_only
+  description: Controlled Alpha158-LightGBM target comparison on repaired U.S. PIT rows
+  development_end: "2023-12-31"
+
+residual_rank_v2:
+  models:
+    feature_counts: [20, 40, all]
+    ridge_alphas: [10.0, 100.0]
+    xgb_candidates: []
+    # Qlib's published Alpha158 LightGBM shape, expressed with sklearn names.
+    lightgbm_candidates:
+      - n_estimators: 500
+        learning_rate: 0.2
+        colsample_bytree: 0.8879
+        subsample: 0.8789
+        subsample_freq: 1
+        reg_alpha: 205.6999
+        reg_lambda: 580.9768
+        max_depth: 8
+        num_leaves: 210
+        n_jobs: 4
+        random_state: 42
+  nested_validation:
+    outer_min_train_weeks: 104
+    outer_test_weeks: 26
+    inner_folds: 2
+    inner_min_train_weeks: 52
+    inner_validation_weeks: 13
+    purge_weeks: 1
+  portfolio:
+    holding_candidates: [40]
+    max_sector_fraction: 0.25
+    invested_weight: 0.70
+    active_l1_budget: 0.10
+    max_active_weight: 0.01
+    calibration_shrink_observations: 500
+    starting_capital: 1000000
+    one_way_cost_bps: 8.0
+    fixed_fee_per_trade: 10.0
+    minimum_trade_weight: 0.0025
+  statistics:
+    permutation_samples: 100
+    moving_block_weeks: 8
+    bootstrap_samples: 5000
+  diagnostics:
+    top_k: 40
+  lockbox:
+    opened: false
+    policy: development comparison only; no confirmation or lockbox data

docs/LIVE_FORWARD_MONITOR.md

@@ -0,0 +1,90 @@
+# Live-Forward Monitoring
+
+## Purpose
+
+The live-forward experiment tests whether two independently tuned ranking models can select stocks
+using only information available at each decision time. It is an evidence-collection workflow, not
+yet a production trading recommendation. Ridge is the stable linear benchmark and the XGBoost
+ranker is the nonlinear challenger. Both select 40 equal-weight names and share SPY as the market
+benchmark.
+
+## Daily after-close workflow
+
+Run the production-style workflow after Yahoo Finance publishes the final regular-session bar:
+
+```bash
+python -m src.cli run-live-forward-daily --as-of YYYY-MM-DD
+```
+
+The command performs these steps in order:
+
+1. Incrementally refreshes market data and evaluates portfolios frozen before the current session.
+2. Waits for the final 15:30 hourly bar and verifies the official daily close is available.
+3. Rebuilds price features through the decision date.
+4. Builds forward-return training labels only where the future outcome was already known before the
+   decision date.
+5. Tunes Ridge and XGBoost independently on temporal validation data, refits the selected
+   configurations, and freezes both top-40 lists for the next evaluation session.
+6. Writes model-agreement and day-over-day selection-stability diagnostics.
+7. Rebuilds `reports/live_forward/dashboard.html`.
+
+If final market data is unavailable, the command retries and then fails closed. It must not create a
+snapshot from a partial trading day.
+
+## Point-in-time and leakage controls
+
+- A portfolio is evaluated only after its holdings have been frozen.
+- Same-session partial prices cannot enter model fitting or stock selection.
+- Training labels end early enough that their complete forward horizon is known at the decision
+  time.
+- Hyperparameters and feature counts are selected using temporal validation, not the live session.
+- Ridge and XGBoost are displayed together; neither is chosen retrospectively using realized live
+  returns.
+- The July 20 session is a setup/test observation. The official comparison baseline begins at the
+  July 20 SPY close, and portfolio performance begins on July 21.
+
+## Dashboard time conventions
+
+Yahoo's hourly timestamps identify the **start** of each bar. The dashboard displays the corresponding
+bar-end time, so a Yahoo 15:30 bar appears as 16:00. Daily compounded returns are anchored to the
+previous official close, not to the session open.
+
+The date controls aggregate every completed observation in the selected range. Because the top-40
+holdings can change each day, each session uses the list frozen before that session.
+
+## Selection-stability metrics
+
+All dashboard insights are deterministic calculations; no LLM API is required.
+
+For two 40-stock sets, A and B:
+
+- **Overlap rate:** `|A intersection B| / 40`
+- **Jaccard similarity:** `|A intersection B| / |A union B|`
+- **Model disagreement:** `1 - overlap rate`
+- **Day-over-day replacement:** `1 - |A_today intersection A_prior| / 40`
+- **Rank stability:** Spearman correlation of ranks among names retained on both dates
+- **Average rank movement:** mean absolute rank change among retained names
+
+The dashboard also reports top-10 retention, entrants, exits, current Ridge-XGBoost consensus, and
+persistent consensus across the available sessions. These describe model agreement and stability;
+they do not by themselves prove forecast quality.
+
+## Generated artifacts
+
+Generated outputs remain under the ignored `reports/live_forward/` directory:
+
+- dated Ridge and XGBoost snapshot directories with manifests, tuning results, predictions, and
+  selected holdings;
+- intraday and close-to-close evaluation files for each frozen portfolio;
+- `selection_stability_YYYY-MM-DD.json`;
+- `dashboard.html`.
+
+Keep source code, configurations, tests, and documentation in Git. Do not commit cached prices,
+trained artifacts, generated reports, credentials, or temporary pytest directories.
+
+## Interpretation boundary
+
+The live sample is currently too short for a performance claim. Compare Ridge, XGBoost, and SPY over
+many independent sessions, then report active return, Rank IC, turnover, drawdown, and uncertainty.
+Model overlap is diagnostically useful, but high consensus is neither necessary nor sufficient for
+outperformance.

docs/README.md

@@ -5,6 +5,7 @@
 - [Final research-reference version](FINAL_VERSION.md)
 - [2026-07-16 consolidation release notes](RELEASE_NOTES_2026-07-16.md)
 - [GitHub push plan](GITHUB_PUSH_PLAN.md)
+- [Live-forward monitoring and daily operations](LIVE_FORWARD_MONITOR.md)
 
 ## Branch Policy
 
@@ -43,6 +44,8 @@ decision explicitly says they are active.
 
 ## Experiment Logs
 
+- [2026-07-20 - Model-specific tuning and ensemble v10](experiments/2026-07-20-double-ensemble-v10.md)
+- [2026-07-20 - Qlib Alpha158 and LightGBM U.S. comparison](experiments/2026-07-20-qlib-alpha158-us.md)
 - [2026-07-17 - Survival-risk filter diagnostic for v8](experiments/2026-07-17-survival-filter-v8.md)
 - [2026-07-17 - Repaired rank-40 ML stock-selection evidence](experiments/2026-07-17-repaired-rank40-selection.md)
 - [2026-07-16 - Transparent smart-universe experiment v7](experiments/2026-07-16-smart-universe-v7.md)