Update for 2026gal: event key, bug fixes, prior event DPR/fouls blending

ejmccalla · claude · ejmccalla · commit e95538c521fb · 2026-04-30T07:47:03.000-05:00
- schedule_picklist.ps1: update event key to 2026gal, start date 2026-04-29
- tba.py: fix prior event selection to skip events with no COPR/OPR data;
  add get_prior_event_oprs() for per-team prior DPR fallback
- frc2026_picklist_analysis.py: fix per-team EPA weight redistribution when
  prior COPR missing; add blended foulPoints and dprs to picklist output;
  fix Unicode arrow in print statement
- frc2026_picklist_runner.py: wire in prior_oprs_df; fix first_event_teams
  to use teams_with_prior set instead of prior_coprs_df index
- .gitignore: exclude *.csv and *.txt local cache files
- CLAUDE.md: add initial codebase documentation

Co-Authored-By: Claude Sonnet 4.6 &lt;noreply@anthropic.com&gt;
diff --git a/.gitignore b/.gitignore
@@ -161,5 +161,9 @@ cython_debug/
 
 .ps_history
 
+# Local data cache
+*.csv
+*.txt
+
 # Claude Code memory
 memory/
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1,97 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Commands
+
+**Setup (first time):**
+```bash
+setup.bat        # Creates venv, installs deps, installs pre-commit hooks
+```
+
+**Run analysis:**
+```bash
+run_picklist --event_key <key>                   # Basic run (e.g. 2026ndgf)
+run_picklist --event_key <key> --save            # Save event-specific JSON to GitHub
+run_picklist --event_key <key> --save --post     # Also update latest_*.json (what webapp reads)
+run_picklist --event_key <key> --force           # Re-fetch all cached data
+run_picklist --event_key <key> --teams 4607 1234 # Analyze specific teams only
+```
+
+**Scheduling during events (Windows):**
+```powershell
+schedule_picklist.ps1     # Register scheduled task (every 30 min, 9am–7pm)
+unschedule_picklist.ps1   # Remove scheduled task
+```
+
+**Linting/formatting:**
+```bash
+ruff format src/          # Format Python
+ruff check src/ --fix     # Lint with auto-fix
+```
+Pre-commit hooks run both automatically on every commit.
+
+## Architecture
+
+This is a Python pipeline that pulls scouting data from multiple sources, blends them into team rankings, and publishes JSON to GitHub where a static webapp reads it.
+
+**Entry point:** `src/scouting_analysis/frc2026_picklist_runner.py` → `main()` (registered as `run_picklist` CLI command in `pyproject.toml`)
+
+### Data Sources
+
+| Module | Source | What it provides |
+|--------|--------|-----------------|
+| `tba.py` | The Blue Alliance API | Team list, match schedule, match breakdowns, COPR, OPR |
+| `sb.py` | Statbotics API | EPA (Expected Points Added) per phase per team |
+| `sdb.py` | `api2.lanzersys.com` (4607 internal) | Raw scouting observations, pit data (hopper size) |
+
+All fetched data is cached locally as CSV files and reused on subsequent runs. `--force` bypasses the cache.
+
+### Core Scoring (frc2026_picklist_analysis.py)
+
+`FRC2026PicklistAnalysis` blends data with **dynamic weights** that shift as the event progresses:
+
+```
+alpha = played_quals / total_quals          # 0.0 → 1.0 through the event
+copr_weight = alpha * 0.75                  # 0% → 75%
+epa_weight  = 1.0 - copr_weight            # 100% → 25%
+
+AUTO/TELEOP score  = 10% scouting + copr_weight*COPR + epa_weight*EPA
+ENDGAME score      = 50% TBA climb data + 50% EPA endgame
+```
+
+This means early in an event the score is EPA-dominated (prediction); late in the event it's COPR-dominated (observed).
+
+### Output & Publishing
+
+The runner generates four JSON files and pushes them to this GitHub repo via REST API:
+- `webapp/{event_key}_picklist.json` / `webapp/{event_key}_planner.json` — historical per-event
+- `webapp/latest_picklist.json` / `webapp/latest_planner.json` — what the webapp actively reads (`--post` flag)
+
+GitHub Actions (`.github/workflows/notify_slack.yaml`) detects pushes to `webapp/latest_*.json` and sends a Slack notification.
+
+### Webapp
+
+Static HTML/CSS/JS in `webapp/` — no build step. `planner.html` and `picklist.html` fetch JSON from `raw.githubusercontent.com`. Changes to HTML deploy immediately on push.
+
+## Environment
+
+Requires a `.env` file (never committed) with:
+```
+X-TBA-Auth-Key=<TBA personal API key>
+GITHUB_TOKEN=<GitHub PAT with repo write access>
+WORKSPACE=<absolute path to project root>
+PYTHONPYCACHEPREFIX=C:\Windows\Temp
+```
+
+## Key Implementation Details
+
+**CSV parsing in `sdb.py`:** The internal scouting database returns malformed CSV (rows split across lines, extra commas in text fields). The parser explicitly handles both cases: stitching split rows together and truncating over-wide rows at the comments column.
+
+**Climb scoring:** Auto=15pts (any level), Endgame Level1=10/Level2=20/Level3=30pts.
+
+**Prior event COPRs:** For teams with prior events, the runner fetches and averages their previous COPR values as a fallback when current-event data is sparse.
+
+**GitHub push:** Uses the GitHub Contents API (requires fetching the existing file's SHA before updating).
+
+**NaN sanitization:** All floats are sanitized before `json.dumps` because `NaN` is invalid JSON.
diff --git a/schedule_picklist.ps1 b/schedule_picklist.ps1
@@ -1,11 +1,11 @@
 $root = "C:\Users\ejmcc\Documents\FRC4607\Scouting-Analysis"
 
 $action = New-ScheduledTaskAction -Execute "cmd.exe" `
-  -Argument "/c cd /d $root && venv\Scripts\run_picklist.exe --event_key 2026mnmi2 --save --post" `
+  -Argument "/c cd /d $root && venv\Scripts\run_picklist.exe --event_key 2026gal --save --post" `
   -WorkingDirectory $root
 
 $trigger = New-ScheduledTaskTrigger -RepetitionInterval (New-TimeSpan -Minutes 30) `
-  -Once -At "2026-04-10 09:00" -RepetitionDuration (New-TimeSpan -Hours 10)
+  -Once -At "2026-04-29 09:00" -RepetitionDuration (New-TimeSpan -Hours 10)
 
 Register-ScheduledTask -TaskName "FRC4607-Picklist" -Action $action -Trigger $trigger
 
diff --git a/src/scouting_analysis/frc2026_picklist_analysis.py b/src/scouting_analysis/frc2026_picklist_analysis.py
@@ -23,6 +23,9 @@ class FRC2026PicklistAnalysis:
         pits_df (pd.DataFrame): Pits database rows for the event.
         coprs_df (pd.DataFrame): TBA COPR data for the event.
         epa_df (pd.DataFrame): Statbotics EPA data for the event.
+        prior_coprs_df (pd.DataFrame): COPR data from each team's most recent prior event.
+        oprs_df (pd.DataFrame): TBA OPR/DPR data for the current event.
+        prior_oprs_df (pd.DataFrame): OPR/DPR data from each team's most recent prior event.
     """
 
     def __init__(
@@ -34,13 +37,17 @@ def __init__(
         coprs_df: pd.DataFrame,
         epa_df: pd.DataFrame = pd.DataFrame(),
         prior_coprs_df: pd.DataFrame = pd.DataFrame(),
+        oprs_df: pd.DataFrame = pd.DataFrame(),
+        prior_oprs_df: pd.DataFrame = pd.DataFrame(),
     ) -> None:
         self.scouting_df = scouting_df
         self.metric = metric
         self.pits_df = pits_df
         self.coprs_df = coprs_df
         self.epa_df = epa_df
         self.prior_coprs_df = prior_coprs_df
+        self.oprs_df = oprs_df
+        self.prior_oprs_df = prior_oprs_df
 
         # Dynamic weights based on quals played.
         # With prior COPR: EPA fixed at 25%, prior→current COPR share the remaining 75%.
@@ -66,7 +73,7 @@ def __init__(
 
         print(
             f"\nMatch progress: {played}/{total} quals played (alpha={alpha:.2f})"
-            f" → current_copr={self.current_copr_weight:.2f},"
+            f" -> current_copr={self.current_copr_weight:.2f},"
             f" prior_copr={self.prior_copr_weight:.2f},"
             f" epa={self.epa_weight:.2f}\n"
         )
@@ -80,6 +87,8 @@ def __init__(
         self.auto_df = self._get_auto_summary()
         self.teleop_df = self._get_teleop_summary()
         self.endgame_df = self._get_tba_endgame_summary()
+        self.fouls_df = self._get_fouls_summary()
+        self.dpr_df = self._get_dpr_summary()
         self.rank_df = self._get_rank_summary()
         self.breakdown_df = self._get_breakdown_summary()
         self.comments_df = self._get_comments_summary()
@@ -99,13 +108,17 @@ def get_picklist_summary(self) -> pd.DataFrame:
         auto_pl = self.auto_df[["team_number", m, "n"]].rename(columns={m: f"auto_{m}"})
         teleop_pl = self.teleop_df[["team_number", m]].rename(columns={m: f"teleop_{m}"})
         endgame_pl = self.endgame_df[["team_number", m]].rename(columns={m: f"endgame_{m}"})
+        fouls_pl = self.fouls_df[["team_number", "foulPoints"]]
+        dpr_pl = self.dpr_df[["team_number", "dprs"]]
         rank_pl = self.rank_df[["team_number", "drive_rank", "defense_rank"]]
         breakdown_pl = self.breakdown_df[["team_number", "breakdown"]]
         comments_pl = self.comments_df[["team_number", "comments"]]
 
         df = (
             auto_pl.merge(teleop_pl, on="team_number")
             .merge(endgame_pl, on="team_number", how="left")
+            .merge(fouls_pl, on="team_number", how="left")
+            .merge(dpr_pl, on="team_number", how="left")
             .merge(rank_pl, on="team_number")
             .merge(breakdown_pl, on="team_number")
             .merge(comments_pl, on="team_number")
@@ -129,7 +142,20 @@ def get_picklist_summary(self) -> pd.DataFrame:
         )
 
         return df[
-            ["team", "score", "auto", "teleop", "endgame", "n", "drive_rank", "defense_rank", "breakdown", "comments"]
+            [
+                "team",
+                "score",
+                "auto",
+                "teleop",
+                "endgame",
+                "n",
+                "foulPoints",
+                "dprs",
+                "drive_rank",
+                "defense_rank",
+                "breakdown",
+                "comments",
+            ]
         ]
 
     # ------------------------------------------------------------------ #
@@ -172,11 +198,13 @@ def _get_auto_summary(self) -> pd.DataFrame:
 
         # Prior event COPR
         prior_copr_auto = pd.Series(0.0, index=df.index)
+        has_prior_copr = pd.Series(False, index=df.index)
         if not self.prior_coprs_df.empty and "Hub Auto Fuel Count" in self.prior_coprs_df.columns:
             prior = self.prior_coprs_df[["Hub Auto Fuel Count"]].copy()
             prior["team_number"] = prior.index.str.replace("frc", "").astype("Int64")
             prior.rename(columns={"Hub Auto Fuel Count": "prior_copr_auto_fuel"}, inplace=True)
             df = df.merge(prior[["team_number", "prior_copr_auto_fuel"]], on="team_number", how="left")
+            has_prior_copr = df["prior_copr_auto_fuel"].notna()
             prior_copr_auto = df["prior_copr_auto_fuel"].fillna(0).clip(lower=0)
 
         # EPA
@@ -187,10 +215,10 @@ def _get_auto_summary(self) -> pd.DataFrame:
             df = df.merge(epa, on="team_number", how="left")
             epa_auto = df["auto_epa"].fillna(0)
 
-        # Weighted blend
-        df["total_auto_points"] = (
-            self.current_copr_weight * copr_auto + self.prior_copr_weight * prior_copr_auto + self.epa_weight * epa_auto
-        )
+        # Weighted blend — teams missing prior COPR data get that weight redistributed to EPA
+        prior_w = self.prior_copr_weight * has_prior_copr.astype(float)
+        epa_w = self.epa_weight + self.prior_copr_weight * (~has_prior_copr).astype(float)
+        df["total_auto_points"] = self.current_copr_weight * copr_auto + prior_w * prior_copr_auto + epa_w * epa_auto
 
         # Debug
         print("\n--- Auto Summary Debug ---")
@@ -247,11 +275,13 @@ def _get_teleop_summary(self) -> pd.DataFrame:
 
         # Prior event COPR
         prior_copr_teleop = pd.Series(0.0, index=df.index)
+        has_prior_copr = pd.Series(False, index=df.index)
         if not self.prior_coprs_df.empty and "Hub Teleop Fuel Count" in self.prior_coprs_df.columns:
             prior = self.prior_coprs_df[["Hub Teleop Fuel Count"]].copy()
             prior["team_number"] = prior.index.str.replace("frc", "").astype("Int64")
             prior.rename(columns={"Hub Teleop Fuel Count": "prior_copr_teleop"}, inplace=True)
             df = df.merge(prior[["team_number", "prior_copr_teleop"]], on="team_number", how="left")
+            has_prior_copr = df["prior_copr_teleop"].notna()
             prior_copr_teleop = df["prior_copr_teleop"].fillna(0).clip(lower=0)
 
         # EPA
@@ -262,11 +292,11 @@ def _get_teleop_summary(self) -> pd.DataFrame:
             df = df.merge(epa, on="team_number", how="left")
             epa_teleop = df["teleop_epa"].fillna(0)
 
-        # Weighted blend
+        # Weighted blend — teams missing prior COPR data get that weight redistributed to EPA
+        prior_w = self.prior_copr_weight * has_prior_copr.astype(float)
+        epa_w = self.epa_weight + self.prior_copr_weight * (~has_prior_copr).astype(float)
         df["total_teleop_points"] = (
-            self.current_copr_weight * copr_teleop
-            + self.prior_copr_weight * prior_copr_teleop
-            + self.epa_weight * epa_teleop
+            self.current_copr_weight * copr_teleop + prior_w * prior_copr_teleop + epa_w * epa_teleop
         )
 
         # Debug
@@ -322,11 +352,13 @@ def _get_tba_endgame_summary(self) -> pd.DataFrame:
 
         # Prior event COPR Hub Endgame Fuel Count (no climb available for prior events)
         prior_endgame = pd.Series(0.0, index=df_merged.index)
+        has_prior_copr = pd.Series(False, index=df_merged.index)
         if not self.prior_coprs_df.empty and "Hub Endgame Fuel Count" in self.prior_coprs_df.columns:
             prior = self.prior_coprs_df[["Hub Endgame Fuel Count"]].copy()
             prior["team_number"] = prior.index.str.replace("frc", "").astype("Int64")
             prior.rename(columns={"Hub Endgame Fuel Count": "prior_endgame_fuel"}, inplace=True)
             df_merged = df_merged.merge(prior[["team_number", "prior_endgame_fuel"]], on="team_number", how="left")
+            has_prior_copr = df_merged["prior_endgame_fuel"].notna()
             prior_endgame = df_merged["prior_endgame_fuel"].fillna(0).clip(lower=0)
 
         # EPA
@@ -337,11 +369,11 @@ def _get_tba_endgame_summary(self) -> pd.DataFrame:
             df_merged = df_merged.merge(epa, on="team_number", how="left")
             epa_endgame = df_merged["endgame_epa"].fillna(0).clip(lower=0)
 
-        # Weighted blend
+        # Weighted blend — teams missing prior COPR data get that weight redistributed to EPA
+        prior_w = self.prior_copr_weight * has_prior_copr.astype(float)
+        epa_w = self.epa_weight + self.prior_copr_weight * (~has_prior_copr).astype(float)
         df_merged["total_endgame_points"] = (
-            self.current_copr_weight * current_tba_endgame
-            + self.prior_copr_weight * prior_endgame
-            + self.epa_weight * epa_endgame
+            self.current_copr_weight * current_tba_endgame + prior_w * prior_endgame + epa_w * epa_endgame
         )
 
         # Debug
@@ -365,6 +397,66 @@ def _get_tba_endgame_summary(self) -> pd.DataFrame:
 
         return self._get_stats_summary(df_merged, "total_endgame_points")
 
+    def _get_fouls_summary(self) -> pd.DataFrame:
+        """Compute per-team blended foul points from current and prior event COPR.
+
+        Blend: alpha * current_foulPoints + (1-alpha) * prior_foulPoints
+        No EPA equivalent; prior COPR fills the full weight when no matches played.
+
+        Returns:
+            pd.DataFrame: foulPoints keyed by team_number.
+        """
+        alpha = self.played_quals / self.total_quals if self.total_quals > 0 else 0.0
+        teams = self.scouting_df[["team_number"]].drop_duplicates().copy()
+        teams["team_number"] = teams["team_number"].astype("Int64")
+
+        current = pd.Series(0.0, index=teams.index)
+        if not self.coprs_df.empty and "foulPoints" in self.coprs_df.columns:
+            c = self.coprs_df[["foulPoints"]].copy()
+            c["team_number"] = c.index.str.replace("frc", "").astype("Int64")
+            teams = teams.merge(c.rename(columns={"foulPoints": "_cur_fouls"}), on="team_number", how="left")
+            current = teams["_cur_fouls"].fillna(0)
+
+        prior = pd.Series(0.0, index=teams.index)
+        if not self.prior_coprs_df.empty and "foulPoints" in self.prior_coprs_df.columns:
+            p = self.prior_coprs_df[["foulPoints"]].copy()
+            p["team_number"] = p.index.str.replace("frc", "").astype("Int64")
+            teams = teams.merge(p.rename(columns={"foulPoints": "_pri_fouls"}), on="team_number", how="left")
+            prior = teams["_pri_fouls"].fillna(0)
+
+        teams["foulPoints"] = alpha * current + (1 - alpha) * prior
+        return teams[["team_number", "foulPoints"]]
+
+    def _get_dpr_summary(self) -> pd.DataFrame:
+        """Compute per-team blended DPR from current and prior event OPRs.
+
+        Blend: alpha * current_dprs + (1-alpha) * prior_dprs
+        No EPA equivalent; prior OPR fills the full weight when no matches played.
+
+        Returns:
+            pd.DataFrame: dprs keyed by team_number.
+        """
+        alpha = self.played_quals / self.total_quals if self.total_quals > 0 else 0.0
+        teams = self.scouting_df[["team_number"]].drop_duplicates().copy()
+        teams["team_number"] = teams["team_number"].astype("Int64")
+
+        current = pd.Series(0.0, index=teams.index)
+        if not self.oprs_df.empty and "dprs" in self.oprs_df.columns:
+            c = self.oprs_df[["dprs"]].copy()
+            c["team_number"] = c.index.str.replace("frc", "").astype("Int64")
+            teams = teams.merge(c.rename(columns={"dprs": "_cur_dpr"}), on="team_number", how="left")
+            current = teams["_cur_dpr"].fillna(0)
+
+        prior = pd.Series(0.0, index=teams.index)
+        if not self.prior_oprs_df.empty and "dprs" in self.prior_oprs_df.columns:
+            p = self.prior_oprs_df[["dprs"]].copy()
+            p["team_number"] = p.index.str.replace("frc", "").astype("Int64")
+            teams = teams.merge(p.rename(columns={"dprs": "_pri_dpr"}), on="team_number", how="left")
+            prior = teams["_pri_dpr"].fillna(0)
+
+        teams["dprs"] = alpha * current + (1 - alpha) * prior
+        return teams[["team_number", "dprs"]]
+
     def _get_rank_summary(self) -> pd.DataFrame:
         """Compute mean drive and defense rankings per team.
 
diff --git a/src/scouting_analysis/frc2026_picklist_runner.py b/src/scouting_analysis/frc2026_picklist_runner.py
diff --git a/src/scouting_analysis/tba.py b/src/scouting_analysis/tba.py
