Skip to content

Commit dd40aae

Browse files
alexhyunminleealexhyunminlee
authored
Merge pull request #461 from switchbox-data/456-unify-md-utility-slugs-to-canonical-std_names-delmarva-potomac_edison
Unify MD utility slugs to canonical std_names (dpl, poted)
2 parents d4a8d8a + c1294f3 commit dd40aae

15 files changed

Lines changed: 132 additions & 83 deletions

File tree

‎context/code/data/pjm_hourly_loads.md‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -130,10 +130,56 @@ Output schema:
130130
Utility aggregation maps each utility to its zone(s) via
131131
`data/pjm/zone_mapping/csv/pjm_utility_zone_mapping.csv` and sums zone loads by
132132
timestamp. Each MD utility maps to exactly one zone (bge→BC, pepco→PEP, dpl→DPL,
133-
potomac-edison→AP), so a utility series equals its zone series. The
133+
poted→AP), so a utility series equals its zone series. The
134134
`capacity_weight` column is for capacity-cost allocation and is **not** applied
135135
to load.
136136

137+
## Zone vs utility load (shared zones, scale-invariance, multi-state zones)
138+
139+
PJM zones and retail utilities are not 1:1, and the aggregation deliberately
140+
treats them as distinct things. Three consequences are worth spelling out, since
141+
they look surprising at first glance.
142+
143+
**Utilities that share a zone get identical load profiles.** A co-op or
144+
municipal that sits inside a host IOU's transmission zone (e.g. `smeco` inside
145+
PEPCO; `choptank`, `easton_muni`, `berlin_muni` inside DPL; `poted`,
146+
`somerset_rec`, `hagerstown_muni` inside APS) is mapped to that zone and receives
147+
the **full zone load series** — `capacity_weight` is for capacity-cost
148+
allocation and is never applied to split load. So every utility in a zone gets
149+
the same hourly shape. Each utility is also computed independently, so
150+
adding or dropping a co-op/municipal in a zone does **not** change any other
151+
utility's load — dropping the phantom slugs that were never assigned in ResStock
152+
(`an-electric`, `thurmont`, `williamsport`) left `poted`/`dpl` etc. unchanged.
153+
154+
**Why identical (and full-zone) profiles are acceptable: the MC allocation is
155+
scale-invariant.** The sub-TX/DX marginal-cost methods that consume these loads
156+
(`generate_utility_tx_dx_mc.py`: peak-of-peak weighting, exceedance weighting)
157+
allocate a fixed `$/kW-yr` cost across hours using only the **shape** of the load
158+
profile — they normalize by the profile's own peak/exceedance, so the absolute
159+
magnitude divides out. Two utilities with the same shape but different absolute
160+
MW get the same hourly MC allocation; a utility assigned its whole zone's MW
161+
rather than just its retail slice gets the same allocation it would from a scaled
162+
copy of that shape. Magnitude is not over- or under-attributed because magnitude
163+
does not enter the allocation.
164+
165+
**Multi-state zones are a shape proxy, not a magnitude error.** Several PJM zones
166+
extend well beyond Maryland — APS (Allegheny Power Systems) covers parts of WV,
167+
PA, and VA; the PEPCO and DPL zones include DC and DE. So `poted`'s load shape is
168+
the whole APS zone's shape, which mixes non-MD load. This is an acknowledged
169+
approximation: we use the zone shape as a **proxy** for the MD retail territory's
170+
shape, accepting that the wider footprint can smear the timing of the local peak.
171+
Because the allocation is scale-invariant, the concern is purely "is the _shape_
172+
representative?", not "are we attributing too many MW?". Customer filtering to the
173+
MD retail territory happens upstream in ResStock utility assignment, not here.
174+
175+
**Consistent with the NY (NYISO) treatment.** NYISO load aggregation
176+
(`data/nyiso/hourly_demand/aggregate_nyiso_utility_loads.py`) follows the same
177+
rule: it sums full zone loads for each utility and uses `capacity_weight` only
178+
for capacity allocation, not to split load. NY utilities that cover the same
179+
zone(s) likewise end up with identical load profiles. The MD approach (shared
180+
APS/PEPCO/DPL profiles, full-zone load) is the PJM analogue of that established
181+
NY precedent.
182+
137183
## Running it
138184

139185
From the repo root (scripts run as modules because they import `data.pjm`):

‎context/code/data/utility_assignment_resstock.md‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -141,16 +141,16 @@ PUMAs that straddle a county line get geographic signal from both sides. A PUMA
141141

142142
Major utilities covered (2023 EIA-861, MD distribution utilities):
143143

144-
| Utility | EIA ID | Std name | Residential customers |
145-
| --------------------------------- | ------ | ---------------- | --------------------- |
146-
| Baltimore Gas & Electric Co | 1167 | `bge` | 1,208k |
147-
| Potomac Electric Power Co (Pepco) | 15270 | `pepco` | 548k |
148-
| The Potomac Edison Company | 15263 | `potomac_edison` | 253k |
149-
| Delmarva Power | 5027 | `delmarva` | 185k |
150-
| Southern Maryland Elec Coop | 17637 | `smeco` | 159k |
151-
| Choptank Electric Cooperative | 3503 | `choptank` | ~30k |
152-
| Somerset Rural Electric Coop | 84 | `somerset_rec` | small |
153-
| Town of Berlin (MD) | 1615 | `berlin_muni` | small |
144+
| Utility | EIA ID | Std name | Residential customers |
145+
| --------------------------------- | ------ | -------------- | --------------------- |
146+
| Baltimore Gas & Electric Co | 1167 | `bge` | 1,208k |
147+
| Potomac Electric Power Co (Pepco) | 15270 | `pepco` | 548k |
148+
| The Potomac Edison Company | 15263 | `poted` | 253k |
149+
| Delmarva Power | 5027 | `dpl` | 185k |
150+
| Southern Maryland Elec Coop | 17637 | `smeco` | 159k |
151+
| Choptank Electric Cooperative | 3503 | `choptank` | ~30k |
152+
| Somerset Rural Electric Coop | 84 | `somerset_rec` | small |
153+
| Town of Berlin (MD) | 1615 | `berlin_muni` | small |
154154

155155
**EIA utility ID → std name mapping** is defined in `_EIA_ID_TO_STD_NAME` in `assign_utility_md.py`. Utilities not in the map fall back to the EIA name string.
156156

‎context/code/marginal_costs/md_supply_energy_marginal_costs.md‎

Lines changed: 12 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -44,21 +44,18 @@ host IOU, map to one of four PJM zone aggregate pricing nodes. Defined in
4444
`utils/data_prep/marginal_costs/supply_utils.py` (`PJM_UTILITY_ZONES`) and
4545
`data/pjm/zone_mapping/generate_zone_mapping_csv.py` (`_MAPPING_ROWS`).
4646

47-
| Utility slug | Type | PJM zone | pnode_id |
48-
| ---------------- | ---------------------------- | -------- | -------- |
49-
| `bge` | IOU | BGE | 51292 |
50-
| `pepco` | IOU | PEPCO | 51298 |
51-
| `dpl` | IOU | DPL | 51293 |
52-
| `potomac-edison` | IOU | APS | 8394954 |
53-
| `smeco` | Co-op (in PEPCO territory) | PEPCO | 51298 |
54-
| `choptank` | Co-op (in DPL territory) | DPL | 51293 |
55-
| `an-electric` | Co-op (in DPL territory) | DPL | 51293 |
56-
| `somerset-rec` | Co-op (in APS territory) | APS | 8394954 |
57-
| `hagerstown` | Municipal (in APS territory) | APS | 8394954 |
58-
| `thurmont` | Municipal (in APS territory) | APS | 8394954 |
59-
| `williamsport` | Municipal (in APS territory) | APS | 8394954 |
60-
| `easton` | Municipal (in DPL territory) | DPL | 51293 |
61-
| `berlin` | Municipal (in DPL territory) | DPL | 51293 |
47+
| Utility slug | Type | PJM zone | pnode_id |
48+
| ----------------- | ---------------------------- | -------- | -------- |
49+
| `bge` | IOU | BGE | 51292 |
50+
| `pepco` | IOU | PEPCO | 51298 |
51+
| `dpl` | IOU | DPL | 51293 |
52+
| `poted` | IOU | APS | 8394954 |
53+
| `smeco` | Co-op (in PEPCO territory) | PEPCO | 51298 |
54+
| `choptank` | Co-op (in DPL territory) | DPL | 51293 |
55+
| `somerset_rec` | Co-op (in APS territory) | APS | 8394954 |
56+
| `hagerstown_muni` | Municipal (in APS territory) | APS | 8394954 |
57+
| `easton_muni` | Municipal (in DPL territory) | DPL | 51293 |
58+
| `berlin_muni` | Municipal (in DPL territory) | DPL | 51293 |
6259

6360
Utilities that share a zone get identical supply energy marginal cost profiles
6461
(the zone LMP is the same for all of them).

‎data/pjm/README.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -112,13 +112,13 @@ The 5 highest non-holiday-weekday RTO unrestricted daily peaks per summer (PJM's
112112

113113
One row per (utility, zone, weight); the single place the per-source zone aliases (`dataminer_zone`, `fivecp_zone_label`, `price_zone`) live. MC scripts use it to filter the region-wide datasets down to one utility.
114114

115-
| | |
116-
| ------------------ | ---------------------------------------------------------------- |
117-
| **Source** | Hardcoded rows in `generate_zone_mapping_csv.py` (nyiso pattern) |
118-
| **Format** | Generated CSV (uploaded as CSV, not parquet) |
119-
| **Coverage** | MD utilities: bge, pepco, dpl, potomac-edison |
120-
| **S3 path** | `s3://data.sb/pjm/zone_mapping/pjm_utility_zone_mapping.csv` |
121-
| **Update cadence** | When onboarding a utility |
115+
| | |
116+
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------ |
117+
| **Source** | Hardcoded rows in `generate_zone_mapping_csv.py` (nyiso pattern) |
118+
| **Format** | Generated CSV (uploaded as CSV, not parquet) |
119+
| **Coverage** | MD utilities (canonical std_names): bge, pepco, dpl, poted, smeco, choptank, somerset_rec, hagerstown_muni, easton_muni, berlin_muni |
120+
| **S3 path** | `s3://data.sb/pjm/zone_mapping/pjm_utility_zone_mapping.csv` |
121+
| **Update cadence** | When onboarding a utility |
122122

123123
**How to update**: Edit `_MAPPING_ROWS` in `generate_zone_mapping_csv.py`, then `just prepare && just upload`. Validation cross-checks every mapped zone against both capacity CSVs (the single enforcement point of zone-code integrity between the three datasets).
124124

@@ -128,7 +128,7 @@ One row per (utility, zone, weight); the single place the per-source zone aliase
128128

129129
- **Three naming systems, one crosswalk**: Data Miner legacy codes (`BC`, `PEP`, `AP`) vs PDF/XLS labels (`BGE`, `PEPCO`, `APS`) — this CSV is the only place the mapping lives.
130130
- **Zone ≠ retail territory**: the PEPCO and DPL zones span MD + DC/DE; `state` here is the _analysis_ state, customer filtering happens upstream in ResStock utility assignment.
131-
- **`potomac-edison`** is a retail brand inside the APS zone — zone-level data includes WV/PA load.
131+
- **`poted`** (Potomac Edison) is a retail brand inside the APS zone — zone-level data includes WV/PA load.
132132
- **UGI / PPL-folded zones**: UGI has no separate row in the RPM price files (it sits inside the PPL LDA), so a mapping row with `price_zone=UGI` will fail the RPM cross-check. Set `price_zone=PPL` for such utilities (UGI is still valid for `fivecp_zone_label` and `dataminer_zone`, which both carry a UGI row).
133133

134134
## Historical source URLs

‎data/pjm/fetch_lmp.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,10 +32,10 @@
3232
Mapping from Switchbox utility std_name to PJM zone codes (pnode_name for
3333
ZONE-type rows):
3434
35-
bge → BGE (pnode_id 51292)
36-
pepco → PEPCO (pnode_id 51298)
37-
delmarva → DPL (pnode_id 51293)
38-
potomac_edison → APS (pnode_id 8394954)
35+
bge → BGE (pnode_id 51292)
36+
pepco → PEPCO (pnode_id 51298)
37+
dpl → DPL (pnode_id 51293)
38+
poted → APS (pnode_id 8394954)
3939
4040
Usage::
4141

‎data/pjm/hourly_demand/aggregate_pjm_utility_loads.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
timestamp, and writes utility-level parquet locally.
88
99
Each Maryland utility maps to exactly one Data Miner zone (bge->BC, pepco->PEP,
10-
dpl->DPL, potomac-edison->AP), so a utility series equals its zone series. The
10+
dpl->DPL, poted->AP), so a utility series equals its zone series. The
1111
``capacity_weight`` column is for capacity-cost allocation and is intentionally
1212
NOT applied here — load is assigned in full to the mapped utility.
1313

‎data/pjm/zone_mapping/Justfile‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,10 @@ validate:
2727
# Prepare: generate then validate
2828
prepare: generate validate
2929

30-
# Upload: sync local CSV to S3
31-
upload: validate
30+
# Upload: regenerate + validate, then sync local CSV to S3.
31+
# Depends on `prepare` (generate → validate) so S3 can never receive a stale or
32+
# hand-edited CSV — the uploaded file is always a fresh render of the generator.
33+
upload: prepare
3234
aws s3 sync "{{ path_local_csv }}/" "{{ path_s3_zone_mapping }}" --exclude "*" --include "*.csv"
3335
@echo "Uploaded zone mapping to {{ path_s3_zone_mapping }}"
3436

‎data/pjm/zone_mapping/csv/pjm_utility_zone_mapping.csv‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
1-
utility,state,dataminer_zone,fivecp_zone_label,price_zone,capacity_weight
2-
bge,md,BC,BGE,BGE,1.0
3-
pepco,md,PEP,PEPCO,PEPCO,1.0
4-
dpl,md,DPL,DPL,DPL,1.0
5-
potomac-edison,md,AP,APS,APS,1.0
1+
utility,state,dataminer_zone,fivecp_zone_label,price_zone,capacity_weight,pnode_id
2+
bge,md,BC,BGE,BGE,1.0,51292
3+
pepco,md,PEP,PEPCO,PEPCO,1.0,51298
4+
dpl,md,DPL,DPL,DPL,1.0,51293
5+
poted,md,AP,APS,APS,1.0,8394954
6+
smeco,md,PEP,PEPCO,PEPCO,1.0,51298
7+
choptank,md,DPL,DPL,DPL,1.0,51293
8+
somerset_rec,md,AP,APS,APS,1.0,8394954
9+
hagerstown_muni,md,AP,APS,APS,1.0,8394954
10+
easton_muni,md,DPL,DPL,DPL,1.0,51293
11+
berlin_muni,md,DPL,DPL,DPL,1.0,51293

‎data/pjm/zone_mapping/generate_zone_mapping_csv.py‎

Lines changed: 11 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -59,17 +59,20 @@
5959
#
6060
# All MD utilities map entirely (1.0) to a single PJM zone — no split-zone
6161
# cases exist among MD electric utilities.
62+
# Slugs are the canonical std_names from utils/utility_codes.py (lowercase,
63+
# underscore-separated — never hyphens). Co-ops and municipals that sit inside
64+
# an IOU transmission zone share that IOU's zone codes and pnode_id.
6265
_MAPPING_ROWS: list[tuple[str, str, str, str, str, float, int]] = [
6366
# ── Investor-Owned Utilities ──────────────────────────────────────────────
6467
# BGE — Baltimore Gas & Electric; only PJM zone entirely within Maryland.
6568
("bge", "md", "BC", "BGE", "BGE", 1.0, 51292),
6669
# Pepco — Potomac Electric Power Co.; zone also covers DC.
6770
("pepco", "md", "PEP", "PEPCO", "PEPCO", 1.0, 51298),
68-
# Delmarva Power & Light; zone also covers DE and parts of VA.
71+
# Delmarva Power & Light (std_name dpl); zone also covers DE and parts of VA.
6972
("dpl", "md", "DPL", "DPL", "DPL", 1.0, 51293),
70-
# Potomac Edison — FirstEnergy subsidiary; operates within the APS zone
71-
# (Allegheny Power Systems), which also covers parts of WV, PA, VA.
72-
("potomac-edison", "md", "AP", "APS", "APS", 1.0, 8394954),
73+
# Potomac Edison (std_name poted) — FirstEnergy subsidiary; operates within
74+
# the APS zone (Allegheny Power Systems), which also covers parts of WV, PA, VA.
75+
("poted", "md", "AP", "APS", "APS", 1.0, 8394954),
7376
# ── Cooperatives ──────────────────────────────────────────────────────────
7477
# SMECO — Southern Maryland Electric Cooperative; distribution co-op whose
7578
# 6 interties connect to the PEPCO transmission zone (per SMECO website
@@ -78,24 +81,17 @@
7881
# Choptank Electric Cooperative — ODEC member serving MD Eastern Shore;
7982
# interconnected to DPL transmission system (ODEC Wholesale Contract, SEC).
8083
("choptank", "md", "DPL", "DPL", "DPL", 1.0, 51293),
81-
# A&N Electric Cooperative — ODEC member; serves Smith Island (MD) and VA
82-
# Eastern Shore; interconnected to DPL transmission (ODEC Wholesale Contract).
83-
("an-electric", "md", "DPL", "DPL", "DPL", 1.0, 51293),
8484
# Somerset Rural Electric Cooperative — Allegheny Electric Cooperative
8585
# member; serves Garrett County MD within the APS zone. (In PA it is in
8686
# the PENELEC zone, but its MD territory is APS.)
87-
("somerset-rec", "md", "AP", "APS", "APS", 1.0, 8394954),
87+
("somerset_rec", "md", "AP", "APS", "APS", 1.0, 8394954),
8888
# ── Municipal Utilities ───────────────────────────────────────────────────
8989
# Hagerstown Light Department — municipal utility within the APS zone.
90-
("hagerstown", "md", "AP", "APS", "APS", 1.0, 8394954),
91-
# Thurmont Municipal Light Company — municipal utility within the APS zone.
92-
("thurmont", "md", "AP", "APS", "APS", 1.0, 8394954),
93-
# Town of Williamsport — municipal utility within the APS zone.
94-
("williamsport", "md", "AP", "APS", "APS", 1.0, 8394954),
90+
("hagerstown_muni", "md", "AP", "APS", "APS", 1.0, 8394954),
9591
# Easton Utilities Commission — municipal utility within the DPL zone.
96-
("easton", "md", "DPL", "DPL", "DPL", 1.0, 51293),
92+
("easton_muni", "md", "DPL", "DPL", "DPL", 1.0, 51293),
9793
# Town of Berlin Municipal Electric Plant — municipal within the DPL zone.
98-
("berlin", "md", "DPL", "DPL", "DPL", 1.0, 51293),
94+
("berlin_muni", "md", "DPL", "DPL", "DPL", 1.0, 51293),
9995
]
10096

10197
# Valid Data Miner legacy transmission-zone codes (hrl_load_metered vocabulary).

‎data/pjm/zone_mapping/validate_pjm_zone_mapping.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,10 @@
7474

7575
VALID_DATAMINER_ZONES = frozenset(CANONICAL_CROSSWALK.values())
7676

77-
UTILITY_SLUG_RE = re.compile(r"^[a-z0-9-]+$")
77+
# Canonical utility std_names are lowercase, underscore-separated — never
78+
# hyphens (see utils/utility_codes.py). Disallowing hyphens here actively
79+
# prevents a hyphenated slug from drifting back into the crosswalk.
80+
UTILITY_SLUG_RE = re.compile(r"^[a-z0-9_]+$")
7881

7982

8083
class ValidationResult:
@@ -129,7 +132,7 @@ def check_utility_slugs(df: pl.DataFrame, result: ValidationResult) -> None:
129132
errors: list[str] = []
130133
for utility in df["utility"].unique().to_list():
131134
if not UTILITY_SLUG_RE.match(utility):
132-
errors.append(f"utility '{utility}' is not a lowercase [a-z0-9-]+ slug")
135+
errors.append(f"utility '{utility}' is not a lowercase [a-z0-9_]+ slug")
133136
bad_states = set(df["state"].unique().to_list()) - KNOWN_STATES
134137
if bad_states:
135138
errors.append(f"unknown states: {sorted(bad_states)} (known: {KNOWN_STATES})")

0 commit comments

Comments
 (0)