Skip to content

Commit ee5a91d

Browse files
jpvelezcursoragent
andauthored
Central utility crosswalk, tariff mapper fixes, and NY customer counts (#155)
* Add fetch-ny-utility-stats Justfile task, EIA-861 utils script, and ny_electric_utility_stats.csv - Move NY data Justfile up to rate_design/ny/hp_rates/data/ - Add fetch-ny-utility-stats recipe (uses utils/get_utility_stats_from_eia861.py) - Add utils/get_utility_stats_from_eia861.py from main - Force-add ny_electric_utility_stats.csv (EIA-861 NY investor-owned utility stats) Co-authored-by: Cursor <cursoragent@cursor.com> * feat(utils): add central utility crosswalk and std_name types - Add utils/utility_codes.py with UTILITIES list (one record per utility) - Derive lookup functions: get_ny_open_data_to_std_name, get_eia_utility_id_to_std_name, get_std_name_to_gas_tariff_key, etc. - Update utils/types.py: electric_utility and gas_utility Literals use std_name (coned, nyseg, nimo, keddy, kedli, etc.) Co-authored-by: Cursor <cursoragent@cursor.com> * test(utils): add tests for utility_codes crosswalk Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(assign_utility_ny): use central crosswalk for NY Open Data names - Replace inline utility_name_map with get_ny_open_data_to_std_name() - Update tests to use _utility_name_map_lazy from utility_codes Co-authored-by: Cursor <cursoragent@cursor.com> * feat(get_utility_stats): add utility_code column (EIA ID → std_name) Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tariff_mappers): use real assign_utility data and central crosswalk - electric_tariff_mapper: use sb.electric_utility when present, support metadata_utility path, accept std_name - gas_tariff_mapper: use get_std_name_to_gas_tariff_key for National Grid split (nimo/keddy/kedli), same metadata handling - Add tests for both mappers Co-authored-by: Cursor <cursoragent@cursor.com> * chore(ny,ri): update Justfiles to std_name, regenerate NY utility stats - Use coned, nimo, nyseg in map-electric-tariff and map-gas-tariff - Add map-electric-rie-default / map-gas-rie-default for RI - Regenerate ny_electric_utility_stats.csv with utility_code column Co-authored-by: Cursor <cursoragent@cursor.com> * docs(utility_codes): add guide for adding a new state Co-authored-by: Cursor <cursoragent@cursor.com> * Make ruff and ty happy --------- Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 3f2d1d6 commit ee5a91d

14 files changed

Lines changed: 788 additions & 173 deletions

rate_design/ny/hp_rates/data/tariff_map/Justfile renamed to rate_design/ny/hp_rates/data/Justfile

Lines changed: 28 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,16 @@
22
project_root := `git rev-parse --show-toplevel`
33
path_s3_resstock_metadata := "s3://data.sb/nrel/resstock/res_2024_amy2018_2/metadata"
44
tariff_map_dir_ny := "{{project_root}}/rate_design/ny/hp_rates/data/tariff_map"
5+
data_dir_ny := project_root + "/rate_design/ny/hp_rates/data"
6+
7+
# Fetch NY investor-owned utility customer counts from EIA-861, write to ny_electric_utility_stats.csv
8+
fetch-ny-utility-stats:
9+
uv run python {{project_root}}/utils/get_utility_stats_from_eia861.py NY > {{data_dir_ny}}/ny_electric_utility_stats.csv
10+
@echo "Wrote {{data_dir_ny}}/ny_electric_utility_stats.csv"
511

612
# Usage: just map-electric-tariff [electric_utility] [SB_scenario_type] [SB_scenario_year] [upgrade_id] [output_dir]
7-
# Example: just map-electric-tariff Coned default 1 00 ./rate_design/ny/hp_rates/data/tariff_map
13+
# Example: just map-electric-tariff coned default 1 00 ./rate_design/ny/hp_rates/data/tariff_map
14+
# electric_utility: std_name (e.g. coned, nyseg, nimo)
815
map-electric-tariff electric_utility SB_scenario_type SB_scenario_year upgrade_id output_dir:
916
uv run python {{project_root}}/utils/electric_tariff_mapper.py \
1017
--metadata_path "{{path_s3_resstock_metadata}}" \
@@ -18,34 +25,34 @@ map-electric-tariff electric_utility SB_scenario_type SB_scenario_year upgrade_i
1825
# Some useful enumerations:
1926

2027
map-electric-coned-default:
21-
just map-electric-tariff Coned default 1 00 {{tariff_map_dir_ny}}
28+
just map-electric-tariff coned default 1 00 {{tariff_map_dir_ny}}
2229

2330
map-electric-coned-seasonal:
24-
just map-electric-tariff Coned seasonal 1 00 {{tariff_map_dir_ny}}
31+
just map-electric-tariff coned seasonal 1 00 {{tariff_map_dir_ny}}
2532

26-
map-celectric-oned-class-specific-seasonal:
27-
just map-electric-tariff Coned class_specific_seasonal 1 00 {{tariff_map_dir_ny}}
33+
map-electric-coned-class-specific-seasonal:
34+
just map-electric-tariff coned class_specific_seasonal 1 00 {{tariff_map_dir_ny}}
2835

2936
map-electric-national-grid-default:
30-
just map-electric-tariff "National Grid" default 1 00 {{tariff_map_dir_ny}}
37+
just map-electric-tariff nimo default 1 00 {{tariff_map_dir_ny}}
3138

3239
map-electric-national-grid-seasonal:
33-
just map-electric-tariff "National Grid" seasonal 1 00 {{tariff_map_dir_ny}}
40+
just map-electric-tariff nimo seasonal 1 00 {{tariff_map_dir_ny}}
3441

3542
map-electric-national-grid-class-specific-seasonal:
36-
just map-electric-tariff "National Grid" class_specific_seasonal 1 00 {{tariff_map_dir_ny}}
43+
just map-electric-tariff nimo class_specific_seasonal 1 00 {{tariff_map_dir_ny}}
3744

3845
map-electric-nyseg-default:
39-
just map-electric-tariff NYSEG default 1 00 {{tariff_map_dir_ny}}
46+
just map-electric-tariff nyseg default 1 00 {{tariff_map_dir_ny}}
4047

4148
map-electric-nyseg-seasonal:
42-
just map-electric-tariff NYSEG seasonal 1 00 {{tariff_map_dir_ny}}
49+
just map-electric-tariff nyseg seasonal 1 00 {{tariff_map_dir_ny}}
4350

4451
map-electric-nyseg-class-specific-seasonal:
45-
just map-electric-tariff NYSEG class_specific_seasonal 1 00 {{tariff_map_dir_ny}}
52+
just map-electric-tariff nyseg class_specific_seasonal 1 00 {{tariff_map_dir_ny}}
4653

4754
# Usage: just map-gas-tariff [electric_utility] [upgrade_id] [output_dir]
48-
# Example: just map-gas-tariff NYSEG 00 ./rate_design/ny/hp_rates/data/tariff_map
55+
# Example: just map-gas-tariff nyseg 00 ./rate_design/ny/hp_rates/data/tariff_map
4956
map-gas-tariff electric_utility upgrade_id output_dir:
5057
uv run python {{project_root}}/utils/gas_tariff_mapper.py \
5158
--metadata_path "{{path_s3_resstock_metadata}}" \
@@ -57,28 +64,28 @@ map-gas-tariff electric_utility upgrade_id output_dir:
5764
# Some useful enumerations:
5865

5966
map-gas-coned-default:
60-
just map-gas-tariff Coned 00 {{tariff_map_dir_ny}}
67+
just map-gas-tariff coned 00 {{tariff_map_dir_ny}}
6168

6269
map-gas-coned-seasonal:
63-
just map-gas-tariff Coned 00 {{tariff_map_dir_ny}}
70+
just map-gas-tariff coned 00 {{tariff_map_dir_ny}}
6471

6572
map-gas-coned-class-specific-seasonal:
66-
just map-gas-tariff Coned 00 {{tariff_map_dir_ny}}
73+
just map-gas-tariff coned 00 {{tariff_map_dir_ny}}
6774

6875
map-gas-national-grid-default:
69-
just map-gas-tariff "National Grid" 00 {{tariff_map_dir_ny}}
76+
just map-gas-tariff nimo 00 {{tariff_map_dir_ny}}
7077

7178
map-gas-national-grid-seasonal:
72-
just map-gas-tariff "National Grid" 00 {{tariff_map_dir_ny}}
79+
just map-gas-tariff nimo 00 {{tariff_map_dir_ny}}
7380

7481
map-gas-national-grid-class-specific-seasonal:
75-
just map-gas-tariff "National Grid" 00 {{tariff_map_dir_ny}}
82+
just map-gas-tariff nimo 00 {{tariff_map_dir_ny}}
7683

7784
map-gas-nyseg-default:
78-
just map-gas-tariff NYSEG 00 {{tariff_map_dir_ny}}
85+
just map-gas-tariff nyseg 00 {{tariff_map_dir_ny}}
7986

8087
map-gas-nyseg-seasonal:
81-
just map-gas-tariff NYSEG 00 {{tariff_map_dir_ny}}
88+
just map-gas-tariff nyseg 00 {{tariff_map_dir_ny}}
8289

8390
map-gas-nyseg-class-specific-seasonal:
84-
just map-gas-tariff NYSEG 00 {{tariff_map_dir_ny}}
91+
just map-gas-tariff nyseg 00 {{tariff_map_dir_ny}}
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
utility_id_eia,utility_code,utility_name,business_model,entity_type,report_date,total_sales_mwh,total_sales_revenue,commercial_sales_mwh,commercial_sales_revenue,commercial_customers,industrial_sales_mwh,industrial_sales_revenue,industrial_customers,other_sales_mwh,other_sales_revenue,other_customers,residential_sales_mwh,residential_sales_revenue,residential_customers,transportation_sales_mwh,transportation_sales_revenue,transportation_customers
2+
4226,coned,Consolidated Edison Co-NY Inc,retail,Investor Owned,2024-01-01,52425972.0,10723693000.0,35041850.0,5565803000.0,572249.0,1303892.0,338525000.0,10498.0,0.0,0.0,0.0,13643968.0,4671811000.0,3064038.0,2436261.0,147553000.0,2.0
3+
13573,nimo,Niagara Mohawk Power Corp.,retail,Investor Owned,2024-01-01,32718092.0,2986044400.0,12576316.0,792737300.0,185072.0,8302774.0,220623000.0,1554.0,0.0,0.0,0.0,11833895.0,1972635400.0,1532294.0,5108.0,48600.0,1.0
4+
13511,nyseg,New York State Elec & Gas Corp,retail,Investor Owned,2024-01-01,15231769.0,1764984600.0,5806928.0,472634400.0,127567.0,2472844.0,62332000.0,1500.0,0.0,0.0,0.0,6919839.0,1228409300.0,792300.0,32158.0,1609000.0,2.0
5+
16183,rge,Rochester Gas & Electric Corp,retail,Investor Owned,2024-01-01,6871716.0,812878850.0,3044406.0,259093600.0,40227.0,984961.0,49085700.0,634.0,0.0,0.0,0.0,2842349.0,504699520.0,351481.0,0.0,0.0,0.0
6+
3249,cenhud,Central Hudson Gas & Elec Corp,retail,Investor Owned,2024-01-01,5018039.0,787803970.0,2009494.0,239328000.0,49472.0,840758.0,31443000.0,861.0,0.0,0.0,0.0,2167787.0,517033020.0,270859.0,0.0,0.0,0.0
7+
14154,or,Orange & Rockland Utils Inc,retail,Investor Owned,2024-01-01,4087784.0,594447900.0,2023377.0,203018600.0,33695.0,313637.0,16500600.0,85.0,0.0,0.0,0.0,1750770.0,374928700.0,211011.0,0.0,0.0,0.0
8+
66101,,FirstEnergy Pennsylvania Electric Co,retail,Investor Owned,2024-01-01,64611.0,8235400.0,30033.0,3107500.0,479.0,4022.0,530100.0,1.0,0.0,0.0,0.0,30556.0,4597800.0,3400.0,0.0,0.0,0.0
9+
14711,,Pennsylvania Electric Co,retail,Investor Owned,2023-01-01,60641.0,7912700.0,28931.0,3099000.0,486.0,1214.0,177500.0,1.0,0.0,0.0,0.0,30496.0,4636200.0,3400.0,0.0,0.0,0.0
10+
6369,,Fishers Island Utility Co Inc,retail,Investor Owned,2024-01-01,7079.0,2372900.0,2061.0,457600.0,113.0,0.0,0.0,0.0,0.0,0.0,0.0,5018.0,1915300.0,657.0,0.0,0.0,0.0

rate_design/ri/hp_rates/data/tariff_map/Justfile

Lines changed: 27 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ path_s3_resstock_metadata := "s3://data.sb/nrel/resstock/res_2024_amy2018_2/meta
44
tariff_map_dir_ri := "{{project_root}}/rate_design/ri/hp_rates/data/tariff_map"
55

66
# Usage: just map-electric-tariff [electric_utility] [SB_scenario_type] [SB_scenario_year] [upgrade_id] [output_dir]
7-
# Example: just map-electric-tariff Coned default 1 00 ./rate_design/ri/hp_rates/data/tariff_map
7+
# Example: just map-electric-tariff rie default 1 00 ./rate_design/ri/hp_rates/data/tariff_map
88
map-electric-tariff electric_utility SB_scenario_type SB_scenario_year upgrade_id output_dir:
99
uv run python {{project_root}}/utils/electric_tariff_mapper.py \
1010
--metadata_path "{{path_s3_resstock_metadata}}" \
@@ -18,35 +18,38 @@ map-electric-tariff electric_utility SB_scenario_type SB_scenario_year upgrade_i
1818
# Some useful enumerations:
1919

2020
map-electric-coned-default:
21-
just map-electric-tariff Coned default 1 00 {{tariff_map_dir_ri}}
21+
just map-electric-tariff coned default 1 00 {{tariff_map_dir_ri}}
2222

2323
map-electric-coned-seasonal:
24-
just map-electric-tariff Coned seasonal 1 00 {{tariff_map_dir_ri}}
24+
just map-electric-tariff coned seasonal 1 00 {{tariff_map_dir_ri}}
2525

26-
map-celectric-oned-class-specific-seasonal:
27-
just map-electric-tariff Coned class_specific_seasonal 1 00 {{tariff_map_dir_ri}}
26+
map-electric-coned-class-specific-seasonal:
27+
just map-electric-tariff coned class_specific_seasonal 1 00 {{tariff_map_dir_ri}}
2828

2929
map-electric-national-grid-default:
30-
just map-electric-tariff "National Grid" default 1 00 {{tariff_map_dir_ri}}
30+
just map-electric-tariff nimo default 1 00 {{tariff_map_dir_ri}}
3131

3232
map-electric-national-grid-seasonal:
33-
just map-electric-tariff "National Grid" seasonal 1 00 {{tariff_map_dir_ri}}
33+
just map-electric-tariff nimo seasonal 1 00 {{tariff_map_dir_ri}}
3434

3535
map-electric-national-grid-class-specific-seasonal:
36-
just map-electric-tariff "National Grid" class_specific_seasonal 1 00 {{tariff_map_dir_ri}}
36+
just map-electric-tariff nimo class_specific_seasonal 1 00 {{tariff_map_dir_ri}}
3737

3838
map-electric-nyseg-default:
39-
just map-electric-tariff NYSEG default 1 00 {{tariff_map_dir_ri}}
39+
just map-electric-tariff nyseg default 1 00 {{tariff_map_dir_ri}}
4040

4141
map-electric-nyseg-seasonal:
42-
just map-electric-tariff NYSEG seasonal 1 00 {{tariff_map_dir_ri}}
42+
just map-electric-tariff nyseg seasonal 1 00 {{tariff_map_dir_ri}}
4343

4444
map-electric-nyseg-class-specific-seasonal:
45-
just map-electric-tariff NYSEG class_specific_seasonal 1 00 {{tariff_map_dir_ri}}
45+
just map-electric-tariff nyseg class_specific_seasonal 1 00 {{tariff_map_dir_ri}}
46+
47+
map-electric-rie-default:
48+
just map-electric-tariff rie default 1 00 {{tariff_map_dir_ri}}
4649

4750

4851
# Usage: just map-gas-tariff [electric_utility] [upgrade_id] [output_dir]
49-
# Example: just map-gas-tariff "National Grid" 00 ./rate_design/ri/hp_rates/data/tariff_map
52+
# Example: just map-gas-tariff rie 00 ./rate_design/ri/hp_rates/data/tariff_map
5053
map-gas-tariff electric_utility upgrade_id output_dir:
5154
uv run python {{project_root}}/utils/gas_tariff_mapper.py \
5255
--metadata_path "{{path_s3_resstock_metadata}}" \
@@ -58,28 +61,31 @@ map-gas-tariff electric_utility upgrade_id output_dir:
5861
# Some useful enumerations:
5962

6063
map-gas-coned-default:
61-
just map-gas-tariff Coned 00 {{tariff_map_dir_ri}}
64+
just map-gas-tariff coned 00 {{tariff_map_dir_ri}}
6265

6366
map-gas-coned-seasonal:
64-
just map-gas-tariff Coned 00 {{tariff_map_dir_ri}}
67+
just map-gas-tariff coned 00 {{tariff_map_dir_ri}}
6568

6669
map-gas-coned-class-specific-seasonal:
67-
just map-gas-tariff Coned 00 {{tariff_map_dir_ri}}
70+
just map-gas-tariff coned 00 {{tariff_map_dir_ri}}
6871

6972
map-gas-national-grid-default:
70-
just map-gas-tariff "National Grid" 00 {{tariff_map_dir_ri}}
73+
just map-gas-tariff nimo 00 {{tariff_map_dir_ri}}
7174

7275
map-gas-national-grid-seasonal:
73-
just map-gas-tariff "National Grid" 00 {{tariff_map_dir_ri}}
76+
just map-gas-tariff nimo 00 {{tariff_map_dir_ri}}
7477

7578
map-gas-national-grid-class-specific-seasonal:
76-
just map-gas-tariff "National Grid" 00 {{tariff_map_dir_ri}}
79+
just map-gas-tariff nimo 00 {{tariff_map_dir_ri}}
7780

7881
map-gas-nyseg-default:
79-
just map-gas-tariff NYSEG 00 {{tariff_map_dir_ri}}
82+
just map-gas-tariff nyseg 00 {{tariff_map_dir_ri}}
8083

8184
map-gas-nyseg-seasonal:
82-
just map-gas-tariff NYSEG 00 {{tariff_map_dir_ri}}
85+
just map-gas-tariff nyseg 00 {{tariff_map_dir_ri}}
8386

8487
map-gas-nyseg-class-specific-seasonal:
85-
just map-gas-tariff NYSEG 00 {{tariff_map_dir_ri}}
88+
just map-gas-tariff nyseg 00 {{tariff_map_dir_ri}}
89+
90+
map-gas-rie-default:
91+
just map-gas-tariff rie 00 {{tariff_map_dir_ri}}

tests/test_assign_utility_ny.py

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,21 @@
77
import polars as pl
88

99
from utils.assign_utility_ny import (
10-
CONFIGS,
1110
_calculate_prior_distributions,
1211
_calculate_utility_probabilities,
1312
_sample_utility_per_building,
1413
)
14+
from utils.utility_codes import get_ny_open_data_to_std_name
15+
16+
17+
def _utility_name_map_lazy() -> pl.LazyFrame:
18+
"""Build utility_name_map from central crosswalk (matches assign_utility_ny)."""
19+
return pl.DataFrame(
20+
[
21+
{"state_name": k, "std_name": v}
22+
for k, v in get_ny_open_data_to_std_name().items()
23+
]
24+
).lazy()
1525

1626

1727
def test_calculate_utility_probabilities_basic():
@@ -29,7 +39,7 @@ def test_calculate_utility_probabilities_basic():
2939
"pct_overlap": [60.0, 30.0, 10.0, 50.0, 50.0],
3040
}
3141
)
32-
utility_name_map = pl.LazyFrame(CONFIGS["utility_name_map"])
42+
utility_name_map = _utility_name_map_lazy()
3343

3444
result = _calculate_utility_probabilities(
3545
puma_overlap,
@@ -60,7 +70,7 @@ def test_calculate_utility_probabilities_filter_none_false():
6070
"pct_overlap": [70.0, 30.0],
6171
}
6272
)
63-
utility_name_map = pl.LazyFrame(CONFIGS["utility_name_map"])
73+
utility_name_map = _utility_name_map_lazy()
6474

6575
result = _calculate_utility_probabilities(
6676
puma_overlap,
@@ -84,7 +94,7 @@ def test_calculate_utility_probabilities_handle_municipal():
8494
"pct_overlap": [40.0, 60.0],
8595
}
8696
)
87-
utility_name_map = pl.LazyFrame(CONFIGS["utility_name_map"])
97+
utility_name_map = _utility_name_map_lazy()
8898

8999
result = _calculate_utility_probabilities(
90100
puma_overlap,
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
"""Tests for electric tariff mapper."""
2+
3+
from typing import cast
4+
5+
import polars as pl
6+
7+
from utils.electric_tariff_mapper import map_electric_tariff
8+
from utils.types import SBScenario
9+
10+
11+
def test_map_electric_tariff_filters_by_std_name():
12+
"""map_electric_tariff filters metadata by sb.electric_utility (std_name)."""
13+
metadata = pl.LazyFrame(
14+
{
15+
"bldg_id": [1, 2, 3, 4],
16+
"sb.electric_utility": ["coned", "coned", "nyseg", "nimo"],
17+
"postprocess_group.has_hp": [True, False, True, False],
18+
}
19+
)
20+
sb_scenario = SBScenario("default", 1)
21+
result = map_electric_tariff(
22+
SB_metadata_df=metadata,
23+
electric_utility="coned",
24+
SB_scenario=sb_scenario,
25+
state="NY",
26+
)
27+
df = cast(pl.DataFrame, result.collect())
28+
assert df.height == 2
29+
assert set(df["bldg_id"].to_list()) == {1, 2}
30+
assert all(df["tariff_key"].str.starts_with("coned_"))
31+
32+
33+
def test_map_electric_tariff_tariff_key_format():
34+
"""tariff_key uses std_name and scenario."""
35+
metadata = pl.LazyFrame(
36+
{
37+
"bldg_id": [1],
38+
"sb.electric_utility": ["nyseg"],
39+
"postprocess_group.has_hp": [True],
40+
}
41+
)
42+
sb_scenario = SBScenario("seasonal", 1)
43+
result = map_electric_tariff(
44+
SB_metadata_df=metadata,
45+
electric_utility="nyseg",
46+
SB_scenario=sb_scenario,
47+
state="NY",
48+
)
49+
df = cast(pl.DataFrame, result.collect())
50+
assert df["tariff_key"][0] == "nyseg_seasonal_1_HP.csv"

tests/test_gas_tariff_mapper.py

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
"""Tests for gas tariff mapper."""
2+
3+
from typing import cast
4+
5+
import polars as pl
6+
7+
from utils.gas_tariff_mapper import map_gas_tariff
8+
9+
10+
def test_map_gas_tariff_uses_crosswalk_for_tariff_key():
11+
"""map_gas_tariff maps sb.gas_utility (std_name) to tariff_key via crosswalk."""
12+
metadata = pl.LazyFrame(
13+
{
14+
"bldg_id": [1, 2, 3],
15+
"sb.electric_utility": ["coned", "coned", "coned"],
16+
"sb.gas_utility": ["nimo", "nyseg", "coned"],
17+
}
18+
)
19+
result = map_gas_tariff(
20+
SB_metadata_df=metadata,
21+
electric_utility_name="coned",
22+
)
23+
df = cast(pl.DataFrame, result.collect())
24+
assert df.height == 3
25+
# nimo -> national_grid, nyseg -> nyseg, coned -> coned
26+
tariff_keys = df["tariff_key"].to_list()
27+
assert "national_grid" in tariff_keys
28+
assert "nyseg" in tariff_keys
29+
assert "coned" in tariff_keys

tests/test_get_utility_stats_from_eia861.py

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
dynamic aggregation is safe. Uses the same parquet URL as the script.
55
"""
66

7+
import subprocess
8+
79
import polars as pl
810

911
# Must match utils/get_utility_stats_from_eia861.py
@@ -71,3 +73,38 @@ def test_eia861_yearly_sales_has_investor_owned_data_for_sample_state():
7173
assert frozenset(classes_in_data) == EXPECTED_CUSTOMER_CLASSES, (
7274
f"NY IOU data missing some customer classes: got {set(classes_in_data)}"
7375
)
76+
77+
78+
def test_utility_code_column_present_and_mapped():
79+
"""Script output includes utility_code; known EIA IDs map to expected std_names."""
80+
from pathlib import Path
81+
82+
project_root = Path(__file__).resolve().parent.parent
83+
result = subprocess.run(
84+
["uv", "run", "python", "utils/get_utility_stats_from_eia861.py", "NY"],
85+
capture_output=True,
86+
text=True,
87+
cwd=project_root,
88+
)
89+
assert result.returncode == 0, f"Script failed: {result.stderr}"
90+
91+
from io import StringIO
92+
93+
df = pl.read_csv(StringIO(result.stdout))
94+
assert "utility_code" in df.columns
95+
96+
# Known EIA ID -> std_name mappings
97+
expected = {
98+
4226: "coned",
99+
13573: "nimo",
100+
13511: "nyseg",
101+
16183: "rge",
102+
3249: "cenhud",
103+
14154: "or",
104+
}
105+
for eia_id, std_name in expected.items():
106+
row = df.filter(pl.col("utility_id_eia") == eia_id)
107+
assert not row.is_empty(), f"EIA ID {eia_id} not in output"
108+
assert row["utility_code"][0] == std_name, (
109+
f"EIA ID {eia_id}: expected utility_code {std_name}, got {row['utility_code'][0]}"
110+
)

0 commit comments

Comments
 (0)