Skip to content

Commit 1d0b7cc

Browse files
authored
Improve query efficiency for profile inputs (#1363)
* Add indexes for common lookups (this changes the order of inputs, so some small changes to test example objective function values are observed) * CTE-based queries for profiles
1 parent 1b5a3ae commit 1d0b7cc

184 files changed

Lines changed: 3167 additions & 3245 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

db/db_schema.sql

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -345,6 +345,11 @@ CREATE TABLE inputs_temporal
345345
FOREIGN KEY (month) REFERENCES mod_months (month)
346346
);
347347

348+
-- Composite index for temporal lookups
349+
CREATE INDEX idx_temporal_scenario_subproblem_stage
350+
ON inputs_temporal(temporal_scenario_id, subproblem_id, stage_id, timepoint);
351+
352+
348353
-- Horizons (with balancing types)
349354
-- How timepoints are organized for operational-decision purposes
350355
-- Each timepoint can belong to more than one balancing_type-horizon (e.g.
@@ -429,6 +434,10 @@ CREATE TABLE inputs_temporal_horizon_timepoints
429434
balancing_type_horizon, horizon)
430435
);
431436

437+
-- Index for horizon timepoints lookups
438+
CREATE INDEX idx_horizon_temporal_scenario_subproblem_stage
439+
ON inputs_temporal_horizon_timepoints(temporal_scenario_id, subproblem_id, stage_id);
440+
432441

433442
---------------------
434443
-- -- GEOGRAPHY -- --
@@ -1669,6 +1678,10 @@ CREATE TABLE inputs_project_portfolios
16691678
FOREIGN KEY (capacity_type) REFERENCES mod_capacity_types (capacity_type)
16701679
);
16711680

1681+
-- Index for filtering by project_portfolio_scenario_id in inputs_project_portfolios
1682+
CREATE INDEX idx_project_portfolios_scenario
1683+
ON inputs_project_portfolios(project_portfolio_scenario_id);
1684+
16721685
-- Existing project capacity and fixed costs
16731686
-- The capacity and fixed costs of 'specified' projects, i.e. exogenously
16741687
-- specified capacity that is not a variable in the model
@@ -2138,6 +2151,11 @@ CREATE TABLE inputs_project_operational_chars
21382151
(operational_type)
21392152
);
21402153

2154+
-- Index for filtering operational_type in inputs_project_operational_chars
2155+
CREATE INDEX idx_operational_chars_scenario_optype
2156+
ON inputs_project_operational_chars(project_operational_chars_scenario_id, operational_type);
2157+
2158+
21412159
-- Variable O&M by period
21422160
DROP TABLE IF EXISTS subscenarios_project_variable_om_cost_by_period;
21432161
CREATE TABLE subscenarios_project_variable_om_cost_by_period

examples/2horizons_w_hydro/inputs/projects.tab

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@ project capacity_type availability_type operational_type balancing_type_project
22
Coal gen_spec exogenous gen_commit_cap day 0 0.0 Coal Zone1 1.0 0.4 6.0 1.0 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . . . . .
33
Gas_CCGT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 1.0 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
44
Gas_CT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 0.0 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-
Hydro gen_spec . gen_hydro day 0 0.0 Hydro Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
65
Nuclear gen_spec . gen_must_run day 0 0.0 Nuclear Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
76
Wind gen_spec . gen_var day 0 0.0 Wind Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7+
Hydro gen_spec . gen_hydro day 0 0.0 Hydro Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .

examples/2horizons_w_hydro_and_nuclear_binary_availability/inputs/projects.tab

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@ project capacity_type availability_type operational_type balancing_type_project
22
Coal gen_spec . gen_commit_lin day 0 0.0 Coal Zone1 1.0 0.4 . . 0.0 . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . . . . .
33
Gas_CCGT gen_spec . gen_commit_lin day 0 0.0 Gas Zone1 2.0 0.4 . . 2.0 . . . . . . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
44
Gas_CT gen_spec . gen_commit_lin day 0 0.0 Gas Zone1 2.0 0.4 . . 1.0 . . . . . . . . . . . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-
Hydro gen_spec . gen_hydro_must_take day 0 0.0 Hydro Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
65
Nuclear gen_spec binary gen_must_run day 0 0.0 Nuclear Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
76
Wind gen_spec . gen_var day 0 0.0 Wind Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7+
Hydro gen_spec . gen_hydro_must_take day 0 0.0 Hydro Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .

examples/2horizons_w_hydro_w_balancing_types/inputs/projects.tab

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@ project capacity_type availability_type operational_type balancing_type_project
22
Coal gen_spec exogenous gen_commit_cap day 0 0.0 Coal Zone1 1.0 0.4 6.0 1.0 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . . . . .
33
Gas_CCGT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 1.0 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
44
Gas_CT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 0.0 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5-
Hydro gen_spec . gen_hydro year 0 0.0 Hydro Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
65
Nuclear gen_spec . gen_must_run day 0 0.0 Nuclear Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
76
Wind gen_spec . gen_var day 0 0.0 Wind Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7+
Hydro gen_spec . gen_hydro year 0 0.0 Hydro Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
project capacity_type availability_type operational_type balancing_type_project load_modifier_flag distribution_loss_adjustment_factor technology load_zone variable_om_cost_per_mwh min_stable_level_fraction unit_size_mw startup_cost_per_mw shutdown_cost_per_mw startup_fuel_mmbtu_per_mw startup_plus_ramp_up_rate shutdown_plus_ramp_down_rate ramp_up_when_on_rate ramp_down_when_on_rate ramp_up_violation_penalty ramp_down_violation_penalty ramp_tuning_cost_per_mw min_up_time_hours min_up_time_violation_penalty min_down_time_hours min_down_time_violation_penalty allow_startup_shutdown_power storage_efficiency charging_efficiency discharging_efficiency charging_capacity_multiplier discharging_capacity_multiplier minimum_duration_hours maximum_duration_hours aux_consumption_frac_capacity aux_consumption_frac_power last_commitment_stage cap_factor_default powerunithour_per_fuelunit soc_penalty_cost_per_energyunit soc_last_tmp_penalty_cost_per_energyunit max_losses_in_hrz_frac_stor_energy_capacity partial_availability_threshold nonfuel_carbon_emissions_per_mwh powerhouse generator_efficiency linked_load_component efficiency_factor energy_requirement_factor losses_factor_in_energy_target losses_factor_curtailment upward_reserves_to_soc_depletion lf_reserves_up_ba lf_reserves_up_derate lf_reserves_down_ba lf_reserves_down_derate regulation_up_ba regulation_up_derate regulation_down_ba regulation_down_derate lf_reserves_up_ramp_rate lf_reserves_down_ramp_rate regulation_up_ramp_rate regulation_down_ramp_rate
2-
Clunky_Old_Gen gen_ret_bin . gen_simple day 0 0.0 Coal Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
Coal gen_spec exogenous gen_commit_cap day 0 0.0 Coal Zone1 1.0 0.4 6.0 1.0 0.0 . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . . . . .
43
Gas_CCGT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 1.0 2.0 . . . . . . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
54
Gas_CT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 0.0 1.0 . . . . . . . . . . . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . .
65
Nuclear gen_spec . gen_must_run day 0 0.0 Nuclear Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
76
Wind gen_spec . gen_var day 0 0.0 Wind Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7+
Clunky_Old_Gen gen_ret_bin . gen_simple day 0 0.0 Coal Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . . . . . . . . .

examples/2periods_gen_bin_econ_retirement/inputs/spec_capacity_period_params.tab

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
11
project period specified_capacity_mw specified_energy_mwh shaping_capacity_mw hyb_gen_specified_capacity_mw hyb_stor_specified_capacity_mw specified_stor_capacity_mwh fuel_production_capacity_fuelunitperhour fuel_release_capacity_fuelunitperhour fuel_storage_capacity_fuelunit fixed_cost_per_mw_yr fixed_cost_per_energy_mwh_yr fixed_cost_per_shaping_mw_yr hyb_gen_fixed_cost_per_mw_yr hyb_stor_fixed_cost_per_mw_yr fixed_cost_per_stor_mwh_yr fuel_production_capacity_fixed_cost_per_fuelunitperhour_yr fuel_release_capacity_fixed_cost_per_fuelunitperhour_yr fuel_storage_capacity_fixed_cost_per_fuelunit_yr
2-
Clunky_Old_Gen 2020 10.0 1000000.0
3-
Clunky_Old_Gen 2030 10.0 1000000.0
42
Coal 2020 6.0
53
Coal 2030 6.0
64
Gas_CCGT 2020 6.0
@@ -11,3 +9,5 @@ Nuclear 2020 6.0
119
Nuclear 2030 6.0
1210
Wind 2020 2.0
1311
Wind 2030 2.0
12+
Clunky_Old_Gen 2020 10.0 1000000.0
13+
Clunky_Old_Gen 2030 10.0 1000000.0
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
project capacity_type availability_type operational_type balancing_type_project load_modifier_flag distribution_loss_adjustment_factor technology load_zone variable_om_cost_per_mwh min_stable_level_fraction unit_size_mw startup_cost_per_mw shutdown_cost_per_mw startup_fuel_mmbtu_per_mw startup_plus_ramp_up_rate shutdown_plus_ramp_down_rate ramp_up_when_on_rate ramp_down_when_on_rate ramp_up_violation_penalty ramp_down_violation_penalty ramp_tuning_cost_per_mw min_up_time_hours min_up_time_violation_penalty min_down_time_hours min_down_time_violation_penalty allow_startup_shutdown_power storage_efficiency charging_efficiency discharging_efficiency charging_capacity_multiplier discharging_capacity_multiplier minimum_duration_hours maximum_duration_hours aux_consumption_frac_capacity aux_consumption_frac_power last_commitment_stage cap_factor_default powerunithour_per_fuelunit soc_penalty_cost_per_energyunit soc_last_tmp_penalty_cost_per_energyunit max_losses_in_hrz_frac_stor_energy_capacity partial_availability_threshold nonfuel_carbon_emissions_per_mwh powerhouse generator_efficiency linked_load_component efficiency_factor energy_requirement_factor losses_factor_in_energy_target losses_factor_curtailment upward_reserves_to_soc_depletion lf_reserves_up_ba lf_reserves_up_derate lf_reserves_down_ba lf_reserves_down_derate regulation_up_ba regulation_up_derate regulation_down_ba regulation_down_derate lf_reserves_up_ramp_rate lf_reserves_down_ramp_rate regulation_up_ramp_rate regulation_down_ramp_rate
2-
Clunky_Old_Gen gen_ret_lin . gen_simple day 0 0.0 Coal Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
Coal gen_spec exogenous gen_commit_cap day 0 0.0 Coal Zone1 1.0 0.4 6.0 1.0 0.0 . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . . . . .
43
Gas_CCGT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 1.0 2.0 . . . . . . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
54
Gas_CT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 0.0 1.0 . . . . . . . . . . . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . .
65
Nuclear gen_spec . gen_must_run day 0 0.0 Nuclear Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
76
Wind gen_spec . gen_var day 0 0.0 Wind Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7+
Clunky_Old_Gen gen_ret_lin . gen_simple day 0 0.0 Coal Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . . . . . . . . .

examples/2periods_gen_lin_econ_retirement/inputs/spec_capacity_period_params.tab

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
11
project period specified_capacity_mw specified_energy_mwh shaping_capacity_mw hyb_gen_specified_capacity_mw hyb_stor_specified_capacity_mw specified_stor_capacity_mwh fuel_production_capacity_fuelunitperhour fuel_release_capacity_fuelunitperhour fuel_storage_capacity_fuelunit fixed_cost_per_mw_yr fixed_cost_per_energy_mwh_yr fixed_cost_per_shaping_mw_yr hyb_gen_fixed_cost_per_mw_yr hyb_stor_fixed_cost_per_mw_yr fixed_cost_per_stor_mwh_yr fuel_production_capacity_fixed_cost_per_fuelunitperhour_yr fuel_release_capacity_fixed_cost_per_fuelunitperhour_yr fuel_storage_capacity_fixed_cost_per_fuelunit_yr
2-
Clunky_Old_Gen 2020 10.0 1000000.0
3-
Clunky_Old_Gen 2030 10.0 1000000.0
42
Coal 2020 6.0
53
Coal 2030 6.0
64
Gas_CCGT 2020 6.0
@@ -11,3 +9,5 @@ Nuclear 2020 6.0
119
Nuclear 2030 6.0
1210
Wind 2020 2.0
1311
Wind 2030 2.0
12+
Clunky_Old_Gen 2020 10.0 1000000.0
13+
Clunky_Old_Gen 2030 10.0 1000000.0
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
project capacity_type availability_type operational_type balancing_type_project load_modifier_flag distribution_loss_adjustment_factor technology load_zone variable_om_cost_per_mwh min_stable_level_fraction unit_size_mw startup_cost_per_mw shutdown_cost_per_mw startup_fuel_mmbtu_per_mw startup_plus_ramp_up_rate shutdown_plus_ramp_down_rate ramp_up_when_on_rate ramp_down_when_on_rate ramp_up_violation_penalty ramp_down_violation_penalty ramp_tuning_cost_per_mw min_up_time_hours min_up_time_violation_penalty min_down_time_hours min_down_time_violation_penalty allow_startup_shutdown_power storage_efficiency charging_efficiency discharging_efficiency charging_capacity_multiplier discharging_capacity_multiplier minimum_duration_hours maximum_duration_hours aux_consumption_frac_capacity aux_consumption_frac_power last_commitment_stage cap_factor_default powerunithour_per_fuelunit soc_penalty_cost_per_energyunit soc_last_tmp_penalty_cost_per_energyunit max_losses_in_hrz_frac_stor_energy_capacity partial_availability_threshold nonfuel_carbon_emissions_per_mwh powerhouse generator_efficiency linked_load_component efficiency_factor energy_requirement_factor losses_factor_in_energy_target losses_factor_curtailment upward_reserves_to_soc_depletion lf_reserves_up_ba lf_reserves_up_derate lf_reserves_down_ba lf_reserves_down_derate regulation_up_ba regulation_up_derate regulation_down_ba regulation_down_derate lf_reserves_up_ramp_rate lf_reserves_down_ramp_rate regulation_up_ramp_rate regulation_down_ramp_rate
22
Coal gen_spec exogenous gen_commit_cap day 0 0.0 Coal Zone1 1.0 0.4 6.0 1.0 0.0 . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . Zone1 . Zone1 . . . . .
33
Gas_CCGT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 1.0 2.0 . . . . . . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
4-
Gas_CCGT_new gen_new_lin . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 1.0 2.0 . . . . . . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
54
Gas_CT gen_spec . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 0.0 1.0 . . . . . . . . . . . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . .
6-
Gas_CT_new gen_new_lin . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 0.0 1.0 . . . . . . . . . . . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . .
75
Nuclear gen_spec . gen_must_run day 0 0.0 Nuclear Zone1 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
86
Wind gen_spec . gen_var day 0 0.0 Wind Zone1 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7+
Gas_CCGT_new gen_new_lin . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 1.0 2.0 . . . . . . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . . . Zone1 . Zone1 . Zone1 . Zone1 . . . . .
8+
Gas_CT_new gen_new_lin . gen_commit_cap day 0 0.0 Gas Zone1 2.0 0.4 6.0 0.0 1.0 . . . . . . . . . . . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . .
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
-10085845900.19
1+
-10085845904.57

0 commit comments

Comments
 (0)