GridPath v2025.10.3

Merge pull request #1322 from blue-marble/develop

Author: anamileva

db/__init__.py

@@ -1,115 +0,0 @@
-# Copyright 2016-2023 Blue Marble Analytics LLC.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-All tables names in the GridPath database start with one of seven prefixes:
-:code:`mod_`, :code:`subscenario_`, :code:`inputs_`, :code:`scenarios`,
-:code:`options_`, :code:`status_`, or :code:`ui_`. This structure is meant to
-organize the tables by their function. Below are descriptions of each table
-type and its role, and of the kind of data tables of this type contain.
-
-***********************
-The :code:`mod_` Tables
-***********************
-The :code:`mod_` should not be modified except by developers. These contain
-various data used by the GridPath platform to describe available
-functionality, help enforce input data consistency and integrity, and aid in
-validation.
-
-***************************************************
-The :code:`subscenario_` and :code:`inputs_` Tables
-***************************************************
-Most tables in the GridPath database have the :code:`subscenario_` and
-:code:`inputs_` prefix. With a few exceptions, for each :code:`subscenario_`
-table, there is a respective :code:`inputs_` table (i.e. the tables have the
-same name except for the prefix). This is because the :code:`subscenario_`
-tables contain the descriptions of the input data contained in the
-:code:`inputs_` tables. For example the :code:`inputs_system_load` may
-contain three different load profiles -- low, mid, and high; the
-:code:`subscenarios_system_load` will then contain three rows, one for each
-load profile, with its description and ID. The pairs of :code:`subscenario_`
-and :code:`inputs_` are linked via an ID column: in the case of the system
-load tables, that is the :code:`load_scenario_id` column. We call these
-shared table keys *subscenario IDs*, as we use them to create a full
-GridPath scenario in the :code:`scenarios` table.
-
-***************************
-The :code:`scenarios` Table
-***************************
-In GridPath, we use the term 'scenario' to describe a model run with a
-particular set of inputs. Some of those inputs stay the same from scenario to
-scenario and others we vary to understand their effect on the results. For
-example, we could keep some input types like the zonal and transmission
-topography, temporal resolution, resource availability, and policy
-requirements the same across scenarios, but vary other input types, e.g. the
-load profile, the cost of solar, and the operational characteristics of coal,
-to create different scenarios. We call each of those inputs types a
-'subscenario' since they are the building blocks of a full scenario. In
-GridPath, you can create a scenario by populating a row of the
-:code:`scenarios` table. The columns of the :code:`scenarios` table are
-linked one of the 'building blocks' -- the data in :code:`inputs_` tables --
-via the respective *subscenario ID*.
-
-For example, the :code:`load_scenario_id` column of the :code:`scenarios` table
-references the :code:`load_scenario_id` column of the
-:code:`subscenarios_system_load` table, which in turn determines which load
-profile contained in the :code:`inputs_system_load` table the scenario
-should use. In our example with three different load profiles, the data for
-which are contained in the :code:`inputs_system_load` table,
-:code:`subscenarios_system_load` will contain three rows with values of 1,
-2, and 3 respectively in the :code:`load_scenario_id` column; in the
-:code:`scenarios` table, the user would then be able to select a value of 1,
-2, or 3 in the :code:`load_scenario_id` column to determine which load
-profile the scenario should use. Similarly, we would select the solar costs
-to use in the scenario via the :code:`projects_new_cost_scenario_id` column
-of the :code:`scenarios` table (which is linked to the
-:code:`subscenarios_project_new_cost` and :code:`inputs_project_new_cost`
-tables) and the operational characteristics of coal to use via the
-:code:`project_operational_chars_scenario_id` column (which is linked to the
-:code:`subscenarios_project_operational_chars` and
-:code:`inputs_project_operational_chars` tables).
-
-***************************
-The :code:`options_` Tables
-***************************
-Some GridPath run options can be specified via the database in the
-:code:`options_` tables. Currently, this includes the solver options that
-can be specified for a scenario run
-
-**************************
-The :code:`status_` Tables
-**************************
-GridPath keeps track of scenario validation and run status. The scenario
-status is recorded in the :code:`scenarios` table (in the
-:code:`validation_status_id` and :code:`run_status_id` columns) and an
-additional detail can be found in the :code:`status_` tables. Currently,
-this includes a single table: the :code:`status_validation` table, which
-contains information about errors encountered during validation for each
-scenario that has been validated.
-
-**********************
-The :code:`ui_` Tables
-**********************
-The :code:`ui_` tables are used to include and exclude components of the
-GridPath user interface.
-
-***********************
-The :code:`viz_` Tables
-***********************
-The :code:`viz_` tables are used in the GridPath visualization suite, for
-instance when determining in which color and order to plot the technologies in
-the dispatch plot.
-
-
-"""

db/doc.py

@@ -0,0 +1,116 @@
+# Copyright 2016-2025 Blue Marble Analytics LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+"""
+All tables names in the GridPath database start with one of seven prefixes:
+:code:`mod_`, :code:`subscenario_`, :code:`inputs_`, :code:`scenarios`,
+:code:`options_`, :code:`status_`, or :code:`ui_`. This structure is meant to
+organize the tables by their function. Below are descriptions of each table
+type and its role, and of the kind of data tables of this type contain.
+
+***********************
+The :code:`mod_` Tables
+***********************
+The :code:`mod_` should not be modified except by developers. These contain
+various data used by the GridPath platform to describe available
+functionality, help enforce input data consistency and integrity, and aid in
+validation.
+
+***************************************************
+The :code:`subscenario_` and :code:`inputs_` Tables
+***************************************************
+Most tables in the GridPath database have the :code:`subscenario_` and
+:code:`inputs_` prefix. With a few exceptions, for each :code:`subscenario_`
+table, there is a respective :code:`inputs_` table (i.e. the tables have the
+same name except for the prefix). This is because the :code:`subscenario_`
+tables contain the descriptions of the input data contained in the
+:code:`inputs_` tables. For example the :code:`inputs_system_load` may
+contain three different load profiles -- low, mid, and high; the
+:code:`subscenarios_system_load` will then contain three rows, one for each
+load profile, with its description and ID. The pairs of :code:`subscenario_`
+and :code:`inputs_` are linked via an ID column: in the case of the system
+load tables, that is the :code:`load_scenario_id` column. We call these
+shared table keys *subscenario IDs*, as we use them to create a full
+GridPath scenario in the :code:`scenarios` table.
+
+***************************
+The :code:`scenarios` Table
+***************************
+In GridPath, we use the term 'scenario' to describe a model run with a
+particular set of inputs. Some of those inputs stay the same from scenario to
+scenario and others we vary to understand their effect on the results. For
+example, we could keep some input types like the zonal and transmission
+topography, temporal resolution, resource availability, and policy
+requirements the same across scenarios, but vary other input types, e.g. the
+load profile, the cost of solar, and the operational characteristics of coal,
+to create different scenarios. We call each of those inputs types a
+'subscenario' since they are the building blocks of a full scenario. In
+GridPath, you can create a scenario by populating a row of the
+:code:`scenarios` table. The columns of the :code:`scenarios` table are
+linked one of the 'building blocks' -- the data in :code:`inputs_` tables --
+via the respective *subscenario ID*.
+
+For example, the :code:`load_scenario_id` column of the :code:`scenarios` table
+references the :code:`load_scenario_id` column of the
+:code:`subscenarios_system_load` table, which in turn determines which load
+profile contained in the :code:`inputs_system_load` table the scenario
+should use. In our example with three different load profiles, the data for
+which are contained in the :code:`inputs_system_load` table,
+:code:`subscenarios_system_load` will contain three rows with values of 1,
+2, and 3 respectively in the :code:`load_scenario_id` column; in the
+:code:`scenarios` table, the user would then be able to select a value of 1,
+2, or 3 in the :code:`load_scenario_id` column to determine which load
+profile the scenario should use. Similarly, we would select the solar costs
+to use in the scenario via the :code:`projects_new_cost_scenario_id` column
+of the :code:`scenarios` table (which is linked to the
+:code:`subscenarios_project_new_cost` and :code:`inputs_project_new_cost`
+tables) and the operational characteristics of coal to use via the
+:code:`project_operational_chars_scenario_id` column (which is linked to the
+:code:`subscenarios_project_operational_chars` and
+:code:`inputs_project_operational_chars` tables).
+
+***************************
+The :code:`options_` Tables
+***************************
+Some GridPath run options can be specified via the database in the
+:code:`options_` tables. Currently, this includes the solver options that
+can be specified for a scenario run
+
+**************************
+The :code:`status_` Tables
+**************************
+GridPath keeps track of scenario validation and run status. The scenario
+status is recorded in the :code:`scenarios` table (in the
+:code:`validation_status_id` and :code:`run_status_id` columns) and an
+additional detail can be found in the :code:`status_` tables. Currently,
+this includes a single table: the :code:`status_validation` table, which
+contains information about errors encountered during validation for each
+scenario that has been validated.
+
+**********************
+The :code:`ui_` Tables
+**********************
+The :code:`ui_` tables are used to include and exclude components of the
+GridPath user interface.
+
+***********************
+The :code:`viz_` Tables
+***********************
+The :code:`viz_` tables are used in the GridPath visualization suite, for
+instance when determining in which color and order to plot the technologies in
+the dispatch plot.
+
+
+"""

db/utilities/scenario.py

@@ -18,6 +18,9 @@
 
 >>> gridpath_load_scenarios --database PATH/DO/DB --csv_path PATH/TO/SCENARIO/CSV
 
+If you are using the csvs_test_examples directory included with GridPath, /PATH/TO/SCENARIO/CSV
+can be set to ../csvs_test_examples/scenarios.csv.
+
 To load a single scenario by name, use the *--scenario* flag. To delete a scenario from
 the database, specify the scenario name with the *--scenario* flag and use the
 *--delete* flag.

doc/graphics/gridpath_workflow.png

@@ -1,4 +1,4 @@
-# Copyright 2016-2024 Blue Marble Analytics LLC.
+# Copyright 2016-2025 Blue Marble Analytics LLC.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -28,7 +28,7 @@
 )
 
 d2i_tbl_hdr_dict = dict(
-    align="right", fill_color="blue", font_color="white", line_color="darkslategray"
+    align="right", fill_color="#5176BF", font_color="white", line_color="darkslategray"
 )
 d2i_tbl_cel_dict = dict(align="right", line_color="darkslategray")
 
@@ -45,24 +45,61 @@
     tbl_header=d2i_tbl_hdr_dict,
     tbl_cells=d2i_tbl_cel_dict,
     show_fig=False,
-    row_fill_color=("#ffffff", "#d7d8d6"),
+    row_fill_color=("#ffffff", "#5176BF"),
     print_index=False,
-    fig_size=(1800, 500),
+    fig_size=(3500, 1000),
     col_width=tuple(col_width_both),
 )
 
 fig_table = fig.data[0]
 rr1, cc1 = np.where(df == "required")
-for validx in range(len(rr1)):
-    ra = rr1[validx]
-    ca = cc1[validx]
-    fig_table.cells.fill.color[ca][ra] = "#FF0000"
-
 rr2, cc2 = np.where(df == "optional")
-for validx in range(len(rr2)):
-    rb = rr2[validx]
-    cb = cc2[validx]
-    fig_table.cells.fill.color[cb][rb] = "#FFFF00"
+
+# python
+# prepare a mutable, normalized column-first list-of-lists for cell fill colors
+colors = list(fig_table.cells.fill.color)
+
+# determine actual number of rows in the table (includes header row)
+num_rows_in_table = 0
+for col in colors:
+    if isinstance(col, (tuple, list)):
+        num_rows_in_table = max(num_rows_in_table, len(col))
+    else:
+        num_rows_in_table = max(num_rows_in_table, 1)
+if num_rows_in_table == 0:
+    num_rows_in_table = len(df) + 1
+
+# normalize each column to be a list of length num_rows_in_table
+normalized = []
+for col in colors:
+    if not isinstance(col, list):
+        col_list = list(col) if isinstance(col, (tuple, list)) else [col]
+    else:
+        col_list = col
+    if len(col_list) < num_rows_in_table:
+        col_list = col_list + ["#ffffff"] * (num_rows_in_table - len(col_list))
+    elif len(col_list) > num_rows_in_table:
+        col_list = col_list[:num_rows_in_table]
+    normalized.append(list(col_list))
+colors = normalized
+
+# compute offset between dataframe row indices and table row indices (header row)
+header_offset = num_rows_in_table - len(df)
+if header_offset < 0:
+    header_offset = 0
+
+# set required cells to red (apply header_offset)
+for r, c in zip(rr1, cc1):
+    if 0 <= c < len(colors) and 0 <= (r + header_offset) < len(colors[c]):
+        colors[c][r + header_offset] = "#BF5176"
+
+# set optional cells to yellow (apply header_offset)
+for r, c in zip(rr2, cc2):
+    if 0 <= c < len(colors) and 0 <= (r + header_offset) < len(colors[c]):
+        colors[c][r + header_offset] = "#51BF9B"
+
+# assign back the modified colors
+fig_table.cells.fill.color = colors
 
 df2img.save_dataframe(fig=fig, filename="graphics/optype_opchar_matrix.png")
 df2img.save_dataframe(fig=fig, filename="graphics/optype_opchar_matrix.svg")

doc/graphics/optype_opchar_matrix.png

@@ -18,6 +18,7 @@
 
 import os
 import sys
+import sphinx_rtd_theme
 
 sys.path.insert(0, os.path.abspath("../../"))
 
@@ -36,6 +37,7 @@
     "sphinx.ext.todo",
     "sphinx.ext.viewcode",
     "sphinxarg.ext",
+    "sphinx_rtd_theme",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -52,7 +54,7 @@
 
 # General information about the project.
 project = "GridPath"
-copyright = "2016-2022, Blue Marble Analytics LLC"
+copyright = "2016-2025, Blue Marble Analytics LLC"
 author = "Blue Marble Analytics LLC"
 
 # The version info for the project you're documenting, acts as replacement for
@@ -95,7 +97,7 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 # http://www.sphinx-doc.org/en/stable/theming.html
-html_theme = "nature"
+html_theme = "sphinx_rtd_theme"
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the