PyPSA · trevorb1 · Jan 3, 2025 · Jan 7, 2025 · Jan 7, 2025 · Jan 7, 2025
diff --git a/.isort.cfg b/.isort.cfg
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -17,19 +17,13 @@ repos:
   - id: pretty-format-yaml
     args: [--autofix, --indent, "2", --preserve-quotes]
 
-- repo: https://github.com/pycqa/isort
-  rev: 5.13.2
-  hooks:
-  - id: isort
-    name: isort (python)
-
 - repo: https://github.com/asottile/add-trailing-comma
   rev: v3.1.0
   hooks:
   - id: add-trailing-comma
 
 - repo: https://github.com/asottile/pyupgrade
-  rev: v3.19.0
+  rev: v3.19.1
   hooks:
   - id: pyupgrade
     args: [--py39-plus]
@@ -45,31 +39,15 @@ repos:
   hooks:
   - id: snakefmt
 
-# - repo: https://github.com/PyCQA/docformatter
-#   rev: v1.7.5
-#   hooks:
-#   - id: docformatter
-#     args: ["--in-place", "--make-summary-multi-line", "--pre-summary-newline"]
-
-
 - repo: https://github.com/keewis/blackdoc
   rev: v0.3.9
   hooks:
   - id: blackdoc
 
-- repo: https://github.com/psf/black-pre-commit-mirror
-  rev: 24.10.0
-  hooks:
-  - id: black
-    language_version: python3.11
-    args: ["--line-length", "120"]
-  - id: black-jupyter
-    language_version: python3.11
-    args: ["--line-length", "120"]
-
-
-- repo: https://github.com/pycqa/flake8
-  rev: 7.1.1
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  rev: v0.1.4  # Check for the latest version
   hooks:
-  - id: flake8
-    args: [--select=F841]    # F841 is the error code for unused variables
+  - id: ruff
+    args: ["--config=pyproject.toml", "--fix"]
+  - id: ruff-format
+    args: ["--config=pyproject.toml"]
diff --git a/README.md b/README.md
@@ -1,5 +1,7 @@
 [![DOI](https://zenodo.org/badge/500892486.svg)](https://zenodo.org/doi/10.5281/zenodo.10815964)
 [![Documentation Status](https://readthedocs.org/projects/pypsa-usa/badge/?version=latest)](https://pypsa-usa.readthedocs.io/en/latest/?badge=latest)
+[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
 # PyPSA-USA: An Open-Source Energy System Optimization Model for the United States
 

diff --git a/docs/source/about-install.md b/docs/source/about-install.md
@@ -27,23 +27,42 @@ templates into the `workflow/config` folder.
 bash init_pypsa_usa.sh
 ```
 
-## Step 3: Create Mamba Environment
+## Step 3: Set-up Environment (mamba or UV)
 
-PyPSA-USA uses conda/mamba to manage project dependencies. You can download and install mamba following the [instructions](https://mamba.readthedocs.io/en/latest/mamba-installation.html). Follow links for mambaforge installation. There are two ways to install mamba, the first (recommended) method will start with a fresh install, meaning if you have previously installed conda environments, you will need to recreate these conda envs. If you already have conda installed and do not wish to install mamba, you can follow the same set of instructions replacing any `mamba` with `conda`
+PyPSA-USA can be managed though either [`UV`](https://github.com/astral-sh/uv) or [`mamba`](https://github.com/mamba-org/mamba). Users only need to install one, not both!
 
-Once mamba is installed, use the environment file within your git repository to activate the `pypsa-usa` conda environment. This step can take ~10-20 minutes. After creating the mamba environment, you will only need to activate it before running the snakemake workflow.
+```{seealso}
+If you are planning to develop `PyPSA-USA`, please see our [contribution guidelines](./contributing.md#code-contributions) for installing additional dependencies.
+```
+
+### Step 3a: `uv` installation
+
+[`UV`](https://docs.astral.sh/uv/) is a new python package managment tool from [`Astral`](https://astral.sh/), the creators of [`ruff`](https://github.com/astral-sh/ruff). It replaces `mamba`, `conda`, and `pip` commands for one package and virtual environment managment tool. Instructions for installing `UV` can be found [here](https://docs.astral.sh/uv/getting-started/installation/).
+
+Once `UV` is installed, you can activate the environemnt with:
 
 ```console
-mamba env create -f workflow/envs/environment.yaml
-mamba activate pypsa-usa
+uv venv
+source .venv/bin/activate
+```
+
+```{warning}
+If you are migrating from `mamba`/`conda`, you may need to install system level dependencies that conda has previously handeled. These include, `HDF5` and `GDAL>=3.1` libraries.
 ```
 
-You also have the option to use miniconda. Download [Miniconda](https://docs.conda.io/en/latest/miniconda.html) following their [instructions](https://docs.conda.io/en/latest/miniconda.html).
+### Step 3b: `mamba` Installation
 
-```{seealso}
-If you are planning to develop PyPSA-USA, please see our [contribution guidelines](./contributing.md#code-contributions) for installing additional dependencies
+Alternatively, PyPSA-USA can manage project dependencies through `conda`/`mamba`. You can download and install `mamba` following the [instructions](https://mamba.readthedocs.io/en/latest/mamba-installation.html). Follow links for mambaforge installation. There are two ways to install `mamba`, the first (recommended) method will start with a fresh install, meaning if you have previously installed `conda` environments, you will need to recreate these `conda` envs. If you already have `conda` installed and do not wish to install `mamba`, you can follow the same set of instructions replacing any `mamba` with `conda`
+
+Once `mamba` is installed, use the environment file within the git repository to create the PyPSA-USA conda environment. This step can take ~10-20 minutes. After creating the mamba environment, you will need to activate it before running the snakemake workflow.
+
+```console
+mamba env create -f workflow/envs/environment.yaml
+mamba activate pypsa-usa
 ```
 
+You also have the option to use `miniconda`. Download [`miniconda`](https://docs.conda.io/en/latest/miniconda.html) following their [instructions](https://docs.conda.io/en/latest/miniconda.html).
+
 ## Step 4: Install a Solver
 
 PyPSA-USA uses an external solver to solve the optimization problem formulated in the workflow. After you install your solver and confirm it is accessible within your conda environment, update your solving configuration to match your solver of choice.
@@ -61,6 +80,6 @@ and the non-free, commercial software (for some of which free academic licenses
 
 ## Step 5: Get an EIA API Key
 
-The pypsa-usa workflow leverages the EIA API in several steps. The default configuration activates dynamic fuel-cost prices, which requires EIA API key. You can quickly get your key by completing this [form](https://www.eia.gov/opendata/register.php).
+The PyPSA-USA workflow leverages the EIA API in several steps. The default configuration activates dynamic fuel-cost prices, which requires EIA API key. You can quickly get your key by completing this [form](https://www.eia.gov/opendata/register.php).
 
 The API key will be emailed to you, and you can copy the key into the `config.api.yaml` file.
diff --git a/docs/source/about-usage.md b/docs/source/about-usage.md
@@ -16,11 +16,16 @@ You can find more information on each configuration setting on the [configuratio
 
 To run the workflow, `cd` into the `workflow` directory and run the `snakemake` from your terminal with your selection of config file:
 
+UV:
 ```console
-snakemake -j1 --configfile config/config.default.yaml
+uv run snakemake -j1 --configfile config/config.default.yaml --scheduler-ilp-solver GUROBI_CMD
 ```
 
-where 1 indicates the number of cores used.
+mamba:
+```console
+mamba activate pypsa-usa
+snakemake -j1 --configfile config/config.default.yaml
+```
 
 ## Running on HPC Cluster
 

diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -1,29 +1,30 @@
-# Configuration file for the Sphinx documentation builder.
-#
+"""Configuration file for the Sphinx documentation builder."""
+
 # For the full list of built-in configuration values, see the documentation:
 # https://www.sphinx-doc.org/en/master/usage/configuration.html
 # -- Project information -----------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
 
+import os
+import sys
+
 project = "pypsa-usa"
 copyright = "2024, Kamran Tehranchi, Trevor Barnes"
 author = "Kamran Tehranchi, Trevor Barnes"
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
 
-import os
-import sys
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 sys.path.insert(0, os.path.abspath("../../workflow/scripts"))
 
 extensions = [
-    #'sphinx.ext.autodoc',
-    #'sphinx.ext.autosummary',
+    # 'sphinx.ext.autodoc',
+    # 'sphinx.ext.autosummary',
     "myst_parser",
     # "sphinx.ext.autosectionlabel",
     "sphinx.ext.intersphinx",
@@ -32,9 +33,9 @@
     "sphinx.ext.napoleon",
     "sphinx.ext.graphviz",
     # "sphinxcontrib.bibtex",
-    #'sphinx.ext.pngmath',
-    #'sphinxcontrib.tikz',
-    #'rinoh.frontend.sphinx',
+    # 'sphinx.ext.pngmath',
+    # 'sphinxcontrib.tikz',
+    # 'rinoh.frontend.sphinx',
     "sphinx.ext.imgconverter",  # for SVG conversion
 ]
 myst_enable_extensions = ["html_image", "colon_fence", "amsmath"]

diff --git a/docs/source/config-co2-base.md b/docs/source/config-co2-base.md
diff --git a/docs/source/config-sectors.md b/docs/source/config-sectors.md
@@ -60,6 +60,23 @@ Only single-period studies are currently supported when running sector studies.
    :file: configtables/sector_service.csv
 ```
 
+:::{note}
+If running demand response at a per-carrier level, put each carrier as a key. For example:
+```yaml
+demand_response:
+  by_carrier: True
+  space-heat:
+    shift: 20
+    marginal_cost: 16
+  elec:
+    shift: 30
+    marginal_cost: 25
+  cool:
+    shift: 10
+    marginal_cost: 30
+```
+:::
+
 ## Transport Sector
 ```{eval-rst}
 .. literalinclude:: ../../workflow/repo_data/config/config.sector.yaml
@@ -84,3 +101,17 @@ Only single-period studies are currently supported when running sector studies.
    :widths: 22,7,22,33
    :file: configtables/sector_industrial.csv
 ```
+
+:::{note}
+If running demand response at a per-carrier level, put each carrier as a key. For example:
+```yaml
+demand_response:
+  by_carrier: True
+  elec:
+    shift: 30
+    marginal_cost: 25
+  heat:
+    shift: 10
+    marginal_cost: 30
+```
+:::
diff --git a/docs/source/config-wildcards.md b/docs/source/config-wildcards.md
@@ -75,6 +75,10 @@ The REM, SAFER, RPS can be defined using either the reeds zone name 'p##"
 the state code (eg, TX, CA, MT), pypsa-usa interconnect name (western, eastern, texas, usa),
 or nerc region name.
 
+```{warning}
+TCT Targets can only be used with renewable generators and utility scale batteries in sector studies.
+```
+
 There are currently:
 
 ```{eval-rst}
@@ -84,14 +88,9 @@ There are currently:
    :file: configtables/opts.csv
 ```
 
-
 (sector)=
 ## The `{sector}` wildcard
 
-```{warning}
-Sector coupling studies are all under active development
-```
-
 The `{sector}` wildcard is used to specify what sectors to include. If `None`
 is provided, an electrical only study is completed.
 
@@ -100,15 +99,6 @@ is provided, an electrical only study is completed.
 | Electricity | E    | Electrical sector. Will always be run.         | Runs        |
 | Natural Gas | G    | All sectors added                              | Development |
 
-(scope)=
-## The `{scope}` wildcard
-
-```{warning}
-Sector coupling studies are all under active development
-```
-
-Takes values `residential`, `urban`, `total`. Used in sector coupling studies to define
-population breakdown.
 
 (cutout_wc)=
 ## The `{cutout}` wildcard

diff --git a/docs/source/configtables/clustering.csv b/docs/source/configtables/clustering.csv
@@ -1,9 +1,11 @@
 ,Unit,Value,Description
 simplify_network:,,,
+weighting_strategy,str,"{population, demand-capacity}","When building multiple networks, Use 'population' if you want to ensure the clusters are assigned constant names."
 to_substations,bool,"{true, false}",Implementation curerntly overrides to true. Network is simplified to substation nodes with positive or negative power injection.
 algorithm,str,{'kmeans'},
 feature,str," {'solar+onwind-time', 'solar+onwind-cap', 'solar-time', 'solar-cap', 'solar+offwind-cap'}",For HAC clustering.
 cluster_network:,,,
+weighting_strategy,str,"{population, demand-capacity}","When building multiple networks, Use 'population' if you want to ensure the clusters are assigned constant names."
 algorithm,str,{'kmeans'},
 feature,str," {'solar+onwind-time', 'solar+onwind-cap', 'solar-time', 'solar-cap', 'solar+offwind-cap'}",For HAC clustering.
 aggregation_strategies:,,,

diff --git a/docs/source/configtables/electricity.csv b/docs/source/configtables/electricity.csv
@@ -4,13 +4,10 @@ renewable_carriers,--,"Any subset of {solar, onwind, offwind-ac, offwind-dc, hyd
 retirement, --,One of ``economic`` or ``technical``,"Sets the retirement method for converntional generators. If ``technical`` all generators ``p_nom_min`` are set to ``p_nom`` to prevent selling off of the asset. Retirements are then tracked in post-proccessing. If ``economic`` existing plants have their ``p_nom_min`` set as ``0``,  ``p_nom_max`` set to ``p_nom``,  and capital costs set to fixed costs. Generators with ``p_nom`` are then added to handle capacity expansion."""
 ,,,
 operational_reserve:,,,Settings for reserve requirements following `GenX <https://genxproject.github.io/GenX.jl/stable/Model_Reference/core/#Operational-Reserves>`_
-#NAME?,bool,true or false,Whether to take operational reserve requirements into account during optimisation
-#NAME?,--,float,share of total load
-#NAME?,--,float,share of total renewable supply
-#NAME?,MW,float,fixed reserve capacity
-,,,
-max_hours:,,,
-battery,h,float,Maximum state of charge capacity of the battery in terms of hours at full output capacity ``p_nom``. Cf. `PyPSA documentation <https://pypsa.readthedocs.io/en/latest/components.html#storage-unit>`_.
+activate,bool,true or false,Whether to take operational reserve requirements into account during optimisation
+epsilon_load,--,float,share of total load
+epsilon_vres,--,float,share of total renewable supply
+contingency,MW,float,fixed reserve capacity
 ,,,
 extendable_carriers:,,,
 Generator,--,Any extendable carrier,"Defines existing or non-existing conventional and renewable power plants to be extendable during the optimization. Conventional generators can only be built/expanded where already existent today. If a listed conventional carrier is not included in the ``conventional_carriers`` list, the lower limit of the capacity expansion is set to 0."
@@ -19,13 +16,15 @@ Store,--,Any subset of {``battery``},Adds extendable storage units (battery and/
 Links,--,Any subset of {},Adds extendable linksat every connection where there are lines or HVDC links without capacity limits and with zero initial capacity. Hydrogen pipelines require hydrogen storage to be modelled as ``Store``.
 ,,,
 demand:,,,
-#NAME?,--,"One of {``efs``, ``eia``}",Datasource for electrical load data. ``EFS`` pulls future state level electrical demand data. ``EIA`` pulls historical balancing level electrical demand dataa.
-#NAME?,--,"One of {``efs``, ``aeo``}, or a float",(UNDER DEVELOPMENT) Used to scale the demand profile data. ``AEO`` will scale according to demand projections from the Annual Energy Outlook. ``EFS`` will scale according to growth rates from the Electrification Futures Study. Or can sclae according to a user defined value.
-#NAME?,--,One of {``pop``},Method to dissagreagate load data. ``pop`` will dissagregate based on population from Breakthrough Energy.
+profile,--,"One of {``efs``, ``eia``}",Datasource for electrical load data. ``EFS`` pulls future state level electrical demand data. ``EIA`` pulls historical balancing level electrical demand dataa.
 scenario:,,,
-#NAME?,--,"One of {``reference``, ``medium``, ``high``}",(UNDER DEVELOPMENT) Extracts EFS data according to level of adoption
-#NAME?,--,"One of {``slow``, ``moderate``, ``fast``}",(UNDER DEVELOPMENT) Extracts EFS data according to speed of electrification
-#NAME?,--,One of the AEO scenarios `here <https://www.eia.gov/outlooks/aeo/data/browser/>`_,(UNDER DEVELOPMENT) Scales future demand according to the AEO scenario
+-efs_case,--,"One of {``reference``, ``medium``, ``high``}",(UNDER DEVELOPMENT) Extracts EFS data according to level of adoption
+-efs_speed,--,"One of {``slow``, ``moderate``, ``fast``}",(UNDER DEVELOPMENT) Extracts EFS data according to speed of electrification
+-aeo,--,One of the AEO scenarios `here <https://www.eia.gov/outlooks/aeo/data/browser/>`_,(UNDER DEVELOPMENT) Scales future demand according to the AEO scenario
+,,,
+demand_response:,,,Settings to activate and configure demand response
+-shift,per_unit,"float {0 <=, >= 1} or 'inf'",Allowable load to be shifted per snapshot. Set to 0 to turn off demand response. Set to 'inf' to not enforce capacity limits.
+-marginal_cost,$/MWh,float {0 <=},Cost to store one unit of energy for one hour
 ,,,
 autarky,,,
 #NAME?,bool,``true`` or ``false``,Require each node to be autarkic by removing all lines and links.

diff --git a/docs/source/configtables/sector_industrial.csv b/docs/source/configtables/sector_industrial.csv
@@ -1,11 +1,12 @@
 ,Unit,Values,Description
 industrial_sector ,,,Options when implementing the industrial sector
 -- brownfield,--,"bool {true, false}",Include brownfield end-use capacity.
--- dynamic_costs,--,"bool {true, false}","Apply sector specific time varrying natural gas, oil, and coal costs. "
+-- dynamic_costs,--,"bool {true, false}","Apply sector specific time varrying natural gas, oil, and coal costs."
 -- technologies,,,Energy conversion technologies to model.
 -- -- gas_furnace,,"bool {true, false}",Include natural gas furnaces
 -- -- coal_furnace,,"bool {true, false}",Include coal boilers
 -- -- heat_pump,,"bool {true, false}",Include heat pumps
 -- demnand_response,,,Demand response options for the service sector.
 -- -- shift,percentage,"float {0 <=, >= 100} or 'inf'",Allowable load to be shifted per snapshot. Set to 0 to turn off demand response. Set to 'inf' to not enforce capacity limits.
 -- -- marginal_cost,$/MWh,float {0 <=},Cost to store one unit of energy for one hour.
+-- -- by_carrier,,"bool {true, false}",Run demand response at a per-carrier level.
diff --git a/docs/source/configtables/sector_service.csv b/docs/source/configtables/sector_service.csv
@@ -28,3 +28,4 @@ service_sector,,,Options when implementing the service sector (Residential and C
 -- demnand_response,,,Demand response options for the service sector.
 -- -- shift,percentage,"float {0 <=, >= 100} or 'inf'",Allowable load to be shifted per snapshot. Set to 0 to turn off demand response. Set to 'inf' to not enforce capacity limits.
 -- -- marginal_cost,$/MWh,float {0 <=},Cost to store one unit of energy for one hour.
+-- -- by_carrier,,"bool {true, false}",Run demand response at a per-carrier level.
diff --git a/docs/source/contributing.md b/docs/source/contributing.md
@@ -72,10 +72,22 @@ copy of the code under your account on the repository service.
 
 ### 3. Install developer dependencies
 
-If you plan on contributing to the respository, please install these packages into your activated mamba environment
+If you plan on contributing to the respository, please install developer dependencies.
+
+#### 3a. Using UV
+
+Run the following command from the activated `pypsa-usa` conda environment.
+
+```console
+uv pip install -r pyproject.toml --extra dev
+```
+
+#### 3b. Using conda/mamba
+
+Run the following command from the activated `pypsa-usa` conda environment.
 
 ```console
-python -m pip install -e ".[dev]"
+conda env update --name pypsa-usa --file workflow/envs/dev.yaml --prune
 ```
 
 ### 4. Install pre-commit hooks: