Skip to content

Commit 5f49781

Browse files
authored
Merge pull request #9 from modelblocks-org/feature-modelblocks
Move to modelblocks
2 parents 2de338a + e78fe22 commit 5f49781

29 files changed

Lines changed: 7705 additions & 455 deletions

.copier-answers.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
# Changes here will be overwritten by Copier
2-
# DO NOT MANUALLY MODIFY THIS FILE
2+
# !!!!! DO NOT MANUALLY MODIFY THIS FILE !!!!!
33
_commit: latest
4-
_src_path: https://github.com/calliope-project/data-module-template.git
4+
_src_path: https://github.com/modelblocks-org/data-module-template.git
55
author_email: i.ruizmanuel@tudelft.nl
66
author_family_name: Ruiz Manuel
77
author_given_name: Ivan
8-
github_org: calliope-project
8+
github_org: modelblocks-org
99
license: Apache-2.0
10-
module_description: A module to aggregate CDR potential from the CO2Stop dataset into
11-
flexible spatial resolutions
12-
module_long_name: Module CO2Stop - Carbon Dioxide Removal
10+
module_description: A module to aggregate CDR potential in Europe into flexible spatial
11+
resolutions, using the CO2Stop dataset
12+
module_long_name: Module CO2Stop - Carbon Dioxide Removal in Europe
1313
module_short_name: module_co2stop_cdr

.github/ISSUE_TEMPLATE/bug_report.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ A clear and concise description of what you expected to happen.
2424
If applicable, add screenshots to help explain your problem.
2525

2626
**Desktop (please complete the following information):**
27-
- OS: [e.g. Linux Fedora 41, Windows 10...]
27+
- OS: [e.g. Linux Fedora 43, Windows 11...]
2828
- Version [e.g. v0.1.1]
2929

3030
**Additional context**

.github/ISSUE_TEMPLATE/config.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
blank_issues_enabled: false
2+
contact_links:
3+
- name: General information
4+
url: https://www.modelblocks.org/
5+
about: Please consult our website for general information on the Modelblocks methodology.

.github/pull_request_template.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,11 @@ Fixes #
88

99
## Reviewer checklist
1010

11-
* [ ] `INTERFACE.yaml` is up-to-date with all relevant user resources and results.
12-
* [ ] The integration example is up-to-date with a minimal use-case of the module.
13-
* [ ] Module documentation is up-to-date.
11+
* [ ] There are no `pip` dependencies in the module's environment files (`workflow/envs/`).
12+
* [ ] All rules use `pathvars` (e.g., `<results>`) in their inputs and outputs.
13+
* [ ] The integration test-suite is successful, including:
14+
* [ ] `pre-commit.ci` tests pass.
15+
* [ ] tests pass for all relevant OS configurations (linux, osx, windows).
16+
* [ ] Module documentation is up-to-date, including:
17+
* [ ] `INTERFACE.yaml` mentions all relevant `pathvars` and `wildcards`.
18+
* [ ] `README.md` describes how to use the module and has the necessary citations.

.github/workflows/check-version.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,4 @@ on:
66

77
jobs:
88
copier-update:
9-
uses: calliope-project/data-module-template/.github/workflows/template-check-version.yml@latest
9+
uses: modelblocks-org/data-module-template/.github/workflows/template-check-version.yml@latest

.github/workflows/pr-ci.yml

Lines changed: 36 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,41 @@
1-
name: CI
1+
# !!!!! DO NOT MODIFY UNLESS ABSOLUTELY NECESSARY !!!!!
2+
# This workflow helps standardise integration tests across data modules
3+
name: Pull Request CI tests for Modelblocks data modules
24
on:
35
pull_request:
46
branches:
57
- "main"
6-
78
jobs:
8-
ci-tests:
9-
uses: calliope-project/data-module-template/.github/workflows/template-pr-ci.yml@latest
9+
build:
10+
name: Build
11+
runs-on: ${{ matrix.os }}
12+
strategy:
13+
fail-fast: false
14+
matrix:
15+
os: [ubuntu-latest, windows-latest, macos-latest]
16+
python-version: ["3.12"]
17+
steps:
18+
- uses: actions/checkout@v4
19+
- name: Setup pixi
20+
uses: prefix-dev/setup-pixi@v0.8.3
21+
- name: Run integration tests
22+
id: tests
23+
run: pixi run test-integration
24+
continue-on-error: true
25+
- name: Run snakemake linting
26+
id: linting
27+
run: |
28+
pixi run snakemake --lint --snakefile workflow/Snakefile
29+
pixi run snakemake --lint --snakefile tests/integration/Snakefile
30+
continue-on-error: true
31+
- name: Save integration logs
32+
if: ${{ always() }}
33+
uses: actions/upload-artifact@v4
34+
with:
35+
name: integration-test-logs-${{ matrix.os }}
36+
path: tests/integration/results/integration_test/logs
37+
if-no-files-found: ignore
38+
retention-days: 30
39+
- name: Fail if integration or linting failed
40+
if: ${{ steps.tests.outcome == 'failure' || steps.linting.outcome == 'failure' }}
41+
run: exit 1

.github/workflows/release.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,4 @@ on:
66

77
jobs:
88
release-workflow:
9-
uses: calliope-project/data-module-template/.github/workflows/template-release.yml@latest
9+
uses: modelblocks-org/data-module-template/.github/workflows/template-release.yml@latest

.pre-commit-config.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ repos:
1212
# Repo quality
1313
- id: check-added-large-files
1414
args: [--enforce-all]
15+
exclude: ^pixi\.lock$
1516
- id: forbid-submodules
1617
- id: check-case-conflict
1718
- id: check-illegal-windows-names
@@ -46,4 +47,4 @@ repos:
4647
- --ignore-words-list=aproximated
4748

4849
ci: # https://pre-commit.ci/
49-
autoupdate_schedule: monthly
50+
autoupdate_schedule: quarterly

INTERFACE.yaml

Lines changed: 28 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,32 @@
11
# Module Input-Output structure for automated doc. generation
2-
resources:
3-
user:
4-
"{shapes}/shapes.parquet": |
5-
Region geometries in parquet format.
6-
These should conform to the schema defined in https://github.com/calliope-project/module_geo_boundaries/.
7-
results:
8-
"{shapes}/{scenario}/{cdr_group}.parquet": |
9-
parquet data table with aggregated CO2 sequestration potential per shape in MtCO2.
10-
This is the total potential, not an annual rate!
11-
"{shapes}/{scenario}/totals.parquet": |
12-
parquet data table with the sum of all cdr_groups.
13-
2+
pathvars:
3+
snakemake_defaults:
4+
logs:
5+
default: "<logs>"
6+
description: location of rule log files.
7+
resources:
8+
default: "<resources>"
9+
description: "location of module resource files."
10+
results:
11+
default: "<results>"
12+
description: "location of module results."
13+
user_resources:
14+
user_shapes:
15+
default: "<resources>/user/{shapes}/shapes.parquet"
16+
description: |
17+
Region geometries in parquet format.
18+
These should conform to the schema defined in https://github.com/calliope-project/module_geo_boundaries/.
19+
results:
20+
cdr_group:
21+
default: "<results>/{shapes}/{scenario}/{cdr_group}.parquet"
22+
description: |
23+
parquet table with aggregated CO2 sequestration potential for a specific CDR group type and scenario.
24+
Potential is in MtCO2. This is the total potential, not an annual rate!
25+
total_aggregate:
26+
default: "<results>/{shapes}/{scenario}/totals.parquet"
27+
description: |
28+
parquet data table with the sum of all CDR groups in a specific scenario.
29+
If you do not distinguish between CDR facility types in your study, this is likely the file you want.
1430
wildcards:
1531
shapes: name of the provided shapefile case.
1632
scenario: requested CO2Stop CDR scenario. Can be 'low', 'medium', or 'high'.

README.md

Lines changed: 80 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,26 @@
1-
# Module CO2Stop - Carbon Dioxide Removal
1+
# Module CO2Stop - Carbon Dioxide Removal in Europe
22

3-
A module to aggregate CDR potential from the CO2Stop dataset into flexible spatial resolutions
3+
A module to aggregate CDR potential in Europe into flexible spatial resolutions, using the CO2Stop dataset
44

5-
A modular `snakemake` workflow built for [`clio`](https://clio.readthedocs.io/) data modules.
5+
## About this module
6+
<!-- Please do not modify this templated section -->
67

7-
## Using this module
8+
This is a modular `snakemake` workflow built for [Modelblocks](https://www.modelblocks.org/) data modules.
89

910
This module can be imported directly into any `snakemake` workflow.
10-
Please consult the integration example in `tests/integration/Snakefile` for more information.
11+
For more information, please consult:
12+
- The Modelblocks [documentation](https://modelblocks.readthedocs.io/en/latest/).
13+
- The integration example in this repository (`tests/integration/Snakefile`).
14+
- The `snakemake` [documentation on modularisation](https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html).
1115

1216
## Development
17+
<!-- Please do not modify this templated section -->
1318

1419
We use [`pixi`](https://pixi.sh/) as our package manager for development.
15-
Once installed, run the following to clone this repo and install all dependencies.
20+
Once installed, run the following to clone this repository and install all dependencies.
1621

1722
```shell
18-
git clone git@github.com:calliope-project/module_co2stop_cdr.git
23+
git clone git@github.com:modelblocks-org/module_co2stop_cdr.git
1924
cd module_co2stop_cdr
2025
pixi install --all
2126
```
@@ -33,3 +38,71 @@ pixi shell # activate this project's environment
3338
cd tests/integration/ # navigate to the integration example
3439
snakemake --use-conda --cores 2 # run the workflow!
3540
```
41+
42+
## Documentation
43+
44+
### Overview
45+
<!-- Please describe the processing stages of this module here -->
46+
47+
The analysis of the module is structured as follows:
48+
49+
<div style="width:50%; margin: auto;">
50+
51+
![scenarios](./figures/rulegraph.png)
52+
</div>
53+
54+
1. The CO2Stop dataset is downloaded and cleaned up following methods described in the [JRC - CO2 transport report](https://publications.jrc.ec.europa.eu/repository/handle/JRC136709).
55+
2. To avoid double-counting, traps within the remaining storage units are removed as their capacity is already included in the storage unit total (please consult the [CO2Stop Final report](https://energy.ec.europa.eu/publications/assessment-co2-storage-potential-europe-co2stop_en) section 2.3.1 for details).
56+
Additionally, the following removal criteria is applied to further clean the data:
57+
- Cases marked as 'not assessed' or as 'undisclosed' in the dataset.
58+
- Ambiguous duplicates (these are a few small traps located in the north sea with repeated IDs and capacities).
59+
- Optionally, details from the dataset are used to remove the following, if configured:
60+
- Qualitative cases marked as having surface or subsurface issues, and artificially created polygons.
61+
- Quantitative values (e.g., porosity, depth, ...).
62+
3. Three scenarios (`low`, `medium`, `high`) are created for each sequestration type (`aquifer`, `gas`, `oil`) for the remaining CO2Stop data.
63+
User-configured lower and upper bounds are applied per-polygon at this stage.
64+
See `bounds_mtco2: co2stop_polygons` in the configuration schema for more information.
65+
<div style="width:50%; margin: auto;">
66+
67+
![scenarios](./figures/aquifer_scenarios.png)
68+
</div>
69+
4. The resulting sequestration potential is aggregated per scenario into user provided shapes.
70+
71+
>[!WARNING]
72+
>Estimates from the CO2Stop dataset are biased by disclosure (or lack thereof), and the filtering settings used.
73+
>Some countries are affected more than others, with Germany having particularly poor disclosure.
74+
>
75+
>Similarly, CO2Stop suffers from poor data handling practices that make unavailable data and 'true' zero values indistinguishable from each other, amplifying the uneven assignation of sequestration. For example: setting `porosity_ratio: 0.1` will completely remove France in most cases.
76+
>
77+
>We provide automated figures and logging (in `logs/storage_units/` and `logs/traps/`) so users can evaluate how their settings affect polygon selection.
78+
>Below is an example for storage unit aquifers where only undisclosed and artificial polygons have been removed. This can be seen as a _MINIMUM_ amount of removals.
79+
><div style="width:50%; margin: auto;">
80+
>
81+
>![scenarios](./figures/aquifer_kept.png)
82+
></div>
83+
84+
### Configuration
85+
<!-- Feel free to describe how to configure this module below -->
86+
87+
Please consult the configuration [README](./config/README.md) and the [configuration example](./config/config.yaml) for a general overview on the configuration options of this module.
88+
89+
### Input / output structure
90+
<!-- Feel free to describe input / output file placement below -->
91+
92+
As input, all you need to provide is a `.parquet` shapes file with the polygons to aggregate capacities into. This file should follow the schema provided by the [geo-boundaries module](https://github.com/calliope-project/module_geo_boundaries/tree/v0.1.6).
93+
94+
Outputs for each input shape file can be requested per potential scenario (low, medium, high), and CDR group (aquifer, gas, oil, and total aggregate sum).
95+
96+
Please consult the [interface file](./INTERFACE.yaml) for more information.
97+
98+
### References
99+
<!-- Please provide thorough referencing below -->
100+
101+
This module is based on the following research and datasets:
102+
103+
- **CO2Stop dataset**
104+
Poulsen, N., Holloway, S., Neele, F., Smith, N.A., Kirk, K., 2012. CO2StoP Executive Summary (No. ENER/C1/154-2011-SI2.611598). GEOLOGICAL SURVEY OF DENMARK AND GREENLAND. <https://energy.ec.europa.eu/publications/assessment-co2-storage-potential-europe-co2stop_en>.
105+
- **Shape schema definition:**
106+
Ruiz Manuel, I. clio - module_geo_boundaries [Computer software]. <https://github.com/calliope-project/module_geo_boundaries/>.
107+
- **Filtering minimum defaults:**
108+
Van Den Broek, M., Brederode, E., Ramírez, A., Kramers, L., Van Der Kuip, M., Wildenborg, T., Turkenburg, W., Faaij, A., 2010. Designing a cost-effective CO2 storage infrastructure using a GIS based linear optimization energy model. Environmental Modelling & Software 25, 1754–1768. <https://doi.org/10.1016/j.envsoft.2010.06.015>.

0 commit comments

Comments
 (0)