Skip to content

Commit 2b14266

Browse files
irm-codebasepre-commit-ci[bot]
irm-codebase
and
pre-commit-ci[bot]
authored
Add scenario processing (#63)
* Add scenario processing * Add antimeridian fixing (DOES NOT WORK 100% OF THE TIME) --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 4bff228 commit 2b14266

33 files changed

Lines changed: 1252 additions & 845 deletions

‎INTERFACE.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ pathvars:
1212
description: "location of module results."
1313
results:
1414
shapes:
15-
default: "<results>/shapes.parquet"
15+
default: "<results>/{scenario}/shapes.parquet"
1616
description: |
17-
Harmonised geoparquet dataset with all the configured countries, including non-contested exclusive economic zones.
17+
Harmonised geoparquet dataset for a configured scenario, including land and maritime non-contested EEZs.
18+
wildcards:
19+
scenario: Scenario name from the module configuration.

‎README.md‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,21 +26,30 @@ Data processing steps:
2626
</p>
2727

2828

29-
1. The configuration file is read to identify the datasets to use as well as the specific countries and regional aggregation (`subtype` in the configuration) to process.
29+
1. The configuration file is read to identify which requested geopolitical combination (`scenarios`) was requested. These can be any number of nations (`countries`), each coming from a distinct dataset (`source`) and with its own subnational aggregation (`subtype`).
3030
- Country landmass data: [eurostat NUTS](https://ec.europa.eu/eurostat/web/gisco/geodata/statistical-units/territorial-units-statistics), [GADM](https://gadm.org/), [geoBoundaries](https://www.geoboundaries.org/), and [Overture Maps](https://overturemaps.org/) are supported.
3131
- Exclusive Economic Zone (EEZ) data: [MarineRegions.org](https://www.marineregions.org/).
3232
2. Individual country files are downloaded and harmonised to fit a standardised schema.
3333
- If identified, contested regions are removed at this stage.
3434
- Land is clipped using maritime Exclusive Economic Zones (EEZ).
3535
- Optionally, a Voronoi algorithm is run to separate EEZ areas to fit subnational regions.
36-
3. Each country file is combined and then clipped using its neighbours to minimise overlapping polygons.
36+
3. The country files requested in the scenario are combined and then clipped using their neighbours to minimise overlapping polygons.
3737

3838
> [!TIP]
39-
> The `subtype` naming matches that of the source database. For example, NUTS uses 0, 1, 2 and 3 (NUTS0, NUTS1, NUTS2, etc.).
40-
> Use the references at the bottom of this page for more details.
39+
> Keep in mind the following
40+
>
41+
> - All downloaded data is kept locally for future reuse across scenarios to minimise stress on the data sources the module relies on.
42+
> - Data source availability can vary.
43+
> Always consult the data source website to identify if a country is available at the desired resolution.
4144
4245
> [!CAUTION]
43-
> To increase the replicability of your workflow, we recommend using NUTS and geoBoundaries as sources whenever possible as they have more stable hosting methods than Overture Maps and GADM.
46+
> Be aware of the following known issues.
47+
>
48+
>- Overture Maps replicability: this data source [does not retain versions](https://github.com/orgs/OvertureMaps/discussions/422) for long.
49+
> If replicability is important to you, we suggest configuring other sources.
50+
>- Anti-meridian distortions: regions near the 180/-180 line (e.g., Fiji, Hawaii, New Zealand, Alaska) might be distorted during processing.
51+
> This is a known issue in the libraries we rely on (`geopandas`, `GDAL`) and something that is being actively [worked on at the moment](https://geopandas.org/en/v1.1.3/about/roadmap.html#s2-geometry-engine).
52+
> For now, we advice to use global projections (EPSG:3857, EPSG:8857) as a way to mitigate it.
4453
4554
## Configuration
4655
<!-- Please describe how to configure this module below -->
@@ -50,7 +59,7 @@ Please consult the configuration [README](./config/README.md) and the [configura
5059
## Input / output structure
5160
<!-- Please describe input / output file placement below -->
5261

53-
This module only has one output: a geoparquet file with your requested geo-boundary "shapes".
62+
This module only has one output: a geoparquet file with your requested geo-boundary "shapes" for each of the the configured `scenarios`.
5463

5564
Please consult the [interface file](./INTERFACE.yaml) for more information.
5665

‎config/china_example.yaml‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎config/config.yaml‎

Lines changed: 208 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,215 @@
11
# A minimal example of how to configure this module
2+
3+
# Default global settings across scenarios
24
voronoi_eez:
35
enabled: True
46
sample_spacing: 10000 # sample every 10 km
57
crs:
68
projected: "epsg:3857"
79
geographic: "epsg:4326"
8-
countries:
9-
DNK:
10-
subtype: "1"
11-
source: "nuts"
12-
resolution: 01M
13-
year: 2024
14-
GBR:
15-
subtype: "country"
16-
source: "overture"
17-
NLD:
18-
subtype: 0
19-
source: "geoboundaries"
20-
release_type: "gbOpen"
21-
BEL:
22-
subtype: 1
23-
source: "geoboundaries"
24-
release_type: "gbOpen"
25-
CHE:
26-
subtype: "country"
27-
source: "overture"
28-
ESP:
29-
subtype: "2"
30-
source: "nuts"
31-
resolution: 01M
32-
year: 2024
33-
extra_eez: 8364
10+
11+
# At least one scenario.
12+
# We include four cases as examples.
13+
scenarios:
14+
mixed_example:
15+
# Assorted countries
16+
countries:
17+
DNK:
18+
subtype: "1"
19+
source: "nuts"
20+
resolution: 01M
21+
year: 2024
22+
GBR:
23+
subtype: "country"
24+
source: "overture"
25+
NLD:
26+
subtype: 0
27+
source: "geoboundaries"
28+
release_type: "gbOpen"
29+
BEL:
30+
subtype: 1
31+
source: "geoboundaries"
32+
release_type: "gbOpen"
33+
CHE:
34+
subtype: "country"
35+
source: "overture"
36+
ESP:
37+
subtype: "2"
38+
source: "nuts"
39+
resolution: 01M
40+
year: 2024
41+
extra_eez: 8364
42+
china_national:
43+
# Large country at national level
44+
# China has contested borders, which will be removed by the module
45+
voronoi_eez:
46+
enabled: False
47+
sample_spacing: 10000
48+
countries:
49+
"CHN":
50+
subtype: "country"
51+
source: "overture"
52+
USA_states:
53+
# Example of regional disaggregation of a large country crossing the antimeridean.
54+
# USA has multiple marine zones, which can be appended as extras
55+
countries:
56+
"USA":
57+
subtype: "1"
58+
source: "gadm"
59+
extra_eez: [8463, 8453]
60+
europe_regions:
61+
# A large continental example.
62+
# Europe at regional resolution.
63+
crs:
64+
projected: "epsg:3035"
65+
geographic: "epsg:4326"
66+
countries:
67+
"ALB":
68+
subtype: "1"
69+
source: "geoboundaries"
70+
release_type: "gbOpen"
71+
"AUT":
72+
subtype: "1"
73+
source: "geoboundaries"
74+
release_type: "gbOpen"
75+
"BEL":
76+
subtype: "1"
77+
source: "geoboundaries"
78+
release_type: "gbOpen"
79+
"BGR":
80+
subtype: "1"
81+
source: "geoboundaries"
82+
release_type: "gbOpen"
83+
"BIH":
84+
subtype: "1"
85+
source: "geoboundaries"
86+
release_type: "gbOpen"
87+
"CHE":
88+
subtype: "1"
89+
source: "geoboundaries"
90+
release_type: "gbOpen"
91+
"CYP":
92+
subtype: "1"
93+
source: "geoboundaries"
94+
release_type: "gbOpen"
95+
"CZE":
96+
subtype: "1"
97+
source: "geoboundaries"
98+
release_type: "gbOpen"
99+
"DEU":
100+
subtype: "1"
101+
source: "geoboundaries"
102+
release_type: "gbOpen"
103+
"DNK":
104+
subtype: "1"
105+
source: "nuts"
106+
resolution: 01M
107+
year: 2024
108+
"ESP":
109+
subtype: "1"
110+
source: "geoboundaries"
111+
release_type: "gbOpen"
112+
extra_eez: 8364
113+
"EST":
114+
subtype: "1"
115+
source: "geoboundaries"
116+
release_type: "gbOpen"
117+
"FIN":
118+
subtype: "1"
119+
source: "geoboundaries"
120+
release_type: "gbOpen"
121+
"FRA":
122+
subtype: "1"
123+
source: "geoboundaries"
124+
release_type: "gbOpen"
125+
"GBR":
126+
subtype: "region"
127+
source: "overture"
128+
"GGY":
129+
subtype: "0"
130+
source: "geoboundaries"
131+
release_type: "gbOpen"
132+
"GRC":
133+
subtype: "1"
134+
source: "geoboundaries"
135+
release_type: "gbOpen"
136+
"HRV":
137+
subtype: "1"
138+
source: "geoboundaries"
139+
release_type: "gbOpen"
140+
"HUN":
141+
subtype: "1"
142+
source: "geoboundaries"
143+
release_type: "gbOpen"
144+
"IMN":
145+
subtype: "0"
146+
source: "geoboundaries"
147+
release_type: "gbOpen"
148+
"IRL":
149+
subtype: "region"
150+
source: "overture"
151+
"ITA":
152+
subtype: "1"
153+
source: "geoboundaries"
154+
release_type: "gbOpen"
155+
"LTU":
156+
subtype: "1"
157+
source: "geoboundaries"
158+
release_type: "gbOpen"
159+
"LUX":
160+
subtype: "1"
161+
source: "geoboundaries"
162+
release_type: "gbOpen"
163+
"LVA":
164+
subtype: "1"
165+
source: "geoboundaries"
166+
release_type: "gbOpen"
167+
"MKD":
168+
subtype: "1"
169+
source: "geoboundaries"
170+
release_type: "gbOpen"
171+
"MNE":
172+
subtype: "1"
173+
source: "geoboundaries"
174+
release_type: "gbOpen"
175+
"NLD":
176+
subtype: "region"
177+
source: "overture"
178+
"NOR":
179+
subtype: "1"
180+
source: "geoboundaries"
181+
release_type: "gbOpen"
182+
"POL":
183+
subtype: "1"
184+
source: "geoboundaries"
185+
release_type: "gbOpen"
186+
"PRT":
187+
subtype: "1"
188+
source: "nuts"
189+
resolution: 01M
190+
year: 2024
191+
extra_eez: [8361, 8363]
192+
"ROU":
193+
subtype: "1"
194+
source: "geoboundaries"
195+
release_type: "gbOpen"
196+
"SRB":
197+
subtype: "1"
198+
source: "geoboundaries"
199+
release_type: "gbOpen"
200+
"SVK":
201+
subtype: "1"
202+
source: "geoboundaries"
203+
release_type: "gbOpen"
204+
"SVN":
205+
subtype: "1"
206+
source: "geoboundaries"
207+
release_type: "gbOpen"
208+
"SWE":
209+
subtype: "1"
210+
source: "geoboundaries"
211+
release_type: "gbOpen"
212+
"XKX":
213+
subtype: "1"
214+
source: "geoboundaries"
215+
release_type: "gbOpen"

0 commit comments

Comments
 (0)