Use Zenodo curated wind dataset instead of scraping MaStR API

[MaykThewessen]

Suggestion: Use Zenodo curated wind dataset as data source

Current approach

The project scrapes the MaStR API directly using paginated curl requests (pageSize=25000, 7 pages). This has several downsides:

• Very slow — the API is rate-limited and downloads ~95 MB per page at low throughput
• Fragile — parallel downloads can produce corrupt/truncated JSON files
• Raw data — coordinates and metadata may contain errors from the registry

Proposed alternative

Use the curated dataset from Zenodo:

Corrected and supplemented unit data on approved wind turbines in Germany
(Version 2026_02_19)

This dataset:

• Contains 40,455 wind turbines (38,606 onshore + 1,849 offshore)
• Has corrected and validated coordinates and metadata
• Is available as both CSV and GeoJSON (pre-geocoded in WGS84 and EPSG:25832)
• Includes all relevant fields: MaStR ID, power (kW), hub height, rotor diameter, manufacturer, type, installation date, decommission date, state, county, municipality, on/offshore status
• Is a one-time download (~13 MB CSV / ~36 MB GeoJSON) vs. ~660 MB from the API
• Is versioned and citable (DOI)

Column mapping

The Zenodo CSV columns map well to the existing PowerPlant dataclass:

Zenodo CSV	PowerPlant field	Notes
nettonennleistung	power	In kW
datum_inbetriebnahme	install_date	ISO format (no .NET date parsing needed)
datum_endgueltige_stilllegung	removal_date	ISO format
lon_x / lat_y	longitude / latitude	WGS84
wind_an_land_oder_auf_see	off_shore	"Windkraft an Land" / "Windkraft auf See"
bundesland	—	State-level info (new)
landkreis	—	County-level info (new)
nabenhoehe	—	Hub height in meters (new)
rotordurchmesser	—	Rotor diameter in meters (new)
hersteller / typenbezeichnung	—	Manufacturer & turbine type (new)

Impact

This would significantly simplify the data pipeline, improve data quality, and make the project more reproducible.

[MaykThewessen]

Implementation update

The approach proposed above has been implemented and tested. Here is what was built:

Data pipeline

The Zenodo CSV loads ~10× faster than the MaStR API scrape and requires no JSON parsing.

Column mapping from Zenodo CSV → notebook fields:

Zenodo	Used as	Notes
nettonennleistung	power	kW — same unit as MaStR Bruttoleistung
datum_inbetriebnahme	install_date	ISO date, no .NET timestamp parsing needed
datum_endgueltige_stilllegung	removal_date	ISO date
lon_x / lat_y	longitude / latitude	WGS84 ready
bundesland	off_shore	"Nordsee …" → "Nordsee", "Ostsee …" → "Ostsee", else None

Map boundaries

germany_kreise.gpkg is now built from Eurostat GISCO NUTS boundaries instead of requiring a full OSM planet file + osmfilter:

# build_kreise_gpkg.py
NUTS3_URL = "https://gisco-services.ec.europa.eu/distribution/v2/nuts/geojson/NUTS_RG_01M_2021_4326_LEVL_3.geojson"
NUTS1_URL = "https://gisco-services.ec.europa.eu/distribution/v2/nuts/geojson/NUTS_RG_01M_2021_4326_LEVL_1.geojson"

• NUTS-3 → admin_level=6 (Kreise)
• NUTS-1 Berlin/Hamburg/Bremen → admin_level=4 (city-states)
• 400 units total, ~2 MB download vs ~4 GB OSM planet file

Results verified

Test run for 2025-01-01:

• 30,321 active turbines → 72.74 GW total
• Top counties: Nordfriesland 2.46 GW, Dithmarschen 2.34 GW, Uckermark 1.64 GW

Capacity export

New script export_capacity_per_region.py produces:

DE_wind_installed_capacity_per_region_MaStR_Zenodo_upto_2026-02-19.csv

315 regions (311 Landkreise + 4 offshore sea areas), 78.36 GW total installed (active turbines only as of 2026-02-19).

[MaykThewessen]

Update: incremental MaStR API fetch + historical capacity snapshots

Since the initial proposal, two new scripts have been added and validated.

---

fetch_mastr.py — fast incremental API fetch

Rather than downloading all ~2 pages of wind data from scratch, this script sorts the API descending by InbetriebnahmeDatum and stops as soon as entries pre-date the Zenodo cutoff. For the current gap (2026-02-19 → now), that's 27 new turbines in ~1 page / 5 seconds.

# Fetch only turbines commissioned after the Zenodo snapshot date
python fetch_mastr.py --energy wind --mode incremental --since 2026-02-19

# Full download (wind ~2 pages; solar warns — ~237 pages)
python fetch_mastr.py --energy wind --mode full
python fetch_mastr.py --energy solar --mode incremental --since 2026-03-01

Supports both wind (Energieträger code 2497) and solar (code 2495).
Output: non-pv-data/mastr-api/mastr_wind_incremental_YYYY-MM-DD.json

---

export_capacity_per_region.py — merged + historical CSV

Merges the Zenodo base with any incremental API files and exports installed capacity per Kreis with historical snapshot columns:

column	description
capacity_MW_2013	end-of-year 2013 (installed, not yet decommissioned)
capacity_MW_2024	end-of-year 2024
capacity_MW_2025	end-of-year 2025
capacity_MW_latest	all currently active turbines

Validated output (313 regions, no duplicates, no null regions):

snapshot	total
end 2013	28.84 GW
end 2024	72.00 GW
end 2025	77.82 GW
latest	78.55 GW

Bugs fixed during development:

• Duplicate region rows: API entries with bundesland=None caused separate rows for the same Kreis. Fixed by filling bundesland from a Zenodo landkreis→bundesland lookup after spatial join.
• Unclassified offshore: 7 API offshore turbines had no bundesland. Fixed with coordinate bounding-box fallback (Nordsee: lon 3–9, lat 53–56; Ostsee: lon 9–15, lat 53–56).
• Timezone mismatch: Zenodo dates are tz-naive; API .NET timestamps parsed as UTC. Fixed with .tz_localize(None).

PR: #3