Skip to content

Commit d7a41c7

Browse files
obielinfranchaise
obielin
and
franchaise
committed
feat(training): real-dataset training pipeline + Colab handoff
Co-authored-by: Francis Umo <franchaise@users.noreply.github.com>
1 parent d050b30 commit d7a41c7

4 files changed

Lines changed: 562 additions & 0 deletions

File tree

‎notebooks/TRAINING_HANDOFF.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# ClimateVision — Training Handoff (run this now)
2+
3+
**Goal:** produce real trained models for ClimateVision so the live site
4+
(climatevision.green) stops serving demo/untrained output. Do **flooding first**
5+
(best dataset fit), then deforestation, then ice.
6+
7+
**You need:** Google Colab (Pro recommended — GPU + longer runtime) and a Google
8+
account with Earth Engine access. You do **not** need any private key file — use
9+
interactive auth (step 2). Est. time: ~30 min setup + 1–3 h GPU training per model.
10+
11+
When done, you hand back **one file per model** (`best_model.pth` and/or
12+
`model.onnx`) — see step 7.
13+
14+
---
15+
16+
## 1. Set up the environment (Colab cell)
17+
18+
```python
19+
!git clone https://github.com/Climate-Vision/ClimateVision.git
20+
%cd ClimateVision
21+
!pip install -q -r requirements.txt
22+
!pip install -q -e .
23+
import torch; print("CUDA available:", torch.cuda.is_available()) # must be True
24+
25+
from google.colab import drive # so checkpoints survive a runtime reset
26+
drive.mount('/content/drive')
27+
!mkdir -p /content/drive/MyDrive/climatevision/models
28+
```
29+
30+
## 2. Earth Engine auth — interactive, NO key file
31+
32+
```python
33+
import ee
34+
ee.Authenticate() # opens an OAuth link, paste the token
35+
ee.Initialize(project='YOUR_GEE_PROJECT_ID')
36+
```
37+
38+
> GEE is only needed if you pull fresh tiles for spot-checking. Training itself
39+
> uses the downloaded dataset, so you can skip this if you just want to train.
40+
41+
## 3. Download a dataset
42+
43+
```bash
44+
# flooding (Sen1Floods11) needs gsutil:
45+
!pip install -q gsutil
46+
!python scripts/download_datasets.py --dataset flooding --out data/datasets
47+
```
48+
49+
| Type | Dataset | Tool | License |
50+
|------|---------|------|---------|
51+
| flooding | Sen1Floods11 | `gsutil` | CC-BY 4.0 |
52+
| deforestation | MultiEarth Amazon | `huggingface_hub` | research use |
53+
| ice_melting | AI4Arctic v2 | `eotdl` | CC-BY 4.0 |
54+
55+
## 4. Convert to the training layout ⚠️ the one manual step
56+
57+
The trainer reads stem-matched GeoTIFF pairs in this structure:
58+
59+
```
60+
data/datasets/<type>/
61+
train/ images/ *.tif masks/ *.tif
62+
val/ images/ *.tif masks/ *.tif
63+
test/ images/ *.tif masks/ *.tif
64+
```
65+
66+
Each dataset ships in its own format, so write a short conversion that:
67+
1. reads each scene + its label,
68+
2. writes the image bands ClimateVision expects for that type
69+
(flooding = B03,B08,B11 → 3 bands; deforestation = B04,B03,B02,B08 → 4 bands),
70+
3. writes the mask as a single-band `uint8` with class indices matching
71+
`config.yaml` (flooding: 0=dry_land, 1=permanent_water, 2=flooded),
72+
4. uses the **same filename stem** for image and mask,
73+
5. splits ~80/10/10 into train/val/test.
74+
75+
Notes per dataset:
76+
- **Sen1Floods11**: GeoTIFF chips already; map its water/flood labels to the
77+
3-class scheme above. SAR bands — run them through
78+
`src/climatevision/data/sar_preprocessing.py` (speckle filter + dB scaling).
79+
- **MultiEarth**: Sentinel-2 chips with binary deforestation masks (0/1) — fits
80+
the 4-band / 2-class deforestation config directly.
81+
- **AI4Arctic**: netCDF — extract the SAR layer + ice chart, tile to 256×256.
82+
83+
## 5. Train
84+
85+
```bash
86+
!python scripts/train_real.py \
87+
--analysis-type flooding \
88+
--data-dir data/datasets/flooding \
89+
--epochs 50 --batch-size 8 \
90+
--out /content/drive/MyDrive/climatevision/models
91+
```
92+
93+
`train_real.py` reads channels/classes from `config.yaml` automatically, uses the
94+
existing `Trainer` (AdamW, warmup→cosine LR, EMA, mixed precision, early stopping)
95+
and saves `best_model.pth` (with `model_state_dict` + `val_iou`) to the run dir.
96+
97+
## 6. Evaluate and gate
98+
99+
```bash
100+
!python scripts/evaluate.py --checkpoint <run>/best_model.pth --data-dir data/datasets/flooding
101+
!python scripts/generate_model_card.py --checkpoint <run>/best_model.pth
102+
!python scripts/governance_ci_gate.py # must pass before promoting a model
103+
```
104+
105+
Only promote a model that clears the governance gate (metrics + fairness). That
106+
is the line between "preview" and something an agency can act on.
107+
108+
## 7. Export ONNX + hand back
109+
110+
```bash
111+
!python scripts/export_model.py --checkpoint <run>/best_model.pth
112+
# produces <run>/model.onnx (+ model_quantized.onnx, export_info.json)
113+
```
114+
115+
Hand back to the project owner, per analysis type, either:
116+
- `best_model.pth` (PyTorch), or
117+
- `model.onnx` (the API serves this automatically via onnxruntime).
118+
119+
Deployment (owner does this): drop the file into `models/<type>_<date>/` and
120+
rebuild the Docker image (the Dockerfile already `COPY models/`), **or** place it
121+
on the Fly volume at `/app/outputs` and point `config.yaml` `weights:` at it.
122+
The serving code (`inference/pipeline.py`) finds `.pth` first, else `model.onnx`.
123+
124+
---
125+
126+
## Quick reference — how serving picks up your model
127+
128+
`inference/pipeline.py` resolution order per type:
129+
1. `config.yaml` `weights:` path (PyTorch `.pth`)
130+
2. `models/best_model.pth`
131+
3. newest `models/*/best_model.pth`
132+
4. `config.yaml` `onnx_weights:` or newest `models/<type>_*/model.onnx` (ONNX)
133+
5. otherwise → untrained demo weights (what's live now)
134+
135+
So just getting a trained file into `models/<type>_*/` is enough to go live with
136+
a real model.

‎notebooks/TRAINING_PIPELINE.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# ClimateVision — Real-Model Training Loop (Colab)
2+
3+
This is the end-to-end loop that turns ClimateVision from a demo (untrained /
4+
synthetic models) into a platform with production models that agencies, NGOs,
5+
and government bodies can rely on. It runs on **Google Colab** (free or Pro;
6+
Pro recommended for the GPU and longer runtimes).
7+
8+
The loop has five stages, repeated per analysis type
9+
(`flooding`, `deforestation`, `ice_melting`):
10+
11+
```
12+
1. Download data -> 2. Train -> 3. Evaluate -> 4. Export ONNX -> 5. Deploy weights
13+
^ |
14+
|________________________ retrain when metrics drift ___________________|
15+
```
16+
17+
---
18+
19+
## 0. One-time Colab setup
20+
21+
```python
22+
!git clone https://github.com/Climate-Vision/ClimateVision.git
23+
%cd ClimateVision
24+
!pip install -q -r requirements.txt
25+
!pip install -e .
26+
# Confirm GPU
27+
import torch; print("CUDA:", torch.cuda.is_available())
28+
```
29+
30+
Mount Google Drive so checkpoints survive a runtime reset:
31+
32+
```python
33+
from google.colab import drive
34+
drive.mount('/content/drive')
35+
```
36+
37+
GEE auth (for pulling fresh tiles during evaluation) — **never paste your
38+
service-account key into the notebook**. Use interactive auth in Colab:
39+
40+
```python
41+
import ee
42+
ee.Authenticate() # opens an OAuth flow
43+
ee.Initialize(project='YOUR_GEE_PROJECT_ID')
44+
```
45+
46+
> Note: `scripts/run_training.py` currently hardcodes `ee.Initialize(project='kinos-473422')`
47+
> and trains on `SyntheticForestDataset`. The steps below replace that synthetic
48+
> path with real data. Rotate any key that has been shared in plaintext.
49+
50+
---
51+
52+
## 1. Download a real dataset
53+
54+
```bash
55+
!python scripts/download_datasets.py --dataset flooding --out data/datasets
56+
```
57+
58+
| Analysis type | Dataset | License | Tool needed |
59+
|---------------|---------|---------|-------------|
60+
| `flooding` | Sen1Floods11 (Sentinel-1 SAR) | CC-BY 4.0 | `gsutil` |
61+
| `deforestation` | MultiEarth Amazon (Sentinel-2) | research use | `huggingface_hub` |
62+
| `ice_melting` | AI4Arctic Sea Ice v2 | CC-BY 4.0 | `eotdl` |
63+
64+
Each download writes a `provenance.json` so `governance/datasheet.py` can record
65+
the source and license — keep this; it is part of what makes the platform
66+
credible for government use.
67+
68+
---
69+
70+
## 2. Train
71+
72+
Train the U-Net for the chosen type. Match `in_channels` / `num_classes` to the
73+
entry in `config.yaml` (e.g. flooding = 3 channels, 3 classes):
74+
75+
```bash
76+
!python scripts/run_training.py \
77+
--analysis-type flooding \
78+
--data-dir data/datasets/flooding \
79+
--epochs 50 \
80+
--batch-size 8 \
81+
--out /content/drive/MyDrive/climatevision/models
82+
```
83+
84+
The trainer should save `best_model.pth` containing at least
85+
`{"model_state_dict", "epoch", "val_iou"}` — this is exactly what
86+
`inference/pipeline.py::_find_best_checkpoint` looks for.
87+
88+
> If `run_training.py` does not yet accept `--analysis-type/--data-dir`, use the
89+
> `training/trainer.py` API directly (see `notebooks/04_model_validation.ipynb`)
90+
> with a `Dataset` that reads the downloaded chips instead of synthetic ones.
91+
92+
---
93+
94+
## 3. Evaluate
95+
96+
```bash
97+
!python scripts/evaluate.py \
98+
--checkpoint /content/drive/MyDrive/climatevision/models/<run>/best_model.pth \
99+
--data-dir data/datasets/flooding
100+
```
101+
102+
Record IoU / F1 in the model card and gate on them:
103+
104+
```bash
105+
!python scripts/generate_model_card.py --checkpoint <run>/best_model.pth
106+
!python scripts/governance_ci_gate.py # enforces metric + fairness thresholds
107+
```
108+
109+
Only promote a model that passes the governance gate. This is the line between a
110+
"preview" and something an NGO can act on.
111+
112+
---
113+
114+
## 4. Export to ONNX
115+
116+
```bash
117+
!python scripts/export_model.py --checkpoint <run>/best_model.pth
118+
# -> <run>/model.onnx (+ model_quantized.onnx, export_info.json)
119+
```
120+
121+
The serving layer now picks this up automatically: `inference/pipeline.py`
122+
loads `models/<analysis_type>_*/model.onnx` via `onnxruntime` when no `.pth` is
123+
present, or you can set `onnx_weights:` per type in `config.yaml`.
124+
125+
---
126+
127+
## 5. Deploy the weights
128+
129+
Weights are **not** committed (`.gitignore` excludes `models/*.pth`). Ship them
130+
one of two ways:
131+
132+
- **Bake into the image** — copy the trained `best_model.pth` (or `model.onnx`)
133+
into `models/` before `docker build`; the Dockerfile already does `COPY models/`.
134+
- **Mount on the Fly volume** — `fly.toml` mounts `climatevision_data` at
135+
`/app/outputs`; place weights there and point `config.yaml` `weights:` at them.
136+
137+
Then redeploy (`./scripts/deploy.sh` for Fly, or push for the Render blueprint).
138+
139+
---
140+
141+
## Retraining cadence
142+
143+
Re-run stages 1-5 when: new labeled data is released, the governance gate flags
144+
metric/fairness drift, or the scheduled freshness check (see the Cowork
145+
"ClimateVision model freshness" task) reports models older than the threshold.
146+
Keep each run's `provenance.json`, model card, and metrics — that audit trail is
147+
the asset, not just the weights.

0 commit comments

Comments
 (0)