Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
293 commits
Select commit Hold shift + click to select a range
36bb69e
update websirenes_defesa_civil_path
NietoCurcio Aug 24, 2024
0a54eb4
improve assertion
NietoCurcio Aug 24, 2024
3796e4a
add era5land_path
NietoCurcio Aug 24, 2024
3dc8fe6
improve square size assertion
NietoCurcio Aug 24, 2024
95c115e
add websirenes_coords_path
NietoCurcio Aug 24, 2024
92dc545
add check_data_requirements
NietoCurcio Aug 24, 2024
812a710
add data instructions
NietoCurcio Aug 24, 2024
77e3982
fix check_data_requirements
NietoCurcio Aug 24, 2024
7103cfc
improve optional date params
NietoCurcio Aug 24, 2024
e531651
add ProcessPoolExecutor
NietoCurcio Aug 24, 2024
8939595
improve process timestamps logging
NietoCurcio Aug 25, 2024
bdc017c
create websirenes features
NietoCurcio Aug 25, 2024
561e842
add ERA5Land in README
NietoCurcio Aug 27, 2024
dbd7f7d
improve target and features folders docs
NietoCurcio Aug 27, 2024
8ec690a
add WebsirenesDataset to get output_dataset.nc
NietoCurcio Sep 8, 2024
afaf299
update README
NietoCurcio Sep 8, 2024
128bac8
update WebSirenesKeys cache
NietoCurcio Sep 8, 2024
4d0045d
update cache + use tqdm in build netcdf
NietoCurcio Sep 8, 2024
e5614cb
add _validate_timestamps
NietoCurcio Sep 8, 2024
9db1d8b
update _validate_timestamps
NietoCurcio Sep 8, 2024
136374d
fix HDF error
NietoCurcio Sep 8, 2024
d5b3fa8
add timestep past to future not future to past
NietoCurcio Sep 8, 2024
5f9ac3e
improve code + setup t2m experiment
NietoCurcio Sep 10, 2024
85619f5
Script download CMI goes16
vkskayo Sep 30, 2024
7ca3506
Merge pull request #38 from vkskayo/main
edubezerra Sep 30, 2024
2f5acca
add flow animation example
NietoCurcio Oct 7, 2024
7f8c6e3
Merge remote-tracking branch 'atmoseer/main' into flow-animation
NietoCurcio Oct 7, 2024
bdb4303
Merge pull request #40 from NietoCurcio/flow-animation
edubezerra Oct 7, 2024
6150a10
Create folder notebooks/GOES16
edubezerra Oct 7, 2024
cb25fe7
Update retrieve_goes16_cmi_for_extent.py
edubezerra Oct 14, 2024
3b150f9
Add goes16_multithreaded_downloader.py
edubezerra Oct 15, 2024
90d9ac7
Add ignore_months command option
edubezerra Oct 15, 2024
3e2b032
Update goes16_multithreaded_downloader.py
edubezerra Oct 15, 2024
3afef98
Update goes16_cropper.py
edubezerra Oct 16, 2024
7b89620
Update goes16_multithreaded_downloader.py
edubezerra Oct 16, 2024
430e66c
Update goes16_multithreaded_downloader.py
edubezerra Oct 18, 2024
b4bc8a2
Update goes16_sync_downloader_cropper.py
edubezerra Oct 19, 2024
bbae16c
Remove png files fom repo
edubezerra Oct 20, 2024
6cb99be
Rename GOES16 plot scripts.
edubezerra Oct 20, 2024
54cb07a
Fix infinite loop
vkskayo Oct 21, 2024
a82ed7e
Merge pull request #42 from vkskayo/main
edubezerra Oct 21, 2024
79a9f29
Add channel number to crop file name.
edubezerra Oct 21, 2024
183efb6
Create generate_samples_to_stconvs2s.py
RaulMartino Oct 25, 2024
e25f737
Update cmd line args definition
edubezerra Oct 27, 2024
66f12ae
Atualizando retrieve_glm
houkaita Oct 29, 2024
8ec19e1
Update GLM retrieval code.
edubezerra Oct 30, 2024
d1f7e17
Merge branch 'AILAB-CEFET-RJ:main' into main
RaulMartino Oct 30, 2024
f06ff88
Create generate_samples_to_stconvs2s_updated.py
RaulMartino Oct 31, 2024
b260cbc
update for 19 features - tp + (t, rh, u, v, spd, w at 200, 700 and 10…
NietoCurcio Nov 5, 2024
71fe556
update validation + improve code
NietoCurcio Nov 5, 2024
f928d79
remove repeated code
NietoCurcio Nov 5, 2024
83fd844
update validate_timestamps
NietoCurcio Nov 5, 2024
8f1a0d3
update task submission logs
NietoCurcio Nov 6, 2024
13ecd63
remove src/spatiotemporal_builder/notebooks/
NietoCurcio Nov 6, 2024
fa6cea3
Merge history from branch 'NietoCurcio-NWP' into branch main
NietoCurcio Nov 6, 2024
6f8db7d
gitignore spatiotemporal_builder/.github
NietoCurcio Nov 6, 2024
290da91
Merge pull request #45 from NietoCurcio/features-pressure-levels
edubezerra Nov 7, 2024
5126bbc
Merge pull request #34 from JenniferRamosdeCarvalho/patch-1
edubezerra Nov 7, 2024
8dbfb3d
Merge pull request #35 from JenniferRamosdeCarvalho/patch-2
edubezerra Nov 7, 2024
7efe6a3
Merge pull request #36 from JenniferRamosdeCarvalho/patch-3
edubezerra Nov 7, 2024
6c98be7
Changing the descriptions to english
houkaita Nov 8, 2024
43b94df
Merge branch 'main' of https://github.com/houkaita/atmoseer
houkaita Nov 8, 2024
c89d945
Merge pull request #46 from houkaita/main
edubezerra Nov 8, 2024
1c3e1c1
Adding aggregation by day
houkaita Nov 12, 2024
7b688af
Merge branch 'AILAB-CEFET-RJ:main' into main
houkaita Nov 12, 2024
c4f4cde
translating to english
houkaita Nov 12, 2024
29e875f
Merge pull request #48 from houkaita/main
edubezerra Nov 12, 2024
c587fce
Add ignored_months command option
edubezerra Nov 12, 2024
0f7c388
Add goes16_sync_downloader_cropper_for_glm.py
edubezerra Nov 16, 2024
0ad9fe6
improve code validations + dask parallel
NietoCurcio Nov 17, 2024
a8a482a
goes16 features
vkskayo Nov 18, 2024
aea32a7
update cells with station logic
NietoCurcio Nov 19, 2024
03e9633
Update generate_samples_to_stconvs2s_updated.py
RaulMartino Nov 22, 2024
4ecc3e8
Merge pull request #49 from vkskayo/main
edubezerra Nov 23, 2024
a3fccbb
Merge pull request #43 from RaulMartino/main
edubezerra Nov 23, 2024
e18c1c2
Fix bug
edubezerra Nov 25, 2024
12ef4dd
websirenes + INMET
NietoCurcio Nov 25, 2024
c7d93b3
fix read_station_id
NietoCurcio Nov 25, 2024
dfd07e4
remove keys interface
NietoCurcio Nov 25, 2024
607b271
improve lock - stations set shared variable
NietoCurcio Nov 26, 2024
ef3d704
update lock code
NietoCurcio Nov 26, 2024
e624307
Revert "update lock code"
NietoCurcio Nov 26, 2024
ce26b16
use concurrent.futures
NietoCurcio Nov 26, 2024
bae22b2
Update path to CMI data.
edubezerra Nov 26, 2024
b0cd8c3
Add main function
edubezerra Nov 26, 2024
727a1d3
Fix path to CMI data
edubezerra Nov 26, 2024
1bc43b9
Fix path to CMI data
edubezerra Nov 26, 2024
619b4e5
check acc15min vs 1 hour script
NietoCurcio Nov 28, 2024
f4c07e1
update title + boxplot label + qcheck folder
NietoCurcio Nov 28, 2024
cc09779
use dask actors
NietoCurcio Nov 28, 2024
4d16a9c
adjusting explicitly warp opt for spatial resolution
vkskayo Nov 28, 2024
2a88b1d
Merge branch 'use-dask' into NietoCurcio-NWP
NietoCurcio Nov 28, 2024
7fcaf33
Replace NietoCurcio-NWP content with use-basemanager
NietoCurcio Nov 28, 2024
12ffe17
fix bottom row and right column first commit
NietoCurcio Nov 29, 2024
b1655e6
Merge pull request #51 from NietoCurcio/acc15min-1h
edubezerra Nov 29, 2024
57665ba
update not zero rows validation
NietoCurcio Nov 30, 2024
c67b584
integrate alertario rain gauges
NietoCurcio Dec 1, 2024
d293ee0
Update code to generate animations.
edubezerra Dec 1, 2024
cb71b20
update minimum and maximum date
NietoCurcio Dec 1, 2024
bdc8261
update xr.Dataset month cache + update logs
NietoCurcio Dec 1, 2024
088a99c
integrate Alertario + update official alertario source
NietoCurcio Dec 2, 2024
18e62b9
fix websisrenes get precipitation in square
NietoCurcio Dec 2, 2024
245e1af
ERA5 Square + update open_dataset logic
NietoCurcio Dec 2, 2024
04efc3a
decouple ERA5 data with inheritance
NietoCurcio Dec 2, 2024
0878fa7
Merge branch 'NietoCurcio-NWP' into era5-square
NietoCurcio Dec 3, 2024
57bf2e4
configure --only-ERA5 flag
NietoCurcio Dec 3, 2024
49b132b
Merge branch 'NietoCurcio-NWP' into era5-square
NietoCurcio Dec 3, 2024
6c057a8
handle no coords files - only ERA5
NietoCurcio Dec 3, 2024
8ee6502
Merge branch 'NietoCurcio-NWP' into era5-square
NietoCurcio Dec 3, 2024
e155fa0
Merge pull request #52 from NietoCurcio/era5-square
edubezerra Dec 3, 2024
546a990
Add stconvs2s.ipynb
edubezerra Dec 3, 2024
800f14b
rm animation.mp4
edubezerra Dec 4, 2024
f270c84
glm feature calculation and formatting
vkskayo Dec 4, 2024
112a1af
Merge branch 'main' of https://github.com/vkskayo/atmoseer
vkskayo Dec 4, 2024
56d5714
Merge pull request #53 from vkskayo/main
edubezerra Dec 4, 2024
2f20a32
Change output directory.
edubezerra Dec 4, 2024
fcc5ac2
adjust timestamp to previous interval
vkskayo Dec 4, 2024
228a241
Merge branch 'main' of https://github.com/vkskayo/atmoseer
vkskayo Dec 4, 2024
a550327
Merge pull request #54 from vkskayo/main
edubezerra Dec 4, 2024
d40e965
add concurrent download for GLM retrieval code
vkskayo Dec 5, 2024
0addaae
Merge branch 'main' of https://github.com/vkskayo/atmoseer
vkskayo Dec 5, 2024
14d8e65
Delete src/goes16_sync_retrieve_glm.py
vkskayo Dec 5, 2024
28c7ea4
Merge pull request #55 from vkskayo/main
edubezerra Dec 5, 2024
8b2017d
Add example to build dataset for stconvs2s
edubezerra Dec 5, 2024
93f1f33
Merge branch 'main' of https://github.com/AIRGOLAB-CEFET-RJ/atmoseer …
edubezerra Dec 5, 2024
b76a451
Add gpm_downloader.py
edubezerra Dec 9, 2024
56fd717
Add plot_half_hourly_data.ipynb
edubezerra Dec 10, 2024
52f6cfc
Add section on reference material
edubezerra Dec 10, 2024
699cbf3
animation glm script
vkskayo Dec 11, 2024
a3b411d
Merge pull request #56 from vkskayo/main
edubezerra Dec 11, 2024
2ceb95a
Change base directory for storing features .
edubezerra Dec 13, 2024
1aa7ce8
Improve error msg on shape compatibility
edubezerra Dec 16, 2024
038a585
Create analyse_nc_file.ipynb
RaulMartino Jan 13, 2025
d5d7994
add parameter spatial resolution to the cmi downloader
vkskayo Jan 14, 2025
30c6fd1
add parameter spatial resolution to the glm extractor
vkskayo Jan 14, 2025
9411538
Merge branch 'main' of https://github.com/vkskayo/atmoseer
vkskayo Jan 14, 2025
ac5f161
Add anl_radar_inea.ipynb
edubezerra Jan 16, 2025
1084b6b
Add anl_radar_inea.ipynb
edubezerra Jan 16, 2025
1e62326
Unify all features in one datset with correct TS
RaulMartino Jan 20, 2025
283d8dc
Add notebook for EDA on radar data
edubezerra Jan 21, 2025
d593b27
Merge pull request #57 from vkskayo/main
edubezerra Jan 21, 2025
5caae19
Collect samples cell
RaulMartino Jan 24, 2025
4bd03ba
Removing unnecessary coords and comparing with reference dataset
RaulMartino Jan 30, 2025
032de62
Update goes16_generate_samples_to_stconvs2s.py
RaulMartino Jan 31, 2025
158314b
Merge branch 'AILAB-CEFET-RJ:main' into main
RaulMartino Jan 31, 2025
79d8eea
Delete src/analyse_nc_file.ipynb
RaulMartino Jan 31, 2025
40ebe17
fix gpm scraping and crop imerg files
vkskayo Jan 31, 2025
bc56535
Merge pull request #59 from vkskayo/main
edubezerra Feb 2, 2025
81a6c50
Add confguration description.
edubezerra Feb 2, 2025
0bfc08d
Add exception catching code when calling session.get(...)
edubezerra Feb 2, 2025
803d147
Change timeout to 100
edubezerra Feb 2, 2025
1ab69f9
Change timeout
edubezerra Feb 3, 2025
80d85f0
Merge branch 'main' of https://github.com/AIRGOLAB-CEFET-RJ/atmoseer …
edubezerra Feb 3, 2025
30ad69d
Change timeout value
edubezerra Feb 3, 2025
19fe475
Adding scripts for Sumare Radar analysis
pedrojoaquim01 Feb 5, 2025
0c1f0c0
Update goes16_generate_samples_to_stconvs2s.py
RaulMartino Feb 9, 2025
6579ed1
Merge branch 'main' of https://github.com/RaulMartino/atmoseer
RaulMartino Feb 9, 2025
52bbdd2
Add processing logs
RaulMartino Feb 9, 2025
76854a7
Adding Rain Gauge Stations from COR to the correlation script
pedrojoaquim01 Feb 17, 2025
3952a05
Fix on the correlation and crop_image scripts
pedrojoaquim01 Feb 24, 2025
9064c90
Add plot_bboxes.py
edubezerra Mar 2, 2025
4a5fd5d
Add notebooks: downscaling and elevation
edubezerra Mar 2, 2025
2b76e8f
Merge branch 'AILAB-CEFET-RJ:main' into main
RaulMartino Apr 1, 2025
6ef83c0
Update environment.yml
edubezerra May 23, 2025
ae65c73
Update environment.yml
edubezerra May 23, 2025
0ee9b59
Fix: sqlite3
edubezerra May 23, 2025
d578126
Update sqlite
edubezerra May 23, 2025
bae0fbb
Fix sqlite config
edubezerra May 23, 2025
6a4ef0a
Remove sqlite config
edubezerra May 23, 2025
33dbc0f
Fiz environment.yml
edubezerra May 23, 2025
ff38650
Add a retry loop with exponential backoff
edubezerra May 27, 2025
d6104ec
Improve setup.sh
edubezerra May 27, 2025
c9afb77
Add ignored_months argument
edubezerra May 28, 2025
5ddf167
Update goes16_generate_samples_to_stconvs2s.py
RaulMartino Jun 20, 2025
2a00593
Refactor GOES-16 and GPM scripts; add Makefile support and reorganize…
edubezerra Jun 29, 2025
9ef09b7
Add more README files.
edubezerra Jun 29, 2025
2089d7b
Modularize feature extraction code (one script per feature)
edubezerra Jun 30, 2025
6213c02
Fix bug
edubezerra Jun 30, 2025
672670d
Add feature extraction code for particle size
vkskayo Jul 3, 2025
dfc7180
Merge pull request #61 from vkskayo/main
edubezerra Jul 3, 2025
e85ddbe
Merge pull request #60 from pedrojoaquim01/radar-sumare
edubezerra Jul 3, 2025
8a0e5af
Add sample checker and improve STConvS2S sample metadata
RaulMartino Jul 6, 2025
d2ae242
Add offset parameter to sample collection logic
RaulMartino Jul 16, 2025
c3e9160
Update README
edubezerra Jul 16, 2025
a2cb754
Sort timestamps and track accepted sample indices
RaulMartino Jul 16, 2025
45c5d5d
Update goes16_generate_samples_to_stconvs2s.py
RaulMartino Jul 16, 2025
d42eebb
Aggregate features and targets by hour
RaulMartino Jul 18, 2025
095ebfe
pipeline com dados do cemaden
JGK19 Jul 19, 2025
a89888d
Add hourly max and mean aggregation for features
RaulMartino Jul 29, 2025
cf60d21
Add notebooks for feature plotting and sample testing
RaulMartino Jul 29, 2025
1e292d3
Merge pull request #62 from JGK19/pipeline-cemaden
edubezerra Jul 31, 2025
75be434
add goes16 cmi plots + docs
vkskayo Aug 7, 2025
3d8c2a1
Refactor sample generation and aggregation logic
RaulMartino Aug 7, 2025
8cbb11f
Merge branch 'main' of https://github.com/RaulMartino/atmoseer
RaulMartino Aug 30, 2025
ff5222e
Add STConvS2S dataset, training, and utils modules
RaulMartino Aug 30, 2025
75a2197
Remove obsolete sample testing scripts and notebooks
RaulMartino Aug 30, 2025
a90384c
Remove duplicate notebook
RaulMartino Aug 30, 2025
afbb768
Create testing_samples.ipynb
RaulMartino Aug 30, 2025
0ad41a5
Merge pull request #63 from vkskayo/main
edubezerra Sep 19, 2025
a39c8f3
Merge branch 'main' of https://github.com/AILAB-CEFET-RJ/atmoseer
edubezerra Sep 19, 2025
082b463
Merge branch 'main' into main
edubezerra Sep 19, 2025
eaeee7c
Merge pull request #64 from RaulMartino/main
edubezerra Sep 19, 2025
5c31125
Create test.txt
sangonvi Sep 20, 2025
b6ec8dc
atualizacoes
Sep 23, 2025
af7ec0a
Merge pull request #65 from sangonvi/main
edubezerra Sep 23, 2025
a4d4e97
Refactor Sumaré radar code
edubezerra Oct 19, 2025
5b349a7
First version of calibration code for Sumaré radar
edubezerra Oct 20, 2025
26907de
Preprocess: load per-system JSON configs, update Makefile, add docs
edubezerra Nov 1, 2025
8b1aee3
Preprocess: load per-system JSON configs, update Makefile, add docs
edubezerra Nov 1, 2025
cd9bc23
Preprocess docs/config: move station metadata into system JSONs, drop…
edubezerra Nov 1, 2025
a6a4f67
Preprocess docs/config: move station metadata into system JSONs, drop…
edubezerra Nov 1, 2025
2a8f67b
Add .DS_Store to .gitignore
edubezerra Nov 1, 2025
fcda721
Add .DS_Store and __pycache__ to .gitignore
edubezerra Nov 1, 2025
b2bd038
Preprocess: add per-system scaling/imputation config, document JSON u…
edubezerra Nov 1, 2025
a8c06dd
Preprocess: add per-system quality report generation and docs
edubezerra Nov 1, 2025
a5beca1
Preprocess: add quality checks with per-system config, expand docs an…
edubezerra Nov 1, 2025
8c899ef
Remove __pycache__ do versionamento
edubezerra Nov 1, 2025
4ef26d8
Remove .DS_Store do versionamento
edubezerra Nov 1, 2025
3f8a654
Atualizar scripts CEMADEN para usar credenciais via .env
edubezerra Nov 1, 2025
007bfdf
Delete src/.DS_Store
edubezerra Nov 1, 2025
6266272
Delete src/goes16/__pycache__ directory
edubezerra Nov 1, 2025
43a335b
Delete src/utils/__pycache__ directory
edubezerra Nov 1, 2025
c7a2cbc
Delete src/websirenes/.DS_Store
edubezerra Nov 1, 2025
5caf52e
Delete src/websirenes/websirenes_defesa_civil directory
edubezerra Nov 1, 2025
2100f75
Move to appropriate folder
edubezerra Nov 1, 2025
279ad14
Fix issues using ruff
edubezerra Nov 1, 2025
95a1242
Fix issues using ruff
edubezerra Nov 1, 2025
d6e4ec8
fix (unused variable)
edubezerra Nov 1, 2025
1dd65d8
Reformat based on ruff
edubezerra Nov 1, 2025
f1d4519
Latest version - RGB Rainfall Calibration notebook
noemicho Nov 11, 2025
9053df6
Add two_stage_undersampling.ipynb
edubezerra Nov 11, 2025
4fa1aab
Merge pull request #66 from noemicho/main
edubezerra Nov 11, 2025
8284faf
feat: Add WebSirene preprocessing pipeline and configuration files
leonardovonrf Nov 14, 2025
caebda3
Merge pull request #69 from leonardovonrf/leonardo
edubezerra Nov 14, 2025
2329559
Updated WebSirene retrieval to parse the XML manually so we capture l…
edubezerra Nov 14, 2025
c80d6ad
Script_IGRA_UOW
EnnyaGoc Nov 15, 2025
1c832ac
Merge pull request #70 from EnnyaGoc/ennya
edubezerra Nov 15, 2025
bd8f399
Add WebSirene fetch, export, and animation tooling
edubezerra Nov 23, 2025
a68fbcb
Make globals configurable via environment variables
edubezerra Nov 23, 2025
7da9c03
Tweak WebSirene animation defaults (imports, bbox fallback, marker si…
edubezerra Nov 23, 2025
140fd7f
Update overlay notebook to load stations from JSON and add DEM altitu…
edubezerra Dec 19, 2025
ee497a2
add script do build corrdiff dataset
sangonvi Apr 23, 2026
b042007
Merge branch 'main' into vinicius
sangonvi Apr 23, 2026
6a5e098
teste
sangonvi Apr 24, 2026
23b482c
teste
sangonvi Apr 24, 2026
8b47fa4
teste
sangonvi Apr 24, 2026
cae0898
teste
sangonvi Apr 24, 2026
803918b
teste
sangonvi Apr 24, 2026
4f62714
teste
sangonvi Apr 24, 2026
7504e8c
teste
sangonvi Apr 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .env
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
INMET_API_BASE_URL=https://apitempo.inmet.gov.br
WS_INMET_DATA_DIR=./data/ws/inmet/
WS_ALERTARIO_DATA_DIR=./data/ws/alertario/ws/
GS_ALERTARIO_DATA_DIR=./data/ws/alertario/rain_gauge_era5_fused/
CEMADEN_DATA_DIR=.data/ws/cemaden/
SUMARE_DATA_DIR=.data/radar_sumare
GOES16_DATA_DIR=./data/goes16/CMI/
GOES16_FEATURES_DIR=./data/goes16/features/
NWP_DATA_DIR=./data/NWP/
AS_DATA_DIR=./data/as/
DSI_DATA_DIR=./data/goes16/DSI
TPW_DATA_DIR=./data/goes16/wsoi
DATASETS_DIR=./data/datasets/
MODELS_DIR=./models/
WEBSIRENE_DATA_DIR=./data/ws/websirene/
REGION_LAT_MIN=-23.801876626302175
REGION_LON_MIN=-45.05290312102409
REGION_LAT_MAX=-21.699774257353113
REGION_LON_MAX=-42.35676996062447
44 changes: 43 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,9 +1,16 @@
.vscode/
clea.vscode/
data/
datasets/
models/
backup/
__pycache__/
*.pyc
*.log

src/spatiotemporal_builder/.github

.DS_Store
src/.DS_Store

src/__pycache__/globals.cpython-38.pyc
src/__pycache__/globals.cpython-310.pyc
Expand All @@ -17,3 +24,38 @@ src/__pycache__/util.cpython-310.pyc
src/__pycache__/globals.cpython-310.pyc
src/__pycache__/util.cpython-310.pyc
src/__pycache__/goes16_utils.cpython-310.pyc
nohup.out
src/__pycache__/
src/surface_stations/__pycache__/
*.out
*.gif
*.png
*.nc
*.parquet
*.log
src/spatiotemporal_builder/__pycache__/
src/spatiotemporal_builder/features/
anim/dF_dt_2024_01_13/animation.mp4
notebooks/GPM/imerg_animation_2024_01_01.mp4
notebooks/GPM/imerg_rio_animation.mp4
notebooks/GPM/imerg_rio_animation_2024_01_01.mp4
anim/PN_GLM/animation.mp4
notebooks/RJ-30m-DEM.tif
*.tgz
src/config/__pycache__/__init__.cpython-310.pyc
src/config/__pycache__/globals.cpython-310.pyc
src/goes16/__pycache__/__init__.cpython-310.pyc
src/goes16/__pycache__/features.cpython-310.pyc
src/goes16/__pycache__/utils.cpython-310.pyc
src/goes16/__pycache__/goes16_cmi_feature_extractor_extended.cpython-311.pyc
src/goes16/__pycache__/goes16_cmi_feature_extractor.cpython-311.pyc
src/goes16/features/__pycache__/__init__.cpython-310.pyc
src/goes16/features/__pycache__/fa.cpython-310.pyc
src/goes16/features/__pycache__/gtn.cpython-310.pyc
src/goes16/features/__pycache__/li_proxy.cpython-310.pyc
src/goes16/features/__pycache__/pn_std.cpython-310.pyc
src/goes16/features/__pycache__/pn.cpython-310.pyc
src/goes16/features/__pycache__/toct.cpython-310.pyc
src/goes16/features/__pycache__/wv_grad.cpython-310.pyc
src/utils/__pycache__/__init__.cpython-310.pyc
src/utils/__pycache__/util.cpython-310.pyc
13 changes: 13 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
repos:
- repo: https://github.com/psf/black
rev: 24.4.2
hooks:
- id: black
args: ["--line-length", "88"]

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.5.6
hooks:
- id: ruff
args: ["--fix"]
- id: ruff-format
112 changes: 112 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# === AtmoSeer Makefile ===

# Diretório base para os dados
DATA_DIR=data

# Diretório base para o código-fonte
SRC_DIR=src

# ========== Surfaces stations - CEMADEN ========================
cemaden_retrieve_data:
PYTHONPATH=src python3 src/surface_stations/retrieve_data_cemaden.py \
$(if $(IBGE),--ibge $(IBGE)) \
$(if $(START),--inicio $(START)) \
$(if $(END),--fim $(END)) \
$(if $(EMAIL),--email $(EMAIL)) \
$(if $(SENHA),--senha $(SENHA)) \
$(if $(ESTACAO),--estacao $(ESTACAO))

cemaden_retrieve_ws:
PYTHONPATH=src python3 src/surface_stations/retrieve_ws_cemaden.py \
$(if $(ESTADO),-e $(ESTADO)) \
$(if $(IDESTACAO),-c $(IDESTACAO)) \
$(if $(INICIO),-start $(INICIO)) \
$(if $(FIM),-end $(FIM)) \
$(if $(SAIDA),-o $(SAIDA)) \

cemaden_mapa:
PYTHONPATH=src python3 src/surface_stations/cemaden_mapa.py

cemaden_mapa_chuva:
PYTHONPATH=src python3 src/surface_stations/cemaden_mapa_chuva.py

# ========== Surfaces stations - INMET ========================
inmet_retrieve_ws:
PYTHONPATH=src python3 src/surface_stations/retrieve_ws_inmet.py \
$(if $(API_TOKEN),-t $(API_TOKEN)) \
$(if $(STATION_ID),-s $(STATION_ID)) \
$(if $(BEGIN_YEAR),-b $(BEGIN_YEAR)) \
$(if $(END_YEAR),-e $(END_YEAR))
# Example usage:
# make inmet_retrieve_ws API_TOKEN=your_token STATION_ID=A601 BEGIN_YEAR=2020 END_YEAR=2023

surface_stations_preprocess:
PYTHONPATH=src python3 src/surface_stations/preprocess.py \
$(if $(STATION_ID),--station_id $(STATION_ID)) \
$(if $(SYSTEM),--station_system $(SYSTEM))
# Example usage:
# make surface_stations_preprocess STATION_ID=A601 SYSTEM=INMET


# === GOES-16 Downloader/Cropper ===
goes16-download-crop:
PYTHONPATH=src python src/goes16/goes16_download_crop.py \
--start_date $(START) \
--end_date $(END) \
--channel $(CHANNEL) \
--crop_dir $(DIR) \
--spatial_resolution 0.1 \
--vars CMI

# === GOES-16 Feature Extractor ===
goes16-features:
PYTHONPATH=src python src/goes16/main_goes16_features.py $(FEATS)

sumare-crop-images:
PYTHONPATH=src python3 src/sumare_radar/crop_image.py

# Validates a GOES-16 feature based on the specified channels
validate-feature:
@echo "Validating feature '$(FEAT)' with channels $(CANAIS)"
PYTHONPATH=src python src/goes16/validate_features.py --feature $(FEAT) --canais $(CANAIS)

# Summarize GOES-16 feature extraction log files
analyze-logs:
@echo "Analyzing log files in src/goes16/features/"
@PYTHONPATH=src python src/goes16/features/analyze_logs.py

# === GPM Download/Cropper ===
gpm-download-crop:
PYTHONPATH=src python src/gpm/gpm_download_crop.py \
--begin_date "$(BEGIN)" \
--end_date "$(END)" \
--user "$(USER)" \
--pwd "$(PWD)" \
$(DEBUG) \
$(IGNORED)


# === Alerta Rio Parser (placeholder) ===
alerta-rio-process:
PYTHONPATH=$(SRC_DIR) python $(SRC_DIR)/surface_stations/alerta_rio_parser.py


# === Clean Outputs ===
clean:
rm -rf $(DATA_DIR)/goes16/features/*

.PHONY: format lint lint-fix pre-commit-install

format:
ruff format src tests
black src tests

lint:
ruff check src tests

lint-fix:
ruff check src tests --fix
black src tests

pre-commit-install:
pre-commit install
147 changes: 120 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,142 @@
# AtmoSeer

## About
AtmoSeer is a set of Python scripts for downloading, processing, and analyzing atmospheric and meteorological data from sources such as NASA GPM and GOES-16. It supports environment setup via Conda and requires authentication for NASA Earthdata access. This project also provides a pipeline to build rainfall forecast models. The pipeline can be configured with different meteorological data sources.

This project provides a pipeline to build rainfall forecast models. The pipeline can be configured with different meteorological data sources.
---

## Install
## Installation

In the root directory of this repository, type the following command (you must have conda installed in your system):

### 1. Clone the repository

```bash
git clone https://github.com/your-org/atmoseer.git
cd atmoseer
````

### 2. Set NASA Earthdata Credentials

Before running the setup script, you must set your NASA Earthdata credentials as environment variables:

```bash
export EARTHDATA_USER="your_username"
export EARTHDATA_PASS="your_password"
```

> You can register for a free NASA Earthdata account at [https://urs.earthdata.nasa.gov](https://urs.earthdata.nasa.gov).

### 3. Run the Setup Script

The `setup.sh` script will:

* Remove any existing Conda environment named `atmoseer`
* Create a new Conda environment using `config/environment.yml`
* Create necessary data directories
* Configure `.netrc` and `.urs_cookies` in your home directory for Earthdata access

```bash
./setup.sh
```

---

## Usage

After the environment is set up, activate it:

```bash
conda activate atmoseer
```

Now, you can start running AtmoSeer scripts.

## Running Scripts with Make

To simplify execution, AtmoSeer supports running preprocessing tasks using `make`. Each data module has its own Makefile targets and README with details.

### ✅ Examples

- **GOES-16 Feature Extraction**

```bash
make goes16-features FEATS="--pn --fa --toct --verbose"
```

See full usage in [`src/goes16/README.md`](src/goes16/README.md)

- **GPM Download and Crop**

```bash
make gpm-download-crop BEGIN=2024/01/01 END=2024/01/03 USER=your_username PWD=your_password
```

See full usage in [`src/gpm/README.md`](src/gpm/README.md)

---

## 📁 Directory Structure

```
atmoseer/
├── config/ # Conda environment and config files
│ └── environment.yml
├── data/ # Automatically created data directories
│ ├── datasets/
│ ├── era5/
│ ├── goes16/
├── gpm/
│ ├── sounding/
│ ├── surface_stations/
│ └──
├── src/ # Scripts for downloading and processing data, and for model training
│ ├── era5/
│ ├── goes16/
├── gpm/
│ ├── sounding/
│ ├── surface_stations/
└── ...
├── setup.sh # Installation script
└── README.md # This file
```

## Project pipeline
---

The project pipeline is defined as a sequence of three steps: (1) data retrieving, (2) data pre-processing and (3) model training. These steps are implemented as Python scripts in the `./src` directory.
## 🛠 Requirements

#### Data retrieval
* [Miniconda](https://docs.conda.io/en/latest/miniconda.html) or [Anaconda](https://www.anaconda.com/)
* Linux or WSL (recommended)
* Internet access for installing packages and accessing NASA endpoints

All datasets retrieved and/or generated by the scripts will be stored in the `./data` folder.
---

- **_retrieve_ws_cor.py_**: This script retrieves observation from a user-provided weather station.
- **_retrieve_ws_inmet.py_**: This script retrieves observations for from a user-provided weather station.
- **_retrieve_as.py_**: this script retrieves atmospheric sounding data.
- **_retrieve_ERA5.py_**: this script retrieves numerical simulation data from the ERA5 portal.
## Guide to Collaborators

Welcome! Please follow these guidelines to keep the project consistent and secure.

##### Script **_gen_sounding_indices.py_**
- Communicate and document **only in English** so every contributor can follow along.
- Never commit secrets, credentials, or private datasets. Store sensitive values in `.env` files that stay out of version control.
- Use GitFlow conventions: branch from `develop` (or `main` if specified), follow feature/hotfix/release prefixes, and open pull requests before merging.
- Prefer small, focused commits with clear messages; reference issues or tasks when possible.
- Keep coding style consistent with the existing modules; run linting/formatting tools provided by the project before submitting code (see the tooling notes below).
- Add or update tests whenever you change behaviour, and ensure the full test suite passes locally.
- Discuss significant architectural or API changes with maintainers before implementation.
- Respect code reviews: respond to feedback promptly and revise your changes as needed.

This script will generate atmospheric instability indices for the data retrieveed by the script **_retrieve_as.py_**. Data from the SBGL sounding (located at the Galeão Airport, Rio de Janeiro - Brazil) will be used to calculate atmospheric instability indices, generating a new dataset. This new dataset contains one entry per sounding probe. SBGL sounding station produces two probes per day (at 00:00h and 12:00h UTC). Each entry in the produced contains the values of the computed instability indices for one probe. The following instability indices are computed:
### Code Style Tooling

- CAPE
- CIN
- Lift
- k
- Total totals
- Show alter
1. Install/update the Conda environment so that `black`, `ruff`, and `pre-commit` are available.
2. Run `make format` for automatic formatting (`ruff format` + `black`) and `make lint` to check for style and static-analysis issues.
3. Optional but recommended: `make pre-commit-install` to set up git hooks that run Ruff and Black before each commit.

#### Preprocessing
---

The preprocessing scripts are responsible for performing several operations on the original dataset, such as creating variables or aggregating data, which can be interesting for model training and its final result.
## 🧾 License

#### Dataset building
This project is licensed under the MIT License. See `LICENSE` for details.

These scripts will build the train, validation and test dataset from the times series produced in the previous steps. These are the datasets to be given as input to the model training step.
---

#### Model training and evaluation
## 👥 Acknowledgments

The model generation script is responsible for performing the training and exporting the results obtained by the model after testing.
# r2t
* NASA GES DISC — [https://disc.gsfc.nasa.gov](https://disc.gsfc.nasa.gov)
* Earthdata Login Services — [https://urs.earthdata.nasa.gov](https://urs.earthdata.nasa.gov)
Loading