Skip to content

Ad-hoc testing of Kerchunk engine compatibility with xCDAT #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tomvothecoder wants to merge 63 commits into
base: main
Choose a base branch
from
Open

Ad-hoc testing of Kerchunk engine compatibility with xCDAT #61

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
25da07d
Add environment file
tomvothecoder Jan 8, 2026
cb399d9
Add modules for experimentation
tomvothecoder Jan 8, 2026
f7c01ab
Extract benchmarking utilities to benchmark.py
tomvothecoder Jan 9, 2026
531913b
Add notes and findings
tomvothecoder Jan 9, 2026
09e0c8c
Add plotting code
tomvothecoder Jan 12, 2026
955d836
Add experiment results
tomvothecoder Jan 12, 2026
24af4af
Fix json netcdf map
tomvothecoder Jan 16, 2026
ea3996d
Refactor code to use pandas and parallelism
tomvothecoder Jan 16, 2026
0044481
Use python logger module
tomvothecoder Jan 16, 2026
d179490
Update FREQ_PATTERN
tomvothecoder Jan 26, 2026
3a5bfc5
Update main.py and benchmark.py for production runs
tomvothecoder Jan 26, 2026
83d4c43
Add latest metrics
tomvothecoder Jan 27, 2026
8424ec2
Update module name
tomvothecoder Jan 27, 2026
2fb2618
Add metric notebooks
tomvothecoder Jan 27, 2026
2bed6ea
Update title of cell
tomvothecoder Jan 27, 2026
a6a545f
Update takeaway
tomvothecoder Jan 27, 2026
693b21e
Fix display of dataframe
tomvothecoder Jan 27, 2026
f69eea6
Add single file metrics notebook
tomvothecoder Feb 5, 2026
9a4cb2b
Add 3hr notebook
tomvothecoder Feb 5, 2026
75d61d4
Add overall takeaway to single file notebook
tomvothecoder Feb 5, 2026
08f7b7b
Update notebook with key findings on reference counts
tomvothecoder Feb 9, 2026
55cb09c
Update notebook
tomvothecoder Feb 9, 2026
47b8072
Fix comment
tomvothecoder Feb 9, 2026
8920f1d
Update 3hr_outlier_metrics.ipynb
tomvothecoder Feb 13, 2026
77654a4
Add initial end-to-end .load() notebook
tomvothecoder Feb 19, 2026
2a46e1c
Add Steve scripts
tomvothecoder Mar 2, 2026
f56ba96
Enhance script for better benchmarking
tomvothecoder Mar 2, 2026
1a0d220
More improvements to head_to_head.py
tomvothecoder Mar 2, 2026
4914902
Update head_to_head.py
tomvothecoder Mar 2, 2026
e252d31
Update docstring
tomvothecoder Mar 2, 2026
b2b5f87
Add benchmark reuslts
tomvothecoder Mar 2, 2026
1e901c3
Add dask metrics
tomvothecoder Mar 2, 2026
5505aef
Remove outdated plots and scripts
tomvothecoder Mar 2, 2026
690355c
Fail fast and skip datasets with malformed cf metadata
tomvothecoder Mar 6, 2026
9f707ba
Remove dataset-level parallelism to prevent noise
tomvothecoder Mar 6, 2026
75c4f51
Add results of benchmark
tomvothecoder Mar 9, 2026
ab8c4e3
Add considerations section
tomvothecoder Mar 9, 2026
0b11faa
Update possible causes section
tomvothecoder Mar 9, 2026
925aa14
Update directories and add script for file-count
tomvothecoder Mar 13, 2026
058287c
Add dataset dir to kerchunk mapping
tomvothecoder Mar 16, 2026
de4fe28
Update with CLI args
tomvothecoder Mar 16, 2026
96c5c67
Add results
tomvothecoder Mar 19, 2026
085e7a2
Add overall summary
tomvothecoder Mar 19, 2026
f58bad7
Update sections by benchmark
tomvothecoder Mar 19, 2026
fd3ccdc
Add latest benchmark scripts
tomvothecoder Apr 16, 2026
1a8915e
Update benchmark script by bin
tomvothecoder Apr 16, 2026
c75dc30
Add results
tomvothecoder Apr 16, 2026
dcbd27f
Remove practical read section from summary.md
tomvothecoder Apr 16, 2026
5d6fa01
Add scripts for latest upscale benchmark
tomvothecoder Apr 23, 2026
4eb4e4a
Update prepare_datasets.py
tomvothecoder Apr 23, 2026
f522014
Improve kerchunk dataset preparation workflow
tomvothecoder Apr 23, 2026
e8f503c
Add loggers to prepare_datasets.py
tomvothecoder Apr 23, 2026
fb92ecc
Add latest benchmark results
tomvothecoder Apr 28, 2026
1d461c4
Add summary.md
tomvothecoder Apr 28, 2026
34b0ba0
Add daily crossover benchmark artifacts and QA
tomvothecoder May 27, 2026
5caa547
Add final total timing by bin plots
tomvothecoder May 27, 2026
a6f4de0
Update summaries
tomvothecoder May 27, 2026
2d28b44
Add metadata CF validation artifacts
tomvothecoder Jun 9, 2026
62962a7
Remove `sasha-tmp` from paths
tomvothecoder Jul 2, 2026
d2db909
Fix .grkerchunk and .gnkerchunk
tomvothecoder Jul 2, 2026
d851abf
Add API comparison workflow and resume support
tomvothecoder Jul 2, 2026
a470663
Update markdowns
tomvothecoder Jul 2, 2026
23b3000
Add overall summary
tomvothecoder Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,3 +144,6 @@ input/
*.nc
*.xml
*.sh.status

riotai/json_to_netcdf_maps/json_to_netcdf_table.csv
riotai/json_to_netcdf_maps/json_to_netcdf.json
148 changes: 148 additions & 0 deletions ecmwf_time_report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
# ECMWF Time Coordinate Report

- Source CSV: `/global/u2/v/vo13/xCDAT/xcdat-validation/riotai/results/20260605-metadata-cf-validation/results.csv`
- Dataset count: 6

## CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.highresSST-present.r5i1p1f1.Amon.pr.gr.v20181119

- NetCDF dir: `/global/cfs/projectdirs/m4931/gsharing/css03_data/CMIP6/HighResMIP/ECMWF/ECMWF-IFS-HR/highresSST-present/r5i1p1f1/Amon/pr/gr/v20181119`
- Kerchunk file: `/global/cfs/projectdirs/m4931/kerchunk/pr/highresSST-present/mon/CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.highresSST-present.r5i1p1f1.Amon.pr.gr.v20181119.kerchunk.json`
- NetCDF files: 65
- Time lengths: kerchunk=618, netcdf=780
- Time start: kerchunk=1950-01-16 12:00:00, netcdf=1950-01-16 12:00:00
- Time end: kerchunk=1984-12-16 12:00:00, netcdf=2014-12-16 12:00:00
- Summary:
- end differs: kerchunk=1984-12-16 12:00:00 vs netcdf=2014-12-16 12:00:00
- length differs: kerchunk=618 vs netcdf=780
- Example values only in kerchunk:
- 1950-02-15 12:00:00
- 1950-03-17 12:00:00
- 1950-04-17 00:00:00
- 1950-05-17 12:00:00
- 1950-06-17 00:00:00
- Example values only in netcdf:
- 1980-01-16 12:00:00
- 1980-02-15 12:00:00
- 1982-02-15 00:00:00
- 1982-03-16 12:00:00
- 1982-04-16 00:00:00

## CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.highresSST-present.r6i1p1f1.Amon.pr.gr.v20181119

- NetCDF dir: `/global/cfs/projectdirs/m4931/gsharing/css03_data/CMIP6/HighResMIP/ECMWF/ECMWF-IFS-HR/highresSST-present/r6i1p1f1/Amon/pr/gr/v20181119`
- Kerchunk file: `/global/cfs/projectdirs/m4931/kerchunk/pr/highresSST-present/mon/CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.highresSST-present.r6i1p1f1.Amon.pr.gr.v20181119.kerchunk.json`
- NetCDF files: 65
- Time lengths: kerchunk=618, netcdf=780
- Time start: kerchunk=1950-01-16 12:00:00, netcdf=1950-01-16 12:00:00
- Time end: kerchunk=1984-12-16 12:00:00, netcdf=2014-12-16 12:00:00
- Summary:
- end differs: kerchunk=1984-12-16 12:00:00 vs netcdf=2014-12-16 12:00:00
- length differs: kerchunk=618 vs netcdf=780
- Example values only in kerchunk:
- 1950-02-15 12:00:00
- 1950-03-17 12:00:00
- 1950-04-17 00:00:00
- 1950-05-17 12:00:00
- 1950-06-17 00:00:00
- Example values only in netcdf:
- 1980-01-16 12:00:00
- 1980-02-15 12:00:00
- 1982-02-15 00:00:00
- 1982-03-16 12:00:00
- 1982-04-16 00:00:00

## CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.hist-1950.r1i1p1f1.Amon.pr.gr.v20170915

- NetCDF dir: `/global/cfs/projectdirs/m4931/gsharing/css03_data/CMIP6/HighResMIP/ECMWF/ECMWF-IFS-HR/hist-1950/r1i1p1f1/Amon/pr/gr/v20170915`
- Kerchunk file: `/global/cfs/projectdirs/m4931/kerchunk/pr/hist-1950/mon/CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.hist-1950.r1i1p1f1.Amon.pr.gr.v20170915.kerchunk.json`
- NetCDF files: 65
- Time lengths: kerchunk=618, netcdf=780
- Time start: kerchunk=1980-01-16 12:00:00, netcdf=1950-01-16 12:00:00
- Time end: kerchunk=2014-12-16 12:00:00, netcdf=2014-12-16 12:00:00
- Summary:
- start differs: kerchunk=1980-01-16 12:00:00 vs netcdf=1950-01-16 12:00:00
- length differs: kerchunk=618 vs netcdf=780
- Example values only in kerchunk:
- 1980-02-15 00:00:00
- 1980-03-15 12:00:00
- 1980-04-15 00:00:00
- 1980-05-15 12:00:00
- 1980-06-15 00:00:00
- Example values only in netcdf:
- 1950-01-16 12:00:00
- 1950-02-15 00:00:00
- 1950-03-16 12:00:00
- 1950-04-16 00:00:00
- 1950-05-16 12:00:00

## CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.hist-1950.r2i1p1f1.Amon.pr.gr.v20181119

- NetCDF dir: `/global/cfs/projectdirs/m4931/gsharing/css03_data/CMIP6/HighResMIP/ECMWF/ECMWF-IFS-HR/hist-1950/r2i1p1f1/Amon/pr/gr/v20181119`
- Kerchunk file: `/global/cfs/projectdirs/m4931/kerchunk/pr/hist-1950/mon/CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.hist-1950.r2i1p1f1.Amon.pr.gr.v20181119.kerchunk.json`
- NetCDF files: 65
- Time lengths: kerchunk=618, netcdf=780
- Time start: kerchunk=1980-01-16 12:00:00, netcdf=1950-01-16 12:00:00
- Time end: kerchunk=2014-12-16 12:00:00, netcdf=2014-12-16 12:00:00
- Summary:
- start differs: kerchunk=1980-01-16 12:00:00 vs netcdf=1950-01-16 12:00:00
- length differs: kerchunk=618 vs netcdf=780
- Example values only in kerchunk:
- 1980-02-15 00:00:00
- 1980-03-15 12:00:00
- 1980-04-15 00:00:00
- 1980-05-15 12:00:00
- 1980-06-15 00:00:00
- Example values only in netcdf:
- 1950-01-16 12:00:00
- 1950-02-15 00:00:00
- 1950-03-16 12:00:00
- 1950-04-16 00:00:00
- 1950-05-16 12:00:00

## CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.hist-1950.r3i1p1f1.Amon.pr.gr.v20181119

- NetCDF dir: `/global/cfs/projectdirs/m4931/gsharing/css03_data/CMIP6/HighResMIP/ECMWF/ECMWF-IFS-HR/hist-1950/r3i1p1f1/Amon/pr/gr/v20181119`
- Kerchunk file: `/global/cfs/projectdirs/m4931/kerchunk/pr/hist-1950/mon/CMIP6.HighResMIP.ECMWF.ECMWF-IFS-HR.hist-1950.r3i1p1f1.Amon.pr.gr.v20181119.kerchunk.json`
- NetCDF files: 65
- Time lengths: kerchunk=618, netcdf=780
- Time start: kerchunk=1980-01-16 12:00:00, netcdf=1950-01-16 12:00:00
- Time end: kerchunk=2014-12-16 12:00:00, netcdf=2014-12-16 12:00:00
- Summary:
- start differs: kerchunk=1980-01-16 12:00:00 vs netcdf=1950-01-16 12:00:00
- length differs: kerchunk=618 vs netcdf=780
- Example values only in kerchunk:
- 1980-02-15 00:00:00
- 1980-03-15 12:00:00
- 1980-04-15 00:00:00
- 1980-05-15 12:00:00
- 1980-06-15 00:00:00
- Example values only in netcdf:
- 1950-01-16 12:00:00
- 1950-02-15 00:00:00
- 1950-03-16 12:00:00
- 1950-04-16 00:00:00
- 1950-05-16 12:00:00

## CMIP6.HighResMIP.ECMWF.ECMWF-IFS-LR.highresSST-present.r8i1p1f1.Amon.pr.gr.v20190410

- NetCDF dir: `/global/cfs/projectdirs/m4931/gsharing/css03_data/CMIP6/HighResMIP/ECMWF/ECMWF-IFS-LR/highresSST-present/r8i1p1f1/Amon/pr/gr/v20190410`
- Kerchunk file: `/global/cfs/projectdirs/m4931/kerchunk/pr/highresSST-present/mon/CMIP6.HighResMIP.ECMWF.ECMWF-IFS-LR.highresSST-present.r8i1p1f1.Amon.pr.gr.v20190410.kerchunk.json`
- NetCDF files: 65
- Time lengths: kerchunk=618, netcdf=780
- Time start: kerchunk=1950-01-16 12:00:00, netcdf=1950-01-16 12:00:00
- Time end: kerchunk=1984-12-16 12:00:00, netcdf=2014-12-16 12:00:00
- Summary:
- end differs: kerchunk=1984-12-16 12:00:00 vs netcdf=2014-12-16 12:00:00
- length differs: kerchunk=618 vs netcdf=780
- Example values only in kerchunk:
- 1950-02-15 12:00:00
- 1950-03-17 12:00:00
- 1950-04-17 00:00:00
- 1950-05-17 12:00:00
- 1950-06-17 00:00:00
- Example values only in netcdf:
- 1980-01-16 12:00:00
- 1980-02-15 12:00:00
- 1982-02-15 00:00:00
- 1982-03-16 12:00:00
- 1982-04-16 00:00:00
Empty file added riotai/__init__.py
View file
Open in desktop
Empty file.
Loading