Xarray GPU optimization#771

Merged

dcherian merged 90 commits into

xarray-contrib:mainfrom

negin513:hackathon

Jul 9, 2025

negin513 commented May 1, 2025 •

edited

Loading

Contributors: @negin513, @weiji14 , @TomAugspurger , @maxrjoes, @akshaysubr, @kafitzgerald

negin513 added 5 commits

April 30, 2025 18:32


          first draft

f69796c


          adding headers

32a8e32


          adding baseline image

d2f7e0d


          update blog post

d23c74f


          update chunking

64b45e1

vercel Bot commented May 1, 2025

@negin513 is attempting to deploy a commit to the xarray Team on Vercel.

A member of the Team first needs to authorize it.


          [pre-commit.ci] auto fixes from pre-commit.com hooks

95e5d65

for more information, see https://pre-commit.ci

maxrjones mentioned this pull request

Publish Xarray blog post on NCAR hackathon NASA-IMPACT/veda-odd#166

Closed

TomAugspurger reviewed

View reviewed changes

TomAugspurger left a comment

Contributor

Thanks for writing this up!

src/posts/gpu-pipeline/index.md Outdated

src/posts/gpu-pipeline/index.md Outdated

src/posts/gpu-pipeline/index.md Outdated

src/posts/gpu-pipeline/index.md Outdated

src/posts/gpu-pipeline/index.md Outdated

src/posts/gpu-pipeline/index.md Outdated

src/posts/gpu-pipeline/index.md Outdated


          Apply suggestions from code review

b52e1e7

Co-authored-by: Tom Augspurger <tom.augspurger88@gmail.com>

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md

+                - name: Katelyn Fitzgerald
+                  github: kafitzgerald
+              summary: 'How to accelerate AI/ML workflows in Earth Sciences with GPU-native Xarray and Zarr.'

dcherian May 8, 2025

Contributor

Can we make this more direct? "X% speedup" or "XMBps throughput"?

negin513 Jul 9, 2025

Author

Added ~17X all in all improvement!💪💪💪

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

dcherian reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

weiji14 reviewed

View reviewed changes

src/posts/gpu-pipeline/index.md Outdated

src/posts/gpu-pipeline/index.md Outdated

src/posts/gpu-pipeline/index.md Outdated

src/posts/gpu-pipeline/index.md Outdated

negin513 added 7 commits

May 12, 2025 13:46


          moving profiling_screenshot1.png over

a2416b3


          adding profiling screenshot

67f9c29


          update

5d168be


          screenshot 1 added

8c292d3


          moving baseline png

e9195c8


          adding pngs for the plots

304acb0


          updates

eccf86c

negin513 and others added 19 commits

June 12, 2025 04:00


          updates

6359e46


          address comments


          update thank you messages

b13036a


          [pre-commit.ci] auto fixes from pre-commit.com hooks

b35db32

for more information, see https://pre-commit.ci


          Update src/posts/gpu-pipeline/index.md

17352a3

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>


          improve TLDR

847ac32


          python3

278d6ab


          icechunk

261bb0a


          hiding I/O latency

90d3acb


          NSight

120502c


          updates

6a37f12


          [pre-commit.ci] auto fixes from pre-commit.com hooks

1c837ec

for more information, see https://pre-commit.ci


          all in all

6b3b5b7


          adding all in all improevemnts

7d1e8b3


          remove extra pictures

61ed1f9


          performance improvement noted

affc047


          performance improvement noted

57d0845


          performance improvement in bold

c6a8cf1


          Merge branch 'main' into hackathon

def6be3

negin513 commented Jul 9, 2025

Author

Hey folks, I think I have addressed all comments & added all the plots, if either one of you can review this: all in all +DALI +nvcomp ~17x improvement... 💪

negin513 and others added 7 commits

July 9, 2025 13:35


          update to the figure

c2c7f45


          moved to images

631f40f


          scaling plot was not rendering

91bb2fe


          update scaling plot

5d17176


          update scaling plots

816838c


          [pre-commit.ci] auto fixes from pre-commit.com hooks

5770be4

for more information, see https://pre-commit.ci


          update date

37fa44c

dcherian approved these changes

View reviewed changes

dcherian commented Jul 9, 2025

Contributor

merging in time for a scipy talk!

dcherian merged commit 98e30f0 into xarray-contrib:main

5 of 6 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

dcherian dcherian approved these changes

+4 more reviewers

TomAugspurger TomAugspurger left review comments

kafitzgerald kafitzgerald left review comments

maxrjones maxrjones left review comments

weiji14 weiji14 left review comments

Labels

None yet