You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/user_guide/examples/explanation_performance.md
+16-4Lines changed: 16 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,12 +2,18 @@
2
2
3
3
In many Parcels simulations, the bottle-neck in terms of performance is the retrieval of the hydrodynamic field data from disk. This is especially true for simulations with a relatively small number of particles, where the time spent on retrieving the fields can be much larger than the time spent on computing the particle trajectories.
4
4
5
-
In this tutorial, we will show how to squeeze performance in Parcels by using a few different techniques. Which technique works best for your case depends on the amount of field data you have and how you can store it on disk.
5
+
In this tutorial, we will show how to squeeze performance in Parcels by using a few different Parcels Backends. Which Backend works best for your case depends on the amount of field data you have and how you can store it on disk.
6
6
7
-
## Option 1: explicitly load the full FieldSet into memory
7
+
```{note}
8
+
You can check which Backend Parcels is using by calling `fieldset.describe()`. The last column shows the Backend that is used for each Field.
9
+
```
10
+
11
+
## Option 1: load the full FieldSet into memory
8
12
9
13
**Best for: small Datasets (less than a few GB)**
10
14
15
+
_Uses Parcels Backend: Numpy_
16
+
11
17
For relatively small Datasets (less than a few GB), it is possible to load the entire FieldSet into memory. This can be done by calling the `load()` method on the `xarray.Dataset` object:
12
18
13
19
```{code-cell}
@@ -22,10 +28,12 @@ This will make Parcels use `numpy` functions in the interpolation routines, whic
| Very fast and simple to implement. | Will only work if the entire Dataset fits into memory. |
24
30
25
-
## Option 2: use cached zarr files
31
+
## Option 2: use (cached) zarr files
26
32
27
33
**Best for: large Datasets (more than a few GB) and particles distributed over a small part of the domain**
28
34
35
+
_Uses Parcels Backend: Zarr_
36
+
29
37
If your Dataset is too large to fit into memory, but your particles are only distributed over a small part of the domain, it could be efficient to use cached zarr files. This can be done by using the (experimental) `zarr.CacheStore` in combination with the `parcels.open_raw_zarr()` function. This will make Parcels only load the chunks that are needed for the particles, and cache these chunks in memory for future use.
In our performance testing, we have found that using zarr files saved without any compression can be considerably faster than using compressed zarr files. However, we are working on an upstream fix in to make caching compressed zarr files faster, so this may change in the future.
50
58
```
51
59
52
-
## Option 3: use `fieldset.to_windowed_arrays()`
60
+
## Option 3: use Windowed Arrays
61
+
62
+
_Uses Parcels Backend: WindowedArray_
53
63
54
64
**Best for: large Datasets (more than a few GB) and particles distributed over the entire domain**
55
65
@@ -72,6 +82,8 @@ fieldset.to_windowed_arrays()
72
82
73
83
**Best for: large Datasets (more than a few GB) and small ParticleSets (less than a few hundred particles)**
74
84
85
+
_Uses Parcels Backend: Dask_
86
+
75
87
If your Dataset is so large that it doesn't fit into memory, and you have very few particles, you can use Dask to perform the interpolation operations. In this case, you don't have to do any special setup, as Parcels will automatically use Dask if the `xarray.Dataset` is a Dask array.
0 commit comments