Skip to content

Commit 0664406

Browse files
Adding info on how to see Backends
1 parent 4dd8f5d commit 0664406

1 file changed

Lines changed: 16 additions & 4 deletions

File tree

docs/user_guide/examples/explanation_performance.md

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,18 @@
22

33
In many Parcels simulations, the bottle-neck in terms of performance is the retrieval of the hydrodynamic field data from disk. This is especially true for simulations with a relatively small number of particles, where the time spent on retrieving the fields can be much larger than the time spent on computing the particle trajectories.
44

5-
In this tutorial, we will show how to squeeze performance in Parcels by using a few different techniques. Which technique works best for your case depends on the amount of field data you have and how you can store it on disk.
5+
In this tutorial, we will show how to squeeze performance in Parcels by using a few different Parcels Backends. Which Backend works best for your case depends on the amount of field data you have and how you can store it on disk.
66

7-
## Option 1: explicitly load the full FieldSet into memory
7+
```{note}
8+
You can check which Backend Parcels is using by calling `fieldset.describe()`. The last column shows the Backend that is used for each Field.
9+
```
10+
11+
## Option 1: load the full FieldSet into memory
812

913
**Best for: small Datasets (less than a few GB)**
1014

15+
_Uses Parcels Backend: Numpy_
16+
1117
For relatively small Datasets (less than a few GB), it is possible to load the entire FieldSet into memory. This can be done by calling the `load()` method on the `xarray.Dataset` object:
1218

1319
```{code-cell}
@@ -22,10 +28,12 @@ This will make Parcels use `numpy` functions in the interpolation routines, whic
2228
| ---------------------------------- | ------------------------------------------------------ |
2329
| Very fast and simple to implement. | Will only work if the entire Dataset fits into memory. |
2430

25-
## Option 2: use cached zarr files
31+
## Option 2: use (cached) zarr files
2632

2733
**Best for: large Datasets (more than a few GB) and particles distributed over a small part of the domain**
2834

35+
_Uses Parcels Backend: Zarr_
36+
2937
If your Dataset is too large to fit into memory, but your particles are only distributed over a small part of the domain, it could be efficient to use cached zarr files. This can be done by using the (experimental) `zarr.CacheStore` in combination with the `parcels.open_raw_zarr()` function. This will make Parcels only load the chunks that are needed for the particles, and cache these chunks in memory for future use.
3038

3139
```{code-cell}
@@ -49,7 +57,9 @@ ds = parcels.open_raw_zarr(store)
4957
In our performance testing, we have found that using zarr files saved without any compression can be considerably faster than using compressed zarr files. However, we are working on an upstream fix in to make caching compressed zarr files faster, so this may change in the future.
5058
```
5159

52-
## Option 3: use `fieldset.to_windowed_arrays()`
60+
## Option 3: use Windowed Arrays
61+
62+
_Uses Parcels Backend: WindowedArray_
5363

5464
**Best for: large Datasets (more than a few GB) and particles distributed over the entire domain**
5565

@@ -72,6 +82,8 @@ fieldset.to_windowed_arrays()
7282

7383
**Best for: large Datasets (more than a few GB) and small ParticleSets (less than a few hundred particles)**
7484

85+
_Uses Parcels Backend: Dask_
86+
7587
If your Dataset is so large that it doesn't fit into memory, and you have very few particles, you can use Dask to perform the interpolation operations. In this case, you don't have to do any special setup, as Parcels will automatically use Dask if the `xarray.Dataset` is a Dask array.
7688

7789
### Advantages and disadvantages

0 commit comments

Comments
 (0)