Skip to content

Commit 6a14807

Browse files
Merge pull request #1494 from OceanParcels/output_chunk_explanation_tutorial
Expanding explanation of output chunks parameter
2 parents 2d17966 + a6fe0a5 commit 6a14807

1 file changed

Lines changed: 16 additions & 1 deletion

File tree

docs/examples/tutorial_parcels_structure.ipynb

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -395,7 +395,22 @@
395395
"cell_type": "markdown",
396396
"metadata": {},
397397
"source": [
398-
"Note the use of the `chunks` argument in the `pset.ParticleFile()` above. This controls the 'chunking' of the output file, which is a way to optimize the writing of the output file. See also [the advanced output in zarr format tutorial](https://docs.oceanparcels.org/en/latest/examples/documentation_advanced_zarr.html) for more information on this. It is worth to optimise this parameter in your runs, as it can significantly speed up the writing of the output file and thus the runtime of `pset.execution()`."
398+
"<div class=\"alert alert-info\">\n",
399+
"\n",
400+
"### A note on output chunking\n",
401+
"\n",
402+
"Note the use of the `chunks` argument in the `pset.ParticleFile()` above. This controls the 'chunking' of the output file, which is a way to optimize the writing of the output file. The default chunking for the output in Parcels is `(number of particles in initial particleset, 1)`. \n",
403+
"Note that this default may not be very efficient if \n",
404+
"1. you use `repeatdt` to release a relatively small number of particles _many_ times during the simulation and/or\n",
405+
"2. you expect to output _a lot of timesteps_ (e.g. more than 1000).\n",
406+
"\n",
407+
"In the first case, it is best to increase the first argument of `chunks` to 10 to 100 times the size of your initial particleset. In the second case, it is best to increase the second argument of `chunks` to 10 to 1000, depending a bit on the size of your initial particleset.\n",
408+
"\n",
409+
"In either case, it will generally be much more efficient if `chunks[0]*chunks[1]` is (much) greater than several thousand.\n",
410+
"\n",
411+
"See also [the advanced output in zarr format tutorial](https://docs.oceanparcels.org/en/latest/examples/documentation_advanced_zarr.html) for more information on this. The details will depend on the nature of the filesystem the data is being written to, so it is worth to optimise this parameter in your runs, as it can significantly speed up the writing of the output file and thus the runtime of `pset.execution()`.\n",
412+
"\n",
413+
"</div>"
399414
]
400415
},
401416
{

0 commit comments

Comments
 (0)