|
| 1 | +{ |
| 2 | + "cells": [ |
| 3 | + { |
| 4 | + "cell_type": "markdown", |
| 5 | + "id": "0", |
| 6 | + "metadata": {}, |
| 7 | + "source": [ |
| 8 | + "# 🎓 Writing output in a Kernel" |
| 9 | + ] |
| 10 | + }, |
| 11 | + { |
| 12 | + "cell_type": "markdown", |
| 13 | + "id": "1", |
| 14 | + "metadata": {}, |
| 15 | + "source": [ |
| 16 | + "The typical way to write particle data to a file is by specifying the `output_file` argument in a `pset.execute()` call. This will write to a file on a regular interval, specified in the `outputdt` argument of the `ParticleFile`. \n", |
| 17 | + "\n", |
| 18 | + "However, sometimes you may want more control of _when_ to write particle data to a file. For example, you may want to write when a particle is deleted, or only when it is in a certain region or when other conditions are met. This can be especially useful if you want to do connectivity studies and don't care about having the full trajectory of each particle available. \n", |
| 19 | + "\n", |
| 20 | + "For these cases, you can use the `pfile.write()` method anywhere in a `Kernel`. You can write to the same file as the one specified in `output_file` or to a different file. \n", |
| 21 | + "\n", |
| 22 | + "This short tutorial will show you how to use `pfile.write()` in a kernel. We will use the same dataset and particle set as in the [output tutorial ](../../getting_started/tutorial_output.ipynb)." |
| 23 | + ] |
| 24 | + }, |
| 25 | + { |
| 26 | + "cell_type": "markdown", |
| 27 | + "id": "2", |
| 28 | + "metadata": {}, |
| 29 | + "source": [ |
| 30 | + "\n", |
| 31 | + "```{warning}\n", |
| 32 | + "Writing output in a Kernel is an experimental feature and may change in the future.\n", |
| 33 | + "```" |
| 34 | + ] |
| 35 | + }, |
| 36 | + { |
| 37 | + "cell_type": "code", |
| 38 | + "execution_count": null, |
| 39 | + "id": "3", |
| 40 | + "metadata": {}, |
| 41 | + "outputs": [], |
| 42 | + "source": [ |
| 43 | + "import numpy as np\n", |
| 44 | + "\n", |
| 45 | + "import parcels\n", |
| 46 | + "import parcels.tutorial" |
| 47 | + ] |
| 48 | + }, |
| 49 | + { |
| 50 | + "cell_type": "markdown", |
| 51 | + "id": "4", |
| 52 | + "metadata": {}, |
| 53 | + "source": [ |
| 54 | + "We start by defining two `ParticleFile` objects: one for writing at regular intervals and one for writing in the kernel. " |
| 55 | + ] |
| 56 | + }, |
| 57 | + { |
| 58 | + "cell_type": "code", |
| 59 | + "execution_count": null, |
| 60 | + "id": "5", |
| 61 | + "metadata": {}, |
| 62 | + "outputs": [], |
| 63 | + "source": [ |
| 64 | + "# The file that will write at regular intervals (every 2 hours in this case)\n", |
| 65 | + "output_file_regular = parcels.ParticleFile(\n", |
| 66 | + " \"output_regular.parquet\",\n", |
| 67 | + " outputdt=np.timedelta64(2, \"h\"),\n", |
| 68 | + ")\n", |
| 69 | + "\n", |
| 70 | + "# The file that will write when the `write_kernel` is executed.\n", |
| 71 | + "output_file_kernel = parcels.ParticleFile(\n", |
| 72 | + " \"output_kernel.parquet\",\n", |
| 73 | + " outputdt=np.timedelta64(\n", |
| 74 | + " 2, \"h\"\n", |
| 75 | + " ), # TODO: this argument is not relevant for this file, since we will write to it manually in the kernel, but it still needs to be specified when creating the ParticleFile object\n", |
| 76 | + ")" |
| 77 | + ] |
| 78 | + }, |
| 79 | + { |
| 80 | + "cell_type": "markdown", |
| 81 | + "id": "6", |
| 82 | + "metadata": {}, |
| 83 | + "source": [ |
| 84 | + "Now we define a kernel that writes to the file when executed. In this example, we write all particles to the file, but you can also choose to write only a subset of the particles by using indexing (e.g., `particles[0]` for the first particle). " |
| 85 | + ] |
| 86 | + }, |
| 87 | + { |
| 88 | + "cell_type": "code", |
| 89 | + "execution_count": null, |
| 90 | + "id": "7", |
| 91 | + "metadata": {}, |
| 92 | + "outputs": [], |
| 93 | + "source": [ |
| 94 | + "def write_kernel(particles, fieldset):\n", |
| 95 | + " output_file_kernel.write(\n", |
| 96 | + " particles,\n", |
| 97 | + " particles.time, # Note that here we use the time of the particles, which may not have started yet.\n", |
| 98 | + " fieldset=fieldset, # Note that we need to specify the fieldset here as well, since the ParticleSetView does not have a reference to the fieldset.\n", |
| 99 | + " )" |
| 100 | + ] |
| 101 | + }, |
| 102 | + { |
| 103 | + "cell_type": "markdown", |
| 104 | + "id": "8", |
| 105 | + "metadata": {}, |
| 106 | + "source": [ |
| 107 | + "Now we run the simulation with both the advection kernel and the write kernel. Note that we specify the `output_file` argument in the `execute()` call, which will write to a file at regular intervals, but we can still use the `write_kernel` to write to a file at specific times." |
| 108 | + ] |
| 109 | + }, |
| 110 | + { |
| 111 | + "cell_type": "code", |
| 112 | + "execution_count": null, |
| 113 | + "id": "9", |
| 114 | + "metadata": {}, |
| 115 | + "outputs": [], |
| 116 | + "source": [ |
| 117 | + "# Load the CopernicusMarine data in the Agulhas region from the example_datasets\n", |
| 118 | + "ds_fields = parcels.tutorial.open_dataset(\n", |
| 119 | + " \"CopernicusMarine_data_for_Argo_tutorial/data\"\n", |
| 120 | + ")\n", |
| 121 | + "ds_fields.load() # load the dataset into memory\n", |
| 122 | + "\n", |
| 123 | + "# Convert to SGRID-compliant dataset and create FieldSet\n", |
| 124 | + "fields = {\"U\": ds_fields[\"uo\"], \"V\": ds_fields[\"vo\"]}\n", |
| 125 | + "ds_fset = parcels.convert.copernicusmarine_to_sgrid(fields=fields)\n", |
| 126 | + "fieldset = parcels.FieldSet.from_sgrid_conventions(ds_fset)\n", |
| 127 | + "\n", |
| 128 | + "pset = parcels.ParticleSet(fieldset=fieldset, lon=[31, 32], lat=[-32, -31], z=[1, 1])\n", |
| 129 | + "\n", |
| 130 | + "pset.execute(\n", |
| 131 | + " kernels=[\n", |
| 132 | + " parcels.kernels.AdvectionRK2,\n", |
| 133 | + " write_kernel, # the kernel that writes to the file when executed\n", |
| 134 | + " ],\n", |
| 135 | + " runtime=np.timedelta64(4, \"h\"),\n", |
| 136 | + " dt=np.timedelta64(15, \"m\"),\n", |
| 137 | + " output_file=output_file_regular, # the file that writes at regular intervals\n", |
| 138 | + ")" |
| 139 | + ] |
| 140 | + }, |
| 141 | + { |
| 142 | + "cell_type": "markdown", |
| 143 | + "id": "10", |
| 144 | + "metadata": {}, |
| 145 | + "source": [ |
| 146 | + "When we inspect the output files, we can see that the file that writes at regular intervals has 6 rows (2 particles * 3 time steps) at intervals of 2 hours, while the file that writes in the kernel has 32 rows (2 particles * 16 time steps) at intervals of 15 minutes." |
| 147 | + ] |
| 148 | + }, |
| 149 | + { |
| 150 | + "cell_type": "code", |
| 151 | + "execution_count": null, |
| 152 | + "id": "11", |
| 153 | + "metadata": {}, |
| 154 | + "outputs": [], |
| 155 | + "source": [ |
| 156 | + "df_loop = parcels.read_particlefile(output_file_regular.path)\n", |
| 157 | + "assert len(df_loop) == 6 # 2 particles * 3 time steps (0h, 2h, 4h)\n", |
| 158 | + "df_loop" |
| 159 | + ] |
| 160 | + }, |
| 161 | + { |
| 162 | + "cell_type": "code", |
| 163 | + "execution_count": null, |
| 164 | + "id": "12", |
| 165 | + "metadata": {}, |
| 166 | + "outputs": [], |
| 167 | + "source": [ |
| 168 | + "output_file_kernel._writer.close() # close the writer to flush the data to disk\n", |
| 169 | + "df_kernel = parcels.read_particlefile(output_file_kernel.path)\n", |
| 170 | + "assert (\n", |
| 171 | + " len(df_kernel) == 32\n", |
| 172 | + ") # 2 particles * 16 time steps (every 15 minutes for 4 hours)\n", |
| 173 | + "df_kernel" |
| 174 | + ] |
| 175 | + }, |
| 176 | + { |
| 177 | + "cell_type": "markdown", |
| 178 | + "id": "13", |
| 179 | + "metadata": {}, |
| 180 | + "source": [ |
| 181 | + "## Writing on particle deletion\n", |
| 182 | + "\n", |
| 183 | + "We can also write to the same file in both the `output_file` and the kernel. This could be useful if e.g. a particle is deleted in the kernel and you want to write the particle data at the time of deletion and you also want to have the full trajectory of the particle available in the output file." |
| 184 | + ] |
| 185 | + }, |
| 186 | + { |
| 187 | + "cell_type": "code", |
| 188 | + "execution_count": null, |
| 189 | + "id": "14", |
| 190 | + "metadata": {}, |
| 191 | + "outputs": [], |
| 192 | + "source": [ |
| 193 | + "output_file_both = parcels.ParticleFile(\n", |
| 194 | + " \"output_both.parquet\",\n", |
| 195 | + " outputdt=np.timedelta64(2, \"h\"),\n", |
| 196 | + ")\n", |
| 197 | + "\n", |
| 198 | + "\n", |
| 199 | + "def write_kernel_on_delete(particles, fieldset):\n", |
| 200 | + " ptcls = particles[particles.time == 8100] # delete particles at 2h15m\n", |
| 201 | + " ptcls.state = parcels.StatusCode.Delete\n", |
| 202 | + " if len(ptcls.time) > 0:\n", |
| 203 | + " output_file_both.write(\n", |
| 204 | + " ptcls,\n", |
| 205 | + " ptcls.time,\n", |
| 206 | + " fieldset=fieldset,\n", |
| 207 | + " )\n", |
| 208 | + "\n", |
| 209 | + "\n", |
| 210 | + "pset = parcels.ParticleSet(fieldset=fieldset, lon=[31, 32], lat=[-32, -31], z=[1, 1])\n", |
| 211 | + "\n", |
| 212 | + "pset.execute(\n", |
| 213 | + " kernels=[\n", |
| 214 | + " parcels.kernels.AdvectionRK2,\n", |
| 215 | + " write_kernel_on_delete, # the kernel that writes to the file when executed\n", |
| 216 | + " ],\n", |
| 217 | + " runtime=np.timedelta64(4, \"h\"),\n", |
| 218 | + " dt=np.timedelta64(15, \"m\"),\n", |
| 219 | + " output_file=output_file_both, # the file that writes at regular intervals\n", |
| 220 | + ")\n", |
| 221 | + "\n", |
| 222 | + "df = parcels.read_particlefile(output_file_both.path)\n", |
| 223 | + "assert len(df) == 6 # 2 particles * 3 time steps (0h, 2h, 2h15m)\n", |
| 224 | + "df" |
| 225 | + ] |
| 226 | + }, |
| 227 | + { |
| 228 | + "cell_type": "markdown", |
| 229 | + "id": "15", |
| 230 | + "metadata": {}, |
| 231 | + "source": [ |
| 232 | + "As you can see in the output above, each particle is now written three times, at 0h, at 2h, and at 2h15m when the particle is deleted.\n", |
| 233 | + "\n", |
| 234 | + "Of course, if you do not add the `output_file` argument to the `execute()` call, then only the kernel will write to the file and there will be one row per particle in the output file, at the time of deletion. Note that you then need to explicitly close the writer to flush the data to disk(with `output_file_both._writer.close()`), since the `execute()` call will not do this for you if you do not specify an `output_file`." |
| 235 | + ] |
| 236 | + } |
| 237 | + ], |
| 238 | + "metadata": { |
| 239 | + "kernelspec": { |
| 240 | + "display_name": "Parcels:docs (3.14.4)", |
| 241 | + "language": "python", |
| 242 | + "name": "python3" |
| 243 | + }, |
| 244 | + "language_info": { |
| 245 | + "codemirror_mode": { |
| 246 | + "name": "ipython", |
| 247 | + "version": 3 |
| 248 | + }, |
| 249 | + "file_extension": ".py", |
| 250 | + "mimetype": "text/x-python", |
| 251 | + "name": "python", |
| 252 | + "nbconvert_exporter": "python", |
| 253 | + "pygments_lexer": "ipython3", |
| 254 | + "version": "3.14.4" |
| 255 | + } |
| 256 | + }, |
| 257 | + "nbformat": 4, |
| 258 | + "nbformat_minor": 5 |
| 259 | +} |
0 commit comments