Skip to content

Commit 57ba36d

Browse files
committed
Fill docs placeholders for old netcdf features.
1 parent 5d7ca14 commit 57ba36d

1 file changed

Lines changed: 81 additions & 11 deletions

File tree

docs/src/user_manual/explanation/netcdf_io.rst

Lines changed: 81 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,87 @@ Iris' optimisation all together, and will take its chunksizes from Dask's behavi
133133
(70, 37, 49)
134134

135135

136+
Split Attributes
137+
-----------------
138+
139+
Since Iris does not provide a "dataset" object, it is not possible to represent netcdf
140+
file-level, aka "global", attributes directly. Instead, the attributes of *cubes*,
141+
specifically, are "split" to discriminate between global (i.e. file) and local (i.e.
142+
variable) attributes.
143+
144+
Prior to Iris v3.8, this was done "automatically" according to a specific lists of
145+
known attribute names which would normally *only* appear as local (variable) attributes
146+
(e.g. "valid_min") or *only* as global (file) ones (e.g. "history").
147+
All other names were saved as "global if the same for all cubes, otherwise local".
148+
149+
Since Iris v3.8, but explicit handling has now been added, so that a ``cube.attributes``
150+
is a :class:`~iris.cube.CubeAttrsDict` object which combines global and local attributes
151+
within a single dictionary-like object.
152+
Finer-grained control of attribute saving is thus now possible, by setting the
153+
:data:`iris.Future.save_split_attrs` control to ``True``.
154+
Ideally, anyone who is saving cubes to netcdf files should now set this, and
155+
a warning is raised if it is not set.
156+
157+
.. note::
158+
The default setting remains ``iris.FUTURE.save_split_attrs = False``, purely for
159+
backwards compatibility, but it is recommended to override this for all new code.
160+
161+
The :class:`~iris.cube.CubeAttrsDict` provides a regular dictionary access which mimics
162+
the older single dictionary, while also containing separate lists in its ``.global`` and
163+
``.local`` properties which allow the user to specifically choose the nature of each
164+
attribute.
165+
On saving, each cube's ".attributes.local"s become variable-level attributes and its
166+
".attributes.global"s become file-level attributes -- *except only*, if globals
167+
have different values in multiple cubes, then global attributes may be 'demoted' to
168+
local ones.
169+
170+
Summary
171+
=======
172+
173+
* You should now always set the ``iris.Future.save_split_attrs=True`` unless you have a
174+
specific need for backwards compatibility.
175+
* all Cube attributes are now stored as specifically "global" or "local"
176+
* you can continue to assign and fetch cube attributes as before, but when required you
177+
can also now access ``.global`` and ``.local`` properties specifically.
178+
* practically, all this is only relevant when **saving to netcdf files**
179+
180+
181+
Deferred Saving
182+
----------------
183+
184+
In some cases it is useful for performance reasons to defer the actual writing of bulk
185+
data to a NetCDF file's variables, from after the "iris.save" call to a later time.
186+
187+
You can do this by adding a ``compute=False`` keyword to the ``iris.save`` call. The
188+
call then returns, instead of None, a :class:``dask.delayed`` object, which can be
189+
computed later to complete the writing of variable data to the file.
190+
191+
Where this can be useful is when you are saving multiple cubes whose lazy data is
192+
computed from shared data, for example multiple statistics over the same data.
193+
194+
To do this, you specify ``compute=False`` to multiple :func:``iris.save`` calls, and
195+
collect the resulting :class:``dask.delayed`` objects. You can then complete the saves
196+
in parallel, by supplying a list of the 'delayed' results to the :func:`Dask.compute`
197+
function.
198+
199+
This mechanism allows Dask to calculate the multiple "derived" results while only
200+
fetching the "common" data from which they derive **once**, whereas otherwise the source
201+
data may need to be fetched multiple times. In this respect, the benefit is the same as
202+
the :meth:``iris.cube.CubeList.realise_cubes`` method. However there is an additional
203+
potential benefit in this case, which is that the actual **writing** of data to file can
204+
be performed in parallel. The scope of this, however, depends on the parallel
205+
capabilities of the Python ``netCDF4`` package, the NetCDF C library, the file
206+
system and the operating system : *Typically* you can write data arrays in parallel
207+
to **separate files**, but not to different variables, or variable sections *within a
208+
single file*.
209+
210+
211+
Character and String data
212+
-------------------------
213+
214+
TBC
215+
216+
136217
Variable-length datatypes
137218
-------------------------
138219

@@ -189,17 +270,6 @@ loader so it can be make a more informed decision on lazy loading:
189270
False
190271

191272

192-
Split Attributes
193-
-----------------
194-
195-
TBC
196-
197-
198-
Deferred Saving
199-
----------------
200-
201-
TBC
202-
203273
.. _save_load_dataless:
204274

205275
Dataless Cubes in NetCDF files

0 commit comments

Comments
 (0)