Skip to content

Commit ac8f94a

Browse files
committed
Fill docs placeholders for old netcdf features.
1 parent eaf35e9 commit ac8f94a

1 file changed

Lines changed: 81 additions & 11 deletions

File tree

docs/src/user_manual/explanation/netcdf_io.rst

Lines changed: 81 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,87 @@ Iris' optimisation all together, and will take its chunksizes from Dask's behavi
131131
(70, 37, 49)
132132

133133

134+
Split Attributes
135+
-----------------
136+
137+
Since Iris does not provide a "dataset" object, it is not possible to represent netcdf
138+
file-level, aka "global", attributes directly. Instead, the attributes of *cubes*,
139+
specifically, are "split" to discriminate between global (i.e. file) and local (i.e.
140+
variable) attributes.
141+
142+
Prior to Iris v3.8, this was done "automatically" according to a specific lists of
143+
known attribute names which would normally *only* appear as local (variable) attributes
144+
(e.g. "valid_min") or *only* as global (file) ones (e.g. "history").
145+
All other names were saved as "global if the same for all cubes, otherwise local".
146+
147+
Since Iris v3.8, but explicit handling has now been added, so that a ``cube.attributes``
148+
is a :class:`~iris.cube.CubeAttrsDict` object which combines global and local attributes
149+
within a single dictionary-like object.
150+
Finer-grained control of attribute saving is thus now possible, by setting the
151+
:data:`iris.Future.save_split_attrs` control to ``True``.
152+
Ideally, anyone who is saving cubes to netcdf files should now set this, and
153+
a warning is raised if it is not set.
154+
155+
.. note::
156+
The default setting remains ``iris.FUTURE.save_split_attrs = False``, purely for
157+
backwards compatibility, but it is recommended to override this for all new code.
158+
159+
The :class:`~iris.cube.CubeAttrsDict` provides a regular dictionary access which mimics
160+
the older single dictionary, while also containing separate lists in its ``.global`` and
161+
``.local`` properties which allow the user to specifically choose the nature of each
162+
attribute.
163+
On saving, each cube's ".attributes.local"s become variable-level attributes and its
164+
".attributes.global"s become file-level attributes -- *except only*, if globals
165+
have different values in multiple cubes, then global attributes may be 'demoted' to
166+
local ones.
167+
168+
Summary
169+
=======
170+
171+
* You should now always set the ``iris.Future.save_split_attrs=True`` unless you have a
172+
specific need for backwards compatibility.
173+
* all Cube attributes are now stored as specifically "global" or "local"
174+
* you can continue to assign and fetch cube attributes as before, but when required you
175+
can also now access ``.global`` and ``.local`` properties specifically.
176+
* practically, all this is only relevant when **saving to netcdf files**
177+
178+
179+
Deferred Saving
180+
----------------
181+
182+
In some cases it is useful for performance reasons to defer the actual writing of bulk
183+
data to a NetCDF file's variables, from after the "iris.save" call to a later time.
184+
185+
You can do this by adding a ``compute=False`` keyword to the ``iris.save`` call. The
186+
call then returns, instead of None, a :class:``dask.delayed`` object, which can be
187+
computed later to complete the writing of variable data to the file.
188+
189+
Where this can be useful is when you are saving multiple cubes whose lazy data is
190+
computed from shared data, for example multiple statistics over the same data.
191+
192+
To do this, you specify ``compute=False`` to multiple :func:``iris.save`` calls, and
193+
collect the resulting :class:``dask.delayed`` objects. You can then complete the saves
194+
in parallel, by supplying a list of the 'delayed' results to the :func:`Dask.compute`
195+
function.
196+
197+
This mechanism allows Dask to calculate the multiple "derived" results while only
198+
fetching the "common" data from which they derive **once**, whereas otherwise the source
199+
data may need to be fetched multiple times. In this respect, the benefit is the same as
200+
the :meth:``iris.cube.CubeList.realise_cubes`` method. However there is an additional
201+
potential benefit in this case, which is that the actual **writing** of data to file can
202+
be performed in parallel. The scope of this, however, depends on the parallel
203+
capabilities of the Python ``netCDF4`` package, the NetCDF C library, the file
204+
system and the operating system : *Typically* you can write data arrays in parallel
205+
to **separate files**, but not to different variables, or variable sections *within a
206+
single file*.
207+
208+
209+
Character and String data
210+
-------------------------
211+
212+
TBC
213+
214+
134215
Variable-length datatypes
135216
-------------------------
136217

@@ -187,17 +268,6 @@ loader so it can be make a more informed decision on lazy loading:
187268
False
188269

189270

190-
Split Attributes
191-
-----------------
192-
193-
TBC
194-
195-
196-
Deferred Saving
197-
----------------
198-
199-
TBC
200-
201271
.. _save_load_dataless:
202272

203273
Dataless Cubes in NetCDF files

0 commit comments

Comments
 (0)