@@ -133,6 +133,302 @@ Iris' optimisation all together, and will take its chunksizes from Dask's behavi
133133 (70, 37, 49)
134134
135135
136+ Character and String datatypes
137+ ------------------------------
138+ Text can be present in NetCDF in a variety of ways
139+ (see : :ref: `cf_strings ` for details).
140+
141+ The main aspect to be explained here is the storage of bulk text data in *variables *.
142+
143+ String data in Iris
144+ ^^^^^^^^^^^^^^^^^^^
145+ Iris objects can store strings in their data arrays, such as a cube ``.data `` or
146+ coordinate ``.points ``.
147+
148+ These are always stored as arrays of numpy dtype "U<xx>", where <xx> is a maximum
149+ string width (either numpy or dask: see :ref: `real_and_lazy_arrays `).
150+
151+ This data is currently **only ** read and written to NetCDF files as
152+ ``char `` type variables (i.e. byte arrays).
153+
154+ .. note ::
155+
156+ In Iris, the NetCDF ``string `` datatype is **not supported at present **, though this
157+ is planned for future releases.
158+ See : `issue #7092 <https://github.com/SciTools/iris/issues/7092 >`_.
159+ See the following section `Variable-length datatypes `_
160+ for an interim solution enabling you at least to *load * variable-length string data.
161+
162+ Encodings
163+ ~~~~~~~~~
164+ String support is fairly simple when strings contain only ASCII characters.
165+ When strings may include non-ascii characters, this requires a specific encoding to be
166+ adopted when translating to and from bytes, and rules for determining what the encoding
167+ is or was.
168+
169+ In some cases a definite record of the byte encoding is needed (though often a default
170+ can be assumed) : An encoding name can appear in the ``_Encoding `` attribute of a file
171+ variable, and likewise as an attribute of the corresponding Iris component object
172+ (e.g. cube or coordinate) : This is loaded and saved as a normal attribute without
173+ modification, but it can also control both loading and saving behaviour.
174+
175+ Iris supports only certain specific encodings :
176+
177+ * "ascii"
178+ * "utf8"
179+ * "utf16"
180+ * "utf32"
181+
182+ (Though, common aliases are also allowed, those recognised by the Python ``codecs ``
183+ module).
184+
185+ When loading
186+ ~~~~~~~~~~~~
187+ If there is a valid ``_Encoding `` attribute this is used to decode the
188+ data, otherwise a default encoding of "utf8" is applied: This works transparently when
189+ only ascii characters are present, and also allows the ``_Encoding `` attribute to be
190+ omitted as long as utf8 was used to write the data.
191+
192+ An invalid or unsupported encoding name will be ignored, with a warning, but the
193+ attribute will still be added to the Iris component object.
194+
195+ When saving
196+ ~~~~~~~~~~~
197+ Any string data with only ascii characters does not require an ``_Encoding `` attribute.
198+
199+ However if there are any non-ascii characters, and no ``_Encoding ``
200+ attribute, then an error will be raised.
201+ This can be fixed by adding a suitable ``_Encoding `` attribute, for example:
202+ ``cube.attributes["_Encoding"] = "utf8" ``.
203+
204+ An invalid or unsupported encoding name will be ignored, with a warning, but the
205+ attribute will still be stored to the file.
206+
207+ So effectively, the **default encoding is 'utf8' for load and 'ascii' for save **.
208+
209+ String widths and string dimensions
210+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
211+ For each supported encoding, Iris defines a specific function relating the string
212+ dimension length in a NetCDF file (i.e. the "maximum byte width"), to the maximum
213+ number of characters in the array dtype, aka string width
214+ (i.e. the "<xx>" in the dtype "U<xx>").
215+
216+ On write, string dimensions are created with the **minimum number of bytes ** which would
217+ be needed to store ascii-only data of the given width in the given encoding.
218+
219+ These are:
220+
221+ * ascii : n-bytes = n-characters
222+ * utf8 : n-bytes = n-characters
223+ * utf16 : n-bytes = 2 * (n-characters + 1)
224+ * utf32 : n-bytes = 4 * (n-characters + 1)
225+
226+ For 'ascii' and 'utf32' this character-to-byte relationship is simple + fixed.
227+
228+ For 'utf8' and 'utf16', however, the number of encoded bytes depends on the actual
229+ characters present **and can exceed the numbers given above **.
230+
231+ String widths in Saving
232+ #######################
233+ If any string in an actual data array encodes to *more * bytes than the above-calculated
234+ string dimension, when written, then Iris will raise an
235+ :class: `iris.exceptions.TranslationError `. In this case, the user should **explicitly
236+ specify ** a longer string dimension, by converting the data to a longer "U<xx>" dtype :
237+ for example, ``cube.data = cube.core_data().astype("U20") ``.
238+
239+ For example:
240+
241+ * "U12" data with encoding of "utf8", "ascii", or none, will be written with a string
242+ dimension of 12 bytes.
243+
244+ * "U7" data with an encoding of "utf16" will be written with a string dimension of
245+ 16 bytes.
246+
247+ .. warning ::
248+
249+ When processing string arrays, Numpy does not routinely preserve the "<xx>" width part
250+ of "U<xx>" type data : instead, some operations will reduce it to the maximum width
251+ occurring. So in these cases also, it may be necessary to explicitly re-assert the
252+ desired string width before saving -- use ``.astype() ``, as above.
253+
254+ String widths in Loading
255+ ########################
256+ On reading, the returned data has a '"U<xx>"' dtype of which the <xx> string width is
257+ determined by **inverting the above relations **.
258+
259+ For example:
260+
261+ * A string dimension of 9 with an encoding of "utf8", "ascii", or none, will read in
262+ as a string array of dtype "U9".
263+
264+ * A string dimension of 24 with an encoding of "utf32" will read in as a
265+ string array of dtype "U5".
266+
267+ The actual maximum number of characters in the data cannot exceed this dtype width,
268+ since the maximum possible string length is achieved when all characters are plain
269+ ascii characters -- i.e. the bytes contain no multi-byte sequences for
270+ extended characters.
271+
272+ The dtype width created by reading will always round-trip correctly, i.e. the dimension
273+ length will be unchanged if data is read and then written back.
274+
275+ Background: NetCDF strings in Iris' dependencies
276+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
277+ The relevant supporting code libraries and standards provide various facilities for
278+ translating between bytes and Python/numpy strings, but not all possibilities are
279+ supported. The facilities and conventions for this have changed over time, and
280+ obsolete methods persist in archive datasets, which must therefore be taken into
281+ consideration.
282+
283+ The above documentation explains how Iris handles the different cases, and this section
284+ details relevant aspects of its supporting projects, which in practice affect its design.
285+ These are:
286+
287+ * the NetCDF file format;
288+ * the CF conventions;
289+ * the ``numpy `` Python module; and
290+ * the ``netCDF4 `` Python module.
291+
292+
293+ .. _cf_strings :
294+
295+ String data in NetCDF
296+ ~~~~~~~~~~~~~~~~~~~~~
297+ In the NetCDF v4 implementation, there are three specific areas where the datatype and
298+ storage characteristics of character data are relevant:
299+
300+ * **The names of file components (variables, dimensions, and attributes) : ** are
301+ natively unicode-capable strings of arbitrary (variable) length.
302+
303+ * **Attributes with string content : ** are likewise "natively" unicode. However, the
304+ actual storage datatype of the attribute may vary, being either ``char `` or ``string ``.
305+
306+ * **The content of variables : ** can be either ``char `` or ``string ``.
307+
308+ * ``string `` type variables contain a variable-length unicode string at each array element.
309+
310+ * ``char `` type variables contain one-byte characters, and generally have a fixed-length
311+ "string dimension". If they contain *only * ascii character values, this is
312+ uncomplicated, but they may also be used to contain non-ascii data (i.e.
313+ including unicode characters). There is no universally defined agreement for
314+ how to indicate that bytes are encoded non-ascii data, but many older datasets
315+ have used a variable attribute ``_Encoding `` indicating the encoding name.
316+
317+ .. note ::
318+
319+ Nearly everything here is written assuming NetCDF version 4 files, which is the newer
320+ NetCDF storage format based on HDF5. The older NetCDF3 format did not provide the
321+ ``string `` datatype, or support unicode in names and attributes.
322+
323+ The NetCDF documentation does also briefly mention that an ``_Encoding `` attribute may be
324+ used to represent non-ascii strings, but only to state that it is "reserved for future use",
325+ and its valid values and effects are not explicitly defined.
326+ See : `here in the NetCDF v3 description <https://docs.unidata.ucar.edu/n ug/current/file_format_specifications.html >`_
327+ : "The variable attribute '_Encoding' is reserved ...".
328+
329+ However, it is also notable that the standard ``ncgen `` and ``ncdump `` tools *do *
330+ correctly interpret an ``_Encoding `` attribute in most cases, despite this not being an
331+ "official" solution.
332+
333+
334+ String data in the NetCDF CF Conventions
335+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
336+ The `CF Conventions <https://cfconventions.org/ >`_ define a subset of
337+ "allowed" datatypes, and various types of data elements represented by variables
338+ -- such as data variables, auxiliary coordinates, cell methods, etc.
339+
340+ CF currently supports the use of either NetCDF ``string `` or ``char `` type arrays for
341+ **any ** variables.
342+ However, *historically *, CF had more limited support, and also "unofficial conventions"
343+ have been used for string data encoded as bytes, which may be encountered
344+ in older datasets, as follows ...
345+
346+ Prior to v1.8
347+ #############
348+ CF required to use ``char `` type only, and provided
349+ **no official means ** of representing non-ascii data.
350+
351+ Since v1.8
352+ ##########
353+ CF has allowed the use of ``string `` data in all variables.
354+ However, up to v1.12 there was still no official way of encoding non-ascii data in
355+ ``char `` arrays.
356+
357+ Since v1.12
358+ ###########
359+ CF now mandates a *default * assumption of utf-8 encoding to store
360+ non-ascii data in ``char `` form. It does also note that some data in the past has used an
361+ ``_Encoding `` attribute -- though this was never an official CF usage.
362+
363+ Characteristics of CF string storage
364+ ####################################
365+ Where strings are stored as ``char `` datatype, which is the more common traditional approach,
366+ the array must have a "string dimension", which is a normal file dimension. Thus, these
367+ strings always have a *fixed byte width *. However, that is not the same as a fixed
368+ *string * width, since in most encodings non-ascii characters require more bytes to
369+ store.
370+
371+ CF states that a string dimension is **always the last dimension of the array **.
372+
373+ Although the variable-length ``string `` datatype is now supported in CF, the use of
374+ fixed-width ``char `` arrays is obviously more efficient for storage and access, and it is
375+ still the most common approach in practice.
376+
377+ String data in numpy
378+ ~~~~~~~~~~~~~~~~~~~~
379+ Numpy provides a number of dtypes which may be used to store string data. Relevant here
380+ are the dtype kinds "U" and "S" : these contain elements which read and write as
381+ Python ``str `` or ``bytes `` objects.
382+
383+ See `Data Types for Strings and Bytes <https://numpy.org/doc/2.5/user/basics.types.html#data-types-for-strings-and-bytes >`_.
384+
385+
386+ String data in the netCDF4 Python module
387+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
388+
389+ Attributes with string content
390+ ##############################
391+ These always appear as Python 'str' (i.e. unicode strings).
392+
393+ It is not possible to distinguish or control the ``char `` and ``string `` datatype in the file :
394+ This is hidden from the user by the Python implementation.
395+
396+ Variables of type ``string ``
397+ ############################
398+ Are presented (read and written) as a variable with a ``.dtype `` of ``str ``
399+ -- that is, the actual Python ``str `` class.
400+
401+ N.B. this is **not a valid numpy dtype ** : the corresponding variable ``.datatype `` is
402+ ``netCDF4.VLType ``, an internal class used to represent variable-length strings.
403+
404+ The variable array has a numpy dtype of "O" -- i.e. "Python objects"", and its individual
405+ elements are Python ``str `` objects.
406+
407+ Variables of type ``char ``
408+ ##########################
409+ Are presented (read and written) as a variable of dtype "S1".
410+ That is, each element is a single byte, which reads as a Python "bytes" object.
411+
412+ Any non-blank character reads as a length-1 byte string, but a blank character
413+ (zero byte) reads as a zero-length ``b'' ``. A blank can be *written * as either b'' or
414+ b'\0 0'.
415+
416+ .. note ::
417+
418+ The netCDF4 package can also automatically translate byte arrays into string
419+ arrays of dtype "U<xx>" on load, if the variable has an ``_Encoding `` attribute.
420+ See in netCDF4 python documentation :
421+ `Dealing with strings <https://unidata.github.io/netcdf4-python/#dealing-with-strings >`_.
422+ **However, ** Iris turns this feature *off *, in order to implement its own
423+ wider-ranging encoding support (as described above).
424+
425+ .. note ::
426+
427+ The netCDF4 package does not allow variables of 'S<xx>' dtype **other than ** "S1".
428+ If you try to create one, it treats it as the equivalent "U" type, so it has
429+ the variable-length NetCDF ``string `` datatype, as detailed above.
430+
431+
136432Variable-length datatypes
137433-------------------------
138434
0 commit comments