Chardata docs#7194
Conversation
![scitools-ci[bot]](https://avatars.githubusercontent.com/in/212324?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## FEATURE_chardata #7194 +/- ##
=================================================
Coverage 90.29% 90.29%
=================================================
Files 92 92
Lines 25171 25171
Branches 4700 4700
=================================================
Hits 22728 22728
Misses 1673 1673
Partials 770 770 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
pp-mo
dismissed
scitools-ci[bot]’s stale review
Changes to pre-commit actually occurred in the target branch 'FEATURE_chardata'.
Only appeared here because I temporarily changed the target to 'main' to get docs building
| encoding support (described below). | ||
|
|
||
|
|
||
| In Iris |
There was a problem hiding this comment.
I think the Iris part of this section should be the first under summary, since I imagine users are most interested in that. The other sections are more to provide context for decisions made in Iris
| cubes and coordinates. | ||
|
|
||
| * the numpy dtype is of the type "U<xx>", where <xx> is a character width. | ||
| * the character width relates to a string *dimension* of the netCDF variable, |
There was a problem hiding this comment.
I'm not following what this bullet point means.
|
|
||
|
|
||
| Character and String datatypes | ||
| ------------------------------ |
There was a problem hiding this comment.
I think an introduction might be warranted. Something like:
In NetCDF, text can be represented in 2 ways:
char: the number of characters in the text is fixedstring: the number of characters in the text can vary
Then text can have different encodings, which affects the number of bytes used to store a character. For example:
ASCIIandUTF-32: the number of bytes are fixedUTF-8andUTF-16: the number of bytes can vary
This section discusses how Iris handles the different cases and how the standards and interpretations have changed.
|
Hi again @pt331 ! |
| determined by **inverting the above relations**. | ||
|
|
||
| The actual maximum number of characters in the data will always be less than or equal to | ||
| this, since the maximum possible character width is for all-ascii data. |
There was a problem hiding this comment.
I assume you meant to say
| this, since the maximum possible character width is for all-ascii data. | |
| this, since the maximum possible string width is 1 byte for all-ascii data. |
There was a problem hiding this comment.
Well, not exactly..
I meant that the longest string you can get is when the bytes are full with all ascii characters.
I'll try + rephrase this more clearly...
| The width created by reading will always round-trip correctly, i.e. the dimension | ||
| length is unchanged if data is read and then written back. | ||
|
|
||
| Summary of Iris string handling |
There was a problem hiding this comment.
I think this summary can be removed now and once you have more concrete examples (mentioned in private), something like "Iris string handling in practice" would be a good replacement.
| See in netCDF4 python documentation : | ||
| `Dealing with strings <https://unidata.github.io/netcdf4-python/#dealing-with-strings>`_. | ||
| **However,** Iris turns this feature *off*, in order to implement its own | ||
| wider-ranging encoding support (described below). |
| adopted when translating to and from bytes, and rules for determining what the encoding | ||
| is or was. | ||
|
|
||
| Background |
There was a problem hiding this comment.
I think the Background can now be moved above "String data in NetCDF", "String data in the NetCDF CF Conventions" and "String data in the netCDF4 Python module" and make them subheadings
There was a problem hiding this comment.
OK that's nice + a simpler way of doing it than creating a separate new page.
Co-authored-by: pt331 <144435193+pt331@users.noreply.github.com>
Co-authored-by: pt331 <144435193+pt331@users.noreply.github.com>
Co-authored-by: pt331 <144435193+pt331@users.noreply.github.com>
pt331
left a comment
pt331
left a comment
There was a problem hiding this comment.
Looks good! I like the idea to link to other sections for further details so that the user can shortcut to them if they feel it's more appropriate for them to know. The only changes left are broken links
| Character and String datatypes | ||
| ------------------------------ | ||
| Text can be present in NetCDF in a variety of ways | ||
| (see : `String data in NetCDF <cf_strings>`_ for details). |
There was a problem hiding this comment.
Link is broken. I think it should be
| (see : `String data in NetCDF <cf_strings>`_ for details). | |
| (see : `String data in NetCDF <cf-strings>`_ for details). |
| coordinate ``.points``. | ||
|
|
||
| These are always stored as arrays of numpy dtype "U<xx>", where <xx> is a maximum | ||
| string width (which could be either `real or lazy <real_and_lazy_data>`_). |
There was a problem hiding this comment.
Link is broken. I think it should be
| string width (which could be either `real or lazy <real_and_lazy_data>`_). | |
| string width (which could be either `real or lazy <real-and-lazy_data>`_). |
2 participants
Closes #6924
In writing this, I have encountered a bit of a doubt about the implementation : #7195
But for the sake of getting this under review, I'm going to treat that as a separate issue :
I believe it can only simplify the account here.