Merge pull request #65 from abkfenris/acdd

Initial ACDD Tests

Author: forman

CHANGES.md

@@ -1,11 +1,28 @@
 # XRLint Change History
 
+
 ## Version 0.5.2 (in development)
 
 ### Adjustments and Enhancements
 
 - Core rule 'time-coordinates' now support ms, µs and ns. (#66)
 
+- Implemented an initial set of 
+  [Attribute Conventions Data Discovery (ACCD)](https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery) 
+  rules adapted from the [IOOS Compliance Checker](https://github.com/ioos/compliance-checker/) 
+  library (many thanks to @abkfenris):
+  - Configs for ACDD 1.0, 1.1, and 1.3, and with selectable levels of severity. 
+    The recommended set uses ACDD 1.3 with the highly recomended rules as errors.
+  - Global attribute existance rules.
+  - Checks that ACDD is in the conventions attribute.
+  - Makes sure the date attributes are ISO formatted.
+  - Metadata links are URLs.
+  - The ID attribute should not be blank.
+
+- Load plugins from entry points allowing plugins to be discovered from installed libraries.
+  - Automatically generate rule documentation removing the manual need to run `mkruleref.py`.
+
+
 ## Version 0.5.1 (from 2025-02-21)
 
 - XRLint now also loads default configuration from files named 

README.md

@@ -38,5 +38,7 @@ The following plugins provide XRLint's [inbuilt rules](https://bcdev.github.io/x
   [xcube datasets](https://xcube.readthedocs.io/en/latest/cubespec.html).
   Note, this plugin is fully optional. You must manually configure 
   it to apply its rules. It may be moved into a separate GitHub repo later. 
+- `xrlint.plugins.acdd`: implements rules for [Attribute Conventions Dataset Discovery](https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery).
+  Note, this plugin is fully optional. You must manually configure it to apply its rules.
 
 

docs/rule-ref.md

@@ -199,3 +199,75 @@ Time coordinate and dimension should be called 'time'.
 
 Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-lightning-bolt:
 
+## ACDD Rules
+
+### :material-bug: `1.0-attrs-highly-recommended`
+
+Global attributes that are highly reccomended by ACDD-1.0.
+[More...](https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery)
+
+Contained in:  `acdd_1.0`-:material-lightning-bolt:
+
+### :material-bug: `1.0-attrs-recommended`
+
+Global attributes that are recommended by ACDD-1.0.
+[More...](https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery)
+
+Contained in:  `acdd_1.0`-:material-alert:
+
+### :material-bug: `1.0-attrs-suggested`
+
+Global attributes that are suggested by ACDD-1.0.
+[More...](https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery)
+
+Contained in:  `acdd_1.0`-:material-alert:
+
+### :material-bug: `1.3-attrs-highly-recommended`
+
+Global attributes that are highly recommended by ACDD-1.3.
+[More...](https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3)
+
+Contained in:  `acdd_1.3`-:material-lightning-bolt: `acdd_1.3_strict`-:material-lightning-bolt: `acdd_1.3_strict_reccomended`-:material-lightning-bolt: `acdd_1.3_warn`-:material-alert: `recommended`-:material-lightning-bolt:
+
+### :material-bug: `1.3-attrs-recommended`
+
+Global attributes that are recommended by ACDD-1.3.
+[More...](https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3)
+
+Contained in:  `acdd_1.3`-:material-alert: `acdd_1.3_strict`-:material-lightning-bolt: `acdd_1.3_strict_reccomended`-:material-lightning-bolt: `acdd_1.3_warn`-:material-alert: `recommended`-:material-alert:
+
+### :material-bug: `1.3-attrs-suggested`
+
+Global attributes that are suggested by ACDD 1.3.
+[More...](https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3)
+
+Contained in:  `acdd_1.3`-:material-alert: `acdd_1.3_strict`-:material-lightning-bolt: `acdd_1.3_strict_reccomended`-:material-alert: `acdd_1.3_warn`-:material-alert: `recommended`-:material-alert:
+
+### :material-bug: `1.3-conventions`
+
+The `Conventions` global attribute should include `ACDD-1.3`.
+[More...](https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3)
+
+Contained in:  `acdd_1.3`-:material-lightning-bolt: `acdd_1.3_strict`-:material-lightning-bolt: `acdd_1.3_strict_reccomended`-:material-lightning-bolt: `acdd_1.3_warn`-:material-alert: `recommended`-:material-lightning-bolt:
+
+### :material-bug: `1.3-dates-iso-format`
+
+ACDD date attributes must be in ISO format.
+[More...](https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3)
+
+Contained in:  `acdd_1.3`-:material-alert: `acdd_1.3_strict`-:material-lightning-bolt: `acdd_1.3_strict_reccomended`-:material-alert: `acdd_1.3_warn`-:material-alert: `recommended`-:material-alert:
+
+### :material-bug: `1.3-metadata-link`
+
+The `metadata` attribute should be a URL.
+[More...](https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3)
+
+Contained in:  `acdd_1.3`-:material-alert: `acdd_1.3_strict`-:material-lightning-bolt: `acdd_1.3_strict_reccomended`-:material-alert: `acdd_1.3_warn`-:material-alert: `recommended`-:material-alert:
+
+### :material-bug: `1.3-no-blanks-in-id`
+
+The `id` attribute should not contain blanks.
+[More...](https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3)
+
+Contained in:  `acdd_1.3`-:material-alert: `acdd_1.3_strict`-:material-lightning-bolt: `acdd_1.3_strict_reccomended`-:material-alert: `acdd_1.3_warn`-:material-alert: `recommended`-:material-alert:
+

mkruleref.py

@@ -27,9 +27,11 @@
 def write_rule_ref_page():
     import xrlint.plugins.core
     import xrlint.plugins.xcube
+    import xrlint.plugins.acdd
 
     core = xrlint.plugins.core.export_plugin()
     xcube = xrlint.plugins.xcube.export_plugin()
+    acdd = xrlint.plugins.acdd.export_plugin()
     with open("docs/rule-ref.md", "w") as stream:
         stream.write("# Rule Reference\n\n")
         stream.write(
@@ -41,6 +43,8 @@ def write_rule_ref_page():
         write_plugin_rules(stream, core)
         stream.write("## xcube Rules\n\n")
         write_plugin_rules(stream, xcube)
+        stream.write("## ACDD Rules\n\n")
+        write_plugin_rules(stream, acdd)
 
 
 def write_plugin_rules(stream, plugin: Plugin):

pyproject.toml

@@ -6,7 +6,8 @@ build-backend = "setuptools.build_meta"
 name = "xrlint"
 dynamic = ["version", "readme"]
 authors = [
-  {name = "Norman Fomferra (Brockmann Consult GmbH)"}
+  {name = "Norman Fomferra (Brockmann Consult GmbH)"},
+  {name = "Alex Kerney (GMRI / NERACOOS)", email = "akerney@gmri.org"}
 ]
 description = "A linter for xarray datasets."
 keywords = [
@@ -17,6 +18,7 @@ requires-python = ">=3.10"
 dependencies = [
   "click",
   "fsspec",
+  "isodate>=0.7.2",
   "pyyaml",
   "tabulate",
   "xarray",
@@ -93,3 +95,10 @@ Documentation = "https://bcdev.github.io/xrlint"
 Repository = "https://github.com/bcdev/xrlint"
 Changelog = "https://github.com/bcdev/xrlint/blob/main/CHANGES.md"
 Issues = "https://github.com/bcdev/xrlint/issues"
+
+[tool.uv]
+default-groups = "all"
+
+# [tool.uv.pip]
+# all-extras = true
+

tests/plugins/acdd/rules/test_attributes.py

@@ -0,0 +1,34 @@
+import xarray as xr
+from xrlint.testing import RuleTest, RuleTester
+
+from xrlint.plugins.acdd.rules.attributes import (
+    Attributes_1_3_Highly_Recommended,
+)
+
+valid_1_3_highly_rec_dataset = xr.Dataset(
+    attrs={
+        "title": "This is only a test",
+        "summary": "This is only a test dataset.",
+        "keywords": "test, example, sample",
+        "Conventions": "ACDD-1.3",
+    },
+)
+invalid_1_3_highly_rec_dataset = xr.Dataset()
+
+
+Attributes_1_3_Highly_RecommendedTest = RuleTester.define_test(
+    "1.3-attrs-highly-recommended",
+    Attributes_1_3_Highly_Recommended,
+    valid=[RuleTest(dataset=valid_1_3_highly_rec_dataset)],
+    invalid=[
+        RuleTest(
+            dataset=invalid_1_3_highly_rec_dataset,
+            expected=[
+                "Missing highly recommended attribute 'title'",
+                "Missing highly recommended attribute 'keywords'",
+                "Missing highly recommended attribute 'summary'",
+                "Missing highly recommended attribute 'Conventions'",
+            ],
+        ),
+    ],
+)

tests/plugins/acdd/rules/test_conventions.py

@@ -0,0 +1,35 @@
+import xarray as xr
+from xrlint.testing import RuleTest, RuleTester
+
+from xrlint.plugins.acdd.rules.conventions import Conventions
+
+valid_dataset_0 = xr.Dataset(attrs={"Conventions": "ACDD-1.3"})
+
+invalid_dataset_0 = xr.Dataset()
+invalid_dataset_1 = xr.Dataset(attrs={"Conventions": "ACDD-1.2"})
+invalid_dataset_2 = xr.Dataset(attrs={"Conventions": 1.3})
+
+
+ConventionsTest = RuleTester.define_test(
+    "1.3-conventions",
+    Conventions,
+    valid=[
+        RuleTest(dataset=valid_dataset_0),
+    ],
+    invalid=[
+        RuleTest(
+            dataset=invalid_dataset_0,
+            expected=["Missing attribute 'Conventions'."],
+        ),
+        RuleTest(
+            dataset=invalid_dataset_1,
+            expected=[
+                "Attribute 'Conventions' needs to contain 'ACDD-1.3' in addition to the current: 'ACDD-1.2'",
+            ],
+        ),
+        RuleTest(
+            dataset=invalid_dataset_2,
+            expected=["Invalid attribute 'Conventions': 1.3"],
+        ),
+    ],
+)

tests/plugins/acdd/rules/test_id_blanks.py

@@ -0,0 +1,26 @@
+import xarray as xr
+from xrlint.testing import RuleTest, RuleTester
+
+from xrlint.plugins.acdd.rules.no_id_blanks import NoBlanksInID
+
+valid_dataset_0 = xr.Dataset(attrs={"id": "testing_dataset"})
+
+invalid_dataset_0 = xr.Dataset()
+invalid_dataset_1 = xr.Dataset(attrs={"id": "testing dataset"})
+
+
+IdBlanksTest = RuleTester.define_test(
+    "1.3-no-blanks-in-id",
+    NoBlanksInID,
+    valid=[RuleTest(dataset=valid_dataset_0)],
+    invalid=[
+        RuleTest(
+            dataset=invalid_dataset_0,
+            expected=["Missing attribute 'id'"],
+        ),
+        RuleTest(
+            dataset=invalid_dataset_1,
+            expected=["There should not be blanks in the id field"],
+        ),
+    ],
+)

tests/plugins/acdd/rules/test_iso_dates.py

@@ -0,0 +1,61 @@
+import xarray as xr
+from xrlint.testing import RuleTest, RuleTester
+
+from xrlint.plugins.acdd.rules.iso_dates import IsoDates
+
+valid_dataset_0 = xr.Dataset(attrs={"date_created": "2023-10-05T12:34:56Z"})
+valid_dataset_1 = xr.Dataset(attrs={"date_modified": "2023-10-05"})
+valid_dataset_2 = xr.Dataset(attrs={"date_issued": "2023-10-05T12:34:56+00:00"})
+valid_dataset_3 = xr.Dataset(
+    attrs={"date_metadata_modified": "2023-10-05T12:34:56-05:00"},
+)
+valid_dataset_4 = xr.Dataset()  # No date attributes
+
+
+invalid_dataset_0 = xr.Dataset(attrs={"date_created": "10/05/2023"})
+invalid_dataset_1 = xr.Dataset(attrs={"date_issued": "2023-13-05"})
+invalid_dataset_2 = xr.Dataset(attrs={"date_modified": "2023-10-05 12:34:56"})
+invalid_dataset_3 = xr.Dataset(attrs={"date_metadata_modified": "2023/10/05"})
+invalid_dataset_4 = xr.Dataset(
+    attrs={"date_created": "2023-10-05T25:34:56Z"},
+)  # Invalid hour
+
+IsoDatesTest = RuleTester.define_test(
+    "1.3-dates-iso-format",
+    IsoDates,
+    valid=[
+        RuleTest(dataset=valid_dataset_0),
+        RuleTest(dataset=valid_dataset_1),
+        RuleTest(dataset=valid_dataset_2),
+        RuleTest(dataset=valid_dataset_3),
+        RuleTest(dataset=valid_dataset_4),
+    ],
+    invalid=[
+        RuleTest(
+            dataset=invalid_dataset_0,
+            expected=["Attribute 'date_created' is not in ISO format: '10/05/2023'"],
+        ),
+        RuleTest(
+            dataset=invalid_dataset_1,
+            expected=["Attribute 'date_issued' is not in ISO format: '2023-13-05'"],
+        ),
+        RuleTest(
+            dataset=invalid_dataset_2,
+            expected=[
+                "Attribute 'date_modified' is not in ISO format: '2023-10-05 12:34:56'",
+            ],
+        ),
+        RuleTest(
+            dataset=invalid_dataset_3,
+            expected=[
+                "Attribute 'date_metadata_modified' is not in ISO format: '2023/10/05'",
+            ],
+        ),
+        RuleTest(
+            dataset=invalid_dataset_4,
+            expected=[
+                "Attribute 'date_created' is not in ISO format: '2023-10-05T25:34:56Z'",
+            ],
+        ),
+    ],
+)

tests/plugins/acdd/rules/test_metadata_link.py

@@ -0,0 +1,28 @@
+import xarray as xr
+from xrlint.testing import RuleTest, RuleTester
+
+from xrlint.plugins.acdd.rules.metadata_link import MetadataLink
+
+valid_dataset_0 = xr.Dataset(attrs={"metadata_link": "http://example.com/metadata"})
+valid_dataset_1 = xr.Dataset(attrs={"metadata_link": "https://example.com/metadata"})
+valid_dataset_2 = xr.Dataset()
+
+invalid_dataset_0 = xr.Dataset(attrs={"metadata_link": "example.com/metadata"})
+
+Metadata_LinkTest = RuleTester.define_test(
+    "1.3-metadata-link",
+    MetadataLink,
+    valid=[
+        RuleTest(dataset=valid_dataset_0),
+        RuleTest(dataset=valid_dataset_1),
+        RuleTest(dataset=valid_dataset_2),
+    ],
+    invalid=[
+        RuleTest(
+            dataset=invalid_dataset_0,
+            expected=[
+                "Metadata URL should include http:// or https://: 'example.com/metadata'",
+            ],
+        ),
+    ],
+)