Add CMIP7 QC — per-variable rules for all 293 ESM1-6 variables + batch report integration (#456)

* Add CMIP7 output QC docs and CLI usage

- add CMIP7 QC validator module and packaged tas range rules

- run QC automatically on CMORised CMIP7 outputs after write/repack

- add moppy-qc CLI entrypoint and CLI tests

- document notebook and CLI workflows in Sphinx docs

* Add per-variable CMIP7 QC rules and batch report integration

- Generate explicit QC rules for all 293 ACCESS-ESM1-6 mapped variables
  Each variable now has units, default min/max, and experiment-specific
  overrides (historical/piControl/ssp*) derived from mapping definitions.
  No more falling back to unit envelopes at runtime — everything is explicit.

- Remove redundant unit_envelopes section from cmip7_ranges.yml
  They were only ever used to seed the per-variable generation, so keeping
  them in the YAML was just noise.

- Add QC section to batch report (moppy_batch_report.json)
  When the batch run finishes, the report now includes a qc block with
  pass/fail counts and per-file failure details (observed vs allowed range).
  Can be disabled with MOPPY_SKIP_QC=1 or --skip-qc / skip_qc=True.

- Add validate_cmip7_output_detailed() for non-raising validation
  Returns a ValidationResult dataclass so batch collection can gather
  failures without stopping on the first bad file.

- Update docs to reflect explicit per-variable setup and batch QC

* Tune QC limits for tasmin and tasmax

tasmax (monthly max temperature): ceiling set 5-10 K above tas since we
expect warmer peaks — 340 K historical, 335 K piControl, 345 K ssp*

tasmin (monthly min temperature): floor dropped 5 K below tas to allow
colder night-time minimums — 175 K floor, 325/320/330 K ceiling per
experiment

* Fix variable selection when file contains *_bnds variables

CMORised files can have lat_bnds/lon_bnds/time_bnds alongside the main
data variable. The previous fallback only triggered when there was exactly
one data_var, so files like tasmax with 4 vars (tasmax + 3 bounds) raised
an error instead of selecting tasmax.

Fix: filter out *_bnds variables before falling back to single-variable
selection. Still raises if multiple non-bounds variables remain.

* Add a blank line before the build_batch_report function for improved readability

* Update rangess

* pre-commit

* Fix test_write_repacks_cmip7_output failing after QC integration

That test is about verifying the repack subprocess is called — it
was written before QC was wired into the write path. The test fixture
dataset doesn't have units on the data variable, so QC was raising.
Patch out validate_cmip7_output since QC is already tested separately.

* Restore CMIP7 tas piControl QC ceiling

The widened tas range made the piControl regression tests pass when they
should fail at 326.5 K. Put the narrower tas limits back in place so the
existing QC behavior stays intact.

* Add comprehensive test coverage for CMIP7 QC validation

Added 15 new unit tests:

QC Validation Tests (5 new):
- test_validate_cmip7_output_requires_variable_id: Validates error handling for missing variable_id
- test_validate_cmip7_output_requires_experiment_id: Validates error handling for missing experiment_id
- test_validate_cmip7_output_detects_all_missing_values: Validates detection of all-NaN data
- test_validate_cmip7_output_detects_infinity_values: Validates detection of infinity values
- test_validate_cmip7_output_experiment_pattern_matching: Validates wildcard pattern matching for experiments

Batch Report QC Integration Tests (10 new):
- test_run_qc_on_output_folder_with_all_passing_files: QC passes with valid files
- test_run_qc_on_output_folder_with_failing_files: QC detects invalid files
- test_run_qc_on_output_folder_with_mixed_pass_fail: QC tallies mixed results
- test_run_qc_on_output_folder_respects_moppy_skip_qc_env_var: MOPPY_SKIP_QC disables QC
- test_run_qc_on_output_folder_with_no_nc_files: QC handles empty folders
- test_run_qc_on_output_folder_with_nested_files: QC finds files in subdirectories
- test_build_batch_report_includes_qc_section_by_default: QC section included by default
- test_build_batch_report_omits_qc_section_when_skip_qc_true: skip_qc parameter works
- test_write_batch_report_passes_skip_qc_to_build: write_batch_report passes skip_qc
- test_run_qc_on_output_folder_includes_detailed_failure_info: Failure details properly included

All 39 tests pass. Coverage improved from 52.2% to estimated 80%+ for changed code.

* Refactor test cases in batch report and CMIP7 QC to remove unnecessary blank linespre-commit

* Enhance time bounds calculation to support single monthly midpoint labels

* Add tests for CLI skip QC functionality and enhance validation for CMIP7 output

* Refactor QC validation to remove positive sign enforcement and update related tests

* Add functions to handle missing-value sentinels and update QC validation

* Add unit tests for standardizing missing values in vocabulary processor

* Add range validation function and corresponding unit test for tiny negative noise

* Update validation function docstrings for clarity on ACCESS mapped variables

* Add unit tests for range resolution and experiment selection in CMIP7 QC

* Enhance CMORiser and vocabulary processor: add support for ACCESS-ESM1-6 and improve nominal resolution handling

* Switch on validation for Ocean variables

* Add unit tests for unit equivalence checks and implement _units_match function

* Increase salinity range in QC

* Enhance DatasetChunker: unify inconsistent chunks before rechunking and add unit test for this behavior

* Clean range file

* update tests

Author: rbeucher

docs/source/getting_started.rst

@@ -243,6 +243,29 @@ After writing the file, we recommend validating it using [PrePARE](https://githu
 
    cmoriser.write()
 
+Running output QC checks
+------------------------
+
+ACCESS-MOPPy includes CMIP7 output QC checks (currently including physical
+range checks for ``tas``). You can run QC from notebooks or the command line.
+
+Notebook/API usage:
+
+.. code-block:: python
+
+   from access_moppy.qc import validate_cmip7_output
+
+   output_file = "/path/to/CMIP7/output.nc"
+   validate_cmip7_output(output_file)
+
+CLI usage:
+
+.. code-block:: bash
+
+   moppy-qc /path/to/output.nc
+
+See :doc:`qc_validation` for complete examples and rule configuration details.
+
 CMIP7 Support with Full Branded Names
 ======================================
 

docs/source/index.rst

@@ -55,6 +55,7 @@ While retaining the core concepts of "custom" and "cmip" modes, ACCESS-MOPPy uni
    esmvaltool_integration
    CMORise_ILAMB_workflow
    mapping_reference
+   qc_validation
    compliance_testing
    testing_cmorisation
 ----

docs/source/mapping_reference.rst

@@ -129,7 +129,7 @@ required fields:
      - Expected physical units of the CMIP output variable (e.g. ``"W m-2"``, ``"kg m-2 s-1"``).
    * - ``positive``
      - Yes
-     - Sign convention: ``"up"``, ``"down"``, or ``null`` if not applicable.
+     - Sign convention metadata: ``"up"``, ``"down"``, or ``null`` if not applicable.
    * - ``model_variables``
      - Yes
      - List of raw model variable names that must be loaded from the input files.

docs/source/qc_validation.rst

@@ -0,0 +1,164 @@
+CMIP7 QC Validation
+===================
+
+This page describes how to run ACCESS-MOPPy output quality-control checks on
+CMORised files.
+
+Scope
+-----
+
+- QC is run on the *CMORised output file*, not the raw model input.
+- For ``source_id=ACCESS-ESM1-6``, QC covers all variables present in
+  ``ACCESS-ESM1-6_mappings.json`` with generic checks (non-missing values,
+  finite values, and units checks where defined).
+- Physical ranges for all 293 ACCESS-ESM1-6 mapped variables are defined
+  explicitly in the QC configuration with defaults and experiment-specific
+  overrides (historical, piControl, ssp*).
+- Each variable's physical range is derived from its definition (mapping units)
+  and stored as a per-variable rule entry.
+- Rules are loaded from: ``access_moppy/resources/qc/cmip7_ranges.yml``.
+
+Running QC in Notebooks
+-----------------------
+
+Use the Python API to validate a CMORised file after ``cmoriser.write()``.
+
+.. code-block:: python
+
+   from access_moppy.qc import validate_cmip7_output
+
+   # Write CMORised output first
+   cmoriser.run()
+   cmoriser.write()
+
+   # Validate the written file
+   output_file = "/path/to/CMIP7/output.nc"
+   validate_cmip7_output(output_file)
+
+If a check fails, ``validate_cmip7_output`` raises ``ValueError`` with details
+about the variable, experiment, observed range, and allowed range.
+
+Running QC from the CLI
+-----------------------
+
+ACCESS-MOPPy provides a CLI command:
+
+.. code-block:: bash
+
+   moppy-qc /path/to/file1.nc /path/to/file2.nc
+
+Exit status:
+
+- ``0``: all files passed
+- ``1``: one or more files failed
+
+Example output:
+
+.. code-block:: text
+
+   PASS /path/to/file1.nc
+   FAIL /path/to/file2.nc: CMIP7 QC failed for tas in experiment piControl using rule piControl: observed range 182.000..329.400 K is outside allowed range 180.000..325.000 K.
+
+Automatic QC during CMORisation
+-------------------------------
+
+For CMIP7 runs, ACCESS-MOPPy automatically validates output in the write path
+after writing and repacking the file. In other words, when you call
+``cmoriser.write()`` for CMIP7 output, QC is already executed.
+
+Batch Report QC Summary
+-----------------------
+
+When running a batch CMORisation, the batch report (``moppy_batch_report.json``)
+automatically includes a QC section summarizing validation results for all
+CMORised output files:
+
+.. code-block:: json
+
+   {
+     "qc": {
+       "passed": 42,
+       "failed": 2,
+       "total": 44,
+       "failures": [
+         {
+           "file": "/output/path/tas.nc",
+           "variable_id": "tas",
+           "experiment_id": "piControl",
+           "error": "Observed range 182.000..329.400 K is outside allowed 180.000..325.000 K.",
+           "observed_range": [182.0, 329.4],
+           "allowed_range": [180.0, 325.0],
+           "units": "K"
+         }
+       ]
+     }
+   }
+
+To disable QC collection during batch report generation, use one of:
+
+.. code-block:: bash
+
+   # Environment variable
+   export MOPPY_SKIP_QC=1
+   moppy-batch-report --db cmor_tasks.db
+
+   # CLI flag
+   moppy-batch-report --db cmor_tasks.db --skip-qc
+
+Or programmatically:
+
+.. code-block:: python
+
+   from access_moppy.batch_report import write_batch_report
+   write_batch_report(db_path, skip_qc=True)
+
+Extending rules
+
+---------------
+
+To add experiment-specific thresholds for a variable, or to override ranges
+for newly added variables, edit:
+
+.. code-block:: text
+
+   src/access_moppy/resources/qc/cmip7_ranges.yml
+
+Under the ``variables`` section, each variable has a ``default`` entry and an
+optional ``experiments`` map for experiment-specific min/max values. For example:
+
+.. code-block:: yaml
+
+   variables:
+     tas:
+       units: K
+       default:
+         min: 180.0
+         max: 330.0
+       experiments:
+         historical:
+           min: 180.0
+           max: 330.0
+         piControl:
+           min: 180.0
+           max: 325.0
+
+Rule structure example:
+
+.. code-block:: yaml
+
+   variables:
+     tas:
+       units: K
+       default:
+         min: 180.0
+         max: 330.0
+       experiments:
+         historical:
+           min: 180.0
+           max: 330.0
+         piControl:
+           min: 180.0
+           max: 325.0
+         ssp*:
+           min: 180.0
+           max: 335.0

pyproject.toml

@@ -91,6 +91,7 @@ moppy-cmorise = "access_moppy.batch_cmoriser:main"
 moppy-dashboard = "access_moppy.dashboard.cmor_dashboard:main"
 moppy-tui = "access_moppy.dashboard.cli_dashboard:main"
 moppy-batch-report = "access_moppy.batch_report:main"
+moppy-qc = "access_moppy.qc.cmip7:main"
 moppy-example-config = "access_moppy.examples.show_config:main"
 moppy-calc-ab-coeffts = "access_moppy.legacy_utilities.calc_hybrid_height_coeffs:main"
 moppy-esmval-prepare = "access_moppy.esmval.cli_commands:main_prepare"

src/access_moppy/base.py

@@ -14,6 +14,7 @@
 import xarray as xr
 from cftime import date2num
 
+from access_moppy.qc import validate_cmip7_output
 from access_moppy.utilities import (
     FrequencyMismatchError,
     IncompatibleFrequencyError,
@@ -111,9 +112,23 @@ def rechunk_dataset(self, ds: xr.Dataset) -> xr.Dataset:
         Returns:
             Rechunked xarray Dataset
         """
-        if not hasattr(ds, "chunks") or not any(
-            ds.chunks.values() if ds.chunks else []
-        ):
+        try:
+            ds_chunks = ds.chunks
+        except AttributeError:
+            logger.debug("Dataset is not chunked, skipping rechunking")
+            return ds
+        except ValueError as err:
+            if "inconsistent chunks along dimension" not in str(err):
+                raise
+
+            logger.debug(
+                "Dataset has inconsistent chunking across variables; "
+                "calling unify_chunks() before rechunking"
+            )
+            ds = ds.unify_chunks()
+            ds_chunks = ds.chunks
+
+        if not any(ds_chunks.values() if ds_chunks else []):
             logger.debug("Dataset is not chunked, skipping rechunking")
             return ds
 
@@ -1253,6 +1268,9 @@ def estimate_data_size(ds):
 
         self._repack_cmip7_output(path)
 
+        if getattr(self.vocab, "mip_era", None) == "CMIP7":
+            validate_cmip7_output(path)
+
         logger.info("CMORised output written to %s", path)
         logger.debug("Optimized layout: metadata -> data chunks")
         if self.enable_compression: