Cloud diagnostics may fail due to wrongly interpreting parametric vertical coordinates

[bouweandela]

There are many (20 out of 50) failures in the cloud-scatterplots-cli-ta diagnostic: https://dashboard.climate-ref.org/diagnostics/esmvaltool/cloud-scatterplots-cli-ta/groups

I suspect several of these are due to ESMValGroup/ESMValCore#3056, even if the error messages vary. Some of these may be data quality issues too.

Example failure: https://dashboard.climate-ref.org/executions/45/logs

The run fail with

esmvalcore.cmor.check.CMORCheckError: There were errors in variable cli:
 There are multiple coordinates with standard_name "air_pressure": ['ps', None]

[bouweandela]

Possibly related, but different failure for FGOALS-f3-L: https://dashboard.climate-ref.org/executions/4032/logs

2026-03-07 13:33:02,394 UTC [1028526] ERROR   Failed to run preprocessor function 'fix_metadata' on the data
[<iris 'Cube' of mass_fraction_of_cloud_ice_in_air / (kg kg-1) (time: 120; atmosphere_hybrid_sigma_pressure_coordinate: 32; latitude: 180; longitude: 288)>,
 <iris 'Cube' of vertical coordinate formula term: a(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 32; -- : 2)>,
 <iris 'Cube' of vertical coordinate formula term: b(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 32; -- : 2)>,
 <iris 'Cube' of Surface Air Pressure / (Pa) (time: 120; latitude: 180; longitude: 288)>]
(and 8 further argument(s) not shown here; refer to the debug log for a full list)
loaded from original input file(s)
[LocalFile('/mnt/data/climate-ref/2026-03-04/scratch/esmvaltool/cloud-scatterplots-cli-ta/ee8de45fb9d17854bd8620c12197f7af9375c1f0/climate_data/CMIP6/CMIP/CAS/FGOALS-f3-L/historical/r1i1p1f1/Amon/cli/gr/v20190927/cli_Amon_FGOALS-f3-L_historical_r1i1p1f1_gr_199001-199912.nc'),
 LocalFile('/mnt/data/climate-ref/2026-03-04/scratch/esmvaltool/cloud-scatterplots-cli-ta/ee8de45fb9d17854bd8620c12197f7af9375c1f0/climate_data/CMIP6/CMIP/CAS/FGOALS-f3-L/historical/r1i1p1f1/Amon/cli/gr/v20190927/cli_Amon_FGOALS-f3-L_historical_r1i1p1f1_gr_200001-200912.nc'),
 LocalFile('/mnt/data/climate-ref/2026-03-04/scratch/esmvaltool/cloud-scatterplots-cli-ta/ee8de45fb9d17854bd8620c12197f7af9375c1f0/climate_data/CMIP6/CMIP/CAS/FGOALS-f3-L/historical/r1i1p1f1/Amon/cli/gr/v20190927/cli_Amon_FGOALS-f3-L_historical_r1i1p1f1_gr_201001-201412.nc')]
with function argument(s)
session = Session(<config>, name=recipe_20260307_133245),
timerange = '1996/2014',
mip = 'Amon',
exp = 'historical',
preprocessor = 'common_grid_levels_cli',
project = 'CMIP6',
activity = 'CMIP',
dataset = 'FGOALS-f3-L',
ensemble = 'r1i1p1f1',
institute = 'CAS',
grid = 'gr',
short_name = 'cli',
variable_group = 'cli',
diagnostic = 'plot_joint_cli_ta_model',
recipe_dataset_index = 0,
alias = 'FGOALS-f3-L',
original_short_name = 'cli',
standard_name = 'mass_fraction_of_cloud_ice_in_air',
long_name = 'Mass Fraction of Cloud Ice',
units = 'kg kg-1',
modeling_realm = ['atmos'],
frequency = 'mon',
version = 'v20190927'
2026-03-07 13:33:06,127 UTC [1025282] INFO    Maximum memory used (estimate): 2.8 GB
2026-03-07 13:33:06,130 UTC [1025282] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2026-03-07 13:33:06,159 UTC [1025282] ERROR   Program terminated abnormally, see stack trace below for more information:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ~~~~^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_task.py", line 1005, in _run_task
    output_files = task.run()
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_task.py", line 301, in run
    self.output_files = self._run(input_files)
                        ~~~~~~~~~^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 859, in _run
    product.apply(step, self.debug)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 675, in apply
    self.cubes,
    ^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 687, in cubes
    self._cubes = [ds.load() for ds in self.datasets]  # type: ignore
                   ~~~~~~~^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/dataset.py", line 805, in load
    cube = self._load()
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/dataset.py", line 887, in _load
    result = preprocess(
        result,
    ...<4 lines>...
        **kwargs,
    )
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 560, in preprocess
    _run_preproc_function(
    ~~~~~~~~~~~~~~~~~~~~~^
        function,
        ^^^^^^^^^
    ...<2 lines>...
        input_files=input_files,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    ),
    ^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 500, in _run_preproc_function
    return function(items, **kwargs)
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/cmor/fix.py", line 188, in fix_metadata
    cube_list = fix.fix_metadata(cube_list)
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/cmor/_fixes/cmip6/fgoals_f3_l.py", line 37, in fix_metadata
    cube_coord = cube.coord(coord)
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/iris/cube.py", line 2413, in coord
    raise iris.exceptions.CoordinateNotFoundError(emsg)
iris.exceptions.CoordinateNotFoundError: "Expected to find exactly 1 'latitude' coordinate, but found none."
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_main.py", line 786, in run
    fire.Fire(ESMValTool())
    ~~~~~~~~~^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ~~~~~~~~~~~~~~~~~~~^
        component,
        ^^^^^^^^^^
    ...<2 lines>...
        treatment='class' if is_class else 'routine',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        target=component.__name__)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_main.py", line 620, in run
    self._run(recipe, session, cli_config_dir)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_main.py", line 675, in _run
    process_recipe(recipe_file=recipe, session=session)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_main.py", line 156, in process_recipe
    recipe.run()
    ~~~~~~~~~~^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_recipe/recipe.py", line 1331, in run
    self.tasks.run(max_parallel_tasks=self.session["max_parallel_tasks"])
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_task.py", line 853, in run
    self._run_parallel(address, max_parallel_tasks)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_task.py", line 963, in _run_parallel
    _copy_results(task, running[task])
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_task.py", line 993, in _copy_results
    task.output_files, task.products = future.get()
                                       ~~~~~~~~~~^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
    ^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_task.py", line 1005, in _run_task
    output_files = task.run()
    ^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/_task.py", line 301, in run
    self.output_files = self._run(input_files)
    ^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 859, in _run
    product.apply(step, self.debug)
    
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 675, in apply
    self.cubes,
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 687, in cubes
    self._cubes = [ds.load() for ds in self.datasets]  # type: ignore
    ^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/dataset.py", line 805, in load
    cube = self._load()
    ^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/dataset.py", line 887, in _load
    result = preprocess(
    ^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 560, in preprocess
    _run_preproc_function(
    ^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/preprocessor/__init__.py", line 500, in _run_preproc_function
    return function(items, **kwargs)
    ^^^^^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/cmor/fix.py", line 188, in fix_metadata
    cube_list = fix.fix_metadata(cube_list)
    ^^^^^^^^^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/esmvalcore/cmor/_fixes/cmip6/fgoals_f3_l.py", line 37, in fix_metadata
    cube_coord = cube.coord(coord)
    ^^^
  File "/mnt/data/climate-ref/2026-03-04/software/conda/esmvaltool-efbbc9d45c9eb22679208cc3eac458e67ddb7308/lib/python3.13/site-packages/iris/cube.py", line 2413, in coord
    raise iris.exceptions.CoordinateNotFoundError(emsg)
    ^^^^^^^^^^^
iris.exceptions.CoordinateNotFoundError: "Expected to find exactly 1 'latitude' coordinate, but found none."