Skip to content

Improve ENSO driver file discovery across realms#1378

Open
zhangshixuan1987 wants to merge 2 commits into
PCMDI:mainfrom
zhangshixuan1987:enso_fix
Open

Improve ENSO driver file discovery across realms#1378
zhangshixuan1987 wants to merge 2 commits into
PCMDI:mainfrom
zhangshixuan1987:enso_fix

Conversation

@zhangshixuan1987
Copy link
Copy Markdown
Collaborator

This updates the ENSO driver to make model input discovery more robust for workflows that mix atmosphere and ocean variables, while preserving the existing successful lookup path as the first choice.

Detailed changes:

  • Add filename token parsing that supports both "." and "_" separators.
  • Pass the inferred realm when discovering models with modnames="all".
  • Use a wildcard realization for file discovery when realization is "all" or "*".
  • Avoid leaving runs_list undefined when wildcard run parsing fails.
  • Normalize table-style realms to path-style realms, e.g. Amon -> atmos and Omon -> ocean.
  • Add fallback model-file lookup for realm/table mismatches: Amon <-> Omon and atmos <-> ocean.
  • Keep the original rendered model path as the first lookup, and only try fallback paths when the original path is missing.
  • Allow ocean variables such as zos/tos to be found when a template is hard-coded with Amon/atmos.
  • Resolve multi-component variables using the concrete loop run instead of the original realization string.
  • Reuse the same model-file fallback logic for multi-component variables.

This helps ENSO_proc metrics that require ssh/zos run when the input exists under an ocean table/path, without changing the downstream metric or derive computation behavior.

This updates the ENSO driver to make model input discovery more robust
for workflows that mix atmosphere and ocean variables, while preserving
the existing successful lookup path as the first choice.

Detailed changes:
- Add filename token parsing that supports both "." and "_" separators.
- Pass the inferred realm when discovering models with modnames="all".
- Use a wildcard realization for file discovery when realization is
  "all" or "*".
- Avoid leaving runs_list undefined when wildcard run parsing fails.
- Normalize table-style realms to path-style realms, e.g. Amon -> atmos
  and Omon -> ocean.
- Add fallback model-file lookup for realm/table mismatches:
  Amon <-> Omon and atmos <-> ocean.
- Keep the original rendered model path as the first lookup, and only
  try fallback paths when the original path is missing.
- Allow ocean variables such as zos/tos to be found when a template is
  hard-coded with Amon/atmos.
- Resolve multi-component variables using the concrete loop run instead
  of the original realization string.
- Reuse the same model-file fallback logic for multi-component variables.

This helps ENSO_proc metrics that require ssh/zos run when the input
exists under an ocean table/path, without changing the downstream metric
or derive computation behavior.
@zhangshixuan1987
Copy link
Copy Markdown
Collaborator Author

@lee1043 Hi Jiwoo, this commit includes changes intended to address one of the issues pointed out by @yyplanton in #95:

EnsoFbTauxSsh, EnsoFbSshSst: not computed. Is it because zos is not accessible or is it because of the curvilinear grid? If the latter, can you regrid before running the package?

In my local Demo 6 test, I saw the following message:

ComputeCollection: metric = EnsoFbTauxSsh
ComputeCollection: ENSO_proc, metric EnsoFbTauxSsh not computed (no model data for: ssh)
ComputeCollection: metric = EnsoSeasonality
ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1

This indicates that the metric was skipped because no model data were found for ssh. I think there are two related issues here.

First, in CMIP-style files, sea surface height is often stored as zos, while ENSO_metrics internally expects this variable as ssh. For example, in the Demo 6 data, the relevant file is:

demo_data/CMIP5_demo_data/zos_Omon_ACCESS1-0_historical_r1i1p1_185001-200512.nc

So there is a vulnerability in the workflow: when zos is passed into ENSO_metrics, the code does not always recognize that zos corresponds to the internal ssh variable needed by the metric.

Second, there appears to be an issue on the PCMDI enso_driver.py side. The zos variable is from the Omon CMIP table, i.e., the ocean model output table, but enso_driver.py appears to use a fixed Amon table when searching for model files. As a result, the file zos_Omon_ACCESS1-0_historical_r1i1p1_185001-200512.nc is not identified and passed downstream to provide the required ssh data for ENSO_metrics.

The code change in this commit is intended to address the second issue on the PCMDI enso_driver.py side, so that variables such as zos can be searched using the appropriate CMIP table rather than assuming Amon. With the changes the zos_Omon_ACCESS1-0_historical_r1i1p1_185001-200512.nc can be passed to the ENSO_METRICS for the calculation of ssh related metrics based on my local test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant