Frequency detec (#123)

* Implement temporal frequency detection and validation

Features:
- Add lazy frequency detection from temporal coordinates using xarray/Dask
- Validate frequency consistency across concatenated input files
- Configurable validation with tolerance settings for slight differences
- Custom FrequencyMismatchError for clear error reporting
- integrate validation into all CMORiser classes with user control
- Comprehensive unit tests covering various frequency scenarios

This prevents issues when concatenating files with different temporal
frequencies and provides early detection of data inconsistencies without
loading full datasets into memory. The implementation is performance-oriented
using lazy evaluation to check only the first few time points of each file.

Usage:
- Validation enabled by default: ACCESS_ESM_CMORiser(...)
- Disable validation: ACCESS_ESM_CMORiser(..., validate_frequency=False)
- Tolerance configurable via validate_consistent_frequency() function

Files modified:
- utilities.py: Core frequency detection functions
- base.py: Integration with load_dataset method
- driver.py: User-facing validate_frequency parameter
- ocean.py: Updated constructor to pass through parameter
- tests/unit/test_frequency_detection.py: Comprehensive test suite

* Enhance CMIP6 frequency validation with compound name support and add new utility functions for frequency parsing and compatibility checks

* Enhance temporal frequency detection and resampling functionality

- Added support for detecting temporal frequency from CF-compliant time bounds in `detect_time_frequency_lazy`.
- Implemented a new utility function `_detect_frequency_from_bounds` for improved frequency detection.
- Introduced `validate_and_resample_if_needed` to validate dataset frequency and resample if necessary for CMIP6 compatibility.
- Enhanced `resample_dataset_temporal` to handle various resampling methods based on variable metadata.
- Updated tests to cover new frequency detection and resampling features, including edge cases for single time points and bounds detection.
- Created a new test suite for temporal resampling functionality, ensuring robust validation and error handling.

* Add ACCESS frequency metadata parsing and detection functions

* Implement smart tolerance for frequency validation and add tests for calendar month variations

* Enhance frequency compatibility validation for monthly data, allowing for calendar month variations. Implement helper functions to validate monthly files and adjust compatibility checks in CMIP6 frequency validation.

* Enhance frequency compatibility validation to handle monthly data without resampling. Special handling for calendar month variations added to improve accuracy in CMIP6 frequency checks.

* Implement efficient frequency detection for monthly files using concatenation; add fallback to individual file analysis. Remove outdated test scripts for exact error simulation and monthly calendar variations.

* Set default validate_frequency to False in CMIP6_CMORiser constructor

* Fix tests

* Fix pre-commit

* Remove codacy

* Revert "Remove codacy"

This reverts commit 936f1c6.

* Add compound_name parameter to CMIP6_CMORiser and update tests

* Refactor CMIP6_CMORiser to extract cmor_name from compound_name and update related tests

* Remove unused import and simplify print statement in CMIP6_CMORiser

* Refactor load_filtered_variables to support model-specific mappings and update integration tests accordingly

* Fix pre-commit

* Fix push trigger in CI workflow to include main branch

Author: rbeucher

.github/workflows/ci.yml

@@ -4,7 +4,7 @@ on:
   pull_request:
     branches: [main]
   push:
-    branches-ignore: [main]
+    branches: [main]
   workflow_dispatch:
     inputs:
       test_suite:

src/access_moppy/base.py

@@ -1,3 +1,4 @@
+import warnings
 from datetime import datetime
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Union
@@ -6,7 +7,14 @@
 import xarray as xr
 from cftime import num2date
 
-from access_moppy.utilities import type_mapping
+from access_moppy.utilities import (
+    FrequencyMismatchError,
+    IncompatibleFrequencyError,
+    ResamplingRequiredWarning,
+    type_mapping,
+    validate_and_resample_if_needed,
+    validate_cmip6_frequency_compatibility,
+)
 
 
 class CMIP6_CMORiser:
@@ -20,20 +28,28 @@ def __init__(
         self,
         input_paths: Union[str, List[str]],
         output_path: str,
-        cmor_name: str,
         cmip6_vocab: Any,
         variable_mapping: Dict[str, Any],
+        compound_name: str,
         drs_root: Optional[Path] = None,
+        validate_frequency: bool = False,
+        enable_resampling: bool = False,
+        resampling_method: str = "auto",
     ):
         self.input_paths = (
             input_paths if isinstance(input_paths, list) else [input_paths]
         )
         self.output_path = output_path
-        self.cmor_name = cmor_name
+        # Extract cmor_name from compound_name
+        _, self.cmor_name = compound_name.split(".")
         self.vocab = cmip6_vocab
         self.mapping = variable_mapping
         self.drs_root = Path(drs_root) if drs_root is not None else None
         self.version_date = datetime.now().strftime("%Y%m%d")
+        self.validate_frequency = validate_frequency
+        self.compound_name = compound_name
+        self.enable_resampling = enable_resampling
+        self.resampling_method = resampling_method
         self.ds = None
 
     def __getitem__(self, key):
@@ -50,9 +66,44 @@ def __repr__(self):
         return repr(self.ds)
 
     def load_dataset(self, required_vars: Optional[List[str]] = None):
+        """
+        Load dataset from input files with optional frequency validation.
+
+        Args:
+            required_vars: Optional list of required variables to extract
+        """
+
         def _preprocess(ds):
             return ds[list(required_vars & set(ds.data_vars))]
 
+        # Validate frequency consistency and CMIP6 compatibility before concatenation
+        if self.validate_frequency and len(self.input_paths) > 0:
+            try:
+                # Enhanced validation with CMIP6 frequency compatibility
+                detected_freq, resampling_required = (
+                    validate_cmip6_frequency_compatibility(
+                        self.input_paths,
+                        self.compound_name,
+                        time_coord="time",
+                        interactive=True,
+                    )
+                )
+                if resampling_required:
+                    print(
+                        f"✓ Temporal resampling will be applied: {detected_freq} → CMIP6 target frequency"
+                    )
+                else:
+                    print(f"✓ Validated compatible temporal frequency: {detected_freq}")
+            except (FrequencyMismatchError, IncompatibleFrequencyError) as e:
+                raise e  # Re-raise these specific errors as-is
+            except InterruptedError as e:
+                raise e  # Re-raise user abort
+            except Exception as e:
+                warnings.warn(
+                    f"Could not validate temporal frequency: {e}. "
+                    f"Proceeding with concatenation but results may be inconsistent."
+                )
+
         self.ds = xr.open_mfdataset(
             self.input_paths,
             combine="nested",  # avoids costly dimension alignment
@@ -64,6 +115,37 @@ def _preprocess(ds):
             parallel=True,  # <--- enables concurrent preprocessing
         )
 
+        # Apply temporal resampling if enabled and needed
+        if self.enable_resampling and self.compound_name:
+            try:
+                print(
+                    f"🔍 Checking if temporal resampling is needed for {self.cmor_name}..."
+                )
+
+                self.ds, was_resampled = validate_and_resample_if_needed(
+                    self.ds,
+                    self.compound_name,
+                    self.cmor_name,
+                    time_coord="time",
+                    method=self.resampling_method,
+                )
+
+                if was_resampled:
+                    print("✅ Applied temporal resampling to match CMIP6 requirements")
+                else:
+                    print("✅ No resampling needed - frequency already compatible")
+
+            except (FrequencyMismatchError, IncompatibleFrequencyError) as e:
+                raise e  # Re-raise validation errors
+            except Exception as e:
+                raise RuntimeError(f"Failed to resample dataset: {e}")
+        elif self.enable_resampling and not self.compound_name:
+            warnings.warn(
+                "Resampling enabled but no compound_name provided. "
+                "Cannot determine target frequency for resampling.",
+                ResamplingRequiredWarning,
+            )
+
     def sort_time_dimension(self):
         if "time" in self.ds.dims:
             self.ds = self.ds.sortby("time")

src/access_moppy/driver.py

@@ -28,6 +28,9 @@ def __init__(
         drs_root: Optional[Union[str, Path]] = None,
         parent_info: Optional[Dict[str, Dict[str, Any]]] = None,
         model_id: Optional[str] = None,
+        validate_frequency: bool = True,
+        enable_resampling: bool = False,
+        resampling_method: str = "auto",
     ):
         """
         Initializes the CMORiser with necessary parameters.
@@ -42,9 +45,15 @@ def __init__(
         :param drs_root: Optional root path for DRS structure.
         :param parent_info: Optional dictionary with parent experiment metadata.
         :param model_id: Optional model identifier for model-specific mappings (e.g., 'ACCESS-ESM1.6').
+        :param validate_frequency: Whether to validate temporal frequency consistency across input files (default: True).
+        :param enable_resampling: Whether to enable automatic temporal resampling when frequency mismatches occur (default: False).
+        :param resampling_method: Method for temporal resampling ('auto', 'mean', 'sum', 'min', 'max', 'first', 'last') (default: 'auto').
         """
 
         self.input_paths = input_paths
+        self.validate_frequency = validate_frequency
+        self.enable_resampling = enable_resampling
+        self.resampling_method = resampling_method
         self.output_path = Path(output_path)
         self.compound_name = compound_name
         self.experiment_id = experiment_id
@@ -75,24 +84,30 @@ def __init__(
         )
 
         # Initialize the CMORiser based on the compound name
-        table, cmor_name = compound_name.split(".")
+        table, _ = compound_name.split(".")  # cmor_name now extracted internally
         if table in ("Amon", "Lmon", "Emon"):
             self.cmoriser = CMIP6_Atmosphere_CMORiser(
                 input_paths=self.input_paths,
                 output_path=str(self.output_path),
-                cmor_name=cmor_name,
                 cmip6_vocab=self.vocab,
                 variable_mapping=self.variable_mapping,
+                compound_name=self.compound_name,
                 drs_root=drs_root if drs_root else None,
+                validate_frequency=self.validate_frequency,
+                enable_resampling=self.enable_resampling,
+                resampling_method=self.resampling_method,
             )
         elif table in ("Oyr", "Oday", "Omon", "SImon"):
             self.cmoriser = CMIP6_Ocean_CMORiser(
                 input_paths=self.input_paths,
                 output_path=str(self.output_path),
-                cmor_name=cmor_name,
                 cmip6_vocab=self.vocab,
                 variable_mapping=self.variable_mapping,
+                compound_name=self.compound_name,
                 drs_root=drs_root if drs_root else None,
+                validate_frequency=self.validate_frequency,
+                enable_resampling=self.enable_resampling,
+                resampling_method=self.resampling_method,
             )
 
     def __getitem__(self, key):

src/access_moppy/ocean.py

@@ -18,18 +18,24 @@ def __init__(
         self,
         input_paths: Union[str, List[str]],
         output_path: str,
-        cmor_name: str,
         cmip6_vocab: CMIP6Vocabulary,
         variable_mapping: Dict[str, Any],
+        compound_name: str,
         drs_root: Optional[Path] = None,
+        validate_frequency: bool = True,
+        enable_resampling: bool = False,
+        resampling_method: str = "auto",
     ):
         super().__init__(
             input_paths=input_paths,
             output_path=output_path,
-            cmor_name=cmor_name,
             cmip6_vocab=cmip6_vocab,
             variable_mapping=variable_mapping,
+            compound_name=compound_name,
             drs_root=drs_root,
+            validate_frequency=validate_frequency,
+            enable_resampling=enable_resampling,
+            resampling_method=resampling_method,
         )
 
         nominal_resolution = cmip6_vocab._get_nominal_resolution()