Add CTD thermal lag correction, YAML constants reader, and GSW salini…

.circleci/config.yml

@@ -8,14 +8,16 @@ jobs:
     executor: python/default
     steps:
       - checkout
+      - run:
+          name: Upgrade pip and setuptools
+          command: python -m ensurepip --upgrade || true && python -m pip install --upgrade pip setuptools
       - run:
           name: Install Python dependencies
           command: |
-            python -m pip install --user \
-                -r docs-requirements.txt
+            python -m pip install -r docs-requirements.txt
       - run:
           name: install module
-          command: python -m pip install --user -ve .
+          command: python -m pip install -ve .
       - run:
           name: Build docs
           command: cd docs/ && make html

.gitignore

@@ -6,3 +6,6 @@ dist
 
 *.png
 .DS_Store
+*.ipynb_checkpoints
+Untitled*.ipynb
+*.py.save

Untitled.ipynb

@@ -0,0 +1,6 @@
+{
+ "cells": [],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/adjust_CTD.md

@@ -0,0 +1,118 @@
+# PyGlider: Adjust CTD variables
+
+PyGlider applies a post-processing protocol to conductivity, temperature, and salinity variables in NetCDF timeseries files, and subsequently generates NetCDF depth–time grids using Python and `xarray`. The resulting NetCDF files are largely CF-compliant.
+
+The basic workflow consists of converting a NetCDF timeseries into an adjusted timeseries and corresponding depth–time grids. This follows the `binary_to_timeseries` (for Slocum gliders) and `raw_to_timeseries` (for Alseamar gliders) protocols, which convert raw glider data into NetCDF format. The variable `outname` refers to the timeseries output from this prior step.
+
+```python
+outname_ctd = pyglider.ncprocess.adjust_CTD(
+    outname,
+    deploymentyaml,
+    l1tsdir,
+    griddir,
+    dTdC=None,
+    tau=None,
+    alpha=None,
+    maskfunction=None,
+    interp_variables=None
+)
+```
+
+Data are read from and written to directories, and metadata are supplied via a YAML file.
+
+---
+
+## Post-processing steps within `pyglider.ncprocess.adjust_CTD`
+
+### 1. Identify anomalous conductivity values
+
+We identify and flag conductivity values that are clearly unphysical, typically caused by air bubbles in the conductivity cell.
+
+A two-step statistical criterion is applied:
+
+- First, data points more than **5 standard deviations** from the mean are temporarily flagged within each depth and profile bin.  
+- The mean and standard deviation are then recomputed excluding these points.  
+- Values still exceeding **3 standard deviations** from the recomputed mean are flagged as **bad (QC = 4)** in `conductivity_QC`.  
+
+However, if the deviation is smaller than the sensor accuracy (0.0003 S/m for the GPCTD), the data are retained.
+
+This procedure is applied using:
+
+- Profile bins of 50 profiles  
+- Depth bins of 5 m  
+
+Using profile-index binning (rather than time or temperature) helps isolate unphysical values.
+
+Salinity (`salinity_QC`) is flagged as bad (QC = 4) wherever `conductivity_QC` is QC4.
+
+---
+
+### 2. Determine what dTdC, tau, and alpha are used in the correction
+
+We correct for:
+
+- Sensor misalignment between temperature and conductivity (`dTdC`)  
+- Thermal lag effects (`tau`, `alpha`)  
+
+Where:
+
+- `dTdC` = time lag (seconds) between temperature and conductivity sensors  
+- `tau` = thermal response time constant (seconds)  
+- `alpha` = scaling of thermal coupling between water and the conductivity cell  
+
+Further details and methods for determining these parameters are available at:  
+https://cproof.uvic.ca/gliderdata/deployments/reports/
+
+For recent C-PROOF missions, these values are included in the YAML file. However, users may:
+
+- Provide custom values for `dTdC`, `tau`, and `alpha`, or  
+- Skip corrections by setting parameters to `None`  
+
+If user-supplied values differ from those in the YAML file, a warning is issued, but the user-provided values are used.
+
+New variables introduced:
+
+- `temperature_adjusted`  
+- `salinity_adjusted`  
+- `temperature_adjusted_QC`  
+- `salinity_adjusted_QC`  
+
+---
+
+### 3. Recalculate derived variables
+
+Using TEOS-10, we recompute:
+
+- `potential_density_adjusted`  
+- `potential_temperature_adjusted`  
+
+Their corresponding QC variables:
+
+- `potential_density_adjusted_QC`  
+- `potential_temperature_adjusted_QC`  
+
+These are flagged as bad (QC = 4) wherever `salinity_adjusted_QC` is QC4.
+
+---
+
+### 4. Convert the adjusted NetCDF timeseries to a NetCDF depth-time grid
+
+The adjusted timeseries is converted into a gridded NetCDF dataset.
+
+- Default binning:
+  - 1 m depth bins  
+  - Profile bins  
+
+- Variables are averaged within each bin.
+
+QC variables use a `QC_protocol` that selects the maximum QC value within each bin, ensuring that bad data are not diluted.
+
+Optional parameters:
+
+- `maskfunction`  
+- `interp_variables`  
+
+C-PROOF applies:
+
+- `pyglider.ncprocess.CPROOF_mask` to exclude QC4 data from gridded products  
+- `pyglider.ncprocess.interpolate_vertical` to interpolate over vertical gaps up to 50 m  

docs/getting-started-seaexplorer.md

@@ -31,7 +31,17 @@ The example script is relatively straight forward if there is no intermediate pr
 
 Data comes from an input directory, and is translated to raw glider-dependent parquet files files and put in a new directory. These files are useful of their own right. Apache Parquet is a columnar oriented format for storing tabular data. Parquet files take up less space than netCDF or csv and are much faster to read and write. These files can be opened with [polars.read_parquet](https://pola-rs.github.io/polars-book/user-guide/howcani/io/parquet.html) or [pandas.read_parquet](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html). These files are then merged into a single monolithic parquet file, and this is translated to a CF-compliant timeseries netCDF file. Finally individual profiles are saved and a 2-D 1-m grid in time-depth is saved.
 
-It is likely that between these steps the user will want to add any screening steps, or adjustments to the calibrations. PyGlider does not provide those steps.
+Users may wish to include additional screening steps or calibration adjustments between these stages.
+
+PyGlider provides an optional function, `pyglider.ncprocess.adjust_CTD`, which:
+
+- identifies anomalous conductivity values  
+- corrects for sensor misalignment between temperature and conductivity  
+- applies a thermal lag correction  
+
+This function can be used as a starting point and adapted depending on the dataset and application.
+
+See the full documentation: [CTD adjustment workflow](adjust_CTD.md). 
 
 (ExDepl)=
 

docs/getting-started-slocum.md

@@ -29,7 +29,17 @@ Data comes from an input directory, and is translated into a single CF-compliant
 There is a version that does not require `dbdreader` to do the initial conversion from the Dinkum format to netCDF. However it is quite slow, particularly for full-resolution datasets, and less robust. We suggest using the `slocum.raw_to_timeseries`.
 :::
 
-It is possible that between these steps the user will want to add any screening steps, or adjustments to the calibrations. PyGlider does not provide those steps, but is designed so they are easy to add.
+Users may wish to include additional screening steps or calibration adjustments between these stages.
+
+PyGlider provides an optional function, `pyglider.ncprocess.adjust_CTD`, which:
+
+- identifies anomalous conductivity values  
+- corrects for sensor misalignment between temperature and conductivity  
+- applies a thermal lag correction  
+
+This function can be used as a starting point and adapted depending on the dataset and application.
+
+See the full documentation: [CTD adjustment workflow](adjust_CTD.md).
 
 (ExDeplSlocum)=
 

docs/index.md

@@ -29,7 +29,7 @@ Install
 getting-started-seaexplorer
 getting-started-slocum
 pyglider/pyglider
-
+adjust_CTD
 ```
 
 ## Acknowledgements