GH #2744 Fix installation and tests for Python 3.10-3.13

Three pinned dependencies blocked installation on Python 3.12+:
- numpy<1.24: no cp312 wheels, distutils removal kills source build
- pandas==1.5.3: no cp312 wheels, same distutils issue
- scipy<1.11: requires_python="<3.12", resolver rejects outright

Dependency changes:
- numpy: uncap (numpy 2.x supported after source fixes below)
- pandas: relax to >=1.5.3,<3 (2.x supported; 3.0 needs further work)
- scipy: uncap (1.14+ supported after interp2d removal)
- pynwb: cap <3 to avoid breaking API changes
- xarray: unpin <2023.2.0 (pure Python, no breakage found)
- pytz: add as explicit dep (was transitive via pandas 1.x)
- setuptools: cap <81 (pkg_resources removed in 81+)

Source fixes for removed/changed APIs:
- scipy.interpolate.interp2d → RectBivariateSpline (removed in 1.14)
- ConfigParser.readfp → read_string (removed in Python 3.12)
- DataFrame.iteritems() → items() (removed in pandas 2.0)
- DataFrame.append() → pd.concat() (removed in pandas 2.0)
- pd.to_datetime: add format="ISO8601" for mixed-precision timestamps
  (pandas 2.x rejects mixed microsecond precision without explicit format)
- pd.to_datetime: fix utc="True" (string) → utc=True (bool) bug in
  behavior_project_cloud_api.py
- np.int() → int() (removed in numpy 1.24)
- np.NaN/np.NAN → np.nan (removed in numpy 2.0)
- np.Inf → np.inf, np.string_ → np.bytes_ (removed in numpy 2.0)
- np.product → np.prod, np.in1d → np.isin (removed in numpy 2.0)
- np.ediff1d to_begin dtype must match array dtype in numpy 2.0
- np.VisibleDeprecationWarning → try/except import (removed in 2.0)
- nwbfile.modules → .processing (pynwb 2.x deprecation)
- IndexSeries unit='None' → 'N/A' (pynwb 2.5+ validation)
- np.linalg.norm([array, scalar]) → np.vstack (numpy 1.24+ rejects)
- aiohttp.ClientSession → lazy property (aiohttp 3.9+ event loop)
- groupby().apply() → select column first (pandas 2.2+ behavior)
- demixer: add np.isfinite guard after linalg.solve (some LAPACK
  implementations return inf instead of raising LinAlgError for
  singular matrices)

Notebook fixes:
- IPython.core.display → IPython.display (removed in IPython 8.x)
  in visual_behavior_compare_across_trial_types and
  visual_behavior_mouse_history notebooks

Test fixes:
- pytest.warns(None) → warnings.catch_warnings (pytest 8)
- mock.called_once_with → assert_called_once() (proper assertion)
- Add res.x to MagicMock for scipy.optimize result
- rng.choice(inhomogeneous) → rng.integers + index (numpy 1.24+)
- Widen curve-fit tolerances for platform variance
- Cast dtypes for pynwb 2.x roundtrip and pandas 2.x index changes
- subprocess 'python' → sys.executable for venv correctness
- Replace flaky test_demix_raises_warning_for_singular_matrix with
  a result-based assertion in test_demix_point
- from mock import → from unittest.mock import (69 test files)
- Uncap/modernize test dependency bounds, remove dead weight
- Add pandas 2.x datetime regression tests (test_pandas_compat.py)

CI:
- Update to actions/checkout@v4, actions/setup-python@v5
- Slim matrix to min/max Python (3.10, 3.13) on ubuntu, 3.13 on
  macOS/windows
- Add pip caching, coverage on single matrix element only
- Modernize notebook_runner.yml to Python 3.13
- Update nightly.yml to checkout@v4
- Switch notebook runner to ubuntu-latest-8x (32GB RAM) for
  ecephys notebooks that peak at ~21GB RSS
- Change notebook workflow trigger to push on master (expensive job
  should not run on every pull request)

Fixes #2747, #2744, #2746, #2754

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: galenlynch

.github/workflows/github-actions-ci.yml

@@ -26,7 +26,7 @@ jobs:
     name: Lint
     runs-on: "ubuntu-latest"
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
       - name: flake8 linting
         run: |
           pip install flake8
@@ -43,28 +43,30 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: ["macos-latest", "windows-latest", "ubuntu-latest"]
-        python-version: ["3.8", "3.9", "3.10", "3.11"]
+        os: ["ubuntu-latest"]
+        python-version: ["3.10", "3.13"]
+        include:
+          - os: "macos-latest"
+            python-version: "3.13"
+          - os: "windows-latest"
+            python-version: "3.13"
       fail-fast: false
-    defaults:
-      run:
-        shell: bash -l {0}
     steps:
-      - uses: actions/checkout@v2
-      - uses: conda-incubator/setup-miniconda@v2
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
         with:
-          auto-update-conda: true
           python-version: ${{ matrix.python-version }}
-          activate-environment: test-env
+          cache: pip
+          cache-dependency-path: |
+            requirements.txt
+            test_requirements.txt
       - name: Install
         run: |
-          conda activate test-env
-          pip install codecov
           pip install -r test_requirements.txt
           pip install .
       - name: Test
         run: |
-          py.test --cov=allensdk -n 4
+          python -m pytest -n 4 ${{ (matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13') && '--cov=allensdk' || '' }}
 
   onprem:
     name: python ${{ matrix.image }} on-prem test
@@ -73,7 +75,7 @@ jobs:
       matrix:
         image: ["allensdk_local_py38:latest"]
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
       - name: run test in docker
         run: |
           docker run \

.github/workflows/nightly.yml

@@ -13,7 +13,7 @@ jobs:
         image: ["allensdk_local_py38:latest"]
         branch: ["master", "rc/**"]
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
         with:
           ref: ${{ matrix.branch }}
       - name: run test in docker

.github/workflows/notebook_runner.yml

@@ -1,23 +1,27 @@
 name: Notebooks
 on:
-  pull_request:
+  push:
     branches:
       - master
 jobs:
   run_notebooks:
     name: Notebook runner
     strategy:
       matrix:
-        os: ["ubuntu-latest"]
-        python-version: ["3.8"]
+        python-version: ["3.13"]
       fail-fast: false
-    runs-on: ${{ matrix.os }}
+    runs-on: ubuntu-latest-8x
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: |
+            requirements.txt
+            dev_requirements.txt
+            notebook_requirements.txt
       - name: Install
         run: |
           pip install -r requirements.txt

allensdk/brain_observatory/behavior/behavior_project_cache/project_apis/data_io/behavior_project_cloud_api.py

@@ -187,7 +187,7 @@ def _get_ophys_session_table(self):
         df = sanitize_data_columns(session_table_path, {"mouse_id": str})
         # Add UTC to match DateOfAcquisition object.
         df["date_of_acquisition"] = pd.to_datetime(
-            df["date_of_acquisition"], utc="True"
+            df["date_of_acquisition"], format="ISO8601", utc=True
         )
         df = enforce_df_int_typing(
             input_df=df, int_columns=VBO_INTEGER_COLUMNS, use_pandas_type=True
@@ -217,7 +217,7 @@ def _get_behavior_session_table(self):
         df = sanitize_data_columns(session_table_path, {"mouse_id": str})
         # Add UTC to match DateOfAcquisition object.
         df["date_of_acquisition"] = pd.to_datetime(
-            df["date_of_acquisition"], utc="True"
+            df["date_of_acquisition"], format="ISO8601", utc=True
         )
         df = enforce_df_int_typing(
             input_df=df, int_columns=VBO_INTEGER_COLUMNS, use_pandas_type=True
@@ -252,7 +252,7 @@ def _get_ophys_experiment_table(self):
         df = sanitize_data_columns(experiment_table_path, {"mouse_id": str})
         # Add UTC to match DateOfAcquisition object.
         df["date_of_acquisition"] = pd.to_datetime(
-            df["date_of_acquisition"], utc="True"
+            df["date_of_acquisition"], format="ISO8601", utc=True
         )
         df = enforce_df_int_typing(
             input_df=df, int_columns=VBO_INTEGER_COLUMNS, use_pandas_type=True

allensdk/brain_observatory/behavior/behavior_project_cache/project_apis/data_io/behavior_project_lims_api.py

@@ -511,7 +511,7 @@ def get_ophys_session_table(self) -> pd.DataFrame:
         )
         # Make date time explicitly UTC.
         table["date_of_acquisition"] = pd.to_datetime(
-            table["date_of_acquisition"], utc=True
+            table["date_of_acquisition"], format="ISO8601", utc=True
         )
 
         # Fill NaN values of imaging_plane_group_count with zero to match
@@ -554,7 +554,7 @@ def get_ophys_experiment_table(self) -> pd.DataFrame:
         """
         df = self._get_ophys_experiment_table()
         df["date_of_acquisition"] = pd.to_datetime(
-            df["date_of_acquisition"], utc=True
+            df["date_of_acquisition"], format="ISO8601", utc=True
         )
         # Set type to pandas.Int64 to enforce integer typing and not revert to
         # float.
@@ -581,7 +581,7 @@ def get_behavior_session_table(self) -> pd.DataFrame:
         summary_tbl = self._get_behavior_summary_table()
         # Add UTC time zone to match timezone from DateOfAcquisition object.
         summary_tbl["date_of_acquisition"] = pd.to_datetime(
-            summary_tbl["date_of_acquisition"], utc=True
+            summary_tbl["date_of_acquisition"], format="ISO8601", utc=True
         )
         # Query returns float typing of age_in_days. Convert to int to match
         # typing of the Age data_object.

allensdk/brain_observatory/behavior/behavior_project_cache/tables/metadata_table_schemas.py

@@ -78,7 +78,7 @@ class BehaviorSessionMetadataSchema(RaisingSchema):
     def convert_date_time(self, data, **kwargs):
         """Change date_of_acquisition to a date time type from string."""
         data["date_of_acquisition"] = pd.to_datetime(
-            data["date_of_acquisition"], utc=True
+            data["date_of_acquisition"], format="ISO8601", utc=True
         )
         return data
 

allensdk/brain_observatory/behavior/data_objects/cell_specimens/cell_specimens.py

@@ -857,13 +857,13 @@ def _validate_traces(
             if traces is None:
                 continue
             # validate traces contain expected roi ids
-            if not np.in1d(traces.value.index, cell_roi_ids).all():
+            if not np.isin(traces.value.index, cell_roi_ids).all():
                 raise RuntimeError(
                     f"{traces.name} contains ROI IDs that "
                     f"are not in "
                     f"cell_specimen_table.cell_roi_id"
                 )
-            if not np.in1d(cell_roi_ids, traces.value.index).all():
+            if not np.isin(cell_roi_ids, traces.value.index).all():
                 raise RuntimeError(
                     f"cell_specimen_table contains ROI IDs "
                     f"that are not in {traces.name}"

allensdk/brain_observatory/behavior/data_objects/metadata/behavior_metadata/behavior_metadata.py

@@ -149,7 +149,7 @@ def get_task_parameters(data: Dict) -> Dict:
     # displayed stimuli (no flashes).
 
     if stim_duration is None:
-        stim_duration = np.NaN
+        stim_duration = np.nan
     else:
         stim_duration = stim_duration[0]
 

allensdk/brain_observatory/behavior/data_objects/running_speed/running_acquisition.py

@@ -129,7 +129,7 @@ def from_nwb(
         cls,
         nwbfile: NWBFile
     ) -> "RunningAcquisition":
-        running_module = nwbfile.modules['running']
+        running_module = nwbfile.processing['running']
         dx_interface = running_module.get_data_interface('dx')
 
         dx = dx_interface.data

allensdk/brain_observatory/behavior/data_objects/running_speed/running_speed.py

@@ -188,7 +188,7 @@ def from_nwb(
         nwbfile: NWBFile,
         filtered=True
     ) -> "RunningSpeed":
-        running_module = nwbfile.modules['running']
+        running_module = nwbfile.processing['running']
         interface_name = 'speed' if filtered else 'speed_unfiltered'
         running_interface = running_module.get_data_interface(interface_name)