Test/filipps update deps (#1757)

SFJohnson24 · filippsatverily · gerrycampion · web-flow · commit 4a2dbb0d3a0f · 2026-06-15T08:49:39.000-04:00
* Define dependencies in pyproject.toml Moves dependency constraints to pyproject.toml. Makes requirements.txt a lockfile. * Support click 8.3.0 Fixes an incompatibility caused by click 8.3.0, which passes the default value as-is. * Support pyreadstat 1.2.9 Fixes an incompatibility caused by pyreadstat 1.2.9, which changed original_variable_type from 'NULL' to None * Support jsonpath-ng 1.8.0 Works around an behavior change in jsonpath-ng 1.8.0 where Child.str gets wrapped in parenthesis. * Suport dask 2024.8.1 Fixes tokenization errors when using dask 2024.8.1+. Starting with this version, dask enforces that tokens remain stable across pickle round-trips (dask/dask#11320). Capturing self in a lambda fails this check because instance objects can have non-deterministic pickle representations. Since calculate_variable_value_length is already a static method, replacing self with the class name is enough to remove the capture. * Support dask 2024.12.1 Fixes an import error caused by dask 2024.12.1, which removed the legacy dask.dataframe.dd submodule (dask/dask#11604). Changes the import to `import dask.dataframe as dd`, consistent with every other file in the codebase. * Support dask 2025.4.0 Dask 2025.4.0 optimizes multiple DataFrames together, which exposes division mismatches when assigning a pandas Series to a dask DataFrame column. The old reset_index/set_index workaround no longer avoids this. Replacing it with compute-assign-rewrap via dd.from_pandas, which builds a clean expression graph. This is safe because __getitem__ already computes the DataFrame to produce the Series being assigned. * Support pandas 2.2.0 Fixes a unit test to support pandas 2.2.0+. The pandas release fixes a sorting bug with pandas-dev/pandas#54611. This commit changes the expected results accordingly. Also fixes a merge type mismatch introduced by upstream #1709: the codelist metadata side was cast to StringDtype but the evaluation dataset side was not. With pandas 2.2.0, empty columns infer as float64, and merging float64 with string is rejected. Casting both sides to string before the merge resolves this. * testing * restore * parentheses strip * stringg to str * black formatting for 26.5.1 * pylock * sync changes, and remove requirements * system flag * no build * pyproject.toml, removes requirements * dep ranges * dask, path rework * full path * str * jsonschema deps and tester * path * path * type coercion fix builder * to dict refactor * dask test * line length * azure-functions as optional-dependency --------- Co-authored-by: Filipp Shpomer <filipps@verily.com> Co-authored-by: Gerry Campion <gcampion@cdisc.org>
diff --git a/.github/workflows/build-binary.yml b/.github/workflows/build-binary.yml
@@ -20,10 +20,9 @@ jobs:
           python-version: "3.12"
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-          pip install setuptools wheel twine
-          pip install --no-deps -r requirements.txt
-          pip install -r requirements.txt --no-cache-dir
+          pip install --upgrade pip
+          pip install twine
+          pip install .
 
       - name: Build Binary (Linux)
         if: runner.os == 'Linux'
diff --git a/.github/workflows/build-version.yml b/.github/workflows/build-version.yml
@@ -18,10 +18,9 @@ jobs:
           python-version: "3.12"
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-          pip install setuptools wheel twine
-          pip install --no-deps -r requirements.txt
-          pip install -r requirements.txt --no-cache-dir
+          pip install --upgrade pip
+          pip install twine
+          pip install .
       - name: Update CDISC standards cache
         env:
           CDISC_LIBRARY_API_KEY: ${{ secrets.CDISC_LIBRARY_API_KEY }}
@@ -70,12 +69,11 @@ jobs:
 #         uses: actions/setup-python@v6
 #         with:
 #           python-version: "3.12"
-#       - name: Install dependencies
-#         run: |
-#           python -m pip install --upgrade pip
-#           pip install setuptools wheel twine
-#           pip install --no-deps -r requirements.txt
-#           pip install -r requirements.txt --no-cache-dir
+#      - name: Install dependencies
+#        run: |
+#          pip install --upgrade pip
+#          pip install twine
+#          pip install .
 #       - name: Update CDISC standards cache
 #         env:
 #           CDISC_LIBRARY_API_KEY: ${{ secrets.CDISC_LIBRARY_API_KEY }}
@@ -114,12 +112,11 @@ jobs:
 #         uses: actions/setup-python@v6
 #         with:
 #           python-version: "3.12"
-#       - name: Install dependencies
-#         run: |
-#           python -m pip install --upgrade pip
-#           pip install setuptools wheel twine
-#           pip install --no-deps -r requirements.txt
-#           pip install -r requirements.txt --no-cache-dir
+#      - name: Install dependencies
+#        run: |
+#          pip install --upgrade pip
+#          pip install twine
+#          pip install .
 #       - name: Update CDISC standards cache
 #         env:
 #           CDISC_LIBRARY_API_KEY: ${{ secrets.CDISC_LIBRARY_API_KEY }}
diff --git a/.github/workflows/deploy-rule-tester.yml b/.github/workflows/deploy-rule-tester.yml
@@ -30,6 +30,11 @@ jobs:
     environment: ${{ github.event_name == 'pull_request' && 'DEV' || inputs.environment || 'PROD' }}
     steps:
       - uses: actions/checkout@v6
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+      - name: Generate requirements.txt from pyproject.toml
+        run: pip install pip-tools && pip-compile pyproject.toml --extra rule-tester -o requirements.txt --no-header
       - uses: azure/login@v2
         with:
           creds: ${{ env.creds }}
diff --git a/.github/workflows/lint-format.yml b/.github/workflows/lint-format.yml
@@ -51,7 +51,7 @@ jobs:
           python-version: "3.12"
       - name: Install linters
         run: |
-          pip install black flake8 -c requirements-dev.txt
+          pip install black flake8
       - name: Run flake8
         run: |
           flake8 ${{needs.get_changed_files.outputs.py}} --count --select=E9,F63,F7,F82 --show-source --statistics
@@ -88,7 +88,7 @@ jobs:
           python-version: "3.12"
       - name: Install yamllint
         run: |
-          pip install yamllint -c requirements-dev.txt
+          pip install ".[dev]"
       - name: Run yamllint
         run: |
           yamllint ${{needs.get_changed_files.outputs.yaml}}
diff --git a/.github/workflows/prerelease-update-cache.yml b/.github/workflows/prerelease-update-cache.yml
@@ -23,8 +23,10 @@ jobs:
         with:
           python-version: "3.12"
 
-      - name: Install requirements
-        run: pip install -r requirements.txt
+      - name: Install dependencies
+        run: |
+          pip install --upgrade pip
+          pip install .
 
       - name: Run Update Cache Script
         run: python core.py update-cache
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -101,9 +101,9 @@ jobs:
           python-version: "3.12"
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-          pip install build setuptools wheel twine
-          pip install -r requirements.txt
+          pip install --upgrade pip
+          pip install twine
+          pip install .
       - name: Build package
         run: |
           python -m build
diff --git a/.github/workflows/test-release.yml b/.github/workflows/test-release.yml
@@ -80,9 +80,9 @@ jobs:
 
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-          pip install build setuptools wheel twine
-          pip install -r requirements.txt
+          pip install --upgrade pip
+          pip install twine
+          pip install .
 
       - name: Build package
         run: |
diff --git a/.github/workflows/test-suite.yml b/.github/workflows/test-suite.yml
@@ -28,7 +28,8 @@ jobs:
           python-version: "3.12"
       - name: Install dependencies
         run: |
-          pip install -r requirements.txt
+          pip install --upgrade pip
+          pip install . --group dev
       - name: Update rules cache
         env:
           CDISC_LIBRARY_API_KEY: ${{ secrets.CDISC_LIBRARY_API_KEY }}
@@ -117,7 +118,7 @@ jobs:
             echo "## Dask Validation" >> $GITHUB_STEP_SUMMARY
             echo "✅ **Success**: Validation completed successfully" >> $GITHUB_STEP_SUMMARY
             python CORE_Test_Suite/scripts/validation_summary.py \
-              dask-results.json >> $GITHUB_STEP_SUMMARY
+              CORE_Test_Suite/dask-results.json >> $GITHUB_STEP_SUMMARY
           else
             echo "Failed to generate dask-results.json"
             echo "dask_success=false" >> $GITHUB_OUTPUT
diff --git a/.github/workflows/test-unit.yml b/.github/workflows/test-unit.yml
@@ -14,10 +14,10 @@ jobs:
         uses: actions/setup-python@v6
         with:
           python-version: "3.12"
-      - name: Install requirements
+      - name: Install dependencies
         run: |
-          pip install -r requirements-dev.txt
-          pip install -e .
+          pip install --upgrade pip
+          pip install . --group dev
       - name: Running Tests
         env:
           CDISC_LIBRARY_API_KEY: fakekey12341234
diff --git a/.github/workflows/validate-published-rules.yml b/.github/workflows/validate-published-rules.yml
@@ -21,7 +21,9 @@ on:
         required: false
         default: "main"
       core_ids:
-        description: "Space-separated list of rule IDs to validate (e.g. CORE-000001 CORE-000002). Leave blank to validate all."
+        description: >-
+          Space-separated list of rule IDs to validate
+          (e.g. CORE-000001 CORE-000002). Leave blank to validate all.
         required: false
         default: ""
 
@@ -80,9 +82,8 @@ jobs:
       - name: Install engine dependencies
         run: |
           python -m venv venv
-          ./venv/bin/pip install --upgrade pip
-          ./venv/bin/pip install -r engine/requirements.txt
-
+          venv/bin/pip install --upgrade pip
+          venv/bin/pip install . --group dev
       # -----------------------------------------------------------------------
       # 5. Run validation for every Published rule
       # -----------------------------------------------------------------------
diff --git a/cdisc_rules_engine/dataset_builders/contents_define_vlm_dataset_builder.py b/cdisc_rules_engine/dataset_builders/contents_define_vlm_dataset_builder.py
@@ -66,7 +66,7 @@ def build(self):
         data_contents_with_vlm["variable_value_length"] = data_contents_with_vlm.data[
             ["variable_value", "define_vlm_data_type"]
         ].apply(
-            lambda row: self.calculate_variable_value_length(
+            lambda row: ValuesDatasetBuilder.calculate_variable_value_length(
                 row["variable_value"], row["define_vlm_data_type"]
             ),
             axis=1,
diff --git a/cdisc_rules_engine/dataset_builders/dataset_metadata_define_dataset_builder.py b/cdisc_rules_engine/dataset_builders/dataset_metadata_define_dataset_builder.py
@@ -59,6 +59,7 @@ def build(self):
         merged._data = merged._data.astype(object).replace({np.nan: None})
 
         # 5. Return all rows (one per dataset)
+        merged._data = merged._data.sort_values("dataset_name").reset_index(drop=True)
         return merged
 
     def _get_define_xml_dataframe(self):
diff --git a/cdisc_rules_engine/models/dataset/dask_dataset.py b/cdisc_rules_engine/models/dataset/dask_dataset.py
@@ -1,6 +1,5 @@
 from cdisc_rules_engine.models.dataset.pandas_dataset import PandasDataset
 import dask.dataframe as dd
-import dask.array as da
 import pandas as pd
 import re
 import dask
@@ -76,14 +75,10 @@ def check_partition(partition):
         )
 
     def __setitem__(self, key, value):
-        if isinstance(value, list):
-            chunks = self._data.map_partitions(lambda x: len(x)).compute().to_numpy()
-            array_values = da.from_array(value, chunks=tuple(chunks))
-            self._data[key] = array_values
-        elif isinstance(value, pd.Series):
-            self._data = self._data.reset_index()
-            self._data = self._data.set_index("index")
-            self._data[key] = value
+        if isinstance(value, (list, pd.Series)):
+            pdf = self._data.compute()
+            pdf[key] = value if isinstance(value, list) else value.reindex(pdf.index)
+            self._data = dd.from_pandas(pdf, npartitions=self._data.npartitions)
         elif isinstance(value, dd.DataFrame):
             for column in value:
                 self._data[column] = value[column]
@@ -367,14 +362,7 @@ def fillna(
     def to_dict(self, **kwargs) -> dict:
         orient = kwargs.get("orient", "dict")
         if orient == "records":
-            reset_df = self._data.reset_index(drop=True)
-            all_partitions = list(
-                reset_df.map_partitions(lambda x: x.to_dict(orient="records"))
-            )
-            flattened = []
-            for partition in all_partitions:
-                flattened.extend(partition)
-            return flattened
+            return self._data.compute().reset_index(drop=True).to_dict(orient="records")
         else:
             return self._data.compute().to_dict(**kwargs)
 
diff --git a/cdisc_rules_engine/operations/codelist_extensible.py b/cdisc_rules_engine/operations/codelist_extensible.py
@@ -28,10 +28,14 @@ def _handle_multiple_versions(self) -> pd.Series:
         ct_df = self.evaluation_dataset.__class__.from_dict(ct_data)
         ct_df = ct_df.astype(
             {
-                "version": "string",
-                "codelist_code": "string",
+                "version": str,
+                "codelist_code": str,
             }
         )
+        cast_cols = {self.params.ct_version: str}
+        if self.params.codelist_code in self.evaluation_dataset.columns:
+            cast_cols[self.params.codelist_code] = str
+        self.evaluation_dataset = self.evaluation_dataset.astype(cast_cols)
         if self.params.codelist_code in self.evaluation_dataset.columns:
             is_extensible = self.evaluation_dataset.merge(
                 ct_df.data,
diff --git a/cdisc_rules_engine/operations/codelist_terms.py b/cdisc_rules_engine/operations/codelist_terms.py
@@ -60,10 +60,14 @@ def _handle_multiple_versions(self) -> pd.Series:
         ct_df = self.evaluation_dataset.__class__.from_dict(ct_data)
         ct_df = ct_df.astype(
             {
-                "version": "string",
-                "codelist_code": "string",
+                "version": str,
+                "codelist_code": str,
             }
         )
+        cast_cols = {self.params.ct_version: str}
+        if self.params.codelist_code in self.evaluation_dataset.columns:
+            cast_cols[self.params.codelist_code] = str
+        self.evaluation_dataset = self.evaluation_dataset.astype(cast_cols)
         if self.params.codelist_code in self.evaluation_dataset.columns:
             result = self.evaluation_dataset.merge(
                 ct_df.data,
diff --git a/cdisc_rules_engine/services/data_readers/csv_reader.py b/cdisc_rules_engine/services/data_readers/csv_reader.py
@@ -1,6 +1,6 @@
 import tempfile
 
-from dask.dataframe import dd
+import dask.dataframe as dd
 
 from cdisc_rules_engine.exceptions.custom_exceptions import InvalidCSVFile
 from cdisc_rules_engine.interfaces import DataReaderInterface
diff --git a/cdisc_rules_engine/services/data_services/usdm_data_service.py b/cdisc_rules_engine/services/data_services/usdm_data_service.py
@@ -399,8 +399,19 @@ def __read_node_metadata(
         }
 
     @staticmethod
-    def __get_full_path(node: DatumInContext):
-        return f"{node.full_path}".replace(".[", "[")
+    def __get_full_path(node: DatumInContext) -> str:
+        parts = []
+        current = node
+        while current is not None and current.context is not None:
+            parts.append(str(current.path))
+            current = current.context
+        result = ""
+        for part in reversed(parts):
+            if part.startswith("["):
+                result += part
+            else:
+                result = (result + "." if result else "") + part
+        return result
 
     def __get_datasets_content_index(self) -> List[dict]:
         """
diff --git a/cdisc_rules_engine/services/datasetxpt_metadata_reader.py b/cdisc_rules_engine/services/datasetxpt_metadata_reader.py
@@ -61,7 +61,7 @@ def read(self) -> dict:
             "variable_labels": list(metadata.column_labels),
             "variable_names": list(metadata.column_names),
             "variable_formats": [
-                "" if data_type == "NULL" else data_type
+                "" if (data_type == "NULL" or data_type is None) else data_type
                 for data_type in metadata.original_variable_types.values()
             ],
             "variable_name_to_label_map": metadata.column_names_to_labels,
diff --git a/cdisc_rules_engine/utilities/data_processor.py b/cdisc_rules_engine/utilities/data_processor.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 from typing import List, TYPE_CHECKING
+import dask.dataframe as dd
 
 from cdisc_rules_engine.models.dataset import PandasDataset, DaskDataset
 from cdisc_rules_engine.models.sdtm_dataset_metadata import SDTMDatasetMetadata
@@ -253,6 +254,10 @@ def merge_pivot_supp_dataset(
                     suffixes=("", "_supp"),
                 )
                 DataProcessor._validate_qnam_dask(left_dataset, qnam_list, common_keys)
+                left_dataset.data = dd.from_pandas(
+                    left_dataset.data.compute().reset_index(drop=True),
+                    npartitions=left_dataset.data.npartitions,
+                )
             else:
                 left_dataset = PandasDataset(
                     pd.merge(  # noqa
diff --git a/core.py b/core.py
@@ -357,15 +357,13 @@ def load_custom_dotenv_from_data_options(_ctx, _param, value):
     "-s",
     "--standard",
     required=True,
-    default=None,
     help="CDISC standard to validate against",
     envvar="PRODUCT",
 )
 @click.option(
     "-v",
     "--version",
     required=True,
-    default=None,
     help="Standard version to validate against",
     envvar="VERSION",
 )
diff --git a/docs/contributing.md b/docs/contributing.md
@@ -41,8 +41,7 @@ This project enforces consistent formatting and linting via pre-commit hooks.
 - [`markdownlint-cli2`](https://github.com/DavidAnson/markdownlint-cli2) — Markdown linter
 - [`check-json`](https://github.com/pre-commit/pre-commit-hooks) — JSON syntax validator
 
-All tools except `markdownlint-cli2` are included in `requirements-dev.txt`.
-After installing dependencies, install the pre-commit hooks:
+Install the dev dependencies (into your venv) and the pre-commit hooks:
 
 ```bash
 pre-commit install
diff --git a/docs/development.md b/docs/development.md
@@ -48,8 +48,9 @@ source venv/bin/activate
 # Activate (Windows)
 .\venv\Scripts\Activate
 
-# Install dependencies
-python -m pip install -r requirements-dev.txt
+# Install dependencies and the package
+pip install --upgrade pip
+pip install . --group dev
 ```
 
 ---
diff --git a/docs/quick-start.md b/docs/quick-start.md
@@ -134,7 +134,8 @@ python -m venv venv
 ### 3. Install Dependencies
 
 ```bash
-python -m pip install -r requirements-dev.txt
+pip install --upgrade pip
+pip install . --group dev
 ```
 
 ### 4. Populate the Cache
diff --git a/pyproject.toml b/pyproject.toml
diff --git a/requirements-dev.txt b/requirements-dev.txt
diff --git a/requirements-rule-tester.txt b/requirements-rule-tester.txt
diff --git a/requirements.txt b/requirements.txt
diff --git a/tests/unit/test_dataset_builders/test_dataset_metadata_define_dataset_builder.py b/tests/unit/test_dataset_builders/test_dataset_metadata_define_dataset_builder.py
