diff --git a/docs/PYPI.md b/docs/PYPI.md index 5c3df1521..a16ba1bf5 100644 --- a/docs/PYPI.md +++ b/docs/PYPI.md @@ -339,7 +339,7 @@ reporting_services = reporting_factory.get_report_services() **`column_prefix_map`** — maps the `--` variable prefix to the dataset domain (e.g. `{"--": "AE"}`), resolving placeholders like `--SEQ` → `AESEQ`. -**External dictionaries** — pass an `ExternalDictionariesContainer` to `RulesEngine` if validating rules that require MedDRA, WHODrug, LOINC, UNII, MedRT, or SNOMED. See the [External Dictionary Reference](https://cdisc-org.github.io/conformance-rules-editor/#/exdictionary). +**External dictionaries** — pass an `ExternalDictionariesContainer` to `RulesEngine` if validating rules that require MedDRA, WHODrug, LOINC, UNII, MedRT, or SNOMED. See the [External Dictionary Reference](https://cdisc-org.github.io/cdisc-open-rules/#/exdictionary). **Dask** — set `max_dataset_size=0` when initializing `DataServiceFactory` to force Dask processing for all datasets. diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 00db2c4c2..49a63e619 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -22,9 +22,6 @@ python core.py validate --help | `-s, --standard TEXT` | CDISC standard to validate against (e.g. `sdtmig`, `tig`). Also via `PRODUCT` env var. | | `-v, --version TEXT` | Standard version (e.g. `3-4`). Also via `VERSION` env var. | | `-ss, --substandard TEXT` | **Required for TIG.** One of `SDTM`, `SEND`, `ADaM`, `CDASH`. Also via `SUBSTANDARD` env var. | -| `-uc, --use-case TEXT` | Use Case for TIG Custom Domains. When performing a TIG validation with custom domain(s), this | -| | must be given to identify the custom domains' use case One of `INDH`, `PROD`, `NONCLIN`, | -| | `ANALYSIS`. Also via `USE_CASE` env var. | ### Dataset Input @@ -33,6 +30,9 @@ python core.py validate --help | `-d, --data TEXT` | Path to directory containing dataset files. Only the last value is used if specified multiple times. | | `-dp, --dataset-path TEXT` | Absolute path to a single dataset file. Can be specified multiple times. | | `-dxp, --define-xml-path TEXT` | Path to Define-XML. Also via `DEFINE_XML` env var. | +| `-uc, --use-case TEXT` | Use Case for TIG Custom Domains. When performing a TIG validation with custom domain(s), this | +| | must be given to identify the custom domains' use case One of `INDH`, `PROD`, `NONCLIN`, | +| | `ANALYSIS`. Also via `USE_CASE` env var. | | `-ft, --filetype TEXT` | File extension filter applied to the `-d` directory (e.g. `xpt`). Has higher priority than --dataset-path parameter | | `-e, --encoding TEXT` | File encoding for reading datasets (default: `utf-8`). Common values: `cp1252`, `latin-1`, `utf-16`. | | `-vcp, --variables-csv-path` | Path to `_variables.csv` when using multiple `-dp` paths across different folders. | diff --git a/docs/contributing.md b/docs/contributing.md index d53a3315e..2d029b694 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -12,7 +12,7 @@ Conformance rules are maintained separately in [`cdisc-open-rules`](https://gith - Report an issue with an existing rule's logic - Contribute a rule implementation -Please open an issue or pull request in that repository. Rule authoring can also be done through the hosted [CORE Rule Editor](https://cdisc-org.github.io/conformance-rules-editor). +Please open an issue or pull request in that repository. For questions about rule contribution workflows, post in [GitHub Discussions](https://github.com/cdisc-org/cdisc-rules-engine/discussions). diff --git a/docs/development.md b/docs/development.md index bb4c27427..1163b48fb 100644 --- a/docs/development.md +++ b/docs/development.md @@ -173,7 +173,7 @@ These are derived from the OpenAPI specs in [`cdisc-org/DDF-RA`](https://github. ## Dataset Format Reference (JSON) -When validating a single rule with `--local-rules`, JSON datasets must match the Dataset-JSON format used by the rule editor: +When validating a single rule with `--local-rules`, JSON datasets must match the json format used by the deprecated rule editor: ```json { diff --git a/resources/schema/rule-merged/Operations.json b/resources/schema/rule-merged/Operations.json index a7307fd5a..6bceb8ebc 100644 --- a/resources/schema/rule-merged/Operations.json +++ b/resources/schema/rule-merged/Operations.json @@ -117,7 +117,7 @@ "properties": { "operator": { "const": "get_codelist_attributes", - "markdownDescription": "\nFetches controlled terminology attribute values from CT packages based on row-specific CT package and version references. The operation constructs CT package names based on the standard being validated and the values in the `name` and `version` columns (e.g., SDTMIG \u2192 \"sdtmct-{version}\"). When the `name` column contains \"CDISC\" or \"CDISC CT\", it uses the validation run's standard to determine the package prefix and the version found in the cell of the specified column. The operation extracts all codes matching the specified ct_attribute from the package.\n\n**Required Parameters:**\n\n- `ct_attribute`: Attribute to extract - `\"Term CCODE\"`, `\"Codelist CCODE\"`, `\"Term Value\"`, `\"Codelist Value\"`, or `\"Term Preferred Term\"`\n- `name`: Column containing CT reference (e.g., \"TSVCDREF\") - identifies which terminology system is referenced\n- `version`: Column containing CT version (e.g., \"TSVCDVER\")\n\n```yaml\n- id: $VALID_TERM_CODES\n name: TSVCDREF\n operator: get_codelist_attributes\n ct_attribute: Term CCODE\n version: TSVCDVER\n```\n\n**Note:** due to editor not containing the cache, if using this operator in rule editor, you must put the ctpackage versions contained within your data in the library tab for it work properly in editor.\n" + "markdownDescription": "\nFetches controlled terminology attribute values from CT packages based on row-specific CT package and version references. The operation constructs CT package names based on the standard being validated and the values in the `name` and `version` columns (e.g., SDTMIG \u2192 \"sdtmct-{version}\"). When the `name` column contains \"CDISC\" or \"CDISC CT\", it uses the validation run's standard to determine the package prefix and the version found in the cell of the specified column. The operation extracts all codes matching the specified ct_attribute from the package.\n\n**Required Parameters:**\n\n- `ct_attribute`: Attribute to extract - `\"Term CCODE\"`, `\"Codelist CCODE\"`, `\"Term Value\"`, `\"Codelist Value\"`, or `\"Term Preferred Term\"`\n- `name`: Column containing CT reference (e.g., \"TSVCDREF\") - identifies which terminology system is referenced\n- `version`: Column containing CT version (e.g., \"TSVCDVER\")\n\n```yaml\n- id: $VALID_TERM_CODES\n name: TSVCDREF\n operator: get_codelist_attributes\n ct_attribute: Term CCODE\n version: TSVCDVER\n```\n\n**Note:** if using this operator with excel data, you must put the ctpackage versions contained within your data in the library tab for it work properly.\n" } }, "required": ["id", "operator", "name", "ct_attribute", "version"], diff --git a/resources/schema/rule/ExDictionary.md b/resources/schema/rule/ExDictionary.md index 8fa5dd1f2..915e35938 100644 --- a/resources/schema/rule/ExDictionary.md +++ b/resources/schema/rule/ExDictionary.md @@ -1,6 +1,6 @@ # External Dictionaries -External dictionaries provide standardized terminology and coding systems for medical terms, drugs, and laboratory tests. This guide covers how to integrate and validate external dictionaries in both the Command Line Interface (CLI) and Rule Editor. +External dictionaries provide standardized terminology and coding systems for medical terms, drugs, and laboratory tests. This guide covers how to integrate and validate external dictionaries in both the Command Line Interface (CLI). ## Supported Dictionaries @@ -123,7 +123,7 @@ Directory must contain the `Loinc.csv` with capital 'L' ``` -## Operations & Rule Editor +## Operations ### Dictionary Version Validation diff --git a/resources/schema/rule/Operations.md b/resources/schema/rule/Operations.md index 3cbdb837b..f5eb48cbd 100644 --- a/resources/schema/rule/Operations.md +++ b/resources/schema/rule/Operations.md @@ -168,7 +168,7 @@ Fetches controlled terminology attribute values from CT packages based on row-sp version: TSVCDVER ``` -**Note:** due to editor not containing the cache, if using this operator in rule editor, you must put the ctpackage versions contained within your data in the library tab for it work properly in editor. +**Note:** if using this operator with excel data, you must put the ctpackage versions contained within your data in the library tab for it work properly. ### valid_codelist_dates diff --git a/resources/schema/rule/customrules.md b/resources/schema/rule/customrules.md index dcc3798ce..92968558e 100644 --- a/resources/schema/rule/customrules.md +++ b/resources/schema/rule/customrules.md @@ -1,119 +1,22 @@ -# Custom Editor Columns and Custom Rules +# CDISC Custom Rule Extensions -This guide explains how to extend CDISC rule definitions with custom attributes and how to access these attributes using the custom columns editor syntax. +This guide explains how to extend CDISC rule definitions with your own custom attributes while maintaining compatibility with the core CDISC schema structure. ## Table of Contents -- [Custom Columns in Editor Syntax](#custom-columns-in-editor-syntax) - - [Overview and Basic Syntax](#overview-and-basic-syntax) - - [Identifying Arrays in YAML](#identifying-arrays-in-yaml) - - [Identifying Arrays in JSON](#identifying-arrays-in-json) - [Custom Attributes Overview](#overview) - [What Can and Cannot Be Changed](#what-can-and-cannot-be-changed) - [Custom Schema Attributes](#custom-attributes) -- [Custom Columns in Editor Syntax](#custom-columns-in-editor-syntax) - [Adding a Custom Organization](#adding-a-custom-organization) - [Example Rule with Custom Attributes](#example-rule-with-custom-attributes) -- [Filtering Rules with Custom Columns](#filtering-rules-with-custom-columns) - [Validation](#validation) - [Best Practices](#best-practices) - [FAQ](#faq) -## Custom Columns in Editor Syntax - -Custom columns allow you to query nested rule structures, including arrays, using a simple path syntax. This allows for better access and filtering for both standard and custom attributes. - -## Overview and Basic Syntax - -Custom columns allow you to query nested rule structures, including arrays, using a simple path syntax. All custom column paths use `.` to separate each nested object property of the rule and use `@` to denote array elements. - -The general rules for constructing paths: - -1. The path is **case-sensitive** -2. The `@` symbol must come directly before the array name -3. Array notation must be used to access array elements -4. The path must reflect the exact structure of your data - -### Identifying Arrays in YAML - -When viewing a YAML document, you can identify arrays by looking for these indicators: - -1. **Hyphen (-) at the Start**: Arrays elements are marked with a leading hyphen - -```yaml -Authorities: #This is an array - - Organization: "Org1" # This is an element within the array - Standards: #This is an array (within the Authorities Array) - - Name: "Standard1" # This is also an element within the array - - Name: "Standard2" # Another element within the array -``` - -This means the Authorities array contains elements with Organization and Standards Properties on them. Similarly, Standards is an array with each element in the array having a Name property. - -2. **No Hyphen**: Regular object properties don't have a hyphen - -```yaml -Core: - Id: "123" - Status: "Active" -``` - -### Example YAML with Path Mapping - -```yaml -Authorities: - - Organization: "Org1" # @Authorities.Organization - Standards: - - Name: "Standard1" # @Authorities.@Standards.Name -Core: - Id: "123" # Core.Id - Status: "Active" # Core.Status -``` - -The presence of hyphens (-) is your key indicator that you need to use @ notation in your custom path - -### Identifying Arrays in JSON - -You can also inspect the rule json looking for `[` square brackets to denote arrays. - -### Single Array Example - -For data structured like: - -```json -{ - "Check": { - "all": [{ "operator": "equal_to" }, { "operator": "not_equal_to" }] - } -} -``` - -Use: `Check.@all.operator` All is an array, containing objects with operator keys - -### Nested Arrays Example - -For data structured like: - -```json -{ - "Authorities": [ - { - "Organization": "Org1", - "Standards": [{ "Name": "Standard1" }, { "Name": "Standard2" }] - } - ] -} -``` - -Use: `@Authorities.@Standards.Name` - -# CDISC Custom Rule Extensions - -This guide explains how to extend CDISC rule definitions with your own custom attributes while maintaining compatibility with the core CDISC schema structure, and how to access these attributes using the custom columns editor syntax. ## Overview -The CDISC Rules Engine schema supports custom extensions to help organizations better categorize, manage, and filter rules. These extensions maintain compatibility with standard CDISC rule definitions while adding organizational metadata tailored to your specific needs. The custom columns editor syntax allows you to query and filter these properties using a simple path notation. +The CDISC Rules Engine schema supports custom extensions to help organizations better categorize, manage, and filter rules. These extensions maintain compatibility with standard CDISC rule definitions while adding organizational metadata tailored to your specific needs. ## What Can and Cannot Be Changed @@ -300,14 +203,6 @@ If you want to formally define custom properties in your schema (recommended for - Provide validation rules or constraints for each property - Share the documentation with all teams that will be using the rules -### Accessing Custom Properties - -To access your custom properties in the editor using custom columns: - -- Simple property: `Category.MyCustomProperty` -- Nested property: `Category.ReviewInfo.LastReviewer` -- Array property: `Category.@ValidationHistory.Result` - ## Example Rule with Custom Attributes Here's a complete example of a rule with custom attributes in the Category section: @@ -374,39 +269,6 @@ Scope: Sensitivity: Record ``` -## Filtering Rules with Custom Columns - -The custom columns editor syntax allows you to create powerful filters based on your custom attributes. Here are some common filtering scenarios: - -### Setting Up Custom Columns - -1. Add a custom column using the column selector -2. Enter the path to the attribute you want to display/filter -3. Set the column name - -### Filter Examples - -| To Filter By | Custom Column Path | Filter Value Example | -| ---------------------- | -------------------------------------------------------- | --------------------- | -| Sponsor | `@Authorities.Category.@Sponsors` | "Sponsor A" | -| Specific vendor | `@Authorities.Category.@Vendors` | "CRO B" | -| Therapeutic area | `@Authorities.Category.@TherapeuticAreas` | "Oncology" | -| Trial | `@Authorities.Category.@Trials` | "ONC-2025-01" | -| Purpose | `@Authorities.Category.Purpose` | "RAW data validation" | -| Keyword | `@Authorities.Category.@Keywords` | "mykeyword" | -| Custom property | `@Authorities.Category.Department` | "Clinical Data Mgmt" | -| Authority organization | `@Authorities.Organization` | "MyCompany" | -| Standard name | `@Authorities.@Standards.Name` | "Internal Standard" | -| Rule ID | `@Authorities.@Standards.@References.Rule_Identifier.Id` | "INT001" | - -### Complex Filtering - -Combine multiple custom columns to create complex filters: - -1. Add column for `@Authorities.Category.@TherapeuticAreas` and filter for "Oncology" -2. Add column for `@Authorities.Category.Purpose` and filter for "RAW data validation" -3. Add column for `@Authorities.Organization` and filter for "MyCompany" - ## Best Practices When adding custom attributes to rules: @@ -416,7 +278,6 @@ When adding custom attributes to rules: 3. **Use controlled vocabularies** - For fields like TherapeuticAreas, maintain a list of standard terms 4. **Consider hierarchies** - Categorize using hierarchical structures when appropriate 5. **Avoid redundancy** - Don't duplicate information already in core CDISC properties -6. **Path naming** - When adding custom columns, use clear names that indicate the data being displayed ## Validation