Implement semantic rule validation for DQ rules (#1169)

docs/dqx/docs/guide/quality_checks_apply.mdx

@@ -43,7 +43,7 @@ The end-to-end `apply_checks_and_save_in_table` and `apply_checks_by_metadata_an
 
 The engine ensures that the specified `column`, `columns`, `filter`, or sql 'expression' fields can be resolved in the input DataFrame. If any of these fields are invalid, the check evaluation is skipped, and the results include the check failure with a message identifying the invalid fields and `skipped=True` in the result struct. You can suppress these entries entirely or identify them downstream — see [Suppressing skipped check entries](/docs/guide/additional_configuration#suppressing-skipped-check-entries).
 The engine will raise an error if you try to apply checks with invalid definition (e.g. wrong syntax).
-In addition, you can also perform a standalone syntax validation of the checks as described [here](/docs/guide/quality_checks_definition#validating-syntax-of-quality-checks).
+In addition, you can also perform standalone validation of the checks as described [here](/docs/guide/quality_checks_definition#validating-quality-checks).
 
 You can apply quality checks to streaming pipelines using the same methods as for batch processing.
 You can either use the end-to-end methods or manage the input stream and output directly with native Spark APIs (e.g. `spark.readStream` and `writeStream`).

docs/dqx/docs/guide/quality_checks_definition.mdx

@@ -794,16 +794,24 @@ In addition to specifying variables during the load or save process, you can def
 
 For technical details and configuration examples, see [Default Variables](/docs/guide/additional_configuration#defining-default-variables-for-substitution) in the Additional Configuration guide.
 
-## Validating syntax of quality checks
+## Validating quality checks
 
-You can validate the syntax of checks loaded from a storage system or checks defined programmatically before applying them.
-This validation ensures that the checks are correctly defined and can be interpreted by the DQX engine.
+You can validate checks loaded from a storage system or checks defined declaratively (metadata) before applying them.
+DQX performs two complementary kinds of validation:
 
-The validation cannot be used for checks defined programmatically using DQX classes.
+- **Syntax (structural) validation** ensures each check is correctly defined and can be interpreted by the DQX engine,
+  for example that the function exists and the provided arguments match its signature. Structural problems are reported
+  as errors in the returned `ChecksValidationStatus`.
+- **Semantic (ruleset-level) validation** inspects the ruleset as a whole and detects:
+  - **Duplicate rules**: two rules with the same function, arguments, criticality and filter.
+  - **Conflicting rules**: two rules targeting the same function and column(s) but with different arguments
+    (e.g. two `is_in_range` checks on the same column with different thresholds).
+
+Validation cannot be used for checks defined programmatically using DQX classes.
 When checks are defined programmatically with DQX classes, syntax validation is unnecessary because the application will fail to interpret them if the DQX objects are constructed incorrectly.
 
 <Admonition type="tip" title="Usage tips">
-Validating quality rules are typically done as part of the CI/CD process to ensure checks are ready to use in the application.
+Validating quality rules is typically done as part of the CI/CD process to ensure checks are ready to use in the application.
 </Admonition>
 
 <Tabs>
@@ -842,3 +850,48 @@ Validating quality rules are typically done as part of the CI/CD process to ensu
     - 'checks_location': file or table location of the quality checks
   </TabItem>
 </Tabs>
+
+### Controlling semantic validation
+
+Semantic validation runs automatically inside `validate_checks`, `load_checks` and `save_checks`. Its behavior is
+controlled by the `semantic_validation_mode` parameter:
+
+| Mode | Behavior |
+| --- | --- |
+| `"warn"` (default) | Log a warning for each duplicate or conflicting rule and continue. |
+| `"fail"` | Raise a `ValueError` listing all issues found. |
+| `None` | Skip semantic validation entirely. |
+
+Duplicate and conflicting rules are not necessarily invalid (a duplicate is redundant, a conflict may be intentional),
+so the default mode only warns. Use `"fail"` to enforce a clean ruleset, for example in a CI/CD pipeline.
+
+<Admonition type="note" title="Limitations">
+Checks that use raw Spark SQL expressions (via the `sql_expression` function) are not deeply inspected — only
+structured metadata (function name, column, arguments, criticality and filter) is compared.
+</Admonition>
+
+```python
+import yaml
+from databricks.labs.dqx.engine import DQEngine
+from databricks.labs.dqx.checks_semantic_validator import ChecksSemanticValidationMode
+
+checks = yaml.safe_load("""
+- criticality: error
+  check:
+    function: is_not_null
+    arguments:
+      column: col1
+- criticality: error
+  check:
+    function: is_not_null
+    arguments:
+      column: col1
+""")
+
+# Fail fast on duplicate or conflicting rules (e.g. in CI/CD)
+DQEngine.validate_checks(checks, semantic_validation_mode=ChecksSemanticValidationMode.FAIL)
+
+# Skip semantic validation when loading checks
+engine = DQEngine(ws)
+engine.load_checks(config=..., semantic_validation_mode=None)
+```

docs/dqx/docs/guide/quality_checks_storage.mdx

@@ -222,8 +222,9 @@ If you create checks as a list of DQRule objects, you can convert them using the
 </Tabs>
 
 <Admonition type="tip" title="Checks validation">
-Wrong types, unknown arguments, and missing required check function parameters are reported by `DQEngine.validate_checks`.
-`load_checks` and `save_checks` (except delta storage) methods do not validate the returned/provided metadata. Call `validate_checks` after load / before save when you want to catch problems before apply (for example hand edited YAML/JSON or checks written to a table without going through DQX).
+Wrong types, unknown arguments, and missing required check function parameters (syntax validation) are reported by `DQEngine.validate_checks`.
+`load_checks` and `save_checks` do not run syntax validation on the returned/provided metadata, so call `validate_checks` after load / before save when you want to catch syntax problems before apply (for example hand edited YAML/JSON or checks written to a table without going through DQX).
+They do, however, run semantic (ruleset-level) validation to detect duplicate and conflicting rules; this is controlled by the `semantic_validation_mode` parameter (`"warn"` by default, `"fail"` to raise, or `None` to skip).
 For field semantics and validation details, see [Quality checks definition](/docs/guide/quality_checks_definition).
 </Admonition>