Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 27 additions & 9 deletions docs/dqx/docs/guide/data_contract_quality_rules_generation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -196,16 +196,18 @@ Predefined rules are automatically derived from schema field constraints defined
properties:
- name: order_id
logicalType: string
physicalType: string
required: true # → is_not_null
unique: true # → is_unique
logicalTypeOptions:
pattern: '^ORD-[0-9]{8}$' # → regex_match

- name: customer_email
logicalType: string
physicalType: string
required: true # → is_not_null
logicalTypeOptions:
pattern: '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$' # → regex_match
pattern: '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$' # → regex_match

- name: order_total
logicalType: number
Expand All @@ -217,6 +219,7 @@ Predefined rules are automatically derived from schema field constraints defined

- name: order_status
logicalType: string
physicalType: string
required: true # → is_not_null
logicalTypeOptions:
pattern: '^(pending|confirmed|shipped|delivered|cancelled)$' # → regex_match
Expand All @@ -235,6 +238,7 @@ Predefined rules are automatically derived from schema field constraints defined
)

# Generated rules will include:
# - has_valid_schema (dataset-level, derived from the contract schema)
# - is_not_null for order_id, customer_email, order_total, order_status
# - is_unique for order_id
# - regex_match for order_id (pattern validation)
Expand Down Expand Up @@ -313,11 +317,13 @@ Explicit rules are defined in the `quality` section of a property or schema usin
properties:
- name: transaction_id
logicalType: string
physicalType: string
required: true
unique: true

- name: account_id
logicalType: string
physicalType: string
required: true
quality:
# Property-level explicit check
Expand Down Expand Up @@ -451,6 +457,7 @@ Text-based expectations are defined using `type: text` in the quality section:
properties:
- name: email
logicalType: string
physicalType: string
required: true
quality:
# Field-level text expectation
Expand All @@ -462,6 +469,7 @@ Text-based expectations are defined using `type: text` in the quality section:

- name: consent_date
logicalType: date
physicalType: date
required: true
quality:
# Another text expectation
Expand Down Expand Up @@ -529,9 +537,12 @@ Here's a complete example showing contract-based rule generation with all three
id: urn:datacontract:iot:sensor_readings
name: IoT Sensor Readings
version: 2.1.0
description: Quality contract for IoT sensor data
status: active
owner: IoT Platform Team
description:
purpose: Quality contract for IoT sensor data
team:
- username: iot-platform-team
role: owner

schema:
- name: sensor_readings
Expand Down Expand Up @@ -565,13 +576,15 @@ Here's a complete example showing contract-based rule generation with all three
properties:
- name: sensor_id
logicalType: string
physicalType: string
required: true # Predefined: is_not_null
unique: true # Predefined: is_unique
logicalTypeOptions:
pattern: '^SENSOR-[A-Z]{2}-[0-9]{4}$' # Predefined: regex_match

- name: temperature
logicalType: decimal
physicalType: decimal(5,2)
required: true # Predefined: is_not_null
logicalTypeOptions:
minimum: -40.0 # Predefined: sql_expression (range check)
Expand All @@ -585,12 +598,10 @@ Here's a complete example showing contract-based rule generation with all three

- name: status
logicalType: string
physicalType: string
required: true # Predefined: is_not_null
logicalTypeOptions:
enum: # Predefined: regex_match
- active
- maintenance
- offline
pattern: '^(active|maintenance|offline)$' # Predefined: regex_match
```
</TabItem>
<TabItem value="Python" label="Python Code">
Expand Down Expand Up @@ -766,23 +777,28 @@ id: urn:datacontract:marketing:customer_360
name: Customer 360 View
version: 1.2.0
status: active
owner: Marketing Analytics Team

team:
- username: marketing-analytics-team
role: owner

schema:
- name: customer_360
physicalType: table
properties:
- name: customer_id
logicalType: string
physicalType: string
required: true
unique: true
- name: email
logicalType: string
physicalType: string
required: true
logicalTypeOptions:
pattern: '^[a-zA-Z0-9._%+-]+@.+$'
- name: lifetime_value
logicalType: decimal
physicalType: decimal(18,2)
logicalTypeOptions:
minimum: 0
"""
Expand Down Expand Up @@ -813,9 +829,11 @@ schema:
properties:
- name: email
logicalType: string
physicalType: string
required: true
- name: consent_date
logicalType: date
physicalType: date
required: true
quality:
- type: text
Expand Down
Loading