Merge pull request #133 from datacontract/develop/1.2.1

jochenchrist · web-flow · commit ec470c03cd85 · 2025-09-24T09:35:50.000+02:00
Develop/1.2.1
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+
+## [1.2.1] - 2025-09-24
+
+### Added
+
+- Support for data quality metrics that align with ODCS 3.1
+
+### Changed
+- Replaced threshold operators mustBeGreaterThanOrEqualTo with mustBeGreaterOrEqualTo and mustBeLessThanOrEqualTo with mustBeLessOrEqualTo to align with ODCS 3.1, even if it feels wrong...
+
 ## [1.2.0] - 2025-07-05
 
 ### Added
diff --git a/README.md b/README.md
@@ -35,15 +35,15 @@ The specification comes along with the [Data Contract CLI](https://github.com/da
 Version
 ---
 
-1.2.0([Changelog](CHANGELOG.md))
+1.2.1([Changelog](CHANGELOG.md))
 
 Example
 ---
 
 View in [Data Contract Catalog](https://datacontract.com/examples/index.html)
 
 ```yaml
-dataContractSpecification: 1.2.0
+dataContractSpecification: 1.2.1
 id: orders-latest
 info:
   title: Orders Latest
@@ -848,16 +848,17 @@ Data can be verified by executing these checks through a data quality engine.
 
 Quality attributes can be:
 - A text in natural language that describes the quality of the data.
+- A predefined metric from the library of commonly used metrics
 - An individual SQL query that returns a single value that can be compared.
 - Engine-specific types: Pre-defined quality checks, as defined by data quality libraries. Currently, the engines `soda` and `great-expectations` are supported.
 
-A quality object can be specified on field level and on model level. 
+A quality object can be specified on the field level and on the model level. 
 The top-level quality object is deprecated.
 
 #### Description Text
 
 A description in natural language that defines the expected quality of the data. 
-This is useful to express requirements or expectation when discussing the data contract with stakeholders.
+This is useful to express requirements or expectations when discussing the data contract with stakeholders.
 Later in the development process, these might be translated into an executable check (such as `sql`).
 It can also be used as a prompt to check the data with an AI engine.
 
@@ -929,6 +930,54 @@ models:
 SQL queries allow powerful checks for custom business logic. 
 A SQL query should run not longer than 10 minutes.
 
+#### Library / Metrics
+
+A set of predefined metrics commonly used in data quality checks, designed to be compatible with all major data quality engines. This simplifies the work for data engineers by eliminating the need to manually write SQL queries.
+These metrics are aligned with ODCS 3.1.
+
+| Field                  | Type                  | Description                                                                      |
+|------------------------|-----------------------|----------------------------------------------------------------------------------|
+| type                   | `string`              | `library` (can be omitted, if `metric` is defined)                               |
+| metric                 | `string`              | `nullValues`, `missingValues`, `invalidValues`, `duplicateValues`, or `rowCount` |
+| arguments              | `object`              | Some metrics require additional arguments                                        |
+| description            | `string`              | A plain text describing the quality of the data.                                 |
+| mustBe                 | `integer`             | The threshold to check the return value of the query                             |
+| mustNotBe              | `integer`             | The threshold to check the return value of the query                             |
+| mustBeGreaterThan      | `integer`             | The threshold to check the return value of the query                             |
+| mustBeGreaterOrEqualTo | `integer`             | The threshold to check the return value of the query                             |
+| mustBeLessThan         | `integer`             | The threshold to check the return value of the query                             |
+| mustBeLessOrEqualTo    | `integer`             | The threshold to check the return value of the query                             |
+| mustBeBetween          | array of two integers | The threshold to check the return value of the query. Boundaries are inclusive.  |
+| mustNotBeBetween       | array of two integers | The threshold to check the return value of the query. Boundaries are inclusive.  |
+| unit                   | `string`              | `rows` (default) or `percent`                                                    |
+
+
+Metrics:
+
+| Metric | Level | Description                                                    | Arguments                                                        | Arguments Example                                                    |
+|--------|--------|----------------------------------------------------------------|------------------------------------------------------------------|----------------------------------------------------------------------|
+| `nullValues` | Property | Counts null values in a column/field                           | None                                                             |                                                                      |
+| `missingValues` | Property | Counts values considered as missing (empty strings, N/A, etc.) | `missingValues`: Array of values considered missing              | `missingValues: [null, '', 'N/A']`                                   |
+| `invalidValues` | Property | Counts values that don't match valid criteria                  | `validValues`: Array of valid values<br>`pattern`: Regex pattern | `validValues: ['pounds', 'kg']`<br>`pattern: '^[A-Z]{2}[0-9]{2}...'` |
+| `duplicateValues` | Property | Counts duplicate values in a column                            | None                                                             |                                                                      |
+| `duplicateValues` | Schema | Counts duplicate values across multiple columns                | `properties`: Array of property names                            | `properties: ['tenant_id', 'order_id']`                              |
+| `rowCount` | Schema | Counts total number of rows in a table/object store            | None                                                             |                                                                      |
+
+
+Example:
+
+```yaml
+properties:
+  - name: email_address
+    quality:
+      - metric: missingValues
+        arguments:
+          missingValues: [null, '', 'N/A', 'n/a']
+        mustBeLessThan: 5
+        unit: percent # rows (default) or percent
+```
+
+
 #### Custom
 
 You can define custom quality attributes that are specific to a data quality engine.
diff --git a/datacontract.init.yaml b/datacontract.init.yaml
@@ -1,4 +1,4 @@
-dataContractSpecification: 1.2.0
+dataContractSpecification: 1.2.1
 id: my-data-contract-id
 info:
   title: My Data Contract
diff --git a/datacontract.schema.json b/datacontract.schema.json
@@ -7,6 +7,7 @@
       "type": "string",
       "title": "DataContractSpecificationVersion",
       "enum": [
+        "1.2.1",
         "1.2.0",
         "1.1.0",
         "0.9.3",
@@ -1816,13 +1817,21 @@
               "mustBeGreaterThan": {
                 "type": "number"
               },
-              "mustBeGreaterThanOrEqualTo": {
+              "mustBeGreaterOrEqualTo": {
                 "type": "number"
               },
+              "mustBeGreaterThanOrEqualTo": {
+                "type": "number",
+                "deprecated": true
+              },
               "mustBeLessThan": {
                 "type": "number"
               },
               "mustBeLessThanOrEqualTo": {
+                "type": "number",
+                "deprecated": true
+              },
+              "mustBeLessOrEqualTo": {
                 "type": "number"
               },
               "mustBeBetween": {
@@ -1849,18 +1858,42 @@
         },
         {
           "if": {
-            "properties": {
-              "type": {
-                "const": "library"
+            "anyOf": [
+              {
+                "properties": {
+                  "type": {
+                    "const": "library"
+                  }
+                }
+              },
+              {
+                "properties": {
+                  "metric": {
+                    "type": "string"
+                  }
+                },
+                "required": ["metric"]
               }
-            }
+            ]
           },
           "then": {
             "properties": {
+              "metric": {
+                "type": "string",
+                "description": "The DataQualityLibrary metric to use for the quality check.",
+                "enum": ["nullValues", "missingValues", "invalidValues", "duplicateValues", "rowCount"]
+              },
               "rule": {
                 "type": "string",
-                "description": "Define a data quality check based on the predefined rules as per ODCS.",
-                "examples": ["duplicateCount", "validValues", "rowCount"]
+                "deprecated": true,
+                "description": "Deprecated. Use metric instead"
+              },
+              "arguments": {
+                "type": "object",
+                "description": "Additional metric-specific parameters for the quality check.",
+                "additionalProperties": {
+                  "type": ["string", "number", "boolean", "array", "object"]
+                }
               },
               "mustBe": {
                 "description": "Must be equal to the value to be valid. When using numbers, it is equivalent to '='."
@@ -1906,7 +1939,7 @@
               }
             },
             "required": [
-              "rule"
+              "metric"
             ]
           }
         },
diff --git a/examples/orders-latest/datacontract.yaml b/examples/orders-latest/datacontract.yaml
@@ -1,4 +1,4 @@
-dataContractSpecification: 1.2.0
+dataContractSpecification: 1.2.1
 id: orders-latest
 info:
   title: Orders Latest
diff --git a/examples/time-example/datacontract.html b/examples/time-example/datacontract.html
@@ -3,7 +3,7 @@
 <head>
     <meta charset="UTF-8">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <title>Time Data Type Example - Data Contract Specification v1.2.0</title>
+    <title>Time Data Type Example - Data Contract Specification v1.2.1</title>
     <style>
         body {
             font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
@@ -106,12 +106,12 @@
 </head>
 <body>
     <div class="container">
-        <div class="version-badge">Data Contract Specification v1.2.0</div>
+        <div class="version-badge">Data Contract Specification v1.2.1</div>
         <h1>Time Data Type Example</h1>
         
         <div class="info-section">
             <h2>Overview</h2>
-            <p>This example demonstrates the usage of the new <strong>time</strong> data type introduced in Data Contract Specification v1.2.0. The time data type is specifically designed for storing time values without date information, making it perfect for business hours, schedules, and time-based data.</p>
+            <p>This example demonstrates the usage of the new <strong>time</strong> data type introduced in Data Contract Specification v1.2.1. The time data type is specifically designed for storing time values without date information, making it perfect for business hours, schedules, and time-based data.</p>
         </div>
 
         <div class="warning">
diff --git a/examples/time-example/datacontract.yaml b/examples/time-example/datacontract.yaml
@@ -1,11 +1,11 @@
-dataContractSpecification: 1.2.0
+dataContractSpecification: 1.2.1
 id: time-demo
 info:
   title: Time Data Type Example
   version: 1.0.0
   description: |
     Example demonstrating the usage of the time data type
-    introduced in Data Contract Specification v1.2.0.
+    introduced in Data Contract Specification v1.2.1.
   owner: Data Contract Team
   contact:
     name: Data Contract Team
@@ -127,6 +127,6 @@ models:
 tags:
   - example
   - time
-  - v1.2.0
+  - v1.2.1
 links:
   datacontractCli: https://cli.datacontract.com 
diff --git a/examples/variant-json-example/datacontract.yaml b/examples/variant-json-example/datacontract.yaml
@@ -1,11 +1,11 @@
-dataContractSpecification: 1.2.0
+dataContractSpecification: 1.2.1
 id: variant-json-demo
 info:
   title: Variant and JSON Data Types Example
   version: 1.0.0
   description: |
     Example demonstrating the usage of variant and json data types
-    introduced in Data Contract Specification v1.2.0.
+    introduced in Data Contract Specification v1.2.1.
   owner: Data Contract Team
   contact:
     name: Data Contract Team
@@ -70,6 +70,6 @@ tags:
   - example
   - variant
   - json
-  - v1.2.0
+  - v1.2.1
 links:
   datacontractCli: https://cli.datacontract.com 
diff --git a/versions/1.2.1/datacontract.init.yaml b/versions/1.2.1/datacontract.init.yaml
@@ -0,0 +1,91 @@
+dataContractSpecification: 1.2.0
+id: my-data-contract-id
+info:
+  title: My Data Contract
+  version: 0.0.1
+#  description:
+#  owner:
+#  contact:
+#    name:
+#    url:
+#    email:
+
+
+### servers
+
+#servers:
+#  production:
+#    type: s3
+#    location: s3://
+#    format: parquet
+#    delimiter: new_line
+
+### terms
+
+#terms:
+#  usage:
+#  limitations:
+#  billing:
+#  noticePeriod:
+
+
+### models
+
+# models:
+#   my_model:
+#     description:
+#     type:
+#     fields:
+#       my_field:
+#         type:
+#         description:
+
+
+### definitions
+
+# definitions:
+#   my_field:
+#     domain:
+#     name:
+#     title:
+#     type:
+#     description:
+#     example:
+#     pii:
+#     classification:
+
+
+### servicelevels
+
+#servicelevels:
+#  availability:
+#    description: The server is available during support hours
+#    percentage: 99.9%
+#  retention:
+#    description: Data is retained for one year because!
+#    period: P1Y
+#    unlimited: false
+#  latency:
+#    description: Data is available within 25 hours after the order was placed
+#    threshold: 25h
+#    sourceTimestampField: orders.order_timestamp
+#    processedTimestampField: orders.processed_timestamp
+#  freshness:
+#    description: The age of the youngest row in a table.
+#    threshold: 25h
+#    timestampField: orders.order_timestamp
+#  frequency:
+#    description: Data is delivered once a day
+#    type: batch # or streaming
+#    interval: daily # for batch, either or cron
+#    cron: 0 0 * * * # for batch, either or interval
+#  support:
+#    description: The data is available during typical business hours at headquarters
+#    time: 9am to 5pm in EST on business days
+#    responseTime: 1h
+#  backup:
+#    description: Data is backed up once a week, every Sunday at 0:00 UTC.
+#    interval: weekly
+#    cron: 0 0 * * 0
+#    recoveryTime: 24 hours
+#    recoveryPoint: 1 week
diff --git a/versions/1.2.1/datacontract.schema.json b/versions/1.2.1/datacontract.schema.json
diff --git a/versions/1.2.1/definition.schema.json b/versions/1.2.1/definition.schema.json

Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,4 @@`
`1`		`-dataContractSpecification: 1.2.0`
	`1`	`+dataContractSpecification: 1.2.1`
`2`	`2`	`id: my-data-contract-id`
`3`	`3`	`info:`
`4`	`4`	`title: My Data Contract`