Allow deduplication of historical snapshops (#13)

* add dedup option for historical snapshots
* fix warning for keys_only dedup on snapshots
* exclude _metadata column from dedup logic for snapshots
* edit docstring for DeduplicateMode for cdc from snapshots

Author: haillew

VERSION

@@ -1 +1 @@
-v0.5.0
+v0.6.0

docs/source/dataflow_spec_ref_cdc.rst

@@ -87,6 +87,12 @@ The ``cdcSnapshotSettings`` object contains the following properties:
    * - **track_history_except_column_list**
      - ``list``
      - (*optional*) A subset of output columns to be excluded from history tracking in the target table. Use this to specify which columns should not be tracked. This cannot be used in conjunction with ``track_history_column_list``.
+   * - **deduplicateMode**
+     - ``string``
+     - (*optional*) How to deduplicate source snapshot data before CDC. Default: ``off`` (no deduplication). Use ``full_row`` to deduplicate based on the full row (deterministic) excluding the ``_metadata`` column if present. Use ``keys_only`` to keep the first row per key(s): This is non-deterministic; as it preserves the first row per key(s) without ordering on any other columns.
+
+  .. warning::
+     The ``keys_only`` option is **non-deterministic**. It preserves the first row per key(s). Use it with caution and only when you accept that which duplicate row is kept may vary between runs.
 
 .. _cdc-apply-changes-from-snapshot-source:
 

samples/bronze_sample/src/dataflows/feature_samples/dataflowspec/historical_snapshot_files_datetime_recursive_and_partitioned_parquet_main.json

@@ -23,7 +23,8 @@
             "path": "{sample_file_location}/snapshot_customer_partitioned_parquet/{version}/customer.parquet",
             "versionType": "timestamp",
             "datetimeFormat": "YEAR=%Y/MONTH=%m/DAY=%d",
-            "recursiveFileLookup": true
+            "recursiveFileLookup": true,
+            "deduplicateMode": "keys_only"
         },
         "track_history_except_column_list":[
             "LOAD_TIMESTAMP"