Add WriteMode with SQL MERGE upsert support including optional DELETE to SQL Server extension

[Copilot]

Implements WriteMode configuration parameter for SQL Server sink to support upsert operations via SQL MERGE, including optional DELETE support for full table synchronization. Addresses requirement to sync source data with destination tables using insert-or-update semantics.

Changes

New Configuration Options

• WriteMode: Enum with Insert (default, preserves existing behavior) and Upsert (SQL MERGE)
• PrimaryKeyColumns: List of columns forming the primary key, required for Upsert mode
• DeleteNotMatchedBySource: Boolean flag to enable DELETE operations for full table sync (defaults to false)

Implementation

• Refactored WriteAsync into WriteInsertAsync (original logic) and WriteUpsertAsync (staging table + MERGE)
• Upsert uses temporary staging table for bulk load, then executes MERGE with WHEN MATCHED UPDATE and WHEN NOT MATCHED INSERT
• When DeleteNotMatchedBySource is true, MERGE includes WHEN NOT MATCHED BY SOURCE THEN DELETE for complete synchronization
• SQL identifier validation prevents injection via table/column names
• Validates at least one non-key column exists for UPDATE operations (unless DeleteNotMatchedBySource is true)
• Validates DeleteNotMatchedBySource can only be used with Upsert mode

Example Configuration (Upsert without DELETE)

{
  "SinkSettings": {
    "ConnectionString": "...",
    "TableName": "AccountTransactions",
    "WriteMode": "Upsert",
    "PrimaryKeyColumns": ["Id"],
    "ColumnMappings": [
      {"ColumnName": "Id"},
      {"ColumnName": "Amount", "DataType": "System.Decimal"}
    ]
  }
}

Example Configuration (Full Sync with DELETE)

{
  "SinkSettings": {
    "ConnectionString": "...",
    "TableName": "Products",
    "WriteMode": "Upsert",
    "PrimaryKeyColumns": ["ProductId"],
    "DeleteNotMatchedBySource": true,
    "ColumnMappings": [
      {"ColumnName": "ProductId"},
      {"ColumnName": "ProductName"},
      {"ColumnName": "Price", "DataType": "System.Decimal"}
    ]
  }
}

Sync Modes

The implementation supports three levels of synchronization:

1. Insert only (WriteMode=Insert): Only adds new records
2. Upsert (WriteMode=Upsert, DeleteNotMatchedBySource=false): Inserts new records and updates existing ones
3. Full Sync (WriteMode=Upsert, DeleteNotMatchedBySource=true): Complete synchronization including deletes

Important Notes

• DeleteNotMatchedBySource is opt-in (defaults to false) to prevent accidental data loss
• When enabled, records in the destination table that don't exist in the source will be permanently deleted
• Use full sync with caution and ensure proper backups are in place
• Slower than bulk insert due to staging table overhead
• Target table must exist with matching schema
• Concurrent updates may cause deadlocks

Testing

Added 14 unit tests covering validation, deserialization, composite key scenarios, and DELETE functionality. All existing tests pass (25 tests passed, 1 skipped).

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Feature Request - SQL Server - WriteMode</issue_title>
<issue_description>I would like to be able to update/insert/delete using the SQL Server extention.

Supporting a option called WriteMode, I would like to be able to upsert data into SQL Server.

Example

  "SinkSettings": 
  {
    "ConnectionString": "<sql connection String>",
    "Database": "PaymentService",
    "TableName": "AccountTransactions",
    "BatchSize": 1000,
    "WriteMode": "upsert",
    "ColumnMappings": [
      {
        "ColumnName": "Id",
        "SourceFieldName": "Id"
      }
    ]
  }
```</issue_description>

<agent_instructions>Users is asking for an upsert operation WriteMode as part of the SQL server extension.  Given the users requirements should provide a WriteMode operation that performs the SQL MERGE operation.  MERGE does the job of INSERT, UPDATE, DELETE, which essentially is syncing the destination with the source. This is also a tricky feature to implement due to the options available in the MERGE SQL statement.  We'll need to come up with a solution that is KISS and makes sense to Users.  I'd say for now the operation should sync the source with the destination and hopefully that addresses 80% of the use case scenarios.  Your change should also include tests to validate this new feature.  Please also identify any pitfalls with the solution.  You will need to update the readme files for the extension to include the new capability.</agent_instructions>

## Comments on the Issue (you are @copilot in this section)

<comments>
</comments>

• Fixes Feature Request - SQL Server - WriteMode #188

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[philnach]

This looks good to me.

[philnach]

@bowencode / @markjbrown, this looks good to me and is ready for review. The strategy here is to perform an upsert, which provides either insert / updating existing options and then there's the optional delete operation. This provides the complete option for "merging" sink with source. It does leverage a temp table as the intermediary, but that's reasonable given the potential amount of data. Copilot updated tests and documentation. Please take a look when you get a chance.