Fix GetPropertyValue to support nested partition key paths

[Copilot]

The GetPropertyValue method fails when partition keys reference nested object properties (e.g., /partitionkeyvalue1/partitionkeyvalue2). It attempts a direct dictionary lookup using the full path as a key instead of traversing the nested structure.

Changes

• Modified GetPropertyValue in CosmosDataSinkExtension.cs

  • Split path on / and traverse nested ExpandoObjects sequentially
  • Return null gracefully for missing properties or null intermediate values
  • Preserve backward compatibility for simple property paths

• Added test coverage in CosmosDataSinkExtensionTests.cs

  • Simple property access (existing behavior)
  • Nested property navigation (2 and 4 levels deep)
  • Null handling (missing properties, null intermediates)
  • Extracted reflection helper to reduce duplication

Example

Given a document:

{
  "id": "123",
  "partitionkeyvalue1": {
    "partitionkeyvalue2": "abc-guid",
    "otherField": "data"
  }
}

With partition key path /partitionkeyvalue1/partitionkeyvalue2, the method now correctly navigates to item["partitionkeyvalue1"]["partitionkeyvalue2"] and returns "abc-guid" instead of throwing KeyNotFoundException.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Cosmos Data Extension GetPropertyValue Fails with Nested Partition Key Path</issue_title>
<issue_description>When transferring from Azure Blob Json source to Cosmos DB sink, the following error occurs if partition key path on destination DB is:

/partitionkeyvalue1/partitionkeyvalue2

Sink Settings are

 "SinkSettings": {
                "Container": "CosmosContainer",
                "IncludeMetaDataFields": true,
                "PartitionKeyPath": "/partitionkeyvalue1/partitionkeyvalue2"
            }

Source Json is:

[
  {
    "_segments": [],
    "_executionLogs": [],
    "partitionkeyvalue1": {
		"partitionkeyvalue2": "<GUID>",
		"somevaule4": "<GUID>",
		"UserName": "<username>",
		"othervalue": "<data>",
		"othervalue3": "<data_2>"
    }
  }
]

This produces an error:

  Data transfer failed
    System.AggregateException: One or more errors occurred. (The specified key 'partitionkeyvalue1/partitionkeyvalue2' does not exist in the ExpandoObject.)
     ---> System.Collections.Generic.KeyNotFoundException: The specified key 'partitionkeyvalue1/partitionkeyvalue2' does not exist in the ExpandoObject.
       at System.Dynamic.ExpandoObject.System.Collections.Generic.IDictionary<System.String,System.Object>.get_Item(String key)
       at Cosmos.DataTransfer.CosmosExtension.CosmosDataSinkExtension.GetPropertyValue(ExpandoObject item, String propertyName) in /__w/data-migration-desktop-tool/data-migration-desktop-tool/Extensions/Cosmos/Cosmos.DataTransfer.CosmosExtension/CosmosDataSinkExtension.cs:line 211

This seems to be directly related to the GetPropertyValue method:

private static string? GetPropertyValue(ExpandoObject item, string propertyName)
{
    return ((IDictionary<string, object?>)item)[propertyName]?.ToString();
}

Is this something that can be addressed, or are we defining the partition key in a not best practice manner?</issue_description>

<agent_instructions>user is hitting an exception when defining keypath values that have "/" in them. Can you take a look at this issue and determine what's wrong. The issue contains exception information and a question on specific line of code believed to be causing the problem. Please throughly read the issue and look at official documentation, the come up with a plan to address this issue. Your solution must test the behavior as well. You might want to start with a test first that cinfirms what the issuer is seeing and then work on a fix.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@markjbrown Am I reading this right? You are defining a partition key that is an object?

This sort of thing is not allowed.

"partitionkeyvalue1": {
		"partitionkeyvalue2": "<GUID>",
		"somevaule4": "<GUID>",
		"UserName": "<username>",
		"othervalue": "<data>",
		"othervalue3": "<data_2>"
    }

Partition keys must be strings. Partition key definitions for a container are an array with 1-3 properties that form a hierarchy of values. Ideally this would be a hierarchy innate to the data itself, (e.g. /companyId, /departmentId, /employeeId) but technically it can be any string values.</comment_new>
<comment_new>@markjbrown
Can you send me the json output for the partition key definition from az cosmosdb sql container show for this container?

You can't create a container with a partition key like what I'm seeing here. I want to be sure I'm clear on your partition key definition.

Thanks.</comment_new>
<comment_new>@markjbrown
Ok got it :) You can't define a hierarchical partition key like this. This would only result in a single partition key of partitionkeyvalue2 with this path, "/partitionkeyvalue1/partitionkeyvalue2". Also the kind is incorrect for hierarchical and you need to specify version number as well. 2 is preferred for either type of partition key.

If you were defining a hierarchical partition key it would need to look like this.

"partitionKey": {
"kind": "MultiHash",
"paths": [
"/partitionkeyvalue1",
"/partitionkeyvalue2"
],
"version": 2
},

@markjbrown Ok, now I got it!!! Your property names were confusing me :)

My spidy senses make me think this is an escaping issue.

@bowencode do you have thoughts on this? Any chance you can try to repro?</comment_new>

• Fixes Cosmos Data Extension GetPropertyValue Fails with Nested Partition Key Path #175

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[philnach]

This looks good to me and I agree with the testing strategy.