Document Azure Table API datetime filter format and add validation tests

[Copilot]

Users reported timestamp query filters producing empty results despite successful migration completion. The QueryFilter property requires specific OData datetime syntax with JSON escaping that was undocumented.

Changes

• Documentation (README.md, ExampleConfigs.md)

  • Added datetime filter examples with correct syntax
  • Documented JSON escaping requirement for single quotes (\u0027)
  • Clarified ISO 8601 format and datetime prefix requirement
  • Added troubleshooting table analyzing 15 query filter attempts from the original issue

• Tests (AzureTableAPIDataSourceExtensionTests.cs)

  • Added 8 tests validating JSON deserialization of datetime filters
  • Coverage for operators: eq, ge, lt, range queries, combined filters

Correct Format

{
  "QueryFilter": "Timestamp ge datetime\u00272023-05-15T03:30:32.663Z\u0027"
}

Key requirements:

• datetime prefix before timestamp value
• ISO 8601 format: YYYY-MM-DDTHH:mm:ss.fffZ
• Single quotes JSON-escaped as \u0027 (not literal ')

Common Mistakes

// ❌ Wrong: Missing datetime prefix
"QueryFilter": "Timestamp ge '2023-05-15T03:30:32.663Z'"

// ❌ Wrong: Unescaped quotes
"QueryFilter": "Timestamp ge datetime'2023-05-15T03:30:32.663Z'"

// ✅ Correct: datetime prefix + escaped quotes
"QueryFilter": "Timestamp ge datetime\u00272023-05-15T03:30:32.663Z\u0027"

Troubleshooting Table

The README now includes a comprehensive table that analyzes each of the 15 query filters attempted in the original issue, identifying specific problems with each attempt:

• Whether the datetime prefix is present or missing
• Whether the date format is correct (ISO 8601) or invalid
• Whether encoding is correct (JSON-escaped quotes) or has issues

This table helps users diagnose why their datetime filters may not be working and provides clear guidance on the correct format.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Azure Table API (Query Filter) For Timestamp</issue_title>
<issue_description>I want to ask about the query filter feature on the Azure Table API Source setting, I'm trying to transfer data from the Azure table API to a JSON file where the filter query uses the following timestamp query: "Timestamp ge datetime'2023-05-15T03:30:32.663Z"' after I run it already appears data transfer complete:

,but when looking at the json file nothing appears:

I want to make sure whether the feature for filtering queries for time already exists or not, or is there a writing error from what I input? thank you for your attention

the following additional notes are my migration settings:

</issue_description>

<agent_instructions>Take a look at this issue and Microsoft documentation for how to query the Azure Table APIs and specificially using DateTime. Look at the code and see if you can see any issues. Are there any gotchas in how the QueryFilter is specified in JSON that might be the issue. The query filter is a OData filter. Is it possible to build tests that can test this as well?</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@bowencode Queries can be tricky because of formatting. I think what you're seeing is probably a combination of the "eq" operator not getting an exact match or JSON encoding not actually reading the query as expected. Here's an example with a full time that matches a table timestamp and the JSON encoded characters: ```json "QueryFilter": "Timestamp eq datetime\u00272023-01-12T16:53:31.1714422Z\u0027" ``` With less precision on the query time it's probably better to use > and < to specify a range.

• Fixes Azure Table API (Query Filter) For Timestamp #58

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[philnach]

@copilot Should any of the queries worked that the issue file listed in the issue: #58 (comment)?

[Copilot]

@copilot Should any of the queries worked that the issue file listed in the issue: #58 (comment)?

No, the query shown in the issue would not have worked. The configuration had three problems:

1. Missing datetime prefix - had '26/07/2021...' instead of datetime'...'
2. Wrong date format - used 26/07/2021 10:31:05 instead of ISO 8601 format 2021-07-26T10:31:05Z
3. Incorrect encoding - used \u002B00:00 (which is +00:00) for timezone

The correct format would be:

"QueryFilter": "Timestamp eq datetime\u00272021-07-26T10:31:05Z\u0027"

All three issues needed to be fixed for the query to work.

[philnach]

@copilot In the PR can you specifically call out in a table format the issue for each of the lines called out in the original issue. Here's that text so you don't have to look it up:

"QueryFilter": "Timestamp ge datetime\u00272023-05-17T03:06:07.691Z\u0027" Success Transfer, Still no data in json file
"QueryFilter": "Timestamp le datetime\u00272023-05-17T03:06:07.691Z\u0027" Success Transfer, Still no data in json file
"QueryFilter": "Timestamp eq datetime\u00272023-05-17T03:06:07.691Z\u0027" Success Transfer, Still no data in json file
"QueryFilter": "Timestamp gt datetime\u00272023-05-17T03:06:07.691Z\u0027" Success Transfer, Still no data in json file
"QueryFilter": "Timestamp ge datetime '2023-05-17T03:06:07.691Z'" Success Transfer, no data in json file
"QueryFilter": "Timestamp le datetime '2023-05-17T03:06:07.691Z'" Success Transfer, no data in json file
"QueryFilter": "Timestamp gt datetime '2023-05-17T03:06:07.691Z'" Success Transfer, no data in json file
"QueryFilter": "Timestamp eq datetime '2023-05-17T03:06:07.691Z'" Success Transfer, no data in json file
"QueryFilter": "Timestamp ge datetime'\u00272023-05-17T03:06:07.691Z\u0027'" Success Transfer, Still no data in json file
"QueryFilter": "Timestamp eq \u00272023-05-17T03:06:07.691Z\u0027" Success Transfer, Still no data in json file
"QueryFilter": "Timestamp ge datetime '2023-05-17T03:10:39.058Z\u002B00:00'" Fail Transfer
"QueryFilter": "Timestamp ge datetime '2023-05-17T03:10:39.058Z\u002B00'" Fail Transfer
"QueryFilter": "Timestamp ge datetime 2023-05-17T03:10:39.058Z\u002B00" Fail Transfer
"QueryFilter": "Timestamp ge datetime'u00272023-05-17T03:06:07.691Zu0027'" Fail Transfer
"QueryFilter": "Timestamp eq '\u00272023-05-17T03:06:07.691Z\u0027'" Fail Transfer

Look at each line and turn it into table with columns for the 3 problems you see are typical. Missing datetime, date format (not ISO 8601) and incorrect encoding.

[Copilot]

@copilot In the PR can you specifically call out in a table format the issue for each of the lines called out in the original issue. Here's that text so you don't have to look it up:

"QueryFilter": ...

Added a comprehensive troubleshooting table to the README in commit 225fab4.

The table analyzes all 15 query filters from the issue with columns for:

• Missing datetime prefix
• Wrong date format (ISO 8601)
• Incorrect encoding

Key findings:

• First 4 queries were syntactically correct but returned no data (likely no entities matched the specific timestamp)
• Queries 5-9 had encoding issues (spaces, literal quotes instead of \u0027)
• Queries 10-15 had multiple issues (missing prefix, wrong encoding, invalid timezone format)

The table is in the "Troubleshooting Common DateTime Filter Issues" section of Extensions/AzureTableAPI/README.md.

[philnach]

Great to have a good extensive documentation and tests. I can see how it's easy to make a mistake when you are supplying the query through JSON.

[philnach]

@markjbrown / @bowencode , Copilot added some good documentation to clarify how to query with DateTime, which I'd imagine is a pretty common QueryFilter.

[markjbrown]

LGTM Thanks!