From a2f9595dfcae114774e838fa6aa088651375b39e Mon Sep 17 00:00:00 2001 From: Michael Vanden Heuvel Date: Tue, 28 Jul 2026 09:41:21 -0700 Subject: [PATCH 1/3] Update ASIM skills for entity placeholders Require generated and reviewed parsers to emit empty entityKey, AdditionalIds, and additionalEntities fields until mappings are defined. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f8c570d9-99f0-4316-af2d-d91b70cce8e4 --- .github/skills/asim-parser-create-parser/SKILL.md | 7 +++++++ .github/skills/asim-parser-pr-reviewer/SKILL.md | 3 +++ 2 files changed, 10 insertions(+) diff --git a/.github/skills/asim-parser-create-parser/SKILL.md b/.github/skills/asim-parser-create-parser/SKILL.md index a444b119aeb..ee4a59c6bdb 100644 --- a/.github/skills/asim-parser-create-parser/SKILL.md +++ b/.github/skills/asim-parser-create-parser/SKILL.md @@ -61,6 +61,13 @@ Parsers are KQL functions that follow a clear flow: **Filter → Parse → Map** ### Required columns - Include the column `Type` in the output of the ASIM parser. This column indicates the source table name. +- Include `entityKey`, `AdditionalIds`, and `additionalEntities` in the parser output. Until mappings for these fields are defined, set `entityKey` to an empty string and both `AdditionalIds` and `additionalEntities` to empty dynamic arrays: + + ```kusto + entityKey = "", + AdditionalIds = dynamic([]), + additionalEntities = dynamic([]) + ``` ### Required parameters diff --git a/.github/skills/asim-parser-pr-reviewer/SKILL.md b/.github/skills/asim-parser-pr-reviewer/SKILL.md index 070145b2165..20887ccec87 100644 --- a/.github/skills/asim-parser-pr-reviewer/SKILL.md +++ b/.github/skills/asim-parser-pr-reviewer/SKILL.md @@ -46,6 +46,8 @@ Review the parameter-less parser for the following: - **`pack` parameter**: If the query uses `AdditionalFields`, verify that a `pack: bool = false` parameter is included. This allows users to choose whether to populate `AdditionalFields` or return an empty dynamic, improving performance for users who do not need the extra information. +- **Placeholder entity fields**: Verify that the parser outputs `entityKey = ""`, `AdditionalIds = dynamic([])`, and `additionalEntities = dynamic([])`. These fields must remain empty until mappings for them are defined. + - **Parsing operator efficiency**: Check that high-performance parsing operators are used (`split`, `parse-kv`, `parse`) and that regular expressions are avoided where simpler operators would work. - **General KQL performance**: Flag any other inefficient patterns such as unnecessary `let` statements, redundant filters, expensive joins, or operations that could be reordered for better performance. @@ -81,6 +83,7 @@ Please review: 2. **Filter efficiency**: Are the parameter-based filters using native columns and indexed fields where possible? 3. **Redundant computation**: Are there any calculated fields or parsing operations that occur before the parameter filters, when they could be moved after? 4. **Parameter completeness**: Are the filtering parameters comprehensive enough to allow efficient querying for common use cases? +5. **Placeholder entity fields**: Does the parameterized parser preserve `entityKey = ""`, `AdditionalIds = dynamic([])`, and `additionalEntities = dynamic([])`? Flag this only when the fields are present in the parameter-less parser but missing or populated in the parameterized parser, so the same issue is not reported twice. **Output format:** From bcfa8bb9f4968e9d27235f73d89c87f312ef6dab Mon Sep 17 00:00:00 2001 From: Michael Vanden Heuvel Date: Wed, 29 Jul 2026 10:40:14 -0700 Subject: [PATCH 2/3] Make ASIM entity placeholders schema aware Derive role-specific entity placeholder fields from the target and Common schemas, preserve exact casing, and clarify review behavior across parser variants. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f8c570d9-99f0-4316-af2d-d91b70cce8e4 --- .github/skills/asim-parser-create-parser/SKILL.md | 11 +++++++---- .github/skills/asim-parser-pr-reviewer/SKILL.md | 4 ++-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/.github/skills/asim-parser-create-parser/SKILL.md b/.github/skills/asim-parser-create-parser/SKILL.md index ee4a59c6bdb..938e2bebdc2 100644 --- a/.github/skills/asim-parser-create-parser/SKILL.md +++ b/.github/skills/asim-parser-create-parser/SKILL.md @@ -61,12 +61,15 @@ Parsers are KQL functions that follow a clear flow: **Filter → Parse → Map** ### Required columns - Include the column `Type` in the output of the ASIM parser. This column indicates the source table name. -- Include `entityKey`, `AdditionalIds`, and `additionalEntities` in the parser output. Until mappings for these fields are defined, set `entityKey` to an empty string and both `AdditionalIds` and `additionalEntities` to empty dynamic arrays: +- Determine the placeholder entity fields applicable to the parser from the target schema and `Common` rows in `ASimTester.csv`. Include every defined column whose name ends in `EntityKey` or `AdditionalIds`, as well as `AdditionalEntities` when it is defined for the schema. +- Preserve the exact field names and casing from `ASimTester.csv`. Entity fields are role-specific (for example, `ActorUserEntityKey`, `SrcSystemEntityKey`, and `ActorUserAdditionalIds`); do not replace them with generic names such as `entityKey` or `AdditionalIds`. +- Until mappings for these fields are defined, set every `*EntityKey` field to an empty string and every `*AdditionalIds` field and `AdditionalEntities` to an empty dynamic array. For example: ```kusto - entityKey = "", - AdditionalIds = dynamic([]), - additionalEntities = dynamic([]) + ActorUserEntityKey = "", + SrcSystemEntityKey = "", + ActorUserAdditionalIds = dynamic([]), + AdditionalEntities = dynamic([]) ``` ### Required parameters diff --git a/.github/skills/asim-parser-pr-reviewer/SKILL.md b/.github/skills/asim-parser-pr-reviewer/SKILL.md index 20887ccec87..59dd5a15dfe 100644 --- a/.github/skills/asim-parser-pr-reviewer/SKILL.md +++ b/.github/skills/asim-parser-pr-reviewer/SKILL.md @@ -46,7 +46,7 @@ Review the parameter-less parser for the following: - **`pack` parameter**: If the query uses `AdditionalFields`, verify that a `pack: bool = false` parameter is included. This allows users to choose whether to populate `AdditionalFields` or return an empty dynamic, improving performance for users who do not need the extra information. -- **Placeholder entity fields**: Verify that the parser outputs `entityKey = ""`, `AdditionalIds = dynamic([])`, and `additionalEntities = dynamic([])`. These fields must remain empty until mappings for them are defined. +- **Placeholder entity fields**: Determine the complete set of applicable fields from the target schema and `Common` rows in `ASimTester.csv`. Verify that the parser includes every defined `*EntityKey` and `*AdditionalIds` field, plus `AdditionalEntities` when defined. Names and casing must exactly match `ASimTester.csv`; generic names such as `entityKey` or `AdditionalIds` are invalid. Every `*EntityKey` must be an empty string, and every `*AdditionalIds` field and `AdditionalEntities` must be an empty dynamic array until mappings are defined. - **Parsing operator efficiency**: Check that high-performance parsing operators are used (`split`, `parse-kv`, `parse`) and that regular expressions are avoided where simpler operators would work. @@ -83,7 +83,7 @@ Please review: 2. **Filter efficiency**: Are the parameter-based filters using native columns and indexed fields where possible? 3. **Redundant computation**: Are there any calculated fields or parsing operations that occur before the parameter filters, when they could be moved after? 4. **Parameter completeness**: Are the filtering parameters comprehensive enough to allow efficient querying for common use cases? -5. **Placeholder entity fields**: Does the parameterized parser preserve `entityKey = ""`, `AdditionalIds = dynamic([])`, and `additionalEntities = dynamic([])`? Flag this only when the fields are present in the parameter-less parser but missing or populated in the parameterized parser, so the same issue is not reported twice. +5. **Placeholder entity field consistency**: Compare the parameterized parser's placeholder fields and values with the parameter-less parser. Report fields missing from both parsers only in the parameter-less review above. In this section, report only differences introduced by the parameterized parser, such as a placeholder that is omitted, renamed, or populated when the parameter-less parser defines it correctly. Matching omissions are not acceptable; they are already reported in the parameter-less review. **Output format:** From 93141dda79ba82f96e93eb54f1461fc5381773b4 Mon Sep 17 00:00:00 2001 From: Michael Vanden Heuvel Date: Thu, 30 Jul 2026 10:05:39 -0700 Subject: [PATCH 3/3] Restrict unsupported ASIM parser patterns Prohibit enrichment, fan-out, mv-* operators, and aggregation in ASIM parser creation and review guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 626b2a6f-5699-493d-bfa5-0c490bf1f420 --- .../asim-parser-create-parameter-parser/SKILL.md | 12 ++++++++++++ .github/skills/asim-parser-create-parser/SKILL.md | 14 +++++++++++++- .github/skills/asim-parser-pr-reviewer/SKILL.md | 15 ++++++++++++++- 3 files changed, 39 insertions(+), 2 deletions(-) diff --git a/.github/skills/asim-parser-create-parameter-parser/SKILL.md b/.github/skills/asim-parser-create-parameter-parser/SKILL.md index 4e1484bccac..8e791f71954 100644 --- a/.github/skills/asim-parser-create-parameter-parser/SKILL.md +++ b/.github/skills/asim-parser-create-parameter-parser/SKILL.md @@ -23,6 +23,18 @@ The file name should be prefixed with vim, followed by the name of the schema, e Add the contents of the parameter-less version of the parser to the new file. +## Preserve parser development restrictions + +The parameterized parser must preserve all development guidelines and prohibited patterns from the `asim-parser-create-parser` skill. Adding parameters or filters must not introduce: + +- Same-table enrichment, such as a second source-table read or self-join. +- Cross-table or other table-shaped enrichment, including `join`, `lookup`, secondary `union` branches, `externaldata`, or inline `datatable` mappings. +- One-to-many fan-out; each source record must still produce at most one normalized record. +- Any `mv-*` operator, including `mv-expand` and `mv-apply`. +- Aggregation, reaggregation, correlation, or deduplication, including `summarize`, `distinct`, `arg_min`, or `arg_max`. + +Use native source columns and scalar predicates for parameter filtering. If a parameter cannot be applied without a prohibited pattern, retain the standard empty-parameter check without adding enrichment, row expansion, or aggregation. + ## Add parameters From the parameters you have gathered, add it to function arguments for both the function and the function call. diff --git a/.github/skills/asim-parser-create-parser/SKILL.md b/.github/skills/asim-parser-create-parser/SKILL.md index 938e2bebdc2..62ba5d27367 100644 --- a/.github/skills/asim-parser-create-parser/SKILL.md +++ b/.github/skills/asim-parser-create-parser/SKILL.md @@ -40,11 +40,23 @@ Parsers are KQL functions that follow a clear flow: **Filter → Parse → Map** - Use indexes so only relevant extents are scanned. - Filter early on native columns before parsing to improve performance. - Use high-performance parsing operators (`split`, `parse-kv`, `parse`) and avoid regular expressions for string parsing. -- Normalize values with `iff`, `case`, or lookup tables rather than copying source values directly. +- Normalize values with scalar expressions such as `iff` or `case` rather than copying source values directly. - Use `project-rename` for mapping, `extend` for calculated or normalized fields. - Try to map as many fields as possible, including optional ones. This increases usefulness and future-proofs the parser for schema changes. - Do not use `project-away` to remove unmapped columns. Use `project` instead, as `project-away` does not protect the parser from schema changes in the source data. +### Prohibited patterns + +ASIM parsers must remain a single pipeline over the declared source table. Do not use: + +- **Same-table enrichment** — including a second read of the source table, a self-join, or a tabular subquery used to add fields. +- **Cross-table or other table-shaped enrichment** — including `join`, `lookup`, a secondary `union` branch, `externaldata`, or inline `datatable` mappings. +- **One-to-many fan-out** — each source record must produce at most one normalized record. Do not expand one record into multiple normalized records; this indicates that the connector or source event shape must be corrected. +- **Any `mv-*` operator** — including `mv-expand` and `mv-apply`. +- **Aggregation, reaggregation, correlation, or deduplication** — including `summarize`, `distinct`, `arg_min`, `arg_max`, or any other operation that combines source records or collapses expanded rows. + +Use only values available on the current source row and preserve its record cardinality. Prefer scalar expressions, scalar dynamic-value access, and direct parsing. If a field cannot be mapped without a prohibited enrichment, fan-out, `mv-*`, or aggregation pattern, leave it unmapped. + ### Parsing operators by performance ranking | Rank | Operator | Description | diff --git a/.github/skills/asim-parser-pr-reviewer/SKILL.md b/.github/skills/asim-parser-pr-reviewer/SKILL.md index 59dd5a15dfe..b3572e68bfa 100644 --- a/.github/skills/asim-parser-pr-reviewer/SKILL.md +++ b/.github/skills/asim-parser-pr-reviewer/SKILL.md @@ -44,13 +44,23 @@ Review the parameter-less parser for the following: - **No `project-away`**: The query must NOT use `project-away` to remove unmapped columns. It should use `project` instead, as `project-away` does not protect the parser from schema changes in the source data. +- **No same-table or cross-table enrichment**: Verify that the query is a single pipeline over the declared source table. Flag any second read of the source table, self-join, cross-table reference, `join`, `lookup`, secondary `union` branch, `externaldata`, inline `datatable` mapping, or equivalent table-shaped enrichment. If a field cannot be produced from the current source row, it should remain unmapped rather than be enriched. + +- **One source record produces at most one normalized record**: Flag any operation that fans out one source record into multiple normalized records, regardless of operator. Fan-out indicates a connector or source event-shape defect that should be corrected rather than supported in the parser. + +- **No `mv-*` operators**: Flag any KQL operator whose name begins with `mv-`, including `mv-expand` and `mv-apply`. The parser should use scalar dynamic-value access or other scalar expressions instead. If a field cannot be mapped without row expansion, it should remain unmapped. + +- **No aggregation, reaggregation, correlation, or deduplication**: Flag `summarize`, `distinct`, `arg_min`, `arg_max`, or any equivalent operation that combines source records or collapses expanded rows. An ASIM parser should normalize each source record independently rather than correlate, deduplicate, or reaggregate records. + - **`pack` parameter**: If the query uses `AdditionalFields`, verify that a `pack: bool = false` parameter is included. This allows users to choose whether to populate `AdditionalFields` or return an empty dynamic, improving performance for users who do not need the extra information. - **Placeholder entity fields**: Determine the complete set of applicable fields from the target schema and `Common` rows in `ASimTester.csv`. Verify that the parser includes every defined `*EntityKey` and `*AdditionalIds` field, plus `AdditionalEntities` when defined. Names and casing must exactly match `ASimTester.csv`; generic names such as `entityKey` or `AdditionalIds` are invalid. Every `*EntityKey` must be an empty string, and every `*AdditionalIds` field and `AdditionalEntities` must be an empty dynamic array until mappings are defined. - **Parsing operator efficiency**: Check that high-performance parsing operators are used (`split`, `parse-kv`, `parse`) and that regular expressions are avoided where simpler operators would work. -- **General KQL performance**: Flag any other inefficient patterns such as unnecessary `let` statements, redundant filters, expensive joins, or operations that could be reordered for better performance. +- **General KQL performance**: Flag any other inefficient patterns such as unnecessary `let` statements, redundant filters, or operations that could be reordered for better performance. + +Treat every same-table enrichment, cross-table/table-shaped enrichment, fan-out, `mv-*`, or aggregation/reaggregation violation as 🔴 High priority. **Output format:** @@ -59,6 +69,7 @@ Return your findings as a markdown table with the following columns: | # | Priority | Issue | Suggestion | | --- | -------- | ----- | ---------- | + Where: - **Priority** is one of: 🔴 High, 🟡 Medium, 🟢 Low @@ -84,6 +95,7 @@ Please review: 3. **Redundant computation**: Are there any calculated fields or parsing operations that occur before the parameter filters, when they could be moved after? 4. **Parameter completeness**: Are the filtering parameters comprehensive enough to allow efficient querying for common use cases? 5. **Placeholder entity field consistency**: Compare the parameterized parser's placeholder fields and values with the parameter-less parser. Report fields missing from both parsers only in the parameter-less review above. In this section, report only differences introduced by the parameterized parser, such as a placeholder that is omitted, renamed, or populated when the parameter-less parser defines it correctly. Matching omissions are not acceptable; they are already reported in the parameter-less review. +6. **Prohibited pattern consistency**: Report any same-table enrichment, cross-table/table-shaped enrichment, fan-out, `mv-*`, or aggregation/reaggregation introduced only by the parameterized parser as 🔴 High priority. If the same violation exists in both parser versions, report it only in the parameter-less review above. **Output format:** @@ -92,5 +104,6 @@ Return findings as a markdown table: | # | Priority | Issue | Suggestion | | --- | -------- | ----- | ---------- | + Where Priority is one of: 🔴 High, 🟡 Medium, 🟢 Low. Only include issues specific to the filtering/parameter logic.