feat(AbnormalSecurity): route CCF Push connector to *_V2_CL tables with native guid types (v3.2.0) by anoopabsec · Pull Request #14772 · Azure/Azure-Sentinel

anoopabsec · 2026-07-28T10:28:31Z

Summary

Move the Abnormal Security CCF Push connector to a dedicated set of ABNORMAL_SECURITY_*_V2_CL tables so a fresh deploy never collides with the Classic tables created by the legacy Azure Functions (Data Collector API) connector — resolving the Invalid output table schema Deploy failure on workspaces that previously ran the legacy connector.

Changes

Renamed all 9 CCF output tables to ABNORMAL_SECURITY_*_V2_CL (table definitions + DCR output streams). DCR input streams (Custom-ABNORMAL_SECURITY_*) are unchanged, so the ingestion contract is unaffected.
Native GUID typing: the GUID-shaped *_g columns are now declared as type guid with matching toguid() conversions in the DCR transforms (fresh V2 tables carry no legacy Classic schema to conflict with).
Content retargeted to V2: Parsers, Hunting Queries, Analytic Rules, and the Overview Workbook (+ Workbooks/WorkbooksMetadata.json dependencies) now reference the *_V2_CL tables.
Regenerated Package (mainTemplate.json, createUiDefinition.json, 3.2.0.zip); bumped solution to v3.2.0.

Test Plan

Table def ↔ DCR parity verified for all 9 tables (guid columns ⇔ toguid(); no _g = tostring( remains):

ABNORMAL_SECURITY_THREAT_LOG_V2_CL   guidCols=3 toguid=3  OK
ABNORMAL_SECURITY_CASE_V2_CL         guidCols=2 toguid=2  OK
... all 9 -> PARITY: ALL OK

Regenerated mainTemplate.json: 9 distinct *_V2_CL tables, 21 "type": "guid" columns, 0 leftover non-V2 table references; input streams intact.
createUiDefinition.json: 0 leftover non-V2 references.
All edited JSON validated; package regenerated at version 3.2.0 (Package version incremented to 3.2.0).
Reviewer: deploy the solution to a clean workspace and confirm the 9 *_V2_CL tables are created and receive data from the CCF Push connector.

Deploy Plan

No infrastructure deploy from this PR. On merge, customers deploy the updated solution from Content Hub; the connector provisions the *_V2_CL tables on Connect. Existing legacy *_CL tables are left untouched and retain their data for the retention period.

…th native guid types (v3.2.0) Rename all nine CCF Push output tables to ABNORMAL_SECURITY_*_V2_CL so the connector always provisions fresh tables and never collides with the Classic tables created by the legacy Azure Functions (Data Collector API) connector — resolving the "Invalid output table schema" Deploy failure on workspaces that previously ran the legacy connector. - DCR input streams (Custom-ABNORMAL_SECURITY_*) unchanged; only output streams, table defs, Parsers, Hunting Queries, Analytic Rules, and the Workbook now target *_V2_CL. - GUID-shaped *_g columns declared as type guid with matching toguid() in the DCR transforms (fresh V2 tables carry no legacy Classic schema to conflict with). - Regenerated Package (mainTemplate/createUiDefinition/3.2.0.zip); bumped v3.2.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Routes the Abnormal Security CCF Push connector and all dependent Sentinel content to new *_V2_CL tables, enabling native guid typing and avoiding schema collisions with legacy Classic (*_CL) tables.

Changes:

Renamed all referenced Abnormal Security CCF Push output tables from *_CL to *_V2_CL across workbooks, parsers, hunting queries, analytic rules, and connector metadata.
Updated table schemas so GUID-shaped *_g columns are typed as guid, and updated DCR transforms to emit toguid() for those columns.
Bumped solution version to 3.2.0 and updated package/UI dependency text to reference the V2 tables.

Reviewed changes

Copilot reviewed 28 out of 38 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
Workbooks/WorkbooksMetadata.json	Updates workbook datatype dependencies to the new `*_V2_CL` tables.
Solutions/AbnormalSecurity/Workbooks/AbnormalSecurityOverview.json	Retargets workbook visualizations/queries to `*_V2_CL` tables.
Solutions/AbnormalSecurity/ReleaseNotes.md	Adds 3.2.0 release notes describing V2 tables + guid typing.
Solutions/AbnormalSecurity/Parsers/AbnormalSecurityVendorCases.yaml	Parser now reads from `ABNORMAL_SECURITY_VENDOR_CASE_V2_CL`.
Solutions/AbnormalSecurity/Parsers/AbnormalSecurityThreatLog.yaml	Parser now reads from `ABNORMAL_SECURITY_THREAT_LOG_V2_CL`.
Solutions/AbnormalSecurity/Parsers/AbnormalSecurityAtoCases.yaml	Parser now reads from `ABNORMAL_SECURITY_ATO_CASE_V2_CL`.
Solutions/AbnormalSecurity/Parsers/AbnormalSecurityAbuseMailbox.yaml	Parser now reads from `ABNORMAL_SECURITY_ABUSE_MAILBOX_V2_CL`.
Solutions/AbnormalSecurity/Package/createUiDefinition.json	Updates UI descriptive text to reference V2 tables.
Solutions/AbnormalSecurity/Hunting Queries/AbnormalSecurity_UnremediatedThreats.yaml	Hunts against `ABNORMAL_SECURITY_THREAT_LOG_V2_CL`.
Solutions/AbnormalSecurity/Hunting Queries/AbnormalSecurity_TopAttackTargets.yaml	Hunts against `ABNORMAL_SECURITY_THREAT_LOG_V2_CL`.
Solutions/AbnormalSecurity/Hunting Queries/AbnormalSecurity_NewVendorDomains.yaml	Hunts against `ABNORMAL_SECURITY_VENDOR_CASE_V2_CL`.
Solutions/AbnormalSecurity/Hunting Queries/AbnormalSecurity_AbuseMailboxCampaigns.yaml	Hunts against `ABNORMAL_SECURITY_ABUSE_MAILBOX_V2_CL`.
Solutions/AbnormalSecurity/Data/Solution_AbnormalSecurity.json	Bumps solution version to 3.2.0.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_table_VendorCase.json	Renames table to V2 and changes relevant `*_g` columns to `guid`.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_table_ThreatLog.json	Renames table to V2 and changes relevant `*_g` columns to `guid`.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_table_Remediation.json	Renames table to V2 and changes relevant `*_g` columns to `guid`.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_table_PostureChange.json	Renames table to V2 and changes relevant `*_g` columns to `guid`.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_table_Case.json	Renames table to V2 and changes relevant `*_g` columns to `guid`.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_table_AuditLog.json	Renames table to V2 and changes relevant `*_g` columns to `guid`.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_table_AtoCase.json	Renames table to V2 and changes relevant `*_g` columns to `guid`.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_table_AbuseMailbox.json	Renames table to V2 and changes relevant `*_g` columns to `guid`.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_table_AbnormalSecurityLogs.json	Renames table to V2 and changes relevant `*_g` columns to `guid`.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_connectorDefinition.json	Retargets connector graphs/queries/connectivity checks and documentation to V2 tables.
Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_DCR.json	Updates output streams to V2 and changes GUID transforms to `toguid()`.
Solutions/AbnormalSecurity/Analytic Rules/AbnormalSecurity_VendorCompromise.yaml	Rule now queries `ABNORMAL_SECURITY_VENDOR_CASE_V2_CL`.
Solutions/AbnormalSecurity/Analytic Rules/AbnormalSecurity_HighRiskEmailAttack.yaml	Rule now queries `ABNORMAL_SECURITY_THREAT_LOG_V2_CL`.
Solutions/AbnormalSecurity/Analytic Rules/AbnormalSecurity_AccountTakeover.yaml	Rule now queries `ABNORMAL_SECURITY_ATO_CASE_V2_CL`.
Solutions/AbnormalSecurity/Analytic Rules/AbnormalSecurity_AbuseMailboxMalicious.yaml	Rule now queries `ABNORMAL_SECURITY_ABUSE_MAILBOX_V2_CL`.

v-maheshbh · 2026-07-29T06:27:02Z

Hi @anoopabsec

Please review the Copilot comments and resolve them accordingly. Once completed, kindly mark the comments as resolved.

Thanks!

… flattened columns - VendorCompromise analytic rule: use isnotnull() instead of isnotempty() on the now-guid-typed abx_body_abx_body_vendorCaseId_g column. - connectorDefinition sample + verification queries: abx_body/abx_metadata are stored as JSON strings by the DCR (tostring(...)), so dot-access (abx_body.severity, abx_body.status, abx_metadata.event_type) doesn't work. Switched to the DCR-flattened columns (abx_body_abx_body_case_status_s, abx_metadata_event_type_s) and, for threat logs which have no severity field, to abx_body_abx_body_attack_type_s. - Regenerated Package (mainTemplate/createUiDefinition/3.2.0.zip). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

anoopabsec · 2026-07-31T05:36:43Z

Hi @v-maheshbh - Resolved all the comments and attached screenshots as well.
We want to keep this PR in a state where we can merge it anytime as needed. This is more of a backup plan for our connector if customers face any issues. Do let me know if that's possible.

v-maheshbh · 2026-07-31T06:24:56Z

Hi @anoopabsec

The table ABNORMAL_SECURITY_ATO_CASE_V2_CL is referenced in the query but is not available in the current environment. Kindly ensure that the required custom table is included in the solution and properly deployed, or update the query to use an existing table.

Thanks!

…r KqlValidations + bump detection versions - KqlValidations: add custom-table schema files for the renamed tables (ABNORMAL_SECURITY_{THREAT_LOG,ATO_CASE,VENDOR_CASE,ABUSE_MAILBOX}_V2_CL) under .script/tests/KqlvalidationsTests/CustomTables so the validator can resolve the tables the analytic rules and hunting queries now reference. GUID-shaped *_g columns are typed `guid` to match the V2 table schema. - check-version-of-detection-was-updated: bump all four modified analytic rules 1.0.0 -> 1.0.1. - Regenerated Package (mainTemplate reflects rule version 1.0.1; solution stays 3.2.0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 32 out of 42 changed files in this pull request and generated no new comments.

Suppressed comments (5)

Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_connectorDefinition.json:63

This sample query unions 9 tables and sorts without any time predicate, which can be very expensive on larger workspaces. Consider adding a bounded where TimeGenerated > ago(...) before sort (e.g., ago(24h)/ago(7d)) so the sample query remains responsive and avoids scanning older extents unnecessarily.

          "query": "union ABNORMAL_SECURITY_THREAT_LOG_V2_CL, ABNORMAL_SECURITY_CASE_V2_CL, ABNORMAL_SECURITY_AUDIT_LOG_V2_CL, ABNORMAL_SECURITY_ABUSE_MAILBOX_V2_CL, ABNORMAL_SECURITY_POSTURE_CHANGE_V2_CL, ABNORMAL_SECURITY_ATO_CASE_V2_CL, ABNORMAL_SECURITY_REMEDIATION_V2_CL, ABNORMAL_SECURITY_VENDOR_CASE_V2_CL, ABNORMAL_SECURITY_LOGS_V2_CL\n | sort by TimeGenerated desc\n | take 100"

Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_connectorDefinition.json:120

The connectivity criteria query computes max(TimeGenerated) across a union of 9 tables without constraining the time range first. Adding | where TimeGenerated > ago(7d) before summarize should produce the same boolean result while significantly reducing scan cost.

            "union ABNORMAL_SECURITY_THREAT_LOG_V2_CL, ABNORMAL_SECURITY_CASE_V2_CL, ABNORMAL_SECURITY_AUDIT_LOG_V2_CL, ABNORMAL_SECURITY_ABUSE_MAILBOX_V2_CL, ABNORMAL_SECURITY_POSTURE_CHANGE_V2_CL, ABNORMAL_SECURITY_ATO_CASE_V2_CL, ABNORMAL_SECURITY_REMEDIATION_V2_CL, ABNORMAL_SECURITY_VENDOR_CASE_V2_CL, ABNORMAL_SECURITY_LOGS_V2_CL\n| summarize LastLogReceived = max(TimeGenerated)\n| project IsConnected = LastLogReceived > ago(7d)"

Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_connectorDefinition.json:63

The connector definition now references several additional V2 tables (e.g., ABNORMAL_SECURITY_CASE_V2_CL, ABNORMAL_SECURITY_AUDIT_LOG_V2_CL, ABNORMAL_SECURITY_POSTURE_CHANGE_V2_CL, ABNORMAL_SECURITY_REMEDIATION_V2_CL, ABNORMAL_SECURITY_LOGS_V2_CL). In this PR’s test additions, only 4 V2 custom-table schemas were added under .script/tests/KqlvalidationsTests/CustomTables/. To keep KQL validation tests green, add matching custom table schema JSON files for any newly-referenced V2 tables that don’t already exist in that test directory.

          "query": "union ABNORMAL_SECURITY_THREAT_LOG_V2_CL, ABNORMAL_SECURITY_CASE_V2_CL, ABNORMAL_SECURITY_AUDIT_LOG_V2_CL, ABNORMAL_SECURITY_ABUSE_MAILBOX_V2_CL, ABNORMAL_SECURITY_POSTURE_CHANGE_V2_CL, ABNORMAL_SECURITY_ATO_CASE_V2_CL, ABNORMAL_SECURITY_REMEDIATION_V2_CL, ABNORMAL_SECURITY_VENDOR_CASE_V2_CL, ABNORMAL_SECURITY_LOGS_V2_CL\n | sort by TimeGenerated desc\n | take 100"

Solutions/AbnormalSecurity/Data Connectors/AbnormalSecurity_CCF/AbnormalSecurity_connectorDefinition.json:75

The connector definition now references several additional V2 tables (e.g., ABNORMAL_SECURITY_CASE_V2_CL, ABNORMAL_SECURITY_AUDIT_LOG_V2_CL, ABNORMAL_SECURITY_POSTURE_CHANGE_V2_CL, ABNORMAL_SECURITY_REMEDIATION_V2_CL, ABNORMAL_SECURITY_LOGS_V2_CL). In this PR’s test additions, only 4 V2 custom-table schemas were added under .script/tests/KqlvalidationsTests/CustomTables/. To keep KQL validation tests green, add matching custom table schema JSON files for any newly-referenced V2 tables that don’t already exist in that test directory.

          "query": "union ABNORMAL_SECURITY_THREAT_LOG_V2_CL, ABNORMAL_SECURITY_CASE_V2_CL, ABNORMAL_SECURITY_AUDIT_LOG_V2_CL, ABNORMAL_SECURITY_ABUSE_MAILBOX_V2_CL, ABNORMAL_SECURITY_POSTURE_CHANGE_V2_CL, ABNORMAL_SECURITY_ATO_CASE_V2_CL, ABNORMAL_SECURITY_REMEDIATION_V2_CL, ABNORMAL_SECURITY_VENDOR_CASE_V2_CL, ABNORMAL_SECURITY_LOGS_V2_CL\n | where TimeGenerated > ago(24h)\n | summarize count() by event_type = abx_metadata_event_type_s\n | order by count_ desc"

Solutions/AbnormalSecurity/ReleaseNotes.md:3

This release-note entry is extremely long for a markdown table cell, which can hurt readability and render poorly in some viewers. Consider splitting the change history into shorter sentences with <br> line breaks (or reducing detail and linking to PR/issue) while keeping the table format intact.

| 3.2.0       | 28-07-2026                     | Renamed the nine CCF Push output tables to `ABNORMAL_SECURITY_*_V2_CL` so the connector always provisions fresh, correctly-typed tables and never collides with Classic tables created by the legacy Azure Functions (Data Collector API) connector — resolving the `Invalid output table schema` Deploy failure on workspaces that previously ran the legacy connector. DCR input streams (`Custom-ABNORMAL_SECURITY_*`) are unchanged; only DCR **output** streams, table definitions, Parsers, Hunting Queries, Analytic Rules, and the Workbook now target the `*_V2_CL` tables. GUID-shaped columns (`*_trace_id_g`, `*_threat_id_g`, `*_campaign_id_g`, `*_vendorCaseId_g`) are now declared as type `guid` with matching `toguid()` conversions in the DCR transforms (fresh V2 tables have no legacy Classic schema to conflict with); all other column names and datatypes are unchanged from 3.1.0. |

anoopabsec requested review from a team as code owners July 28, 2026 10:28

v-atulyadav requested a review from Copilot July 28, 2026 11:34

v-atulyadav assigned v-maheshbh Jul 28, 2026

v-atulyadav added Solution Solution specialty review needed Codeless Connector Framework (CCF) Connector labels Jul 28, 2026

Copilot AI reviewed Jul 28, 2026

View reviewed changes

anoopabsec marked this pull request as draft July 29, 2026 07:25

anoopabsec marked this pull request as ready for review July 31, 2026 05:34

anoopabsec requested a review from a team as a code owner July 31, 2026 08:17

anoopabsec requested a review from Copilot July 31, 2026 08:36

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(AbnormalSecurity): route CCF Push connector to *_V2_CL tables with native guid types (v3.2.0) - #14772

feat(AbnormalSecurity): route CCF Push connector to *_V2_CL tables with native guid types (v3.2.0)#14772
anoopabsec wants to merge 3 commits into
Azure:masterfrom
anoopabsec:anoop/pr/abnormal-ccf-v2-tables

anoopabsec commented Jul 28, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v-maheshbh commented Jul 29, 2026

Uh oh!

anoopabsec commented Jul 31, 2026

Uh oh!

v-maheshbh commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

anoopabsec commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test Plan

Deploy Plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v-maheshbh commented Jul 29, 2026

Uh oh!

anoopabsec commented Jul 31, 2026

Uh oh!

v-maheshbh commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anoopabsec commented Jul 28, 2026 •

edited

Loading