Merge pull request #14765 from prodaft/prodaft-usta-atp

usta account takeover prevention playbook fix

Author: v-atulyadav

Solutions/PRODAFT USTA - Account Takeover Prevention/Data/Solution_PRODAFTUstaATP.json

@@ -23,7 +23,7 @@
   "Playbooks": [
     "Playbooks/PRODAFTUstaATP-Backfill/azuredeploy.json"
   ],
-  "Version": "3.0.0",
+  "Version": "3.0.1",
   "Metadata": "SolutionMetadata.json",
   "TemplateSpec": true,
   "Is1PConnector": false

Solutions/PRODAFT USTA - Account Takeover Prevention/Package/3.0.1.zip

@@ -3,14 +3,14 @@
   "contentVersion": "1.0.0.0",
   "metadata": {
     "title": "PRODAFTUstaATP-Backfill",
-    "description": "On-demand backfill playbook for PRODAFT USTA - Account Takeover Prevention. Pages through the USTA compromised-credentials API and pushes historical tickets to the connector's Data Collection Endpoint (Logs Ingestion API), so the same DCR transform (password redaction, field mapping) is applied as for live polling. The codeless connector only polls forward from connection time; run this playbook once after connecting to load history.",
+    "description": "On-demand backfill playbook for PRODAFT USTA - Account Takeover Prevention. Pages through the USTA compromised-credentials API and pushes historical tickets into the workspace via the Logs Ingestion API, applying the same transform (password redaction, field mapping) as live polling. The deployment is self-contained: it creates its own Data Collection Endpoint and Rule (writing into the existing PRODAFTUstaCompromisedCredentials_CL table) and assigns its identity the required role. The codeless connector only polls forward from connection time; run this playbook once after installing the solution to load history.",
     "prerequisites": [
-      "The PRODAFT USTA - Account Takeover Prevention solution installed and its data connector connected (this creates the DCE, DCR, and table).",
-      "The Data Collection Endpoint logs-ingestion URI and the DCR immutable ID (see readme.md for how to find them).",
-      "A PRODAFT USTA long-lived API key."
+      "The PRODAFT USTA - Account Takeover Prevention solution installed (this creates the PRODAFTUstaCompromisedCredentials_CL table the backfill writes into); connecting the data connector starts forward polling.",
+      "A PRODAFT USTA long-lived API key.",
+      "Permission to create role assignments in the workspace resource group (Owner or User Access Administrator), since the template grants its identity 'Monitoring Metrics Publisher' automatically."
     ],
     "postDeployment": [
-      "Grant the playbook's system-assigned managed identity the 'Monitoring Metrics Publisher' role on the connector's Data Collection Rule, then run the trigger once (see readme.md)."
+      "Run the playbook's trigger once to start the backfill (see readme.md)."
     ],
     "lastUpdateTime": "2026-07-06T00:00:00.000Z",
     "entities": [],
@@ -22,6 +22,11 @@
       "name": "PRODAFT"
     },
     "releaseNotes": [
+      {
+        "version": "1.1",
+        "title": "PRODAFTUstaATP-Backfill",
+        "notes": [ "Playbook now self-provisions its Data Collection Endpoint and Rule and assigns its identity the Monitoring Metrics Publisher role; removes manual ingestion-endpoint and DCR immutable-ID setup." ]
+      },
       {
         "version": "1.0",
         "title": "PRODAFTUstaATP-Backfill",
@@ -35,13 +40,18 @@
       "type": "string",
       "metadata": { "description": "Name of the Logic App playbook." }
     },
-    "DataCollectionEndpointUri": {
+    "WorkspaceName": {
       "type": "string",
-      "metadata": { "description": "Logs-ingestion URI of the connector's Data Collection Endpoint, e.g. https://<dce-name>-a1b2.westeurope-1.ingest.monitor.azure.com" }
+      "metadata": { "description": "Name of the existing Microsoft Sentinel / Log Analytics workspace that the ATP connector ingests into." }
     },
-    "DataCollectionRuleImmutableId": {
+    "WorkspaceLocation": {
       "type": "string",
-      "metadata": { "description": "Immutable ID of the connector's Data Collection Rule, e.g. dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }
+      "metadata": { "description": "Region of the workspace. The backfill Data Collection Endpoint and Rule are created in this region." }
+    },
+    "WorkspaceResourceGroup": {
+      "type": "string",
+      "defaultValue": "[resourceGroup().name]",
+      "metadata": { "description": "Resource group of the workspace. Leave as default if it is in the same resource group as this deployment." }
     },
     "UstaBaseUrl": {
       "type": "string",
@@ -58,25 +68,91 @@
       "metadata": { "description": "Number of days of history to backfill." }
     }
   },
+  "variables": {
+    "WorkspaceResourceId": "[resourceId(parameters('WorkspaceResourceGroup'), 'Microsoft.OperationalInsights/workspaces', parameters('WorkspaceName'))]",
+    "DceName": "[concat(parameters('PlaybookName'), '-DCE')]",
+    "DcrName": "[concat(parameters('PlaybookName'), '-DCR')]",
+    "DceResourceId": "[resourceId('Microsoft.Insights/dataCollectionEndpoints', variables('DceName'))]",
+    "DcrResourceId": "[resourceId('Microsoft.Insights/dataCollectionRules', variables('DcrName'))]",
+    "MonitoringMetricsPublisherRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '3913510d-42f4-4e42-8a64-420c390055eb')]"
+  },
   "resources": [
+    {
+      "type": "Microsoft.Insights/dataCollectionEndpoints",
+      "apiVersion": "2024-03-11",
+      "name": "[variables('DceName')]",
+      "location": "[parameters('WorkspaceLocation')]",
+      "properties": {
+        "description": "Data Collection Endpoint for the PRODAFT USTA ATP backfill playbook (Logs Ingestion API).",
+        "networkAcls": {
+          "publicNetworkAccess": "Enabled"
+        }
+      }
+    },
+    {
+      "type": "Microsoft.Insights/dataCollectionRules",
+      "apiVersion": "2024-03-11",
+      "name": "[variables('DcrName')]",
+      "location": "[parameters('WorkspaceLocation')]",
+      "dependsOn": [
+        "[variables('DceResourceId')]"
+      ],
+      "properties": {
+        "description": "DCR for the PRODAFT USTA ATP backfill playbook. Applies the same transform as the connector DCR and writes to the existing PRODAFTUstaCompromisedCredentials_CL table.",
+        "dataCollectionEndpointId": "[variables('DceResourceId')]",
+        "streamDeclarations": {
+          "Custom-PRODAFTUstaCompromisedCredentials_CL": {
+            "columns": [
+              { "name": "id", "type": "long" },
+              { "name": "status", "type": "string" },
+              { "name": "status_timestamp", "type": "datetime" },
+              { "name": "created", "type": "datetime" },
+              { "name": "content_type", "type": "string" },
+              { "name": "company", "type": "dynamic" },
+              { "name": "content", "type": "dynamic" }
+            ]
+          }
+        },
+        "destinations": {
+          "logAnalytics": [
+            {
+              "workspaceResourceId": "[variables('WorkspaceResourceId')]",
+              "name": "clv2ws1"
+            }
+          ]
+        },
+        "dataFlows": [
+          {
+            "streams": [ "Custom-PRODAFTUstaCompromisedCredentials_CL" ],
+            "destinations": [ "clv2ws1" ],
+            "transformKql": "source | extend c = content, co = company | project TimeGenerated = now(), TicketId = tolong(id), Status = tostring(status), StatusTimestamp = todatetime(status_timestamp), Created = todatetime(created), ContentType = tostring(content_type), CompanyId = toint(co.id), CompanyName = tostring(co.name), Username = tostring(c.username), Url = tostring(c.url), Source = tostring(c.source), IsCorporate = tobool(c.is_corporate), PasswordScore = tostring(c.password_complexity.score), PasswordLength = toint(c.password_complexity.length), VictimUid = tostring(c.victim_detail.victim_uid), VictimUsername = tostring(c.victim_detail.username), VictimCountry = tostring(c.victim_detail.country), VictimIp = tostring(c.victim_detail.ip), VictimComputerName = tostring(c.victim_detail.computer_name), VictimOs = tostring(c.victim_detail.victim_os), VictimMalware = tostring(c.victim_detail.malware), InfectionDate = todatetime(c.victim_detail.infection_date)",
+            "outputStream": "Custom-PRODAFTUstaCompromisedCredentials_CL"
+          }
+        ]
+      }
+    },
     {
       "type": "Microsoft.Logic/workflows",
       "apiVersion": "2019-05-01",
       "name": "[parameters('PlaybookName')]",
       "location": "[resourceGroup().location]",
       "identity": { "type": "SystemAssigned" },
+      "dependsOn": [
+        "[variables('DceResourceId')]",
+        "[variables('DcrResourceId')]"
+      ],
       "tags": {
         "hidden-SentinelTemplateName": "PRODAFTUstaATP-Backfill",
-        "hidden-SentinelTemplateVersion": "1.0"
+        "hidden-SentinelTemplateVersion": "1.1"
       },
       "properties": {
         "state": "Enabled",
         "definition": {
           "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
           "contentVersion": "1.0.0.0",
           "parameters": {
-            "DataCollectionEndpointUri": { "type": "String", "defaultValue": "[parameters('DataCollectionEndpointUri')]" },
-            "DataCollectionRuleImmutableId": { "type": "String", "defaultValue": "[parameters('DataCollectionRuleImmutableId')]" },
+            "DataCollectionEndpointUri": { "type": "String", "defaultValue": "[reference(variables('DceResourceId'), '2024-03-11').logsIngestion.endpoint]" },
+            "DataCollectionRuleImmutableId": { "type": "String", "defaultValue": "[reference(variables('DcrResourceId'), '2024-03-11').immutableId]" },
             "UstaBaseUrl": { "type": "String", "defaultValue": "[parameters('UstaBaseUrl')]" },
             "UstaApiKey": { "type": "SecureString", "defaultValue": "[parameters('UstaApiKey')]" },
             "BackfillDays": { "type": "Int", "defaultValue": "[parameters('BackfillDays')]" }
@@ -164,12 +240,39 @@
         },
         "parameters": {}
       }
+    },
+    {
+      "type": "Microsoft.Authorization/roleAssignments",
+      "apiVersion": "2022-04-01",
+      "name": "[guid(variables('DcrResourceId'), parameters('PlaybookName'), variables('MonitoringMetricsPublisherRoleId'))]",
+      "scope": "[variables('DcrResourceId')]",
+      "dependsOn": [
+        "[resourceId('Microsoft.Logic/workflows', parameters('PlaybookName'))]",
+        "[variables('DcrResourceId')]"
+      ],
+      "properties": {
+        "roleDefinitionId": "[variables('MonitoringMetricsPublisherRoleId')]",
+        "principalId": "[reference(resourceId('Microsoft.Logic/workflows', parameters('PlaybookName')), '2019-05-01', 'Full').identity.principalId]",
+        "principalType": "ServicePrincipal"
+      }
     }
   ],
   "outputs": {
-    "playbookPrincipalId": {
+    "playbookName": {
+      "type": "string",
+      "value": "[parameters('PlaybookName')]"
+    },
+    "dataCollectionEndpoint": {
+      "type": "string",
+      "value": "[reference(variables('DceResourceId'), '2024-03-11').logsIngestion.endpoint]"
+    },
+    "dataCollectionRuleName": {
+      "type": "string",
+      "value": "[variables('DcrName')]"
+    },
+    "dataCollectionRuleImmutableId": {
       "type": "string",
-      "value": "[reference(resourceId('Microsoft.Logic/workflows', parameters('PlaybookName')), '2019-05-01', 'full').identity.principalId]"
+      "value": "[reference(variables('DcrResourceId'), '2024-03-11').immutableId]"
     }
   }
 }

Solutions/PRODAFT USTA - Account Takeover Prevention/Package/mainTemplate.json

@@ -4,75 +4,64 @@ On-demand backfill playbook for the **PRODAFT USTA - Account Takeover Prevention
 
 The codeless (CCF) data connector only polls **forward** from the moment it is connected.
 This playbook loads history: it pages through the USTA compromised-credentials API and
-pushes the records to the connector's **Data Collection Endpoint** (Logs Ingestion API),
-so the same DCR transform — password redaction and field mapping — is applied exactly as
-for live polling. Plaintext passwords are never written to the workspace.
+pushes the records into the workspace via the **Logs Ingestion API**, applying the same
+transform — password redaction and field mapping — as live polling. Plaintext passwords are
+never written to the workspace.
+
+The deployment is **self-contained**: it provisions its own Data Collection Endpoint (DCE)
+and Data Collection Rule (DCR) — the DCR uses the same schema and transform as the connector
+and writes into the existing `PRODAFTUstaCompromisedCredentials_CL` table — and grants its
+own managed identity the required role. You do **not** need to look up any endpoint URI or
+DCR immutable ID by hand.
 
 ## Prerequisites
 
-1. The solution is **installed** (this deploys the `PRODAFTUstaATPDCR` data collection
-   rule, its data collection endpoint, and the `PRODAFTUstaCompromisedCredentials_CL`
-   table) and the **PRODAFT USTA - Account Takeover Prevention** data connector is
-   **connected** (this creates the poller and starts polling forward).
+1. The **PRODAFT USTA - Account Takeover Prevention** solution is **installed**, which
+   creates the `PRODAFTUstaCompromisedCredentials_CL` table the backfill writes into.
+   (Connecting the data connector is recommended so forward polling is also active.)
 2. A PRODAFT USTA long-lived API key.
-3. Azure CLI signed in with permission to deploy into the workspace resource group and to
-   create role assignments on the DCR (`Microsoft.Authorization/roleAssignments/write`,
-   i.e. Owner or User Access Administrator on that scope).
+3. Permission to deploy into the workspace resource group **and to create role assignments**
+   (`Microsoft.Authorization/roleAssignments/write` — i.e. Owner or User Access
+   Administrator on that scope), since the template assigns *Monitoring Metrics Publisher*
+   to the playbook's identity automatically.
 
 ## Scripted deployment (run from this folder)
 
 ```bash
 # ---- configuration ----
 SUB="<subscription-id>"
 RG="<usta-sentinel-resource-group>"        # resource group of the Sentinel workspace
+WORKSPACE="<sentinel-workspace-name>"
+LOCATION="<workspace-region>"              # e.g. westeurope
 USTA_API_KEY="<usta-api-key>"
 BACKFILL_DAYS=90
 PLAYBOOK="PRODAFTUstaATP-Backfill"
 
 az account set --subscription "$SUB"
 
-# 1. Locate the connector's DCR (deployed by the solution as 'PRODAFTUstaATPDCR')
-DCR_ID=$(az monitor data-collection rule list --resource-group "$RG" \
-  --query "[?contains(name, 'PRODAFTUstaATP')].id | [0]" -o tsv)
-DCR_IMMUTABLE_ID=$(az monitor data-collection rule show --ids "$DCR_ID" \
-  --query immutableId -o tsv)
-DCE_ID=$(az monitor data-collection rule show --ids "$DCR_ID" \
-  --query dataCollectionEndpointId -o tsv)
-# If DCE_ID comes back empty, list endpoints directly:
-#   az monitor data-collection endpoint list -g "$RG" -o table
-
-# 2. Resolve the DCE logs-ingestion URI
-DCE_URI=$(az monitor data-collection endpoint show --ids "$DCE_ID" \
-  --query logsIngestion.endpoint -o tsv)
-echo "DCR immutable ID: $DCR_IMMUTABLE_ID"
-echo "DCE ingestion URI: $DCE_URI"
-
-# 3. Deploy the playbook (captures its managed identity from the deployment output)
-PRINCIPAL_ID=$(az deployment group create \
+# Deploy the playbook. It creates its own DCE + DCR, derives the ingestion
+# endpoint and DCR immutable ID automatically, and assigns its identity the
+# 'Monitoring Metrics Publisher' role on that DCR.
+az deployment group create \
   --resource-group "$RG" \
   --template-file azuredeploy.json \
   --parameters PlaybookName="$PLAYBOOK" \
-               DataCollectionEndpointUri="$DCE_URI" \
-               DataCollectionRuleImmutableId="$DCR_IMMUTABLE_ID" \
+               WorkspaceName="$WORKSPACE" \
+               WorkspaceLocation="$LOCATION" \
                UstaApiKey="$USTA_API_KEY" \
-               BackfillDays=$BACKFILL_DAYS \
-  --query properties.outputs.playbookPrincipalId.value -o tsv)
-
-# 4. Grant the playbook's identity 'Monitoring Metrics Publisher' on the DCR
-az role assignment create \
-  --assignee-object-id "$PRINCIPAL_ID" \
-  --assignee-principal-type ServicePrincipal \
-  --role "Monitoring Metrics Publisher" \
-  --scope "$DCR_ID"
+               BackfillDays=$BACKFILL_DAYS
 
-# 5. Run the backfill once (or use 'Run Trigger' on the Logic App in the portal)
+# Run the backfill once (or use 'Run Trigger' on the Logic App in the portal)
 az rest --method POST \
   --url "https://management.azure.com/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.Logic/workflows/$PLAYBOOK/triggers/manual/run?api-version=2016-10-01"
 ```
 
-> **RBAC propagation:** the role assignment in step 4 can take a minute to become
-> effective. If the first run shows 403 responses from the ingestion API in the run
-> history, simply run the trigger again (step 5).
+> If the workspace is in a **different resource group** than this deployment, add
+> `WorkspaceResourceGroup="<workspace-rg>"` to the parameters.
+
+> **RBAC propagation:** the role assignment created by the deployment can take a minute to
+> become effective. If the first run shows 403 responses from the ingestion API in the run
+> history, simply run the trigger again.
 
 Monitor progress under **Logic App → Runs history**, then verify data:
 
@@ -84,6 +73,9 @@ PRODAFTUstaCompromisedCredentials
 
 ## Behavior and notes
 
+* The backfill writes through its own DCR into the same `PRODAFTUstaCompromisedCredentials_CL`
+  table as the connector, using an identical transform, so backfilled and live rows are
+  indistinguishable to the solution's content.
 * Pages of 100 records are posted per request — well under the Logs Ingestion API's 1 MB
   request limit. The loop follows the API's `next` URL until exhausted (up to 1000 pages / 4 hours).
 * The API key and the fetched credential data are hidden from the Logic App run history:

Solutions/PRODAFT USTA - Account Takeover Prevention/Playbooks/PRODAFTUstaATP-Backfill/azuredeploy.json

@@ -2,4 +2,5 @@
 
 | **Version** | **Date Modified (DD-MM-YYYY)** | **Change History** |
 |-------------|--------------------------------|--------------------|
+| 3.0.1       | 27-07-2026                     | Updated the backfill **Playbook** to self-provision its Data Collection Endpoint (DCE) and Data Collection Rule (DCR) and to grant its managed identity the *Monitoring Metrics Publisher* role on the DCR. Removes the manual ingestion-endpoint URI and DCR immutable-ID setup previously required to run the backfill. |
 | 3.0.0       | 06-07-2026                     | Initial Solution Release. Codeless (CCF) data connector for compromised-credential tickets with ingestion-time password redaction (only strength signals are stored). Two **Analytic Rules** (corporate credential compromised; compromised credential used in a successful Entra ID sign-in), one **Hunting Query**, an overview **Workbook**, a query-time dedup **Parser**, and an on-demand backfill **Playbook** (Logs Ingestion API via managed identity) for loading historical data. |