Skip to content

Commit ce28db2

Browse files
terrancedejesusterrancedejesus
authored
[Azure Logs] Add dedicated aadgraphactivitylogs data stream (#18880)
Add a dedicated data stream for the AzureADGraphActivityLogs diagnostic category from Microsoft Entra ID. Without this, legacy Azure AD Graph (graph.windows.net) events fall through to the platformlogs catch-all and lose schema-aware parsing. The events router maps routing.category == "AzureADGraphActivityLogs" to the new dataset. The ingest pipeline extracts ECS fields: event.action from HTTP method + URI collection, event.outcome from response status, event.category [iam, web], and related.user including the OAuth app_id for client correlation. Legacy AAD Graph is still actively used by Microsoft first-party tooling, older line-of-business apps, and adversary tooling (ROADtools, AzureHound v1, AADInternals). The dedicated dataset makes these events available for detection rules and dashboards.
1 parent 6a9d08b commit ce28db2

24 files changed

Lines changed: 2300 additions & 5 deletions

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,7 @@
100100
/packages/awsfargate @elastic/obs-infraobs-integrations
101101
/packages/awsfirehose @elastic/obs-ds-hosted-services
102102
/packages/azure @elastic/obs-infraobs-integrations @elastic/obs-ds-hosted-services @elastic/security-service-integrations
103+
/packages/azure/data_stream/aadgraphactivitylogs @elastic/security-service-integrations
103104
/packages/azure/data_stream/activitylogs @elastic/obs-infraobs-integrations
104105
/packages/azure/data_stream/application_gateway @elastic/security-service-integrations
105106
/packages/azure/data_stream/auditlogs @elastic/obs-infraobs-integrations

‎packages/azure/_dev/build/docs/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Azure Logs Integration
22

3-
The Azure Logs integration collects logs for specific Azure services like Microsoft Entra ID (Sign-in, Audit, Identity Protection, and Provisioning logs), Azure Spring Apps, Azure Firewall, Microsoft Graph Activity, and several others using the Activity and Platform logs.
3+
The Azure Logs integration collects logs for specific Azure services like Microsoft Entra ID (Sign-in, Audit, Identity Protection, and Provisioning logs), Azure Spring Apps, Azure Firewall, Microsoft Graph Activity, Azure AD Graph Activity, and several others using the Activity and Platform logs.
44

55
You can then visualize that data in Kibana, create alerts if something goes wrong, and reference data when troubleshooting an issue.
66

@@ -16,7 +16,7 @@ fail to start due to an exceed quota limit.
1616
The Azure Logs integration collects logs.
1717

1818
**Logs** help you keep a record of events that happen on your Azure account.
19-
Log data streams collected by the Azure Logs integration include Activity, Platform, Microsoft Entra ID (Sign-in, Audit, Identity Protection, Provisioning), Microsoft Graph Activity, and Spring Apps logs.
19+
Log data streams collected by the Azure Logs integration include Activity, Platform, Microsoft Entra ID (Sign-in, Audit, Identity Protection, Provisioning), Microsoft Graph Activity, Azure AD Graph Activity, and Spring Apps logs.
2020

2121
## Requirements
2222

‎packages/azure/_dev/build/docs/aadgraphactivitylogs.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Azure AD Graph Activity Logs
2+
3+
Azure AD Graph Activity Logs provide an audit trail of all HTTP requests that the legacy Azure AD Graph service (`graph.windows.net`) has received and processed for a tenant. Although Microsoft has deprecated Azure AD Graph in favor of Microsoft Graph, the API is still actively used by Microsoft first-party tooling, older line-of-business applications, third-party SaaS connectors, and adversary tooling (for example ROADtools, AzureHound v1, AADInternals). Refer to the [AADGraphActivityLogs table reference](https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/aadgraphactivitylogs) for the canonical schema.
4+
5+
Tenant administrators can configure the collection and storage destinations of Azure AD Graph Activity Logs through Diagnostic Setting in the Entra Portal. This integration uses Azure Event Hubs destination to stream Azure AD Graph Activity Logs to Elastic.
6+
7+
## Requirements and Setup
8+
9+
### What do I need to use this integration?
10+
11+
The following privileges are required to collect Azure AD Graph Activity Logs:
12+
- A Microsoft Entra ID P1 or P2 tenant license in your tenant.
13+
- A `Security Administrator` or `Global Administrator` Microsoft Entra ID role to configure the diagnostic settings.
14+
15+
### Setup
16+
17+
Refer to the [Azure Logs](https://docs.elastic.co/integrations/azure) page for more information about setting up and using this integration.
18+
19+
### Limitations
20+
21+
- Activities of multi-tenant applications belonging to another tenant are not available.
22+
- In rare cases, events might take up to 2 hours to be delivered to Event Hubs.
23+
- Azure AD Graph is deprecated by Microsoft. New workloads should target Microsoft Graph; this dataset is intended for visibility into legacy traffic that still exists in the tenant.
24+
25+
## Settings
26+
27+
`eventhub` :
28+
_string_
29+
It is a fully managed, real-time data ingestion service. Elastic recommends using only letters, numbers, and the hyphen (-) character for Event Hub names to maximize compatibility. You _can_ use existing Event Hubs having underscores (_) in the Event Hub name. In this case, the integration will replace underscores with hyphens (-) when it uses the Event Hub name to create dependent Azure resources behind the scenes (for example, the storage account container to store Event Hub consumer offsets). Elastic also recommends using a separate event hub for each log type as the field mappings of each log type differ.
30+
Default value `insights-operational-logs`.
31+
32+
`consumer_group` :
33+
_string_
34+
The publish/subscribe mechanism of Event Hubs is enabled through consumer groups. A consumer group is a view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets.
35+
Default value: `$Default`
36+
37+
`connection_string` :
38+
_string_
39+
The connection string required to communicate with Event Hubs, steps [here](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string).
40+
41+
A Blob Storage account is required to store, retrieve, and update the offset or state of the eventhub messages. This means that after stopping the filebeat azure module it can start back up at the spot that it stopped processing messages.
42+
43+
`storage_account` :
44+
_string_
45+
The name of the storage account the state/offsets will be stored and updated.
46+
47+
`storage_account_key` :
48+
_string_
49+
The storage account key, this key will be used to authorize access to data in your storage account.
50+
51+
`storage_account_container` :
52+
_string_
53+
The storage account container where the integration stores the checkpoint data for the consumer group. It is an advanced option to use with extreme care. You MUST use a dedicated storage account container for each Azure log type (activity, sign-in, audit logs, and others). DO NOT REUSE the same container name for more than one Azure log type. See [Container Names](https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata#container-names) for details on naming rules from Microsoft. The integration generates a default container name if not specified.
54+
55+
`resource_manager_endpoint` :
56+
_string_
57+
Optional. By default the integration uses the Azure public environment. To override, provide a specific resource manager endpoint to use a different Azure environment.
58+
59+
Resource manager endpoints:
60+
61+
```text
62+
# Azure ChinaCloud
63+
https://management.chinacloudapi.cn/
64+
65+
# Azure GermanCloud
66+
https://management.microsoftazure.de/
67+
68+
# Azure PublicCloud
69+
https://management.azure.com/
70+
71+
# Azure USGovernmentCloud
72+
https://management.usgovcloudapi.net/
73+
```
74+
75+
## Logs
76+
77+
### aadgraphactivitylogs
78+
79+
The `aadgraphactivitylogs` data stream of the Azure Logs package collects Azure AD Graph activity events that have been streamed through an Azure event hub. The events ingest pipeline matches `category == "AzureADGraphActivityLogs"` and sets `event.dataset = azure.aadgraphactivitylogs`. The events data stream's routing rules then reroute the document from `logs-azure.events-*` directly to `logs-azure.aadgraphactivitylogs-*`, where this data stream's pipeline applies full ECS field extraction.
80+
81+
Before this data stream existed, AAD Graph events had no specific override in the events router and fell through to the `azure.platformlogs` catch-all, landing in `logs-azure.platformlogs-default` with only generic platform-log parsing. Those previously-indexed events are not backfilled. Only new events are routed to the dedicated dataset.
82+
83+
{{event "aadgraphactivitylogs"}}
84+
85+
**ECS Field Reference**
86+
87+
Refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
88+
89+
{{fields "aadgraphactivitylogs"}}

‎packages/azure/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
- version: "1.37.0"
2+
changes:
3+
- description: Add azure.aadgraphactivitylogs dataset to ingest legacy Azure AD Graph (graph.windows.net) AzureADGraphActivityLogs diagnostic events.
4+
type: enhancement
5+
link: https://github.com/elastic/integrations/pull/18880
16
- version: "1.36.1"
27
changes:
38
- description: Add missing event.kind pipeline_error handling to ingest pipeline on_failure handlers.

‎packages/azure/data_stream/aadgraphactivitylogs/_dev/test/pipeline/test-aadgraph-activity.log‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{"category":"AzureADGraphActivityLogs","location":"WestUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639140000000000000,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000001-0001-0001-0001-000000000001","actorType":"User","apiVersion":"1.6","appId":"04b07795-8ddb-461a-bbee-02f9e1bf7b46","callerIpAddress":"81.2.69.143","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":59,"env_cloud_role":"restdirectoryservice","httpMethod":"GET","httpStatusCode":200,"identityProvider":"https://sts.windows.net/ab30785b-417f-42a4-b5dc-8f9051718acb/","issuedAt":"5/7/2026 1:50:39 PM","location":"WestUS","requestId":"00000001-0001-0001-0001-000000000001","requestUri":"/v2/ab30785b-417f-42a4-b5dc-8f9051718acb/users","responseSizeBytes":54662,"roles":"","scopes":"62e90394-69f5-4237-9190-012177145e10","servicePrincipalId":"","sessionId":"5a5a5a5a-5a5a-5a5a-5a5a-5a5a5a5a5a5a","signInActivityId":"AAAAAAAAAAAAAAAAAAAAAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-07T15:19:33.5368860Z","userAgent":"azure-graph-test-client/1.0","userId":"b37ec517-0a34-4266-b627-f7bb0d679d70","wids":""},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"}
2+
{"category":"AzureADGraphActivityLogs","location":"WestUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639140000000000001,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000002-0002-0002-0002-000000000002","actorType":"User","apiVersion":"1.61-internal","appId":"1b730954-1685-4b74-9bfd-dac224a7b894","callerIpAddress":"81.2.69.143","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":25,"env_cloud_role":"restdirectoryservice","httpMethod":"GET","httpStatusCode":403,"identityProvider":"","issuedAt":"5/8/2026 4:11:19 PM","location":"WestUS","requestId":"00000002-0002-0002-0002-000000000002","requestUri":"/v2/ab30785b-417f-42a4-b5dc-8f9051718acb/tenantDetails","responseSizeBytes":-1,"roles":"","scopes":"user_impersonation","servicePrincipalId":"","sessionId":"004d8baa-891d-139e-633f-6c194e566b78","signInActivityId":"BCF50s3Ik0q7Ke-mIBAGAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-08T16:37:21.2823997Z","userAgent":"AADInternals","userId":"b37ec517-0a34-4266-b627-f7bb0d679d70","wids":""},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"}
3+
{"category":"AzureADGraphActivityLogs","location":"WestUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639141000000000000,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000003-0003-0003-0003-000000000003","actorType":"Application","apiVersion":"AGSbeta-internal","appId":"65d91a3d-ab74-42e6-8a2f-0add61688c74","callerIpAddress":"81.2.69.144","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":44,"env_cloud_role":"restdirectoryservice","httpMethod":"GET","httpStatusCode":200,"identityProvider":"https://sts.windows.net/ab30785b-417f-42a4-b5dc-8f9051718acb/","issuedAt":"5/11/2026 12:35:58 PM","location":"WestUS","requestId":"00000003-0003-0003-0003-000000000003","requestUri":"/v2/ab30785b-417f-42a4-b5dc-8f9051718acb/groups","responseSizeBytes":695,"roles":" Group.Read.All,","scopes":"","servicePrincipalId":"fa5cc200-f464-4068-88b9-c6b8335c8c70","sessionId":"","signInActivityId":"Rr_bbVOVu0WHOTzl0BhgAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-11T12:41:10.9430757Z","userAgent":"","userId":"","wids":" 0997a1d0-0d1d-4acb-b408-d5ca73121e90,"},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"}
4+
{"category":"AzureADGraphActivityLogs","location":"SouthCentralUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639141000000000001,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000004-0004-0004-0004-000000000004","actorType":"Application","apiVersion":"1.61-internal","appId":"66244124-575c-4284-92bc-fdd00e669cea","callerIpAddress":"89.160.20.156","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":152,"env_cloud_role":"restdirectoryservice","httpMethod":"POST","httpStatusCode":202,"identityProvider":"https://sts.windows.net/ab30785b-417f-42a4-b5dc-8f9051718acb/","issuedAt":"5/9/2026 12:28:05 PM","location":"SouthCentralUS","requestId":"00000004-0004-0004-0004-000000000004","requestUri":"/ab30785b-417f-42a4-b5dc-8f9051718acb/$batch","responseSizeBytes":9580,"roles":"","scopes":"","servicePrincipalId":"4ed697af-4af9-43f7-8bf2-7b2172300068","sessionId":"","signInActivityId":"-TZsVeDWj0e-jamRtuUnAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-09T12:33:05.9908603Z","userAgent":"","userId":"","wids":""},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"}
5+
{"category":"AzureADGraphActivityLogs","location":"WestUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639142000000000000,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000005-0005-0005-0005-000000000005","actorType":"User","apiVersion":"1.6","appId":"04b07795-8ddb-461a-bbee-02f9e1bf7b46","callerIpAddress":"81.2.69.143","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":71,"env_cloud_role":"restdirectoryservice","httpMethod":"PATCH","httpStatusCode":204,"identityProvider":"https://sts.windows.net/ab30785b-417f-42a4-b5dc-8f9051718acb/","issuedAt":"5/7/2026 1:50:39 PM","location":"WestUS","requestId":"00000005-0005-0005-0005-000000000005","requestUri":"/v2/ab30785b-417f-42a4-b5dc-8f9051718acb/applications/fda2d70f-7df3-4556-8bcb-30acc0a45082","responseSizeBytes":0,"roles":"","scopes":"62e90394-69f5-4237-9190-012177145e10","servicePrincipalId":"","sessionId":"5a5a5a5a-5a5a-5a5a-5a5a-5a5a5a5a5a5a","signInActivityId":"AAAAAAAAAAAAAAAAAAAAAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-07T15:15:44.4909654Z","userAgent":"azure-graph-test-client/1.0","userId":"b37ec517-0a34-4266-b627-f7bb0d679d70","wids":""},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"}

0 commit comments

Comments
 (0)