Skip to content

Commit bcf1da7

Browse files
efd6efd6
committed
entityanalytics_{ad,entra_id,okta}: add agentless deployment mode and minimal-state sync
Enable agentless deployment for all three entity analytics packages by bumping format_version to 3.3.2, adding deployment_modes to the policy templates, and introducing a use_minimal_state configuration variable (default true, hidden from users in both deployment modes). Existing system tests are pinned to use_minimal_state: false to keep them on the legacy sync path. New script tests verify the minimal-state path routes entities to per-type data streams (AD and EntraID). Okta minimal-state script tests are blocked on elastic/beats#52061 (the entcollect Okta provider hardcodes https:// but does not accept custom TLS config, preventing testing against a self-signed mock). The EntraID mock gains /v1.0/groups and /v1.0/groups/{id}/members endpoints required by the entcollect provider's non-delta group fetch.
1 parent 0137f93 commit bcf1da7

34 files changed

Lines changed: 404 additions & 11 deletions

‎packages/entityanalytics_ad/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "0.22.0"
3+
changes:
4+
- description: Add agentless deployment mode and minimal-state sync support.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/20212
27
- version: "0.21.0"
38
changes:
49
- description: Add the Advanced Analytics (UEBA) subcategory

‎packages/entityanalytics_ad/data_stream/entity/_dev/test/scripts/minimal_state.txt‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Verify the minimal-state sync path routes entity documents to per-type data
2+
# streams without producing marker events in the entity data stream.
3+
#
4+
# This test explicitly sets use_minimal_state: true.
5+
#
6+
# The mock LDAP server provides:
7+
# - 1 user (Administrator) → entityanalytics_ad.user
8+
# - 1 device (WORKSTATION-01) → entityanalytics_ad.device
9+
# - 2 empty groups (Decommissioned, Contractors) → entityanalytics_ad.group
10+
11+
[!external_stack] skip 'Skipping external stack test.'
12+
[!exec:jq] skip 'Skipping test requiring absent jq command'
13+
14+
use_stack -profile ${CONFIG_PROFILES}/${PROFILE}
15+
16+
install_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME
17+
18+
exec ln -s ${PACKAGE_ROOT}/_dev/deploy/docker ldap-mock-service
19+
docker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} ldap-mock-service
20+
21+
add_package -profile ${CONFIG_PROFILES}/${PROFILE}
22+
23+
add_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml ENTITY_DS
24+
25+
# Verify user documents arrive in the user data stream.
26+
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -timeout 5m logs-entityanalytics_ad.user-*
27+
cp stdout user_docs.json
28+
exec jq '[.hits.hits[]._source.user.id] | unique[]' user_docs.json
29+
stdout 'S-1-5-21-374105552-1189110957-4047727897-500'
30+
exec jq '[.hits.hits[]._source.user.name] | unique[]' user_docs.json
31+
stdout 'Administrator'
32+
33+
# Verify device documents arrive in the device data stream.
34+
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -timeout 2m logs-entityanalytics_ad.device-*
35+
cp stdout device_docs.json
36+
exec jq '[.hits.hits[]._source.device.id] | unique[]' device_docs.json
37+
stdout 'S-1-5-21-374105552-1189110957-4047727897-1001'
38+
39+
# Verify empty group documents arrive in the group data stream.
40+
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 2 -timeout 2m logs-entityanalytics_ad.group-*
41+
cp stdout group_docs.json
42+
exec jq '[.hits.hits[]._source.group.id] | unique | sort[]' group_docs.json
43+
stdout 'S-1-5-21-374105552-1189110957-4047727897-1101'
44+
stdout 'S-1-5-21-374105552-1189110957-4047727897-1102'
45+
exec jq '[.hits.hits[]._source.group.name] | unique | sort[]' group_docs.json
46+
stdout 'Contractors'
47+
stdout 'Decommissioned'
48+
49+
# The entity data stream should have no documents (no markers in minimal mode).
50+
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 0 -timeout 30s ${ENTITY_DS}
51+
52+
# Clean up.
53+
remove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${ENTITY_DS}
54+
uninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m
55+
docker_down ldap-mock-service
56+
57+
-- test_config.yaml --
58+
input: entity-analytics
59+
data_stream:
60+
vars:
61+
ad_base_dn: "DC=testserver,DC=local"
62+
ad_url: "ldap://svc-ldap-mock-service:10389"
63+
ad_user: "cn=admin,dc=testserver,dc=local"
64+
ad_password: "password"
65+
dataset: all
66+
use_minimal_state: true
67+
include_empty_groups: true
68+
sync_interval: 24h
69+
update_interval: 15m

‎packages/entityanalytics_ad/data_stream/entity/_dev/test/scripts/minimal_state_default.txt‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Verify that the default configuration (use_minimal_state not set) uses the
2+
# minimal-state sync path. The package default is true, so the behavior should
3+
# be identical to setting it explicitly.
4+
#
5+
# The mock LDAP server provides:
6+
# - 1 user (Administrator) → entityanalytics_ad.user
7+
# - 1 device (WORKSTATION-01) → entityanalytics_ad.device
8+
# - 2 empty groups (Decommissioned, Contractors) → entityanalytics_ad.group
9+
10+
[!external_stack] skip 'Skipping external stack test.'
11+
[!exec:jq] skip 'Skipping test requiring absent jq command'
12+
13+
use_stack -profile ${CONFIG_PROFILES}/${PROFILE}
14+
15+
install_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME
16+
17+
exec ln -s ${PACKAGE_ROOT}/_dev/deploy/docker ldap-mock-service
18+
docker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} ldap-mock-service
19+
20+
add_package -profile ${CONFIG_PROFILES}/${PROFILE}
21+
22+
add_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml ENTITY_DS
23+
24+
# Verify user documents arrive in the user data stream.
25+
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -timeout 5m logs-entityanalytics_ad.user-*
26+
cp stdout user_docs.json
27+
exec jq '[.hits.hits[]._source.user.id] | unique[]' user_docs.json
28+
stdout 'S-1-5-21-374105552-1189110957-4047727897-500'
29+
exec jq '[.hits.hits[]._source.user.name] | unique[]' user_docs.json
30+
stdout 'Administrator'
31+
32+
# Verify device documents arrive in the device data stream.
33+
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -timeout 2m logs-entityanalytics_ad.device-*
34+
cp stdout device_docs.json
35+
exec jq '[.hits.hits[]._source.device.id] | unique[]' device_docs.json
36+
stdout 'S-1-5-21-374105552-1189110957-4047727897-1001'
37+
38+
# Verify empty group documents arrive in the group data stream.
39+
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 2 -timeout 2m logs-entityanalytics_ad.group-*
40+
cp stdout group_docs.json
41+
exec jq '[.hits.hits[]._source.group.id] | unique | sort[]' group_docs.json
42+
stdout 'S-1-5-21-374105552-1189110957-4047727897-1101'
43+
stdout 'S-1-5-21-374105552-1189110957-4047727897-1102'
44+
exec jq '[.hits.hits[]._source.group.name] | unique | sort[]' group_docs.json
45+
stdout 'Contractors'
46+
stdout 'Decommissioned'
47+
48+
# The entity data stream should have no documents (no markers in minimal mode).
49+
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 0 -timeout 30s ${ENTITY_DS}
50+
51+
# Clean up.
52+
remove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${ENTITY_DS}
53+
uninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m
54+
docker_down ldap-mock-service
55+
56+
-- test_config.yaml --
57+
input: entity-analytics
58+
data_stream:
59+
vars:
60+
ad_base_dn: "DC=testserver,DC=local"
61+
ad_url: "ldap://svc-ldap-mock-service:10389"
62+
ad_user: "cn=admin,dc=testserver,dc=local"
63+
ad_password: "password"
64+
dataset: all
65+
include_empty_groups: true
66+
sync_interval: 24h
67+
update_interval: 15m

‎packages/entityanalytics_ad/data_stream/entity/_dev/test/scripts/routed_data_streams.txt‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -41,14 +41,10 @@ exec jq '[.hits.hits[]._source.user.name] | unique[]' user_docs.json
4141
stdout 'Administrator'
4242

4343
# Verify device documents were routed to the device data stream.
44-
# NOTE: The device pipeline (device.yml) has a pre-existing bug — it references
45-
# activedirectory.device.* but the beats input puts device attributes under
46-
# activedirectory.user.*. The pipeline fails, so device.id retains the DN set
47-
# by the input rather than being overwritten with the SID.
4844
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -timeout 2m logs-entityanalytics_ad.device-*
4945
cp stdout device_docs.json
5046
exec jq '[.hits.hits[]._source.device.id] | unique[]' device_docs.json
51-
stdout 'WORKSTATION-01'
47+
stdout 'S-1-5-21-374105552-1189110957-4047727897-1001'
5248

5349
# Verify empty group documents were routed to the group data stream.
5450
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 2 -timeout 2m logs-entityanalytics_ad.group-*
@@ -74,6 +70,7 @@ data_stream:
7470
ad_user: "cn=admin,dc=testserver,dc=local"
7571
ad_password: "password"
7672
dataset: all
73+
use_minimal_state: false
7774
include_empty_groups: true
7875
sync_interval: 24h
7976
update_interval: 15m

‎packages/entityanalytics_ad/data_stream/entity/_dev/test/system/test-default-config.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,6 @@ data_stream:
1111
include_empty_groups: true
1212
sync_interval: 24h
1313
update_interval: 15m
14+
use_minimal_state: false
1415
assert:
1516
hit_count: 2

‎packages/entityanalytics_ad/data_stream/entity/_dev/test/system/test-special-chars-config.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,6 @@ data_stream:
1111
include_empty_groups: true
1212
sync_interval: 24h
1313
update_interval: 15m
14+
use_minimal_state: false
1415
assert:
1516
hit_count: 2

‎packages/entityanalytics_ad/data_stream/entity/agent/stream/entity-analytics.yml.hbs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
11
provider: activedirectory
2+
{{#if use_minimal_state}}
3+
use_minimal_state: true
4+
{{/if}}
25
sync_interval: {{sync_interval}}
36
update_interval: {{update_interval}}
47
ad_base_dn: {{ad_base_dn}}

‎packages/entityanalytics_ad/data_stream/entity/manifest.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,19 @@ streams:
174174
show_user: false
175175
description: >-
176176
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
177+
- name: use_minimal_state
178+
type: bool
179+
title: Use minimal state
180+
description: >-
181+
Use the minimal-state sync implementation. Required for agentless
182+
deployments. Uses reduced local storage and supports ES-backed state.
183+
multi: false
184+
required: false
185+
show_user: false
186+
default: true
187+
hide_in_deployment_modes:
188+
- default
189+
- agentless
177190
elasticsearch:
178191
dynamic_dataset: true
179192
dynamic_namespace: true

‎packages/entityanalytics_ad/manifest.yml‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
format_version: "3.0.2"
1+
format_version: "3.3.2"
22
name: entityanalytics_ad
33
title: Active Directory Entity Analytics
4-
version: "0.21.0"
4+
version: "0.22.0"
55
description: "Collect User Identities from Active Directory Entity with Elastic Agent."
66
type: integration
77
categories:
@@ -22,6 +22,15 @@ policy_templates:
2222
- name: entity
2323
title: Identities
2424
description: Collect identities.
25+
deployment_modes:
26+
default:
27+
enabled: true
28+
agentless:
29+
enabled: true
30+
release: beta
31+
organization: security
32+
division: engineering
33+
team: security-service-integrations
2534
inputs:
2635
- type: entity-analytics
2736
title: Collect identities

‎packages/entityanalytics_entra_id/_dev/deploy/docker/files/config.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,24 @@ rules:
1717
- status_code: 200
1818
body: |-
1919
{"@odata.deltaLink":"http://localhost:8080/v1.0/groups/delta?$skiptoken=1","value":[{"id":"e7089e3a-2c83-4f08-8280-7530ed39b6ca","name":"Group 5202","members@delta":[{"id":"aa534e49-edfd-4541-8256-8bbf34f122b4","type":"#microsoft.graph.user"},{"id":"feb6a386-612a-4ed1-9b13-2adc73074a19","type":"#microsoft.graph.user"}]},{"id":"526588ce-2828-4cb1-9c9b-e57026e94b82","name":"Group 16739","members@delta":[{"id":"aa534e49-edfd-4541-8256-8bbf34f122b4","type":"#microsoft.graph.user"}]}]}
20+
- path: /v1.0/groups
21+
methods: ["GET"]
22+
responses:
23+
- status_code: 200
24+
body: |-
25+
{"value":[{"id":"e7089e3a-2c83-4f08-8280-7530ed39b6ca","displayName":"Group 5202"},{"id":"526588ce-2828-4cb1-9c9b-e57026e94b82","displayName":"Group 16739"}]}
26+
- path: /v1.0/groups/e7089e3a-2c83-4f08-8280-7530ed39b6ca/members
27+
methods: ["GET"]
28+
responses:
29+
- status_code: 200
30+
body: |-
31+
{"value":[{"id":"aa534e49-edfd-4541-8256-8bbf34f122b4","@odata.type":"#microsoft.graph.user"},{"id":"feb6a386-612a-4ed1-9b13-2adc73074a19","@odata.type":"#microsoft.graph.user"}]}
32+
- path: /v1.0/groups/526588ce-2828-4cb1-9c9b-e57026e94b82/members
33+
methods: ["GET"]
34+
responses:
35+
- status_code: 200
36+
body: |-
37+
{"value":[{"id":"aa534e49-edfd-4541-8256-8bbf34f122b4","@odata.type":"#microsoft.graph.user"}]}
2038
- path: /v1.0/reports/authenticationMethods/userRegistrationDetails
2139
methods: ["GET"]
2240
responses:

0 commit comments

Comments
 (0)