diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 70fb4d23359..1ebcfc23e2d 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -431,6 +431,7 @@ /packages/panw_metrics @elastic/obs-infraobs-integrations /packages/pfsense @elastic/integration-experience /packages/php_fpm @elastic/obs-infraobs-integrations +/packages/ping_directory @elastic/security-service-integrations /packages/ping_federate @elastic/security-service-integrations @elastic/sit-crest-contractors /packages/ping_one @elastic/security-service-integrations @elastic/sit-crest-contractors /packages/platform_observability @elastic/stack-monitoring @@ -613,4 +614,4 @@ /packages/zscaler_zia @elastic/security-service-integrations @elastic/sit-crest-contractors /packages/zscaler_zpa @elastic/security-service-integrations @elastic/sit-crest-contractors /packages/otel_rum_dashboards @elastic/apm-agent-rum -/packages/otel_android_dashboards @elastic/apm-agent-android @elastic/apm-agent-approvers \ No newline at end of file +/packages/otel_android_dashboards @elastic/apm-agent-android @elastic/apm-agent-approvers diff --git a/packages/ping_directory/_dev/build/build.yml b/packages/ping_directory/_dev/build/build.yml new file mode 100644 index 00000000000..32921896292 --- /dev/null +++ b/packages/ping_directory/_dev/build/build.yml @@ -0,0 +1,3 @@ +dependencies: + ecs: + reference: git@v9.3.0 diff --git a/packages/ping_directory/_dev/build/docs/README.md b/packages/ping_directory/_dev/build/docs/README.md new file mode 100644 index 00000000000..b4aab5ce96e --- /dev/null +++ b/packages/ping_directory/_dev/build/docs/README.md @@ -0,0 +1,149 @@ +# PingDirectory Integration + +## Overview + +[PingDirectory](https://www.pingidentity.com/en/product/pingdirectory.html) is an **enterprise-grade LDAP directory server** and identity data store. It provides high-performance, scalable directory services for managing user identities, credentials, and access control across hybrid environments — combining authentication, authorization, and directory data management into a unified platform for **critical identity infrastructure**. + +This integration collects group membership data from PingDirectory via the SCIM v2 API using the Elastic Agent CEL input. It provides visibility into group accounts and membership attributes managed within your PingDirectory environment. + +### Compatibility + +The PingDirectory integration is compatible with **PingDirectory version 11.0.0.2 and above**. + +### How it works + +The integration uses the Elastic Agent CEL input to collect group data from the PingDirectory SCIM v2 API. Each collection cycle: + +1. Authenticates against the PingDirectory REST API (`/directory/v1/authenticate`) using the configured `bind_dn` and `password` to obtain a short-lived access token. +2. Paginates through all groups via the SCIM v2 Groups endpoint (`/scim/v2/Groups`) using the configured batch size. +3. Emits each group record as an individual event for ingestion and enrichment via the built-in ingest pipeline. +4. Caches the access token across collection cycles and re-authenticates automatically when the token expires. + +## What data does this integration collect? + +The PingDirectory integration collects the following types of data: + +| Data stream | Description | Endpoint | +|---|---|---| +| `group` | Group membership records retrieved from the PingDirectory SCIM v2 API, including group names, display names, members, and group metadata. | `/scim/v2/Groups` | + +### Supported use cases + +* **Group inventory and membership visibility**: Track all groups managed in PingDirectory, including their members, display names, and group types. + +* **Identity data enrichment**: Correlate PingDirectory group records with other security and operational data in Elastic for unified identity context. + +## What do I need to use this integration? + +### From PingDirectory + +* **PingDirectory deployment**: An active PingDirectory server with the SCIM v2 API enabled and accessible. +* **Service account credentials**: A bind DN (`bind_dn`) and password with sufficient permissions to authenticate via `/directory/v1/authenticate` and read groups from `/scim/v2/Groups`. +* **Network access**: Elastic Agent must be able to reach the PingDirectory HTTPS endpoint. +* **SSL certificate**: If PingDirectory uses a self-signed certificate, the certificate must be trusted by the Elastic Agent host. +* **Elastic Agent**: Version 8.18+ or 9.0+ with Fleet enrollment. + +## How do I deploy this integration? + +This integration supports both Elastic Agentless-based and Agent-based installations. + +### Agent-based deployment + +Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host. + +Elastic Agent is required to poll the PingDirectory SCIM v2 API and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines. + +### Agentless deployment + +Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. Agentless deployments provide a means to ingest data while avoiding the orchestration, management, and maintenance needs associated with standard ingest infrastructure. Using an agentless deployment makes manual agent deployment unnecessary, allowing you to focus on your data instead of the agent that collects it. + +For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html) + +### Set up steps in PingDirectory + +Before configuring the integration, ensure that a PingDirectory account is available for API authentication and group retrieval. + +1. Create or identify a service account in PingDirectory. +2. Grant the account permission to: + + * Authenticate through the `/directory/v1/authenticate` endpoint. + * Read group records through the `/scim/v2/Groups` endpoint. + +3. Record the account's bind DN and password. These values are required when configuring the integration in Kibana. +4. Verify that the SCIM v2 API is enabled and accessible from the Elastic Agent host. +5. If HTTPS is configured with a self-signed certificate, export the certificate so it can be trusted by Elastic Agent. + +Refer to the PingDirectory documentation for configuring SCIM 2.0 and authentication requirements. + +### Configure + +1. In Kibana, navigate to **Fleet → Integrations** and search for **PingDirectory**. +2. Click **Add PingDirectory**. +3. Configure the integration settings: + + * **URL**: The base URL of your PingDirectory instance, for example: + `https://pingdirectory.example.com:2443` + * **Bind DN**: The distinguished name used to authenticate, for example: + `cn=admin,dc=example,dc=com` + * **Password**: The password for the bind DN. + * **Batch Size**: Number of group records to retrieve per API page (default: `500`). + * **Interval**: How frequently to poll for new data (default: `24h`). +4. If using a self-signed SSL certificate, configure the SSL settings under **Advanced options**. +5. Select **Save and continue** to save the integration. +6. Add the integration to an existing Agent policy or create a new one. +7. Verify that group records are being ingested into Elasticsearch. + +### Validation + +#### Dashboard populated + +1. In the top search bar in Kibana, search for **Dashboards**. +2. In the search bar, type **PingDirectory**. +3. Open the **[Logs PingDirectory] Group** dashboard. +4. Verify that the visualizations are populated with group membership data, including group names, members, and membership counts. + +## Scaling + +For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation. + +## Troubleshooting + +* **No data collected**: Verify that the PingDirectory SCIM v2 API is enabled and reachable from the Elastic Agent host. Confirm that the configured URL, bind DN, and password are correct. +* **Authentication failures**: Ensure the bind DN has permission to authenticate via `/directory/v1/authenticate` and read from `/scim/v2/Groups`. +* **SSL certificate errors**: If PingDirectory uses a self-signed certificate, extract the certificate and configure it under the SSL settings of the integration, or add it to the Elastic Agent's trusted certificate store. +* **Token expiry**: The integration automatically re-authenticates when the access token expires. If repeated auth failures occur, verify the bind DN password has not changed. + +For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems). + +## Reference + +#### Vendor documentation links + +- [Configuring SCIM 2.0 on your server](https://docs.pingidentity.com/pingdirectory/9.3/pingdirectory_server_administration_guide/pd_ds_config_scim_2_server.html) +- [PingDirectory Directory REST API Getting Started](https://developer.pingidentity.com/pingdirectory/directory/getting-started.html) +- [PingDirectory REST API Authentication](https://developer.pingidentity.com/pingdirectory/directory/authentication.html) +- [SCIM 2.0 Users API](https://developer.pingidentity.com/pingdirectory/directory-proxy-scim/overview.html) +- [SCIM 2.0 Groups API](https://developer.pingidentity.com/pingdirectory/directory-proxy-scim/user-profile-endpoints/get-read-search-group-members-display-name.html) + +### Group + +The `group` data stream provides group membership records collected from PingDirectory. + +#### group fields + +{{fields "group"}} + +{{event "group"}} + +### Inputs used + +{{ inputDocs }} + +### API usage + +These PingDirectory REST API endpoints are used by this integration: + +| Endpoint | Method | Data stream | Description | +|---|---|---|---| +| `/directory/v1/authenticate` | POST | group | Authenticate with bind DN and password to obtain an access token | +| `/scim/v2/Groups` | GET | group | Retrieve paginated group membership records | \ No newline at end of file diff --git a/packages/ping_directory/_dev/deploy/docker/docker-compose.yml b/packages/ping_directory/_dev/deploy/docker/docker-compose.yml new file mode 100644 index 00000000000..53ef21de0e1 --- /dev/null +++ b/packages/ping_directory/_dev/deploy/docker/docker-compose.yml @@ -0,0 +1,15 @@ +version: '3.8' +services: + ping_directory: + image: docker.elastic.co/observability/stream:v0.22.0 + hostname: ping_directory + ports: + - 8090 + volumes: + - ./files:/files:ro + environment: + PORT: '8090' + command: + - http-server + - --addr=:8090 + - --config=/files/config.yml diff --git a/packages/ping_directory/_dev/deploy/docker/files/config.yml b/packages/ping_directory/_dev/deploy/docker/files/config.yml new file mode 100644 index 00000000000..656be1ff64e --- /dev/null +++ b/packages/ping_directory/_dev/deploy/docker/files/config.yml @@ -0,0 +1,162 @@ +rules: + - path: /directory/v1/authenticate + methods: ['POST'] + request_headers: + Content-Type: + - 'application/json' + request_body: '{"credentials":{"authenticationType":"password","dn":"xxxx","staticPassword":"xxxx"}}' + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "accessToken": "xxxx" + } + `}} + - path: /scim/v2/Groups + methods: ['GET'] + query_params: + startIndex: 5 + count: 2 + request_headers: + Authorization: + - "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "schemas": ["urn:ietf:params:scim:api:messages:2.0:ListResponse"], + "totalResults": 5, + "startIndex": 5, + "Resources": [ + { + "displayName": "platform-admins", + "members": [ + "uid=john.doe,ou=People,dc=example,dc=com", + "uid=alice.smith,ou=People,dc=example,dc=com" + ], + "id": "d1964i47-f0he-723i-1e27-057e01574h46", + "meta": { + "resourceType": "Group", + "location": "https://10.50.15.29:2443/scim/v2/Groups/d1964i47-f0he-723i-1e27-057e01574h46" + }, + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:Group" + ] + } + ] + } + `}} + - path: /scim/v2/Groups + methods: ['GET'] + query_params: + startIndex: 3 + count: 2 + request_headers: + Authorization: + - "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "schemas": ["urn:ietf:params:scim:api:messages:2.0:ListResponse"], + "totalResults": 5, + "startIndex": 3, + "Resources": [ + { + "displayName": "engineering", + "members": [ + "uid=bob.jones,ou=Engineering,dc=example,dc=com" + ], + "id": "b8742g25-d8fc-501g-9c05-835c89352f24", + "meta": { + "resourceType": "Group", + "location": "https://10.50.15.29:2443/scim/v2/Groups/b8742g25-d8fc-501g-9c05-835c89352f24" + }, + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:Group" + ] + }, + { + "displayName": "finance", + "members": [ + "uid=carol.white,ou=Finance,dc=corp,dc=internal" + ], + "id": "c9853h36-e9gd-612h-0d16-946d90463g35", + "meta": { + "resourceType": "Group", + "location": "https://10.50.15.29:2443/scim/v2/Groups/c9853h36-e9gd-612h-0d16-946d90463g35" + }, + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:Group" + ] + } + ] + } + `}} + - path: /scim/v2/Groups + methods: ['GET'] + query_params: + startIndex: 1 + count: 2 + request_headers: + Authorization: + - "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "schemas": ["urn:ietf:params:scim:api:messages:2.0:ListResponse"], + "totalResults": 5, + "startIndex": 1, + "Resources": [ + { + "displayName": "security", + "members": [ + "uid=alice.smith,ou=People,dc=example,dc=com" + ], + "id": "a7631f14-c7eb-490f-8b94-724b78241e13", + "meta": { + "resourceType": "Group", + "location": "https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13" + }, + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:Group" + ] + }, + { + "displayName": "test-group", + "members": { + "value": [ + "uid=user.0,ou=People,dc=example,dc=com", + "uid=user.1,ou=People,dc=example,dc=com", + "uid=user.2,ou=People,dc=example,dc=com" + ] + }, + "id": "1a8e021b-c7b0-4bd9-8e2d-af5691c035ce", + "meta": { + "resourceType": "Groups", + "location": "https://example.com:443/scim/v2/Groups/1a8e021b-c7b0-4bd9-8e2d-af5691c035ce" + }, + "schemas": [ + "urn:pingidentity:schemas:Group:1.0" + ] + } + ] + } + `}} diff --git a/packages/ping_directory/changelog.yml b/packages/ping_directory/changelog.yml new file mode 100644 index 00000000000..bd3cb8acdc1 --- /dev/null +++ b/packages/ping_directory/changelog.yml @@ -0,0 +1,6 @@ +# newer versions go on top +- version: '0.1.0' + changes: + - description: Add support of group datastream. + type: enhancement + link: https://github.com/elastic/integrations/pull/19883 diff --git a/packages/ping_directory/data_stream/group/_dev/test/pipeline/test-group.log b/packages/ping_directory/data_stream/group/_dev/test/pipeline/test-group.log new file mode 100644 index 00000000000..8d1de16a9be --- /dev/null +++ b/packages/ping_directory/data_stream/group/_dev/test/pipeline/test-group.log @@ -0,0 +1,4 @@ +{"displayName":"security","members":["uid=alice.smith,ou=People,dc=example,dc=com"],"id":"a7631f14-c7eb-490f-8b94-724b78241e13","meta":{"resourceType":"Group","location":"https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:Group"]} +{"displayName":"test-group","members":{"value":["uid=user.0,ou=People,dc=example,dc=com","uid=user.1,ou=People,dc=example,dc=com","uid=user.2,ou=People,dc=example,dc=com","uid=user.3,ou=People,dc=example,dc=com","uid=user.4,ou=People,dc=example,dc=com","uid=user.5,ou=People,dc=example,dc=com","uid=user.6,ou=People,dc=example,dc=com","uid=user.7,ou=People,dc=example,dc=com","uid=user.8,ou=People,dc=example,dc=com","uid=user.9,ou=People,dc=example,dc=com"]},"id":"1a8e021b-c7b0-4bd9-8e2d-af5691c035ce","meta":{"resourceType":"Groups","location":"https://example.com:443/scim/v2/Groups/1a8e021b-c7b0-4bd9-8e2d-af5691c035ce"},"schemas":["urn:pingidentity:schemas:Group:1.0"]} +{"displayName":"engineering","members":["uid=bob.jones,ou=Engineering,dc=example,dc=com"],"id":"b8742g25-d8fc-501g-9c05-835c89352f24","meta":{"resourceType":"Group","location":"https://10.50.15.29:2443/scim/v2/Groups/b8742g25-d8fc-501g-9c05-835c89352f24"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:Group"]} +{"displayName":"finance","members":["uid=carol.white,ou=Finance,dc=corp,dc=internal"],"id":"c9853h36-e9gd-612h-0d16-946d90463g35","meta":{"resourceType":"Group","location":"https://10.50.15.29:2443/scim/v2/Groups/c9853h36-e9gd-612h-0d16-946d90463g35"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:Group"]} diff --git a/packages/ping_directory/data_stream/group/_dev/test/pipeline/test-group.log-expected.json b/packages/ping_directory/data_stream/group/_dev/test/pipeline/test-group.log-expected.json new file mode 100644 index 00000000000..0766506d41f --- /dev/null +++ b/packages/ping_directory/data_stream/group/_dev/test/pipeline/test-group.log-expected.json @@ -0,0 +1,180 @@ +{ + "expected": [ + { + "ecs": { + "version": "9.3.0" + }, + "event": { + "category": [ + "iam" + ], + "kind": "event", + "original": "{\"displayName\":\"security\",\"members\":[\"uid=alice.smith,ou=People,dc=example,dc=com\"],\"id\":\"a7631f14-c7eb-490f-8b94-724b78241e13\",\"meta\":{\"resourceType\":\"Group\",\"location\":\"https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13\"},\"schemas\":[\"urn:ietf:params:scim:schemas:core:2.0:Group\"]}", + "type": [ + "group" + ] + }, + "group": { + "id": "a7631f14-c7eb-490f-8b94-724b78241e13", + "name": "security" + }, + "ping_directory": { + "group": { + "members_metadata": "ou=People,dc=example,dc=com", + "meta": { + "location": "https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13", + "resource_type": "Group" + }, + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:Group" + ] + } + }, + "related": { + "user": [ + "alice.smith" + ] + }, + "user": { + "id": "alice.smith" + } + }, + { + "ecs": { + "version": "9.3.0" + }, + "event": { + "category": [ + "iam" + ], + "kind": "event", + "original": "{\"displayName\":\"test-group\",\"members\":{\"value\":[\"uid=user.0,ou=People,dc=example,dc=com\",\"uid=user.1,ou=People,dc=example,dc=com\",\"uid=user.2,ou=People,dc=example,dc=com\",\"uid=user.3,ou=People,dc=example,dc=com\",\"uid=user.4,ou=People,dc=example,dc=com\",\"uid=user.5,ou=People,dc=example,dc=com\",\"uid=user.6,ou=People,dc=example,dc=com\",\"uid=user.7,ou=People,dc=example,dc=com\",\"uid=user.8,ou=People,dc=example,dc=com\",\"uid=user.9,ou=People,dc=example,dc=com\"]},\"id\":\"1a8e021b-c7b0-4bd9-8e2d-af5691c035ce\",\"meta\":{\"resourceType\":\"Groups\",\"location\":\"https://example.com:443/scim/v2/Groups/1a8e021b-c7b0-4bd9-8e2d-af5691c035ce\"},\"schemas\":[\"urn:pingidentity:schemas:Group:1.0\"]}", + "type": [ + "group" + ] + }, + "group": { + "id": "1a8e021b-c7b0-4bd9-8e2d-af5691c035ce", + "name": "test-group" + }, + "ping_directory": { + "group": { + "members_metadata": "ou=People,dc=example,dc=com", + "meta": { + "location": "https://example.com:443/scim/v2/Groups/1a8e021b-c7b0-4bd9-8e2d-af5691c035ce", + "resource_type": "Groups" + }, + "schemas": [ + "urn:pingidentity:schemas:Group:1.0" + ] + } + }, + "related": { + "user": [ + "user.0", + "user.1", + "user.2", + "user.3", + "user.4", + "user.5", + "user.6", + "user.7", + "user.8", + "user.9" + ] + }, + "user": { + "id": [ + "user.0", + "user.1", + "user.2", + "user.3", + "user.4", + "user.5", + "user.6", + "user.7", + "user.8", + "user.9" + ] + } + }, + { + "ecs": { + "version": "9.3.0" + }, + "event": { + "category": [ + "iam" + ], + "kind": "event", + "original": "{\"displayName\":\"engineering\",\"members\":[\"uid=bob.jones,ou=Engineering,dc=example,dc=com\"],\"id\":\"b8742g25-d8fc-501g-9c05-835c89352f24\",\"meta\":{\"resourceType\":\"Group\",\"location\":\"https://10.50.15.29:2443/scim/v2/Groups/b8742g25-d8fc-501g-9c05-835c89352f24\"},\"schemas\":[\"urn:ietf:params:scim:schemas:core:2.0:Group\"]}", + "type": [ + "group" + ] + }, + "group": { + "id": "b8742g25-d8fc-501g-9c05-835c89352f24", + "name": "engineering" + }, + "ping_directory": { + "group": { + "members_metadata": "ou=Engineering,dc=example,dc=com", + "meta": { + "location": "https://10.50.15.29:2443/scim/v2/Groups/b8742g25-d8fc-501g-9c05-835c89352f24", + "resource_type": "Group" + }, + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:Group" + ] + } + }, + "related": { + "user": [ + "bob.jones" + ] + }, + "user": { + "id": "bob.jones" + } + }, + { + "ecs": { + "version": "9.3.0" + }, + "event": { + "category": [ + "iam" + ], + "kind": "event", + "original": "{\"displayName\":\"finance\",\"members\":[\"uid=carol.white,ou=Finance,dc=corp,dc=internal\"],\"id\":\"c9853h36-e9gd-612h-0d16-946d90463g35\",\"meta\":{\"resourceType\":\"Group\",\"location\":\"https://10.50.15.29:2443/scim/v2/Groups/c9853h36-e9gd-612h-0d16-946d90463g35\"},\"schemas\":[\"urn:ietf:params:scim:schemas:core:2.0:Group\"]}", + "type": [ + "group" + ] + }, + "group": { + "id": "c9853h36-e9gd-612h-0d16-946d90463g35", + "name": "finance" + }, + "ping_directory": { + "group": { + "members_metadata": "ou=Finance,dc=corp,dc=internal", + "meta": { + "location": "https://10.50.15.29:2443/scim/v2/Groups/c9853h36-e9gd-612h-0d16-946d90463g35", + "resource_type": "Group" + }, + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:Group" + ] + } + }, + "related": { + "user": [ + "carol.white" + ] + }, + "user": { + "id": "carol.white" + } + } + ] +} diff --git a/packages/ping_directory/data_stream/group/_dev/test/system/test-default-config.yml b/packages/ping_directory/data_stream/group/_dev/test/system/test-default-config.yml new file mode 100644 index 00000000000..80c658439b3 --- /dev/null +++ b/packages/ping_directory/data_stream/group/_dev/test/system/test-default-config.yml @@ -0,0 +1,12 @@ +input: cel +service: ping_directory +vars: + url: http://{{Hostname}}:{{Port}} + bind_dn: "xxxx" + password: "xxxx" +data_stream: + vars: + preserve_original_event: true + batch_size: 2 +assert: + hit_count: 5 diff --git a/packages/ping_directory/data_stream/group/agent/stream/cel.yml.hbs b/packages/ping_directory/data_stream/group/agent/stream/cel.yml.hbs new file mode 100644 index 00000000000..dd6484c737a --- /dev/null +++ b/packages/ping_directory/data_stream/group/agent/stream/cel.yml.hbs @@ -0,0 +1,158 @@ +interval: {{interval}} +resource.tracer: + enabled: {{enable_request_tracer}} + filename: "../../logs/cel/http-request-trace-*.ndjson" + maxbackups: 5 +{{#if proxy_url}} +resource.proxy_url: {{proxy_url}} +{{/if}} +{{#if ssl}} +resource.ssl: {{ssl}} +{{/if}} +{{#if http_client_timeout}} +resource.timeout: {{http_client_timeout}} +{{/if}} +resource.url: {{url}} +state: + bind_dn: {{bind_dn}} + password: {{password}} + batch_size: {{batch_size}} +redact: + fields: + - bind_dn + - password +program: | + state.url.trim_right("/").as(base, + state.with( + ( + state.?cursor.access_token.orValue("") == "" || + state.?cursor.need_reauth.orValue(false) + ) ? + post_request( + base + "/directory/v1/authenticate", + "application/json", + { + "credentials": { + "authenticationType": "password", + "dn": state.bind_dn, + "staticPassword": state.password, + }, + }.encode_json() + ).do_request().as(auth_resp, + (auth_resp.StatusCode == 200) ? + auth_resp.Body.decode_json().as(auth, { + "ok": true, + "access": string(auth.accessToken), + }) + : + { + "ok": false, + "err_code": string(auth_resp.StatusCode), + "err_status": string(auth_resp.Status), + "err_body": (size(auth_resp.Body) != 0) ? string(auth_resp.Body) : string(auth_resp.Status), + } + ) + : + { + "ok": true, + "access": state.?cursor.access_token.orValue(""), + } + ).as(tok, + !tok.ok ? + { + "events": [{ + "error": { + "code": tok.err_code, + "id": tok.err_status, + "message": "POST " + base + "/directory/v1/authenticate: " + tok.err_body, + } + }], + "want_more": false, + "cursor": { + "need_reauth": true, // First reauth, true + }, + "bind_dn": state.bind_dn, + "password": state.password, + "batch_size": state.batch_size, + } + : + request( + "GET", + base + "/scim/v2/Groups?" + { + "startIndex": [string(state.?start_index.orValue(1))], + "count": [string(state.batch_size)], + }.format_query() + ).with({ + "Header": { + "Authorization": ["Bearer " + tok.access], + "Accept": ["application/json"], + }, + }).do_request().as(resp, + (resp.StatusCode == 200) ? + resp.Body.decode_json().as(data, + (int(data.startIndex) + size(data.Resources)).as(next_idx, { + "events": data.?Resources.orValue([]).map(u, {"message": u.encode_json()}), + "cursor": { + "access_token": tok.access, + "need_reauth": false, // ← all good + }, + "want_more": next_idx <= int(data.totalResults), + "start_index": next_idx <= int(data.totalResults) ? next_idx : 1, + "bind_dn": state.bind_dn, + "password": state.password, + "batch_size": state.batch_size, + }) + ) + : (resp.StatusCode == 401 || resp.StatusCode == 419) ? + { + "events": [{ + "message": "retry"}], + "cursor": { + "need_reauth": true, // Second reauth, true + }, + "want_more": true, // ← retry immediately + "start_index": state.?start_index.orValue(1), + "bind_dn": state.bind_dn, + "password": state.password, + "batch_size": state.batch_size, + } + : + { + "events": [{ + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET " + base + "/scim/v2/Groups: " + ( + size(resp.Body) != 0 ? + string(resp.Body) + : + string(resp.Status) + " (" + string(resp.StatusCode) + ")" + ), + } + }], + "want_more": false, + "start_index": state.?start_index.orValue(1), + "cursor": { + "access_token": tok.access, + "need_reauth": false, // ← third reauth, false + }, + + "batch_size": state.batch_size, + } + ) + ) + ) +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/ping_directory/data_stream/group/elasticsearch/ingest_pipeline/default.yml b/packages/ping_directory/data_stream/group/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..8fedbcde951 --- /dev/null +++ b/packages/ping_directory/data_stream/group/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,240 @@ +--- +description: Pipeline for processing group logs. +processors: + - set: + field: ecs.version + tag: set_ecs_version + value: 9.3.0 + - terminate: + tag: data_collection_error + if: ctx.error?.message != null && ctx.message == null && ctx.event?.original == null + description: error message set and no data to process. + - drop: + tag: drop_retry_events + if: ctx.message == 'retry' + - remove: + field: + - organization + - division + - team + tag: remove_agentless_tags + if: ctx.organization instanceof String && ctx.division instanceof String && ctx.team instanceof String + ignore_missing: true + description: Removes the fields added by Agentless as metadata, as they can collide with ECS fields. + - rename: + field: message + tag: rename_message_to_event_original + if: ctx.event?.original == null + target_field: event.original + ignore_missing: true + description: Renames the original `message` field to `event.original` to store a copy of the original message. + - remove: + field: + - message + tag: remove_message + if: ctx.event?.original != null + ignore_missing: true + description: The `message` field is no longer required if the document has an `event.original` field. + - json: + field: event.original + tag: json_event_original + target_field: ping_directory.group + - fingerprint: + tag: fingerprint_with_id + fields: + - ping_directory.group.id + target_field: _id + ignore_missing: true + - script: + description: > + Convert field names from camelCase to snake_case recursively. + tag: script_convert_camelcase_to_snake_case + lang: painless + source: |- + String camelToSnake(String str) { + def result = ""; + for (int i = 0; i < str.length(); i++) { + char c = str.charAt(i); + if (Character.isUpperCase(c)) { + if (i > 0 && Character.isLowerCase(str.charAt(i - 1))) { + result += "_"; + } + result += Character.toLowerCase(c); + } else { + result += c; + } + } + return result; + } + def convertToSnakeCase(def obj) { + if (obj instanceof Map) { + def newObj = [:]; + for (entry in obj.entrySet()) { + String newKey = camelToSnake(entry.getKey()); + newObj[newKey] = convertToSnakeCase(entry.getValue()); + } + return newObj; + } else if (obj instanceof List) { + def newList = []; + for (item in obj) { + newList.add(convertToSnakeCase(item)); + } + return newList; + } else { + return obj; + } + } + if (ctx.ping_directory?.group != null) { + ctx.ping_directory.group = convertToSnakeCase(ctx.ping_directory.group); + } + on_failure: + - append: + tag: append_script_convert_camelcase_to_snake_case_error + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: event.kind + tag: set_event_kind + value: event + - append: + field: event.category + tag: append_iam_into_event_category + value: iam + - append: + field: event.type + tag: append_group_into_event_type + value: group + - set: + field: group.id + tag: set_group_id_from_group_id + copy_from: ping_directory.group.id + ignore_empty_value: true + - set: + field: group.name + tag: set_group_name_from_group_display_name + copy_from: ping_directory.group.display_name + ignore_empty_value: true + - script: + tag: script_append_members_to_related_user + description: > + Process group members (plain array or object with value array). + Extracts uid from DN, appends to related.user and user.id. + Stores remaining DN (without uid) deduplicated as plain string into members_metadata. + lang: painless + source: | + def members = ctx.ping_directory?.group?.members; + if (members instanceof Map && members.value instanceof List) { + members = members.value; + } + if (!(members instanceof List)) return; + if (ctx.related == null) ctx.related = new HashMap(); + if (ctx.related.user == null) ctx.related.user = []; + if (ctx.user == null) ctx.user = new HashMap(); + def uidList = new ArrayList(); + def metaSet = new LinkedHashSet(); + for (def member : members) { + if (!(member instanceof String)) continue; + def uid = null; + def dnWithoutUid = new ArrayList(); + def parts = member.splitOnToken(','); + for (def part : parts) { + def kv = part.trim().splitOnToken('='); + if (kv.length == 2) { + def key = kv[0].trim().toLowerCase(); + if (key == 'uid') { + uid = kv[1].trim(); + } else { + dnWithoutUid.add(part.trim()); + } + } + } + if (uid != null) { + if (!ctx.related.user.contains(uid)) { + ctx.related.user.add(uid); + } + if (!uidList.contains(uid)) { + uidList.add(uid); + } + } + if (!dnWithoutUid.isEmpty()) { + metaSet.add(String.join(',', dnWithoutUid)); + } + } + if (!uidList.isEmpty()) { + ctx.user.id = uidList.size() == 1 ? uidList[0] : uidList; + } + if (!metaSet.isEmpty()) { + def metaList = new ArrayList(metaSet); + ctx.ping_directory.group.members_metadata = metaList.size() == 1 ? metaList[0] : metaList; + } + ctx.ping_directory.group.remove('members'); + on_failure: + - append: + tag: append_script_append_members_to_related_user_error + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - remove: + field: + - ping_directory.group.id + - ping_directory.group.display_name + tag: remove_fields_mapped_to_ecs + ignore_missing: true + - script: + tag: remove_null_values + lang: painless + description: Remove null or empty values from the document. + source: |- + void handleMap(Map map) { + map.values().removeIf(v -> { + if (v instanceof Map) { + handleMap(v); + } else if (v instanceof List) { + handleList(v); + } + return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0) + }); + } + void handleList(List list) { + list.removeIf(v -> { + if (v instanceof Map) { + handleMap(v); + } else if (v instanceof List) { + handleList(v); + } + return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0) + }); + } + handleMap(ctx); + on_failure: + - append: + tag: append_remove_null_values_error + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: event.kind + tag: set_pipeline_error_into_event_kind + if: ctx.error?.message != null + value: pipeline_error + - append: + field: tags + tag: append_tags + if: ctx.error?.message != null + value: preserve_original_event + allow_duplicates: false +on_failure: + - append: + tag: append_pipeline_failure_message + field: error.message + value: |- + Processor '{{{_ingest.on_failure_processor_type}}}' + {{{#_ingest.on_failure_processor_tag}}}with tag '{{{_ingest.on_failure_processor_tag}}}' + {{{/_ingest.on_failure_processor_tag}}}failed with message '{{{_ingest.on_failure_message}}}' + - set: + field: event.kind + tag: set_pipeline_error_to_event_kind + value: pipeline_error + - append: + field: tags + tag: append_tags + value: preserve_original_event + allow_duplicates: false diff --git a/packages/ping_directory/data_stream/group/fields/base-fields.yml b/packages/ping_directory/data_stream/group/fields/base-fields.yml new file mode 100644 index 00000000000..a50514dad56 --- /dev/null +++ b/packages/ping_directory/data_stream/group/fields/base-fields.yml @@ -0,0 +1,22 @@ +- name: data_stream.dataset + external: ecs + description: Name of the dataset associated with the event. +- name: data_stream.namespace + external: ecs + description: Namespace for the data stream. +- name: data_stream.type + external: ecs + description: Type of data stream. +- name: event.dataset + type: constant_keyword + external: ecs + value: ping_directory.group + description: Dataset name for events collected from PingDirectory group records. +- name: event.module + type: constant_keyword + external: ecs + value: ping_directory + description: Module name for PingDirectory events. +- name: '@timestamp' + external: ecs + description: Date and time when the event occurred. diff --git a/packages/ping_directory/data_stream/group/fields/beats.yml b/packages/ping_directory/data_stream/group/fields/beats.yml new file mode 100644 index 00000000000..3382e376e77 --- /dev/null +++ b/packages/ping_directory/data_stream/group/fields/beats.yml @@ -0,0 +1,3 @@ +- name: input.type + type: keyword + description: Type of filebeat input. diff --git a/packages/ping_directory/data_stream/group/fields/ecs.yml b/packages/ping_directory/data_stream/group/fields/ecs.yml new file mode 100644 index 00000000000..8c340d0b9e6 --- /dev/null +++ b/packages/ping_directory/data_stream/group/fields/ecs.yml @@ -0,0 +1,10 @@ +- name: observer.product + external: ecs + type: constant_keyword + value: PingDirectory + description: Product name of the observer that generated the event. +- name: observer.vendor + external: ecs + type: constant_keyword + value: PingIdentity + description: Vendor name of the observer that generated the event. diff --git a/packages/ping_directory/data_stream/group/fields/fields.yml b/packages/ping_directory/data_stream/group/fields/fields.yml new file mode 100644 index 00000000000..3ecdaba1fdc --- /dev/null +++ b/packages/ping_directory/data_stream/group/fields/fields.yml @@ -0,0 +1,21 @@ +- name: ping_directory + type: group + fields: + - name: group + type: group + fields: + - name: members_metadata + type: keyword + description: Member DN metadata with uid removed (ou and dc components only). + - name: meta + type: group + fields: + - name: location + type: keyword + description: Self-link URL of the group resource. + - name: resource_type + type: keyword + description: Always Group. Identifies the SCIM resource type. + - name: schemas + type: keyword + description: SCIM schema URN for the group resource. diff --git a/packages/ping_directory/data_stream/group/manifest.yml b/packages/ping_directory/data_stream/group/manifest.yml new file mode 100644 index 00000000000..2e69f79ab07 --- /dev/null +++ b/packages/ping_directory/data_stream/group/manifest.yml @@ -0,0 +1,65 @@ +title: 'Group' +type: logs +streams: + - input: cel + title: 'Collect PingDirectory Group logs via SCIM v2 API' + description: 'Collect group logs from PingDirectory via SCIM v2 REST API.' + template_path: cel.yml.hbs + vars: + - name: interval + type: text + title: Interval + description: Duration between full sync requests to PingDirectory SCIM API. Supported units h/m/s. + multi: false + required: true + show_user: true + default: 24h + - name: batch_size + type: integer + title: Batch Size + description: Number of records fetched per page from PingDirectory SCIM API. Maps to the count parameter in SCIM pagination. Maximum supported value is 500. + multi: false + required: true + show_user: false + default: 500 + - name: http_client_timeout + type: text + title: HTTP Client Timeout + description: Duration before declaring that the HTTP client connection has timed out. Supported time units are ns, us, ms, s, m, h. + multi: false + required: true + show_user: false + default: 60s + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - ping_directory-group + - name: preserve_original_event + required: true + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field event.original. + type: bool + multi: false + default: false + - name: enable_request_tracer + type: bool + title: Enable request tracing + multi: false + default: false + required: false + show_user: false + description: >- + The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. Disabling the request tracer will delete any stored traces. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_enable) for details. + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. diff --git a/packages/ping_directory/data_stream/group/sample_event.json b/packages/ping_directory/data_stream/group/sample_event.json new file mode 100644 index 00000000000..686e19ddffc --- /dev/null +++ b/packages/ping_directory/data_stream/group/sample_event.json @@ -0,0 +1,68 @@ +{ + "@timestamp": "2026-06-23T05:26:00.102Z", + "agent": { + "ephemeral_id": "9deb4381-f13b-460f-a113-d50474f421d6", + "id": "b8e1b118-3146-4215-8504-c18df1838432", + "name": "elastic-agent-98118", + "type": "filebeat", + "version": "8.18.0" + }, + "data_stream": { + "dataset": "ping_directory.group", + "namespace": "73116", + "type": "logs" + }, + "ecs": { + "version": "9.3.0" + }, + "elastic_agent": { + "id": "b8e1b118-3146-4215-8504-c18df1838432", + "snapshot": false, + "version": "8.18.0" + }, + "event": { + "agent_id_status": "verified", + "category": [ + "iam" + ], + "dataset": "ping_directory.group", + "ingested": "2026-06-23T05:26:03Z", + "kind": "event", + "original": "{\"displayName\":\"security\",\"id\":\"a7631f14-c7eb-490f-8b94-724b78241e13\",\"members\":[\"uid=alice.smith,ou=People,dc=example,dc=com\"],\"meta\":{\"location\":\"https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13\",\"resourceType\":\"Group\"},\"schemas\":[\"urn:ietf:params:scim:schemas:core:2.0:Group\"]}", + "type": [ + "group" + ] + }, + "group": { + "id": "a7631f14-c7eb-490f-8b94-724b78241e13", + "name": "security" + }, + "input": { + "type": "cel" + }, + "ping_directory": { + "group": { + "members_metadata": "ou=People,dc=example,dc=com", + "meta": { + "location": "https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13", + "resource_type": "Group" + }, + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:Group" + ] + } + }, + "related": { + "user": [ + "alice.smith" + ] + }, + "tags": [ + "preserve_original_event", + "forwarded", + "ping_directory-group" + ], + "user": { + "id": "alice.smith" + } +} diff --git a/packages/ping_directory/docs/README.md b/packages/ping_directory/docs/README.md new file mode 100644 index 00000000000..eff10fa9fca --- /dev/null +++ b/packages/ping_directory/docs/README.md @@ -0,0 +1,262 @@ +# PingDirectory Integration + +## Overview + +[PingDirectory](https://www.pingidentity.com/en/product/pingdirectory.html) is an **enterprise-grade LDAP directory server** and identity data store. It provides high-performance, scalable directory services for managing user identities, credentials, and access control across hybrid environments — combining authentication, authorization, and directory data management into a unified platform for **critical identity infrastructure**. + +This integration collects group membership data from PingDirectory via the SCIM v2 API using the Elastic Agent CEL input. It provides visibility into group accounts and membership attributes managed within your PingDirectory environment. + +### Compatibility + +The PingDirectory integration is compatible with **PingDirectory version 11.0.0.2 and above**. + +### How it works + +The integration uses the Elastic Agent CEL input to collect group data from the PingDirectory SCIM v2 API. Each collection cycle: + +1. Authenticates against the PingDirectory REST API (`/directory/v1/authenticate`) using the configured `bind_dn` and `password` to obtain a short-lived access token. +2. Paginates through all groups via the SCIM v2 Groups endpoint (`/scim/v2/Groups`) using the configured batch size. +3. Emits each group record as an individual event for ingestion and enrichment via the built-in ingest pipeline. +4. Caches the access token across collection cycles and re-authenticates automatically when the token expires. + +## What data does this integration collect? + +The PingDirectory integration collects the following types of data: + +| Data stream | Description | Endpoint | +|---|---|---| +| `group` | Group membership records retrieved from the PingDirectory SCIM v2 API, including group names, display names, members, and group metadata. | `/scim/v2/Groups` | + +### Supported use cases + +* **Group inventory and membership visibility**: Track all groups managed in PingDirectory, including their members, display names, and group types. + +* **Identity data enrichment**: Correlate PingDirectory group records with other security and operational data in Elastic for unified identity context. + +## What do I need to use this integration? + +### From PingDirectory + +* **PingDirectory deployment**: An active PingDirectory server with the SCIM v2 API enabled and accessible. +* **Service account credentials**: A bind DN (`bind_dn`) and password with sufficient permissions to authenticate via `/directory/v1/authenticate` and read groups from `/scim/v2/Groups`. +* **Network access**: Elastic Agent must be able to reach the PingDirectory HTTPS endpoint. +* **SSL certificate**: If PingDirectory uses a self-signed certificate, the certificate must be trusted by the Elastic Agent host. +* **Elastic Agent**: Version 8.18+ or 9.0+ with Fleet enrollment. + +## How do I deploy this integration? + +This integration supports both Elastic Agentless-based and Agent-based installations. + +### Agent-based deployment + +Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host. + +Elastic Agent is required to poll the PingDirectory SCIM v2 API and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines. + +### Agentless deployment + +Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. Agentless deployments provide a means to ingest data while avoiding the orchestration, management, and maintenance needs associated with standard ingest infrastructure. Using an agentless deployment makes manual agent deployment unnecessary, allowing you to focus on your data instead of the agent that collects it. + +For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html) + +### Set up steps in PingDirectory + +Before configuring the integration, ensure that a PingDirectory account is available for API authentication and group retrieval. + +1. Create or identify a service account in PingDirectory. +2. Grant the account permission to: + + * Authenticate through the `/directory/v1/authenticate` endpoint. + * Read group records through the `/scim/v2/Groups` endpoint. + +3. Record the account's bind DN and password. These values are required when configuring the integration in Kibana. +4. Verify that the SCIM v2 API is enabled and accessible from the Elastic Agent host. +5. If HTTPS is configured with a self-signed certificate, export the certificate so it can be trusted by Elastic Agent. + +Refer to the PingDirectory documentation for configuring SCIM 2.0 and authentication requirements. + +### Configure + +1. In Kibana, navigate to **Fleet → Integrations** and search for **PingDirectory**. +2. Click **Add PingDirectory**. +3. Configure the integration settings: + + * **URL**: The base URL of your PingDirectory instance, for example: + `https://pingdirectory.example.com:2443` + * **Bind DN**: The distinguished name used to authenticate, for example: + `cn=admin,dc=example,dc=com` + * **Password**: The password for the bind DN. + * **Batch Size**: Number of group records to retrieve per API page (default: `500`). + * **Interval**: How frequently to poll for new data (default: `24h`). +4. If using a self-signed SSL certificate, configure the SSL settings under **Advanced options**. +5. Select **Save and continue** to save the integration. +6. Add the integration to an existing Agent policy or create a new one. +7. Verify that group records are being ingested into Elasticsearch. + +### Validation + +#### Dashboard populated + +1. In the top search bar in Kibana, search for **Dashboards**. +2. In the search bar, type **PingDirectory**. +3. Open the **[Logs PingDirectory] Group** dashboard. +4. Verify that the visualizations are populated with group membership data, including group names, members, and membership counts. + +## Scaling + +For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation. + +## Troubleshooting + +* **No data collected**: Verify that the PingDirectory SCIM v2 API is enabled and reachable from the Elastic Agent host. Confirm that the configured URL, bind DN, and password are correct. +* **Authentication failures**: Ensure the bind DN has permission to authenticate via `/directory/v1/authenticate` and read from `/scim/v2/Groups`. +* **SSL certificate errors**: If PingDirectory uses a self-signed certificate, extract the certificate and configure it under the SSL settings of the integration, or add it to the Elastic Agent's trusted certificate store. +* **Token expiry**: The integration automatically re-authenticates when the access token expires. If repeated auth failures occur, verify the bind DN password has not changed. + +For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems). + +## Reference + +#### Vendor documentation links + +- [Configuring SCIM 2.0 on your server](https://docs.pingidentity.com/pingdirectory/9.3/pingdirectory_server_administration_guide/pd_ds_config_scim_2_server.html) +- [PingDirectory Directory REST API Getting Started](https://developer.pingidentity.com/pingdirectory/directory/getting-started.html) +- [PingDirectory REST API Authentication](https://developer.pingidentity.com/pingdirectory/directory/authentication.html) +- [SCIM 2.0 Users API](https://developer.pingidentity.com/pingdirectory/directory-proxy-scim/overview.html) +- [SCIM 2.0 Groups API](https://developer.pingidentity.com/pingdirectory/directory-proxy-scim/user-profile-endpoints/get-read-search-group-members-display-name.html) + +### Group + +The `group` data stream provides group membership records collected from PingDirectory. + +#### group fields + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Date and time when the event occurred. | date | +| data_stream.dataset | Name of the dataset associated with the event. | constant_keyword | +| data_stream.namespace | Namespace for the data stream. | constant_keyword | +| data_stream.type | Type of data stream. | constant_keyword | +| event.dataset | Dataset name for events collected from PingDirectory group records. | constant_keyword | +| event.module | Module name for PingDirectory events. | constant_keyword | +| input.type | Type of filebeat input. | keyword | +| observer.product | Product name of the observer that generated the event. | constant_keyword | +| observer.vendor | Vendor name of the observer that generated the event. | constant_keyword | +| ping_directory.group.members_metadata | Member DN metadata with uid removed (ou and dc components only). | keyword | +| ping_directory.group.meta.location | Self-link URL of the group resource. | keyword | +| ping_directory.group.meta.resource_type | Always Group. Identifies the SCIM resource type. | keyword | +| ping_directory.group.schemas | SCIM schema URN for the group resource. | keyword | + + +An example event for `group` looks as following: + +```json +{ + "@timestamp": "2026-06-23T05:26:00.102Z", + "agent": { + "ephemeral_id": "9deb4381-f13b-460f-a113-d50474f421d6", + "id": "b8e1b118-3146-4215-8504-c18df1838432", + "name": "elastic-agent-98118", + "type": "filebeat", + "version": "8.18.0" + }, + "data_stream": { + "dataset": "ping_directory.group", + "namespace": "73116", + "type": "logs" + }, + "ecs": { + "version": "9.3.0" + }, + "elastic_agent": { + "id": "b8e1b118-3146-4215-8504-c18df1838432", + "snapshot": false, + "version": "8.18.0" + }, + "event": { + "agent_id_status": "verified", + "category": [ + "iam" + ], + "dataset": "ping_directory.group", + "ingested": "2026-06-23T05:26:03Z", + "kind": "event", + "original": "{\"displayName\":\"security\",\"id\":\"a7631f14-c7eb-490f-8b94-724b78241e13\",\"members\":[\"uid=alice.smith,ou=People,dc=example,dc=com\"],\"meta\":{\"location\":\"https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13\",\"resourceType\":\"Group\"},\"schemas\":[\"urn:ietf:params:scim:schemas:core:2.0:Group\"]}", + "type": [ + "group" + ] + }, + "group": { + "id": "a7631f14-c7eb-490f-8b94-724b78241e13", + "name": "security" + }, + "input": { + "type": "cel" + }, + "ping_directory": { + "group": { + "members_metadata": "ou=People,dc=example,dc=com", + "meta": { + "location": "https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13", + "resource_type": "Group" + }, + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:Group" + ] + } + }, + "related": { + "user": [ + "alice.smith" + ] + }, + "tags": [ + "preserve_original_event", + "forwarded", + "ping_directory-group" + ], + "user": { + "id": "alice.smith" + } +} +``` + +### Inputs used + +These inputs can be used with this integration: +
+cel + +## Setup + +For more details about the CEL input settings, check the [Filebeat documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html). + +Before configuring the CEL input, make sure you have: +- Network connectivity to the target API endpoint +- Valid authentication credentials (API keys, tokens, or certificates as required) +- Appropriate permissions to read from the target data source + +### Collecting logs from CEL + +To configure the CEL input, you must specify the `request.url` value pointing to the API endpoint. The interval parameter controls how frequently requests are made and is the primary way to balance data freshness with API rate limits and costs. Authentication is often configured through the `request.headers` section using the appropriate method for the service. + +NOTE: To access the API service, make sure you have the necessary API credentials and that the Filebeat instance can reach the endpoint URL. Some services may require IP whitelisting or VPN access. + +To collect logs via API endpoint, configure the following parameters: + +- API Endpoint URL +- API credentials (tokens, keys, or username/password) +- Request interval (how often to fetch data) +
+ + +### API usage + +These PingDirectory REST API endpoints are used by this integration: + +| Endpoint | Method | Data stream | Description | +|---|---|---|---| +| `/directory/v1/authenticate` | POST | group | Authenticate with bind DN and password to obtain an access token | +| `/scim/v2/Groups` | GET | group | Retrieve paginated group membership records | \ No newline at end of file diff --git a/packages/ping_directory/img/ping_directory-group-dashboard.png b/packages/ping_directory/img/ping_directory-group-dashboard.png new file mode 100644 index 00000000000..35fb3f46fbd Binary files /dev/null and b/packages/ping_directory/img/ping_directory-group-dashboard.png differ diff --git a/packages/ping_directory/img/ping_directory-logo.svg b/packages/ping_directory/img/ping_directory-logo.svg new file mode 100644 index 00000000000..7c894bfcded --- /dev/null +++ b/packages/ping_directory/img/ping_directory-logo.svg @@ -0,0 +1,3 @@ + + + \ No newline at end of file diff --git a/packages/ping_directory/kibana/dashboard/ping_directory-b2380ddf-991e-4c72-8fb4-abb3206db74d.json b/packages/ping_directory/kibana/dashboard/ping_directory-b2380ddf-991e-4c72-8fb4-abb3206db74d.json new file mode 100644 index 00000000000..012c04240e0 --- /dev/null +++ b/packages/ping_directory/kibana/dashboard/ping_directory-b2380ddf-991e-4c72-8fb4-abb3206db74d.json @@ -0,0 +1,505 @@ +{ + "attributes": { + "controlGroupInput": { + "chainingSystem": "HIERARCHICAL", + "controlStyle": "oneLine", + "ignoreParentSettingsJSON": { + "ignoreFilters": false, + "ignoreQuery": false, + "ignoreTimerange": false, + "ignoreValidations": false + }, + "panelsJSON": { + "428f076c-d10c-42fc-98b6-d11886d59b37": { + "explicitInput": { + "dataViewId": "logs-*", + "exclude": null, + "existsSelected": null, + "fieldName": "group.name", + "hideActionBar": null, + "hideExclude": null, + "hideExists": null, + "hideSort": null, + "placeholder": null, + "runPastTimeout": null, + "searchTechnique": "prefix", + "selectedOptions": [], + "singleSelect": null, + "sort": { + "by": "_count", + "direction": "desc" + }, + "title": "Group Name" + }, + "grow": false, + "order": 0, + "type": "optionsListControl", + "width": "medium" + }, + "8fbed66e-e30d-4972-b113-5418c4cdeca4": { + "explicitInput": { + "dataViewId": "logs-*", + "exclude": null, + "existsSelected": null, + "fieldName": "user.id", + "hideActionBar": null, + "hideExclude": null, + "hideExists": null, + "hideSort": null, + "placeholder": null, + "runPastTimeout": null, + "searchTechnique": "prefix", + "selectedOptions": [], + "singleSelect": null, + "sort": { + "by": "_count", + "direction": "desc" + }, + "title": "User ID" + }, + "grow": false, + "order": 1, + "type": "optionsListControl", + "width": "medium" + } + }, + "showApplySelections": false + }, + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "**Overview**\n\nThis dashboard provides visibility into group membership and directory structure collected from PingDirectory.\n\nIt highlights group membership trends and user distribution across groups in the environment. The dashboard helps administrators and analysts monitor group provisioning activity, identify changes in group membership, and understand the composition of synced groups.\n\nVisualizations include group sync events over time, top groups by membership, users with most group memberships, and a group inventory table. Interactive filters allow scoping the view to relevant synced group records for focused analysis and troubleshooting.\n\n[Integration Page](https://127.0.0.1:5601/app/integrations/detail/ping_directory-0.1.0/overview)", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 22, + "i": "d85b0dab-4b5a-45aa-8c4a-74e49190e960", + "w": 11, + "x": 0, + "y": 0 + }, + "panelIndex": "d85b0dab-4b5a-45aa-8c4a-74e49190e960", + "type": "visualization" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 9, + "i": "a3be92fe-afd8-4bee-a500-a1de715583a4", + "w": 21, + "x": 11, + "y": 9 + }, + "panelIndex": "a3be92fe-afd8-4bee-a500-a1de715583a4", + "panelRefName": "panel_a3be92fe-afd8-4bee-a500-a1de715583a4", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 9, + "i": "537711c1-24db-452f-a5e6-fde67ae4eb87", + "w": 15, + "x": 32, + "y": 9 + }, + "panelIndex": "537711c1-24db-452f-a5e6-fde67ae4eb87", + "panelRefName": "panel_537711c1-24db-452f-a5e6-fde67ae4eb87", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "hidePanelTitles": true, + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 9, + "i": "2a4bd4cb-a3a1-4fd7-8793-a1cc0ef3834a", + "w": 6, + "x": 11, + "y": 0 + }, + "panelIndex": "2a4bd4cb-a3a1-4fd7-8793-a1cc0ef3834a", + "panelRefName": "panel_2a4bd4cb-a3a1-4fd7-8793-a1cc0ef3834a", + "title": "Total Groups", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "hidePanelTitles": true, + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 9, + "i": "e790cc53-68f2-4200-8155-2c91485a6b48", + "w": 6, + "x": 17, + "y": 0 + }, + "panelIndex": "e790cc53-68f2-4200-8155-2c91485a6b48", + "panelRefName": "panel_e790cc53-68f2-4200-8155-2c91485a6b48", + "title": "Total Memberships", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 9, + "i": "cfa8058b-75fe-49e7-ab39-986a00238661", + "w": 24, + "x": 23, + "y": 0 + }, + "panelIndex": "cfa8058b-75fe-49e7-ab39-986a00238661", + "panelRefName": "panel_cfa8058b-75fe-49e7-ab39-986a00238661", + "type": "lens" + } + ], + "timeRestore": false, + "title": "[Logs PingDirectory] Group", + "version": 3 + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2026-06-23T06:27:03.791Z", + "id": "ping_directory-b2380ddf-991e-4c72-8fb4-abb3206db74d", + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "ping_directory-d82eb1fc-ee25-4605-a2fa-8496e18d1235", + "name": "a3be92fe-afd8-4bee-a500-a1de715583a4:panel_a3be92fe-afd8-4bee-a500-a1de715583a4", + "type": "lens" + }, + { + "id": "ping_directory-ee497d59-c6b9-47ba-87c3-ba4c42872251", + "name": "537711c1-24db-452f-a5e6-fde67ae4eb87:panel_537711c1-24db-452f-a5e6-fde67ae4eb87", + "type": "lens" + }, + { + "id": "ping_directory-83bcd4a6-2174-46f0-924b-2e778b8759fd", + "name": "2a4bd4cb-a3a1-4fd7-8793-a1cc0ef3834a:panel_2a4bd4cb-a3a1-4fd7-8793-a1cc0ef3834a", + "type": "lens" + }, + { + "id": "ping_directory-ff4c6268-9001-414d-92a8-ffbbc2a79c45", + "name": "e790cc53-68f2-4200-8155-2c91485a6b48:panel_e790cc53-68f2-4200-8155-2c91485a6b48", + "type": "lens" + }, + { + "id": "ping_directory-302ed42a-03bf-483d-a056-41f99d43908f", + "name": "cfa8058b-75fe-49e7-ab39-986a00238661:panel_cfa8058b-75fe-49e7-ab39-986a00238661", + "type": "lens" + }, + { + "id": "logs-*", + "name": "a3be92fe-afd8-4bee-a500-a1de715583a4:indexpattern-datasource-layer-6f9c6419-e6e2-4959-a3a8-4809558fe9ba", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "a3be92fe-afd8-4bee-a500-a1de715583a4:0b080d27-3177-4002-9291-e415764fe790", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "537711c1-24db-452f-a5e6-fde67ae4eb87:indexpattern-datasource-layer-d8b3687e-2cb3-4a70-873a-b194749db60e", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "537711c1-24db-452f-a5e6-fde67ae4eb87:03c7840f-c7c0-4110-a971-174fcbee2c78", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "2a4bd4cb-a3a1-4fd7-8793-a1cc0ef3834a:indexpattern-datasource-layer-922cc685-7e3c-413c-8cff-f835d704b45c", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "2a4bd4cb-a3a1-4fd7-8793-a1cc0ef3834a:35036783-3609-40e4-b97b-3150815e052b", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "e790cc53-68f2-4200-8155-2c91485a6b48:indexpattern-datasource-layer-9039dbbc-0af1-4e65-bdb4-65ee87511350", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "cfa8058b-75fe-49e7-ab39-986a00238661:indexpattern-datasource-layer-6c6e3d9f-40f4-44ab-9322-e28123ce33b2", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "cfa8058b-75fe-49e7-ab39-986a00238661:f7f978ae-aa36-4c7f-b8af-3651181cdcc7", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_428f076c-d10c-42fc-98b6-d11886d59b37:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_8fbed66e-e30d-4972-b113-5418c4cdeca4:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + } + ], + "type": "dashboard", + "typeMigrationVersion": "10.2.0" +} \ No newline at end of file diff --git a/packages/ping_directory/kibana/lens/ping_directory-302ed42a-03bf-483d-a056-41f99d43908f.json b/packages/ping_directory/kibana/lens/ping_directory-302ed42a-03bf-483d-a056-41f99d43908f.json new file mode 100644 index 00000000000..9bfe54f5efb --- /dev/null +++ b/packages/ping_directory/kibana/lens/ping_directory-302ed42a-03bf-483d-a056-41f99d43908f.json @@ -0,0 +1,146 @@ +{ + "attributes": { + "description": "", + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "6c6e3d9f-40f4-44ab-9322-e28123ce33b2": { + "columnOrder": [ + "36dd2e72-963a-4543-8dc9-fa58bf7c70f2", + "59ab2135-7c63-4444-96a1-44a4b4af01ad" + ], + "columns": { + "36dd2e72-963a-4543-8dc9-fa58bf7c70f2": { + "dataType": "date", + "isBucketed": true, + "label": "@timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": true, + "interval": "m" + }, + "scale": "interval", + "sourceField": "@timestamp" + }, + "59ab2135-7c63-4444-96a1-44a4b4af01ad": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "f7f978ae-aa36-4c7f-b8af-3651181cdcc7", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "accessors": [ + "59ab2135-7c63-4444-96a1-44a4b4af01ad" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "6c6e3d9f-40f4-44ab-9322-e28123ce33b2", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "xAccessor": "36dd2e72-963a-4543-8dc9-fa58bf7c70f2" + } + ], + "legend": { + "isVisible": true, + "position": "right", + "shouldTruncate": false, + "showSingleSeries": false + }, + "preferredSeriesType": "line", + "title": "Empty XY chart", + "valueLabels": "hide" + } + }, + "title": "Groups over Time", + "visualizationType": "lnsXY" + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2026-06-23T06:25:31.000Z", + "id": "ping_directory-302ed42a-03bf-483d-a056-41f99d43908f", + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-6c6e3d9f-40f4-44ab-9322-e28123ce33b2", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "f7f978ae-aa36-4c7f-b8af-3651181cdcc7", + "type": "index-pattern" + } + ], + "type": "lens", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/ping_directory/kibana/lens/ping_directory-83bcd4a6-2174-46f0-924b-2e778b8759fd.json b/packages/ping_directory/kibana/lens/ping_directory-83bcd4a6-2174-46f0-924b-2e778b8759fd.json new file mode 100644 index 00000000000..eec0a75dc3f --- /dev/null +++ b/packages/ping_directory/kibana/lens/ping_directory-83bcd4a6-2174-46f0-924b-2e778b8759fd.json @@ -0,0 +1,96 @@ +{ + "attributes": { + "description": "", + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "922cc685-7e3c-413c-8cff-f835d704b45c": { + "columnOrder": [ + "15bf09bf-22c9-4030-b7e7-97f669e9eabb" + ], + "columns": { + "15bf09bf-22c9-4030-b7e7-97f669e9eabb": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Total Groups", + "operationType": "count", + "params": { + "emptyAsNull": false + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "35036783-3609-40e4-b97b-3150815e052b", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#6092C0", + "layerId": "922cc685-7e3c-413c-8cff-f835d704b45c", + "layerType": "data", + "metricAccessor": "15bf09bf-22c9-4030-b7e7-97f669e9eabb" + } + }, + "title": "Total Groups", + "visualizationType": "lnsMetric" + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2026-06-23T06:25:31.000Z", + "id": "ping_directory-83bcd4a6-2174-46f0-924b-2e778b8759fd", + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-922cc685-7e3c-413c-8cff-f835d704b45c", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "35036783-3609-40e4-b97b-3150815e052b", + "type": "index-pattern" + } + ], + "type": "lens", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/ping_directory/kibana/lens/ping_directory-d82eb1fc-ee25-4605-a2fa-8496e18d1235.json b/packages/ping_directory/kibana/lens/ping_directory-d82eb1fc-ee25-4605-a2fa-8496e18d1235.json new file mode 100644 index 00000000000..e0e0bcbdcd1 --- /dev/null +++ b/packages/ping_directory/kibana/lens/ping_directory-d82eb1fc-ee25-4605-a2fa-8496e18d1235.json @@ -0,0 +1,134 @@ +{ + "attributes": { + "description": "", + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "6f9c6419-e6e2-4959-a3a8-4809558fe9ba": { + "columnOrder": [ + "4d864658-c234-43d4-bd4e-f9ebc2f2ee0f", + "d239e568-d015-4c92-a9fe-3fdaf8ad9a05" + ], + "columns": { + "4d864658-c234-43d4-bd4e-f9ebc2f2ee0f": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "User ID", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "d239e568-d015-4c92-a9fe-3fdaf8ad9a05", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "user.id" + }, + "d239e568-d015-4c92-a9fe-3fdaf8ad9a05": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "0b080d27-3177-4002-9291-e415764fe790", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "4d864658-c234-43d4-bd4e-f9ebc2f2ee0f", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "d239e568-d015-4c92-a9fe-3fdaf8ad9a05", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "6f9c6419-e6e2-4959-a3a8-4809558fe9ba", + "layerType": "data" + } + }, + "title": "Top Users", + "visualizationType": "lnsDatatable" + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2026-06-23T06:36:52.486Z", + "created_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", + "id": "ping_directory-d82eb1fc-ee25-4605-a2fa-8496e18d1235", + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-6f9c6419-e6e2-4959-a3a8-4809558fe9ba", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "0b080d27-3177-4002-9291-e415764fe790", + "type": "index-pattern" + } + ], + "type": "lens", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/ping_directory/kibana/lens/ping_directory-ee497d59-c6b9-47ba-87c3-ba4c42872251.json b/packages/ping_directory/kibana/lens/ping_directory-ee497d59-c6b9-47ba-87c3-ba4c42872251.json new file mode 100644 index 00000000000..4b21be6d208 --- /dev/null +++ b/packages/ping_directory/kibana/lens/ping_directory-ee497d59-c6b9-47ba-87c3-ba4c42872251.json @@ -0,0 +1,160 @@ +{ + "attributes": { + "description": "", + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "d8b3687e-2cb3-4a70-873a-b194749db60e": { + "columnOrder": [ + "eee03565-f217-456b-b41d-dece39e896b7", + "53b4c7df-784a-45bf-ac48-31fce992b529" + ], + "columns": { + "53b4c7df-784a-45bf-ac48-31fce992b529": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "eee03565-f217-456b-b41d-dece39e896b7": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Group Name", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "53b4c7df-784a-45bf-ac48-31fce992b529", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "group.name" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "03c7840f-c7c0-4110-a971-174fcbee2c78", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "accessors": [ + "53b4c7df-784a-45bf-ac48-31fce992b529" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "d8b3687e-2cb3-4a70-873a-b194749db60e", + "layerType": "data", + "position": "top", + "seriesType": "bar_stacked", + "showGridlines": false, + "xAccessor": "eee03565-f217-456b-b41d-dece39e896b7" + } + ], + "legend": { + "isVisible": true, + "position": "right", + "shouldTruncate": false, + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "title": "Empty XY chart", + "valueLabels": "hide" + } + }, + "title": "Groups by Department", + "visualizationType": "lnsXY" + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2026-06-23T06:39:46.017Z", + "created_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", + "id": "ping_directory-ee497d59-c6b9-47ba-87c3-ba4c42872251", + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-d8b3687e-2cb3-4a70-873a-b194749db60e", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "03c7840f-c7c0-4110-a971-174fcbee2c78", + "type": "index-pattern" + } + ], + "type": "lens", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/ping_directory/kibana/lens/ping_directory-ff4c6268-9001-414d-92a8-ffbbc2a79c45.json b/packages/ping_directory/kibana/lens/ping_directory-ff4c6268-9001-414d-92a8-ffbbc2a79c45.json new file mode 100644 index 00000000000..4b5b3ed64f7 --- /dev/null +++ b/packages/ping_directory/kibana/lens/ping_directory-ff4c6268-9001-414d-92a8-ffbbc2a79c45.json @@ -0,0 +1,101 @@ +{ + "attributes": { + "description": "", + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "9039dbbc-0af1-4e65-bdb4-65ee87511350": { + "columnOrder": [ + "047f56ce-3cd8-4852-81b1-02b9b9dc75e8" + ], + "columns": { + "047f56ce-3cd8-4852-81b1-02b9b9dc75e8": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Total Memberships", + "operationType": "count", + "params": { + "emptyAsNull": false + }, + "scale": "ratio", + "sourceField": "ping_directory.group.members_metadata" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "currentIndexPatternId": "logs-*", + "layers": {} + }, + "textBased": { + "indexPatternRefs": [ + { + "id": "logs-*", + "timeField": "@timestamp", + "title": "logs-*" + } + ], + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "ping_directory.group" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "ping_directory.group" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#6092C0", + "layerId": "9039dbbc-0af1-4e65-bdb4-65ee87511350", + "layerType": "data", + "metricAccessor": "047f56ce-3cd8-4852-81b1-02b9b9dc75e8" + } + }, + "title": "Total Memberships", + "visualizationType": "lnsMetric" + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2026-06-23T06:25:31.000Z", + "id": "ping_directory-ff4c6268-9001-414d-92a8-ffbbc2a79c45", + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-9039dbbc-0af1-4e65-bdb4-65ee87511350", + "type": "index-pattern" + } + ], + "type": "lens", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/ping_directory/manifest.yml b/packages/ping_directory/manifest.yml new file mode 100644 index 00000000000..ce75579ee8d --- /dev/null +++ b/packages/ping_directory/manifest.yml @@ -0,0 +1,103 @@ +format_version: '3.3.2' +name: ping_directory +title: PingDirectory +version: '0.1.0' +description: 'Collect group logs from PingDirectory via SCIM v2 REST API.' +type: integration +categories: + - security + - iam +conditions: + kibana: + version: '^8.18.0 || ^9.1.0' + elastic: + subscription: 'basic' +screenshots: + - src: /img/ping_directory-group-dashboard.png + title: Group Directory Dashboard + size: 600x600 + type: image/png +icons: + - src: /img/ping_directory-logo.svg + title: ping_directory logo + size: 32x32 + type: image/svg+xml +policy_templates: + - name: ping_directory + title: PingDirectory + description: Collect group data from PingDirectory via SCIM v2 REST API. + deployment_modes: + default: + enabled: true + agentless: + enabled: true + organization: security + division: engineering + team: security-service-integrations + inputs: + - type: cel + title: Collect PingDirectory log via SCIM v2 API + description: Collecting group log from PingDirectory via SCIM v2 REST API. + vars: + - name: url + type: url + title: URL + description: Base URL of the PingDirectory server. Example - https://10.40.16.33:2443. + multi: false + required: true + show_user: true + - name: bind_dn + type: text + title: Bind DN + description: Admin DN for authenticating to PingDirectory. Example - cn=Directory Manager. + multi: false + required: true + show_user: true + - name: password + type: password + title: Password + description: Password for the admin Bind DN. + multi: false + required: true + show_user: true + secret: true + - name: ssl + type: yaml + title: SSL Configuration + description: SSL configuration options. Set verification_mode to none for self-signed certificates. + multi: false + required: false + show_user: false + default: | + #certificate_authorities: + # - | + # -----BEGIN CERTIFICATE----- + # MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF + # ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2 + # MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB + # BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n + # fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl + # 94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t + # /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP + # PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41 + # CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O + # BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux + # 8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D + # 874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw + # 3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA + # H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu + # 8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0 + # yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk + # sxSmbIUfc2SGJGCJD4I= + # -----END CERTIFICATE----- + # verification_mode: none + - name: proxy_url + type: text + title: Proxy URL + description: URL to proxy connections in the form of http[s]://:@:. + multi: false + required: false + show_user: false +owner: + github: elastic/security-service-integrations + type: elastic diff --git a/packages/ping_directory/validation.yml b/packages/ping_directory/validation.yml new file mode 100644 index 00000000000..1189aa63c89 --- /dev/null +++ b/packages/ping_directory/validation.yml @@ -0,0 +1,3 @@ +errors: + exclude_checks: + - SVR00004 # References in dashboards.