[PingDirectory][Group] Add Group data stream

.github/CODEOWNERS

@@ -431,6 +431,7 @@
 /packages/panw_metrics @elastic/obs-infraobs-integrations
 /packages/pfsense @elastic/integration-experience
 /packages/php_fpm @elastic/obs-infraobs-integrations
+/packages/ping_directory @elastic/security-service-integrations
 /packages/ping_federate @elastic/security-service-integrations @elastic/sit-crest-contractors
 /packages/ping_one @elastic/security-service-integrations @elastic/sit-crest-contractors
 /packages/platform_observability @elastic/stack-monitoring
@@ -613,4 +614,4 @@
 /packages/zscaler_zia @elastic/security-service-integrations @elastic/sit-crest-contractors
 /packages/zscaler_zpa @elastic/security-service-integrations @elastic/sit-crest-contractors
 /packages/otel_rum_dashboards @elastic/apm-agent-rum
-/packages/otel_android_dashboards @elastic/apm-agent-android @elastic/apm-agent-approvers
+/packages/otel_android_dashboards @elastic/apm-agent-android @elastic/apm-agent-approvers

packages/ping_directory/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: git@v9.3.0

packages/ping_directory/_dev/build/docs/README.md

@@ -0,0 +1,149 @@
+# PingDirectory Integration
+
+## Overview
+
+[PingDirectory](https://www.pingidentity.com/en/product/pingdirectory.html) is an **enterprise-grade LDAP directory server** and identity data store. It provides high-performance, scalable directory services for managing user identities, credentials, and access control across hybrid environments — combining authentication, authorization, and directory data management into a unified platform for **critical identity infrastructure**.
+
+This integration collects group membership data from PingDirectory via the SCIM v2 API using the Elastic Agent CEL input. It provides visibility into group accounts and membership attributes managed within your PingDirectory environment.
+
+### Compatibility
+
+The PingDirectory integration is compatible with **PingDirectory version 11.0.0.2 and above**.
+
+### How it works
+
+The integration uses the Elastic Agent CEL input to collect group data from the PingDirectory SCIM v2 API. Each collection cycle:
+
+1. Authenticates against the PingDirectory REST API (`/directory/v1/authenticate`) using the configured `bind_dn` and `password` to obtain a short-lived access token.
+2. Paginates through all groups via the SCIM v2 Groups endpoint (`/scim/v2/Groups`) using the configured batch size.
+3. Emits each group record as an individual event for ingestion and enrichment via the built-in ingest pipeline.
+4. Caches the access token across collection cycles and re-authenticates automatically when the token expires.
+
+## What data does this integration collect?
+
+The PingDirectory integration collects the following types of data:
+
+| Data stream | Description | Endpoint |
+|---|---|---|
+| `group` | Group membership records retrieved from the PingDirectory SCIM v2 API, including group names, display names, members, and group metadata. | `/scim/v2/Groups` |
+
+### Supported use cases
+
+* **Group inventory and membership visibility**: Track all groups managed in PingDirectory, including their members, display names, and group types.
+
+* **Identity data enrichment**: Correlate PingDirectory group records with other security and operational data in Elastic for unified identity context.
+
+## What do I need to use this integration?
+
+### From PingDirectory
+
+* **PingDirectory deployment**: An active PingDirectory server with the SCIM v2 API enabled and accessible.
+* **Service account credentials**: A bind DN (`bind_dn`) and password with sufficient permissions to authenticate via `/directory/v1/authenticate` and read groups from `/scim/v2/Groups`.
+* **Network access**: Elastic Agent must be able to reach the PingDirectory HTTPS endpoint.
+* **SSL certificate**: If PingDirectory uses a self-signed certificate, the certificate must be trusted by the Elastic Agent host.
+* **Elastic Agent**: Version 8.18+ or 9.0+ with Fleet enrollment.
+
+## How do I deploy this integration?
+
+This integration supports both Elastic Agentless-based and Agent-based installations.
+
+### Agent-based deployment
+
+Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host.
+
+Elastic Agent is required to poll the PingDirectory SCIM v2 API and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.
+
+### Agentless deployment
+
+Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. Agentless deployments provide a means to ingest data while avoiding the orchestration, management, and maintenance needs associated with standard ingest infrastructure. Using an agentless deployment makes manual agent deployment unnecessary, allowing you to focus on your data instead of the agent that collects it.
+
+For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html)
+
+### Set up steps in PingDirectory
+
+Before configuring the integration, ensure that a PingDirectory account is available for API authentication and group retrieval.
+
+1. Create or identify a service account in PingDirectory.
+2. Grant the account permission to:
+
+   * Authenticate through the `/directory/v1/authenticate` endpoint.
+   * Read group records through the `/scim/v2/Groups` endpoint.
+
+3. Record the account's bind DN and password. These values are required when configuring the integration in Kibana.
+4. Verify that the SCIM v2 API is enabled and accessible from the Elastic Agent host.
+5. If HTTPS is configured with a self-signed certificate, export the certificate so it can be trusted by Elastic Agent.
+
+Refer to the PingDirectory documentation for configuring SCIM 2.0 and authentication requirements.
+
+### Configure
+
+1. In Kibana, navigate to **Fleet → Integrations** and search for **PingDirectory**.
+2. Click **Add PingDirectory**.
+3. Configure the integration settings:
+
+   * **URL**: The base URL of your PingDirectory instance, for example:
+     `https://pingdirectory.example.com:2443`
+   * **Bind DN**: The distinguished name used to authenticate, for example:
+     `cn=admin,dc=example,dc=com`
+   * **Password**: The password for the bind DN.
+   * **Batch Size**: Number of group records to retrieve per API page (default: `500`).
+   * **Interval**: How frequently to poll for new data (default: `24h`).
+4. If using a self-signed SSL certificate, configure the SSL settings under **Advanced options**.
+5. Select **Save and continue** to save the integration.
+6. Add the integration to an existing Agent policy or create a new one.
+7. Verify that group records are being ingested into Elasticsearch.
+
+### Validation
+
+#### Dashboard populated
+
+1. In the top search bar in Kibana, search for **Dashboards**.
+2. In the search bar, type **PingDirectory**.
+3. Open the **[Logs PingDirectory] Group** dashboard.
+4. Verify that the visualizations are populated with group membership data, including group names, members, and membership counts.
+
+## Scaling
+
+For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation.
+
+## Troubleshooting
+
+* **No data collected**: Verify that the PingDirectory SCIM v2 API is enabled and reachable from the Elastic Agent host. Confirm that the configured URL, bind DN, and password are correct.
+* **Authentication failures**: Ensure the bind DN has permission to authenticate via `/directory/v1/authenticate` and read from `/scim/v2/Groups`.
+* **SSL certificate errors**: If PingDirectory uses a self-signed certificate, extract the certificate and configure it under the SSL settings of the integration, or add it to the Elastic Agent's trusted certificate store.
+* **Token expiry**: The integration automatically re-authenticates when the access token expires. If repeated auth failures occur, verify the bind DN password has not changed.
+
+For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems).
+
+## Reference
+
+#### Vendor documentation links
+
+- [Configuring SCIM 2.0 on your server](https://docs.pingidentity.com/pingdirectory/9.3/pingdirectory_server_administration_guide/pd_ds_config_scim_2_server.html)
+- [PingDirectory Directory REST API Getting Started](https://developer.pingidentity.com/pingdirectory/directory/getting-started.html)
+- [PingDirectory REST API Authentication](https://developer.pingidentity.com/pingdirectory/directory/authentication.html)
+- [SCIM 2.0 Users API](https://developer.pingidentity.com/pingdirectory/directory-proxy-scim/overview.html)
+- [SCIM 2.0 Groups API](https://developer.pingidentity.com/pingdirectory/directory-proxy-scim/user-profile-endpoints/get-read-search-group-members-display-name.html)
+
+### Group
+
+The `group` data stream provides group membership records collected from PingDirectory.
+
+#### group fields
+
+{{fields "group"}}
+
+{{event "group"}}
+
+### Inputs used
+
+{{ inputDocs }}
+
+### API usage
+
+These PingDirectory REST API endpoints are used by this integration:
+
+| Endpoint | Method | Data stream | Description |
+|---|---|---|---|
+| `/directory/v1/authenticate` | POST | group | Authenticate with bind DN and password to obtain an access token |
+| `/scim/v2/Groups` | GET | group | Retrieve paginated group membership records |

packages/ping_directory/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,15 @@
+version: '3.8'
+services:
+  ping_directory:
+    image: docker.elastic.co/observability/stream:v0.22.0
+    hostname: ping_directory
+    ports:
+      - 8090
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: '8090'
+    command:
+      - http-server
+      - --addr=:8090
+      - --config=/files/config.yml

packages/ping_directory/_dev/deploy/docker/files/config.yml

@@ -0,0 +1,162 @@
+rules:
+  - path: /directory/v1/authenticate
+    methods: ['POST']
+    request_headers:
+      Content-Type:
+        - 'application/json'
+    request_body: '{"credentials":{"authenticationType":"password","dn":"xxxx","staticPassword":"xxxx"}}'
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {{ minify_json `
+          {
+            "accessToken": "xxxx"
+          }
+          `}}
+  - path: /scim/v2/Groups
+    methods: ['GET']
+    query_params:
+      startIndex: 5
+      count: 2
+    request_headers:
+      Authorization:
+        - "Bearer xxxx"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {{ minify_json `
+          {
+            "schemas": ["urn:ietf:params:scim:api:messages:2.0:ListResponse"],
+            "totalResults": 5,
+            "startIndex": 5,
+            "Resources": [
+              {
+                "displayName": "platform-admins",
+                "members": [
+                  "uid=john.doe,ou=People,dc=example,dc=com",
+                  "uid=alice.smith,ou=People,dc=example,dc=com"
+                ],
+                "id": "d1964i47-f0he-723i-1e27-057e01574h46",
+                "meta": {
+                  "resourceType": "Group",
+                  "location": "https://10.50.15.29:2443/scim/v2/Groups/d1964i47-f0he-723i-1e27-057e01574h46"
+                },
+                "schemas": [
+                  "urn:ietf:params:scim:schemas:core:2.0:Group"
+                ]
+              }
+            ]
+          }
+          `}}
+  - path: /scim/v2/Groups
+    methods: ['GET']
+    query_params:
+      startIndex: 3
+      count: 2
+    request_headers:
+      Authorization:
+        - "Bearer xxxx"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {{ minify_json `
+          {
+            "schemas": ["urn:ietf:params:scim:api:messages:2.0:ListResponse"],
+            "totalResults": 5,
+            "startIndex": 3,
+            "Resources": [
+              {
+                "displayName": "engineering",
+                "members": [
+                  "uid=bob.jones,ou=Engineering,dc=example,dc=com"
+                ],
+                "id": "b8742g25-d8fc-501g-9c05-835c89352f24",
+                "meta": {
+                  "resourceType": "Group",
+                  "location": "https://10.50.15.29:2443/scim/v2/Groups/b8742g25-d8fc-501g-9c05-835c89352f24"
+                },
+                "schemas": [
+                  "urn:ietf:params:scim:schemas:core:2.0:Group"
+                ]
+              },
+              {
+                "displayName": "finance",
+                "members": [
+                  "uid=carol.white,ou=Finance,dc=corp,dc=internal"
+                ],
+                "id": "c9853h36-e9gd-612h-0d16-946d90463g35",
+                "meta": {
+                  "resourceType": "Group",
+                  "location": "https://10.50.15.29:2443/scim/v2/Groups/c9853h36-e9gd-612h-0d16-946d90463g35"
+                },
+                "schemas": [
+                  "urn:ietf:params:scim:schemas:core:2.0:Group"
+                ]
+              }
+            ]
+          }
+          `}}
+  - path: /scim/v2/Groups
+    methods: ['GET']
+    query_params:
+      startIndex: 1
+      count: 2
+    request_headers:
+      Authorization:
+        - "Bearer xxxx"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {{ minify_json `
+          {
+            "schemas": ["urn:ietf:params:scim:api:messages:2.0:ListResponse"],
+            "totalResults": 5,
+            "startIndex": 1,
+            "Resources": [
+              {
+                "displayName": "security",
+                "members": [
+                  "uid=alice.smith,ou=People,dc=example,dc=com"
+                ],
+                "id": "a7631f14-c7eb-490f-8b94-724b78241e13",
+                "meta": {
+                  "resourceType": "Group",
+                  "location": "https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13"
+                },
+                "schemas": [
+                  "urn:ietf:params:scim:schemas:core:2.0:Group"
+                ]
+              },
+              {
+                "displayName": "test-group",
+                "members": {
+                  "value": [
+                    "uid=user.0,ou=People,dc=example,dc=com",
+                    "uid=user.1,ou=People,dc=example,dc=com",
+                    "uid=user.2,ou=People,dc=example,dc=com"
+                  ]
+                },
+                "id": "1a8e021b-c7b0-4bd9-8e2d-af5691c035ce",
+                "meta": {
+                  "resourceType": "Groups",
+                  "location": "https://example.com:443/scim/v2/Groups/1a8e021b-c7b0-4bd9-8e2d-af5691c035ce"
+                },
+                "schemas": [
+                  "urn:pingidentity:schemas:Group:1.0"
+                ]
+              }
+            ]
+          }
+          `}}

packages/ping_directory/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: '0.1.0'
+  changes:
+    - description: Add support of group datastream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/19883

packages/ping_directory/data_stream/group/_dev/test/pipeline/test-group.log

@@ -0,0 +1,4 @@
+{"displayName":"security","members":["uid=alice.smith,ou=People,dc=example,dc=com"],"id":"a7631f14-c7eb-490f-8b94-724b78241e13","meta":{"resourceType":"Group","location":"https://10.50.15.29:2443/scim/v2/Groups/a7631f14-c7eb-490f-8b94-724b78241e13"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:Group"]}
+{"displayName":"test-group","members":{"value":["uid=user.0,ou=People,dc=example,dc=com","uid=user.1,ou=People,dc=example,dc=com","uid=user.2,ou=People,dc=example,dc=com","uid=user.3,ou=People,dc=example,dc=com","uid=user.4,ou=People,dc=example,dc=com","uid=user.5,ou=People,dc=example,dc=com","uid=user.6,ou=People,dc=example,dc=com","uid=user.7,ou=People,dc=example,dc=com","uid=user.8,ou=People,dc=example,dc=com","uid=user.9,ou=People,dc=example,dc=com"]},"id":"1a8e021b-c7b0-4bd9-8e2d-af5691c035ce","meta":{"resourceType":"Groups","location":"https://example.com:443/scim/v2/Groups/1a8e021b-c7b0-4bd9-8e2d-af5691c035ce"},"schemas":["urn:pingidentity:schemas:Group:1.0"]}
+{"displayName":"engineering","members":["uid=bob.jones,ou=Engineering,dc=example,dc=com"],"id":"b8742g25-d8fc-501g-9c05-835c89352f24","meta":{"resourceType":"Group","location":"https://10.50.15.29:2443/scim/v2/Groups/b8742g25-d8fc-501g-9c05-835c89352f24"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:Group"]}
+{"displayName":"finance","members":["uid=carol.white,ou=Finance,dc=corp,dc=internal"],"id":"c9853h36-e9gd-612h-0d16-946d90463g35","meta":{"resourceType":"Group","location":"https://10.50.15.29:2443/scim/v2/Groups/c9853h36-e9gd-612h-0d16-946d90463g35"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:Group"]}