[sailpoint_identity_sc] Add machine_identities data stream

packages/sailpoint_identity_sc/_dev/build/build.yml

@@ -1,3 +1,3 @@
 dependencies:
   ecs:
-    reference: git@v8.11.0
+    reference: git@v9.4.0

packages/sailpoint_identity_sc/_dev/build/docs/README.md

@@ -9,24 +9,30 @@
 
 ## Data Streams
 
-- **`events`**: Provides audit data that includes actions such as `USER_MANAGEMENT`, `PASSWORD_ACTIVITY`, `PROVISIONING`, `ACCESS_ITEM`, `SOURCE_MANAGEMENT`, `CERTIFICATION`, `AUTH`, `SYSTEM_CONFIG`, `ACCESS_REQUEST`, `SSO`, `WORKFLOW`, `SEGMENT` and more.
-- [Audit Events](https://community.sailpoint.com/t5/IdentityNow-Wiki/Audit-Events-in-Cloud-Audit/ta-p/218727) are records that a user took action in an [IdentityNow](https://www.sailpoint.com/products/identitynow) tenant, or other service like [IdentityAI](https://www.sailpoint.com/products/ai-driven-identity-security). Audit Events are structurally and conceptually very similar to [IdentityIQ's](https://www.sailpoint.com/products/identity-security-software/identity-iq)Audit Events, but have evolved in several ways.
-- This data stream leverages the Sailpoint identity security cloud API's [/v2025/search](https://developer.sailpoint.com/docs/api/v2025/search-post) endpoint to retrieve event logs.
+- **`events`**: Provides audit data that includes actions such as `USER_MANAGEMENT`, `PASSWORD_ACTIVITY`, `PROVISIONING`, `ACCESS_ITEM`, `SOURCE_MANAGEMENT`, `CERTIFICATION`, `AUTH`, `SYSTEM_CONFIG`, `ACCESS_REQUEST`, `SSO`, `WORKFLOW`, `SEGMENT` and more. [Audit Events](https://community.sailpoint.com/t5/IdentityNow-Wiki/Audit-Events-in-Cloud-Audit/ta-p/218727) are records that a user took action in an [IdentityNow](https://www.sailpoint.com/products/identitynow) tenant, or other service like [IdentityAI](https://www.sailpoint.com/products/ai-driven-identity-security). This data stream leverages the [/v2026/search](https://developer.sailpoint.com/docs/api/v2026/search-post) endpoint.
+
+- **`machine_identities`**: Collects machine/non-human identity records (service accounts, application accounts, bots, and AI agents) governed by SailPoint ISC. Each document represents a machine identity with its owner relationships and held entitlements. Uses the experimental [Machine Identities API](https://developer.sailpoint.com/docs/api/v2026/list-machine-identities/) with a full offset-based scan each collection cycle. **Note:** This data stream depends on an experimental SailPoint API (`X-SailPoint-Experimental: true`). The API may change without notice in future SailPoint releases. HTTP 404 and 501 responses are treated as "feature not enabled" rather than errors.
 
 ## Requirements
 
-### Generate a Personal Access Token (PAT)
+### Create an OAuth2 API Client
+
+This integration uses OAuth2 `client_credentials` to authenticate against the SailPoint ISC API.
 
-Log in to the application with an administrator account and generate a **Personal Access Token (PAT)**. Personal access tokens are associated with a user in **Sailpoint identity security cloud** and inherit the user's permission level (e.g., Admin, Helpdesk, etc.) to determine access.
+1. Log in to the SailPoint ISC admin console.
+2. Navigate to **Admin → Security Settings → API Management**.
+3. Click **Create API Client**, select **Client Credentials** as grant type, and grant the following scopes:
+   - `sp:search:read` — required for the `events` and `identities` data streams
+   - `idn:mis-identity:read` and `idn:mis-identity:manage` — required for the `machine_identities` data stream (experimental API)
+4. Note the generated **Client ID** and **Client Secret** for use in the integration configuration.
 
-To create a **Personal Access Token (PAT)** using an **admin account**, follow the instructions provided in the official documentation:  
-[Generate a Personal Access Token](https://developer.sailpoint.com/docs/api/v2024/authentication#generate-a-personal-access-token).
+For further details see the official [Authentication documentation](https://developer.sailpoint.com/docs/api/authentication).
 
 ## Logs
 
 ### Events
 
-Event documents can be found by setting the following filter: 
+Event documents can be found by setting the following filter:
 `event.dataset : "sailpoint_identity_sc.events"`
 
 {{event "events"}}
@@ -39,3 +45,19 @@
 
 {{fields "events"}}
 
+### Machine Identities
+
+Machine identity documents can be found by setting the following filter:
+`event.dataset : "sailpoint_identity_sc.machine_identities"`
+
+Machine identity documents carry ECS entity fields (`service.entity.*`) that enable entity analytics for non-human identities such as service accounts, application accounts, bots, and AI agents.
+
+{{event "machine_identities"}}
+
+**ECS Field Reference**
+
+Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
+
+The following non-ECS fields are used in machine_identities documents:
+
+{{fields "machine_identities"}}

packages/sailpoint_identity_sc/_dev/deploy/docker/docker-compose.yml

@@ -12,3 +12,16 @@ services:
       - http-server
       - --addr=:8080
       - --config=/files/config.yml
+  sailpoint-machine-identities:
+    image: docker.elastic.co/observability/stream:v0.20.0
+    hostname: sailpoint-machine-identities
+    ports:
+      - 8091
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: '8091'
+    command:
+      - http-server
+      - --addr=:8091
+      - --config=/files/config-machine-identities.yml

packages/sailpoint_identity_sc/_dev/deploy/docker/files/config-machine-identities.yml

@@ -0,0 +1,179 @@
+rules:
+  # OAuth Token Endpoint
+  # Used by the system test agent for OAuth2 client_credentials flow.
+  # SailPoint ISC rejects an explicit scope parameter in the token request body,
+  # so no scope is sent in the request.
+  - path: /oauth/token
+    methods:
+      - POST
+    query_params:
+      grant_type: client_credentials
+      client_id: test-client-id
+      client_secret: test-client-secret
+    request_headers:
+      Content-Type:
+        - application/x-www-form-urlencoded
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |
+          {
+            "access_token": "test-token-machine-identities",
+            "expires_in": 3600,
+            "token_type": "Bearer"
+          }
+  # Round 2, Page 1 (offset=0) — cursor resume after Round 1 completes.
+  # Same as Round 1 Page 1. Returns 2 records, pagination will continue.
+  # Rule ordering: Round 2 termination (offset=2) is placed BEFORE this rule
+  # so the more specific offset=2 rule matches before the fallback.
+  # This rule uses a separate response block to indicate Round 2; response
+  # data is identical to Round 1 Page 1 (full scan restarted).
+  - path: /v2026/machine-identities
+    methods:
+      - GET
+    query_params:
+      limit: "2"
+      offset: "0"
+    request_headers:
+      Authorization:
+        - Bearer test-token-machine-identities
+      X-SailPoint-Experimental:
+        - "true"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |
+          [
+            {
+              "id": "mi-id-aabbccdd-1111-2222-3333-000000000001",
+              "name": "svc-example-service",
+              "created": "2024-01-15T10:30:00Z",
+              "modified": "2025-01-10T08:45:00Z",
+              "nativeIdentity": "svc-example:prod:00001",
+              "description": "Example service account for integration tests.",
+              "attributes": {
+                "env": "production",
+                "team": "Platform"
+              },
+              "subtype": "Application",
+              "owners": {
+                "primaryIdentity": {
+                  "type": "IDENTITY",
+                  "id": "00000000000000000000000000000001",
+                  "name": "alice.example"
+                },
+                "secondaryIdentities": []
+              },
+              "sourceId": "src-id-aabbccdd-1111-2222-3333-000000000001",
+              "uuid": "uuid-aabbccdd-1111-2222-3333-000000000001",
+              "manuallyEdited": false,
+              "manuallyCreated": false,
+              "source": {
+                "type": "SOURCE",
+                "id": "src-id-aabbccdd-1111-2222-3333-000000000001",
+                "name": "Example Directory"
+              },
+              "datasetId": "ds-id-aabbccdd-1111-2222-3333-000000000001",
+              "userEntitlements": []
+            },
+            {
+              "id": "mi-id-aabbccdd-1111-2222-3333-000000000002",
+              "name": "svc-reporting-job",
+              "created": "2024-02-20T14:00:00Z",
+              "modified": "2025-01-11T09:00:00Z",
+              "nativeIdentity": "svc-reporting:prod:00002",
+              "description": "Service account for scheduled reporting tasks.",
+              "attributes": {
+                "env": "production",
+                "team": "Analytics"
+              },
+              "subtype": "Application",
+              "owners": {
+                "primaryIdentity": {
+                  "type": "IDENTITY",
+                  "id": "00000000000000000000000000000002",
+                  "name": "bob.example"
+                },
+                "secondaryIdentities": []
+              },
+              "sourceId": "src-id-aabbccdd-1111-2222-3333-000000000002",
+              "uuid": "uuid-aabbccdd-1111-2222-3333-000000000002",
+              "manuallyEdited": false,
+              "manuallyCreated": false,
+              "source": {
+                "type": "SOURCE",
+                "id": "src-id-aabbccdd-1111-2222-3333-000000000002",
+                "name": "Example Cloud Apps"
+              },
+              "datasetId": "ds-id-aabbccdd-1111-2222-3333-000000000002",
+              "userEntitlements": [
+                {
+                  "sourceId": "src-id-aabbccdd-1111-2222-3333-000000000003",
+                  "entitlementId": "ent-id-aabbccdd-1111-2222-3333-000000000001",
+                  "displayName": "Reports DB Read Access",
+                  "source": {
+                    "type": "SOURCE",
+                    "id": "src-id-aabbccdd-1111-2222-3333-000000000003",
+                    "name": "Example SQL DB"
+                  }
+                }
+              ]
+            }
+          ]
+  # Round 1, Terminal page (offset=2): returns 1 record (< batch_size=2), triggers termination.
+  # Also serves as Round 2 termination for offset=2.
+  - path: /v2026/machine-identities
+    methods:
+      - GET
+    query_params:
+      limit: "2"
+      offset: "2"
+    request_headers:
+      Authorization:
+        - Bearer test-token-machine-identities
+      X-SailPoint-Experimental:
+        - "true"
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |
+          [
+            {
+              "id": "mi-id-aabbccdd-1111-2222-3333-000000000003",
+              "name": "svc-audit-collector",
+              "created": "2024-03-10T08:00:00Z",
+              "modified": "2025-01-12T07:30:00Z",
+              "nativeIdentity": "svc-audit:prod:00003",
+              "description": "Service account for audit log collection.",
+              "attributes": {
+                "env": "production",
+                "team": "Security"
+              },
+              "subtype": "Application",
+              "owners": {
+                "primaryIdentity": {
+                  "type": "IDENTITY",
+                  "id": "00000000000000000000000000000003",
+                  "name": "carol.example"
+                },
+                "secondaryIdentities": []
+              },
+              "sourceId": "src-id-aabbccdd-1111-2222-3333-000000000001",
+              "uuid": "uuid-aabbccdd-1111-2222-3333-000000000003",
+              "manuallyEdited": false,
+              "manuallyCreated": false,
+              "source": {
+                "type": "SOURCE",
+                "id": "src-id-aabbccdd-1111-2222-3333-000000000001",
+                "name": "Example Directory"
+              },
+              "datasetId": "ds-id-aabbccdd-1111-2222-3333-000000000001",
+              "userEntitlements": []
+            }
+          ]

packages/sailpoint_identity_sc/changelog.yml

@@ -1,3 +1,8 @@
+- version: "2.0.0-next"
+  changes:
+    - description: Add `machine_identities` data stream for machine identity records via the Machine Identities API.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/20198
 - version: "1.3.0"
   changes:
     - description: Enable Agentless deployment.

packages/sailpoint_identity_sc/data_stream/machine_identities/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,3 @@
+fields:
+  tags:
+    - preserve_original_event

packages/sailpoint_identity_sc/data_stream/machine_identities/_dev/test/pipeline/test-machine-identity-full-sample.log

@@ -0,0 +1 @@
+{"id":"mi-id-aabbccdd-1111-2222-3333-000000000002","name":"svc-reporting-job","created":"2024-02-20T14:00:00Z","modified":"2025-01-11T09:00:00Z","nativeIdentity":"svc-reporting:prod:00002","description":"Service account for scheduled reporting tasks.","attributes":{"env":"production","team":"Analytics"},"subtype":"Application","owners":{"primaryIdentity":{"type":"IDENTITY","id":"00000000000000000000000000000002","name":"bob.example"},"secondaryIdentities":[{"type":"IDENTITY","id":"00000000000000000000000000000004","name":"alice.example"}]},"sourceId":"src-id-aabbccdd-1111-2222-3333-000000000002","uuid":"uuid-aabbccdd-1111-2222-3333-000000000002","manuallyEdited":false,"manuallyCreated":false,"source":{"type":"SOURCE","id":"src-id-aabbccdd-1111-2222-3333-000000000002","name":"Example Cloud Apps"},"datasetId":"ds-id-aabbccdd-1111-2222-3333-000000000002","userEntitlements":[{"sourceId":"src-id-aabbccdd-1111-2222-3333-000000000003","entitlementId":"ent-id-aabbccdd-1111-2222-3333-000000000001","displayName":"Reports DB Read Access","source":{"type":"SOURCE","id":"src-id-aabbccdd-1111-2222-3333-000000000003","name":"Example SQL DB"}}]}

packages/sailpoint_identity_sc/data_stream/machine_identities/_dev/test/pipeline/test-machine-identity-full-sample.log-expected.json

@@ -0,0 +1,88 @@
+{
+    "expected": [
+        {
+            "@timestamp": "2025-01-11T09:00:00.000Z",
+            "ecs": {
+                "version": "9.4.0"
+            },
+            "event": {
+                "category": [
+                    "iam"
+                ],
+                "created": "2024-02-20T14:00:00.000Z",
+                "kind": "asset",
+                "original": "{\"id\":\"mi-id-aabbccdd-1111-2222-3333-000000000002\",\"name\":\"svc-reporting-job\",\"created\":\"2024-02-20T14:00:00Z\",\"modified\":\"2025-01-11T09:00:00Z\",\"nativeIdentity\":\"svc-reporting:prod:00002\",\"description\":\"Service account for scheduled reporting tasks.\",\"attributes\":{\"env\":\"production\",\"team\":\"Analytics\"},\"subtype\":\"Application\",\"owners\":{\"primaryIdentity\":{\"type\":\"IDENTITY\",\"id\":\"00000000000000000000000000000002\",\"name\":\"bob.example\"},\"secondaryIdentities\":[{\"type\":\"IDENTITY\",\"id\":\"00000000000000000000000000000004\",\"name\":\"alice.example\"}]},\"sourceId\":\"src-id-aabbccdd-1111-2222-3333-000000000002\",\"uuid\":\"uuid-aabbccdd-1111-2222-3333-000000000002\",\"manuallyEdited\":false,\"manuallyCreated\":false,\"source\":{\"type\":\"SOURCE\",\"id\":\"src-id-aabbccdd-1111-2222-3333-000000000002\",\"name\":\"Example Cloud Apps\"},\"datasetId\":\"ds-id-aabbccdd-1111-2222-3333-000000000002\",\"userEntitlements\":[{\"sourceId\":\"src-id-aabbccdd-1111-2222-3333-000000000003\",\"entitlementId\":\"ent-id-aabbccdd-1111-2222-3333-000000000001\",\"displayName\":\"Reports DB Read Access\",\"source\":{\"type\":\"SOURCE\",\"id\":\"src-id-aabbccdd-1111-2222-3333-000000000003\",\"name\":\"Example SQL DB\"}}]}",
+                "type": [
+                    "info"
+                ]
+            },
+            "related": {
+                "user": [
+                    "bob.example",
+                    "alice.example"
+                ]
+            },
+            "sailpoint_identity_sc": {
+                "machine_identity": {
+                    "attributes": {
+                        "env": "production",
+                        "team": "Analytics"
+                    },
+                    "created": "2024-02-20T14:00:00.000Z",
+                    "dataset_id": "ds-id-aabbccdd-1111-2222-3333-000000000002",
+                    "description": "Service account for scheduled reporting tasks.",
+                    "manually_created": false,
+                    "manually_edited": false,
+                    "modified": "2025-01-11T09:00:00.000Z",
+                    "native_identity": "svc-reporting:prod:00002",
+                    "owners": {
+                        "primary": {
+                            "id": "00000000000000000000000000000002",
+                            "name": "bob.example"
+                        },
+                        "secondary": [
+                            {
+                                "id": "00000000000000000000000000000004",
+                                "name": "alice.example",
+                                "type": "IDENTITY"
+                            }
+                        ]
+                    },
+                    "source": {
+                        "id": "src-id-aabbccdd-1111-2222-3333-000000000002",
+                        "name": "Example Cloud Apps"
+                    },
+                    "source_id": "src-id-aabbccdd-1111-2222-3333-000000000002",
+                    "user_entitlements": [
+                        {
+                            "displayName": "Reports DB Read Access",
+                            "entitlementId": "ent-id-aabbccdd-1111-2222-3333-000000000001",
+                            "source": {
+                                "id": "src-id-aabbccdd-1111-2222-3333-000000000003",
+                                "name": "Example SQL DB",
+                                "type": "SOURCE"
+                            },
+                            "sourceId": "src-id-aabbccdd-1111-2222-3333-000000000003"
+                        }
+                    ],
+                    "uuid": "uuid-aabbccdd-1111-2222-3333-000000000002"
+                }
+            },
+            "service": {
+                "entity": {
+                    "id": "mi-id-aabbccdd-1111-2222-3333-000000000002",
+                    "name": "svc-reporting-job",
+                    "type": [
+                        "service"
+                    ]
+                },
+                "id": "mi-id-aabbccdd-1111-2222-3333-000000000002",
+                "name": "svc-reporting-job",
+                "type": "Application"
+            },
+            "tags": [
+                "preserve_original_event"
+            ]
+        }
+    ]
+}

packages/sailpoint_identity_sc/data_stream/machine_identities/_dev/test/pipeline/test-machine-identity-minimal-sample.log

@@ -0,0 +1 @@
+{"id":"mi-id-aabbccdd-1111-2222-3333-000000000001","name":"svc-example-service","created":"2024-01-15T10:30:00Z","modified":"2025-01-10T08:45:00Z","nativeIdentity":"svc-example:prod:00001","description":"Example service account for integration tests.","attributes":{"env":"production","team":"Platform"},"subtype":"Application","owners":{"primaryIdentity":{"type":"IDENTITY","id":"00000000000000000000000000000001","name":"alice.example"},"secondaryIdentities":[]},"sourceId":"src-id-aabbccdd-1111-2222-3333-000000000001","uuid":"uuid-aabbccdd-1111-2222-3333-000000000001","manuallyEdited":false,"manuallyCreated":false,"source":{"type":"SOURCE","id":"src-id-aabbccdd-1111-2222-3333-000000000001","name":"Example Directory"},"datasetId":"ds-id-aabbccdd-1111-2222-3333-000000000001","userEntitlements":[]}