Add SCIM API incomplete events and large group timeout troubleshooting (#61691)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: mc <42146119+mchammer01@users.noreply.github.com>

Author: 3 people

content/admin/managing-iam/provisioning-user-accounts-with-scim/troubleshooting-team-membership-with-identity-provider-groups.md

@@ -78,3 +78,59 @@ If synchronization of team membership with a group on your IdP fails due to a pr
 {% data variables.product.prodname_dotcom %} will try to resolve this problem automatically during the next sync, which occurs at least once daily. You may be able to resolve the problem by unlinking the impacted team from the IdP group and then linking it to the same group again. For more information, see [AUTOTITLE](/admin/managing-iam/provisioning-user-accounts-with-scim/managing-team-memberships-with-identity-provider-groups#managing-the-connection-between-an-existing-organization-team-and-an-idp-group).
 
 If the problem persists, contact {% data variables.contact.contact_ent_support %} and provide details about the organization, team, and the IdP group you're experiencing problems with.
+
+## SCIM API incomplete events
+
+If you see an `external_identity.scim_api_incomplete` or `external_group.scim_api_incomplete` event in your enterprise audit log, a SCIM request from your identity provider was received by {% data variables.product.github %} but did not complete successfully. No response was sent back to your identity provider, which may report the operation as failed or timed out.
+
+### Resolving the issue
+
+Re-trigger provisioning from your identity provider for the affected user or group. SCIM operations are idempotent, so re-provisioning will not create duplicates.
+
+* **Entra ID:** In the Microsoft Entra admin center, go to **Enterprise Applications** > your SCIM app > **Provisioning**, and use **Provision on demand** for the affected user or group, or **Restart provisioning** for a full sync. For more information, see [On-demand provisioning in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity/app-provisioning/provision-on-demand) in the Microsoft documentation.
+* **Okta:** Re-push the affected group from **Push Groups**, or re-assign the app to the affected user. For more information, see [Push Groups](https://help.okta.com/en-us/content/topics/users-groups/usgr-push-groups.htm) in the Okta documentation.
+* **Other identity providers:** Consult your identity provider's documentation for how to re-trigger SCIM provisioning for a specific user or group.
+
+### Checking if the change was applied
+
+If you have audit log streaming configured, you can search your streamed logs for other events with the same `request_id` value from the `scim_api_incomplete` event. For more information, see [AUTOTITLE](/admin/monitoring-activity-in-your-enterprise/reviewing-audit-logs-for-your-enterprise/streaming-the-audit-log-for-your-enterprise).
+
+A single group SCIM API call can trigger any of the following events during processing:
+
+| Audit log event | Description |
+| --- | --- |
+| `external_group.provision` | Group was created |
+| `external_group.delete` | Group was deleted |
+| `external_group.update` | Group metadata was updated |
+| `external_group.update_display_name` | Display name was changed |
+| `external_group.add_member` | A specific member was added |
+| `external_group.remove_member` | A specific member was removed |
+
+To determine which member changes were applied before the interruption, the `add_member` and `remove_member` events are the most useful. They identify the specific member affected. If you find fewer member events than the request intended, the remaining members were not processed.
+
+> [!NOTE]
+> The enterprise audit log UI and REST API do not currently support filtering by `request_id`. Audit log streaming to a SIEM or log platform is required for this step.
+
+### Common causes
+
+### Common causes of incomplete events
+
+* Processing time exceeds the connection timeout, often because of large groups.
+* A network interruption occurs between the identity provider and {% data variables.product.github %}.
+* A transient issue occurs on {% data variables.product.github %}'s infrastructure.
+* Your network environment, such as corporate proxies, firewalls, or CASB solutions, interferes with the connection.
+* The identity provider's SCIM client timeout settings are too restrictive.
+
+If this event recurs for the same group or user, contact {% data variables.contact.contact_ent_support %} with the `request_id` values from the affected events.
+
+## Large group timeouts
+
+A single SCIM `PUT` or `PATCH` request for a group with a large number of members can exceed the request timeout. When this happens, your identity provider may report the operation as failed, and you may see an `external_group.scim_api_incomplete` event in your enterprise audit log.
+
+The SCIM provisioning rate limits describe a limit of 1,000 users per group per hour, but a single `PUT` or `PATCH` request that changes membership for a large group can also exceed the request timeout before all members are processed. For more information, see [AUTOTITLE](/admin/managing-iam/provisioning-user-accounts-with-scim/provisioning-users-and-groups-with-scim-using-the-rest-api#understand-rate-limits-on-github).
+
+### Preventing timeouts
+
+* **Break large groups into smaller groups.** If your identity provider supports it, consider splitting groups that frequently time out into multiple smaller groups. This reduces the processing time per SCIM request.
+* **Use incremental updates.** Where possible, use `PATCH` requests to add or remove individual members rather than `PUT` requests that replace the entire membership list.
+* **Monitor for incomplete events.** Set up audit log streaming and alert on `scim_api_incomplete` events so you can re-trigger provisioning promptly.