Power BI Admin API ingestion fails with throttling on large tenants (9000+ workspaces) — v1.12.6

[jothinathan]

Environment

OpenMetadata version: 1.12.6
Deployment: Docker (local) and OCI (production)
Airflow: bundled ingestion container
Power BI connector: Admin API mode (useAdminApis: true)

Issue Summary
Power BI metadata ingestion consistently fails with 0 records processed when connecting to a large shared Power BI tenant (~9993 workspaces). The ingestion retries indefinitely against https://api.powerbi.com/v1.0/myorg/admin/groups and eventually gets killed by Airflow as a timed-out task.

What is working correctly

• Access token generated successfully every run
• Test connection passes every time
• Direct Postman call to admin/groups returns 200 OK with all workspaces
• Network connectivity from inside the ingestion container confirmed via curl
• Power BI tenant settings correctly configured by tenant admin (all three required settings enabled)
• Service principal correctly added to security group scoped to Admin API settings

Root cause identified
The Power BI Admin API enforces a rate limit of 200 calls per hour per service principal. When the OpenMetadata ingestion pipeline starts, it calls admin/groups with pagination_entity_per_page=100 to paginate through all workspaces. With ~9993 workspaces in the tenant, this requires approximately 100 sequential API calls just to retrieve the workspace list, which exhausts the hourly quota almost immediately.
Once the rate limit is hit, the API begins throttling — returning either a 429 response or in some cases behaving as 403. OpenMetadata's retry handler in client.py then enters a retry loop (100 retries, starting at 30s backoff, escalating to 150s) consuming the remaining quota with each retry, making recovery impossible within the same hour. The ingestion task is eventually killed by Airflow before any records are processed.
Evidence of the throttling:
sleep 30 seconds and retrying .../admin/groups 100 more time(s)...
sleep 60 seconds and retrying .../admin/groups 99 more time(s)...
sleep 90 seconds and retrying .../admin/groups 98 more time(s)...
sleep 120 seconds and retrying .../admin/groups 97 more time(s)...
sleep 150 seconds and retrying .../admin/groups 96 more time(s)...
Task killed, sending failed status for workflow
After the ingestion run, subsequent Postman calls to admin/groups return:
json{
"message": "You have exceeded the amount of requests allowed in the current time frame and further requests will fail. Retry in 1778 seconds."
}
The Dashboard Filter Pattern does not help — the filter is applied after OpenMetadata fetches all workspaces, so all 9993 workspaces are still retrieved before filtering occurs. The API quota is exhausted regardless of what filter is configured.

Questions for the community

Is there a way to configure OpenMetadata's Power BI connector to use server-side filtering (e.g. passing a $filter parameter to the admin/groups API) rather than fetching all workspaces and filtering client-side? The Power BI Admin Groups API supports OData $filter — for example $filter=startswith(name,'AU') — which would reduce API calls dramatically for scoped tenants.
Is there a workaround in v1.12.6 to limit which workspaces are fetched at the API level rather than post-fetch?
Is this a known issue for large shared tenants? Are there plans to support OData server-side filtering in the Power BI connector?

Workaround being considered
Switching to Non-Admin API mode (useAdminApis: false) and manually adding the service principal as a member to specific workspaces — this avoids the full tenant pagination but sacrifices full lineage capability (only Push Dataset lineage supported in non-admin mode).

The OData $filter approach feels like the right long-term fix — happy to raise a GitHub issue if the community confirms this is a gap in the connector.

[jothinathan]

Some screenshots to support

[jothinathan]

[harshach]

Thansk @jothinathan for reporting the issues. We are looking into this @harshsoni2024 @ulixius9

[harshsoni2024]

Hello @jothinathan ,
On this question

Is there a way to configure OpenMetadata's Power BI connector to use server-side filtering (e.g. passing a $filter parameter to the admin/groups API) rather than fetching all workspaces and filtering client-side? The Power BI Admin Groups API supports OData $filter — for example $filter=startswith(name,'AU') — which would reduce API calls dramatically for scoped tenants.
Is there a workaround in v1.12.6 to limit which workspaces are fetched at the API level rather than post-fetch?

Yes we've active support for passing OData $filter in Getgroups API.
You can use the ProjectFilterPattern in the metadata ingestion agent to apply the same filter condition on workspaces to include/exclude those using regex.