sync: add --concurrency and --retry-timeout flags#5201
Conversation
The sync engine hardcoded its in-flight request limit at 20 and relied on the SDK's retry layer, which only retries 429 and 504 (databricks-sdk-go httpclient/errors.go DefaultErrorRetriable). 502 and 503 surface as fatal errors that fail the entire sync. - Add SyncOptions.Concurrency and SyncOptions.RetryTimeout; default to MaxRequestsInFlight and DefaultRetryTimeout respectively when zero so existing in-process callers (bundle deploy/files) need no changes. - Wrap each filer call (PUT, delete, mkdir, rmdir) in retries.Poll, retrying on 502/503/504 within the deadline and logging each retry at warn level. Uses the same retries.Poll helper used elsewhere in the CLI for state polling. - Surface --concurrency (default 5) and --retry-timeout (default 30s) on 'databricks sync' and 'databricks bundle sync'. 429 stays out of the new retry layer because the SDK already handles it with rate-limit-aware backoff. 500 stays out because it is not always transient for Databricks APIs. Slack thread: https://replit.slack.com/archives/C0A2Z9042FR/p1778082184841149
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Approval status: pending
|
The sync engine hardcoded its in-flight request limit at 20 and relied on the SDK's retry layer, which only retries 429 and 504 (databricks-sdk-go httpclient/errors.go DefaultErrorRetriable). 502 and 503 surface as fatal errors that fail the entire sync.
429 stays out of the new retry layer because the SDK already handles it with rate-limit-aware backoff. 500 stays out because it is not always transient for Databricks APIs.
Slack thread:
https://replit.slack.com/archives/C0A2Z9042FR/p1778082184841149
Changes
Why
Tests