Is your feature request related to a problem? Please describe.
Yes. Some SaaS APIs return temporary client-side errors for specific endpoints, such as HTTP 409 during in-progress inventory generation, and also use non-standard rate-limit headers instead of Retry-After.
Today, default retry behavior treats most 4xx responses as non-retryable and only supports a fixed set of rate-limit headers. This causes avoidable crawl failures or forces coarse workarounds like increasing polling intervals globally.
Describe the solution you'd like
Add configurable, endpoint-level retry behavior in source-crawler so specific client status codes can be retried only where needed.
Also add configurable rate-limit header parsing so connectors can use vendor-specific headers for retry delay calculation, while preserving fallback backoff behavior.
Keep defaults backward-compatible so existing connectors behave exactly as before unless they opt in.
Describe alternatives you've considered (Optional)
- Make HTTP 409 globally retryable: rejected because 409 often means true business conflict and should not always be retried.
- Keep fixed headers only and tune polling intervals: rejected because this is connector-specific and can increase latency or reduce throughput unnecessarily.
- Implement connector-specific custom handlers in each plugin: works but increases duplication and maintenance cost compared to shared framework extensions.
Additional context
The change is intentionally scoped to shared retry utilities and tests in source-crawler.
Key goals are minimal blast radius, explicit opt-in behavior, and compatibility with declarative-source adoption in a follow-up step.
Is your feature request related to a problem? Please describe.
Yes. Some SaaS APIs return temporary client-side errors for specific endpoints, such as HTTP 409 during in-progress inventory generation, and also use non-standard rate-limit headers instead of Retry-After.
Today, default retry behavior treats most 4xx responses as non-retryable and only supports a fixed set of rate-limit headers. This causes avoidable crawl failures or forces coarse workarounds like increasing polling intervals globally.
Describe the solution you'd like
Add configurable, endpoint-level retry behavior in source-crawler so specific client status codes can be retried only where needed.
Also add configurable rate-limit header parsing so connectors can use vendor-specific headers for retry delay calculation, while preserving fallback backoff behavior.
Keep defaults backward-compatible so existing connectors behave exactly as before unless they opt in.
Describe alternatives you've considered (Optional)
Additional context
The change is intentionally scoped to shared retry utilities and tests in source-crawler.
Key goals are minimal blast radius, explicit opt-in behavior, and compatibility with declarative-source adoption in a follow-up step.