Add rate limiting and retry configuration to Contentstack provider#44
Conversation
- Add rate_limit, rate_burst, and max_retries provider attributes - Update provider schema to support new rate limiting options - Add automatic retry logic integration with contentstack-go-sdk - Update provider documentation with new attributes - Add simplified examples for provider and rate-limiting configurations - Update go.mod to use enhanced contentstack-go-sdk with rate limiting This enables reliable Terraform operations by automatically handling Contentstack API rate limits (10 req/sec) with configurable throttling and exponential backoff retry logic for 429 responses. Related: contentstack-go-sdk rate limiting implementation
- Add retryableRoundTripper for seamless SDK integration - Implement retry policy for 429 responses only - Add max_retries, retry_wait_min, retry_wait_max provider attributes - Update examples and documentation - Use exponential backoff with configurable wait times - Maintain compatibility with existing rate limiting
|
Hi @demeyerthom , |
demeyerthom
left a comment
demeyerthom
left a comment
There was a problem hiding this comment.
Overall looks good! I left 2 comments to fix.
Can you also add a changie file for the changelog? It triggers our deployment processes:
- install changie (https://changie.dev/guide/installation/)
- run
changie new - fill in the questions
- add generated file to the commit
| BaseURL: config.BaseURL.Value, | ||
| AuthToken: config.AuthToken.Value, | ||
| HTTPClient: httpClient, | ||
| RateLimit: config.RateLimit.Value, |
There was a problem hiding this comment.
I think those fields are not part of the config are they?
There was a problem hiding this comment.
Hi @demeyerthom,
Good catch! Those RateLimit and RateBurst fields are part of the SDK changes I've proposed in contentstack-go-sdk PR labd/contentstack-go-sdk#17.
The fields aren't in the published SDK (v0.1.0) yet, so this terraform provider PR depends on those SDK changes being merged first.
Would you be open to reviewing and merging the SDK PR first? Once that's released, this provider PR will work with the new SDK version. Happy to make any adjustments needed on either PR!
Thanks for your time reviewing this!
| @@ -0,0 +1,33 @@ | |||
| terraform { | |||
There was a problem hiding this comment.
This bit will not show in the docs. You want to add it to examples/provider/main.tf, and then run go generate to generate the correct docs for it to show
There was a problem hiding this comment.
@robertolosanno-e2x are you okay to sort this comment out? Or alternatively allow me to push to your fork :)
|
I've recently started using this provider on a project, and am unfortunately running into the same issue as @robertolosanno-e2x, of getting a bunch of 429 errors when running tf plan. Any chance we can have this and the dependant pr reviewed? |
|
Hi @demeyerthom , Following your feedback, I've updated this PR to address the dependency on unreleased SDK changes. What changed in this PR:
What this PR now ships:
Plan for rate limiting: This should unblock users hitting 429 errors today (like @rasmus-lisborg-19) while keeping the dependency chain clean. |
|
I think all comments have been addressed. Any chance we could get it re-reviewed @demeyerthom ? |
3 participants
This PR adds automatic retry configuration to the Contentstack Terraform provider to address API rate limit issues that cause Terraform operations to fail with 429 errors.
Changes implemented:
Provider Retry Configuration: Added three new optional provider attributes:
max_retries(int): Maximum retry attempts for 429 responses with exponential backoff (defaults to 3)retry_wait_min(int): Minimum wait time in seconds between retries (defaults to 1)retry_wait_max(int): Maximum wait time in seconds between retries (defaults to 30)Retry Implementation: Uses
go-retryablehttpto wrap the SDK's HTTP client with automatic retry logic:retry_wait_minandretry_wait_maxcontentstack-go-sdkDocumentation Updates: Auto-generated provider documentation now includes all new retry attributes with detailed descriptions.
Example Configuration: Updated
examples/provider/main.tfto demonstrate retry configuration alongside the standard provider setup.Rationale:
Contentstack's Management API has a 10 requests/second rate limit. When Terraform makes concurrent requests during plan/apply operations with multiple resources, it frequently hits 429 errors causing operations to fail. This is particularly problematic for users managing large numbers of content types and global fields (~35+ resources as mentioned in the related issue).
This PR ships retry logic as a self-contained change with no SDK dependency. Proactive rate limiting (token bucket) will follow in a separate PR once contentstack-go-sdk#17 is merged.
Fixes #3
NEW FEATURES | BUG FIXES
max_retriesprovider attribute to configure automatic retry attempts for 429 responsesretry_wait_minprovider attribute to configure minimum wait time between retriesretry_wait_maxprovider attribute to configure maximum wait time between retries