feat(source-amazon-seller-partner): configurable cooldown retry and 429 error handling#77837
Conversation
…L report failures Add failed_retry_wait_time_in_seconds: 1800 (30 minutes) to the shared basic_async_retriever definition. When Amazon SP-API returns FATAL status due to its undocumented per-report-type cooldown, the CDK will now defer the retry until the cooldown elapses instead of retrying immediately. This requires the CDK change from airbytehq/airbyte-python-cdk#1016. Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
…d update CDK to 7.19.0 - Make failed_retry_wait_time_in_seconds configurable via config with default 1800 (30m) - Add separate 429 error handler on creation_requester with configurable max_retries - Update base image to source-declarative-manifest:7.19.0 (includes deferred retry feature) Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
devin-ai-integration
Bot
changed the title
….6-rc.3 Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
…ields Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
|
Deploy preview for airbyte-docs ready!
Deployed with vercel-action |
…section Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
…ection Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
… section Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
|
…on_requester_429_max_retries spec field Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
…K to 7.19.0 which supports interpolated max_retries Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
|
/publish-connectors-prerelease
|
|
Important Active progressive rollout warning for This PR can bypass the warning only after the PR-description ACK checkbox is checked. Detected signals:
To bypass this warning, check this box in the PR description:
What happens if this PR is merged?Checking the ACK box does not stop the active rollout by itself. It only allows this workflow's required check to pass. If the PR then merges, the result depends on what connector version is published after merge. Expected outcomes by version-change type
|
- Add with_failed_retry_wait_time_in_seconds() to ConfigBuilder for test config override - Add without_amz_date() to RequestBuilder for tick=True compatibility - Update FATAL tests to use 1s cooldown + tick=True + 3 response mocks for retry cycles - All 314 tests pass in ~16 minutes (FATAL tests complete in ~2 minutes) Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
…_response Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
|
/force-merge reason="long running integration tests"
|
octavia-bot-admin
Bot
deleted the
devin/1778160571-amazon-sp-failed-retry-cooldown
branch
3 participants
What
Configures cooldown-aware deferred retry for report streams that receive Amazon's
FATALstatus due to SP-API's undocumented per-report-type cooldown, and adds a dedicated 429 error handler oncreation_requesterwith configurable max retries and custom backoff.Previously, when a report request FATALed, the CDK retried immediately 3 times — all of which failed because the cooldown (~30 min for near-real-time FBA reports) hadn't elapsed. This wasted all retry attempts and surfaced as a
system_errorto users.Depends on CDK v7.19.0 which includes:
DefaultErrorHandler.max_retries: feat(declarative): allow DefaultErrorHandler.max_retries to be interpolated from config airbyte-python-cdk#1017How
Configurable deferred retry —
failed_retry_wait_time_in_secondsonbasic_async_retrieveris interpolated from config with a default of 1800 (30 min), minimum of 1, and maximum of 14400 (4 hours). Connector users or support can override it via the config API without a code change.Dedicated 429 error handler — A new
DefaultErrorHandlerentry in thecreation_requester'sCompositeErrorHandlermatches HTTP 429 specifically, withmax_retriesinterpolated fromconfig.get('creation_requester_429_max_retries', 5)and a user-facingerror_message. The handler uses the existingAmazonSellerPartnerWaitTimeFromHeaderBackoffStrategyfor backoff. Other errors (e.g. 403) retain the existing handler with default max retries.Hidden spec fields — Two fields added to the connector spec with
airbyte_hidden: true(not visible in UI, settable via API):failed_retry_wait_time_in_seconds(integer, default 1800, range 1–14400)creation_requester_429_max_retries(integer, default 5, minimum 0)CDK base image bump —
source-declarative-manifestupdated from 7.18.0 → 7.19.0. Unit test CDK dependency bumped from^7.18.0→^7.19.0(required formax_retriesinterpolation support).Unit test fixes for deferred retry — FATAL status tests updated to work with cooldown-aware retry:
ConfigBuilder.with_failed_retry_wait_time_in_seconds(1)overrides the 30-min default to 1 second so tests complete quickly@freezegun.freeze_time(NOW.isoformat(), tick=True)allows time to advance during the test so the deferred retry cooldown actually elapsesRequestBuilder.without_amz_date()removes the dynamicx-amz-dateheader from mock matchers — withtick=True, the header's timestamp changes on each request, butHttpRequest.matches()uses subset matching so excluding it lets mocks still match_DEFAULT_MAX_JOB_RETRY = 3retry cyclesTroubleshooting documentation — Three documentation updates:
failed_retry_wait_time_in_secondstuningcreation_requester_429_max_retries, andmax_done_report_age_hoursas mitigationsmax_done_report_age_hours,creation_requester_429_max_retries,failed_retry_wait_time_in_seconds) as recommended stepsReview guide
manifest.yaml— spec properties (~line 336): two hidden config fields with defaults, descriptions, and validation constraintsmanifest.yaml—basic_async_retrieverdefinition (~line 3050): configurablefailed_retry_wait_time_in_secondsvia Jinja interpolationmanifest.yaml—creation_requester.error_handler(~line 3098): new 429 handler with interpolatedmax_retries,error_message, and custom backoff strategy, placed before existing 403 handlermetadata.yaml— base image bump (7.18.0 → 7.19.0) and version bump (5.7.6-rc.2 → 5.7.6-rc.3)unit_tests/pyproject.toml+poetry.lock— CDK test dependency bump to^7.19.0unit_tests/integration/config.py— newwith_failed_retry_wait_time_in_seconds()builder methodunit_tests/integration/request_builder.py— newwithout_amz_date()helper for tick-compatible mock matchingunit_tests/integration/test_report_based_streams.py— FATAL tests updated withtick=True, 1s cooldown override, and 3-response mocksdocs/integrations/sources/amazon-seller-partner.md— changelog entry, two new troubleshooting sections, updated Rate Limit sectionItems for reviewer attention:
config.get('failed_retry_wait_time_in_seconds', 1800)andconfig.get('creation_requester_429_max_retries', 5)evaluate correctly when the keys are absent from user config (relies on CDK Jinja interpolation with dict.get())airbyte_hidden: truecreation_requester_429_max_retrieshasminimum: 0— setting to 0 disables 429 retries entirely; verify this is acceptable_DEFAULT_MAX_JOB_RETRY = 3— if the CDK default changes, the[...] * 3response mocking will need updatingwithout_amz_date()relies onHttpRequest.matches()using subset matching for headers (_is_subdict) — confirm this CDK behavior is stableUser Impact
Reports that previously failed with
system_errordue to Amazon's cooldown will now succeed on deferred retry. The 429-specific handler gives clearer error messages and dedicated backoff for rate-limit errors on report creation. No user-facing configuration changes are required — defaults preserve current behavior — but support can tune bothfailed_retry_wait_time_in_secondsandcreation_requester_429_max_retriesvia connector config if needed.Can this PR be safely reverted and rolled back?
Important
Active progressive rollout warning for source-amazon-seller-partner.
Link to Devin session: https://app.devin.ai/sessions/2cbe53a30ea14f9f8151f407f423283b
Requested by: Daryna Ishchenko (@darynaishchenko)