feat: retry HTTP 429 responses with jittered backoff by dgarros · Pull Request #1149 · opsmill/infrahub-sdk-python

dgarros · 2026-07-08T08:35:11Z

Why

When the Infrahub server returns HTTP 429 (Too Many Requests), the SDK surfaces the error immediately — so every caller (infrahubctl, the Ansible collection, custom scripts) has to hand-roll retry logic or just fail. Background workloads (batching, syncs, generators) are the traffic most likely to be rate-limited and had no cooperative way to back off.

Goal: the SDK transparently retries 429s with jittered exponential backoff, honours the server's Retry-After, and raises a dedicated catchable error once a configurable budget is exhausted — so callers ride through transient rate-limiting with no retry code of their own.

Non-goals: retrying other status codes (503, etc.); server-side rate limiting / prioritisation (INFP-636/635); changing the existing connectivity retry_on_failure mechanism.

Closes #1124 (Jira IHS-249)

What changed

Behavioural:

429 responses are retried automatically (default 5 retries) with jittered exponential backoff — callers receive the eventual success transparently.
The server's Retry-After header (delta-seconds and HTTP-date) is honoured in place of computed backoff, clamped to a max; malformed headers fall back to computed backoff.
On sustained rate-limiting the SDK raises a new RateLimitError (carrying url, attempts, last Retry-After; chains the underlying httpx.HTTPStatusError as __cause__).
Identical behaviour across InfrahubClient (async) and InfrahubClientSync (sync), applied to all request paths: queries, mutations, multipart uploads, streaming initiation, and auth.
Each retry logs at WARNING (url, attempt, delay) — no headers/credentials logged.

Config (all additive): rate_limit_retry_enabled (True), rate_limit_max_retries (5), rate_limit_backoff_base (0.5), rate_limit_backoff_max (60). Set rate_limit_retry_enabled=False to fully restore the previous behaviour.

Implementation notes: a new pure-logic RateLimitRetryHandler (no I/O) owns parsing/backoff/decision; thin async & sync drivers wrap the three send sites. Multipart retries rewind/re-materialize the body per attempt.

What stayed the same: no existing method signatures changed; retry_on_failure untouched; no new dependencies (httpx + stdlib only).

How to review

Suggested order: infrahub_sdk/rate_limit.py (pure logic) → exceptions.py + config.py → infrahub_sdk/client.py (drivers + three wired send sites, async & sync) → tests. Extra scrutiny on the async/sync driver parity and the streaming-init retry resource handling. Full design rationale in dev/specs/ihs-249-sdk-429-retry/.

How to test

uv run pytest tests/unit/test_rate_limit.py tests/unit/sdk/test_rate_limit_retry.py

Expected: 62 passing — handler unit tests + client-level tests parametrized across async/sync (transparent 429→200, Retry-After honouring, exhaustion→RateLimitError, disable/tune, all-paths coverage, multipart body-survives-retry, jitter growth/divergence).

Impact & rollout

Backward compatibility: one caller-visible change — a persistent 429 now raises RateLimitError after retries exhaust instead of httpx.HTTPStatusError (raw error still available via __cause__). Documented in changelog/1124.changed.md. Retry is on by default; rate_limit_retry_enabled=False opts out.
Performance: no hot-path impact; worst-case blocking is bounded (~max_retries × backoff_max ≈ 300s with defaults).
Config/env changes: four additive rate_limit_* Config fields (env-overridable).
Deployment notes: safe to deploy; additive.

Checklist

Tests added/updated (62 tests)
Changelog entry added (1124.added.md, 1124.changed.md)
External docs updated (config.mdx regenerated)
Internal .md docs updated (dev/specs/ihs-249-sdk-429-retry/)

Summary by cubic

Adds transparent retries for HTTP 429 responses with jittered exponential backoff across both clients, honoring Retry-After, and raising RateLimitError when the retry budget is exhausted. Also hardens exhaustion and streaming edge cases, and fixes markdownlint/Vale warnings in spec docs.

New Features
- Automatically retries 429s with jittered exponential backoff; honors Retry-After (delta-seconds or HTTP-date) and clamps to a max.
- Works across InfrahubClient and InfrahubClientSync for all paths: standard requests, multipart uploads (with safe body rewind), streaming initiation, and auth.
- New RateLimitError is raised after retries are exhausted and chains the underlying httpx.HTTPStatusError when available.
- Configurable via rate_limit_retry_enabled (True), rate_limit_max_retries (5), rate_limit_backoff_base (0.5), rate_limit_backoff_max (60). Hardened: caps backoff exponent, guards negative/overflow Retry-After, always raises on request-less 429 exhaustion, and closes streaming resources on failure.
Migration
- Persistent 429s now raise RateLimitError instead of httpx.HTTPStatusError. Catch RateLimitError (or the SDK base Error); the original httpx.HTTPStatusError is available via __cause__ when present.
- To revert to previous behavior, set rate_limit_retry_enabled=False.

^{Written for commit c01b7dd. Summary will update on new commits.}

Specify phase for IHS-249. Adds spec.md (user journeys P1-P3 + tune, FR-001..009, success criteria, edge cases, out-of-scope) and the requirements quality checklist. Resolves the PRD open question by chaining the underlying transport error as the RateLimitError cause. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Plan phase for IHS-249. Adds plan.md, research.md, data-model.md, contracts/ (Config fields, RateLimitError, RateLimitRetryHandler), and quickstart.md. Points the agent-context plan reference at the new plan. Key design finding: the retry chokepoint is not singular — login routes through _request, but _request_multipart and _get_streaming bypass it, so the retry driver is applied at all three send sites per client to satisfy FR-006. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Verdict: PROCEED WITH UPDATES. Records the dual-lens critique and applies its findings: - E2/X1 (Must-Address): multipart retry could re-send a consumed file body; plan + data-model now require rewinding/re-materializing the payload per attempt, plus a regression test. - P3: build-vs-buy rationale (custom vs tenacity/httpx) in research. - P4: worst-case cumulative wait documented in plan. - E4: retry logs constrained to URL/attempt/delay (no secrets). - E6: multipart full-body-on-retry validation added to quickstart. - E3: mutation-retry assumption accepted (PRD pre-processing 429). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Tasks phase for IHS-249: 21 tasks across setup, foundational retry machinery (handler, RateLimitError, Config fields, drivers on all three send sites of both clients incl. the E2/X1 multipart re-read fix), four user-story validation phases, and polish (FR-006 coverage, E2 regression test, towncrier fragments, docs/lint gates). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Compares spec.md against the Jira IHS-249 PRD. Verdict: ALIGNED. All FR-001..009, journeys, acceptance criteria, success criteria, and out-of-scope boundaries carried over faithfully. Only additions are the authorized open-question resolution (RateLimitError __cause__) and SC-006 (derived from FR-009). No remediation needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Verified against client.py that InfrahubClient and InfrahubClientSync are symmetric: each has three direct send sites (_request, _request_multipart, _get_streaming) and multipart/streaming bypass _request on both. Send-site audit confirms exactly six sites total, no fourth path. Tightened plan source-code section, research R1, and tasks T009/T010 with concrete per-client method line refs so the retry driver is wired on both clients (FR-008). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add transparent retry-with-backoff on HTTP 429 across both clients (T001-T010): - New pure RateLimitRetryHandler (Retry-After parsing, jittered/clamped exponential backoff, retry-budget decision) in infrahub_sdk/rate_limit.py - Four rate_limit_* fields on ConfigBase; new RateLimitError(Error) - Async/sync _send_with_rate_limit_retry drivers wired into _request, _request_multipart (with per-attempt file rewind, critique E2/X1), and _get_streaming (retry on stream initiation) - Handler unit tests; client-level test skeleton Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add async/sync parametrized tests exercising the real _request -> _send_with_rate_limit_retry path: a scripted [429, 200] sequence retries transparently and returns the 200 after exactly two sends, and non-429 responses pass through untouched with a single send. Driver sleep is patched via monkeypatch to avoid real waits. Covers T011 and T012 (SC-001). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Cover honouring Retry-After on 429 across async and sync clients: delta-seconds, HTTP-date, zero/past-date (~0s), clamp above rate_limit_backoff_max, and malformed header falling back to computed jittered backoff while still retrying. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add parametrized async+sync tests asserting persistent 429 exhausts the retry budget: exactly max_retries + 1 sends, one RateLimitError with url/attempts/retry_after and a chained httpx.HTTPStatusError cause, and one WARNING log per retry carrying url, attempt number, and delay. Covers T014; T015 verified — driver synthesizes and chains the terminal error and tracks last_retry_after with no source change needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add client-level tests for the rate-limit retry disabled path, the rate_limit_max_retries budget, and explicit async/sync parity on an identical 429 sequence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add FR-006 all-paths coverage (regular request, multipart upload, streaming initiation on both async and sync clients) and the E2/X1 regression test that asserts a retried multipart upload re-sends the full body byte-for-byte (modulo the random boundary), driving 429->200 at the httpx transport layer via pytest-httpx. Add towncrier fragments for the transparent 429 retry feature and regenerate the Config reference for the four new rate_limit_* fields. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

parse_retry_after now floors negative delta-seconds at 0.0 (preventing a negative wait that would crash the sync driver's time.sleep while asyncio tolerated it) and returns None on OverflowError from pathological digit strings so the caller falls back to computed backoff. Adds handler unit tests for both cases, driver-level tests proving exponential backoff growth/clamping and per-instance jitter divergence, and a direct unit test of _rewind_multipart_files across its files shapes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Records the implement + review tail: 6 impl chunks (T001-T021, all ticked), 3-agent review of a55cbaa..HEAD, and inline fixes for the high-severity findings (negative/overflow Retry-After crash; unguarded SC-003 growth + jitter tests; direct multipart-rewind guard test). 62 tests passing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

codecov · 2026-07-08T08:38:01Z

Codecov Report

❌ Patch coverage is 89.59538% with 18 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
infrahub_sdk/client.py	89.91%	10 Missing and 2 partials ⚠️
infrahub_sdk/rate_limit.py	88.09%	3 Missing and 2 partials ⚠️
infrahub_sdk/exceptions.py	87.50%	0 Missing and 1 partial ⚠️

@@             Coverage Diff             @@
##           develop    #1149      +/-   ##
===========================================
+ Coverage    82.35%   82.47%   +0.12%     
===========================================
  Files          138      139       +1     
  Lines        12065    12194     +129     
  Branches      1805     1823      +18     
===========================================
+ Hits          9936    10057     +121     
- Misses        1573     1576       +3     
- Partials       556      561       +5

Flag	Coverage Δ
integration-tests	`40.79% <36.41%> (-0.19%)`	⬇️
python-3.10	`56.04% <76.87%> (+0.23%)`	⬆️
python-3.11	`56.06% <76.87%> (+0.23%)`	⬆️
python-3.12	`56.04% <76.87%> (+0.22%)`	⬆️
python-3.13	`56.06% <76.87%> (+0.23%)`	⬆️
python-3.14	`56.05% <76.87%> (+0.25%)`	⬆️
python-filler-3.12	`22.37% <12.71%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
infrahub_sdk/config.py	`91.41% <100.00%> (+0.21%)`	⬆️
infrahub_sdk/exceptions.py	`89.69% <87.50%> (-0.12%)`	⬇️
infrahub_sdk/rate_limit.py	`88.09% <88.09%> (ø)`
infrahub_sdk/client.py	`77.31% <89.91%> (+1.76%)`	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

cubic-dev-ai

3 issues found across 24 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="infrahub_sdk/client.py">

<violation number="1" location="infrahub_sdk/client.py:96">
P2: Multipart uploads backed by non-seekable streams can be retried with an already-consumed body, so the second attempt may upload empty/truncated content. Consider buffering/rebuilding the multipart payload per attempt, or disabling 429 retry for non-rewindable multipart bodies instead of silently re-sending them.</violation>
</file>

<file name="tests/unit/sdk/test_rate_limit_retry.py">

<violation number="1" location="tests/unit/sdk/test_rate_limit_retry.py:564">
P3: This assertion is probabilistic: two correct jittered runs can occasionally produce the same wait sequence, so the test can fail spuriously. A deterministic seed or patched jitter source would make the check stable.</violation>
</file>

<file name="dev/specs/ihs-249-sdk-429-retry/opsmill-implement-report.md">

<violation number="1" location="dev/specs/ihs-249-sdk-429-retry/opsmill-implement-report.md:81">
P3: This is a product feature, so steering the PR to `stable` conflicts with the repo's release-vehicle guidance. `develop` is the expected base for feature work; `stable` should stay reserved for hotfixes and non-product changes.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai · 2026-07-08T08:48:58Z

+        file_obj = value[1] if isinstance(value, tuple) and len(value) > 1 else value
+        seek = getattr(file_obj, "seek", None)
+        if callable(seek):
+            # Non-seekable streams cannot be rewound; ignore and re-send as-is.


P2: Multipart uploads backed by non-seekable streams can be retried with an already-consumed body, so the second attempt may upload empty/truncated content. Consider buffering/rebuilding the multipart payload per attempt, or disabling 429 retry for non-rewindable multipart bodies instead of silently re-sending them.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At infrahub_sdk/client.py, line 96: <comment>Multipart uploads backed by non-seekable streams can be retried with an already-consumed body, so the second attempt may upload empty/truncated content. Consider buffering/rebuilding the multipart payload per attempt, or disabling 429 retry for non-rewindable multipart bodies instead of silently re-sending them.</comment> <file context> @@ -79,6 +81,23 @@ class ProxyConfig(TypedDict): + file_obj = value[1] if isinstance(value, tuple) and len(value) > 1 else value + seek = getattr(file_obj, "seek", None) + if callable(seek): + # Non-seekable streams cannot be rewound; ignore and re-send as-is. + with suppress(OSError, ValueError): + seek(0) </file context>

cubic-dev-ai · 2026-07-08T08:48:58Z

+    # Both instances performed the same number of jittered waits ...
+    assert len(first) == len(second) == max_retries
+    # ... but real full jitter makes at least one recorded wait diverge between the two instances.
+    assert first != second


P3: This assertion is probabilistic: two correct jittered runs can occasionally produce the same wait sequence, so the test can fail spuriously. A deterministic seed or patched jitter source would make the check stable.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/unit/sdk/test_rate_limit_retry.py, line 564: <comment>This assertion is probabilistic: two correct jittered runs can occasionally produce the same wait sequence, so the test can fail spuriously. A deterministic seed or patched jitter source would make the check stable.</comment> <file context> @@ -0,0 +1,756 @@ + # Both instances performed the same number of jittered waits ... + assert len(first) == len(second) == max_retries + # ... but real full jitter makes at least one recorded wait diverge between the two instances. + assert first != second + + </file context>

cubic-dev-ai · 2026-07-08T08:48:58Z

+
+## 7. Suggested next steps
+
+1. **Open a PR** for branch `dga/feat-409-retry-ivj0i` (base `stable`) — the feature is complete, tested (62 passing), and reviewed. Ensure both towncrier fragments (`changelog/1124.added.md`, `1124.changed.md`) are included; the `429 → RateLimitError` behaviour change is a caller-visible change flagged in the changed fragment.


P3: This is a product feature, so steering the PR to stable conflicts with the repo's release-vehicle guidance. develop is the expected base for feature work; stable should stay reserved for hotfixes and non-product changes.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At dev/specs/ihs-249-sdk-429-retry/opsmill-implement-report.md, line 81: <comment>This is a product feature, so steering the PR to `stable` conflicts with the repo's release-vehicle guidance. `develop` is the expected base for feature work; `stable` should stay reserved for hotfixes and non-product changes.</comment> <file context> @@ -0,0 +1,84 @@ + +## 7. Suggested next steps + +1. **Open a PR** for branch `dga/feat-409-retry-ivj0i` (base `stable`) — the feature is complete, tested (62 passing), and reviewed. Ensure both towncrier fragments (`changelog/1124.added.md`, `1124.changed.md`) are included; the `429 → RateLimitError` behaviour change is a caller-visible change flagged in the changed fragment. +2. **Optional hardening** (deferred LOW findings): decide whether a 429 retry on a non-seekable multipart stream should raise a clear error instead of silently sending an empty body; add the small missing tests (`retry_after is None` exhaustion, Config defaults/validators). +3. **Separate follow-up** for the repo's `docs-generate`/markdownlint tooling breakage (missing `.markdownlint.yaml`; unrelated `.mdx` drift) — outside this feature's scope. </file context>

markdown-lint: pad table delimiter rows (MD060 compact style), add blank lines around headings/tables/lists (MD022/MD032/MD058), and replace emphasis-as-heading with plain text (MD036) in the generated spec artifacts under dev/specs/ihs-249-sdk-429-retry/. vale: add "backoff" to the spelling exception vocabulary so the generated config.mdx retry docs pass Infrahub.spelling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cloudflare-workers-and-pages · 2026-07-08T09:14:24Z

Deploying infrahub-sdk-python with Cloudflare Pages

Latest commit:	`c01b7dd`
Status:	✅ Deploy successful!
Preview URL:	https://a507059e.infrahub-sdk-python.pages.dev
Branch Preview URL:	https://dga-feat-409-retry-ivj0i.infrahub-sdk-python.pages.dev

View logs

Addresses code-review findings on the 429 retry driver: - Exhaustion no longer leaks RuntimeError when the final 429 response has no attached request (e.g. a custom requester): the driver now captures the cause (HTTPStatusError or none) and always raises RateLimitError, removing the previously-unreachable trailing raise. - Streaming init: the ExitStack/AsyncExitStack is now closed via try/finally so a raise during the failed-429 read cannot leak the stream. - compute_backoff caps the exponent (2 ** min(attempt, 63)) so a very large rate_limit_max_retries can no longer overflow float before the clamp. - next_delay drops a redundant no-op clamp on the jittered branch. - Adds a regression test: request-less 429 exhaustion raises RateLimitError (cause None), async and sync. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

saltas888 · 2026-07-09T16:48:39Z

+        handler = RateLimitRetryHandler(
+            max_retries=self.config.rate_limit_max_retries,
+            backoff_base=self.config.rate_limit_backoff_base,
+            backoff_max=self.config.rate_limit_backoff_max,
+        )


I was thinking whether we can actually have the attempts state inside the class so that we don't always need to keep track of the attempts when using. However it seems thats intentionally designed to be stateless.

If we prefer it to be stateless, we should initialize the handler once globally. We do it now twice

dgarros and others added 14 commits July 7, 2026 11:09

test(sdk): add US4 disable/tune/parity 429 retry tests (T016)

1727b96

Add client-level tests for the rate-limit retry disabled path, the rate_limit_max_retries budget, and explicit async/sync parity on an identical 429 sequence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

dgarros added type/feature New feature or request effort/medium This issue should be completed in a less than a day labels Jul 8, 2026

github-actions Bot added the type/documentation Improvements or additions to documentation label Jul 8, 2026

cubic-dev-ai Bot reviewed Jul 8, 2026

View reviewed changes

dgarros marked this pull request as ready for review July 9, 2026 06:38

dgarros requested a review from a team as a code owner July 9, 2026 06:38

saltas888 reviewed Jul 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: retry HTTP 429 responses with jittered backoff#1149

feat: retry HTTP 429 responses with jittered backoff#1149
dgarros wants to merge 16 commits into
developfrom
dga/feat-409-retry-ivj0i

dgarros commented Jul 8, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

codecov Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cubic-dev-ai Bot Jul 8, 2026

Uh oh!

cubic-dev-ai Bot Jul 8, 2026

Uh oh!

cubic-dev-ai Bot Jul 8, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

saltas888 Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		## 7. Suggested next steps

		1. Open a PR for branch `dga/feat-409-retry-ivj0i` (base `stable`) — the feature is complete, tested (62 passing), and reviewed. Ensure both towncrier fragments (`changelog/1124.added.md`, `1124.changed.md`) are included; the `429 → RateLimitError` behaviour change is a caller-visible change flagged in the changed fragment.

Uh oh!

Conversation

dgarros commented Jul 8, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changed

How to review

How to test

Impact & rollout

Checklist

Summary by cubic

Uh oh!

codecov Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying infrahub-sdk-python with Cloudflare Pages

Uh oh!

saltas888 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dgarros commented Jul 8, 2026 •

edited by cubic-dev-ai Bot

Loading

codecov Bot commented Jul 8, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cloudflare-workers-and-pages Bot commented Jul 8, 2026 •

edited

Loading