Skip to content

Commit e343ed5

Browse files
vdusekclaude
andauthored
test: Add exponential backoff for shared RQ propagation delay in integration tests (#831)
## Summary - Added `call_with_exp_backoff(rq.fetch_next_request)` for shared mode in all integration tests that fetch requests right after adding/reclaiming them - Restructured while-loops to use backoff on the first fetch, preventing premature loop exit when the API hasn't propagated yet - Added `request: pytest.FixtureRequest` parameter to tests that were missing it for shared mode detection - All changes reference the known propagation delay issue #808 ## Test plan - [ ] Integration tests pass in CI (both `single` and `shared` parameterizations) - [ ] No more sporadic `test_rq_long_url[shared]` failures (and similar tests) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 519f6d1 commit e343ed5

File tree

2 files changed

+136
-138
lines changed

2 files changed

+136
-138
lines changed

tests/integration/_utils.py

Lines changed: 24 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
from __future__ import annotations
22

33
import asyncio
4-
from typing import TYPE_CHECKING, TypeVar
4+
from typing import TYPE_CHECKING, Literal, TypeVar
55

66
from crawlee._utils.crypto import crypto_random_object_id
77

@@ -13,26 +13,39 @@
1313
T = TypeVar('T')
1414

1515

16-
async def call_with_exp_backoff(fn: Callable[[], Awaitable[T]], *, max_retries: int = 3) -> T | None:
16+
async def call_with_exp_backoff(
17+
fn: Callable[[], Awaitable[T]],
18+
*,
19+
rq_access_mode: Literal['single', 'shared'],
20+
max_retries: int = 3,
21+
) -> T | None:
1722
"""Call an async callable with exponential backoff retries until it returns a truthy value.
1823
1924
In shared request queue mode, there is a propagation delay before newly added, reclaimed, or handled requests
2025
become visible in the API (see https://github.com/apify/apify-sdk-python/issues/808). This helper retries with
2126
exponential backoff to handle that delay in integration tests.
27+
28+
When `rq_access_mode` is `'single'`, the function is called once without retries.
2229
"""
23-
result = None
30+
if rq_access_mode == 'single':
31+
return await fn()
32+
33+
if rq_access_mode == 'shared':
34+
result = None
35+
36+
for attempt in range(max_retries):
37+
result = await fn()
2438

25-
for attempt in range(max_retries):
26-
result = await fn()
39+
if result:
40+
return result
2741

28-
if result:
29-
return result
42+
delay = 2**attempt
43+
Actor.log.info(f'{fn} returned {result!r}, retrying in {delay}s (attempt {attempt + 1}/{max_retries})')
44+
await asyncio.sleep(delay)
3045

31-
delay = 2**attempt
32-
Actor.log.info(f'{fn} returned {result!r}, retrying in {delay}s (attempt {attempt + 1}/{max_retries})')
33-
await asyncio.sleep(delay)
46+
return result
3447

35-
return result
48+
raise ValueError(f'Invalid rq_access_mode: {rq_access_mode}')
3649

3750

3851
def generate_unique_resource_name(label: str) -> str:

0 commit comments

Comments
 (0)