Skip to content

Commit 8cb4a8a

Browse files
vdusekclaude
andcommitted
test: fix flaky test_large_batch_operations by adding retry on empty fetches
The test was flaky because fetch_next_request() can temporarily return None due to eventual consistency in the Apify API, even when is_empty() reports requests still exist. Replace the while-not-is_empty loop with a retry pattern that tolerates up to 10 consecutive empty fetches before concluding the queue is drained. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent cf36cc2 commit 8cb4a8a

File tree

1 file changed

+10
-6
lines changed

1 file changed

+10
-6
lines changed

tests/integration/apify_api/test_request_queue.py

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -548,15 +548,19 @@ async def test_large_batch_operations(request_queue_apify: RequestQueue) -> None
548548
total_count = await rq.get_total_count()
549549
assert total_count == 500, f'total_count={total_count}'
550550

551-
# Process all in chunks to test performance
551+
# Process all requests using a retry loop to handle eventual consistency —
552+
# fetch_next_request() can temporarily return None even when requests exist.
552553
processed_count = 0
554+
max_consecutive_empty_fetches = 10
555+
consecutive_empty_fetches = 0
553556

554-
while not await rq.is_empty():
557+
while consecutive_empty_fetches < max_consecutive_empty_fetches:
555558
request = await rq.fetch_next_request()
556-
557-
# The RQ is_empty should ensure we don't get None
558-
assert request is not None, f'request={request}'
559-
559+
if request is None:
560+
consecutive_empty_fetches += 1
561+
await asyncio.sleep(0.5)
562+
continue
563+
consecutive_empty_fetches = 0
560564
await rq.mark_request_as_handled(request)
561565
processed_count += 1
562566

0 commit comments

Comments
 (0)