Skip to content

Flaky tests since 0.33.0 #277

Description

@wRAR

I was able to reproduce this locally on 0.33.0 but not on 0.32.0. Some tests (e.g. some parameter of tests/test_sessions_pool.py::test_size) randomly fail with the same problem:

------------------------------ Captured log call -------------------------------
WARNING  scrapy_zyte_api._session:_session.py:628 ZYTE_API_SESSION_POOL_SIZES is deprecated, use ZYTE_API_SESSION_POOLS instead
ERROR    scrapy.core.scraper:scraper.py:232 Error downloading <GET https://example.com>
Traceback (most recent call last):
  File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 887, in _next_from_queue
    session_id, next_use = self._queues[pool].popleft()
IndexError: pop from an empty deque

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/.tox/min-scrapy-2x6/lib/python3.10/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
    result = current_context.run(
  File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/.tox/min-scrapy-2x6/lib/python3.10/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/.tox/min-scrapy-2x6/lib/python3.10/site-packages/scrapy/core/downloader/middleware.py", line 41, in process_request
    response = yield deferred_from_coro(method(request=request, spider=spider))
  File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/.tox/min-scrapy-2x6/lib/python3.10/site-packages/twisted/internet/defer.py", line 1031, in adapt
    extracted = result.result()
  File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 1086, in process_request
    return await self._sessions.assign(request)
  File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 1022, in assign
    session_id = await self._next(request)
  File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 930, in _next
    session_id = await self._next_from_queue(request, pool)
  File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 891, in _next_from_queue
    raise RuntimeError(
RuntimeError: Could not get a session ID from the session rotation queue after 60 attempts, waiting at least 0.0 seconds between attempts. See https://scrapy-zyte-api.readthedocs.io/en/latest/usage/session.html#session-troubleshooting-could-not-get-session-id

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions