I was able to reproduce this locally on 0.33.0 but not on 0.32.0. Some tests (e.g. some parameter of tests/test_sessions_pool.py::test_size) randomly fail with the same problem:
------------------------------ Captured log call -------------------------------
WARNING scrapy_zyte_api._session:_session.py:628 ZYTE_API_SESSION_POOL_SIZES is deprecated, use ZYTE_API_SESSION_POOLS instead
ERROR scrapy.core.scraper:scraper.py:232 Error downloading <GET https://example.com>
Traceback (most recent call last):
File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 887, in _next_from_queue
session_id, next_use = self._queues[pool].popleft()
IndexError: pop from an empty deque
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/.tox/min-scrapy-2x6/lib/python3.10/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
result = current_context.run(
File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/.tox/min-scrapy-2x6/lib/python3.10/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/.tox/min-scrapy-2x6/lib/python3.10/site-packages/scrapy/core/downloader/middleware.py", line 41, in process_request
response = yield deferred_from_coro(method(request=request, spider=spider))
File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/.tox/min-scrapy-2x6/lib/python3.10/site-packages/twisted/internet/defer.py", line 1031, in adapt
extracted = result.result()
File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 1086, in process_request
return await self._sessions.assign(request)
File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 1022, in assign
session_id = await self._next(request)
File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 930, in _next
session_id = await self._next_from_queue(request, pool)
File "/home/runner/work/scrapy-zyte-api/scrapy-zyte-api/scrapy_zyte_api/_session.py", line 891, in _next_from_queue
raise RuntimeError(
RuntimeError: Could not get a session ID from the session rotation queue after 60 attempts, waiting at least 0.0 seconds between attempts. See https://scrapy-zyte-api.readthedocs.io/en/latest/usage/session.html#session-troubleshooting-could-not-get-session-id
I was able to reproduce this locally on 0.33.0 but not on 0.32.0. Some tests (e.g. some parameter of
tests/test_sessions_pool.py::test_size) randomly fail with the same problem: