File / lines: src/gmaps_scraper/scraper.py, lines 213-277; src/gmaps_scraper/place_scraper.py, lines 2736-2778
Problem: Both collect_http_artifacts and _collect_preview_place_enrichment create a single session.get(...) for the primary request and any preload request. There is no retry loop, no jitter, and no handling of Retry-After.
Why it matters: Transient Google rate-limiting, TLS handshake failures, or short network blips immediately surface as ScrapeError or degraded place output. Since these are read-only GET requests, retrying is safe and expected for a production scraper.
Suggested fix:
- Introduce a small shared helper that wraps idempotent GET calls with configurable retries (e.g., 3 attempts), exponential backoff with jitter, and explicit handling of 429 / 5xx / timeout.
- Respect
Retry-After when present.
- Add tests for retry behavior using mocked responses.
Difficulty: medium
Impact: High — reduces transient failures and improves batch reliability.
File / lines:
src/gmaps_scraper/scraper.py, lines 213-277;src/gmaps_scraper/place_scraper.py, lines 2736-2778Problem: Both
collect_http_artifactsand_collect_preview_place_enrichmentcreate a singlesession.get(...)for the primary request and any preload request. There is no retry loop, no jitter, and no handling ofRetry-After.Why it matters: Transient Google rate-limiting, TLS handshake failures, or short network blips immediately surface as
ScrapeErroror degraded place output. Since these are read-only GET requests, retrying is safe and expected for a production scraper.Suggested fix:
Retry-Afterwhen present.Difficulty: medium
Impact: High — reduces transient failures and improves batch reliability.