Skip to content

Resilience: Add retry/backoff for idempotent HTTP GET requests #59

Description

@MozzamShahid

File / lines: src/gmaps_scraper/scraper.py, lines 213-277; src/gmaps_scraper/place_scraper.py, lines 2736-2778

Problem: Both collect_http_artifacts and _collect_preview_place_enrichment create a single session.get(...) for the primary request and any preload request. There is no retry loop, no jitter, and no handling of Retry-After.

Why it matters: Transient Google rate-limiting, TLS handshake failures, or short network blips immediately surface as ScrapeError or degraded place output. Since these are read-only GET requests, retrying is safe and expected for a production scraper.

Suggested fix:

  1. Introduce a small shared helper that wraps idempotent GET calls with configurable retries (e.g., 3 attempts), exponential backoff with jitter, and explicit handling of 429 / 5xx / timeout.
  2. Respect Retry-After when present.
  3. Add tests for retry behavior using mocked responses.

Difficulty: medium
Impact: High — reduces transient failures and improves batch reliability.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions