Skip to content

Bug: Entitylist preload expansion ignores continuation tokens and requests whole list at once #56

Description

@MozzamShahid

File / lines: src/gmaps_scraper/scraper.py, lines 601-632, 635-660

Problem: _extract_entitylist_payload returns only payload[0] and throws away payload[1], which is the continuation token. _expand_entitylist_preload_text then rewrites the page-size parameter to total_rows and makes a single enormous request. If Google rejects that size, the except Exception: return None silently keeps the truncated first payload.

Evidence:

candidate = payload[0]          # payload[1] (continuation token) is ignored

Suggested fix: Parse and use the continuation token. Implement a paginated fetch loop:

  1. Fetch the preload.
  2. If len(rows) < total, use the token from payload[1] to request the next page.
  3. Accumulate rows (or the full payloads) into script_texts.
  4. Cap the single-request page size and log a quality flag when pagination is used.

Difficulty: high-value
Impact: High — saved lists with many places may be truncated or silently incomplete.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions