File / lines: src/gmaps_scraper/scraper.py, lines 601-632, 635-660
Problem: _extract_entitylist_payload returns only payload[0] and throws away payload[1], which is the continuation token. _expand_entitylist_preload_text then rewrites the page-size parameter to total_rows and makes a single enormous request. If Google rejects that size, the except Exception: return None silently keeps the truncated first payload.
Evidence:
candidate = payload[0] # payload[1] (continuation token) is ignored
Suggested fix: Parse and use the continuation token. Implement a paginated fetch loop:
- Fetch the preload.
- If
len(rows) < total, use the token from payload[1] to request the next page.
- Accumulate rows (or the full payloads) into
script_texts.
- Cap the single-request page size and log a quality flag when pagination is used.
Difficulty: high-value
Impact: High — saved lists with many places may be truncated or silently incomplete.
File / lines:
src/gmaps_scraper/scraper.py, lines 601-632, 635-660Problem:
_extract_entitylist_payloadreturns onlypayload[0]and throws awaypayload[1], which is the continuation token._expand_entitylist_preload_textthen rewrites the page-size parameter tototal_rowsand makes a single enormous request. If Google rejects that size, theexcept Exception: return Nonesilently keeps the truncated first payload.Evidence:
Suggested fix: Parse and use the continuation token. Implement a paginated fetch loop:
len(rows) < total, use the token frompayload[1]to request the next page.script_texts.Difficulty: high-value
Impact: High — saved lists with many places may be truncated or silently incomplete.