Skip to content

Make scraper resilient to Google Maps DOM changes (selector fallbacks, consent handling, geo data)#16

Open
Rmaawn wants to merge 3 commits into
zohaibbashir:mainfrom
Rmaawn:robust-scraper
Open

Make scraper resilient to Google Maps DOM changes (selector fallbacks, consent handling, geo data)#16
Rmaawn wants to merge 3 commits into
zohaibbashir:mainfrom
Rmaawn:robust-scraper

Conversation

@Rmaawn

@Rmaawn Rmaawn commented Jun 17, 2026

Copy link
Copy Markdown

Summary

The scraper relied on hard-coded XPaths tied to Google's obfuscated CSS classes
(e.g. TIHn2, DUwDvf lfPIob). Google rotates these frequently, which silently
breaks extraction. This PR reworks the core to be resilient and adds several
commonly-needed fields.

What changed

  • Selector fallback layer — new safe_text / safe_attr helpers try a
    prioritized list of CSS + XPath selectors and return the first valid match,
    so a single class change no longer breaks a field.
  • Consent / GDPR handling — pre-seeds CONSENT/SOCS cookies and adds a
    dismiss_consent fallback, preventing the scraper from getting stuck on
    consent.google.com.
  • Direct search-URL navigation — navigates straight to /maps/search/...
    instead of typing into the search box (more stable, fewer race conditions).
  • Smarter result loadingscroll_results detects the end-of-list marker
    and stops on stale rounds instead of fixed sleeps.
  • New fieldslatitude, longitude, plus_code, and google_maps_url
    parsed from the place URL.
  • Single-result handling — detects and extracts a direct one-place hit.
  • QoL--headless flag and utf-8-sig CSV output (correct Unicode in Excel).

Notes

  • Backward compatible with the existing CLI flags; no breaking changes.
  • DOM selectors will still need occasional updates as Google evolves, but the
    fallback layer makes failures degrade gracefully instead of breaking outright.

Hasnain190 and others added 3 commits October 29, 2025 21:20
modified: added conditional logic for continuous check to match all the searches,
modified requirements.txt to install latest packages for python 3.13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants