Background
The washdev raw data is produced by two Python scripts in inst/python/:
washdev_scraping.py uses plain HTTP requests with a hardcoded browser cookie header. The cookies are long expired and iwaponline.com sits behind Cloudflare, so this script no longer works.washdev_scraping_selenium.py drives a real Chrome browser to get past Cloudflare. It works but hardcodes a chromedriver path from another contributor's machine (inst/python/washdev_scraping_selenium.py:22) and depends on a committed 17 MB chromedriver binary that is version-locked to an old Chrome.
Neither script supports incremental updates; both re-scrape everything from volume 1 (open TODO in the code).
Proposal
Port the scraper to R so the whole package is one language:
- Use
chromote (headless Chrome) to fetch pages, mirroring what Selenium does now, and rvest to parse them. The CSS selectors and parsing logic are already worked out in the Python scripts; porting is mechanical.
- Two-stage design, same as the Python version: (1) issue table-of-contents pages to build the article overview table, (2) per-article pages for supplementary materials, author metadata, data availability statement, and keywords.
- Make it incremental: read
data-raw/washdev.csv, find the last volume/issue present, scrape only newer issues, and append.
- Fix a bug carried in the requests version:
get_issues() unconditionally raises an error for any end volume above 11 (inst/python/washdev_scraping.py:273), making the final-volume branch unreachable. The Selenium version worked around it by commenting the check out. The R port should encode the publication history correctly: volumes 1-10 have 4 issues, volume 11 has 6, volume 12 onward has 12.
Script location: data-raw/washdev_scraping.R, following the openwashdata convention of keeping acquisition code next to data_processing.R.
Alternative
Patch and rerun the Selenium script (fix the chromedriver path, download a current chromedriver, restrict to new volumes). Less new code, but keeps a Python stack that the current maintainer does not work in.
Background
The washdev raw data is produced by two Python scripts in
inst/python/:washdev_scraping.pyuses plain HTTP requests with a hardcoded browser cookie header. The cookies are long expired and iwaponline.com sits behind Cloudflare, so this script no longer works.washdev_scraping_selenium.pydrives a real Chrome browser to get past Cloudflare. It works but hardcodes a chromedriver path from another contributor's machine (inst/python/washdev_scraping_selenium.py:22) and depends on a committed 17 MB chromedriver binary that is version-locked to an old Chrome.Neither script supports incremental updates; both re-scrape everything from volume 1 (open TODO in the code).
Proposal
Port the scraper to R so the whole package is one language:
chromote(headless Chrome) to fetch pages, mirroring what Selenium does now, andrvestto parse them. The CSS selectors and parsing logic are already worked out in the Python scripts; porting is mechanical.data-raw/washdev.csv, find the last volume/issue present, scrape only newer issues, and append.get_issues()unconditionally raises an error for any end volume above 11 (inst/python/washdev_scraping.py:273), making the final-volume branch unreachable. The Selenium version worked around it by commenting the check out. The R port should encode the publication history correctly: volumes 1-10 have 4 issues, volume 11 has 6, volume 12 onward has 12.Script location:
data-raw/washdev_scraping.R, following the openwashdata convention of keeping acquisition code next todata_processing.R.Alternative
Patch and rerun the Selenium script (fix the chromedriver path, download a current chromedriver, restrict to new volumes). Less new code, but keeps a Python stack that the current maintainer does not work in.