Complete Carpentries Materials Scrapers

[njall]

There are three well-established Carpentries; Software, Data, and Library; and some nascent/in-development ones.

All carpentries events are included in TeSS, but none of their extensive and useful training materials are. Ideally we would like to automatically scrape them.

Work with Francois Michonneau francois@carpentries.org to devise some form of structured data that can be rolled out across each carpentries repository.

I have worked with Christopher Erdmann of Library Carpentries to create this metadata template that might be useful if roles out across all materials.

https://github.com/LibraryCarpentry/librarycarpentry.github.io/blob/master/_data/lessons.yml

Current state of Carpentry scraping

• Software Carpentry Events - is scraped via ICS feed https://software-carpentry.org/workshops.ics
• Software Carpentry Materials - Is now broken. We used to parse yaml files from the github repo - but that repo has since become legacy.

---

• Data Carpentry Events - is scraped via JSON feed https://raw.githubusercontent.com/datacarpentry/datacarpentry.github.io/master/_data/dc_upcoming_workshops.json
• Data Carpentry Materials - Is now broken. It used to parse markdown descriptions on lessons

---

• Library Carpentry Events - is scraped via workshop yaml https://raw.githubusercontent.com/LibraryCarpentry/librarycarpentry.github.io/master/_data/workshops.yml
• Library Carpentry Materials - Never scraped