Skip to content

Legislator Profile - Scrape Biographies from Legislature Site #2161

Description

@Mephistic

Summary

The Legislator pages on the Legislature's site actually have biographies:https://malegislature.gov/Legislators/Profile/RLR0/Biography - we should use those as a default for our Legislator Profile page's biography section. Given that this is relevant data for all members, not just members with MAPLE accounts, it makes the most sense to store this on the Member document itself.

I would guess they only update these once a session at most, so I think this is fine as a one-off script that will run and scrape this for all legislators. (We may eventually want this longer-term, but given that we also want to avoid overwriting a biography the legislator sets in MAPLE itself, that may be self-defeating).

Success Criteria

  • Add biography field to Members (in generalCourts/194/members)
    • (There may also eventually be an override in profiles for legislator who make accounts on MAPLE and want to customize them, but that is out of scope for this ticket)
  • Create a scraper script that:
    • Iterates over all legislators in the current court (better yet, make court configurable by a param and just default to the current general court)
    • Scrapes the biography from the html of the legislature's legislator page (found at https://malegislature.gov/Legislators/Profile/<<MEMBER_CODE>>/Biography)
    • Writes the biographies to the corresponding member in Firestore

Metadata

Metadata

Assignees

Labels

Ready for DevelopmentAutomatically assigned to issues that are ready to be picked upbackendBackend Developmentgood first issueGood for newcomersscraperBackend work related to content scraping

Type

No type
No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions