Skip to content

Implement site wide scraping limit #1530

@grossir

Description

@grossir

Meaning, we only execute the scraper on some hours of the day; maybe once or twice daily, instead of every hour
This will be a time check at init, which just aborts the scraper
This way, we can keep the hourly scraper cronjob in CL, but control individual scraper execution from juriscraper only

Part of this is already implemented here https://github.com/freelawproject/juriscraper/pull/1497/files#diff-5a1da7e0c51f215c8d638407892fd45ef750bf6dc19c8431444de3f9ceb556e7

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    PR'd Issues 🤞

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions