Make Europeana fetch viable

[TimidRobot]

Problem

The Europeana data set is too big to fetch all at once, or in its entirety.

• data providers: 4206
• rights: 64

Assuming approximately 1 second per query, the expected duration is approximately 75 hours.

Description

1. Add counts for rights only (not including data providers)
2. Write a query plan on first run per quarter to map out which queries to make over 30 runs (30 days)

Alternatives

Parallel queries (probably considered impolite, Timid Robot will ask)

Additional context

• https://github.com/creativecommons/quantifying/blob/main/sources.md#europeana

Implementation

• I would be interested in implementing this feature.

[oree-xx]

@TimidRobot This is interesting to see that Europeana has a lot of data providers.
But why is the status of the issue blocked?
Are you trying to identify or investigate something?

[TimidRobot]

@TimidRobot This is interesting to see that Europeana has a lot of data providers. But why is the status of the issue blocked? Are you trying to identify or investigate something?

Yes

Parallel queries (probably considered impolite, Timid Robot will ask)

[oree-xx]

@TimidRobot Is there a verdict yet on whether parallel queries can be considered?

[TimidRobot]

@TimidRobot Is there a verdict yet on whether parallel queries can be considered?

This issue will be updated when there is more information.

[TimidRobot]

TODO:

1. Contact Europeana to either:
   • allow parallel API queries for authenticated account (allow parallel queries)
   • update API to provide totals (so fewer queries are required)
2. Update script, as necessary
3. Rename script and make it executable