To perform multiple Stage Domain Crawls we face at the moment multiple problems:
- We can not pass Limits per seeds to browsertrix, only for the entirely crawl
- We do not know which seed is done completly and should not pass to the next stage of the domain crawls
A few words to the Stages:
In the past (Heritrix via NetArchiveSuite) we had the following settings for the first stage (Limit per domain, 10 k domains in one job):
The most seeds where hitting the 100 MB Limit and would be passed to the 2nd Stage, with the following config:
- Hops: 30
- Size: 1GB
(We had also objects, but this we don't need anymore)
We would need the parameters --size-limit-per-seed and --page-limit-per-seed to get aquivalent results with browsertrix AND, this is also important, we need to know after the Crawl which seed was crawled entirely (not hit size limit or depth neither page limit) and which seeds would have more content to capture and can be passed to the next stage
To perform multiple Stage Domain Crawls we face at the moment multiple problems:
A few words to the Stages:
In the past (Heritrix via NetArchiveSuite) we had the following settings for the first stage (Limit per domain, 10 k domains in one job):
The most seeds where hitting the 100 MB Limit and would be passed to the 2nd Stage, with the following config:
(We had also objects, but this we don't need anymore)
We would need the parameters --size-limit-per-seed and --page-limit-per-seed to get aquivalent results with browsertrix AND, this is also important, we need to know after the Crawl which seed was crawled entirely (not hit size limit or depth neither page limit) and which seeds would have more content to capture and can be passed to the next stage