Currently, when crawling with multiple seeds with scopeType: host, it is possible that some pages that are in scope when considering all seeds at once may be rejected as out of scope when checking against only a single seed.
For example, with example-A.com and example-B.com, if example-A.com/ links to example-B.com/some/page.html and example-B.com links to example-A.com/some/page.html, these pages won't be included because the function that checks if pages are in-scope only considers the seed they were discovered from.
These URLs are added to the seen list when they are determined out of scope, so even if example-B.com later finds example-B.com/some/page.html, that page will have already been marked as seen and won't be added to the crawl.
We should swap the default behavior of scope checks to see if a URL is in scope _for the crawl across all possible seeds, rather than only checking one seed at a time.
If there are instances where checking scope per-seed may still be necessary or useful, we could add an option to keep the current behavior. At this point, however, it's unclear is there is a strong supporting use case for per-seed scope checks within a multi-seed crawl.
Currently, when crawling with multiple seeds with
scopeType: host, it is possible that some pages that are in scope when considering all seeds at once may be rejected as out of scope when checking against only a single seed.For example, with
example-A.comandexample-B.com, ifexample-A.com/links toexample-B.com/some/page.htmlandexample-B.comlinks toexample-A.com/some/page.html, these pages won't be included because the function that checks if pages are in-scope only considers the seed they were discovered from.These URLs are added to the
seenlist when they are determined out of scope, so even ifexample-B.comlater findsexample-B.com/some/page.html, that page will have already been marked as seen and won't be added to the crawl.We should swap the default behavior of scope checks to see if a URL is in scope _for the crawl across all possible seeds, rather than only checking one seed at a time.
If there are instances where checking scope per-seed may still be necessary or useful, we could add an option to keep the current behavior. At this point, however, it's unclear is there is a strong supporting use case for per-seed scope checks within a multi-seed crawl.