discoverValidSitemaps (from Crawlee) sends network requests to robots & sitemap files without timeouts.
If any of those requests hangs before a response, the promise never resolves -> the async iterator stalls -> crawler startup is blocked.
Need to implement timeout-workaround until it's fixed in Crawlee
discoverValidSitemaps (from Crawlee) sends network requests to robots & sitemap files without timeouts.
If any of those requests hangs before a response, the promise never resolves -> the async iterator stalls -> crawler startup is blocked.
Need to implement timeout-workaround until it's fixed in Crawlee