Browsertrix Host
Self-Hosted
What change would you like to see?
I would like to be able to choose if Browsertrix should use the prefetch/direct fetch or not.
The reason is we are having a lot of "Direct fetch of page URL timed out" on domains protected by WAF (Cloudflare, Azure Front Door etc.). I believe the prefetch/direct fetch is causing this as it is not normal human behavior.
Turning off the prefetch/direct fetch might make us behave more like a friendly bot thus reducing ratelimiting and blocking.
Additional details
Try crawling https://favrskov.dk/ to see what happens.
It would be great if it could be combined with some kind of domain preanalysis: If the domain uses Cloudflare or other WAFs, turn of direct fetch.
Browsertrix Host
Self-Hosted
What change would you like to see?
I would like to be able to choose if Browsertrix should use the prefetch/direct fetch or not.
The reason is we are having a lot of "Direct fetch of page URL timed out" on domains protected by WAF (Cloudflare, Azure Front Door etc.). I believe the prefetch/direct fetch is causing this as it is not normal human behavior.
Turning off the prefetch/direct fetch might make us behave more like a friendly bot thus reducing ratelimiting and blocking.
Additional details
Try crawling https://favrskov.dk/ to see what happens.
It would be great if it could be combined with some kind of domain preanalysis: If the domain uses Cloudflare or other WAFs, turn of direct fetch.