Can add some simple optimizations to only try direct fetch of pages with certain known extensions, otherwise, assume HTML.
This avoids the extra fetch that happens for every page (and may contribute to rate limits).