The following exception block catches all exceptions without logging anything, making it incredibly difficult to debug the library. For example, bs4 fails to parse the HTML because of an optional lxml parser dependency, but the exception is never raised and the library fails to crawl silently.
|
) |
|
except: |
|
child_urls = [] |
|
page_type = "other" |
|
|
The following exception block catches all exceptions without logging anything, making it incredibly difficult to debug the library. For example,
bs4fails to parse the HTML because of an optionallxmlparser dependency, but the exception is never raised and the library fails to crawl silently.webtranspose/src/webtranspose/crawl.py
Lines 174 to 178 in 61d207a