Fallback to GET for 500 responses.

[Pigankle]

django-linkcheck/linkcheck/models.py

Line 413 in c13ea89

if response.status_code in [HTTPStatus.BAD_REQUEST, HTTPStatus.METHOD_NOT_ALLOWED]:

Some servers return 500 to HEAD requests but 200 to GET. The current fallback only retries with GET on 400 and 405. This adds 500 to that list so those URLs will be rechecked (and not incorrectly marked as broken.)
An example of such a site:
https://swefc.unm.edu/home/resources/?_sft_topic=water-conservation

# Fails (HEAD request - what linkcheck uses)
curl -I "https://swefc.unm.edu/home/resources/?_sft_topic=water-conservation"

# Succeeds (GET request - what browsers use)
curl -L "https://swefc.unm.edu/home/resources/?_sft_topic=water-conservation"

I think the fix would be changing line 413 from

            if response.status_code in [HTTPStatus.BAD_REQUEST, HTTPStatus.METHOD_NOT_ALLOWED]:

to

            if response.status_code in [HTTPStatus.BAD_REQUEST, HTTPStatus.METHOD_NOT_ALLOWED, HTTPStatus.INTERNAL_SERVER_ERROR]:

[Pigankle]

I have a fork with that single change in it - if you like it I can PR it.

[claudep]

Please do!

[Pigankle]

#225