Downloads interface doesn't report failures; task stays at pending

[UnwashedMeme]

If the backend download system fails the portal keeps displaying that as Status "pending".

E.g. on a current download, the status url is requesting: https://api.idigbio.org/v2/download/f1f91566-3b1f-4478-b3ed-af32703f5cca yield response:

{
  "complete": false, 
  "core_source": "indexterms", 
  "core_type": "records", 
  "expires": "2017-01-11T12:22:30.268094", 
  "form": "dwca-csv", 
  "mediarecord_fields": null, 
  "mq": null, 
  "record_fields": null, 
  "rq": {
    "scientificname": "alligator mississippiensis"
  }, 
  "status_url": "https://api.idigbio.org/v2/download/f1f91566-3b1f-4478-b3ed-af32703f5cca", 
  "task_status": "FAILURE"
}

but the portal keeps requesting that about every 4s and the status pane in the download tab keeps saying "pending"

[UnwashedMeme]

I deleted the ID from redis so we could generate a new download request instead of just fetching the failed one. The portal keeps requesting the status, even when it is getting back a 404 from the server too.

10.5.145.178 - - [2017-01-10 14:44:53] "GET /v2/download/598debb2-c8f3-4fcf-9d07-ea3161239516 HTTP/1.1" 404 368 0.001939
10.5.145.178 - - [2017-01-10 14:44:53] "GET /v2/download/598debb2-c8f3-4fcf-9d07-ea3161239516 HTTP/1.1" 404 368 0.001086
10.5.145.178 - - [2017-01-10 14:44:53] "GET /v2/download/598debb2-c8f3-4fcf-9d07-ea3161239516 HTTP/1.1" 404 368 0.001130
10.5.145.178 - - [2017-01-10 14:44:53] "GET /v2/download/598debb2-c8f3-4fcf-9d07-ea3161239516 HTTP/1.1" 404 368 0.001067

[UnwashedMeme]

also note that the above is all in the same second; it was producing a lot of logs. I kind of suspect that when it fails with a 404 it was trying to restart the job; but then not removing the old setInterval or something which led to an increasing number of workers trying to do this