Skip to content

Failures in /api/v2/bulk_download/awards #4283

Description

@jalehman

I'm noticing frequent failures with the bulk_download endpoint when requesting ten days of data. Here's an example of a request I'm issuing (this is the response from the API, which shows the parameters):

{
   "status_url":"https://api.usaspending.gov/api/v2/download/status?file_name=All_SubawardsAndPrimeTransactions_2025-02-18_H05M23S53477012.zip",
   "file_name":"All_SubawardsAndPrimeTransactions_2025-02-18_H05M23S53477012.zip",
   "file_url":"https://files.usaspending.gov/generated_downloads/All_SubawardsAndPrimeTransactions_2025-02-18_H05M23S53477012.zip",
   "download_request":{
      "download_types":[
         "elasticsearch_sub_awards",
         "prime_awards"
      ],
      "file_format":"csv",
      "filters":{
         "prime_and_sub_award_types":{
            "elasticsearch_sub_awards":[
               "grant",
               "procurement"
            ],
            "prime_awards":[
               "-1",
               "02",
               "03",
               "04",
               "05",
               "06",
               "07",
               "08",
               "09",
               "10",
               "11",
               "A",
               "B",
               "C",
               "D",
               "IDV_A",
               "IDV_B",
               "IDV_B_A",
               "IDV_B_B",
               "IDV_B_C",
               "IDV_C",
               "IDV_D",
               "IDV_E"
            ]
         },
         "time_period":[
            {
               "date_type":"last_modified_date",
               "end_date":"2025-02-07",
               "start_date":"2025-01-28"
            }
         ]
      },
      "request_type":"award"
   }
}

Here's an example of a failed request that was made using the same parameters as above, as seen from the status URL:

{
    "status": "failed",
    "message": "An exception was raised while attempting to process the DownloadJob:\nTraceback (most recent call last):\n  File \"/data-act/backend/usaspending_api/download/filestreaming/download_generation.py\", line 99, in generate_download\n    parse_source(\n  File \"/data-act/backend/usaspending_api/download/filestreaming/download_generation.py\", line 441, in parse_source\n    raise e\n  File \"/data-act/backend/usaspending_api/download/filestreaming/download_generation.py\", line 416, in parse_source\n    wait_for_process(psql_process, start_time, download_job)\n  File \"/data-act/backend/usaspending_api/download/filestreaming/download_generation.py\", line 550, in wait_for_process\n    raise e\nException: Command failed. Please see the logs for details.\n",
    "file_name": "All_SubawardsAndPrimeTransactions_2025-02-17_H17M45S17255664.zip",
    "file_url": "https://files.usaspending.gov/generated_downloads/All_SubawardsAndPrimeTransactions_2025-02-17_H17M45S17255664.zip",
    "total_size": null,
    "total_columns": 118,
    "total_rows": 0,
    "seconds_elapsed": "26992.228396"
}

Any idea what's going on? Let me know if there's any other information I can provide. Thank you!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions