Skip to content

fix: handle retrying and non-200 status codes when downloading advisories #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
G-Rath merged 2 commits into from
Aug 18, 2025
Merged

fix: handle retrying and non-200 status codes when downloading advisories #124

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2638cd4
fix: properly error if a non-200 response happens when downloading ad…
G-Rath Aug 13, 2025
c93774b
fix: retry a url once before giving up when downloading advisories
G-Rath Aug 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions scripts/download_sa_advisories.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

import json
import os
import time
import typing

import requests
Expand Down Expand Up @@ -49,12 +50,28 @@ def download_sa_advisories_from_rest_api(last_modified_timestamp: int) -> None:

print(f'fetching sa advisories modified after {last_modified_timestamp}')
url = 'https://www.drupal.org/api-d7/node.json?type=sa&sort=changed&direction=DESC&field_is_psa=0'
retry = True
while url != '':
print(f'fetching {url}')
response = requests.get(url, headers={'user-agent': user_agent})

# if we're making too many requests and have not already retried the current
# url, wait the requested number of seconds before doing a retry
if retry and response.status_code == 429:
retry = False # just give up if we get told to back off again
seconds = int(response.headers.get('Retry-After', 0))
print(f' |* (waiting {seconds} seconds before retrying)')
time.sleep(seconds)
continue

if response.status_code != 200:
print(f'X API responded {response.status_code}')
break
raise Exception(f'unexpected {response.status_code} response when fetching {url}')

# allow (re)retrying for future requests
retry = True

# grab each advisory from the response and write it to disk
# before then continuing to the next page (if there is one)
data: drupal.ApiResponse[drupal.Advisory] = response.json()
url = data.get('next', '').replace('api-d7/node?', 'api-d7/node.json?')
for item in data['list']:
Expand Down