Skip to content

Handle ReliefWeb 410 Gone during prefetch #2707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
szabozoltan69 merged 3 commits into from
Apr 5, 2026
Merged

Handle ReliefWeb 410 Gone during prefetch #2707

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e9c6001
chore: add the ssh public key
ranjan-stha Apr 2, 2026
2d148ef
Merge pull request #2706 from IFRCGo/chore/ssh-access
szabozoltan69 Apr 2, 2026
c3ab8d7
Handle ReliefWeb 410 Gone during prefetch
szabozoltan69 Apr 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 30 additions & 6 deletions databank/management/commands/sources/RELIEFWEB.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,30 @@ def parse_date(date):
return datetime.datetime.strptime(date.split("T")[0], RELIEFWEB_DATETIME_FORMAT)


def _post_reliefweb(query_params, url, context):
try:
response = requests.post(url, data=query_params)
response.raise_for_status()
return response.json()
except requests.HTTPError as exc:
status_code = exc.response.status_code if exc.response else None
if status_code == 410:
logger.warning(
"ReliefWeb API returned 410 Gone for %s. Skipping %s prefetch. URL: %s",
DISASTER_API,
context,
url,
)
return None
raise
except requests.RequestException:
logger.exception("ReliefWeb API request failed for %s. URL: %s", context, url)
raise
except ValueError:
logger.exception("ReliefWeb API returned invalid JSON for %s. URL: %s", context, url)
raise


def _crises_event_prefetch():
query_params = json.dumps(
{
Expand All @@ -42,9 +66,9 @@ def _crises_event_prefetch():
url = DISASTER_API
data = {}
while True:
response = requests.post(url, data=query_params)
response.raise_for_status()
response = response.json()
response = _post_reliefweb(query_params, url, "crises_event")
if response is None:
return {}

for disaster in response["data"]:
disaster = disaster["fields"]
Expand Down Expand Up @@ -90,9 +114,9 @@ def _epidemics_prefetch():
url = DISASTER_API
data = {}
while True:
response = requests.post(url, data=query_params)
response.raise_for_status()
response = response.json()
response = _post_reliefweb(query_params, url, "epidemics")
if response is None:
return {}

for epidemic in response["data"]:
epidemic = epidemic["fields"]
Expand Down
1 change: 1 addition & 0 deletions deploy/helm/ifrcgo-helm/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,3 +315,4 @@ sshBastion:
thenav56.pub: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN/f/A3qkaTHSdbKn8Hv75YiJvRMEXvWTDdIiR7tyAjJ navin@nav-machine
david.pub: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3FzrQdVh5Qwp5Y6KQGcpqHxKErxCW103iEECuutR/jBZe6X0xjD+cW7e+H8SrUsPQwj87fzOsMAc6v6n+3hdYFa6ekgRG/USEIUR5C/GD1Xjva3Xpp45PasBhJEtYt2ON+dlzwvRyOuv2hvqv2WHBO020ewIlVuQ4pU4Qj5ysvwWGj8GAv/jITiVERmjLTStbFwxeIDT3jQEbwnfV1zZZKiGxIecB/y51nk6oIQ00ZGrYEo5ieWsUSVfLHOX0/lZ0mtrdqxDEgMaCbNaUbICAimsJPamNpoirKc7FoKIKKrLQsK8qE1lClWQEecbW+dgSiwxracooKeWhHq+BkKUCNgEL/C0ff2l9e8sJcLmYZUdPtDCdtUDC8BAlELA5HR6tdCTfFcc0nXltclSSODMnZkQohh5/2fixJTwN5p5csEfBLzbdrturKtT/TbYSoaodg4muPqY4YE5jiJfrHVAGS1DVWz/cRcm1vOxT2V4iW2SNvo8fS2PZOpU5furrvbM= ifrcds\david.muchatiza@5CG41911S1
paola.pub: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGql4RrbxSQTW5QrTh+P+94jGCXOCeZgc23hxL9zFCYQrzL0SMw1F53Z5SFZimIhJswYPqV2pT8L4oTRqIrTCM+looWi7b9/9u+m/KmA+FWbo3u6uRrckkA3nVIKsKHvlOucX2GxE6i+tXdeXEisW49ZpMtuvxMLJ3Eg4MK10d/2d3FKuzTsrxCTlJn8FAE3yOsVow0jdu+381IrkAqRE2GINeQ87hVlQpbo+bL2N/2QZmNjDhBBQkRJLDisW0+UNgo+S9wN7HbpV5LheSJS9wGN7LlmcqlpZFrDO/lVyoMxEQ0588wUI8BVfqAZDEBJPdGtzq513r+5iXEX/9A1Mendlvxfl6ANNRcH9PVZHkRN1dxY3rckQ+Lk3qqIjjfYFYvl5Gybidb1BM2VNWHAuzaDDQzJpeTHIbQnDt7Ke4oX2xWYgyu+kVhqz0HnAV28qMXbMEsrMIrtwl7IjcrorgdduHghZvWFbaJZNtXOfgnf1IYNXkZ9eWPS+Bz9nWMhE= ifrcds\paola.yela@5CG41911RT
ranjan.pub: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGJA0ec4Gavc+m1MjEZGoUce51yWouMTRTYJZV3s/jgD rsh@rsh-XPS-15-9510
Loading