Performance optimization for multi-country market forecasting with rate limiting - seeking guidance

[methunraj]

We're using gdeltdoc for a market forecasting application that analyzes news sentiment across 50+ countries with 15-20 search topics. We need to respect the 5-second rate limit to avoid IP
bans, but this creates significant performance challenges.

Current Implementation

• Processing 50 countries × 15 topics (in batches of 3)
• Each country requires 5 batches (15÷3)
• With 5-second rate limiting: 5 batches × 5 seconds = 25 seconds per country
• Total time: 50 countries × 25 seconds = 1,250 seconds (21 minutes minimum)
• With random 5-10 second delays for safety: ~35-40 minutes

Code Structure

# Current approach - sequential due to rate limiting
for country in countries:
    for topic_batch in topic_batches:  # 3 topics per batch
        time.sleep(random.uniform(5, 10))  # Rate limit
        results = gdelt.article_search(filters)

## Question

Is there a recommended approach to optimize performance while respecting rate limits? Specifically:

1. Can we make parallel requests for different countries? (e.g., 4 different country queries simultaneously, each respecting 5-second spacing)
2. Is there a higher rate limit available for academic/commercial use cases?
3. Are there any best practices for bulk country-topic analysis that we should follow?
4. Would caching be acceptable for 24-hour periods to avoid repeated queries?

## Use Case

Market forecasting system analyzing news sentiment impact on infrastructure/technology markets across global regions. Need to balance comprehensive coverage with reasonable processing time.

Any guidance on optimizing this workflow while being a responsible API user would be greatly appreciated!

## Environment

• Python 3.x
• gdeltdoc latest version
• ThreadPoolExecutor for parallel processing (currently serialized due to global rate limiter)

[alex9smith]

Please note - this is an unofficial API client package and I've not ever been employed by or had any professional relationship with GDELT. I don't have access to any information other than what's on the public website.

I'll give my best answers to your question below but in general I would recommend the Big Query export over repeated API calls for this sort of large scale analysis. If you can't get what you need from Big Query then all the 'normal' practices for scaling up ETL jobs will apply. You need to split your queries up from running on a single machine to multiple workers with different IP addresses. It seems like you're building a commercial product here - do you already have a preferred data warehouse / workflow orchestration tool like Airflow running in a cloud platform?

You could also try changing the user agent set by this package but I'm not going to provide a feature to do that - in my opinion GDELT provide us an excellent free product and the rate limit is part of what allows them to keep running it.

1. Can we make parallel requests for different countries? (e.g., 4 different country queries simultaneously, each respecting 5-second spacing)

I think this will still mean you get rate limited.

2. Is there a higher rate limit available for academic/commercial use cases?

There are no paid API tiers.

3. Are there any best practices for bulk country-topic analysis that we should follow?

Use the Big Query export as I mentioned above if it supports your use case.

4. Would caching be acceptable for 24-hour periods to avoid repeated queries?

Yes. In my experience data older than 15 minutes ago doesn't change so you can (and should) cache or just save data to a database.