Task rate limits

[mbertrand]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/11394

Description (What does it do?)

Adds a cooldown_task decorator that makes use of redis lock keys to prevent a task from running more than once every n seconds (3600 for get_mit_edx_data, 900 for get_xpro_data, get_oll_data, and get_mitxonline_data).

How can this be tested?

• docker compose up

• Run ./manage.py backpopulate_mitxonline_data

• Try running it again afterward. You should see this in the logs:

  Skipping ratelimit:learning_resources.tasks.get_mitxonline_data: rate-limited (900s)

  And the command should finish quickly:

  Population of MITX Online data finished, took 0.041112 seconds
  Task skipped (rate-limited). See celery logs for details.

• Try again with --force. This time it should process normally:

  Population of MITX Online data finished, took 77.92632 seconds
  Populated 268 resources. See celery logs for details.
  

Additional Context

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

Adds a reusable task rate-limiting decorator (backed by the configured Redis cache) and applies it to several ETL Celery tasks to prevent repeated invocations within a fixed time window.

Changes:

• Introduces rate_limited_task(wait_time, key=None) in main/decorators.py using an atomic cache.add on the redis cache.
• Adds unit tests covering rate-limiting behavior and key selection.
• Applies the decorator to selected ETL tasks in learning_resources/tasks.py (3600s for MIT edX; 900s for MITxOnline/xPro/OLL).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
main/decorators.py	Adds the Redis-cache-backed rate limit decorator used to skip frequent task invocations.
main/decorators_test.py	Adds tests for rate-limiting behavior and cache key selection.
learning_resources/tasks.py	Wraps several ETL tasks with the new rate-limiting decorator.

[rhysyngsun]

Generally looking good, I'll test tomorrow

[rhysyngsun]

I'm going to sleep on it overnight but I'm a little concerned that letting the task succeed is going to cause confusion later when tasks appear to be successful but they aren't actually getting run.

[rhysyngsun]

I think we'll just leave it as-is and this should come up in the logs if anyone investigates.

[mbertrand]

I think I might have just come up with a way to make it more obvious in the celery monitoring dashboard - raise Reject so the status is REJECTED instead of SUCCEEDED - I think that should not raise a sentry error (which is what this is intended to avoid). And the mgmt commands should probably override the cooldown by default without needing to supply --force - what do you think of that idea?

[mbertrand]

I went ahead and pushed those changes in a new commit, can revert if you prefer the original approach.

[rhysyngsun]

LGTM