Mitxonline ETL: switch to consuming internal courses endpoint

[shanbady]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/11671

Description (What does it do?)

This PR allows us to be able to use the internal mitxonline courses api which includes variant and b2b courses

How can this be tested?

1. checkout this branch
2. ssh into an rc mitxonline pod and get into a django shell
3. once in the shell execute the following to create an api key:

from rest_framework_api_key.models import *
api_key, key = APIKey.objects.create_key(name="test-learn-etl-service")
print(key)

4. the key will only displayed/available once (its stored as a hash). note the api key down
5. in settings_course_etl.py set MITX_ONLINE_COURSES_API_URL = "https://api.rc.mitxonline.mit.edu/api/internal/courses/" and MITX_ONLINE_ETL_API_KEY to the api key you have just created.
6. restart celery and run the mitxonline backpopulate (optionally delete existing mitxionline courses) ./manage.py backpopulate_mitxonline_data
7. verify that mitxonline learning resources populate.
8. once done testing delete the test api key created on mitonline:

from rest_framework_api_key.models import *
APIKey.objects.get(name="test-learn-etl-service").delete()

Additional Context

1. This Pr also contains a small change to coerce the protocol used when fetching data to whatever the initial request was - this is because the mitonline api for some reason uses http for next urls.
2. when using mitxonline rc as the endpoint - you will end up with the same amount of resources as using the regular api since I dont think there are variant courses there.

merging this will be blocked on getting the actual api key secrets onto mitlearn rc/prod via https://github.com/mitodl/hq/issues/12088

[github-actions]

OpenAPI Changes

24 changes: 0 error, 9 warning, 15 info

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

This PR updates the MITx Online ETL to support consuming MITx Online’s internal courses endpoint by optionally attaching an API key, and hardens pagination behavior when the upstream API returns next URLs with an unexpected scheme.

Changes:

• Add MITX_ONLINE_ETL_API_KEY setting and send it as Authorization: Api-Key <key> for MITx Online ETL requests.
• Preserve the original request scheme (e.g., keep https) when following pagination next links.
• Add unit tests covering scheme preservation and authorization header behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
main/settings_course_etl.py	Adds a new optional env setting for the MITx Online internal endpoint API key.
learning_resources/etl/mitxonline.py	Sends the API key header (when configured) and preserves scheme across pagination.
learning_resources/etl/mitxonline_test.py	Adds test coverage for scheme preservation and auth header injection.

[mbertrand]

The API returns all published runs per course — including B2B and variant runs that v2 filtered out. _transform_course (:352) keeps all of them and publishes any run with is_enrollable AND page.live, so they surface in the catalog. (include_in_learn_catalog gates the course, not its runs.)

With RC data, published courses expose published B2B/variant published runs in API responses.

Also, these runs can impact the assignment of best_run (models.py:581-634), which picks purely by enrollment/start date with no B2B/variant awareness — some of these runs now resolve to a B2B run as their best_run , which drives the displayed dates/price/URL and gates opensearch content-file indexing, so the wrong run could be shown and indexed in opensearch.

Example: http://api.open.odl.local:8065/api/v1/learning_resources/?readable_id=course-v1%3AMITxT%2BSTL.162x

Maybe we need a new field on LearningResourceRun to differentiate B2B/variant runs and exclude them from API results and calculation of best_run?

[mbertrand]

Not sure if we want these b2b/variant runs in the API results or not, but pretty sure they should not be assigned as the best_run for published courses.

[mbertrand]

Maybe can just set these runs as always unpublished, so they get their contentfiles indexed but that's it (assuming they don't have to show up anywhere else in mit-learn).

[shanbady]

that is a good point..let me see if i can filter those out some how

[shanbady]

I added a is_b2b attribute for runs which gets set based on b2b_contract from the api

[mbertrand]

README.md:300-301 and env/codespaces.env:29-30 still point at /api/v2/courses/ — maybe worth updating to the internal endpoint (and mentioning that MITX_ONLINE_ETL_API_KEY is now required for that ETL).

[shanbady]

I was intending this to be backwards compatable (local setups default to the public endpoint ~ less configuration for default environments)

[mbertrand]

Nit: scheme is invariant across loop iterations — each rewritten url already carries the preserved scheme — so it could be captured once before the while. Not a bug, just slightly clearer.

[shanbady]

fixed

[claude]

⚠️ Code review skipped — your organization's overage spend limit has been reached.

Code review is billed via overage credits. To resume reviews, an organization admin can raise the monthly limit at claude.ai/admin-settings/claude-code.

Once credits are available, push a new commit or reopen this pull request to trigger a review.

[mbertrand]

LGTM, one question I have is if this will handle variant courses (other languages?) that are not B2B, or do those not exist?

Other than that, 1 minor nit about a trailing slash for individual course endpoint urls

[mbertrand]

Maybe add a trailing slash to avoid a 301 redirect