Ingest contentfile archives for all runs (two-tier indexing)

[mbertrand]

What are the relevant tickets?

Closes mitodl/hq#11872

Description (What does it do?)

Ingests and embeds every run's contentfiles into Qdrant, not just the best published run.

• Qdrant (powers the VideoGPT bot, TutorBot, vector_content_files_search endpoint): should now hold contentfiles from all runs of published/test-mode courses, so any run's content is findable.
• OpenSearch (resource search): unchanged — should still only contain contentfiles from the best published run (or any published run of a test_mode course), as children or nested documents of resources.

In this branch, all runs, both published and unpublished, of a published/test_mode course are loaded into the DB and embedded into Qdrant. Unpublishing a run removes that run's contentfiles from OpenSearch without flipping ContentFile.published, keeping it in Qdrant and the REST API. Ditto for an published course that is later unpublished. Only deleting a course/run will remove its contentfiles from Qdrant.

Opensearch behavior should be unaffected (still only contentfiles from best published run should be indexed, and deindexed if that course/run is unpublished).

How can this be tested?

Pick a mitxonline course that has both published and unpublished runs, then backpopulate its files. A good candidate is course-v1:MITxT+15.356.2x (3 runs with about 67 content files each).

./manage.py backpopulate_mitxonline_files --resource-ids <resource_id> --overwrite

Then verify:

• Qdrant / vector_content_file_search contains results from all runs of the course (published and unpublished):
  http://open.odl.local:8065/api/v0/vector_content_files_search/?run_readable_id=course-v1%3AMITxT%2B15.356.2x%2B2T2023
  http://open.odl.local:8065/api/v0/vector_content_files_search/?run_readable_id=course-v1%3AMITxT%2B15.356.2x%2B3T2025
  http://open.odl.local:8065/api/v0/vector_content_files_search/?run_readable_id=course-v1%3AMITxT%2B15.356.2x%2B3T2026

• But it returns only results from the best run if resource_id is a parameter without run_readable_id or edx_module_id:

• http://open.odl.local:8065/api/v0/vector_content_files_search/?resource_readable_id=course-v1%3AMITxT%2B15.356.2x

• OpenSearch / learning_resources_search and `content_files_search still returns contentfiles only from the best published run (plus any "runless" content file(s) for the resource):
  http://open.odl.local:8065/api/v1/content_file_search/?resource_id=<resource_id for course-v1:MITxT+15.356.2x>

• Unpublishing one run drops it from OpenSearch but keeps it in Qdrant; unpublishing the whole course works the same.

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

This PR updates the ingestion/indexing pipeline so all runs’ content files for published/test-mode courses are loaded and embedded into Qdrant, while OpenSearch continues to expose only the “best run” (or any published run for test_mode) for resource search semantics. It also adjusts unpublish/delete behavior so a run can be removed from OpenSearch without necessarily flipping ContentFile.published, keeping the run’s files available in Qdrant/REST as intended.

Changes:

• Add request-time gating in vector_content_files_search so resource-scoped queries default to best-run unless an explicit run/module filter is provided.
• Update embedding + ETL logic to process/embed content files across all runs, not just the best run.
• Extend deindexing APIs/tasks/plugins to support OpenSearch-only removal (keep_published) and align Qdrant purge behavior with run/course lifecycle events.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
vector_search/views.py	Rewrites resource-scoped content-file vector queries to best-run run filters unless a run/module filter is present.
vector_search/views_test.py	Adds tests verifying best-run restriction behavior and exceptions (test_mode, explicit run filter, no published runs).
vector_search/utils.py	Introduces best_run_ids_for_resources helper to resolve which run_ids to filter for resource-scoped queries.
vector_search/utils_test.py	Adds unit tests for best_run_ids_for_resources across course/test_mode/no-published-run scenarios.
vector_search/tasks.py	Embeds published content files across all runs for courses during bulk embedding + by-id embedding; updates healthcheck accordingly.
vector_search/tasks_test.py	Updates/expands task tests to assert embedding covers all runs and excludes unpublished content files where appropriate.
learning_resources/etl/edx_shared.py	Removes “best-run only” gating during archive sync; skips only fully retired courses; expands run lookup matching.
learning_resources/etl/edx_shared_test.py	Updates ETL tests to assert all runs processed for published courses and retired courses are excluded.
learning_resources_search/tasks.py	Extends deindex task to pass through keep_published to the indexing API.
learning_resources_search/tasks_test.py	Updates task test expectation to include keep_published=False default.
learning_resources_search/plugins.py	Adjusts plugin hooks so Qdrant embeds all loaded runs; OpenSearch indexes only best-run/test_mode rules; adds keep_published behavior on run unpublish.
learning_resources_search/plugins_test.py	Adds coverage for new lifecycle semantics (keep Qdrant on run-unpublish for published course; purge on full retirement; delete purges Qdrant).
learning_resources_search/indexing_api.py	Adds keep_published option to deindex run content files without flipping ContentFile.published.
learning_resources_search/indexing_api_test.py	Adds test verifying OpenSearch deindex without flipping published when keep_published=True.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

[shanbady]

changes here look good and function as needed.

some lingering concerns i had (but not something to address here):

• how many new records will qdrant have to handle? (we are close to 5 million points on the contentfiles collection) - it might make sense to switch to 1024 for chunk size instead of 512 if this becomes an issue (to reduce the size of the collection)
• performance concerns related to the above
• growing complexity around all the conditionals for what gets ingested/visible etc

[shanbady]

looks good - just minor nit with the name "was_published" - maybe "previously_published" might be more clear