feat(ingest): add YouTube dlt source to ol_dlt by blarghmatey · Pull Request #2355 · mitodl/ol-data-platform

blarghmatey · 2026-07-01T13:02:06Z

What are the relevant tickets?

N/A (part of the MIT Learn ETL → Data Platform migration, Cohort 3 media/feed sources — epic https://github.com/mitodl/hq/issues/11512)

Description (What does it do?)

Migrates MIT Learn's YouTube catalog ETL (learning_resources/etl/youtube.py) onto the data platform as a new pure-dlt source, so channel/playlist/video/transcript extraction is observable and backfillable in Dagster instead of opaque in the Django app.

New source src/ol_dlt/ol_dlt/sources/youtube/ — reads per-channel YAML configs from mitodl/open-video-data (youtube/*.yaml) and calls the YouTube Data API v3 directly over requests (API-key auth via the key param), consistent with the other ol_dlt sources — no google-api-python-client SDK. Follows the restructured pattern: pure dlt (no Dagster imports), shared ol_dlt.config for destination/dataset/table_format (config.pipeline_for("youtube"), config.active_table_format()) and lazy credentials (config.resolve_secret / require_secrets), exposing youtube_source(), youtube_pipeline, and build_source().
Four merge-disposition tables: raw__youtube__api__channels, raw__youtube__api__playlists, raw__youtube__api__videos, raw__youtube__api__transcripts. Playlist resolution supports explicit playlist ids, the all wildcard (lists every playlist on the channel), and ignore flags; video ids are collected via playlistItems pagination and deduplicated across playlists.
Transcripts are fetched with youtube-transcript-api (added as a dependency, >=1.0,<2), imported lazily. Coded against the 1.x instance API (YouTubeTranscriptApi().fetch() → FetchedTranscript → .to_raw_data(); TextFormatter().format_transcript()) — note this is a breaking change from the 0.6.x classmethod API that MIT Learn currently uses. Videos with transcripts disabled/unavailable are skipped, not raised.
Dagster wiring (dg_projects/data_loading/data_loading/defs/ingestion/): registered as @dlt_assets via build_ingest_assets; the default RawDataDltTranslator maps the tables to ["ol_warehouse_raw_data", …] under group youtube. Added youtube_ingest_daily_schedule (30 3 * * *).

How can this be tested?

Automated (no AWS/YouTube credentials needed):

# ol_dlt source unit + materialization tests (runs the real dlt pipeline
# against an ephemeral tmp filesystem profile)
cd src/ol_dlt && uv run pytest            # 52 passed (14 new for youtube)

# data_loading code location loads the new assets + schedule
cd dg_projects/data_loading && uv run dg check defs
uv run dg list defs | grep youtube        # 4 raw__youtube__api__* assets + youtube_ingest_daily_schedule

A live end-to-end run against the real YouTube Data API has not been performed — it needs YOUTUBE_DEVELOPER_KEY and S3/Glue credentials in a deployed environment.

Additional Context

YOUTUBE_DEVELOPER_KEY must be provided to the data_loading deployment environment for the assets to materialize.
The raw tables land the ingest/warehouse layer only; MIT Learn delivery (webhook) is a separate follow-up task in Cohort 3.

Migrates MIT Learn's YouTube catalog ETL to the data platform as a pure-dlt source in src/ol_dlt, so channel/playlist/video/transcript extraction is observable and backfillable in Dagster rather than opaque in the Django app. Reads per-channel configs from mitodl/open-video-data and calls the YouTube Data API v3 over plain requests (consistent with the other ol_dlt sources) plus youtube-transcript-api for transcripts, landing raw__youtube__api__{channels,playlists,videos,transcripts}. Wired into the data_loading code location as @dlt_assets with a daily schedule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Adds a new ol_dlt YouTube source (plus Dagster data_loading wiring) to migrate MIT Learn’s YouTube catalog extraction into an observable/backfillable dlt pipeline, including transcripts via youtube-transcript-api.

Changes:

Introduces ol_dlt.sources.youtube (GitHub YAML config → YouTube Data API v3 → 4 raw tables) with standalone runner + README.
Wires the source into dg_projects/data_loading via build_ingest_assets and adds a daily schedule targeting the youtube asset group.
Adds youtube-transcript-api dependency (and lockfile updates) and adds unit/materialization tests for the YouTube source.

Reviewed changes

Copilot reviewed 7 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/ol_dlt/uv.lock	Locks new dependency (`youtube-transcript-api` + `defusedxml`).
src/ol_dlt/pyproject.toml	Adds `youtube-transcript-api>=1.0,<2` to `ol_dlt` deps.
src/ol_dlt/ol_dlt/sources/youtube/README.md	Documents the new YouTube dlt source and config expectations.
src/ol_dlt/ol_dlt/sources/youtube/main.py	Adds a simple standalone smoke-run entrypoint.
src/ol_dlt/ol_dlt/sources/youtube/init.py	Implements the YouTube dlt source, resources, and pipeline entrypoints.
src/ol_dlt/tests/sources/test_youtube.py	Adds unit + materialization tests for the YouTube source.
dg_projects/data_loading/uv.lock	Updates locks due to `ol_dlt` dependency change.
dg_projects/data_loading/data_loading/defs/ingestion/schedules.py	Adds `youtube_ingest_daily_schedule` selecting the `youtube` group.
dg_projects/data_loading/data_loading/defs/ingestion/assets.py	Registers `youtube_assets` via `build_ingest_assets`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

blarghmatey · 2026-07-01T13:15:32Z

+        try:
+            channel_config = yaml.safe_load(raw_content)
+        except yaml.YAMLError:
+            logger.exception("Failed to parse YAML config: %s", file_meta["name"])
+            continue
+
+        if not isinstance(channel_config, dict):
+            logger.warning("Skipping non-dict youtube config: %s", file_meta["name"])
+            continue
+
+        if "channel_id" not in channel_config:
+            logger.warning(
+                "Skipping config missing required key (channel_id): %s",
+                file_meta["name"],
+            )
+            continue
+
+        configs.append(channel_config)


Confirmed and fixed in 8f53aee. I verified the real repo: youtube/channels.yaml and shorts.yaml are YAML lists of channel dicts. _fetch_channel_configs now flattens list-format files (and still accepts a single dict). Added test_fetch_channel_configs_parses_yaml_list.

blarghmatey · 2026-07-01T13:15:36Z

+**Config source:** [mitodl/open-video-data](https://github.com/mitodl/open-video-data)
+(`youtube/*.yaml`, one file per channel). Each config provides a `channel_id`
+and an optional `playlists` list (`{id, ignore?}`). A config with no `playlists`,
+or one containing the `all` wildcard id, ingests every playlist on the channel.


Fixed in 8f53aee — README now documents the real format (list-based channels.yaml/shorts.yaml on main) with an example entry.

blarghmatey · 2026-07-01T13:15:39Z

+            json_data=[{"name": "c.yaml", "url": "https://api/file/c.yaml"}]
+        )
+    if "api/file" in url:
+        yaml_bytes = b"channel_id: CHAN\noffered_by: ocw\n"


Fixed in 8f53aee — _fake_channel_get now serves a list-format channels.yaml, and a new test_fetch_channel_configs_parses_yaml_list guards the list-vs-dict regression directly.

Address PR review: the mitodl/open-video-data youtube/*.yaml files (channels.yaml, shorts.yaml) are YAML *lists* of channel dicts on the `main` branch, not one dict per file on `master` — the previous parser loaded zero configs against the real repo. Parse both list and single-dict files, default github_branch to "main", and accept bare-string playlist entries so `playlists: [all]` still triggers the wildcard instead of silently ingesting nothing. Update the README format docs and tests to match the real list-based config shape. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings July 1, 2026 13:02

Copilot started reviewing on behalf of blarghmatey July 1, 2026 13:02 View session

sentry Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread src/ol_dlt/ol_dlt/sources/youtube/__init__.py

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ingest): add YouTube dlt source to ol_dlt#2355

feat(ingest): add YouTube dlt source to ol_dlt#2355
blarghmatey wants to merge 2 commits into
mainfrom
tmacey/youtube-dlt-ingest

blarghmatey commented Jul 1, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

blarghmatey Jul 1, 2026

Uh oh!

blarghmatey Jul 1, 2026

Uh oh!

blarghmatey Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

blarghmatey commented Jul 1, 2026

What are the relevant tickets?

Description (What does it do?)

How can this be tested?

Additional Context

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

blarghmatey Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

blarghmatey Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

blarghmatey Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants