feat(dimensional): add semester + passing_grade to dim_course_run from MicroMasters exam runs by quazi-h · Pull Request #2319 · mitodl/ol-data-platform

quazi-h · 2026-06-17T19:52:51Z

What are the relevant tickets?

Partially addresses #2088

Scope note: This PR only adds semester and passing_grade to dim_course_run as the dimensional
source of truth. The mart (marts__micromasters_dedp_exam_grades) is not migrated in this PR — it
still reads examrun_semester/examrun_passing_grade from intermediate models. Fully migrating the
mart requires MicroMasters exam grade data to be added to tfact_grade (the larger remaining work in
epic #2072), tracked as a follow-on to close #2088.

Description (What does it do?)

Adds two nullable columns to dim_course_run:

semester (varchar) — academic term identifier for a proctored exam run (e.g. 2T2022)
passing_grade (double) — passing score threshold set per exam run

Sourced from stg__micromasters__app__postgres__exams_examrun via a new micromasters_examruns CTE, left-joined onto MITxOnline course runs on courserun_readable_id = examrun_readable_id. All other platforms get NULL. Both fields are included in the SCD2 change-detection predicate (using -1.0 as sentinel for passing_grade to distinguish NULL from an actual 0.0 value).

How can this be tested?

ol-dbt run --target dev_qa --select dim_course_run

-- Should return rows for MITxOnline proctored exam runs
select semester, passing_grade, count(*)
from dim_course_run
where semester is not null
group by 1, 2

Additional Context

semester and passing_grade belong on dim_course_run (not tfact_grade) because they describe the exam run configuration, not the individual grade event.

Copilot

Pull request overview

This PR extends the dimensional dim_course_run table with MicroMasters-derived proctored exam metadata (semester, passing_grade) and begins migrating the MITxOnline portion of the DEDP proctored exam grades mart to use dimensional models (tfact_*/dim_*) instead of int__*.

Changes:

Add semester and passing_grade columns to dim_course_run, sourced (for MITxOnline only) from stg__micromasters__app__postgres__exams_examrun, and include them in the SCD2 change-detection predicate.
Rewrite the MITxOnline branch of marts__micromasters_dedp_exam_grades to use tfact_grade + dim_course_run + dim_course + dim_user, with dedup via an anti-join against the MicroMasters branch.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
`src/ol_dbt/models/dimensional/dim_course_run.sql`	Adds `semester`/`passing_grade` to the course-run SCD2 dimension via a MicroMasters examrun left join, and updates incremental change detection + expiration logic.
`src/ol_dbt/models/marts/micromasters/marts__micromasters_dedp_exam_grades.sql`	Migrates the MITxOnline branch to dimensional sources and replaces prior join-based dedup/email-bridge logic with a NOT EXISTS anti-join and `dim_user.email`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…partially migrate marts__micromasters_dedp_exam_grades - Add micromasters_examruns CTE to dim_course_run sourced from stg__micromasters__app__postgres__exams_examrun; left-join to mitxonline_courseruns on courserun_readable_id = examrun_readable_id to populate semester and passing_grade for proctored exam runs. - Add nullable semester (varchar) and passing_grade (double) columns to all non-mitxonline platform CTEs and thread through to final SELECT and records_to_expire for SCD2 correctness. - Include semester and passing_grade in the incremental not-exists change-detection predicate so exam run config corrections trigger new SCD2 records. - Rewrite MITxOnline branch of marts__micromasters_dedp_exam_grades to use tfact_grade + dim_course_run + dim_user instead of int__mitxonline__proctored_exam_grades and int__micromasters__users. - MicroMasters branch retained on int__micromasters__dedp_proctored_exam_grades pending MicroMasters being added to tfact_grade (epic #2072). - user_micromasters_email cast to NULL in MITxOnline branch pending dim_user exposing platform-specific emails (epic #2072). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Restore dim_course_run.sql to original single-quote/comma-leading style so PR diff against main shows only the ~27 functional added lines - Fix coalesce(passing_grade, 0.0) -> coalesce(passing_grade, -1.0) in SCD2 change-detection predicate to distinguish NULL from a real 0.0 value - Add YAML column docs for semester and passing_grade in _dim_course_run.yml Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The mart's course_run_dim CTE referenced dim_course_run.semester and dim_course_run.passing_grade which don't exist in the production table until this PR is deployed, causing COLUMN_NOT_FOUND on dbt build. Additionally, joining dim_course_run for the MicroMasters branch would produce null semester for ~9,987 older edxorg-era exam run rows (where the examrun_readable_id maps to an edxorg platform row in dim_course_run that has semester=null), breaking the not_null test on semester. Both int models already carry semester and passing_grade directly, so removing the dim join loses no data. The value of this PR is Part 1: making semester/passing_grade available on dim_course_run for future consumers — the mart migration itself is deferred to the follow-up work in epic #2072. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tional changes in this PR) The mart is unchanged from main — semester/passing_grade continue to source from int models. The dim_course_run additions in this PR are for future dimensional consumers; the mart migration is deferred to epic #2072 follow-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

KatelynGit

Build completed without errors now looks good. But may need to look over copilot comments

…cs and null test - Add QUALIFY ROW_NUMBER() dedup to micromasters_examruns CTE to prevent 1:N fan-out if multiple exam run records share the same examrun_readable_id; picks the most recently updated row per readable ID. - Fix semester column description in _dim_course_run.yml: value format is '2T2022' (term identifier), not 'Fall 2023' (human-readable label). - Remove not_null test on semester in _marts_micromasters__models.yml: MITxOnline exam grade rows without a corresponding MicroMasters exam run record will have semester = NULL, so the constraint was incorrect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

…no compatibility Trino does not support the QUALIFY clause. Replace the micromasters_examruns dedup with the equivalent ROW_NUMBER() inner-subquery pattern, consistent with the existing pattern documented and used in dim_course.sql. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add examrun_id desc as secondary ORDER BY key so the ROW_NUMBER dedup is deterministic when two exam run rows share the same examrun_updated_on timestamp, preventing nondeterministic semester/passing_grade selection and unnecessary SCD2 churn. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

+                        partition by examrun_readable_id order by examrun_updated_on desc nulls last, examrun_id desc
+                    ) as _row_num
+                from {{ ref("stg__micromasters__app__postgres__exams_examrun") }}
+            )
+        where _row_num = 1


Copilot AI review requested due to automatic review settings June 17, 2026 19:52

Copilot started reviewing on behalf of quazi-h June 17, 2026 19:53 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

quazi-h force-pushed the copilot/2088-dim-course-run-exam-semester-grade branch from 073c4e3 to d5e4681 Compare June 17, 2026 20:03

rachellougee reviewed Jun 18, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

KatelynGit self-assigned this Jun 24, 2026

KatelynGit reviewed Jun 24, 2026

View reviewed changes

Comment thread src/ol_dbt/models/marts/micromasters/marts__micromasters_dedp_exam_grades.sql Outdated

quazi-h and others added 2 commits June 24, 2026 15:36

quazi-h requested a review from Copilot June 24, 2026 21:49

Copilot started reviewing on behalf of quazi-h June 24, 2026 21:49 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql Outdated

Comment thread src/ol_dbt/models/dimensional/_dim_course_run.yml

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql Outdated

KatelynGit approved these changes Jun 25, 2026

View reviewed changes

quazi-h requested a review from Copilot June 30, 2026 21:06

Copilot started reviewing on behalf of quazi-h June 30, 2026 21:07 View session

Copilot AI reviewed Jun 30, 2026

quazi-h requested a review from Copilot July 1, 2026 15:21

Copilot started reviewing on behalf of quazi-h July 1, 2026 15:21 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

quazi-h and others added 2 commits July 1, 2026 11:55

quazi-h requested a review from Copilot July 1, 2026 19:12

Copilot started reviewing on behalf of quazi-h July 1, 2026 19:13 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql Outdated

style: split else branch onto separate line for consistency

6dfc8dd

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

quazi-h requested a review from Copilot July 1, 2026 21:19

Copilot started reviewing on behalf of quazi-h July 1, 2026 21:20 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

quazi-h changed the title ~~feat(dimensional): add semester + passing_grade to dim_course_run; migrate MITxOnline branch of dedp exam grades mart~~ feat(dimensional): add semester + passing_grade to dim_course_run from MicroMasters exam runs Jul 2, 2026

quazi-h requested a review from Copilot July 2, 2026 18:13

Copilot started reviewing on behalf of quazi-h July 2, 2026 18:14 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

quazi-h requested a review from Copilot July 2, 2026 19:47

Copilot started reviewing on behalf of quazi-h July 2, 2026 19:47 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

Comment on lines +24 to +28

partition by examrun_readable_id order by examrun_updated_on desc nulls last, examrun_id desc

) as _row_num

from {{ ref("stg__micromasters__app__postgres__exams_examrun") }}

)

where _row_num = 1

Uh oh!

Conversation

quazi-h commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What are the relevant tickets?

Description (What does it do?)

How can this be tested?

Additional Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KatelynGit left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

quazi-h commented Jun 17, 2026 •

edited

Loading