feat(#2075): migrate marts__combined_course_enrollment_detail to dimensional models#2314
Draft
quazi-h wants to merge 2 commits into
Draft
feat(#2075): migrate marts__combined_course_enrollment_detail to dimensional models#2314quazi-h wants to merge 2 commits into
quazi-h wants to merge 2 commits into
Conversation
…nsional models - Add courserun_upgrade_deadline to dim_course_run (MITxOnline only; null for other platforms) - Add enrollment_is_edx_enrolled to tfact_enrollment (per-platform: true for edxorg/residential, from source field for mitxonline/mitxpro, null for program/bootcamps) - Rewrite marts__combined_course_enrollment_detail to source from dimensional layer: - tfact_enrollment + dim_course_run + dim_course + dim_user replace 6-way UNION ALL - Grade data from tfact_grade (joined on user_fk + courserun_fk) - Certificate join uses user_email (from dim_user) as cross-table key - edX.org MITxOnline certificate cross-lookup uses dim_user.user_mitxonline_username - Bootcamps order join changed from username-based to user_id-based - MITxPro order join bridged via supplemental ecommerce_order_id CTE - edX.org MicroMasters order join uses dim_user.micromasters_user_id - All 38 output columns preserved and verified via systematic diff Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These changes are now in a separate prerequisite PR: #2317 This branch now contains only the mart rewrite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
Closes #2075 (mart migration — depends on prerequisite PR #2317)
Description (What does it do?)
Migrates
marts__combined_course_enrollment_detailfrom intermediate combined models (int__combined__courserun_enrollments,int__combined__course_runs,int__combined__users) to the dimensional layer.The 6-way per-platform UNION ALL is replaced by a single flat query joining dimensional models:
tfact_enrollment(filtered toenrollment_type = 'course') — enrollment base +enrollment_is_edx_enrolled(added in feat(#2075): add courserun_upgrade_deadline to dim_course_run and enrollment_is_edx_enrolled to tfact_enrollment #2317)dim_course_run(filtered tois_current = true) — course run attributes +courserun_upgrade_deadline(added in feat(#2075): add courserun_upgrade_deadline to dim_course_run and enrollment_is_edx_enrolled to tfact_enrollment #2317)dim_course—course_titleandcourse_readable_id(viadim_course_run.course_fk)dim_user— user demographics and all platform-specific user IDs/usernamestfact_grade— grade data joined onuser_fk + courserun_fkint__combined__courserun_certificates— certificate data (ecommerce order CTEs retained from int__ models)Notable join strategy decisions:
user_email(fromdim_user) as the join key for all platforms. Bootcamps username is not stored indim_user, so email is the reliable cross-table key.int__micromasters__usersCTE; now usesdim_user.user_mitxonline_usernamedirectly, eliminating that CTE.dim_user.micromasters_user_idreplacing themicromasters_usersCTE user_id lookup.user_username-based touser_id-based (dim_user.bootcamps_application_user_id = bootcamps_completed_orders.order_purchaser_user_id), since Bootcamps username is not indim_user.mitxpro_enrollment_ordersCTE sourcingecommerce_order_idfromint__mitxpro__courserunenrollments(not yet intfact_enrollment).tfact_gradejoined onuser_fk + courserun_fk— clean dimensional join, no per-platform grade CTEs needed.All 38 output columns preserved and verified via systematic diff against the original final SELECT.
How can this be tested?
Prerequisite: #2317 must be deployed first (with
--full-refreshon both dimensional models).Additional Context
Deployment order (no breaking change if followed):
dim_course_run --full-refreshandtfact_enrollment --full-refreshmarts__combined_course_enrollment_detailnormally (no full-refresh needed)Downstream consumers (dashboards, reports) using
courserun_upgrade_deadlineorcourserunenrollment_is_edx_enrolledwill see no regression as long as step 1 completes before step 3.