feat(#2077): migrate marts__combined__users to dim_user

[quazi-h]

What are the relevant tickets?

Closes #2077
Part of epic #2072

Description (What does it do?)

Migrates marts__combined__users from a 4-way UNION ALL over int__mitx__users + int__combined__users to use dim_user as the single canonical user identity source. This is part of the epic to migrate mart and reporting models to the dimensional layer.

What changed:

• dim_user replaces the old user identity UNIONs as the source of truth for all user-level fields
• A users_supplement CTE is added from int__combined__users to supply 5 fields not yet promoted to dim_user: user_last_login, user_street_address, user_address_city, user_address_state_or_territory, user_address_postal_code
• user_hashed_id, platforms, user_joined_on, user_is_active are re-derived from dim_user's per-platform fields using the same priority logic as the original
• A WHERE clause excludes MicroMasters-only users from dim_user (who the original mart never emitted) to preserve row-level population scope
• All stats CTEs (course_stats, program_stats, orders_stats) are unchanged

Side effects (see ⚠️ note below):

• The old mart double-counted cross-platform users (e.g., a user on both MITx Online and xPro appeared in two UNION branches). dim_user correctly deduplicates these to one row per user — total row count drops ~6.6% (~524k rows), but only ~9k of those are genuinely different email-level users
• ~417k users gain the more accurate platforms = 'MITx Online and edX.org' label (previously incorrectly labeled 'MITx Online' because their edX link wasn't detected by the old mart)
• Stats join coverage improves slightly (94.24% → 94.52%) because dim_user.email is already lowercased, making the enrollment join more robust

---

⚠️ Breaking Migration Side Effect — Requires Team Discussion Before Merge

~49,852 users will have their user_hashed_id change after this migration.

These users were labeled 'edX.org' in the old mart (edX-only, no MITx Online account detected). dim_user now correctly links them to a MITx Online account (mitxonline_application_user_id IS NOT NULL). Because the new mart checks MITx Online first in the user_hashed_id CASE, their hash changes:

Old: sha256(cast(edxorg_openedx_user_id AS VARCHAR) || 'edX.org')
New: sha256(cast(mitxonline_application_user_id AS VARCHAR) || 'MITx Online')

This is intentionally correct behavior — these users genuinely have MITx Online accounts that the old mart missed. However, user_hashed_id is used as a primary key by Hightouch and as a join key in learner_demographics_and_cert_info and cheating_detection_report. A hard cutover would orphan Hightouch records for these ~50k users.

Mitigation included in this PR:

A user_hashed_id_edxorg_legacy column has been added. For any user with an edX.org account, it emits the edX.org-based hash regardless of their MITx Online status:

case
    when users.edxorg_openedx_user_id is not null
        then sha256(cast(edxorg_openedx_user_id AS VARCHAR) || 'edX.org')
end as user_hashed_id_edxorg_legacy

This allows a graceful migration:

1. Deploy this PR — both the new user_hashed_id and legacy column are available
2. Hightouch team updates their sync to reconcile old edX-based PKs via user_hashed_id_edxorg_legacy
3. Downstream models (learner_demographics_and_cert_info, cheating_detection_report) are updated to join on user_hashed_id
4. Legacy column is removed in a follow-up PR once all consumers are migrated

Suggested follow-up ticket: Open a new issue to track removal of user_hashed_id_edxorg_legacy and coordinate the Hightouch key migration. Suggested title: "Remove user_hashed_id_edxorg_legacy from marts__combined__users after Hightouch migration".

Do not merge this PR until the migration plan for the ~49,852 affected users is agreed with the team.

---

How can this be tested?

The following 4 queries can be run directly against production Trino to validate the migration logic without deploying to QA. They compare the current production marts__combined__users against the new logic sourced directly from dim_user.

Connection: trino --server https://mitol-ol-data-lake-production.trino.galaxy.starburst.io:443 --external-authentication --catalog ol_data_lake_production

---

Q1 — Row count parity by platform

What to look for: The MITx Online + "MITx Online and edX.org" combined totals should be nearly identical between old and new (~813k each). Other platforms will drop slightly due to cross-platform deduplication (expected). A large drop in any single platform that cannot be explained by deduplication would be a regression.

WITH new_logic AS (
    SELECT
        CASE
            WHEN mitxonline_application_user_id IS NOT NULL
                AND edxorg_openedx_user_id IS NOT NULL
                THEN 'MITx Online and edX.org'
            WHEN mitxonline_application_user_id IS NOT NULL THEN 'MITx Online'
            WHEN edxorg_openedx_user_id IS NOT NULL THEN 'edX.org'
            WHEN mitxpro_application_user_id IS NOT NULL THEN 'xPro'
            WHEN bootcamps_application_user_id IS NOT NULL THEN 'Bootcamps'
            WHEN emeritus_user_id IS NOT NULL THEN 'xPRO Emeritus'
            WHEN global_alumni_user_id IS NOT NULL THEN 'xPRO Global Alumni'
        END AS platforms
    FROM ol_data_lake_production.ol_warehouse_production_dimensional.dim_user
    WHERE
        mitxonline_application_user_id IS NOT NULL
        OR edxorg_openedx_user_id IS NOT NULL
        OR mitxpro_application_user_id IS NOT NULL
        OR bootcamps_application_user_id IS NOT NULL
        OR emeritus_user_id IS NOT NULL
        OR global_alumni_user_id IS NOT NULL
)
SELECT 'new' AS version, platforms, COUNT(*) AS user_count
FROM new_logic
GROUP BY platforms
UNION ALL
SELECT 'old' AS version, platforms, COUNT(*) AS user_count
FROM ol_data_lake_production.ol_warehouse_production_mart.marts__combined__users
GROUP BY platforms
ORDER BY platforms, version

---

Q2 — Email-level population delta

What to look for: in_old_not_new should be 0 or very small. These are users in the production mart whose email is genuinely absent from dim_user — they would be lost after migration. in_new_not_old may be a small positive number (new users dim_user now surfaces).

SELECT
    'in_old_not_new' AS direction,
    COUNT(*) AS user_count,
    APPROX_PERCENTILE(LENGTH(old_mart.user_email), 0.5) AS sanity_check_avg_email_len
FROM ol_data_lake_production.ol_warehouse_production_mart.marts__combined__users AS old_mart
WHERE NOT EXISTS (
    SELECT 1
    FROM ol_data_lake_production.ol_warehouse_production_dimensional.dim_user AS du
    WHERE
        du.email = LOWER(old_mart.user_email)
        AND (
            du.mitxonline_application_user_id IS NOT NULL
            OR du.edxorg_openedx_user_id IS NOT NULL
            OR du.mitxpro_application_user_id IS NOT NULL
            OR du.bootcamps_application_user_id IS NOT NULL
            OR du.emeritus_user_id IS NOT NULL
            OR du.global_alumni_user_id IS NOT NULL
        )
)
UNION ALL
SELECT
    'in_new_not_old' AS direction,
    COUNT(*) AS user_count,
    NULL AS sanity_check_avg_email_len
FROM ol_data_lake_production.ol_warehouse_production_dimensional.dim_user AS du
WHERE
    (
        du.mitxonline_application_user_id IS NOT NULL
        OR du.edxorg_openedx_user_id IS NOT NULL
        OR du.mitxpro_application_user_id IS NOT NULL
        OR du.bootcamps_application_user_id IS NOT NULL
        OR du.emeritus_user_id IS NOT NULL
        OR du.global_alumni_user_id IS NOT NULL
    )
    AND NOT EXISTS (
        SELECT 1
        FROM ol_data_lake_production.ol_warehouse_production_mart.marts__combined__users AS old_mart
        WHERE LOWER(old_mart.user_email) = du.email
    )

---

Q3 — user_hashed_id stability for MITx users (critical for Hightouch)

What to look for: pct_hash_match = 1.0 for MITx Online and MITx Online and edX.org. For edX.org, expect ~0.9928 — the ~0.72% mismatch corresponds to the ~49,852 users documented in the ⚠️ section above whose hash legitimately changes. These users can be migrated using user_hashed_id_edxorg_legacy.

SELECT
    old_mart.platforms,
    COUNT(*) AS matched_users,
    SUM(CASE
        WHEN old_mart.platforms LIKE '%MITx Online%'
            THEN CASE WHEN old_mart.user_hashed_id = lower(to_hex(sha256(cast(
                    CAST(du.mitxonline_application_user_id AS VARCHAR) || 'MITx Online'
                    AS VARBINARY)))) THEN 1 ELSE 0 END
        WHEN old_mart.platforms = 'edX.org'
            THEN CASE WHEN old_mart.user_hashed_id = lower(to_hex(sha256(cast(
                    CAST(du.edxorg_openedx_user_id AS VARCHAR) || 'edX.org'
                    AS VARBINARY)))) THEN 1 ELSE 0 END
        ELSE 1
    END) AS hash_matches,
    CAST(SUM(CASE
        WHEN old_mart.platforms LIKE '%MITx Online%'
            THEN CASE WHEN old_mart.user_hashed_id = lower(to_hex(sha256(cast(
                    CAST(du.mitxonline_application_user_id AS VARCHAR) || 'MITx Online'
                    AS VARBINARY)))) THEN 1 ELSE 0 END
        WHEN old_mart.platforms = 'edX.org'
            THEN CASE WHEN old_mart.user_hashed_id = lower(to_hex(sha256(cast(
                    CAST(du.edxorg_openedx_user_id AS VARCHAR) || 'edX.org'
                    AS VARBINARY)))) THEN 1 ELSE 0 END
        ELSE 1
    END) AS DOUBLE) / COUNT(*) AS pct_hash_match
FROM ol_data_lake_production.ol_warehouse_production_mart.marts__combined__users AS old_mart
JOIN ol_data_lake_production.ol_warehouse_production_dimensional.dim_user AS du
    ON du.email = LOWER(old_mart.user_email)
WHERE old_mart.platforms IN ('MITx Online', 'edX.org', 'MITx Online and edX.org')
GROUP BY old_mart.platforms
ORDER BY old_mart.platforms

---

Q4 — Enrollment stats join coverage

What to look for: new_coverage_rate should be greater than or equal to old_coverage_rate. A lower new rate would indicate the canonical email from dim_user is causing more missed enrollment joins than the old per-branch email logic.

WITH enrollment_emails AS (
    SELECT LOWER(user_email) AS email
    FROM ol_data_lake_production.ol_warehouse_production_intermediate.int__combined__courserun_enrollments
    GROUP BY LOWER(user_email)
),
new_users AS (
    SELECT email
    FROM ol_data_lake_production.ol_warehouse_production_dimensional.dim_user
    WHERE
        mitxonline_application_user_id IS NOT NULL
        OR edxorg_openedx_user_id IS NOT NULL
        OR mitxpro_application_user_id IS NOT NULL
        OR bootcamps_application_user_id IS NOT NULL
        OR emeritus_user_id IS NOT NULL
        OR global_alumni_user_id IS NOT NULL
)
SELECT
    'new' AS version,
    COUNT(nu.email) AS total_users,
    COUNT(ee.email) AS users_with_enrollments,
    ROUND(CAST(COUNT(ee.email) AS DOUBLE) / NULLIF(COUNT(nu.email), 0), 4) AS coverage_rate
FROM new_users nu
LEFT JOIN enrollment_emails ee ON nu.email = ee.email
UNION ALL
SELECT
    'old' AS version,
    COUNT(m.user_email) AS total_users,
    COUNT(ee.email) AS users_with_enrollments,
    ROUND(CAST(COUNT(ee.email) AS DOUBLE) / NULLIF(COUNT(m.user_email), 0), 4) AS coverage_rate
FROM ol_data_lake_production.ol_warehouse_production_mart.marts__combined__users m
LEFT JOIN enrollment_emails ee ON LOWER(m.user_email) = ee.email
ORDER BY version

Additional Context

Do not merge until the Hightouch key migration plan is agreed. See the ⚠️ section above.

The follow-up ticket to remove user_hashed_id_edxorg_legacy should be created and linked here before this PR is marked ready for review.