Loop r5 Phase C + KEYSTONE morb-fix: 68.2% -> 72.1% (+183 cols), closes #26

KEYSTONE (morbidity.sql): add PATSBJ participation morb Order 20080010 -> patient 20000000 so
nrt_observation.patient_id resolves -> sp_nrt_morbidity_report_postprocessing stops throwing Error 515
-> OBSERVATION entity no longer fails -> the service's fail-fast batch short-circuit no longer skips
CONTACT/VACCINATION/lab. This DETERMINISTICALLY fixes the 'flakiness' (covid_contact +51, d_contact_record
+39, f_contact_record_case +11, d_place +6 now stable) AND populates morb_rpt_user_comment 0->8 (closes
the PR #837 / task #26 morb PATIENT_KEY gap at its source) + lab_rpt_user_comment +8, lab100 +5.

Phase C COVID (zz_covid_dedicated_entities.sql + enriched covid_investigation_full_chain.sql PHC +
coordinated SubjOfPHC ownership in zz_investigation_patient_links.sql): dedicated rich patient/provider/
org (22055xxx) -> covid_case_datamart +54 (318->372) PATIENT_*/PHYS_*/RPT_*/HOSPITAL + PHC-core scalars.

Net +183 cols, idle-verified, zero per-table regressions.

Author: nullflux

utilities/comparison-fixtures/LOOP_round5.md

@@ -190,3 +190,8 @@ the morb report so sp_nrt_morbidity_report_postprocessing succeeds -> OBSERVATIO
 contact/vaccination/lab never skipped -> those tables populate DETERMINISTICALLY. Lab fixture (B) also
 needs its date-children remapped to {LAB334,349,350,356,357,361,362} via the FROM_TIME channel (routine
 020 hard convert(datetime) throws otherwise). Spawned morb-fix agent (parallel with COVID-C).
+- PHASE C (COVID) + KEYSTONE morb-fix COMMITTED: 68.2% -> 72.1% (+183 cols!), NO regressions.
+  covid_case_datamart +54 (dedicated patient/provider/org 22055xxx + enriched PHC), covid_contact +51,
+  d_contact_record +39, f_contact_record_case +11, morb_rpt_user_comment +8 (CLOSES #26), lab_rpt_user_comment
+  +8, lab100 +5, d_place +6 (all the formerly-"flaky" tables now STABLE+populated - LESSON 12 confirmed:
+  they were fail-fast-skipped by the morb-515 throw). morb 515 throw GONE. Phase B (lab) now safe.

utilities/comparison-fixtures/catalog/uid_ranges.md

@@ -1620,4 +1620,4 @@ tail-EXEC sequences these correctly for standalone verification.
 | 22052000 - 22052999 | R4-M TB tail debug/fix (tb_datamart 2nd row) |
 | 22053000 - 22053999 | R4-N lab100 + lab101 — **allocated 2026-06-03**, `zz_lab100_101_fill.sql`. ODSE-ONLY (NO-SHORTCUT). Part A (LAB100 36/69→demographics): new fully-attributed Order `22053010` + Result child `22053011` (LOINC 13950-1, cond 10110 baseline-seeded) with participations PATSBJ→20000000 / ORD,VRF→20000010 / AUT,ORD(org)→20000020 / PRF→20000020 + role SPP(PSN)→20000010 (specimen collector) → CDC sets nrt_observation.patient_id/ordering_person_id/author_organization_id/ordering_organization_id/specimen_collector_id → sp_d_labtest_result_postprocessing (017) resolves D_PATIENT 4 / D_PROVIDER 12 / D_ORGANIZATION 7 → fills PERSON_*/PATIENT_*/PROVIDER_*/ORDERING_FACILITY/REPORTING_FACILITY*/ACCESSION_NBR. ROOT CAUSE of NULLs: v2 lab 20070010 had ZERO participations. Part B (LAB101 0/46→unblock): full ODSE I_Order `22053500` + I_Result `22053501` + 'Result' `22053502` + 35 LABxxx I_Result children `22053600-22053634` (cd LAB329a,LAB330..LAB363) each + obs_value_coded(display_name→TEST_RESULT_VAL_CD_DESC) + obs_value_txt, wired child→I_Order via COMP act_relationship so sp_observation_event (055) emits them in the I_Order followup_observations JSON → service sets nrt_observation.followup_observation_uid → sp_lab101 (020) step 2 #tmp_I_Result_vals resolves → 35-col trtdN pivot (LAB1..LAB35) lands. ROOT CAUSE of 0/46: the shortcut-era zz_lab101_unblock.sql injected RDB LAB_TEST/LAB_RESULT_VAL directly but its nrt_observation INSERT was stripped on no-shortcut → root-order nrt_observation never existed → chain starved. UIDs consumed: 22053010/22053011 (Part A acts/obs), 22053500/22053501/22053502 (Part B I_Order/I_Result/Result), 22053600-22053634 (35 LABxxx children). **ORCH_TODO:** (1) add `22053010,22053011,22053500,22053501,22053502` to BOTH the `sp_observation_event @obs_id_list` and the two postprocessing `@obs_ids` lists in `scripts/merge_and_verify.sh run_lab_chain()` (~lines 312-315); (2) add `22053011,22053501` to `LAB_OBS_UIDS` (~line 473). SEED note: LAB100/LAB101 are NOT subject to the covid_lab 11065-LOINC seed gate (bug #16) — the chain uses baseline-seeded LOINC 13950-1→cond 10110 and culture codes, so CONDITION_CD/PROGRAM_AREA fill without seed edits. |
 | 22054000 - 22054999 | R4-O hepatitis_datamart remainder |
-| 22055000 - 22055999 | R5-C COVID dedicated patient/provider/org + enriched PHC |
+| 22055000 - 22055999 | R5-C COVID dedicated patient/provider/org + enriched PHC — **allocated 2026-06-04**, `zz_covid_dedicated_entities.sql` + edits to `covid_investigation_full_chain.sql` (PHC-core scalars) + `zz_investigation_patient_links.sql` (omit 22003000). ODSE-ONLY, ADDITIVE, no shared-dim UPDATE. Fills the ~27 shared-dim NULL cols of `covid_case_datamart` for COVID PHC 22003000 by authoring DEDICATED rich entities and repointing 22003000's participations to them: rich patient `22055000` (PSN/PAT, full name/middle/suffix/DOB/sex/race/ethnicity/marital/age+unit/deceased/full addr/home+work+cell phone w/ work-ext/email) → SubjOfPHC; investigator `22055010` → InvestgrOfPHC; physician `22055020` → PhysicianOfPHC; person-reporter `22055030` → PerAsReporterOfPHC (all PSN/PRV, self-parented, work addr/phone+ext/cell); reporter org `22055040` → OrgAsReporterOfPHC; hospital org `22055050` → HospOfADT (both ORG, addr + work phone+ext). Locators 22055001-006 / 011-013 / 021-023 / 031-033 / 041-042 / 051-052. nbs_act_entity edges (PerAsReporterOfPHC/OrgAsReporterOfPHC/HospOfADT) auto-IDENTITY + natural-key guard (LESSON 11). The pipeline (CDC→nrt_*→sp_*_event/sp_nrt_*_postprocessing) builds D_PATIENT/D_PROVIDER/D_ORGANIZATION + nrt_investigation.{patient,investigator,physician,person_as_reporter,organization,hospital}_id from these per ProcessInvestigationDataUtil.java:211-292; routine 310 reads them. The foundation SubjOfPHC(22003000→20000000) is superseded (DELETE + re-INSERT to 22055000); `zz_investigation_patient_links.sql` no longer lists 22003000 (it sorts AFTER and would otherwise re-add the foundation link → dup SubjOfPHC). **ORCH_TODO: none** — 22003000 already in `PHC_UIDS`; Step-8 Tier-3 drain (900s) projects ODSE→dims before Step-9 `sp_covid_case_datamart_postprocessing`. |

utilities/comparison-fixtures/coverage/coverage_merged.md

@@ -1,6 +1,6 @@
 # Coverage: merged fixture (full chain)
 
-Generated: 2026-06-04 06:37:39 UTC
+Generated: 2026-06-04 07:02:16 UTC
 
 This report is produced by `scripts/coverage_summary.sh` against the
 RDB_MODERN state after `scripts/merge_and_verify.sh` has run end-to-end.
@@ -13,14 +13,14 @@ A column is "populated" if at least one row has a non-NULL value for it.
 ## Summary
 
 - In-scope target tables: 118
-- Fully covered (all columns populated for at least one row): 63
+- Fully covered (all columns populated for at least one row): 67
 - Partially covered (some columns populated): 31
-- Empty (table exists, 0 rows): 23
+- Empty (table exists, 0 rows): 19
 - Missing (table not present in live RDB_MODERN): 1
 
 - Total columns across in-scope tables: 4652
-- Columns with ≥1 populated row: 3173
-- Overall column coverage: 68.2%
+- Columns with ≥1 populated row: 3356
+- Overall column coverage: 72.1%
 
 ## Per-table coverage
 
@@ -35,15 +35,15 @@ A column is "populated" if at least one row has a non-NULL value for it.
 | dbo.condition | 269 | 15 | 14/15 |
 | dbo.confirmation_method | 4 | 3 | **3/3** |
 | dbo.confirmation_method_group | 23 | 3 | **3/3** |
-| dbo.covid_case_datamart | 1 | 383 | 318/383 |
-| dbo.covid_contact_datamart | 0 | 94 | 0/94 |
+| dbo.covid_case_datamart | 1 | 383 | 372/383 |
+| dbo.covid_contact_datamart | 1 | 94 | 51/94 |
 | dbo.covid_lab_celr_datamart | 0 | 101 | 0/101 |
 | dbo.covid_lab_datamart | 0 | 120 | 0/120 |
 | dbo.covid_vaccination_datamart | 1 | 60 | 39/60 |
 | dbo.d_addl_risk | 5 | 6 | **6/6** |
 | dbo.d_addl_risk_group | 3 | 1 | **1/1** |
 | dbo.d_case_management | 16 | 67 | 41/67 |
-| dbo.d_contact_record | 1 | 66 | 1/66 |
+| dbo.d_contact_record | 4 | 66 | 40/66 |
 | dbo.d_disease_site | 5 | 6 | **6/6** |
 | dbo.d_disease_site_group | 3 | 1 | **1/1** |
 | dbo.d_gt_12_reas | 4 | 6 | **6/6** |
@@ -63,14 +63,14 @@ A column is "populated" if at least one row has a non-NULL value for it.
 | dbo.d_move_state_group | 3 | 1 | **1/1** |
 | dbo.d_moved_where | 4 | 6 | **6/6** |
 | dbo.d_moved_where_group | 3 | 1 | **1/1** |
-| dbo.d_organization | 13 | 30 | **30/30** |
+| dbo.d_organization | 15 | 30 | **30/30** |
 | dbo.d_out_of_cntry | 5 | 6 | **6/6** |
 | dbo.d_out_of_cntry_group | 3 | 1 | **1/1** |
-| dbo.d_patient | 9 | 81 | 60/81 |
+| dbo.d_patient | 10 | 81 | 60/81 |
 | dbo.d_pcr_source | 3 | 6 | **6/6** |
 | dbo.d_pcr_source_group | 2 | 1 | **1/1** |
-| dbo.d_place | 4 | 37 | 31/37 |
-| dbo.d_provider | 19 | 34 | 32/34 |
+| dbo.d_place | 10 | 37 | **37/37** |
+| dbo.d_provider | 22 | 34 | 32/34 |
 | dbo.d_rash_loc_gen | 3 | 6 | **6/6** |
 | dbo.d_rash_loc_gen_group | 2 | 1 | **1/1** |
 | dbo.d_smr_exam_ty | 3 | 6 | **6/6** |
@@ -80,9 +80,9 @@ A column is "populated" if at least one row has a non-NULL value for it.
 | dbo.d_vaccination | 4 | 21 | **21/21** |
 | dbo.d_var_pam | 1 | 129 | 127/129 |
 | dbo.etl_dq_log | 1572 | 15 | **15/15** |
-| dbo.event_metric | 26 | 28 | **28/28** |
-| dbo.event_metric_inc | 26 | 28 | **28/28** |
-| dbo.f_contact_record_case | 0 | 11 | 0/11 |
+| dbo.event_metric | 29 | 28 | **28/28** |
+| dbo.event_metric_inc | 29 | 28 | **28/28** |
+| dbo.f_contact_record_case | 3 | 11 | **11/11** |
 | dbo.f_interview_case | 2 | 10 | **10/10** |
 | dbo.f_page_case | 4 | 35 | **35/35** |
 | dbo.f_std_page_case | 12 | 52 | **52/52** |
@@ -94,19 +94,19 @@ A column is "populated" if at least one row has a non-NULL value for it.
 | dbo.hepatitis_datamart | 3 | 209 | 160/209 |
 | dbo.inv_hiv | 2 | 19 | 17/19 |
 | dbo.inv_summ_datamart | 22 | 58 | **58/58** |
-| dbo.investigation | 23 | 71 | 61/71 |
+| dbo.investigation | 23 | 71 | 62/71 |
 | dbo.job_batch_rebuild_log | MISSING | - | - |
-| dbo.job_flow_log | 29681 | 15 | 14/15 |
+| dbo.job_flow_log | 30180 | 15 | 14/15 |
 | dbo.l_inv_place_repeat | 1 | 2 | 1/2 |
 | dbo.l_investigation_repeat | 12 | 2 | **2/2** |
-| dbo.l_investigation_repeat_inc | 9 | 2 | **2/2** |
-| dbo.lab100 | 1 | 69 | 31/69 |
+| dbo.l_investigation_repeat_inc | 6 | 2 | **2/2** |
+| dbo.lab100 | 2 | 69 | 36/69 |
 | dbo.lab101 | 0 | 46 | 0/46 |
 | dbo.lab_result_comment | 5 | 6 | **6/6** |
-| dbo.lab_result_val | 48 | 20 | **20/20** |
-| dbo.lab_rpt_user_comment | 0 | 8 | 0/8 |
-| dbo.lab_test | 19 | 66 | 55/66 |
-| dbo.lab_test_result | 15 | 20 | 19/20 |
+| dbo.lab_result_val | 59 | 20 | **20/20** |
+| dbo.lab_rpt_user_comment | 1 | 8 | **8/8** |
+| dbo.lab_test | 32 | 66 | 55/66 |
+| dbo.lab_test_result | 28 | 20 | 19/20 |
 | dbo.ldf_bmird | 0 | 7 | 0/7 |
 | dbo.ldf_data | 1 | 17 | 9/17 |
 | dbo.ldf_datamart_column_ref | 2662 | 8 | **8/8** |
@@ -117,11 +117,11 @@ A column is "populated" if at least one row has a non-NULL value for it.
 | dbo.ldf_mumps | 0 | 7 | 0/7 |
 | dbo.ldf_tetanus | 0 | 7 | 0/7 |
 | dbo.ldf_vaccine_prevent_diseases | 1 | 8 | **8/8** |
-| dbo.lookup_table_n_rept | 7 | 2 | **2/2** |
-| dbo.morb_rpt_user_comment | 0 | 8 | 0/8 |
+| dbo.lookup_table_n_rept | 2 | 2 | **2/2** |
+| dbo.morb_rpt_user_comment | 1 | 8 | **8/8** |
 | dbo.morbidity_report | 3 | 30 | **30/30** |
-| dbo.morbidity_report_datamart | 1 | 133 | 130/133 |
-| dbo.morbidity_report_event | 1 | 17 | **17/17** |
+| dbo.morbidity_report_datamart | 2 | 133 | 130/133 |
+| dbo.morbidity_report_event | 2 | 17 | **17/17** |
 | dbo.notification | 2 | 6 | **6/6** |
 | dbo.notification_event | 2 | 8 | **8/8** |
 | dbo.organization_ldf_group | 0 | 3 | 0/3 |
@@ -138,17 +138,17 @@ A column is "populated" if at least one row has a non-NULL value for it.
 | dbo.tb_datamart | 2 | 318 | 293/318 |
 | dbo.tb_hiv_datamart | 2 | 322 | 297/322 |
 | dbo.tb_pam_ldf | 0 | 3 | 0/3 |
-| dbo.test_result_grouping | 14 | 3 | **3/3** |
-| dbo.treatment | 4 | 16 | **16/16** |
-| dbo.treatment_event | 3 | 11 | **11/11** |
+| dbo.test_result_grouping | 25 | 3 | **3/3** |
+| dbo.treatment | 7 | 16 | **16/16** |
+| dbo.treatment_event | 6 | 11 | **11/11** |
 | dbo.user_profile | 12 | 8 | **8/8** |
 | dbo.var_datamart | 0 | 231 | 0/231 |
 | dbo.var_pam_ldf | 0 | 3 | 0/3 |
 
 
 ## Categorization
 
-### Fully covered (63)
+### Fully covered (67)
 
 Tables where every column has at least one row with a non-NULL value.
 
@@ -179,6 +179,7 @@ Tables where every column has at least one row with a non-NULL value.
 - dbo.d_out_of_cntry_group
 - dbo.d_pcr_source
 - dbo.d_pcr_source_group
+- dbo.d_place
 - dbo.d_rash_loc_gen
 - dbo.d_rash_loc_gen_group
 - dbo.d_smr_exam_ty
@@ -188,6 +189,7 @@ Tables where every column has at least one row with a non-NULL value.
 - dbo.etl_dq_log
 - dbo.event_metric
 - dbo.event_metric_inc
+- dbo.f_contact_record_case
 - dbo.f_interview_case
 - dbo.f_page_case
 - dbo.f_std_page_case
@@ -199,10 +201,12 @@ Tables where every column has at least one row with a non-NULL value.
 - dbo.l_investigation_repeat_inc
 - dbo.lab_result_comment
 - dbo.lab_result_val
+- dbo.lab_rpt_user_comment
 - dbo.ldf_datamart_column_ref
 - dbo.ldf_group
 - dbo.ldf_vaccine_prevent_diseases
 - dbo.lookup_table_n_rept
+- dbo.morb_rpt_user_comment
 - dbo.morbidity_report
 - dbo.morbidity_report_event
 - dbo.notification
@@ -224,6 +228,7 @@ candidates for Tier 3 gap-driven coverage work.
 - dbo.bmird_strep_pneumo_datamart
 - dbo.condition
 - dbo.covid_case_datamart
+- dbo.covid_contact_datamart
 - dbo.covid_vaccination_datamart
 - dbo.d_case_management
 - dbo.d_contact_record
@@ -232,7 +237,6 @@ candidates for Tier 3 gap-driven coverage work.
 - dbo.d_investigation_repeat
 - dbo.d_ldf_meta_data
 - dbo.d_patient
-- dbo.d_place
 - dbo.d_provider
 - dbo.d_tb_pam
 - dbo.d_var_pam
@@ -253,27 +257,23 @@ candidates for Tier 3 gap-driven coverage work.
 - dbo.tb_datamart
 - dbo.tb_hiv_datamart
 
-### Empty (23)
+### Empty (19)
 
 Tables that exist in RDB_MODERN but have zero rows after the merged
 chain runs. Most are datamart-side fact tables that depend on Merge
 contract step 9 (Datamart SPs — out of scope for v1).
 
 - dbo.aggregate_report_datamart
-- dbo.covid_contact_datamart
 - dbo.covid_lab_celr_datamart
 - dbo.covid_lab_datamart
 - dbo.d_interview_note
-- dbo.f_contact_record_case
 - dbo.f_var_pam
 - dbo.lab101
-- dbo.lab_rpt_user_comment
 - dbo.ldf_bmird
 - dbo.ldf_foodborne
 - dbo.ldf_hepatitis
 - dbo.ldf_mumps
 - dbo.ldf_tetanus
-- dbo.morb_rpt_user_comment
 - dbo.organization_ldf_group
 - dbo.patient_ldf_group
 - dbo.provider_ldf_group

utilities/comparison-fixtures/fixtures/10_subjects/morbidity.sql

@@ -461,6 +461,68 @@ VALUES
      @superuser_id, N'ACTIVE', '2026-04-04T00:00:00', 16, N'OBS', N'OBS',
      N'A', '2026-04-04T00:00:00', N'Component');
 
+-- =====================================================================
+-- v2 Morb Order patient-subject participation (PATSBJ).
+--
+-- KEYSTONE FIX (Round 5 LESSON 12): the Morbidity postprocessing SP
+-- (016-sp_nrt_morbidity_report_postprocessing) resolves
+-- MORBIDITY_REPORT_EVENT.PATIENT_KEY at line 986 via
+--     left join dbo.d_patient pat ON n.patient_id = pat.patient_uid
+-- where n = #morb_obs_reference (= dbo.nrt_observation). PATIENT_KEY is
+-- NOT COALESCEd and the target column is NOT NULL, so a NULL
+-- nrt_observation.patient_id throws SQL Error 515 ("Cannot insert the
+-- value NULL into column 'PATIENT_KEY'") on EVERY run.
+--
+-- The suite is now fully CDC-driven: morbidity.sql no longer hand-writes
+-- nrt_observation; that row is materialized by the pipeline
+-- (observation -> sp_observation_event JSON -> reporting-pipeline-service
+-- -> nrt_observation). The service sets nrt_observation.patient_id ONLY
+-- when the Order observation carries a PERSON participation whose
+-- type_cd is 'PATSBJ' or 'SubjOfMorbReport' and subject_class_cd='PSN'
+-- (ProcessObservationDataUtil.transformPersonParticipations, lines
+-- 113-122: case "PATSBJ","SubjOfMorbReport" -> setPatientId(entityId)).
+-- The morb Order acts had ZERO participations, so patient_id stayed
+-- NULL -> the 515 throw -> the fail-fast short-circuit in
+-- PostProcessingService.processIdCache skipped CONTACT/VACCINATION/lab in
+-- the same CDC batch (LESSON 12).
+--
+-- This additive PATSBJ participation links the v2 Morb Order (20080010)
+-- to the foundation Patient (20000000, which HAS a D_PATIENT row,
+-- PATIENT_KEY=4). The event SP projects it (its filter is
+-- p.act_uid = o.observation_uid AND p.record_status_cd='ACTIVE', joining
+-- nbs_odse.dbo.person; 20000000 is an ACTIVE PSN person) -> the service
+-- sets nrt_observation.patient_id=20000000 -> the SP join resolves
+-- PATIENT_KEY -> MORBIDITY_REPORT_EVENT INSERT succeeds -> no 515.
+--
+-- (Mirrors how investigations get their patient via the SubjOfPHC
+-- participation in fixtures/20_links/patient_phc.sql; PAR_TYPE 'PATSBJ'
+-- is the production patient-subject participation type for observations.)
+-- Composite PK is (act_uid, subject_entity_uid, type_cd) — no surrogate
+-- UID needed.
+-- =====================================================================
+INSERT INTO [dbo].[participation]
+    ([act_uid], [subject_entity_uid], [type_cd],
+     [act_class_cd], [subject_class_cd],
+     [add_time], [add_user_id], [last_chg_time], [last_chg_user_id],
+     [record_status_cd], [record_status_time],
+     [status_cd], [status_time],
+     [type_desc_txt])
+VALUES
+    (@dbo_Act_morb_v2_order_uid,      -- act_uid (OBS; v2 Morb Order)
+     @foundation_patient_uid,         -- subject_entity_uid (PSN; foundation Patient 20000000)
+     N'PATSBJ',                       -- type_cd (patient-subject of observation)
+     N'OBS',                          -- act_class_cd
+     N'PSN',                          -- subject_class_cd (PERSON; required for setPatientId)
+     '2026-04-04T00:00:00',           -- add_time
+     @superuser_id,                   -- add_user_id
+     '2026-04-04T00:00:00',           -- last_chg_time
+     @superuser_id,                   -- last_chg_user_id
+     N'ACTIVE',                       -- record_status_cd (event SP filters = 'ACTIVE')
+     '2026-04-04T00:00:00',           -- record_status_time
+     'A',                             -- status_cd
+     '2026-04-04T00:00:00',           -- status_time
+     N'Patient Subject');
+
 -- =====================================================================
 -- v2 Morb Order act_id row.
 -- =====================================================================