Err/app 556/null columns dm inv generic v2

[eliSkylight]

Description

sp_organization_event has been updated to properly handing empty spaces for several organization columns. Previously, these empty spaces caused downstream datamart tables to contain inaccurate data. A functional test step has also been added to solidfy the changes. I also modified the expected values for the organizationEvent unit test to account for the NULL values that are now being returned from the stored procedure.

Related Issue

APP-556

Additional Notes

The bmirdCase functional test was renamed to casesBmirdGeneric since a GCD investigation was added as a step. Also, I added ALTER statements to the D_INVESTIGATION_REPEAT table for columns known to cause datamart-related issues in our local deployment. More information on that here.

Checklist

• I have ensured that the pull request is of a manageable size, allowing it to be reviewed within a single session.
• I have reviewed my changes to ensure they are clear, concise, and well-documented.
• I have updated the documentation, if applicable.
• I have added or updated test cases to cover my changes, if applicable.

[ericnagel]

Also, I added ALTER statements to the D_INVESTIGATION_REPEAT table for columns known to cause datamart-related issues in our local deployment.

Were the ALTER statements to the D_INVESTIGATION_REPEAT table supposed to be in this PR?
Look at PR #871 - I think the change to liquibase-service/src/main/resources/db/005-rdb_modern/routines/245-sp_dyn_dm_createdm_postprocessing-001.sql eliminates (or, gets around) the need for these alter statements.

[eliSkylight]

Also, I added ALTER statements to the D_INVESTIGATION_REPEAT table for columns known to cause datamart-related issues in our local deployment.

Were the ALTER statements to the D_INVESTIGATION_REPEAT table supposed to be in this PR? Look at PR #871 - I think the change to liquibase-service/src/main/resources/db/005-rdb_modern/routines/245-sp_dyn_dm_createdm_postprocessing-001.sql eliminates (or, gets around) the need for these alter statements.

Didn't know there was a PR for that. I'll wait for it to merge and see it if fixes the known error.

[ericnagel]

The ALTER statements are in https://github.com/CDCgov/NEDSS-DataReporting/pull/882/changes#diff-bf11e505bf0f9c4859f55a85601aa54cd63205a2d7753b6a3a43e23f6c9a4929R23

[ericbuckley]

Also, I added ALTER statements to the D_INVESTIGATION_REPEAT table for columns known to cause datamart-related issues in our local deployment.

Were the ALTER statements to the D_INVESTIGATION_REPEAT table supposed to be in this PR? Look at PR #871 - I think the change to liquibase-service/src/main/resources/db/005-rdb_modern/routines/245-sp_dyn_dm_createdm_postprocessing-001.sql eliminates (or, gets around) the need for these alter statements.

Didn't know there was a PR for that. I'll wait for it to merge and see it if fixes the known error.

I'm wondering about the order of our db initialization scripts. Would it help if we ran the 04 and 05 migrations before we copy RDB -> RDB_Modern?

[mehansen]

does this fix the issue in the ticket of the discrepancy in DM_INV_GENERIC_V2.EVENT_DATE?

[eliSkylight]

does this fix the issue in the ticket of the discrepancy in DM_INV_GENERIC_V2.EVENT_DATE?

In the ticket I reported that the EVENT_DATE is not discrepant in a local environment. These columns were determined as discrepant however from local testing.

[eliSkylight]

Also, I added ALTER statements to the D_INVESTIGATION_REPEAT table for columns known to cause datamart-related issues in our local deployment.

Were the ALTER statements to the D_INVESTIGATION_REPEAT table supposed to be in this PR? Look at PR #871 - I think the change to liquibase-service/src/main/resources/db/005-rdb_modern/routines/245-sp_dyn_dm_createdm_postprocessing-001.sql eliminates (or, gets around) the need for these alter statements.

Didn't know there was a PR for that. I'll wait for it to merge and see it if fixes the known error.

I'm wondering about the order of our db initialization scripts. Would it help if we ran the 04 and 05 migrations before we copy RDB -> RDB_Modern?

@ericbuckley not sure what issues reordering would cause.

[ericbuckley]

Also, I added ALTER statements to the D_INVESTIGATION_REPEAT table for columns known to cause datamart-related issues in our local deployment.

Were the ALTER statements to the D_INVESTIGATION_REPEAT table supposed to be in this PR? Look at PR #871 - I think the change to liquibase-service/src/main/resources/db/005-rdb_modern/routines/245-sp_dyn_dm_createdm_postprocessing-001.sql eliminates (or, gets around) the need for these alter statements.

Didn't know there was a PR for that. I'll wait for it to merge and see it if fixes the known error.

I'm wondering about the order of our db initialization scripts. Would it help if we ran the 04 and 05 migrations before we copy RDB -> RDB_Modern?

@ericbuckley not sure what issues reordering would cause.

@eliSkylight its mainly this call. https://github.com/CDCgov/NEDSS-DataReporting/blob/main/containers/db/initialize/005-prep-for-masterEtl-trace.sql#L6. If we delete the D_INVESTIGATION_REPEAT table in RDB, before we duplicate that to RDB_Modern, then it will be missing when RDB_Modern starts up and we run RTR. I was thinking if its missing then RTR would create the table with the proper table structure and the ALTER statements wouldn't be required. maybe not though 🤷