Skip to content

fix(ingest/teradata): lowercase database container URNs under convert_urns_to_lowercase#18232

Merged
treff7es merged 3 commits into
masterfrom
fix-teradata-lowercase-container-urns
Jul 9, 2026
Merged

fix(ingest/teradata): lowercase database container URNs under convert_urns_to_lowercase#18232
treff7es merged 3 commits into
masterfrom
fix-teradata-lowercase-container-urns

Conversation

@treff7es

@treff7es treff7es commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

When convert_urns_to_lowercase is enabled, the Teradata connector lower-cased dataset URNs — via get_identifier() and the central AutoLowercaseUrnsProcessor — but built the database container URN from the source-case name. This split one physical database into two URN identities: datasets were parented under a container whose key did not match their own (lower-cased) name.

Because a container URN is an opaque hash of its key, the stream-level AutoLowercaseUrnsProcessor (which only rewrites dataset/schemaField URNs) structurally cannot fix container casing — it has to be normalized at key-build time in the source. This mirrors how Snowflake already handles it (routing the database/schema name through snowflake_identifier before gen_database_key/gen_schema_key).

Changes

  • Add _maybe_lower_urn_name() helper and apply it in the two seams that build the database container key:
    • get_database_container_key() — dataset → container parenting
    • gen_database_containers() — container entity creation
  • get_db_name() is intentionally left in source case: it is reused verbatim as a quoted SQL identifier (DATABASE "{schema}" set before view HELP commands), which must match the stored case in CASESPECIFIC installations. Only URN identity is normalized; SQL resolution is unaffected.

Behavior

  • Default config (convert_urns_to_lowercase: false) is unchanged.
  • With the flag enabled, container URNs are now consistent with the (already lower-cased) dataset URNs.

Testing

  • TestConvertUrnsToLowercaseContainerConsistency: flag-on consistency, flag-off no-op, and that the container creation and parenting paths resolve to the same container URN.
  • Full Teradata unit suites pass; ruff + mypy clean.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly PR Title Format)
  • Tests for the changes have been added/updated
  • Breaking change documented in docs/how/updating-datahub.md

Note: this is a URN-changing migration for deployments already running with the flag enabled (their container URNs shift from source-case to lower-case on the next run). See the entry added to docs/how/updating-datahub.md.

…_urns_to_lowercase

When `convert_urns_to_lowercase` is enabled, dataset URNs were lower-cased
(via `get_identifier` and the central `AutoLowercaseUrnsProcessor`) while the
database container URN was keyed on the source-case name. That split one
physical database into two URN identities: datasets ended up parented under a
container URN whose key did not match their own (lower-cased) name, which
surfaces as mismatched / duplicate "lowercase" containers once any
consistently-lower-cased producer touches the same database.

The central lowercase processor cannot fix this: container URNs are opaque
hashes, so casing must be normalized at key-build time in the source (the same
approach Snowflake takes by routing names through `snowflake_identifier`).

Normalize the container name in the two seams that build the key:
- `get_database_container_key` (dataset -> container parenting)
- `gen_database_containers` (container entity creation)

`get_db_name` is intentionally left in source case: it is reused verbatim as a
quoted SQL identifier (`DATABASE "{schema}"` before view HELP commands), which
must match the stored case in CASESPECIFIC installations. Only URN identity is
normalized; SQL resolution is unaffected.

Tests cover flag-on consistency, flag-off no-op, and that the container
creation and parenting paths resolve to the same URN.
@github-actions github-actions Bot added ingestion PR or Issue related to the ingestion of metadata docs Issues and Improvements to docs labels Jul 7, 2026
@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@maggiehays maggiehays added the needs-review Label for PRs that need review from a maintainer. label Jul 7, 2026
Comment on lines +2193 to +2195
# The container *entity* is created here (via gen_database_key), a path
# separate from get_database_container_key() used for dataset parenting;
# both must agree, so apply the same normalization to the name.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment like this is quite dangerous, it describes places which this function is called from, which function itself has no control over.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trimmed — the comment no longer references the other call paths, it just states the local reason (normalize the name so the container URN matches the dataset URNs). Did the same for the sibling comments.

@skrydal skrydal left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, comments are overly verbose though.

Do I understand correctly, that if user actively uses Teradata's capability to have 2 different databases differing by casing (e.g. Foo and foo) we will generate a single container for them if lowercasing is enabled? Effectively lowercase flag can be used only by users who can ensure they do not have such cases in their data source?

@maggiehays maggiehays added pending-submitter-merge and removed needs-review Label for PRs that need review from a maintainer. labels Jul 8, 2026
@treff7es

treff7es commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@skrydal on the casing question: for Teradata specifically this can't happen — object identifiers (database/table names) are case-insensitive, so you can't have two databases differing only by case (CASESPECIFIC only affects CHAR data comparison, not names). More generally the caveat isn't new to this PR: with convert_urns_to_lowercase on, the datasets under such names already collapse to a single URN via get_identifier()/AutoLowercaseUrnsProcessor — this change just makes the container consistent with the datasets it already parents.

Also trimmed all the comments per your note.

@treff7es treff7es enabled auto-merge (squash) July 9, 2026 14:22
@treff7es treff7es merged commit c6912d9 into master Jul 9, 2026
56 of 58 checks passed
@treff7es treff7es deleted the fix-teradata-lowercase-container-urns branch July 9, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Issues and Improvements to docs ingestion PR or Issue related to the ingestion of metadata pending-submitter-merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants