fix(ingest/teradata): lowercase database container URNs under convert_urns_to_lowercase#18232
Conversation
…_urns_to_lowercase
When `convert_urns_to_lowercase` is enabled, dataset URNs were lower-cased
(via `get_identifier` and the central `AutoLowercaseUrnsProcessor`) while the
database container URN was keyed on the source-case name. That split one
physical database into two URN identities: datasets ended up parented under a
container URN whose key did not match their own (lower-cased) name, which
surfaces as mismatched / duplicate "lowercase" containers once any
consistently-lower-cased producer touches the same database.
The central lowercase processor cannot fix this: container URNs are opaque
hashes, so casing must be normalized at key-build time in the source (the same
approach Snowflake takes by routing names through `snowflake_identifier`).
Normalize the container name in the two seams that build the key:
- `get_database_container_key` (dataset -> container parenting)
- `gen_database_containers` (container entity creation)
`get_db_name` is intentionally left in source case: it is reused verbatim as a
quoted SQL identifier (`DATABASE "{schema}"` before view HELP commands), which
must match the stored case in CASESPECIFIC installations. Only URN identity is
normalized; SQL resolution is unaffected.
Tests cover flag-on consistency, flag-off no-op, and that the container
creation and parenting paths resolve to the same URN.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
| # The container *entity* is created here (via gen_database_key), a path | ||
| # separate from get_database_container_key() used for dataset parenting; | ||
| # both must agree, so apply the same normalization to the name. |
There was a problem hiding this comment.
Comment like this is quite dangerous, it describes places which this function is called from, which function itself has no control over.
There was a problem hiding this comment.
Trimmed — the comment no longer references the other call paths, it just states the local reason (normalize the name so the container URN matches the dataset URNs). Did the same for the sibling comments.
skrydal
left a comment
There was a problem hiding this comment.
LGTM, comments are overly verbose though.
Do I understand correctly, that if user actively uses Teradata's capability to have 2 different databases differing by casing (e.g. Foo and foo) we will generate a single container for them if lowercasing is enabled? Effectively lowercase flag can be used only by users who can ensure they do not have such cases in their data source?
|
@skrydal on the casing question: for Teradata specifically this can't happen — object identifiers (database/table names) are case-insensitive, so you can't have two databases differing only by case ( Also trimmed all the comments per your note. |
Summary
When
convert_urns_to_lowercaseis enabled, the Teradata connector lower-cased dataset URNs — viaget_identifier()and the centralAutoLowercaseUrnsProcessor— but built the database container URN from the source-case name. This split one physical database into two URN identities: datasets were parented under a container whose key did not match their own (lower-cased) name.Because a container URN is an opaque hash of its key, the stream-level
AutoLowercaseUrnsProcessor(which only rewritesdataset/schemaFieldURNs) structurally cannot fix container casing — it has to be normalized at key-build time in the source. This mirrors how Snowflake already handles it (routing the database/schema name throughsnowflake_identifierbeforegen_database_key/gen_schema_key).Changes
_maybe_lower_urn_name()helper and apply it in the two seams that build the database container key:get_database_container_key()— dataset → container parentinggen_database_containers()— container entity creationget_db_name()is intentionally left in source case: it is reused verbatim as a quoted SQL identifier (DATABASE "{schema}"set before viewHELPcommands), which must match the stored case in CASESPECIFIC installations. Only URN identity is normalized; SQL resolution is unaffected.Behavior
convert_urns_to_lowercase: false) is unchanged.Testing
TestConvertUrnsToLowercaseContainerConsistency: flag-on consistency, flag-off no-op, and that the container creation and parenting paths resolve to the same container URN.ruff+mypyclean.Checklist
docs/how/updating-datahub.mdNote: this is a URN-changing migration for deployments already running with the flag enabled (their container URNs shift from source-case to lower-case on the next run). See the entry added to
docs/how/updating-datahub.md.