Skip to content

perf: skip unnecessary metadata fetch calls for tags when not configured#1387

Open
tejassp-db wants to merge 2 commits intomainfrom
PECOBLR-2497/skip-unused-metadata-calls
Open

perf: skip unnecessary metadata fetch calls for tags when not configured#1387
tejassp-db wants to merge 2 commits intomainfrom
PECOBLR-2497/skip-unused-metadata-calls

Conversation

@tejassp-db
Copy link
Copy Markdown
Collaborator

Summary

  • Skips fetch_tags and fetch_column_tags queries to information_schema during incremental and view materializations when the model has no tags configured
  • Falls back to fetching metadata when model config is unavailable (safe default)
  • No behavior change for models with tags configured — diffs still computed correctly

What changed

  • Added requires_server_metadata_for_diff() to TagsConfig and ColumnTagsConfig to signal when server metadata is needed
  • IncrementalTableAPI._describe_relation and ViewAPI._describe_relation now check model config before making tag fetch calls
  • get_relation_config accepts the model config to pass through to _describe_relation
  • Fixed duplicate TagsProcessor in StreamingTableConfig.config_components

Test plan

  • Unit tests: empty tags (skip), non-empty tags (fetch), null config fallback (fetch), both tags + column tags present, hive_metastore (skip)

  • Functional tests: override fetch_tags/fetch_column_tags macros to raise errors, confirming calls are actually skipped when expected

    PECOBLR-2497

Skip fetch_tags and fetch_column_tags information_schema queries during
incremental and view materializations when the model has no tags configured.
This avoids unnecessary server roundtrips on every run for models that don't
use tags, while preserving full fetch behavior when tags are present or
when the model config is unavailable.

PECOBLR-2497
@github-actions
Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  dbt/adapters/databricks
  impl.py 931, 935-944, 1027-1029, 1054-1057, 1076-1077, 1101-1102
  dbt/adapters/databricks/relation_configs
  base.py 41
  column_tags.py
  tags.py
Project Total  

This report was generated by python-coverage-comment-action

Add unit and functional coverage for skipping tag metadata queries when model
config does not require them. This protects the new fetch-planning logic
across incremental, view, streaming table, and materialized view test paths
without changing unrelated unstaged work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant