fix(query): refresh ATTACH table schema in system.columns/statistics#19966
Draft
TCeason wants to merge 2 commits into
Draft
fix(query): refresh ATTACH table schema in system.columns/statistics#19966TCeason wants to merge 2 commits into
TCeason wants to merge 2 commits into
Conversation
system.columns, system.statistics and the information_schema.columns view expose per-column metadata via table.schema(). For read-only ATTACH tables the schema is not persisted on the meta server; it is derived from the source table's latest snapshot and only becomes current after a refresh. dump_tables disabled refresh for performance, so these tables reported the schema frozen at ATTACH time, hiding columns added to the source afterwards (while DESC, which refreshes, showed them). Naively dropping disable_catalog_refresh would regress the resilience added in "avoid SHOW TABLES refresh failures": a single unreachable ATTACH table makes its whole-database refresh fail, dropping the columns of healthy sibling tables. So keep listing through the refresh-disabled catalog (fast, resilient, zero S3 for normal tables), then refresh ATTACH tables individually through the original catalog, falling back to the cached schema with a warning on failure so one broken table never drops its siblings' columns. system.tables keeps refresh disabled on purpose: it only needs meta-server level table info and never reads the snapshot.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
system.columns,system.statisticsandinformation_schema.columnsdon't reflect columns added to a read-only ATTACH table's source table.DESCshows them, but these system tables keep the schema frozen at ATTACH time.system.tables keeps refresh disabled on purpose: it only needs meta-server level table info and never reads the snapshot.
Add setting
enable_table_schema_refresh(default0). When enabled,system.columns/system.statisticsre-fetches the schema from storage for each ATTACH table, picking up source-side schema changes.Tests
Type of change
This change is