Skip to content

Config for refresh catalog and max fetch tables concurrency#42

Merged
osipovartem merged 5 commits into
mainfrom
aosipov/concurrent_table_fetches
Nov 25, 2025
Merged

Config for refresh catalog and max fetch tables concurrency#42
osipovartem merged 5 commits into
mainfrom
aosipov/concurrent_table_fetches

Conversation

@osipovartem
Copy link
Copy Markdown
Contributor

  • default max_concurrent_fetches for tables is 5
  • by default refresh_catalog is disabled
  • added fetch_table_providers func which allows to get table_providers in concurrently to reduce time of catalog tree preparing
  • with disabled refresh catalog flag we add missing entities to the catalog list cache during tables calls (if the table is missing in cache but exists in query, we will add it to cache for the next queries)

Copy link
Copy Markdown
Contributor

@DanCodedThis DanCodedThis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@osipovartem LGTM, with some tweaks

Comment thread crates/embucketd/src/cli.rs Outdated
Comment thread crates/embucketd/src/cli.rs Outdated
Comment thread crates/catalog/src/utils.rs
Comment thread crates/catalog/src/information_schema/config.rs
Comment thread crates/catalog/src/catalog_list.rs Outdated
@osipovartem osipovartem merged commit d6affa4 into main Nov 25, 2025
3 checks passed
@osipovartem osipovartem deleted the aosipov/concurrent_table_fetches branch November 25, 2025 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants