Skip to content

feat(bigquery-jdbc): add queryProcessThreadCount connection property#13480

Closed
Neenu1995 wants to merge 1 commit into
pr-1-executor-migrationfrom
pr-2-query-thread-count
Closed

feat(bigquery-jdbc): add queryProcessThreadCount connection property#13480
Neenu1995 wants to merge 1 commit into
pr-1-executor-migrationfrom
pr-2-query-thread-count

Conversation

@Neenu1995

Copy link
Copy Markdown
Contributor

No description provided.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the background thread management in the BigQuery JDBC driver by replacing raw Thread instances with Future<?> tasks managed by connection-scoped executor services. It introduces a configurable QueryProcessThreadCount property to bound the query executor pool, mitigating thread starvation and potential deadlocks. Feedback on these changes suggests dynamically referencing the configured thread pool limit in exception messages rather than hardcoding '100 threads', and dynamically adjusting the pool saturation warning message to point to the correct configuration property depending on which executor pool is saturating.

Comment on lines +862 to +866
if (ex instanceof OutOfMemoryError || ex instanceof RejectedExecutionException) {
throw new BigQueryJdbcException(
"Failed to execute query: Unable to allocate background threads to process the query results. Connection-scoped thread pool limit of 100 threads was reached or system is out of memory.",
ex);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The error message hardcodes the thread pool limit of 100 threads. Since this limit is now configurable via the QueryProcessThreadCount connection property, consider dynamically retrieving the configured limit using connection.getQueryProcessThreadCount() to avoid misleading users when a custom thread count is used.

      if (ex instanceof OutOfMemoryError || ex instanceof RejectedExecutionException) {
        throw new BigQueryJdbcException(
            String.format(
                "Failed to execute query: Unable to allocate background threads to process the query results. Connection-scoped thread pool limit of %d threads was reached or system is out of memory.",
                connection.getQueryProcessThreadCount()),
            ex);
      }

Comment on lines +1102 to +1106
if (e instanceof RejectedExecutionException || e instanceof OutOfMemoryError) {
throw new BigQueryJdbcException(
"Failed to execute query: Unable to allocate background threads to process the query results. Connection-scoped thread pool limit of 100 threads was reached or system is out of memory.",
e);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The error message hardcodes the thread pool limit of 100 threads. Since this limit is now configurable via the QueryProcessThreadCount connection property, consider dynamically retrieving the configured limit using connection.getQueryProcessThreadCount() to avoid misleading users when a custom thread count is used.

      if (e instanceof RejectedExecutionException || e instanceof OutOfMemoryError) {
        throw new BigQueryJdbcException(
            String.format(
                "Failed to execute query: Unable to allocate background threads to process the query results. Connection-scoped thread pool limit of %d threads was reached or system is out of memory.",
                connection.getQueryProcessThreadCount()),
            e);
      }

Comment on lines 154 to 159
if (queueSize >= warnThreshold) {
if (warningLogged.compareAndSet(false, true)) {
LOG.warning(
"Thread pool is saturating. Max pool size: %d, Active threads: %d, Queued tasks: %d. Consider increasing the thread count property.",
maxPoolSize, getActiveCount(), queueSize);
"[%s] Thread pool is saturating. Max pool size: %d, Active threads: %d, Queued tasks: %d. Consider increasing the metadataFetchThreadCount property.",
poolName, maxPoolSize, getActiveCount(), queueSize);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The warning message hardcodes metadataFetchThreadCount (which is also lowercase compared to the actual property name MetaDataFetchThreadCount). Since MdcThreadPoolExecutor is now shared between the metadata fetch pool and the query executor pool, this warning will be misleading if the query executor pool saturates.

Consider dynamically determining the property name based on the pool name to ensure the warning points to the correct configuration property.

      if (queueSize >= warnThreshold) {
        if (warningLogged.compareAndSet(false, true)) {
          String propertyName = "Query Executor Pool".equals(poolName)
              ? BigQueryJdbcUrlUtility.QUERY_PROCESS_THREAD_COUNT_PROPERTY_NAME
              : BigQueryJdbcUrlUtility.METADATA_FETCH_THREAD_COUNT_PROPERTY_NAME;
          LOG.warning(
              "[%s] Thread pool is saturating. Max pool size: %d, Active threads: %d, Queued tasks: %d. Consider increasing the %s property.",
              poolName, maxPoolSize, getActiveCount(), queueSize, propertyName);
        }

@Neenu1995 Neenu1995 changed the base branch from main to pr-1-executor-migration June 15, 2026 17:17
@Neenu1995 Neenu1995 closed this Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant