feat(bigquery-jdbc): add queryProcessThreadCount connection property#13480
feat(bigquery-jdbc): add queryProcessThreadCount connection property#13480Neenu1995 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request refactors the background thread management in the BigQuery JDBC driver by replacing raw Thread instances with Future<?> tasks managed by connection-scoped executor services. It introduces a configurable QueryProcessThreadCount property to bound the query executor pool, mitigating thread starvation and potential deadlocks. Feedback on these changes suggests dynamically referencing the configured thread pool limit in exception messages rather than hardcoding '100 threads', and dynamically adjusting the pool saturation warning message to point to the correct configuration property depending on which executor pool is saturating.
| if (ex instanceof OutOfMemoryError || ex instanceof RejectedExecutionException) { | ||
| throw new BigQueryJdbcException( | ||
| "Failed to execute query: Unable to allocate background threads to process the query results. Connection-scoped thread pool limit of 100 threads was reached or system is out of memory.", | ||
| ex); | ||
| } |
There was a problem hiding this comment.
The error message hardcodes the thread pool limit of 100 threads. Since this limit is now configurable via the QueryProcessThreadCount connection property, consider dynamically retrieving the configured limit using connection.getQueryProcessThreadCount() to avoid misleading users when a custom thread count is used.
if (ex instanceof OutOfMemoryError || ex instanceof RejectedExecutionException) {
throw new BigQueryJdbcException(
String.format(
"Failed to execute query: Unable to allocate background threads to process the query results. Connection-scoped thread pool limit of %d threads was reached or system is out of memory.",
connection.getQueryProcessThreadCount()),
ex);
}| if (e instanceof RejectedExecutionException || e instanceof OutOfMemoryError) { | ||
| throw new BigQueryJdbcException( | ||
| "Failed to execute query: Unable to allocate background threads to process the query results. Connection-scoped thread pool limit of 100 threads was reached or system is out of memory.", | ||
| e); | ||
| } |
There was a problem hiding this comment.
The error message hardcodes the thread pool limit of 100 threads. Since this limit is now configurable via the QueryProcessThreadCount connection property, consider dynamically retrieving the configured limit using connection.getQueryProcessThreadCount() to avoid misleading users when a custom thread count is used.
if (e instanceof RejectedExecutionException || e instanceof OutOfMemoryError) {
throw new BigQueryJdbcException(
String.format(
"Failed to execute query: Unable to allocate background threads to process the query results. Connection-scoped thread pool limit of %d threads was reached or system is out of memory.",
connection.getQueryProcessThreadCount()),
e);
}| if (queueSize >= warnThreshold) { | ||
| if (warningLogged.compareAndSet(false, true)) { | ||
| LOG.warning( | ||
| "Thread pool is saturating. Max pool size: %d, Active threads: %d, Queued tasks: %d. Consider increasing the thread count property.", | ||
| maxPoolSize, getActiveCount(), queueSize); | ||
| "[%s] Thread pool is saturating. Max pool size: %d, Active threads: %d, Queued tasks: %d. Consider increasing the metadataFetchThreadCount property.", | ||
| poolName, maxPoolSize, getActiveCount(), queueSize); | ||
| } |
There was a problem hiding this comment.
The warning message hardcodes metadataFetchThreadCount (which is also lowercase compared to the actual property name MetaDataFetchThreadCount). Since MdcThreadPoolExecutor is now shared between the metadata fetch pool and the query executor pool, this warning will be misleading if the query executor pool saturates.
Consider dynamically determining the property name based on the pool name to ensure the warning points to the correct configuration property.
if (queueSize >= warnThreshold) {
if (warningLogged.compareAndSet(false, true)) {
String propertyName = "Query Executor Pool".equals(poolName)
? BigQueryJdbcUrlUtility.QUERY_PROCESS_THREAD_COUNT_PROPERTY_NAME
: BigQueryJdbcUrlUtility.METADATA_FETCH_THREAD_COUNT_PROPERTY_NAME;
LOG.warning(
"[%s] Thread pool is saturating. Max pool size: %d, Active threads: %d, Queued tasks: %d. Consider increasing the %s property.",
poolName, maxPoolSize, getActiveCount(), queueSize, propertyName);
}
No description provided.