Skip to content

fix(bigquery): retry cancel job and routine creation on transient HTTP 5xx errors #13416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jinseopkim0 wants to merge 8 commits into
base: main
Choose a base branch
from
+109 −16
Open

fix(bigquery): retry cancel job and routine creation on transient HTTP 5xx errors #13416

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f195928
fix(bigquery): wrap default ResultRetryAlgorithm to retry transient H…
jinseopkim0 Jun 10, 2026
4a1bcdb
fix(bigquery): selectively apply HTTP error retries to safe/idempoten…
jinseopkim0 Jun 12, 2026
7aab43a
refactor(bigquery): rename retry-related helper methods to reflect co…
jinseopkim0 Jun 12, 2026
9845a87
fix(bigquery): limit HTTP retries to safe operations and enable for c…
jinseopkim0 Jun 12, 2026
337e8b1
docs(bigquery): add javadoc comments to retry-wrapping helper methods
jinseopkim0 Jun 12, 2026
fdcde0a
docs(bigquery): update retry helper comments as suggested
jinseopkim0 Jun 12, 2026
dc01c5e
docs(bigquery): add javadoc comment to wrapDefaultAlgorithm
jinseopkim0 Jun 12, 2026
f6aa837
Merge branch 'main' into fix-cancel-job-b467066417
jinseopkim0 Jun 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 29 additions & 16 deletions ...-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import com.google.api.core.BetaApi;
import com.google.api.core.InternalApi;
import com.google.api.gax.paging.Page;
import com.google.api.gax.retrying.ResultRetryAlgorithm;
import com.google.api.services.bigquery.model.ErrorProto;
import com.google.api.services.bigquery.model.GetQueryResultsResponse;
import com.google.api.services.bigquery.model.QueryRequest;
Expand Down Expand Up @@ -396,7 +397,7 @@ public com.google.api.services.bigquery.model.Routine call() throws IOException
}
},
getOptions().getRetrySettings(),
getOptions().getResultRetryAlgorithm(),
getRetryAlgorithmWithHttpRetry(),

@lqiu96 lqiu96 Jun 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just use BigQueryRetryHelper.maybeWrapForHttpRetry for all the calls for consistency?

getOptions().getClock(),
EMPTY_RETRY_CONFIG,
getOptions().isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -592,7 +593,7 @@ public com.google.api.services.bigquery.model.Dataset call() throws IOException
}
},
getOptions().getRetrySettings(),
getOptions().getResultRetryAlgorithm(),
getRetryAlgorithmWithHttpRetry(),
getOptions().getClock(),
EMPTY_RETRY_CONFIG,
getOptions().isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -658,7 +659,7 @@ private static Page<Dataset> listDatasets(
}
},
serviceOptions.getRetrySettings(),
serviceOptions.getResultRetryAlgorithm(),
BigQueryRetryHelper.maybeWrapForHttpRetry(serviceOptions.getResultRetryAlgorithm()),
serviceOptions.getClock(),
EMPTY_RETRY_CONFIG,
serviceOptions.isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1131,7 +1132,7 @@ public com.google.api.services.bigquery.model.Table call() throws IOException {
}
},
getOptions().getRetrySettings(),
getOptions().getResultRetryAlgorithm(),
getRetryAlgorithmWithHttpRetry(),
getOptions().getClock(),
EMPTY_RETRY_CONFIG,
getOptions().isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1190,7 +1191,7 @@ public com.google.api.services.bigquery.model.Model call() throws IOException {
}
},
getOptions().getRetrySettings(),
getOptions().getResultRetryAlgorithm(),
getRetryAlgorithmWithHttpRetry(),
getOptions().getClock(),
EMPTY_RETRY_CONFIG,
getOptions().isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1249,7 +1250,7 @@ public com.google.api.services.bigquery.model.Routine call() throws IOException
}
},
getOptions().getRetrySettings(),
getOptions().getResultRetryAlgorithm(),
getRetryAlgorithmWithHttpRetry(),
getOptions().getClock(),
EMPTY_RETRY_CONFIG,
getOptions().isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1467,7 +1468,7 @@ public Tuple<String, Iterable<com.google.api.services.bigquery.model.Table>> cal
}
},
serviceOptions.getRetrySettings(),
serviceOptions.getResultRetryAlgorithm(),
BigQueryRetryHelper.maybeWrapForHttpRetry(serviceOptions.getResultRetryAlgorithm()),
serviceOptions.getClock(),
EMPTY_RETRY_CONFIG,
serviceOptions.isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1508,7 +1509,7 @@ public Tuple<String, Iterable<com.google.api.services.bigquery.model.Model>> cal
}
},
serviceOptions.getRetrySettings(),
serviceOptions.getResultRetryAlgorithm(),
BigQueryRetryHelper.maybeWrapForHttpRetry(serviceOptions.getResultRetryAlgorithm()),
serviceOptions.getClock(),
EMPTY_RETRY_CONFIG,
serviceOptions.isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1549,7 +1550,7 @@ private static Page<Routine> listRoutines(
}
},
serviceOptions.getRetrySettings(),
serviceOptions.getResultRetryAlgorithm(),
BigQueryRetryHelper.maybeWrapForHttpRetry(serviceOptions.getResultRetryAlgorithm()),
serviceOptions.getClock(),
EMPTY_RETRY_CONFIG,
serviceOptions.isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1725,7 +1726,7 @@ public TableDataList call() throws IOException {
}
},
serviceOptions.getRetrySettings(),
serviceOptions.getResultRetryAlgorithm(),
BigQueryRetryHelper.maybeWrapForHttpRetry(serviceOptions.getResultRetryAlgorithm()),
serviceOptions.getClock(),
EMPTY_RETRY_CONFIG,
serviceOptions.isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1802,7 +1803,7 @@ public com.google.api.services.bigquery.model.Job call() throws IOException {
}
},
getOptions().getRetrySettings(),
getOptions().getResultRetryAlgorithm(),
getRetryAlgorithmWithHttpRetry(),
getOptions().getClock(),
EMPTY_RETRY_CONFIG,
getOptions().isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1859,7 +1860,7 @@ public Tuple<String, Iterable<com.google.api.services.bigquery.model.Job>> call(
}
},
serviceOptions.getRetrySettings(),
serviceOptions.getResultRetryAlgorithm(),
BigQueryRetryHelper.maybeWrapForHttpRetry(serviceOptions.getResultRetryAlgorithm()),
serviceOptions.getClock(),
EMPTY_RETRY_CONFIG,
serviceOptions.isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -1914,7 +1915,7 @@ public Boolean call() throws IOException {
}
},
getOptions().getRetrySettings(),
getOptions().getResultRetryAlgorithm(),
getRetryAlgorithmWithHttpRetry(),
getOptions().getClock(),
EMPTY_RETRY_CONFIG,
getOptions().isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -2169,7 +2170,7 @@ public GetQueryResultsResponse call() throws IOException {
}
},
serviceOptions.getRetrySettings(),
serviceOptions.getResultRetryAlgorithm(),
BigQueryRetryHelper.maybeWrapForHttpRetry(serviceOptions.getResultRetryAlgorithm()),
serviceOptions.getClock(),
DEFAULT_RETRY_CONFIG,
serviceOptions.isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -2240,7 +2241,7 @@ public com.google.api.services.bigquery.model.Policy call() throws IOException {
}
},
getOptions().getRetrySettings(),
getOptions().getResultRetryAlgorithm(),
getRetryAlgorithmWithHttpRetry(),
getOptions().getClock(),
EMPTY_RETRY_CONFIG,
getOptions().isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -2334,7 +2335,7 @@ public com.google.api.services.bigquery.model.TestIamPermissionsResponse call()
}
},
getOptions().getRetrySettings(),
getOptions().getResultRetryAlgorithm(),
getRetryAlgorithmWithHttpRetry(),
getOptions().getClock(),
EMPTY_RETRY_CONFIG,
getOptions().isOpenTelemetryTracingEnabled(),
Expand Down Expand Up @@ -2411,4 +2412,16 @@ private static boolean isRetryErrorCodeHttpNotFound(BigQueryRetryHelperException
}
return false;
}

/**
* Helper to retrieve the retry algorithm wrapped for HTTP error retries.
*
* <p>This delegates to {@link BigQueryRetryHelper#maybeWrapForHttpRetry} to ensure safe
* conditional wrapping of the default algorithm while leaving custom user algorithms untouched.
*/
@SuppressWarnings("unchecked")
private <V> ResultRetryAlgorithm<V> getRetryAlgorithmWithHttpRetry() {
return BigQueryRetryHelper.maybeWrapForHttpRetry(
(ResultRetryAlgorithm<V>) getOptions().getResultRetryAlgorithm());
}
}
45 changes: 45 additions & 0 deletions ...ry/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryRetryHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
*/
package com.google.cloud.bigquery;

import com.google.api.client.http.HttpResponseException;
import com.google.api.core.ApiClock;
import com.google.api.gax.retrying.DirectRetryingExecutor;
import com.google.api.gax.retrying.ExponentialRetryAlgorithm;
Expand All @@ -23,6 +24,7 @@
import com.google.api.gax.retrying.RetrySettings;
import com.google.api.gax.retrying.RetryingExecutor;
import com.google.api.gax.retrying.RetryingFuture;
import com.google.api.gax.retrying.TimedAttemptSettings;
import com.google.api.gax.retrying.TimedRetryAlgorithm;
import com.google.cloud.RetryHelper;
import io.opentelemetry.api.trace.Span;
Expand Down Expand Up @@ -119,6 +121,49 @@ private static <V> V run(
return retryingFuture.get();
}

/**
* Conditionally wraps the provided retry algorithm with a wrapper that retries on transient HTTP
* errors.
*
* <p>Wrapping only occurs if the provided algorithm is the default {@link
* BigQueryBaseService#DEFAULT_BIGQUERY_EXCEPTION_HANDLER}. Custom user-defined retry algorithms
* are returned unmodified to preserve custom retry policies.
*/
static <V> ResultRetryAlgorithm<V> maybeWrapForHttpRetry(ResultRetryAlgorithm<V> algorithm) {
if (algorithm == BigQueryBaseService.DEFAULT_BIGQUERY_EXCEPTION_HANDLER) {
return wrapDefaultAlgorithm(algorithm);
}
return algorithm;
}

/**
* Wraps the default retry algorithm to additionally retry on transient HTTP status codes 500,
* 502, 503, and 504. Other retry decisions and timing logic are delegated back to the default
* algorithm.
*/
private static <V> ResultRetryAlgorithm<V> wrapDefaultAlgorithm(
ResultRetryAlgorithm<V> defaultAlgorithm) {
return new ResultRetryAlgorithm<V>() {
@Override
public TimedAttemptSettings createNextAttempt(
Throwable previousThrowable, V previousResponse, TimedAttemptSettings previousSettings) {
return defaultAlgorithm.createNextAttempt(
previousThrowable, previousResponse, previousSettings);
}
Comment thread
jinseopkim0 marked this conversation as resolved.

@Override
public boolean shouldRetry(Throwable previousThrowable, V previousResponse) {
if (previousThrowable instanceof HttpResponseException) {
int statusCode = ((HttpResponseException) previousThrowable).getStatusCode();
if (statusCode == 500 || statusCode == 502 || statusCode == 503 || statusCode == 504) {
return true;
}
}
return defaultAlgorithm.shouldRetry(previousThrowable, previousResponse);
}
};
}

public static class BigQueryRetryHelperException extends RuntimeException {

private static final long serialVersionUID = -8519852520090965314L;
Expand Down
35 changes: 35 additions & 0 deletions ...query/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/BigQueryImplTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;

import com.google.api.client.googleapis.json.GoogleJsonError;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.http.HttpHeaders;
import com.google.api.client.http.HttpResponseException;
import com.google.api.gax.paging.Page;
import com.google.api.services.bigquery.model.ErrorProto;
import com.google.api.services.bigquery.model.GetQueryResultsResponse;
Expand Down Expand Up @@ -935,6 +939,37 @@ void testGetTable() throws IOException {
.getTableSkipExceptionTranslation(PROJECT, DATASET, TABLE, EMPTY_RPC_OPTIONS);
}

@Test
void testGetTableFailureShouldRetryServerErrors() throws IOException {

@lqiu96 lqiu96 Jun 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a case where there is a custom retry config from the user?

GoogleJsonError error = new GoogleJsonError();
error.setMessage("Visibility check was unavailable. Please retry the request");
error.setCode(503);
GoogleJsonError.ErrorInfo errorInfo = new GoogleJsonError.ErrorInfo();
errorInfo.setReason("backendError");
error.setErrors(ImmutableList.of(errorInfo));

when(bigqueryRpcMock.getTableSkipExceptionTranslation(
PROJECT, DATASET, TABLE, EMPTY_RPC_OPTIONS))
.thenThrow(new GoogleJsonResponseException(serverErrorResponse(), error))
.thenReturn(TABLE_INFO_WITH_PROJECT.toPb());

bigquery =
options.toBuilder()
.setRetrySettings(ServiceOptions.getDefaultRetrySettings())
.build()
.getService();

Table table = bigquery.getTable(DATASET, TABLE);

assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table);
verify(bigqueryRpcMock, times(2))
.getTableSkipExceptionTranslation(PROJECT, DATASET, TABLE, EMPTY_RPC_OPTIONS);
}

private static HttpResponseException.Builder serverErrorResponse() {
return new HttpResponseException.Builder(503, "Service Unavailable", new HttpHeaders());
}

@Test
void testGetModel() throws IOException {
when(bigqueryRpcMock.getModelSkipExceptionTranslation(
Expand Down
Loading