diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md deleted file mode 100644 index a07a9fe80e3c..000000000000 --- a/TROUBLESHOOTING.md +++ /dev/null @@ -1,255 +0,0 @@ -# Troubleshooting - -## Logging - -Our libraries use the Java logging API via `java.util.logging` package. -Configuring logging level reveals various facts that help your troubleshooting, -including: - -- The timing of underlying client-server communication -- Request and response message headers -- Verbose messages in underlying dependency libraries - -While there are various ways to configure the logging, -to quickly enable verbose logging for the Google Cloud Java libraries, create -a file `logging.properties` with the following content: - -``` -# run java program pointing to this properties file with the java arg -# -Djava.util.logging.config.file=path/to/logging.properties -handlers=java.util.logging.ConsoleHandler -java.util.logging.SimpleFormatter.format=%1$tF %1$tT,%1$tL %4$-8s %3$-50s - %5$s %6$s%n - -# --- ConsoleHandler --- -java.util.logging.ConsoleHandler.level=ALL -java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter -.level=INFO - -# --- Specify logging level for certain packages --- -# com.google.api is for HTTP 1.1 layer -com.google.api.level=ALL -# io.grpc is for gRPC + Netty layer -io.grpc.level=FINE -# com.google.auth is for authentication -com.google.auth.level=FINE - -# Example when we want to specify storge library's level. This works when -# the target Cloud library uses the logging API. -com.google.cloud.storage.level=INFO -``` - -and run your application with `-Djava.util.logging.config.file=path/to/logging.properties` -as the "VM argument" (not "Program argument"). - -If you use IntelliJ, you specify the VM argument in "Run/Debug Configuration": - -![Screenshot of IntelliJ configuration](docs/logging_vm_options.png) - -If the JVM of your program is running with the configuration correctly, you see -the FINE-level logging in your console. Example output: - -``` -2023-04-05 13:03:01,761 FINE com.google.auth.oauth2.DefaultCredentialsProvider - Attempting to load credentials from well known file: /usr/local/google/home/suztomo/.config/gcloud/application_default_credentials.json -2023-04-05 13:03:01,847 FINE io.grpc.ManagedChannelRegistry - Unable to find OkHttpChannelProvider -java.lang.ClassNotFoundException: io.grpc.okhttp.OkHttpChannelProvider - at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) - at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) - at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) - at java.base/java.lang.Class.forName0(Native Method) - at java.base/java.lang.Class.forName(Class.java:315) -... -``` - -Note that you may see many stacktraces printed there. -As long as it's "FINE" level, they are not errors. - -This is just one way to configure logging level. -For more details about Java logging API usage, see [Java Logging Overview]( -https://docs.oracle.com/javase/8/docs/technotes/guides/logging/overview.html). - -## ALPN is not configured properly - -If you see exceptions related to `ALPN is not configured properly`, such as: - -``` -Caused by: java.lang.IllegalArgumentException: ALPN is not configured properly. See https://github.com/grpc/grpc-java/blob/master/SECURITY.md#troubleshooting for more information. -``` - -Please use the [compatibility checker](https://github.com/googleapis/google-cloud-java/tree/master/google-cloud-util/google-cloud-compat-checker) to see if your environment is compatible with grpc-based clients. The incompatibility can mean that: -- You are not on a [supported platform](https://github.com/googleapis/google-cloud-java/#supported-platforms). -- There are classpath conflicts with `netty`. -- Or, you are seeing any of the conflicts specified in [gRPC Troubleshooting guide](https://github.com/grpc/grpc-java/blob/master/SECURITY.md#troubleshooting). - -## ClassNotFoundException, NoSuchMethodError, NoClassDefFoundError - -These errors are usually caused by having multiple versions or conflicting versions of the same dependency in the classpath. -Usually these dependency conflicts occur with `guava` or `protobuf-java`. - -There may be multiple sources for classpath conflicts: -- Multiple versions of the same transitive dependency in the dependency tree -- Your runtime classpath has different versions of dependencies than what you specified in the build - -For example, if you have a direct or a transitive dependency on Guava version 19.0, -and `google-cloud-java` uses Guava version 30.0, -then `google-cloud-java` could be using Guava methods that don't exist in Guava 19.0, -and cause `NoSuchMethodError`. - -Similarily, if your classpath has an older version of `protobuf-java`, -but `google-cloud-java` requires a newer version, - then you might see `NoClassDefFoundError` that fails to initialize `google-cloud-java` classes. For example: - -``` -java.lang.NoClassDefFoundError: Could not initialize class com.google.pubsub.v1.PubsubMessage$AttributesDefaultEntryHolder -``` - -### Validate the conflict - -Check the dependency tree to see if you have multiple versions of the same dependencies: - -``` -$ mvn dependency:tree -``` - -Look for versions of potentially conflicting dependencies like `guava`, `protobuf-java`, etc. - -If you experience the error only during runtime, then your runtime environment -might be introducing conflicting JARs into your runtime classpath. A typical case -is that Hadoop, Spark, or other server software that your application runs on -has conflicting versions `netty`, `guava`, or `protobuf-java` JARs in the classpath. - -### Detecting conflicts during build - -To detect dependency linkage errors at compile time, add the -[Linkage Checker Enforcer Rule](https://github.com/GoogleCloudPlatform/cloud-opensource-java/tree/master/enforcer-rules) -in your pom.xml: - -``` - - org.apache.maven.plugins - maven-enforcer-plugin - 3.0.0-M3 - - - com.google.cloud.tools - linkage-checker-enforcer-rules - 1.5.7 - - - - - enforce-linkage-checker - - verify - - enforce - - - - - - - - - -``` - -There is no way to detect runtime classpath conflicts though. You need to be fully -aware of which JARs/classes are included in the runtime classpath -as every server environment is different. - -### Resolving the conflict - -There are different strategies to resolve conflicts, but you must understand the root cause of the conflicts. For example: - -- If you have control over the dependency tree, upgrade - offending dependencies (for example, upgrading Guava version). This is the - least hackish approach, but it is a lot of work that can require multiple releases - of multiple libraries to sync everything up. -- If you can't modify and push new versions of your dependencies, import - `com.google.cloud:libraries-bom:25.1.0` (or a more recent version) and use that to - select consistent dependency versions. This is the easiest route. - For example, this is how you can depend on consistent versions of Guava and - `com.google.cloud:google-cloud-storage` without explicitly setting the version of either one: - -``` - ... - - - - com.google.cloud - libraries-bom - 25.1.0 - pom - import - - - - ... - - - com.google.cloud - google-cloud-storage - - - com.google.guava - guava - - ... - - ... -``` - -- The release note of the libraries-bom shows the compatible dependency libraries. - For example, https://github.com/googleapis/java-cloud-bom/releases/tag/v26.31.0 shows - - > These client libraries are built with the following Java libraries: - > - > - Guava: 32.1.3-jre - > - Protobuf Java: 3.25.2 - > - Google Auth Library: 1.22.0 - > - Google API Client: 2.2.0 - > - gRPC: 1.61.0 - > - GAX: 2.41.0 - > - Google Cloud Core: 2.31.0 - - By examining the dependency graph of your project (`mvn dependency:tree -Dverbose`, - `gradle dependencies`, or `sbt dependencyTree`), you may find some of the dependencies - having unexpected versions. They might cause dependency conflicts. - -- If changing dependency versions causes other failures, - consider [shading dependencies](https://maven.apache.org/plugins/maven-shade-plugin/) - that conflict with Google Cloud Java libraries. - - For example, to shade `guava` and `protobuf-java`: - -``` - - org.apache.maven.plugins - maven-shade-plugin - ... - - - package - - shade - - - false - - - - com.google.protobuf - myapp.shaded.com.google.protobuf - - - - com.google.common - myapp.shaded.com.google.common - - - - - - -``` diff --git a/docs/client_retries.md b/docs/client_retries.md deleted file mode 100644 index d69550bbba27..000000000000 --- a/docs/client_retries.md +++ /dev/null @@ -1,327 +0,0 @@ -# Client Side Retries -Client Libraries use retries to handle unexpected, transient failures (i.e. server is temporarily unavailable). -Multiple attempts, hopefully, will result in a successful response from the server. - -Default retry values are selected by the team operating the cloud service. These retry values are configured -per RPC. A service *may* choose to only enable retries for a subset of RPCs. It is possible that each RPC for -a service is configured differently. - -## Retry Parameters -Client libraries have two types of retry parameters to configure: -1. Retry Status Code: Set of status codes to retry on -2. Retry Timeout/ Attempt Bounds: Configurable [RetrySettings](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings) to define the bounds - -### Default RPC Retry Configuration Location -The default retry configurations are defined in the generated `{Client}StubSettings` file. Using the ExportAssets RPC in -Java-Asset v3.64.0 as an example, the default retry configurations are defined in the following places: -
-- Retry Status Codes are configured [here](https://github.com/googleapis/google-cloud-java/blob/d9da511b4b56302e509abe8b2d919a15ea7dcae7/java-asset/google-cloud-asset/src/main/java/com/google/cloud/asset/v1/stub/AssetServiceStubSettings.java#L1058-L1082). Example: - ```java - ImmutableMap.Builder> definitions = ImmutableMap.builder(); - definitions.put("no_retry_0_codes", ImmutableSet.copyOf(Lists.newArrayList())); - // ... More StatusCode configurations - RETRYABLE_CODE_DEFINITIONS = definitions.build(); - ``` - -- Retry parameters are configured [here](https://github.com/googleapis/google-cloud-java/blob/40ca7a8a267ca29837f9a6eceb3933c73551ac11/java-asset/google-cloud-asset/src/main/java/com/google/cloud/asset/v1/stub/AssetServiceStubSettings.java#L1105-L1172). Example: - ```java - ImmutableMap.Builder definitions = ImmutableMap.builder(); - RetrySettings settings = null; - settings = - RetrySettings.newBuilder() - .setInitialRpcTimeoutDuration(Duration.ofMillis(60000L)) - .setRpcTimeoutMultiplier(1.0) - .setMaxRpcTimeoutDuration(Duration.ofMillis(60000L)) - .setTotalTimeoutDuration(Duration.ofMillis(60000L)) - .build(); - definitions.put("no_retry_0_params", settings); - // ... More RetrySettings configurations - RETRY_PARAM_DEFINITIONS = definitions.build(); - ``` - -- The configurations above are mapped to the RPC [here](https://github.com/googleapis/google-cloud-java/blob/40ca7a8a267ca29837f9a6eceb3933c73551ac11/java-asset/google-cloud-asset/src/main/java/com/google/cloud/asset/v1/stub/AssetServiceStubSettings.java#L1325-L1491). Example: - ```java - builder - .exportAssetsSettings() - .setRetryableCodes(RETRYABLE_CODE_DEFINITIONS.get("no_retry_0_codes")) - .setRetrySettings(RETRY_PARAM_DEFINITIONS.get("no_retry_0_params")); - ``` - -## Client Library Retry Concepts -Enabling retries allow an RPC multiple attempts to try and achieve a successful call. A successful call -is a response from a server that returns an `OK` Status Code (from gRPC) or a `2xx` Status Code (from HttpJson). - -### Attempt vs Operation -Take a sample RetrySettings configuration -```java -settings = - RetrySettings.newBuilder() - .setInitialRetryDelayDuration(Duration.ofMillis(100L)) - .setRetryDelayMultiplier(1.3) - .setMaxRetryDelayDuration(Duration.ofMillis(60000L)) - .setInitialRpcTimeoutDuration(Duration.ofMillis(60000L)) - .setRpcTimeoutMultiplier(1.0) - .setMaxRpcTimeoutDuration(Duration.ofMillis(60000L)) - .setTotalTimeoutDuration(Duration.ofMillis(60000L)) - .build(); -``` -The configuration above modifies the retry settings for both an RPC's attempt and operation. An RPC attempt is the -individual attempt made and an RPC operation is collection of all attempts made. A single RPC invocation will -have one or more attempts in a single operation. - -Individual RPC Bounds (an attempt) are controlled by the following settings: -- [setInitialRetryDelayDuration](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings.Builder#com_google_api_gax_retrying_RetrySettings_Builder_setInitialRetryDelayDuration_java_time_Duration_): Delay before the first attempt -- [setRetryDelayMultiplier](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings.Builder#com_google_api_gax_retrying_RetrySettings_Builder_setRetryDelayMultiplier_double_) - Delay multiplier applied between each attempt -- [setMaxRetryDelayDuration](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings.Builder#com_google_api_gax_retrying_RetrySettings_Builder_setMaxRetryDelayDuration_java_time_Duration_) - Max Delay possible for an attempt -- [setInitialRpcTimeoutDuration](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings.Builder#com_google_api_gax_retrying_RetrySettings_Builder_setInitialRpcTimeoutDuration_java_time_Duration_) - Timeout for the first attempt -- [setRpcTimeoutMultiplier](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings.Builder#com_google_api_gax_retrying_RetrySettings_Builder_setRpcTimeoutMultiplier_double_) - Timeout multiplier applied between each attempt -- [setMaxRpcTimeoutDuration](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings.Builder#com_google_api_gax_retrying_RetrySettings_Builder_setMaxRpcTimeoutDuration_java_time_Duration_) - Max Timeout possible for an attempt - -Total RPC Bounds (an operation) are controlled by the following settings: -- [setTotalTimeout](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings.Builder#com_google_api_gax_retrying_RetrySettings_Builder_setTotalTimeoutDuration_java_time_Duration_) - Total timeout allowed the entire operation -- [setMaxAttempts](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings.Builder#com_google_api_gax_retrying_RetrySettings_Builder_setMaxAttempts_int_) - Max number of attempts allowed - -### When is an RPC retried -An RPC will be retried when _both_ of the following scenarios occur: -- Non-successful status code is received by the library and the status code is marked as retryable* -- An RPC invocation exceeds the individual RPC bounds, but still falls within total RPC bounds** - -Note: If only one (or neither) of the scenarios above are true, then the RPC will _NOT_ be retried. -i.e. If the total timeout has not been exceeded, but the latest attempt receives a non-retryable status code. - -*The client library will check RPC's list of retryable status codes and mark a status code accordingly. -It is marked as retryable if the response's status code matches with any of the RPC's configured retryable status -codes. - -**When configuring the RPC bounds, you may configure the bounds for each attempt as well as the -total RPC's bounds. The retry algorithm will ensure that an individual attempt's bounds falls within -the total RPC's bounds. - -### Exponential Backoff -Exponential backoff will retry requests with an increasing delay between each retry attempt. This retry delay value -can be capped with a maximum retry delay value. - -For example, the following retry configurations may result in the following delay times: -``` -Initial Retry Delay: 100ms -Retry Delay Multiplier: 2.0 -Max Retry Delay: 500ms -``` -- Attempt 1: Delay 100ms -- Attempt 2: Delay 200ms -- Attempt 3: Delay 400ms -- Attempt 4: Delay 500ms -- ... -- Attempt X: Delay 500ms - -Note: The example above is configured to retry indefinitely. Do set an RPC operation cap by setting the TotalTimeout -and/or MaxAttempts value. - -### Jitter -Jitter is added variance via randomness to spread out when the RPCs are invoked. Google Cloud Client Libraries -always enable jitter for retries. This is to help spread out the retry attempts without overwhelming the server. - -The jitter random value is computed as the retry delay. Before each attempt, the retry algorithm will compute a random -value between `[1, RETRY_DELAY]`. This computed value is the *approximate* delay before the request is sent to the -server. - -For example, the following retry configurations utilizes Jitter and Exponential Backoff, which may result in the -following delays times: -``` -Initial Retry Delay: 100ms -Retry Delay Multiplier: 2.0 -Max Retry Delay: 500ms -``` -- Attempt 1: Delay a random value between `[1, 100]`ms -- Attempt 2: Delay a random value between `[1, 200]`ms -- Attempt 3: Delay a random value between `[1, 400]`ms -- Attempt 4: Delay a random value between `[1, 500]`ms -- ... -- Attempt X: Delay a random value between `[1, 500]`ms - -Note: The example above is configured to retry indefinitely. Do set an RPC operation cap by setting the TotalTimeout -and/or MaxAttempts value. - -## Retry Examples -The following examples below show the behavior of some retry configurations. -
-Note: These examples below assume that jitter is disabled. The retry delay is computed to be the attempt's maximum -value at the end of each attempt. - -### No Retry -```java -RetrySettings defaultNoRetrySettings = - RetrySettings.newBuilder() - // Use the default configurations for other settings - .setTotalTimeoutDuration(Duration.ofMillis(5000L)) - // Explicitly set retries as disabled (maxAttempts == 1) - .setMaxAttempts(1) - .build(); -``` -Alternatively, this same behavior can be configured with -```java -RetrySettings defaultNoRetrySettings = - RetrySettings.newBuilder() - .setLogicalTimeoutDuration(Duration.ofMillis(5000L)) - .build(); -``` - -The following table shows the attempts: - -| Attempt Number | RPC Timeout | Retry Delay | Call Invoked | Call Ended | -|---------------- |------------- |------------- |-------------- |----------- | -| 1 | 5000ms | 0ms | 0ms | 5000ms | - -### Retry Examples -#### Example 1 -```java -RetrySettings.newBuilder() - .setInitialRetryDelayDuration(Duration.ofMillis(200L)) - .setRetryDelayMultiplier(2.0) - .setMaxRetryDelayDuration(Duration.ofMillis(500L)) - .setInitialRpcTimeoutDuration(Duration.ofMillis(1500L)) - .setRpcTimeoutMultiplier(2.0) - .setMaxRpcTimeoutDuration(Duration.ofMillis(3000L)) - .setTotalTimeoutDuration(Duration.ofMillis(5000L)) - .build(); -``` - -The following table shows the attempts: - -| Attempt Number | RPC Timeout | Retry Delay | Call Invoked | Call Ended | -|------------------------- |------------- |------------- |-------------- |------------ | -| 1 | 1500ms | 0ms | 0ms | 1500ms | -| 2 (Retry) | 3000ms | 200ms | 1700ms | 4700ms | -| 3 (Retry Not Attempted) | - | 400ms | - | - | -The third retry attempt is not attempted because the computed retry delay (400ms) would invoke the -RPC after the total timeout (400 + 4700 > 5000). The RPC will return a failure message after 4700ms. - -#### Example 2 -This example is similar to Example #1, but has a longer total timeout to showcase an additional -retry attempt and the capped RPC Timeout for the last retry attempt. -```java -RetrySettings.newBuilder() - .setInitialRetryDelayDuration(Duration.ofMillis(200L)) - .setRetryDelayMultiplier(2.0) - .setMaxRetryDelayDuration(Duration.ofMillis(500L)) - .setInitialRpcTimeoutDuration(Duration.ofMillis(1500L)) - .setRpcTimeoutMultiplier(2.0) - .setMaxRpcTimeoutDuration(Duration.ofMillis(3000L)) - .setTotalTimeoutDuration(Duration.ofMillis(10000L)) - .build(); -``` - -The following table shows the attempts: - -| Attempt Number | RPC Timeout | Retry Delay | Call Invoked | Call Ended | -|---------------- |------------- |------------- |-------------- |------------ | -| 1 | 1500ms | 0ms | 0ms | 1500ms | -| 2 (Retry) | 3000ms | 200ms | 1700ms | 4700ms | -| 3 (Retry) | 4900ms | 400ms | 5100ms | 10000ms | - -The third retry’s RPC Timeout value is limited due the Total Timeout value. Using the multiplier (2.0) with the -previous timeout value (3000ms) should result in an RPC Timeout of 6000ms. However, the RPC Timeout should not -exceed the Total Timeout and is reduced to be the "time left" (10000 - 5100 = 4900). - -#### Example 3 -```java -RetrySettings defaultRetrySettings = - RetrySettings.newBuilder() - .setInitialRetryDelayDuration(Duration.ofMillis(200L)) - .setRetryDelayMultiplier(2.0) - .setMaxRetryDelayDuration(Duration.ofMillis(500L)) - .setInitialRpcTimeoutDuration(Duration.ofMillis(500L)) - .setRpcTimeoutMultiplier(2.0) - .setMaxRpcTimeoutDuration(Duration.ofMillis(2000L)) - .setTotalTimeoutDuration(Duration.ofMillis(4000L)) - .build(); -``` - -The following table shows the attempts: - -| Attempt Number | RPC Timeout | Retry Delay | Call Invoked | Call Ended | -|---------------- |------------- |------------- |-------------- |------------ | -| 1 | 500ms | 0ms | 0ms | 500ms | -| 2 (Retry) | 1000ms | 200ms | 700ms | 1700ms | -| 3 (Retry) | 1900ms | 400ms | 2100ms | 4000ms | - -Another example where the RPC Timeout is capped to not exceed the total timeout. - -## How to configure a custom retry parameters for an RPC -The following example is using the Java-Asset client library: - -1. Create the RetrySettings class with your custom configurations -```java -RetrySettings customRetrySettings = - RetrySettings.newBuilder() - // ... Retry Configurations - .build(); -RetrySettings customRetrySettings2 = - RetrySettings.newBuilder() - // ... Retry Configurations - .build(); -``` -2. Create the StubSettings.Builder for your client and configure it for the RPC -```java -AssetServiceStubSettings.Builder assetStubSettingsBuilder = AssetServiceStubSettings.newBuilder(); -assetStubSettingsBuilder - .exportAssetsSettings() - // Set your custom Retry Settings - .setRetrySettings(customRetrySettings) - // Set your custom Retryable Codes - .setRetryableCodes(ImmutableSet.of(StatusCode.Code.DEADLINE_EXCEEDED)); -``` -The code snippet above is setting custom retry configurations for AssetServiceClient's ExportAssets -RPC. It configures the ExportAssets RPC to use the retry settings configured in `customRetrySettings` -and sets the retryable codes to be `DEADLINE_EXCEEDED`. - -3. Create the Settings for the Client -```java -AssetServiceSettings assetSettings = AssetServiceSettings.create(assetStubSettingsBuilder.build()); -``` -4.Create the Client with the Settings -```java -try (AssetServiceClient assetClient = AssetServiceClient.create(assetSettings)) { - ... -} -``` - -Repeat Step #2 for each RPC that you want to configure. For example: -```java -AssetServiceStubSettings.Builder assetStubSettingsBuilder = AssetServiceStubSettings.newBuilder(); - -// Modify the retry params for ExportAssets RPC -assetStubSettingsBuilder - .exportAssetsSettings() - .setRetrySettings(customRetrySettings) - .setRetryableCodes(ImmutableSet.of(StatusCode.Code.DEADLINE_EXCEEDED)); - -// Modify the retry params for ListAssets RPC -assetStubSettingsBuilder - .listAssetsSettings() - .setRetrySettings(customRetrySettings2) - .setRetryableCodes(ImmutableSet.of(StatusCode.Code.UNAVAILABLE)); -``` - -## FAQ -### I expected X retry attempts, but it attempted Y times -Unless you explicitly specify the number of max attempts (along with disabling the timeout configurations), -you may not consistently see the same number of retry attempts made. [Jitter's](#jitter) random values for -RPC delay make it difficult predict when the request is actually sent. - -### The RPC returned a failure before the Total Timeout value was reached -The retry algorithm will calculate the jittered retry delay value during each retry attempt. The calculated -retry delay will be scheduled to run in the future (i.e. `currentTime() + jitteredRetryDelay`). If that attempt's -scheduled attempt time exceeds the total timeout, the "final" retry attempt will not be made. - -See this [example](#example-1) as an example of this behavior. - -### I configured custom settings and am seeing quota issues -You may have configured the RetrySettings to run too aggressively. The default retry values are chosen by -the team operation the service. - -Consider increasing the retry delay (initial retry delay and retry multiplier) so that the retry attempts -are spaced out and less frequent. Note that this *may* result in a slower response. - -Your use case may require a quicker response and/or more frequent retry attempts. If that is the case, try to -increase the quota limits. \ No newline at end of file diff --git a/docs/logging_vm_options.png b/docs/logging_vm_options.png deleted file mode 100644 index 0109fd0d0d86..000000000000 Binary files a/docs/logging_vm_options.png and /dev/null differ