Doc for new config

Author: ludoch

runtime/runtime_impl_jetty121/API_CLIENTS.md

@@ -1,3 +1,18 @@
+<!--
+ Copyright 2021 Google LLC
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+     https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
 # App Engine API Client Configuration
 
 The App Engine Java runtime communicates with Google Cloud APIs (such as
@@ -16,21 +31,24 @@ the [Eclipse Jetty](https://eclipse.dev/jetty/) HTTP client and is optimized for
 high performance and efficient connection management.
 
 By default, the client is configured to allow a maximum of 100 concurrent
-connections to API backends. This limit helps prevent memory exhaustion on
-smaller App Engine instance types (like F1 or F2) and avoids overwhelming
-backend services during sudden traffic spikes or high rates of failing requests
-with aggressive retry logic. If the connection limit is reached, subsequent
-requests are queued until a connection becomes available.
+threads and 100 concurrent connections for API calls. These limits help
+prevent memory exhaustion on smaller App Engine instance types (like F1 or F2)
+and avoid overwhelming backend services during sudden traffic spikes or high
+rates of failing requests with aggressive retry logic. If the thread limit is
+reached, subsequent requests are queued until a thread becomes available.
 
 ### Configuration
 
 You can configure the Jetty client using the following environment variables
 and system properties:
 
 *   **`APPENGINE_API_MAX_CONNECTIONS`** (Environment Variable): Sets the
-    maximum number of concurrent connections for API calls. If you observe API
-    call latency or connection-related errors under high load, you might
-    consider adjusting this value.
+    maximum number of concurrent connections in the HTTP client pool.
+    *   Default: `100`
+*   **`APPENGINE_API_MAX_THREADS`** (Environment Variable): Sets the
+    maximum number of concurrent threads for executing API calls. If unset,
+    this also defaults to 100. This is the most direct way to control API
+    call throughput and prevent backend overload.
     *   Default: `100`
 *   **`APPENGINE_API_CALLS_IDLE_TIMEOUT_MS`** (Environment Variable): Sets
     the idle timeout in milliseconds for connections in the connection pool.
@@ -66,6 +84,9 @@ traditional thread pool model. In this mode, you can control the thread pool
 size using an environment variable:
 
 *   **`APPENGINE_API_MAX_CONNECTIONS`** (Environment Variable): Sets the
+    maximum number of concurrent connections in the HTTP client pool.
+    *   Default: `100`
+*   **`APPENGINE_API_MAX_THREADS`** (Environment Variable): Sets the
     maximum number of threads in the pool for handling API calls. This limits
     the number of concurrent API calls, similar to how it works for the Jetty
     client. This variable is **ignored** if `appengine.api.use.virtualthreads`
@@ -101,19 +122,29 @@ configure the number of retry attempts using a system property:
 </system-properties>
 ```
 
+## Recommended value for Jetty 12.1 / Java 25
+The Java 25 runtime environment is highly performant, and in rare cases of very high throughput,
+this can lead to backend services (like Datastore) being temporarily overloaded,
+which may result in exceptions like `DatastoreFailureException: Internal Datastore Error`.
+If you encounter such issues, you can throttle the rate of API calls by reducing
+the maximum number of concurrent threads used by the API client.
+We recommend starting with a lower value and adjusting as needed:
+`APPENGINE_API_MAX_THREADS=50`
+
 ## Configuring via `appengine-web.xml`
 
 You can set these options by adding `<env-variables>` and
 `<system-properties>` sections to your `appengine-web.xml` file.
 
 For example, to switch to the JDK client, enable virtual threads, increase
-the connection limit to 200, and set Datastore `beginTransaction` retries to 3,
+the thread limit to 200, and set Datastore `beginTransaction` retries to 3,
 you would add:
 
 ```xml
 <env-variables>
   <env-var name="APPENGINE_API_CALLS_USING_JDK_CLIENT" value="true" />
   <env-var name="APPENGINE_API_MAX_CONNECTIONS" value="200" />
+  <env-var name="APPENGINE_API_MAX_THREADS" value="200" />
 </env-variables>
 <system-properties>
   <property name="appengine.api.use.virtualthreads" value="true" />