fix: ensure client keepalive < server keepalive to avoid client keepalive desync errors (#1555)

uvicorn closes idle keep-alive connections after 5s by default
([ref](https://uvicorn.dev/settings/#timeouts))

while the aiohttp client keeps them pooled for 15s by default
([ref](https://docs.aiohttp.org/en/stable/client_reference.html#connectors)),
causing ServerDisconnectedError when the client reuses a socket the
server already closed.

This PR set the client TCPConnector `keepalive_timeout` to 15s and the
uvicorn `timeout_keep_alive` to 30s so pooled sockets are guaranteed
live when reused.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

Author: ananthsub

nemo_gym/server_utils.py

@@ -110,6 +110,7 @@ def set_global_aiohttp_client(cfg: GlobalAIOHTTPAsyncClientConfig) -> ClientSess
         connector=TCPConnector(
             limit=cfg.global_aiohttp_connector_limit // num_workers,
             limit_per_host=cfg.global_aiohttp_connector_limit_per_host // num_workers,
+            keepalive_timeout=15.0,
         ),
         timeout=ClientTimeout(),
         cookie_jar=DummyCookieJar(),
@@ -672,6 +673,8 @@ def filter(self, record: LogRecord) -> bool:
             timeout_graceful_shutdown=0.5,
             # Some workers may take a while for imports and setup_webserver.
             timeout_worker_healthcheck=30,
+            # Ensure server keepalive > client keepalive
+            timeout_keep_alive=30,
         )
 
         if server.config.num_workers and server.config.num_workers > 1: