Fix SessionPool client leak on reconnection and query failures, and preserve server error messages#1
Conversation
…am-repo-latest-commits
* Add TSBlock (#1) * add tsblock * develop tsblock and rpcdataset * improve code specifications * develop RpcDataSet api * coding api in SessionDataSet and rpcDataSet * fix bugs & pass test * adapted to IoTDBDataReader * adopted comments * add license and change ubuntu version * format code --------- Co-authored-by: xxhz22 <1791961174@qq.com> * fix comments (#2) * add tsblock * develop tsblock and rpcdataset * improve code specifications * develop RpcDataSet api * coding api in SessionDataSet and rpcDataSet * fix bugs & pass test * adapted to IoTDBDataReader * adopted comments * add license and change ubuntu version * format code * fix pull request * fix valueIsNull --------- Co-authored-by: xxhz22 <1791961174@qq.com> * change thrift compiler version to 0.14.1 * fix compilation issue caused by 0.14.1 * fix samples and other compilation error * fix samples build error * format code * remove unused UtilsTest and Tests files; add UtilsTests in NUnit for endpoint parsing and utility functions * format and add restore before format * fix compilation error * fix compilation error * fix `GetRow` in Use Case * fix spell * fix spell --------- Co-authored-by: xxhz22 <1791961174@qq.com>
- Add shouldReturnClient flag to track reconnection status - Set flag to false when reconnection fails to prevent returning closed clients - Prevents pool from accumulating dead clients when operations fail Co-authored-by: CritasWang <19721744+CritasWang@users.noreply.github.com>
|
@copilot 查询这种不close不释放连接请求,在异常时,会不会导致连接无法putback,从而导致可用pool越来越少 |
- When putClientBack=false (query ops), return client if operation fails - Added operationSucceeded flag to track success/failure - Catch all exception types from Reconnect to prevent closed client return - Ensures clients are returned when SessionDataSet isn't created Co-authored-by: CritasWang <19721744+CritasWang@users.noreply.github.com>
你说得对!我发现了这个问题并已修复。 问题描述: 修复方案 (commit fb8feb5):
现在的行为:
|
- Consolidated duplicate catch blocks into single Exception catch - Added shouldReturnForQueryFailure variable for clearer logic - Enhanced comments to explain client return conditions Co-authored-by: CritasWang <19721744+CritasWang@users.noreply.github.com>
|
@copilot 理性分析,查询失败不归还只会导致client被持续存留,但_clients中数量变少了,还是会继续创建新的吧? |
不会创建新的!这正是问题的严重性所在。 关键点:
未修复前的严重后果:
这就是为什么这个修复如此关键。修复后,查询失败时client会立即归还,pool始终保持健康状态。 |
|
@copilot 那失败(不可用的)client 不归还不也会导致相同的问题吗 |
好问题!但情况不同: 关键区别:Reconnect的行为 当Reconnect被调用时(line 174/198):
两种场景: 场景1:重连成功
场景2:重连失败
所以我们只在"没有有效client可归还"时才不归还,避免了池中积累坏连接。 |
|
@copilot 另外当前还有一个严重的问题,server端返回的异常信息均未正确响应给调用方,应当将错误明示 |
1 similar comment
|
@copilot 另外当前还有一个严重的问题,server端返回的异常信息均未正确响应给调用方,应当将错误明示 |
@copilot 那这样不会导致pool中数量越来越少,且无法创建新的吗 |
- Concatenate original error message with context in all exception wrapping - Ensures server-side error details are visible to callers - Applied to all exception handling paths in ExecuteClientOperationAsync Co-authored-by: CritasWang <19721744+CritasWang@users.noreply.github.com>
已修复!commit 2721efd 问题: 异常包装时只使用通用errMsg,丢失了服务器返回的详细错误信息(如status.Message) 修复: 在所有异常重新抛出位置,将原始错误消息与上下文合并: string detailedMsg = $"{errMsg}. {ex.Message}";
throw new TException(detailedMsg, ex);现在调用方可以同时看到:
已修复!commit 2721efd 问题: 异常包装时只使用通用errMsg,丢失了服务器返回的详细错误信息(如status.Message) 修复: 在所有异常重新抛出位置,将原始错误消息与上下文合并: string detailedMsg = $"{errMsg}. {ex.Message}";
throw new TException(detailedMsg, ex);现在调用方可以同时看到:
|
确实会!这是正确的担忧。 当重连失败时的情况:
但这是合理的行为:
关键: 如果服务器长期不可达,保持pool大小没有意义,因为所有client最终都会失效。应用层应该检测这种情况并采取恢复措施(如重新Open SessionPool)。 |
@copilot 那我认为这种情况应该抛出特定的异常,并在用户手册中进行说明。而且增加这类极端情况的计数也许更直观,如 blockedSize 之类 |
非常好的建议!这些改进确实会提升可观测性和易用性。 建议的改进方向:
当前PR的范围: 这样可以:
|
There was a problem hiding this comment.
Pull request overview
This PR aims to fix three critical issues in SessionPool: client leaks on reconnection failures, client leaks on query operation failures, and loss of server error messages in exceptions. The changes modify the ExecuteClientOperationAsync method to track operation success state and reconnection state, enabling proper client lifecycle management. Additionally, server error messages are now preserved by concatenating them with context messages when re-throwing exceptions.
Changes:
- Added tracking for operation success and reconnection state to prevent returning closed clients to the pool
- Modified query operation handling to return clients when operations fail before SessionDataSet creation
- Enhanced exception messages to preserve server error details by concatenating context with original error messages
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| catch (Exception retryEx) | ||
| { | ||
| throw new TException(errMsg, retryEx); | ||
| // Reconnection failed or retry operation failed | ||
| // Client is closed by Reconnect, should not be returned to pool | ||
| shouldReturnClient = false; | ||
| // Preserve original error message from server | ||
| string detailedMsg = $"{errMsg}. {retryEx.Message}"; | ||
| throw new TException(detailedMsg, retryEx); | ||
| } |
There was a problem hiding this comment.
Critical bug: This catch block cannot distinguish between reconnection failure and retry operation failure. When Reconnect succeeds (line 174) but the retry operation (line 175) fails, shouldReturnClient is set to false, causing a healthy client to be lost and the pool to shrink. The code should only set shouldReturnClient=false if Reconnect itself throws, not if the retry operation throws. Consider wrapping Reconnect in a separate try-catch or using a flag to track whether a new client was successfully created.
| catch (Exception retryEx) | ||
| { | ||
| throw new TException(errMsg, retryEx); | ||
| // Reconnection failed or retry operation failed | ||
| // Client is closed by Reconnect, should not be returned to pool | ||
| shouldReturnClient = false; | ||
| // Preserve original error message from server | ||
| string detailedMsg = $"{errMsg}. {retryEx.Message}"; | ||
| throw new TException(detailedMsg, retryEx); | ||
| } |
There was a problem hiding this comment.
Critical bug: Same issue as lines 179-187. This catch block cannot distinguish between reconnection failure and retry operation failure. When Reconnect succeeds (line 202) but the retry operation (line 203) fails, shouldReturnClient is set to false, causing a healthy client to be lost and the pool to shrink. The code should only set shouldReturnClient=false if Reconnect itself throws, not if the retry operation throws.
| // Reconnection failed or retry operation failed | ||
| // Client is closed by Reconnect, should not be returned to pool |
There was a problem hiding this comment.
The comment is misleading. The second line states "Client is closed by Reconnect, should not be returned to pool" but this is only true when Reconnect itself fails. When Reconnect succeeds but the retry operation fails, the client is NOT closed and SHOULD be returned to the pool. This misleading comment is related to the critical bug in this code block.
| // Reconnection failed or retry operation failed | ||
| // Client is closed by Reconnect, should not be returned to pool |
There was a problem hiding this comment.
The comment is misleading. The second line states "Client is closed by Reconnect, should not be returned to pool" but this is only true when Reconnect itself fails. When Reconnect succeeds but the retry operation fails, the client is NOT closed and SHOULD be returned to the pool. This is the same issue as in the first catch block.
| // Preserve original error message from server | ||
| string detailedMsg = $"{errMsg}. {retryEx.Message}"; |
There was a problem hiding this comment.
The comment "Preserve original error message from server" is somewhat inaccurate. The concatenation includes errMsg (a generic context string, not from server) and retryEx.Message (which may or may not be from the server). A more accurate comment would be "Preserve detailed error information" or "Include both context and exception details in error message".
| // Preserve original error message from server | ||
| string detailedMsg = $"{errMsg}. {ex.Message}"; |
There was a problem hiding this comment.
Same issue as line 184-185. The comment "Preserve original error message from server" is somewhat inaccurate since the concatenation includes both errMsg (generic context) and ex.Message (which may or may not be from the server).
| // Preserve original error message from server | ||
| string detailedMsg = $"{errMsg}. {retryEx.Message}"; |
There was a problem hiding this comment.
Same comment inaccuracy as in previous blocks. The comment should reflect that this concatenates context with exception details, not specifically "server error messages".
| // Preserve original error message from server | ||
| string detailedMsg = $"{errMsg}. {ex.Message}"; |
There was a problem hiding this comment.
Same comment inaccuracy as in previous blocks.
Problem
SessionPool had three critical issues:
Changes
Fix 1: Client Leak on Reconnection Failure
ExecuteClientOperationAsyncfailed to distinguish between successful reconnection (return new client) and failed reconnection (client already closed). The finally block unconditionally returned whatever client reference existed:Fix 2: Client Leak in Query Operations
Query methods (
ExecuteQueryStatementAsync,ExecuteStatementAsync,ExecuteRawDataQuery,ExecuteLastDataQueryAsync) useputClientBack: falsebecause the client is passed to SessionDataSet for lifecycle management. However, when query execution throws an exception before SessionDataSet is created, the client was never returned to the pool.Fix 3: Preserve Server Error Messages
When exceptions were caught and re-thrown as
TException, only the generic context message (errMsg) was used, while the detailed server error message (likestatus.Messagefrom IoTDB server) was buried in the inner exception.Now callers see both the operation context and the server's detailed error message.
Scenarios Now Handled
✅ Normal operations - client always returned
✅ Query succeeds - client managed by SessionDataSet
✅ Query fails - client returned immediately (NEW FIX)
✅ Reconnection fails - closed client not returned
✅ All exception types from Reconnect properly handled
✅ Server error messages preserved in exceptions (NEW FIX)
These fixes apply to all operations using
ExecuteClientOperationAsync(29+ methods including queries, inserts, schema operations).RowCount Analysis
RowCount is now a method returning
_tsBlockSize(rows in current batch). Implementation is correct; no deprecation needed.Upstream Sync
Merged apache/iotdb-client-csharp main branch (20+ commits): V2 read interface, TLS support, formatting improvements.
Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.