Skip to content

Commit d489815

Browse files
Copilotfruch
andcommitted
Add comprehensive documentation for SSL connection error tests
Created detailed README documenting: - Background on the "Bad file descriptor" issue - Test structure and organization - How to run the tests - Implementation details - Contributing guidelines Co-authored-by: fruch <340979+fruch@users.noreply.github.com>
1 parent 887a1bd commit d489815

1 file changed

Lines changed: 129 additions & 0 deletions

File tree

tests/unit/SSL_CONNECTION_TESTS.md

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# SSL Connection Error Handling Tests
2+
3+
This directory contains comprehensive tests for SSL connection error handling, specifically addressing the "Bad file descriptor" errors that occur when nodes reboot while using client encryption.
4+
5+
## Background
6+
7+
### Issue
8+
When using client encryption and nodes reboot, the driver reported errors like:
9+
```
10+
cassandra.connection.ConnectionShutdown: [Errno 9] Bad file descriptor
11+
```
12+
13+
The issue occurred because:
14+
1. A connection is forcefully closed (e.g., due to node reboot)
15+
2. Parallel operations attempt to read/write to the closed socket
16+
3. This results in "Bad file descriptor" errors
17+
4. The original cause of the connection closure could be lost
18+
19+
See: https://github.com/scylladb/python-driver/issues/614
20+
21+
### Solution
22+
The driver already has proper error handling via the `last_error` mechanism in the `Connection` class. When a connection becomes defunct, the error that caused it is stored in `last_error` and included in subsequent `ConnectionShutdown` exception messages.
23+
24+
This test suite verifies that this mechanism works correctly across all SSL error scenarios.
25+
26+
## Test Files
27+
28+
### Unit Tests (`test_ssl_connection_errors.py`)
29+
14 unit tests that mock SSL connection failures:
30+
31+
1. **Basic SSL Socket Errors**
32+
- `test_ssl_socket_bad_file_descriptor_on_send`: Simulates EBADF during send
33+
- `test_ssl_socket_bad_file_descriptor_on_recv`: Simulates EBADF during recv
34+
- `test_ssl_socket_broken_pipe_error`: Simulates EPIPE (broken pipe)
35+
- `test_ssl_socket_connection_aborted_error`: Simulates ECONNABORTED
36+
- `test_ssl_socket_errno_enotconn`: Simulates ENOTCONN
37+
38+
2. **SSL-Specific Errors**
39+
- `test_ssl_connection_error_during_handshake`: SSL handshake failures
40+
- `test_ssl_unwrap_error_on_close`: SSL unwrap errors during close
41+
42+
3. **Race Condition Tests**
43+
- `test_concurrent_operations_on_closed_ssl_socket`: Multiple threads using closed socket
44+
- `test_parallel_send_on_defuncting_connection`: Thread trying to send while connection defuncts
45+
- `test_node_reboot_scenario`: Complete node reboot simulation
46+
47+
4. **Error Preservation Tests**
48+
- `test_multiple_error_scenarios_last_error_preserved`: Verifies first error is preserved
49+
- `test_wait_for_responses_includes_ssl_error`: Error included in wait_for_responses
50+
- `test_ssl_error_on_closed_connection_send_msg`: Error included in send_msg
51+
52+
### Integration Tests (`test_ssl_connection_failures.py`)
53+
8 integration tests with actual connection implementations:
54+
55+
1. **AsyncoreConnection Tests** (5 tests)
56+
- Socket closed forcefully during send
57+
- Connection reset during recv
58+
- SSL handshake failure
59+
- Broken pipe on SSL socket
60+
- Concurrent operations on closing connection
61+
62+
2. **AsyncioConnection Tests** (2 tests)
63+
- Socket error preservation
64+
- Connection reset handling
65+
66+
3. **Error Message Quality Tests** (2 tests)
67+
- Error message includes root cause
68+
- Multiple errors preserve first error
69+
70+
## Running the Tests
71+
72+
### Unit Tests Only
73+
```bash
74+
pytest tests/unit/test_ssl_connection_errors.py -v
75+
```
76+
77+
### Integration Tests (requires running cluster)
78+
```bash
79+
pytest tests/integration/standard/test_ssl_connection_failures.py -v
80+
```
81+
82+
### All Connection Tests
83+
```bash
84+
pytest tests/unit/test_connection.py tests/unit/test_ssl_connection_errors.py -v
85+
```
86+
87+
## Test Coverage
88+
89+
The tests verify:
90+
- ✅ Original error is captured in `last_error` field
91+
-`ConnectionShutdown` exceptions include root cause
92+
- ✅ Concurrent operations see original error, not just "Bad file descriptor"
93+
- ✅ First error is preserved when multiple errors occur
94+
- ✅ Mechanism works across different error types (EBADF, EPIPE, ECONNRESET, etc.)
95+
- ✅ Both AsyncoreConnection and AsyncioConnection handle errors correctly
96+
97+
## Code Changes
98+
99+
**No driver code changes were needed**. The existing error handling mechanism was already correct. These tests document and verify the expected behavior.
100+
101+
## Key Implementation Details
102+
103+
When a connection error occurs:
104+
105+
1. The error is caught in the reactor's `handle_error()`, `handle_write()`, or `handle_read()` method
106+
2. `defunct(exc)` is called with the exception
107+
3. `defunct()` stores the exception in `self.last_error`
108+
4. Subsequent operations check `is_defunct` and raise `ConnectionShutdown` with the original error
109+
110+
Example from `connection.py`:
111+
```python
112+
def send_msg(self, msg, request_id, cb, ...):
113+
if self.is_defunct:
114+
msg = "Connection to %s is defunct" % self.endpoint
115+
if self.last_error:
116+
msg += ": %s" % (self.last_error,) # Include original error
117+
raise ConnectionShutdown(msg)
118+
```
119+
120+
## Contributing
121+
122+
When adding new connection error handling code:
123+
1. Add corresponding tests to `test_ssl_connection_errors.py`
124+
2. Ensure `last_error` is set when connections become defunct
125+
3. Verify error messages include the root cause
126+
4. Test concurrent operation scenarios
127+
128+
## Related Issues
129+
- https://github.com/scylladb/python-driver/issues/614

0 commit comments

Comments
 (0)