|
| 1 | +# SSL Connection Error Handling Tests |
| 2 | + |
| 3 | +This directory contains comprehensive tests for SSL connection error handling, specifically addressing the "Bad file descriptor" errors that occur when nodes reboot while using client encryption. |
| 4 | + |
| 5 | +## Background |
| 6 | + |
| 7 | +### Issue |
| 8 | +When using client encryption and nodes reboot, the driver reported errors like: |
| 9 | +``` |
| 10 | +cassandra.connection.ConnectionShutdown: [Errno 9] Bad file descriptor |
| 11 | +``` |
| 12 | + |
| 13 | +The issue occurred because: |
| 14 | +1. A connection is forcefully closed (e.g., due to node reboot) |
| 15 | +2. Parallel operations attempt to read/write to the closed socket |
| 16 | +3. This results in "Bad file descriptor" errors |
| 17 | +4. The original cause of the connection closure could be lost |
| 18 | + |
| 19 | +See: https://github.com/scylladb/python-driver/issues/614 |
| 20 | + |
| 21 | +### Solution |
| 22 | +The driver already has proper error handling via the `last_error` mechanism in the `Connection` class. When a connection becomes defunct, the error that caused it is stored in `last_error` and included in subsequent `ConnectionShutdown` exception messages. |
| 23 | + |
| 24 | +This test suite verifies that this mechanism works correctly across all SSL error scenarios. |
| 25 | + |
| 26 | +## Test Files |
| 27 | + |
| 28 | +### Unit Tests (`test_ssl_connection_errors.py`) |
| 29 | +14 unit tests that mock SSL connection failures: |
| 30 | + |
| 31 | +1. **Basic SSL Socket Errors** |
| 32 | + - `test_ssl_socket_bad_file_descriptor_on_send`: Simulates EBADF during send |
| 33 | + - `test_ssl_socket_bad_file_descriptor_on_recv`: Simulates EBADF during recv |
| 34 | + - `test_ssl_socket_broken_pipe_error`: Simulates EPIPE (broken pipe) |
| 35 | + - `test_ssl_socket_connection_aborted_error`: Simulates ECONNABORTED |
| 36 | + - `test_ssl_socket_errno_enotconn`: Simulates ENOTCONN |
| 37 | + |
| 38 | +2. **SSL-Specific Errors** |
| 39 | + - `test_ssl_connection_error_during_handshake`: SSL handshake failures |
| 40 | + - `test_ssl_unwrap_error_on_close`: SSL unwrap errors during close |
| 41 | + |
| 42 | +3. **Race Condition Tests** |
| 43 | + - `test_concurrent_operations_on_closed_ssl_socket`: Multiple threads using closed socket |
| 44 | + - `test_parallel_send_on_defuncting_connection`: Thread trying to send while connection defuncts |
| 45 | + - `test_node_reboot_scenario`: Complete node reboot simulation |
| 46 | + |
| 47 | +4. **Error Preservation Tests** |
| 48 | + - `test_multiple_error_scenarios_last_error_preserved`: Verifies first error is preserved |
| 49 | + - `test_wait_for_responses_includes_ssl_error`: Error included in wait_for_responses |
| 50 | + - `test_ssl_error_on_closed_connection_send_msg`: Error included in send_msg |
| 51 | + |
| 52 | +### Integration Tests (`test_ssl_connection_failures.py`) |
| 53 | +8 integration tests with actual connection implementations: |
| 54 | + |
| 55 | +1. **AsyncoreConnection Tests** (5 tests) |
| 56 | + - Socket closed forcefully during send |
| 57 | + - Connection reset during recv |
| 58 | + - SSL handshake failure |
| 59 | + - Broken pipe on SSL socket |
| 60 | + - Concurrent operations on closing connection |
| 61 | + |
| 62 | +2. **AsyncioConnection Tests** (2 tests) |
| 63 | + - Socket error preservation |
| 64 | + - Connection reset handling |
| 65 | + |
| 66 | +3. **Error Message Quality Tests** (2 tests) |
| 67 | + - Error message includes root cause |
| 68 | + - Multiple errors preserve first error |
| 69 | + |
| 70 | +## Running the Tests |
| 71 | + |
| 72 | +### Unit Tests Only |
| 73 | +```bash |
| 74 | +pytest tests/unit/test_ssl_connection_errors.py -v |
| 75 | +``` |
| 76 | + |
| 77 | +### Integration Tests (requires running cluster) |
| 78 | +```bash |
| 79 | +pytest tests/integration/standard/test_ssl_connection_failures.py -v |
| 80 | +``` |
| 81 | + |
| 82 | +### All Connection Tests |
| 83 | +```bash |
| 84 | +pytest tests/unit/test_connection.py tests/unit/test_ssl_connection_errors.py -v |
| 85 | +``` |
| 86 | + |
| 87 | +## Test Coverage |
| 88 | + |
| 89 | +The tests verify: |
| 90 | +- ✅ Original error is captured in `last_error` field |
| 91 | +- ✅ `ConnectionShutdown` exceptions include root cause |
| 92 | +- ✅ Concurrent operations see original error, not just "Bad file descriptor" |
| 93 | +- ✅ First error is preserved when multiple errors occur |
| 94 | +- ✅ Mechanism works across different error types (EBADF, EPIPE, ECONNRESET, etc.) |
| 95 | +- ✅ Both AsyncoreConnection and AsyncioConnection handle errors correctly |
| 96 | + |
| 97 | +## Code Changes |
| 98 | + |
| 99 | +**No driver code changes were needed**. The existing error handling mechanism was already correct. These tests document and verify the expected behavior. |
| 100 | + |
| 101 | +## Key Implementation Details |
| 102 | + |
| 103 | +When a connection error occurs: |
| 104 | + |
| 105 | +1. The error is caught in the reactor's `handle_error()`, `handle_write()`, or `handle_read()` method |
| 106 | +2. `defunct(exc)` is called with the exception |
| 107 | +3. `defunct()` stores the exception in `self.last_error` |
| 108 | +4. Subsequent operations check `is_defunct` and raise `ConnectionShutdown` with the original error |
| 109 | + |
| 110 | +Example from `connection.py`: |
| 111 | +```python |
| 112 | +def send_msg(self, msg, request_id, cb, ...): |
| 113 | + if self.is_defunct: |
| 114 | + msg = "Connection to %s is defunct" % self.endpoint |
| 115 | + if self.last_error: |
| 116 | + msg += ": %s" % (self.last_error,) # Include original error |
| 117 | + raise ConnectionShutdown(msg) |
| 118 | +``` |
| 119 | + |
| 120 | +## Contributing |
| 121 | + |
| 122 | +When adding new connection error handling code: |
| 123 | +1. Add corresponding tests to `test_ssl_connection_errors.py` |
| 124 | +2. Ensure `last_error` is set when connections become defunct |
| 125 | +3. Verify error messages include the root cause |
| 126 | +4. Test concurrent operation scenarios |
| 127 | + |
| 128 | +## Related Issues |
| 129 | +- https://github.com/scylladb/python-driver/issues/614 |
0 commit comments