Fix COMMIT/ROLLBACK/savepoint retry data-loss; harden MySQL connection-loss detection#1218
Conversation
… detection Transaction terminators must not be retried on a dropped backend: retrying a COMMIT/ROLLBACK/savepoint would reconnect, replay an empty transaction and report success, silently losing writes. Route them through with_raw_connection(allow_retry: false, materialize_transactions: true) to match ActiveRecord's native adapters, and forward the retry kwargs through exec_query so ordinary statements stay retryable. Also detect lost MySQL/MariaDB server connections (SQLState class 08, vendor "server gone" codes, recoverable/non-transient causes) and translate them to ActiveRecord::ConnectionFailed so the reconnect machinery engages, mirroring the PostgreSQL adapter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
headius
left a comment
There was a problem hiding this comment.
In the interest of moving the critical part of this fix forward (correct handling of restored connections with active tranactions) I approve these changes, but my comments here need to be addressed in another PR:
- More effort to try to align our copies of DB logic with Rails, such as adding the
internal_executemethod that defaults these retry/materialization kwargs. - Eliminate likely-duplicate tests added here where they already exist in Rails.
Claude does not have enough context to know that this repository includes copied code from rails/rails nor that we use the tests from rails/rails as our primary acceptance suite.
| def create_savepoint(name = current_savepoint_name) | ||
| log("SAVEPOINT #{name}", 'TRANSACTION') do | ||
| with_raw_connection(allow_retry: true, materialize_transactions: false) do |conn| | ||
| with_raw_connection(allow_retry: false, materialize_transactions: true) do |conn| |
There was a problem hiding this comment.
Equivalent code in Rails just falls back on default kwarg parameters for allow_retry: false and materialize_transactions: true in the internal_execute method they have and we do not. We probably want to try to match that relationship in the future.
| def exec_rollback_to_savepoint(name = current_savepoint_name) | ||
| log("ROLLBACK TO SAVEPOINT #{name}", 'TRANSACTION') do | ||
| with_raw_connection(allow_retry: true, materialize_transactions: false) do |conn| | ||
| with_raw_connection(allow_retry: false, materialize_transactions: true) do |conn| |
There was a problem hiding this comment.
Inconsistently, Rails does not fall back in internal_execute defaults here and passes these values explicitly. 🤨
| def release_savepoint(name = current_savepoint_name) | ||
| log("RELEASE SAVEPOINT #{name}", 'TRANSACTION') do | ||
| with_raw_connection(allow_retry: true, materialize_transactions: false) do |conn| | ||
| with_raw_connection(allow_retry: false, materialize_transactions: true) do |conn| |
There was a problem hiding this comment.
And this one falls back on defaults again.
| ::ActiveRecord::ConnectionAdapters::MySQL::Column | ||
| end | ||
|
|
||
| # MySQL / MariaDB surface a dropped server connection as a JDBC error in |
There was a problem hiding this comment.
I'd have preferred this be a separate PR but I understand they are somewhat interrelated.
Note that this mimics the same changes for PG in #1220 and these changes were suggested and encouraged there.
| @@ -0,0 +1,42 @@ | |||
| require 'db/mysql' | |||
|
|
|||
| # Regression tests for the COMMIT-retry data-loss footgun (#1) on MySQL. | |||
There was a problem hiding this comment.
We can merge these generated tests for now, but Claude is not considering the presence of similar tests in Rails. We are very likely duplicating tests and should review and remove where if that is the case.
|
@skunkworker This is still marked as a draft with your "Draft: ..." subject. If you're satisfied with this please update the topic appropriately (and if it's really a draft, use a draft PR in the future). |
Summary
Splits the connection-retry correctness fix out of #1216 into a focused PR.
Transaction terminators must not be retried on a dropped backend. On a
networked database, retrying a
COMMIT/ROLLBACK/ savepoint statementthrough
with_raw_connection(allow_retry: true)would reconnect, replay anempty transaction (the original writes died with the old backend), commit it
successfully, and report success — silently losing the transaction's writes.
This routes those statements through
with_raw_connection(allow_retry: false, materialize_transactions: true),matching ActiveRecord's native adapters, and forwards the retry kwargs through
exec_queryso ordinary statements stay retryable.It also hardens MySQL/MariaDB connection-loss detection: a dropped server
connection (SQLState class
08, vendor "server gone" codes likeCR_SERVER_GONE_ERROR/CR_SERVER_LOST, or a recoverable/non-transient cause)is now translated to
ActiveRecord::ConnectionFailedso the reconnect/retrymachinery engages — mirroring the PostgreSQL adapter's handling of proxy-dropped
connections (e.g. ProxySQL / pgbouncer).
Changes
abstract/transaction_support.rb—allow_retry: false, materialize_transactions: truefor COMMIT/ROLLBACK + all savepoint opsabstract/database_statements.rb— forwardallow_retry/materialize_transactionsthroughexec_querymysql/adapter.rb—connection_lost?detection (SQLStates, vendor error codes, message patterns, recoverable/non-transient causes) +translate_exceptionhookmysql/commit_no_retry_test.rb,sqlite3/transaction_no_retry_test.rb,mysql/connection_lost_test.rb,postgresql/connection_lost_test.rbNotes
Part of a larger split of #1216 into reviewable, topic-scoped PRs. Targets
72-stable. No conflicts with the other carved branches.🤖 Used Claude Opus 4.8
Justification
Existing functionality today:
#commit_db_transactionhttps://github.com/rails/rails/blob/7-2-stable/activerecord/lib/active_record/connection_adapters/postgresql/database_statements.rb#L111-L113
#exec_rollback_db_transactionhttps://github.com/rails/rails/blob/7-2-stable/activerecord/lib/active_record/connection_adapters/postgresql/database_statements.rb#L116-L119
create_savepointchangesThe change in this PR
Native AR defines the savepoint methods in Savepoints, routing through internal_execute with no explicit flags — so it takes internal_execute's defaults, which
are
allow_retry: false, materialize_transactions: true:https://github.com/rails/rails/blob/7-2-stable/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb
So the PR brings ArJdbc's savepoint handling from the unsafe allow_retry: true, materialize_transactions: false into exact parity with native AR's allow_retry:
false, materialize_transactions: true — the same fix and reasoning as the COMMIT/ROLLBACK methods, and master still needs this change applied separately.
Why shouldn't MySQL be broken into it's own PR?
The changes to bring
transaction_support.rblogic in line with what is on AR 7.2, would expose Mysql to these situations where the connection could be dropped, an error is raised (currently a non-retryable JDBCError, not handling situations where the connection is need to be retried. And shipping these two separately could open MySQL to data loss.Having them be both in this MR allows for the proper connection retry logic to bubble up to ActiveRecord, and prevents data loss from occuring.
The sqlite test
SQLITE is just a testing change without any actual logic needed. And will run if PG/MYSQL is not present but ensure the code works as expected.