diff --git a/README.md b/README.md index 097fa0712..50b791e70 100644 --- a/README.md +++ b/README.md @@ -104,6 +104,13 @@ encounter problems, try adding this to your database configuration: The correct timezone depends on the system setup, but the one shown is a good place to start and is actually the correct setting for many systems. +#### PostgreSQL specific notes + +PostgreSQL connection/resilience options (timeouts, TCP keep-alive, +prepared-statement tuning) and guidance for running behind PgBouncer +(including transaction/statement pooling mode) are documented in the +[activerecord-jdbcpostgresql-adapter README](activerecord-jdbcpostgresql-adapter/README.md). + ### Standalone with ActiveRecord diff --git a/activerecord-jdbc-adapter.gemspec b/activerecord-jdbc-adapter.gemspec index a32506441..ddcdfe166 100644 --- a/activerecord-jdbc-adapter.gemspec +++ b/activerecord-jdbc-adapter.gemspec @@ -41,7 +41,7 @@ Gem::Specification.new do |gem| gem.executables = gem.files.grep(%r{^bin/}).map { |f| File.basename(f) } gem.test_files = gem.files.grep(%r{^test/}) - gem.add_dependency "activerecord", "~> 8.0.0" + gem.add_dependency "activerecord", "~> 8.1.0" #gem.add_development_dependency 'test-unit', '2.5.4' #gem.add_development_dependency 'test-unit-context', '>= 0.3.0' diff --git a/activerecord-jdbcpostgresql-adapter/README.md b/activerecord-jdbcpostgresql-adapter/README.md new file mode 100644 index 000000000..4f6e759ba --- /dev/null +++ b/activerecord-jdbcpostgresql-adapter/README.md @@ -0,0 +1,114 @@ +# activerecord-jdbcpostgresql-adapter + +* https://github.com/jruby/activerecord-jdbc-adapter/ + +## Description + +This is an ActiveRecord driver for PostgreSQL using JDBC running under JRuby. + +It is part of the [activerecord-jdbc-adapter](https://github.com/jruby/activerecord-jdbc-adapter) +project; see the top-level README for general installation and configuration. + +## Connection and resilience options + +In addition to the standard ActiveRecord PostgreSQL settings, the adapter +recognizes the following `database.yml` keys. Each is translated to the +corresponding [pgjdbc connection parameter](https://jdbc.postgresql.org/documentation/use/#connection-parameters). +Setting timeouts and TCP keep-alive is strongly recommended when running behind a +connection pooler or a firewall that drops idle connections, so a dead/reaped +backend surfaces as an error instead of hanging the JVM thread. + +| `database.yml` key | pgjdbc property | Meaning | Default | +| --- | --- | --- | --- | +| `connect_timeout` | `connectTimeout` | Timeout (seconds) for establishing a new connection. Also read from `PGCONNECT_TIMEOUT`. | pgjdbc default (10s) | +| `socket_timeout` | `socketTimeout` | Per-query read timeout (seconds) on an established connection. `0` = infinite. | unset (infinite) | +| `login_timeout` | `loginTimeout` | Timeout (seconds) for the connection + authentication handshake. | unset | +| `prepare_threshold` | `prepareThreshold` | How many executions before a query is promoted to a *named* server-side prepared statement. `0` keeps statements unnamed. Overrides the value derived from `prepared_statements`. | derived from `prepared_statements` | +| `keepalives` | `tcpKeepAlive` | Enable TCP keep-alive probes. | unset (off) | + +You can also pass any pgjdbc property directly under `properties:` (e.g. +`tcpKeepAlive: true`). Example: + +```yml +production: + adapter: postgresql + database: blog + username: blog + password: blog + connect_timeout: 5 # cap connection establishment (pgjdbc connectTimeout) + socket_timeout: 30 # cap per-query reads (pgjdbc socketTimeout) + properties: + tcpKeepAlive: true # detect dead backends instead of hanging +``` + +> **Behavior change:** prior to this release `connect_timeout` was mapped to +> pgjdbc's `socketTimeout` (a per-query read timeout). It now maps to +> `connectTimeout`, matching the pg gem / libpq meaning of `connect_timeout`. If +> you relied on the old behavior to bound query execution time, set +> `socket_timeout` instead. + +## Running behind PgBouncer (or another connection pooler) + +The PostgreSQL adapter works out of the box with PgBouncer in **`session`** +pooling mode — every checkout keeps one backend, so per-session state is stable +and no special configuration is required. + +In **`transaction`** or **`statement`** pooling mode PgBouncer reassigns the +server backend between transactions, which breaks anything that relies on +per-session server state. See the next section before using those modes. + +| Concern | `session` mode | `transaction` / `statement` mode | +| --- | --- | --- | +| Prepared statements | works as-is | `prepared_statements: false` (or PgBouncer >= 1.21, see below) | +| Advisory locks (migrations) | works as-is | `advisory_locks: false` | +| Session `SET`s (timezone, search_path, intervalstyle) | applied at connect | must be enforced per backend (see below) | + +## PgBouncer transaction (and statement) pooling mode + +Use this configuration as a starting point: + +```yml +production: + adapter: postgresql + host: 127.0.0.1 + port: 6432 # PgBouncer listening port + database: blog + username: blog + password: blog + prepared_statements: false # required: named server statements can't span backends + advisory_locks: false # required: session-scoped locks can't span backends + connect_timeout: 5 + socket_timeout: 30 + properties: + tcpKeepAlive: true +``` + +**Prepared statements.** With `prepared_statements: true` (the default) the +pgjdbc driver promotes a query to a *named* server-side prepared statement after +it has executed `prepareThreshold` (5 by default) times. That named statement +lives on one backend, so once PgBouncer hands the next transaction a different +backend you get `ERROR: prepared statement "S_1" does not exist`. Two options: + +* Set `prepared_statements: false` — this forces pgjdbc `prepareThreshold=0`, so + only unnamed statements are used, which are safe across backends. +* Or run PgBouncer >= 1.21 with `max_prepared_statements > 0` and keep prepared + statements on. Tune the promotion point with `prepare_threshold:` if needed. + +**Advisory locks.** Schema migrations take a session-level advisory lock that is +acquired and released across transaction boundaries. Under transaction/statement +pooling the unlock may land on a different backend and orphan the lock, so set +`advisory_locks: false`. + +**Session `SET` statements.** On connect the adapter issues `SET TIME ZONE`, +`SET intervalstyle = iso_8601`, applies `schema_search_path`, etc. These do +**not** follow backend reassignment, which can *silently* corrupt results +(interval parsing, timestamps, querying the wrong schema). Because PgBouncer in +transaction/statement mode disallows per-session state, enforce these per backend +instead — for example: + +* `ALTER ROLE blog SET intervalstyle = 'iso_8601';` +* `ALTER ROLE blog SET timezone = 'UTC';` (when using ActiveRecord UTC time zone support) +* set `search_path` via `ALTER ROLE`/`ALTER DATABASE` or PgBouncer's server-side + connect options instead of `schema_search_path:` + +If you cannot enforce that state globally, use `session` pooling for that database. diff --git a/activerecord-jdbcpostgresql-adapter/README.txt b/activerecord-jdbcpostgresql-adapter/README.txt deleted file mode 100644 index c2dde810d..000000000 --- a/activerecord-jdbcpostgresql-adapter/README.txt +++ /dev/null @@ -1,7 +0,0 @@ -= activerecord-jdbcpostgresql-adapter - -* https://github.com/jruby/activerecord-jdbc-adapter/ - -== DESCRIPTION: - -This is an ActiveRecord driver for PostgreSQL using JDBC running under JRuby. diff --git a/lib/arjdbc/abstract/core.rb b/lib/arjdbc/abstract/core.rb index 2ed5d8c63..7aafc5d35 100644 --- a/lib/arjdbc/abstract/core.rb +++ b/lib/arjdbc/abstract/core.rb @@ -58,7 +58,7 @@ def translate_exception(exception, message:, sql:, binds:) end # this version of log() automatically fills type_casted_binds from binds if necessary - def log(sql, name = "SQL", binds = [], type_casted_binds = [], async: false, &block) + def log(sql, name = "SQL", binds = [], type_casted_binds = [], async: false, allow_retry: false, &block) if binds.any? && (type_casted_binds.nil? || type_casted_binds.empty?) type_casted_binds = lambda { # extract_raw_bind_values diff --git a/lib/arjdbc/abstract/database_statements.rb b/lib/arjdbc/abstract/database_statements.rb index 64a6a82f6..eeb428bed 100644 --- a/lib/arjdbc/abstract/database_statements.rb +++ b/lib/arjdbc/abstract/database_statements.rb @@ -19,13 +19,16 @@ def exec_insert(sql, name = nil, binds = NO_BINDS, pk = nil, sequence_name = nil binds = convert_legacy_binds_to_attributes(binds) if binds.first.is_a?(Array) with_raw_connection do |conn| - if without_prepared_statement?(binds) - log(sql, name) { conn.execute_insert_pk(sql, pk) } - else - log(sql, name, binds) do - conn.execute_insert_pk(sql, binds, pk) + result = + if without_prepared_statement?(binds) + log(sql, name) { conn.execute_insert_pk(sql, pk) } + else + log(sql, name, binds) do + conn.execute_insert_pk(sql, binds, pk) + end end - end + handle_warnings(sql) + result end end @@ -41,15 +44,18 @@ def internal_exec_query(sql, name = nil, binds = NO_BINDS, prepare: false, async binds = convert_legacy_binds_to_attributes(binds) if binds.first.is_a?(Array) with_raw_connection do |conn| - if without_prepared_statement?(binds) - log(sql, name, async: async) { conn.execute_query(sql) } - else - log(sql, name, binds, async: async) do - # this is different from normal AR that always caches - cached_statement = fetch_cached_statement(sql) if prepare && @jdbc_statement_cache_enabled - conn.execute_prepared_query(sql, binds, cached_statement) + result = + if without_prepared_statement?(binds) + log(sql, name, async: async) { conn.execute_query(sql) } + else + log(sql, name, binds, async: async) do + # this is different from normal AR that always caches + cached_statement = fetch_cached_statement(sql) if prepare && @jdbc_statement_cache_enabled + conn.execute_prepared_query(sql, binds, cached_statement) + end end - end + handle_warnings(sql) + result end end @@ -63,11 +69,14 @@ def exec_update(sql, name = 'SQL', binds = NO_BINDS) binds = convert_legacy_binds_to_attributes(binds) if binds.first.is_a?(Array) with_raw_connection do |conn| - if without_prepared_statement?(binds) - log(sql, name) { conn.execute_update(sql) } - else - log(sql, name, binds) { conn.execute_prepared_update(sql, binds) } - end + result = + if without_prepared_statement?(binds) + log(sql, name) { conn.execute_update(sql) } + else + log(sql, name, binds) { conn.execute_prepared_update(sql, binds) } + end + handle_warnings(sql) + result end end alias :exec_delete :exec_update @@ -90,10 +99,17 @@ def convert_legacy_binds_to_attributes(binds) end end - def preprocess_query(sql) - check_if_write_query(sql) if respond_to?(:check_if_write_query, true) - mark_transaction_written_if_write(sql) if respond_to?(:mark_transaction_written_if_write, true) - sql + # AR >= 8.1 removed these helpers, folding them into #preprocess_query via + # the primitives #write_query?, #ensure_writes_are_allowed and + # #mark_transaction_written. The JDBC adapters' custom query paths + # (e.g. #exec_insert, #execute_and_clear) still call them, so reintroduce + # them here on top of those primitives. + def mark_transaction_written_if_write(sql) + mark_transaction_written if write_query?(sql) + end + + def check_if_write_query(sql) + ensure_writes_are_allowed(sql) if write_query?(sql) end def raw_execute(sql, name, binds = [], prepare: false, async: false, allow_retry: false, materialize_transactions: true, batch: false) @@ -101,11 +117,19 @@ def raw_execute(sql, name, binds = [], prepare: false, async: false, allow_retry with_raw_connection(allow_retry: allow_retry, materialize_transactions: materialize_transactions) do |conn| result = conn.execute(sql) verified! + handle_warnings(sql) result end end end + # Drains and dispatches any SQL warnings collected during the last + # execution. No-op by default; adapters that surface DB warnings (e.g. + # PostgreSQL) override this. Mirrors AbstractAdapter#handle_warnings, which + # the JDBC query paths bypass since they don't route through #raw_execute. + def handle_warnings(sql) + end + end end end diff --git a/lib/arjdbc/abstract/transaction_support.rb b/lib/arjdbc/abstract/transaction_support.rb index d513230c3..7e7052991 100644 --- a/lib/arjdbc/abstract/transaction_support.rb +++ b/lib/arjdbc/abstract/transaction_support.rb @@ -64,6 +64,19 @@ def exec_rollback_db_transaction end end + # Rolls back the current transaction and immediately starts a fresh one, + # the equivalent of PostgreSQL's `ROLLBACK AND CHAIN`. Implemented via the + # JDBC transaction API (rollback + begin) since issuing the raw SQL would + # bypass the driver's transaction state tracking. + def exec_restart_db_transaction + log('ROLLBACK AND CHAIN', 'TRANSACTION') do + with_raw_connection(allow_retry: false, materialize_transactions: true) do |conn| + conn.rollback + conn.begin + end + end + end + ########################## Savepoint Interface ############################ # Creates a (transactional) save-point one can rollback to. diff --git a/lib/arjdbc/postgresql/adapter.rb b/lib/arjdbc/postgresql/adapter.rb index 88adbb878..debec2817 100644 --- a/lib/arjdbc/postgresql/adapter.rb +++ b/lib/arjdbc/postgresql/adapter.rb @@ -90,6 +90,10 @@ def configure_connection execute("SET time zone '#{tz}'", 'SCHEMA') end unless redshift? + # Track the timezone the session was configured for so #update_typemap_for_default_timezone + # only reconfigures when ActiveRecord.default_timezone actually changes at runtime. + @default_timezone = ActiveRecord.default_timezone + # Set interval output format to ISO 8601 for ease of parsing by ActiveSupport::Duration.parse execute("SET intervalstyle = iso_8601", "SCHEMA") @@ -183,6 +187,10 @@ def supports_expression_index? true end + def supports_index_include? + database_version >= 11_00_00 # >= 11.0 + end + def supports_transaction_isolation? true end @@ -199,6 +207,18 @@ def supports_validate_constraints? true end + def supports_deferrable_constraints? + true + end + + def supports_exclusion_constraints? + true + end + + def supports_unique_constraints? + true + end + def supports_views? true end @@ -219,6 +239,14 @@ def supports_savepoints? true end + def supports_restart_db_transaction? + database_version >= 12_00_00 # >= 12.0 + end + + def supports_close_prepared? + false # no JDBC protocol-level equivalent; pgjdbc manages deallocation + end + def supports_native_partitioning? database_version >= 100_000 end @@ -509,8 +537,8 @@ def build_insert_sql(insert) # :nodoc: end def check_version # :nodoc: - if database_version < 90300 - raise "Your version of PostgreSQL (#{database_version}) is too old. Active Record supports PostgreSQL >= 9.3." + if database_version < 9_05_00 # < 9.5 + raise "Your version of PostgreSQL (#{database_version}) is too old. Active Record supports PostgreSQL >= 9.5." end end @@ -741,39 +769,98 @@ def can_perform_case_insensitive_comparison_for?(column) end end + # SQLSTATE codes, see https://www.postgresql.org/docs/current/errcodes-appendix.html + VALUE_LIMIT_VIOLATION = "22001" + NUMERIC_VALUE_OUT_OF_RANGE = "22003" + NOT_NULL_VIOLATION = "23502" + FOREIGN_KEY_VIOLATION = "23503" + UNIQUE_VIOLATION = "23505" + CHECK_VIOLATION = "23514" + EXCLUSION_VIOLATION = "23P01" + SERIALIZATION_FAILURE = "40001" + DEADLOCK_DETECTED = "40P01" + DUPLICATE_DATABASE = "42P04" + LOCK_NOT_AVAILABLE = "55P03" + QUERY_CANCELED = "57014" + def translate_exception(exception, message:, sql:, binds:) return super unless exception.is_a?(ActiveRecord::JDBCError) - # TODO: Can we base these on an error code of some kind? - case exception.message - when /could not create unique index/ - ::ActiveRecord::RecordNotUnique.new(message, sql: sql, binds: binds, connection_pool: @pool) - when /duplicate key value violates unique constraint/ - ::ActiveRecord::RecordNotUnique.new(message, sql: sql, binds: binds) + # Prefer the driver-provided SQLSTATE; fall back to message matching for + # drivers/versions that do not populate a code we recognize. + klass = exception_class_for_sql_state(exception.sql_state) || + exception_class_for_message(exception.message) + return super unless klass + + klass.new(message, sql: sql, binds: binds, connection_pool: @pool) + end + + def exception_class_for_sql_state(sql_state) + case sql_state + when UNIQUE_VIOLATION then ::ActiveRecord::RecordNotUnique + when FOREIGN_KEY_VIOLATION then ::ActiveRecord::InvalidForeignKey + when NOT_NULL_VIOLATION then ::ActiveRecord::NotNullViolation + when CHECK_VIOLATION then ::ActiveRecord::CheckViolation + when EXCLUSION_VIOLATION then ::ActiveRecord::ExclusionViolation + when VALUE_LIMIT_VIOLATION then ::ActiveRecord::ValueTooLong + when NUMERIC_VALUE_OUT_OF_RANGE then ::ActiveRecord::RangeError + when SERIALIZATION_FAILURE then ::ActiveRecord::SerializationFailure + when DEADLOCK_DETECTED then ::ActiveRecord::Deadlocked + when DUPLICATE_DATABASE then ::ActiveRecord::DatabaseAlreadyExists + when LOCK_NOT_AVAILABLE then ::ActiveRecord::LockWaitTimeout + when QUERY_CANCELED then ::ActiveRecord::QueryCanceled + end + end + private :exception_class_for_sql_state + + def exception_class_for_message(msg) + case msg + when /could not create unique index/, /duplicate key value violates unique constraint/ + ::ActiveRecord::RecordNotUnique when /violates not-null constraint/ - ::ActiveRecord::NotNullViolation.new(message, sql: sql, binds: binds) + ::ActiveRecord::NotNullViolation when /violates foreign key constraint/ - ::ActiveRecord::InvalidForeignKey.new(message, sql: sql, binds: binds) + ::ActiveRecord::InvalidForeignKey when /value too long/ - ::ActiveRecord::ValueTooLong.new(message, sql: sql, binds: binds) + ::ActiveRecord::ValueTooLong when /out of range/ - ::ActiveRecord::RangeError.new(message, sql: sql, binds: binds) + ::ActiveRecord::RangeError when /could not serialize/ - ::ActiveRecord::SerializationFailure.new(message, sql: sql, binds: binds) + ::ActiveRecord::SerializationFailure when /deadlock detected/ - ::ActiveRecord::Deadlocked.new(message, sql: sql, binds: binds) + ::ActiveRecord::Deadlocked when /lock timeout/ - ::ActiveRecord::LockWaitTimeout.new(message, sql: sql, binds: binds) - when /canceling statement/ # This needs to come after lock timeout because the lock timeout message also contains "canceling statement" - ::ActiveRecord::QueryCanceled.new(message, sql: sql, binds: binds) - when /relation .* does not exist/i - ::ActiveRecord::StatementInvalid.new(message, sql: sql, binds: binds, connection_pool: @pool) - when /syntax error at or near/i - ::ActiveRecord::StatementInvalid.new(message, sql: sql, binds: binds, connection_pool: @pool) - else - super + ::ActiveRecord::LockWaitTimeout + when /canceling statement/ # must follow lock timeout (its message also contains "canceling statement") + ::ActiveRecord::QueryCanceled + when /relation .* does not exist/i, /syntax error at or near/i + ::ActiveRecord::StatementInvalid end end + private :exception_class_for_message + + # Drains the SQL warnings collected by the JDBC layer for the last statement + # and dispatches each via +ActiveRecord.db_warnings_action+. Called from the + # JDBC query paths (ArJdbc::Abstract::DatabaseStatements), which bypass the + # abstract adapter's #raw_execute-based warning handling. Always drains so the + # Java-side buffer can't grow even when warnings are configured to be ignored. + def handle_warnings(sql) + warnings = @raw_connection.take_warnings + return if warnings.empty? || ActiveRecord.db_warnings_action.nil? + + warnings.each do |message, code, level| + warning = ActiveRecord::SQLWarning.new(message, code, level, sql, @pool) + next if warning_ignored?(warning) + + ActiveRecord.db_warnings_action.call(warning) + end + end + private :handle_warnings + + def warning_ignored?(warning) + ["WARNING", "ERROR", "FATAL", "PANIC"].exclude?(warning.level) || super + end + private :warning_ignored? # @private `Utils.extract_schema_and_table` from AR def extract_schema_and_table(name) @@ -986,6 +1073,7 @@ def exec_no_cache(sql, name, binds, async: false) with_raw_connection do |conn| result = conn.exec_params(sql, type_casted_binds) verified! + handle_warnings(sql) result end end @@ -1030,9 +1118,14 @@ def exec_cache(sql, name, binds, async: false) # Check here for more details: # https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/utils/cache/plancache.c#l573 def is_cached_plan_failure?(e) - pgerror = e.cause - pgerror.result.result_error_field(PG::PG_DIAG_SQLSTATE) == FEATURE_NOT_SUPPORTED && - pgerror.result.result_error_field(PG::PG_DIAG_SOURCE_FUNCTION) == "RevalidateCachedQuery" + # Under JDBC we don't have access to PG::Result#result_error_field, so we + # classify via the JDBC SQLSTATE (0A000 / FEATURE_NOT_SUPPORTED) together + # with the server message emitted by RevalidateCachedQuery. + error = e.cause || e + return false unless error.respond_to?(:sql_state) + + error.sql_state == FEATURE_NOT_SUPPORTED && + error.message.to_s.include?("cached plan must not change result type") rescue false end diff --git a/lib/arjdbc/postgresql/adapter_hash_config.rb b/lib/arjdbc/postgresql/adapter_hash_config.rb index bc042ac1f..fdb72e1f0 100644 --- a/lib/arjdbc/postgresql/adapter_hash_config.rb +++ b/lib/arjdbc/postgresql/adapter_hash_config.rb @@ -50,10 +50,19 @@ def database_driver_name def build_properties(config) properties = config[:properties] || {} - # PG :connect_timeout - maximum time to wait for connection to succeed + # PG :connect_timeout - maximum time to wait for connection to succeed. + # Maps to pgjdbc's connectTimeout (connection establishment), matching the + # pg gem / libpq meaning of connect_timeout. connect_timeout = config[:connect_timeout] || ENV["PGCONNECT_TIMEOUT"] - properties["socketTimeout"] ||= connect_timeout if connect_timeout + properties["connectTimeout"] ||= connect_timeout if connect_timeout + + # :socket_timeout - per-query read timeout (pgjdbc socketTimeout). Useful + # behind a connection pooler (e.g. PgBouncer) so a dead/reaped backend + # surfaces as an error instead of hanging the JVM thread indefinitely. + socket_timeout = config[:socket_timeout] + + properties["socketTimeout"] ||= socket_timeout if socket_timeout login_timeout = config[:login_timeout] @@ -92,6 +101,16 @@ def build_properties(config) properties["prepareThreshold"] = 0 end + # :prepare_threshold - explicit pgjdbc prepareThreshold passthrough. Lets + # you control when (or whether) pgjdbc promotes a query to a *named* + # server-side prepared statement. Set to 0 to keep using unnamed statements, + # which is required behind PgBouncer in transaction/statement pooling mode + # (named statements are bound to a single backend). Overrides the value + # derived from :prepared_statements above. + unless config[:prepare_threshold].nil? + properties["prepareThreshold"] = config[:prepare_threshold] + end + properties end end diff --git a/lib/arjdbc/postgresql/database_statements.rb b/lib/arjdbc/postgresql/database_statements.rb index 2c1ddc85e..c745e929b 100644 --- a/lib/arjdbc/postgresql/database_statements.rb +++ b/lib/arjdbc/postgresql/database_statements.rb @@ -15,6 +15,35 @@ def build_explain_clause(options = []) "EXPLAIN (#{options.join(", ").upcase})" end + + # From https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-CURRENT + HIGH_PRECISION_CURRENT_TIMESTAMP = Arel.sql("CURRENT_TIMESTAMP", retryable: true).freeze # :nodoc: + private_constant :HIGH_PRECISION_CURRENT_TIMESTAMP + + def high_precision_current_timestamp + HIGH_PRECISION_CURRENT_TIMESTAMP + end + + # Set when constraints will be checked for the current transaction. + # + # Not passing any specific constraint names will set the value for all + # deferrable constraints. + # + # +deferred+ must be +:deferred+ or +:immediate+. + # + # See https://www.postgresql.org/docs/current/sql-set-constraints.html + def set_constraints(deferred, *constraints) + unless %i[deferred immediate].include?(deferred) + raise ArgumentError, "deferred must be :deferred or :immediate" + end + + constraints = if constraints.empty? + "ALL" + else + constraints.map { |c| quote_table_name(c) }.join(", ") + end + execute("SET CONSTRAINTS #{constraints} #{deferred.to_s.upcase}") + end end end end diff --git a/lib/arjdbc/postgresql/oid_types.rb b/lib/arjdbc/postgresql/oid_types.rb index cd6e6cff9..ee1ca5275 100644 --- a/lib/arjdbc/postgresql/oid_types.rb +++ b/lib/arjdbc/postgresql/oid_types.rb @@ -232,19 +232,13 @@ def load_types_queries(initializer, oids) end def update_typemap_for_default_timezone - if @default_timezone != ActiveRecord.default_timezone && @timestamp_decoder - decoder_class = ActiveRecord.default_timezone == :utc ? - PG::TextDecoder::TimestampUtc : - PG::TextDecoder::TimestampWithoutTimeZone - - @timestamp_decoder = decoder_class.new(@timestamp_decoder.to_h) - @connection.type_map_for_results.add_coder(@timestamp_decoder) - + # Unlike the pg gem (which swaps PG::TextDecoder timestamp coders), the + # JDBC driver returns timestamps based on the session time zone set in + # #configure_connection. So if ActiveRecord.default_timezone changes at + # runtime we just re-run #configure_connection to apply the new zone. + if @default_timezone != ActiveRecord.default_timezone @default_timezone = ActiveRecord.default_timezone - - # if default timezone has changed, we need to reconfigure the connection - # (specifically, the session time zone) - configure_connection + configure_connection if @raw_connection end end diff --git a/lib/arjdbc/postgresql/schema_statements.rb b/lib/arjdbc/postgresql/schema_statements.rb index 41f3ef91b..3f08344dc 100644 --- a/lib/arjdbc/postgresql/schema_statements.rb +++ b/lib/arjdbc/postgresql/schema_statements.rb @@ -52,6 +52,28 @@ def foreign_keys(table_name) ForeignKeyDefinition.new(table_name, to_table, options) end end + + # Override to avoid the gem's PG::TextDecoder::Array (pg gem) when decoding + # the result of `current_schemas(false)`. + def current_schemas + schemas = query_value("SELECT current_schemas(false)", "SCHEMA") + decode_string_array(schemas) + end + + # Rails' PostgreSQL::SchemaStatements#decode_string_array (used by #indexes + # and schema dumping since AR 8.x) relies on PG::TextDecoder::Array from the + # pg gem, which is unavailable under JDBC. The JDBC driver already decodes + # array-typed result columns into Ruby Arrays, so pass those through; only + # parse when given a raw PostgreSQL array string. + def decode_string_array(value) + return value if value.is_a?(::Array) + return [] if value.nil? + + @string_array_decoder ||= + ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Array::PG::TextDecoder::Array.new(name: nil, delimiter: ",") + @string_array_decoder.decode(value) + end + private :decode_string_array end end end diff --git a/src/java/arjdbc/jdbc/RubyJdbcConnection.java b/src/java/arjdbc/jdbc/RubyJdbcConnection.java index d88d9a8d6..b555149f9 100644 --- a/src/java/arjdbc/jdbc/RubyJdbcConnection.java +++ b/src/java/arjdbc/jdbc/RubyJdbcConnection.java @@ -41,6 +41,7 @@ import java.sql.ResultSet; import java.sql.ResultSetMetaData; import java.sql.SQLException; +import java.sql.SQLWarning; import java.sql.SQLXML; import java.sql.Statement; import java.sql.Date; @@ -142,6 +143,10 @@ public class RubyJdbcConnection extends RubyObject { private boolean configureConnection = true; // final once initialized private int fetchSize = 0; // 0 = JDBC default + // SQL warnings collected from recent statement executions; drained via #take_warnings. + // Each entry is a [message, sqlstate, level] Ruby array. + private final List warnings = new ArrayList<>(0); + protected RubyJdbcConnection(Ruby runtime, RubyClass metaClass) { super(runtime, metaClass); var context = runtime.getCurrentContext(); @@ -822,6 +827,8 @@ public IRubyObject execute(final ThreadContext context, final IRubyObject sql) { updateCount = statement.getUpdateCount(); } + collectWarnings(context, statement); + return result; } catch (final SQLException e) { @@ -833,6 +840,76 @@ public IRubyObject execute(final ThreadContext context, final IRubyObject sql) { }); } + /** + * Collects any SQL warnings from the given statement into the per-connection + * buffer, then clears them from the statement. Best-effort: surfacing warnings + * must never break query execution. + * @param context current thread context + * @param statement the statement just executed + */ + protected void collectWarnings(final ThreadContext context, final Statement statement) { + if (!collectsWarnings()) return; + try { + SQLWarning warning = statement.getWarnings(); + if (warning == null) return; + while (warning != null) { + warnings.add(warningToRuby(context, warning)); + warning = warning.getNextWarning(); + } + statement.clearWarnings(); + } catch (SQLException e) { + // ignore - warnings are advisory and must not fail the query + } + } + + private IRubyObject warningToRuby(final ThreadContext context, final SQLWarning warning) { + final Ruby runtime = context.runtime; + final String message = warning.getMessage(); + final String state = warning.getSQLState(); + final IRubyObject[] parts = new IRubyObject[] { + message == null ? context.nil : runtime.newString(message), + state == null ? context.nil : runtime.newString(state), + warningLevel(context, warning) + }; + return RubyArray.newArray(runtime, parts); + } + + /** + * Whether this connection should gather SQL warnings after each execution. + * Off by default to avoid the per-statement getWarnings() cost; + * adapters that surface DB warnings (e.g. PostgreSQL) override this. + * @return true to collect warnings + */ + protected boolean collectsWarnings() { + return false; + } + + /** + * The severity/level of a warning (e.g. "WARNING", "NOTICE"). The standard + * JDBC API does not expose this, so the base implementation returns nil; + * adapters with driver-specific support override this. + * @param context current thread context + * @param warning the warning to inspect + * @return the level as a Ruby string, or nil + */ + protected IRubyObject warningLevel(final ThreadContext context, final SQLWarning warning) { + return context.nil; + } + + /** + * Returns the SQL warnings accumulated since the previous call and clears the + * buffer. Each entry is a [message, sqlstate, level] array. + * @param context current thread context + * @return a Ruby array of warning triples (possibly empty) + */ + @JRubyMethod(name = "take_warnings") + public IRubyObject take_warnings(final ThreadContext context) { + if (warnings.isEmpty()) return newArray(context); + final RubyArray result = context.runtime.newArray(warnings); + warnings.clear(); + return result; + } + protected Statement createStatement(final ThreadContext context, final Connection connection) throws SQLException { final Statement statement = connection.createStatement(); @@ -905,6 +982,7 @@ public IRubyObject execute_insert_pk(final ThreadContext context, final IRubyObj statement.executeUpdate(query, createStatementPk(pk)); } + collectWarnings(context, statement); return mapGeneratedKeys(context, connection, statement); } catch (final SQLException e) { debugErrorSQL(context, query); @@ -945,6 +1023,7 @@ public IRubyObject execute_insert_pk(final ThreadContext context, final IRubyObj setStatementParameters(context, connection, statement, (RubyArray) binds); statement.executeUpdate(); + collectWarnings(context, statement); return mapGeneratedKeys(context, connection, statement); } catch (final SQLException e) { debugErrorSQL(context, query); @@ -978,6 +1057,7 @@ public IRubyObject execute_update(final ThreadContext context, final IRubyObject statement = createStatement(context, connection); final int rowCount = statement.executeUpdate(query); + collectWarnings(context, statement); return context.runtime.newFixnum(rowCount); } catch (final SQLException e) { debugErrorSQL(context, query); @@ -1006,6 +1086,7 @@ public IRubyObject execute_prepared_update(final ThreadContext context, final IR statement = connection.prepareStatement(query); setStatementParameters(context, connection, statement, (RubyArray) binds); final int rowCount = statement.executeUpdate(); + collectWarnings(context, statement); return context.runtime.newFixnum(rowCount); } catch (final SQLException e) { debugErrorSQL(context, query); @@ -1119,7 +1200,9 @@ public IRubyObject execute_query(final ThreadContext context, final IRubyObject statement = createStatement(context, connection); // At least until AR 5.1 #exec_query still gets called for things that don't return results in some cases :( - if (statement.execute(query)) { + final boolean hasResultSet = statement.execute(query); + collectWarnings(context, statement); + if (hasResultSet) { return mapQueryResult(context, connection, statement.getResultSet()); } @@ -1207,7 +1290,9 @@ public IRubyObject execute_prepared_query(final ThreadContext context, final IRu setStatementParameters(context, connection, statement, (RubyArray) binds); - if (statement.execute()) { + final boolean hasResultSet = statement.execute(); + collectWarnings(context, statement); + if (hasResultSet) { ResultSet resultSet = statement.getResultSet(); IRubyObject results = mapQueryResult(context, connection, resultSet); resultSet.close(); diff --git a/src/java/arjdbc/postgresql/PostgreSQLResult.java b/src/java/arjdbc/postgresql/PostgreSQLResult.java index 4643db42a..020935263 100644 --- a/src/java/arjdbc/postgresql/PostgreSQLResult.java +++ b/src/java/arjdbc/postgresql/PostgreSQLResult.java @@ -193,7 +193,12 @@ public IRubyObject toARResult(final ThreadContext context) throws SQLException { } } - return super.toARResult(context); + final IRubyObject arResult = super.toARResult(context); + // Expose the affected/returned row count on ActiveRecord::Result#affected_rows + // (added in Rails 8.1). Set the ivar directly to avoid keyword-arg plumbing + // through the Java -> Ruby constructor call. + arResult.getInstanceVariables().setInstanceVariable("@affected_rows", length(context)); + return arResult; } /** @@ -258,12 +263,12 @@ public IRubyObject aref(ThreadContext context, IRubyObject rowArg) { return resultHash; } - // Note: this is # of commands (insert/update/selects performed) and not number of rows. In practice, - // so far users always just check this as to when it is 0 which ends up being the same as an update/insert - // where no rows were affected...so wrong value but the important value will be the same (I do not see - // how jdbc can do this). + // pg's PG::Result#cmd_tuples reports the number of rows affected by the command + // (for SELECT, the number of rows returned). The JDBC driver doesn't expose + // libpq's command-tuple count directly, but for the result sets we build it is + // equivalent to the number of rows we hold. @PG @JRubyMethod(name = {"cmdtuples", "cmd_tuples"}) public IRubyObject cmdtuples(ThreadContext context) { - return values.isEmpty() ? context.runtime.newFixnum(0) : aref(context, context.runtime.newFixnum(0)); + return length(context); } } diff --git a/src/java/arjdbc/postgresql/PostgreSQLRubyJdbcConnection.java b/src/java/arjdbc/postgresql/PostgreSQLRubyJdbcConnection.java index 51274ce95..ed55a36fa 100644 --- a/src/java/arjdbc/postgresql/PostgreSQLRubyJdbcConnection.java +++ b/src/java/arjdbc/postgresql/PostgreSQLRubyJdbcConnection.java @@ -66,6 +66,8 @@ import org.postgresql.geometric.PGpolygon; import org.postgresql.util.PGInterval; import org.postgresql.util.PGobject; +import org.postgresql.util.PSQLWarning; +import org.postgresql.util.ServerErrorMessage; /** * @@ -267,6 +269,27 @@ protected PostgreSQLResult mapExecuteResult(final ThreadContext context, final C return PostgreSQLResult.newResult(context, resultClass, this, resultSet); } + @Override + protected boolean collectsWarnings() { + return true; + } + + /** + * Extracts the PostgreSQL severity (e.g. "WARNING", "NOTICE") from a server + * warning so AR's db_warnings_action can filter by level. + */ + @Override + protected IRubyObject warningLevel(final ThreadContext context, final java.sql.SQLWarning warning) { + if (warning instanceof PSQLWarning) { + final ServerErrorMessage serverError = ((PSQLWarning) warning).getServerErrorMessage(); + if (serverError != null) { + final String severity = serverError.getSeverity(); + if (severity != null) return context.runtime.newString(severity); + } + } + return context.nil; + } + /** * Maps a query result set into a ActiveRecord result. * @param context diff --git a/test/db/postgresql/db_warnings_test.rb b/test/db/postgresql/db_warnings_test.rb new file mode 100644 index 000000000..cc17c3b75 --- /dev/null +++ b/test/db/postgresql/db_warnings_test.rb @@ -0,0 +1,87 @@ +require 'db/postgres' + +# Tests for SQL warning capture and dispatch via ActiveRecord.db_warnings_action. +# Most are negative: warnings that should NOT be surfaced (ignored by level, by +# the configured ignore list, or absent entirely). +class PostgresDbWarningsTest < Test::Unit::TestCase + + def setup + @connection = ActiveRecord::Base.connection + @saved_action = ActiveRecord.db_warnings_action + @saved_ignore = ActiveRecord.db_warnings_ignore + end + + def teardown + # The setter rejects nil (the :ignore default resolves to nil), so restore + # the resolved value directly. + ActiveRecord.instance_variable_set(:@db_warnings_action, @saved_action) + ActiveRecord.db_warnings_ignore = @saved_ignore + end + + def test_warning_is_captured_and_dispatched + warnings = capture_warnings do + @connection.execute("DO $$ BEGIN RAISE WARNING 'arjdbc test warning'; END $$;") + end + + assert_equal 1, warnings.size + warning = warnings.first + assert_kind_of ActiveRecord::SQLWarning, warning + assert_match(/arjdbc test warning/, warning.message) + assert_equal "WARNING", warning.level + assert_not_nil warning.code + end + + def test_notice_is_ignored_by_level + # NOTICE is below WARNING, so warning_ignored? filters it even with :raise. + warnings = capture_warnings do + @connection.execute("DO $$ BEGIN RAISE NOTICE 'just a notice'; END $$;") + end + + assert_equal 0, warnings.size + end + + def test_warning_filtered_by_db_warnings_ignore_message + ActiveRecord.db_warnings_ignore = [/arjdbc test warning/] + + warnings = capture_warnings do + @connection.execute("DO $$ BEGIN RAISE WARNING 'arjdbc test warning'; END $$;") + end + + assert_equal 0, warnings.size + end + + def test_no_warning_means_no_dispatch + warnings = capture_warnings do + @connection.select_value("SELECT 1") + end + + assert_equal 0, warnings.size + end + + def test_ignored_action_does_not_dispatch_but_drains_buffer + # Default action (:ignore -> nil) must not dispatch, and the underlying + # Java-side buffer must still be drained so it can't grow unbounded. + ActiveRecord.instance_variable_set(:@db_warnings_action, nil) + @connection.execute("DO $$ BEGIN RAISE WARNING 'drain me'; END $$;") + + assert_equal [], @connection.raw_connection.take_warnings + end + + def test_raise_action_raises_sql_warning + ActiveRecord.db_warnings_action = :raise + + assert_raises(ActiveRecord::SQLWarning) do + @connection.execute("DO $$ BEGIN RAISE WARNING 'boom'; END $$;") + end + end + + private + + def capture_warnings + collected = [] + ActiveRecord.db_warnings_action = ->(warning) { collected << warning } + yield + collected + end + +end diff --git a/test/db/postgresql/exception_translation_test.rb b/test/db/postgresql/exception_translation_test.rb new file mode 100644 index 000000000..3193fc83e --- /dev/null +++ b/test/db/postgresql/exception_translation_test.rb @@ -0,0 +1,129 @@ +require 'db/postgres' + +# Negative tests: invalid operations must be translated from the driver's +# SQLSTATE into the matching ActiveRecord exception subclass. +# See ArJdbc::PostgreSQL#translate_exception / #exception_class_for_sql_state. +class PostgresExceptionTranslationTest < Test::Unit::TestCase + + def setup + @connection = ActiveRecord::Base.connection + end + + def teardown + %w[et_child et_parent et_unique et_unique_ps et_notnull et_check et_excl + et_varchar et_int].each do |t| + @connection.execute("DROP TABLE IF EXISTS #{t}") + end + end + + def test_unique_violation + @connection.execute("CREATE TABLE et_unique (id integer PRIMARY KEY)") + @connection.execute("INSERT INTO et_unique (id) VALUES (1)") + + err = assert_raises(ActiveRecord::RecordNotUnique) do + @connection.execute("INSERT INTO et_unique (id) VALUES (1)") + end + assert_equal "23505", err.cause.sql_state + end + + def test_unique_violation_with_prepared_statement + omit "prepared statements disabled" unless @connection.prepared_statements + + @connection.execute("CREATE TABLE et_unique_ps (id integer PRIMARY KEY)") + # exec_query with binds exercises the prepared-statement path (execute_prepared_query). + # The JDBC driver uses '?' placeholders (not libpq's $1). + @connection.exec_query("INSERT INTO et_unique_ps (id) VALUES (?)", "SQL", [int_bind("id", 1)]) + + err = assert_raises(ActiveRecord::RecordNotUnique) do + @connection.exec_query("INSERT INTO et_unique_ps (id) VALUES (?)", "SQL", [int_bind("id", 1)]) + end + assert_equal "23505", err.cause.sql_state + end + + def test_foreign_key_violation + @connection.execute("CREATE TABLE et_parent (id integer PRIMARY KEY)") + @connection.execute("CREATE TABLE et_child (id integer PRIMARY KEY, parent_id integer REFERENCES et_parent (id))") + + err = assert_raises(ActiveRecord::InvalidForeignKey) do + @connection.execute("INSERT INTO et_child (id, parent_id) VALUES (1, 999)") + end + assert_equal "23503", err.cause.sql_state + end + + def test_not_null_violation + @connection.execute("CREATE TABLE et_notnull (id integer NOT NULL)") + + err = assert_raises(ActiveRecord::NotNullViolation) do + @connection.execute("INSERT INTO et_notnull (id) VALUES (NULL)") + end + assert_equal "23502", err.cause.sql_state + end + + def test_check_violation + @connection.execute("CREATE TABLE et_check (n integer, CONSTRAINT n_positive CHECK (n > 0))") + + err = assert_raises(ActiveRecord::CheckViolation) do + @connection.execute("INSERT INTO et_check (n) VALUES (-1)") + end + assert_equal "23514", err.cause.sql_state + end + + def test_exclusion_violation + # gist over a range type needs no extension (unlike btree_gist for scalars). + @connection.execute("CREATE TABLE et_excl (during tsrange, EXCLUDE USING gist (during WITH &&))") + @connection.execute("INSERT INTO et_excl (during) VALUES ('[2024-01-01, 2024-02-01)')") + + err = assert_raises(ActiveRecord::ExclusionViolation) do + @connection.execute("INSERT INTO et_excl (during) VALUES ('[2024-01-15, 2024-03-01)')") + end + assert_equal "23P01", err.cause.sql_state + end + + def test_value_too_long + @connection.execute("CREATE TABLE et_varchar (s varchar(3))") + + err = assert_raises(ActiveRecord::ValueTooLong) do + @connection.execute("INSERT INTO et_varchar (s) VALUES ('abcd')") + end + assert_equal "22001", err.cause.sql_state + end + + def test_numeric_value_out_of_range + @connection.execute("CREATE TABLE et_int (n integer)") + + err = assert_raises(ActiveRecord::RangeError) do + @connection.execute("INSERT INTO et_int (n) VALUES (2147483648)") + end + assert_equal "22003", err.cause.sql_state + end + + def test_statement_invalid_on_missing_relation + assert_raises(ActiveRecord::StatementInvalid) do + @connection.execute("SELECT * FROM et_does_not_exist") + end + end + + def test_statement_invalid_on_syntax_error + assert_raises(ActiveRecord::StatementInvalid) do + @connection.execute("SELECT FROM WHERE") + end + end + + # Serialization/deadlock failures are timing-dependent; we only assert the + # SQLSTATE mapping table covers them rather than provoking a flaky race. + def test_serialization_and_deadlock_codes_are_mapped + adapter = @connection + mapped = adapter.send(:exception_class_for_sql_state, "40001") + assert_equal ActiveRecord::SerializationFailure, mapped + assert_equal ActiveRecord::Deadlocked, adapter.send(:exception_class_for_sql_state, "40P01") + assert_equal ActiveRecord::LockWaitTimeout, adapter.send(:exception_class_for_sql_state, "55P03") + assert_equal ActiveRecord::QueryCanceled, adapter.send(:exception_class_for_sql_state, "57014") + end + + private + + def int_bind(name, value) + ActiveRecord::Relation::QueryAttribute.new(name, value, ActiveRecord::Type::Integer.new) + end + +end diff --git a/test/db/postgresql/version_test.rb b/test/db/postgresql/version_test.rb index ddd143801..7cdb8a59f 100644 --- a/test/db/postgresql/version_test.rb +++ b/test/db/postgresql/version_test.rb @@ -53,6 +53,7 @@ def connection_stub(version, full_version = false) raw_connection = mock('raw_connection') raw_connection.stubs(:execute) raw_connection.stubs(:exec_params) + raw_connection.stubs(:take_warnings).returns([]) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.stubs(:new_client).returns(raw_connection) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.any_instance.stubs(:initialize_type_map) diff --git a/test/simple.rb b/test/simple.rb index 2e6610d71..d79597248 100644 --- a/test/simple.rb +++ b/test/simple.rb @@ -573,9 +573,11 @@ def test_empty_insert_statement end def test_column_default - assert_equal '3.14', DbType.columns_hash['sample_small_decimal'].default - assert_equal '-1', DbType.columns_hash['sample_integer_neg_default'].default - assert_equal '42', DbType.columns_hash['sample_integer_no_limit'].default + # Since Rails 8.1 ConnectionAdapters::Column#default returns the value + # deserialized through the column's cast type (was the raw string before). + assert_equal BigDecimal('3.14'), DbType.columns_hash['sample_small_decimal'].default + assert_equal(-1, DbType.columns_hash['sample_integer_neg_default'].default) + assert_equal 42, DbType.columns_hash['sample_integer_no_limit'].default assert_equal '', DbType.columns_hash['sample_string'].default assert_equal(-1, DbType.column_defaults['sample_integer_neg_default'])