Merge pull request #51 from Sage/connection-pooling

FOX-3833 - Connection pooling

Author: PawelKlapec

.github/workflows/rspec.yml

@@ -7,7 +7,7 @@ jobs:
 
     services:
       mysql:
-        image: mysql:5.7
+        image: mysql:8.0
         ports:
           - 3306:3306
         env:
@@ -19,19 +19,11 @@ jobs:
           - 6379:6379
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v6
       - uses: ruby/setup-ruby@v1
         with:
-          ruby-version: 2.4
+          ruby-version: 4.0
           bundler-cache: true
 
       - name: Run tests
         run: bundle exec rspec
-
-      - name: Code Coverage
-        uses: paambaati/codeclimate-action@v2.7.5
-        env:
-          CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}
-        with:
-          coverageLocations: |
-            ${{github.workspace}}/coverage/.resultset.json:simplecov

Gemfile

@@ -7,5 +7,6 @@ group :test, :development do
 end
 
 group :test do
-  gem 'simplecov', '0.17.1', require: false
+  gem 'simplecov', require: false
+  gem 'debug', require: false
 end

README.md

@@ -34,8 +34,11 @@ gem 'mysql_framework'
 
 #### MySQL Connection Pooling Variables
 
-* `MYSQL_START_POOL_SIZE` - how many connections should be created by default (default: `1`)
+* `MYSQL_CONNECTION_POOL_ENABLED` - enables/disables pooling (default: `true`)
 * `MYSQL_MAX_POOL_SIZE` - how many connections should the pool be allowed to grow to (default: `5`)
+* `MYSQL_POOL_TIMEOUT` - how long to wait for a pooled connection before timing out (default: `5` seconds)
+* `MYSQL_POOL_IDLE_TIMEOUT` - how long a pooled connection can remain idle before being reaped (default: `300` seconds)
+* `MYSQL_POOL_IDLE_REAP_TIME` - time interval between background thread checking for idle connections to reap (default: `60` seconds)
 
 #### MySQL Migration Variables
 
@@ -161,31 +164,33 @@ MysqlFramework::Connector.new(options)
 
 #### #setup
 
-Sets up the connection pooling. Creates `ENV['MYSQL_START_POOL_SIZE']` `Mysql2::Client` instances up front. This is provided as a separate method to allow for use within process forking where connections would need to be created after forking the process.
+Sets up connection pooling using `connection_pool` with `ENV['MYSQL_MAX_POOL_SIZE']` and `ENV['MYSQL_POOL_TIMEOUT']`. Connections are created lazily by the pool when first needed.
 
 ```ruby
 connector.setup
 ```
 
 #### #dispose
 
-Closes all the `Mysql2::Client` connections and removes the connection pool. Intended as a clean-up method to be used on process fork shutdown.
+Closes pooled `Mysql2::Client` connections and removes the pool. Intended as a clean-up method to be used on process fork shutdown.
 
 ```ruby
 connector.dispose
 ```
 
 #### #check_out
 
-Check out a client from the connection pool. Will create new `Mysql2::Client` instances up-to `ENV['MYSQL_MAX_POOL_SIZE']` times if no idle connections are available.
+Checks out a `Mysql2::Client` instance from the pool, sanitizes it, and returns it.
+When pooling is disabled, it returns a newly created client.
 
 ```ruby
 client = connector.check_out
 ```
 
 #### #check_in
 
-Check in a client to the connection pool
+Checks a client back in to the pool.
+When pooling is disabled, it closes the provided client.
 
 ```ruby
 client = connector.check_out
@@ -195,7 +200,17 @@ connector.check_in(client)
 
 #### #with_client
 
-Called with a block. The method checks out a client from the pool and yields it to the block. Finally it ensures that the client is always checked back into the pool.
+Called with a block. The method obtains a client (from the pool when enabled), yields it to the block, and guarantees cleanup.
+
+When pooling is enabled, it uses the pool lifecycle (`ConnectionPool#with`) and supports optional discarding of the current pooled connection:
+
+```ruby
+connector.with_client(discard_current_pool_connection: true) do |client|
+  # use client
+end
+```
+
+When pooling is disabled, it creates a fresh client for the block and closes it afterwards.
 
 ```ruby
 connector.with_client do |client|
@@ -205,6 +220,17 @@ connector.with_client do |client|
 end
 ```
 
+**Warning: re-entrant connections within the same thread**
+
+The `connection_pool` gem implements thread-local connection tracking. When a thread already holds a connection via `with_client` (or `check_out`), any nested call to `with_client` or `check_out` on the **same thread** returns the **same connection** — it does not check out a second one from the pool.
+
+This means that if you fire an async query on a connection and then attempt to run a second query (e.g. via `run_query` or `connector.query`) from within the same `with_client` block, the nested call will receive the already-checked-out connection. Sanitization will then fail with:
+
+```
+Connection sanitization failed: This connection is still waiting for a result,
+try again once you have the result
+```
+
 It can optionally accept an existing client to avoid starting new connections in the middle of a transaction. This can be used to ensure that a series of queries are wrapped by the same transaction.
 
 ```ruby
@@ -280,10 +306,65 @@ The default options used to initialise MySQL2::Client instances:
   database: ENV.fetch('MYSQL_DATABASE'),
   username: ENV.fetch('MYSQL_USERNAME'),
   password: ENV.fetch('MYSQL_PASSWORD'),
-  reconnect: true
+  reconnect: true,
+  read_timeout: Integer(ENV.fetch('MYSQL_READ_TIMEOUT', 30)),
+  write_timeout: Integer(ENV.fetch('MYSQL_WRITE_TIMEOUT', 10))
 }
 ```
 
+### MysqlFramework::Stats::AwsMetricPublisher
+
+Publishes connection-pool metrics (`size`, `available`, `idle`) to AWS CloudWatch on a configurable interval via a background thread.
+
+**Setup sequence** — the connector must be set up before the publisher is started:
+
+```ruby
+connector = MysqlFramework::Connector.new
+connector.setup  # must come first
+
+publisher = MysqlFramework::Stats::AwsMetricPublisher.new(
+  connector: connector,
+  publish_interval: 300  # seconds, default
+)
+publisher.start
+```
+
+On shutdown, stop the publisher before disposing the connector:
+
+```ruby
+publisher.stop
+connector.dispose
+```
+
+#### Customising CloudWatch dimensions and namespace
+
+Use `MysqlFramework::Stats::DimensionMap` to configure the CloudWatch namespace and dimensions. Each attribute falls back to the corresponding environment variable when not set explicitly:
+
+| Attribute | ENV fallback | CloudWatch dimension |
+|---|---|---|
+| `service_name` | `SERVICE_NAME` | `ServiceName` |
+| `application` | `APPLICATION` | `Application` |
+| `environment` | `ENVIRONMENT` | `Environment` |
+| `landscape` | `LANDSCAPE` | `Landscape` |
+| `namespace` | `AWS_METRICS_NAMESPACE` | (namespace, default: `MysqlFramework`) |
+
+```ruby
+dimension_map = MysqlFramework::Stats::DimensionMap.new(
+  service_name: 'my-service',
+  application:  'my-app',
+  environment:  'production',
+  landscape:    'us-east',
+  namespace:    'MyCompany/MySQL'
+)
+
+publisher = MysqlFramework::Stats::AwsMetricPublisher.new(
+  connector:        connector,
+  dimension_map:    dimension_map,
+  publish_interval: 60
+)
+publisher.start
+```
+
 ### MysqlFramework::SqlCondition
 
 A representation of a MySQL Condition for a column. Created automatically by SqlColumn

docker-compose.yml

@@ -1,8 +1,6 @@
-version: '2.1'
-
 services:
   test-runner:
-    image: ruby:2.4
+    image: ruby:4.0
     working_dir: /usr/src/app
     container_name: test-runner
     command: sh -c "while true; do echo 'Container is running..'; sleep 5; done"
@@ -21,7 +19,7 @@ services:
 
   test-mysql:
     container_name: test-mysql
-    image: mysql:5.7
+    image: mysql:8
     restart: always
     environment:
       MYSQL_ROOT_PASSWORD: admin