Example code for automatic reconnection

[candlerb]

Related to #13: it would be really helpful if there was some example code showing the correct pattern to reconnect to a server when a problem occurs.

The following appears to work (tested using HornetQ 2.2.5 as the server) but I'm not sure if this is a safe or recommended approach.

connection = nil
reconnect = lambda do |exc|
  begin
    connection = JMS::Connection.new(config)
    connection.on_exception(&reconnect)
    connection.start
  rescue => e
    $stderr.puts e
    sleep 5
    retry
  end
end
reconnect.call(nil)

Producer:

count = 0
loop do
  begin
    connection.session do |session|
      session.producer(:queue_name => 'test') do |producer|
        m = "Hello World #{count}"
        puts m
        producer.send(session.message(m))
        count += 1
        sleep 3
      end
    end
  rescue javax.jms.JMSException => e
    puts "Send failed (#{e})"
    sleep 1
  end
end

Consumer:

loop do
  begin
    connection.session do |session|
      session.consume(:queue_name => 'test', :timeout => -1) do |message|
          p message
          p message.jms_reply_to
      end
    end
  rescue javax.jms.JMSException => e
    puts "Consume failed (#{e})"
    sleep 1
  end
end

[candlerb]

I should add that the following does NOT work:

Thread.abort_on_exception = true
connection = JMS::Connection.new(config)
connection.on_exception { exit 1 }
connection.start

It turns out that the SystemExit exception is being caught and preventing termination. The program continues to run after logging:

Oct 01, 2015 4:06:07 PM org.hornetq.core.logging.impl.JULLogDelegate warn
WARNING: Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
Exception in thread "Thread-4" org.jruby.exceptions.RaiseException: (SystemExit) exit
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:705)
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:668)
    at x.block in x.rb(x.rb:27)

Solution here is to change exit to exit! but it's non-obvious that exceptions in the on_exception handler will be caught and discarded.

[reidmorrison]

The age old challenge of what to do when a server connection is lost. In Rails ActiveRecord performs a ping on the database connection every time it gets a connection out the pool before using. It ensures that the connection is good before trying to use it. The obvious cost is the additional network hop to the database server, but the upside is much better resilience.

I recently had to add transparent retries for our MongoDB client connections and created a gem called mongo_ha. It was very difficult to add the automatic re-connect capability to an existing client library.

Reconnecting to a server is hard, especially considering all the various calls that would need to have a retry. The "ping" solution is a good option, but need to figure out a portable way of "pinging" over a connection before using the connection. However the ping does not resolve transient network failures that occur during processing.

The last time I added a retry was with a native WebSphere MQ client. It rolled back the current transaction and then started it from the beginning again when a connection failure occurred. Of course we would have to rollback any database changes made during the incomplete processing. It also assumes no calls were made to external providers.

I will keep thinking about this one.

Thank you for the feedback, keep it coming.

[candlerb]

For my use I'm not really worried about retrying if a message send fails; that can be left to application logic, as long as it can detect the failure, and has a way to re-establish the connection.

It's a bit painful if the JMS::Connection is established at one point in the code, but the connection.session() fails at a distant part of the code. Hence using on_exception to create a new connection object automatically is attractive.

I had to go digging and reverse-engineering to find out about this, so it would be nice if there were something in the jruby-jms examples directory. Arguably this really belongs in JMS API documentation, but the on_exception(&block) method is a jruby-jms method :-)

[reidmorrison]

I agree, there has to be a better way to determine connection loss early. Ideally without having to resort to the on_exception callback. This will be an interesting challenge.