AopInvocationException when connecting to IBM MQ from an EAR application when deployed on CloudFoundry

[javierpalau]

Hi,

we are integrating our EAR application with an IBM MQ Broker and when we send a message we obtain de following error:

2018-04-04T22:49:16.41+0200 [App/0] ERR org.springframework.aop.AopInvocationException: AOP configuration seems to be invalid: tried calling method [public abstract javax.jms.Connection javax.jms.ConnectionFactory.createConnection() throws javax.jms.JMSException] on target [com.ibm.mq.connector.outbound.ConnectionFactoryImpl@7dce618e]; nested exception is java.lang.IllegalArgumentException: object is not an instance of declaring class 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:325) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201) 2018-04-04T22:49:16.41+0200 [App/0] ERR at com.sun.proxy.$Proxy66.createConnection(Unknown Source) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:180) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:484) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:580) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:706) 2018-04-04T22:49:16.41+0200 [App/0] ERR at internal.JournalWriterImpl.sendMessage(JournalWriterImpl.java:40) 2018-04-04T22:49:16.41+0200 [App/0] ERR at internal.JournalBusinessWriterImpl.sendMessage(JournalBusinessWriterImpl.java:75) 2018-04-04T22:49:16.41+0200 [App/0] ERR at internal.JournalRecordAdapterBusinessImpl.writeData(JournalRecordAdapterBusinessImpl.java:132) 2018-04-04T22:49:16.41+0200 [App/0] ERR at internal.JournalRecordAdapterBusinessImpl.writeReceive(JournalRecordAdapterBusinessImpl.java:73) 2018-04-04T22:49:16.41+0200 [App/0] ERR at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2018-04-04T22:49:16.41+0200 [App/0] ERR at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 2018-04-04T22:49:16.41+0200 [App/0] ERR at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2018-04-04T22:49:16.41+0200 [App/0] ERR at java.lang.reflect.Method.invoke(Method.java:606) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:775) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) 2018-04-04T22:49:16.41+0200 [App/0] ERR at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:965) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:867) 2018-04-04T22:49:16.42+0200 [App/0] ERR at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:841) 2018-04-04T22:49:16.42+0200 [App/0] ERR at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.openejb.server.httpd.EEFilter.doFilter(EEFilter.java:65) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.cloudfoundry.router.ClientCertificateMapper.doFilter(ClientCertificateMapper.java:77) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:44) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.tomee.catalina.OpenEJBSecurityListener$RequestCapturer.invoke(OpenEJBSecurityListener.java:97) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1457) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) 2018-04-04T22:49:16.42+0200 [App/0] ERR at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2018-04-04T22:49:16.42+0200 [App/0] ERR at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) 2018-04-04T22:49:16.42+0200 [App/0] ERR at java.lang.Thread.run(Thread.java:745) 2018-04-04T22:49:16.42+0200 [App/0] ERR Caused by: java.lang.IllegalArgumentException: object is not an instance of declaring class 2018-04-04T22:49:16.42+0200 [App/0] ERR at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2018-04-04T22:49:16.42+0200 [App/0] ERR at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 2018-04-04T22:49:16.42+0200 [App/0] ERR at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2018-04-04T22:49:16.42+0200 [App/0] ERR at java.lang.reflect.Method.invoke(Method.java:606) 2018-04-04T22:49:16.42+0200 [App/0] ERR at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) 2018-04-04T22:49:16.42+0200 [App/0] ERR ... 60 more

This problem only occurs when we use the TomEE Buildpack deployed on CloudFoundry. If you deploy the same EAR in a TomEE running locally, the message is sent correctly.

We are using (locally and in CloudFoundry) the Tomee-7.0.4 buildpack (https://java-buildpack.cloudfoundry.org/tomee/tomee-7.0.4.tar.gz).

Our deployed EAR has the following structure:

MyApp.ear
| META-INF
| | application.xml
| | resource.xml
| drivers
| | com.ibm.*.jar (when we deploy this ear locally, jars in this folder are copied in tomme/lib manually)
| MyApp.war

MQ resources (Resource adapter, Connection factory and Queue) are defined in the resource.xml file.

Is there any differences between a "local" TomEE and a TomEE deployed in CloudFoundry that could generate this problem?

Thanks.

[javierpalau]

Hi, we have tried more configurations and we have found that if we define the MQQueueConnectionFactory using an Spring bean (not a TomEE resource) it works.

For example:

### <bean id="jmsConnectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory"> 
<property name="queueManager"> 
<value>QM_XXX</value> 
</property> 
<property name="hostName"> 
<value>localhost</value> 
</property> 
<property name="port"> 
<value>1414</value> 
</property> 
<property name="channel"> 
<value>XXX_SER_CH</value> 
</property> 

Although this configuration works, we would prefer to configure this resources using the tomee.xml because it provides a better SoC for us.
We think that this problem is caused by a different classloader hierarchy between a local Tomee, and a deployed CloudFoundry Tomee. Is it possible to configure the TomEE buildpack to use the same classloader hierarchy as a “local” TomEE?

Thanks,
Javi

[violetagg]

Hi,

This problem only occurs when we use the TomEE Buildpack deployed on CloudFoundry. If you deploy the same EAR in a TomEE running locally, the message is sent correctly.

What TomEE flavour do you use for the local development?

TomEE flavour that is provided by TomEE buildpack is the Web Profile.
https://github.com/cloudfoundry-community/tomee-buildpack/blob/master/docs/container-tomee.md#support-for-deploying-an-ear-enterprise-application-archive-file

Regards,
Violeta

[javierpalau]

Hi, we are using the same flavour and version (apache-tomee-webprofile-7.0.4) that is provided by the TomEE buildpack.
It is downloaded from https://java-buildpack.cloudfoundry.org/tomee/tomee-7.0.4.tar.gz when we deploy/publish our application.

Regards,
Javi.