[fix][admin] Add javax.xml.bind:jaxb-api to shade by nodece · Pull Request #20106 · apache/pulsar

nodece · 2023-04-14T17:40:07Z

Fixes #19856

Motivation

javax.xml.bind:jaxb-api didn't package into the shade package.

Add a context, why add javax.xml.bind:jaxb-api to the admin module, see https://www.jesperdj.com/2018/09/30/jaxb-on-java-9-10-11-and-beyond/

Modifications

Add javax.xml.bind:jaxb-api to pulsar-client-admin-shaded/pom.xml.

Documentation

doc
doc-required
doc-not-needed
doc-complete

merlimat · 2023-04-15T16:33:19Z

/pulsarbot rerun-failure-checks

tisonkun

Thank you!

onobc · 2023-04-15T21:17:42Z

Is there test coverage for a case where the admin API PulsarAdmin.topics().getList("public/default") is executed against a standalone server? The reason I ask is because the 2.11.1 branch w/o the changes in this PR are fine but w/ these changes in the PR a call to the above API just times out w/ the following output in the logs:

2023-04-15T16:11:43.776-05:00 WARN 4356 --- [ main] o.a.p.s.o.g.j.m.i.MessagingBinders : A class org.apache.pulsar.shade.javax.xml.transform.stream.StreamSource for a default provider MessageBodyReader<org.apache.pulsar.shade.javax.xml.transform.stream.StreamSource> was not found. The provider is not available.
2023-04-15T16:11:43.777-05:00 WARN 4356 --- [ main] o.a.p.s.o.g.j.m.i.MessagingBinders : A class org.apache.pulsar.shade.javax.xml.transform.sax.SAXSource for a default provider MessageBodyReader<org.apache.pulsar.shade.javax.xml.transform.sax.SAXSource> was not found. The provider is not available.
2023-04-15T16:11:43.777-05:00 WARN 4356 --- [ main] o.a.p.s.o.g.j.m.i.MessagingBinders : A class org.apache.pulsar.shade.javax.xml.transform.Source for a default provider MessageBodyWriter<org.apache.pulsar.shade.javax.xml.transform.Source> was not found. The provider is not available.
2023-04-15T16:11:43.777-05:00 WARN 4356 --- [ main] o.a.p.s.o.g.j.m.i.MessagingBinders : A class javax.activation.DataSource for a default provider MessageBodyWriter<javax.activation.DataSource> was not found. The provider is not available.
2023-04-15T16:11:43.777-05:00 WARN 4356 --- [ main] o.a.p.s.o.g.j.m.i.MessagingBinders : A class org.apache.pulsar.shade.javax.xml.transform.dom.DOMSource for a default provider MessageBodyReader<org.apache.pulsar.shade.javax.xml.transform.dom.DOMSource> was not found. The provider is not available.

I am concerned that shading the javax.xml.bind may need updates in other areas? If someone else could verify that they are NOT seeing this behaviour w/ these changes that would be comforting.

Thanks

onobc · 2023-04-16T03:21:28Z

I am concerned that shading the javax.xml.bind may need updates in other areas? If someone else could verify that they are NOT seeing this behaviour w/ these changes that would be comforting.

Thanks

I believe the current relocation is too aggressive as it attempts to relocate JDK classes (javax.xml.DOMSource) in MessageBinders but will not shade those classes actually because they are JDK classes.

I got further by tightening the shading to javax.xml.bind w/

 <include>javax.xml.bind:jaxb-api</include>

instead of

<include>javax.xml.bind:*</include>

and for relocation:

<relocation>
  <pattern>javax.xml.bind</pattern>
  <shadedPattern>org.apache.pulsar.shade.javax.xml.bind</shadedPattern>
</relocation>

instead of

<relocation>
  <pattern>javax.xml</pattern>
  <shadedPattern>org.apache.pulsar.shade.javax.xml</shadedPattern>
</relocation>

The warnings now go away in the app log and the call does not timeout. ~~However, another error now occurs. I am debugging and hopefully will have some info soon.~~
EDIT
The new exception must have been due to something else as it has since went away and everything is running smoothly now for me.

@nodece Can you adjust the shading as described above?

Thanks

nodece · 2023-04-16T09:05:49Z

Hi @onobc, thank you for your research! This PR has been updated.

tisonkun · 2023-04-16T13:53:28Z

Test failure can be related:

Error:  Tests run: 18, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 55.518 s <<< FAILURE! - in org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest
  Error:  initializeState(org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest)  Time elapsed: 34.049 s  <<< FAILURE!
  org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: HTTP 500 {}
  	at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
  	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352)
  	at org.apache.pulsar.client.admin.internal.NamespacesImpl.unload(NamespacesImpl.java:792)
  	at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.initializeState(ExtensibleLoadManagerImplTest.java:173)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:590)
  	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
  	at org.testng.TestRunner.run(TestRunner.java:602)
  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
  	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
  	at org.testng.TestNG.runSuites(TestNG.java:1099)
  	at org.testng.TestNG.run(TestNG.java:1067)
  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90)
  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
  	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
  	Suppressed: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: HTTP 500 {}
  		at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:272)
  		at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:136)
  		at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
  		at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863)
  		at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
  		at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
  		at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
  		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
  		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
  		at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
  		at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
  		at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
  		at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
  		at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
  		at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254)
  		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
  		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
  		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
  		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
  		at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296)
  		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
  		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
  		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
  		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
  		at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
  		at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
  		at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
  		at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
  		at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
  		at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  		at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
  		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
  		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
  		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
  		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
  		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
  		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
  		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
  		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
  		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
  		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  		at java.base/java.lang.Thread.run(Thread.java:833)
  	Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {}
  		at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960)
  		at org.glassfish.jersey.client.JerseyInvocation.access$[700](https://github.com/apache/pulsar/actions/runs/4712460077/jobs/8357595863#step:11:701)(JerseyInvocation.java:82)
  		... 54 more
  Caused by: [CIRCULAR REFERENCE: javax.ws.rs.InternalServerErrorException: HTTP 500 {}]

nodece · 2023-04-16T16:21:29Z

/pulsarbot rerun-failure-checks

Signed-off-by: nodece <nodeces@gmail.com>

cbornet · 2023-04-18T17:01:02Z

@merlimat we are normally in code freeze for 3.0. Do we do an exception for this one ?

onobc · 2023-04-18T17:32:37Z

FWIW - this change is one that is required for Spring Pulsar to get it's auto-configuration moved into Spring Boot (3.2) project. Thanks.

merlimat · 2023-04-18T19:14:06Z

@cbornet Yes, this a very safe and understood change, that would not add more risk to the RC-2.

Signed-off-by: nodece <nodeces@gmail.com>

cbornet · 2023-04-18T22:39:55Z

Cherry-picked to branch-3.0

onobc · 2023-05-03T03:22:34Z

I don't see this one in the 3.0.0 Release notes but I do see the changes in the branch-3.0. Really I just want to make sure the fix is in 3.0.0 and there is not something else I am missing.

Thanks

tisonkun · 2023-05-03T03:34:34Z

cc @RobertIndie perhaps you can investigate why the release note is unsync?

RobertIndie · 2023-05-03T04:06:51Z

The release note is collected by filtering the label like component/admin. This PR is not labeled with component/admin. So it's not included in the release note.

It's hard for the release manager to collect and classify all the unlabeled PRs. I encourage we should label all the PRs correctly before merging the PR.

tisonkun · 2023-07-12T15:01:44Z

Cross-reference - apache/flink-connector-pulsar#54

This include a new dependency rejected by Flink's LicenseChecker, while it should be a false positive.

github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Apr 14, 2023

nodece mentioned this pull request Apr 14, 2023

Update jaxb and jax-rs libs #19856

Closed

2 tasks

nodece requested review from lhotari and tisonkun April 14, 2023 17:42

onobc reviewed Apr 14, 2023

View reviewed changes

Comment thread pulsar-client-admin-shaded/pom.xml Outdated

nodece force-pushed the fix-admin-java.xml branch from c94f3f0 to 8d65b1a Compare April 15, 2023 15:51

merlimat approved these changes Apr 15, 2023

View reviewed changes

merlimat added the ready-to-test label Apr 15, 2023

merlimat added this to the 3.0.0 milestone Apr 15, 2023

tisonkun approved these changes Apr 15, 2023

View reviewed changes

nodece force-pushed the fix-admin-java.xml branch from 8d65b1a to 0eeab81 Compare April 16, 2023 09:02

nodece changed the title ~~[fix][admin] Add javax.xml to shade~~ [fix][admin] Add javax.xml.bind:jaxb-api to shade Apr 16, 2023

[fix][admin] Add javax.xml.bind:jaxb-api to shade

c30cb73

Signed-off-by: nodece <nodeces@gmail.com>

nodece force-pushed the fix-admin-java.xml branch from 0eeab81 to c30cb73 Compare April 18, 2023 15:30

nodece merged commit 78cf5a6 into apache:master Apr 18, 2023

cbornet pushed a commit that referenced this pull request Apr 18, 2023

[fix][admin] Add javax.xml.bind:jaxb-api to shade (#20106)

43ef457

Signed-off-by: nodece <nodeces@gmail.com>

tisonkun mentioned this pull request May 3, 2023

[improve][misc] Allow custom Netty DnsServerAddressStreamProviders #18646

Merged

5 tasks

nodece added area/admin area/client labels May 3, 2023

Uh oh!

Conversation

nodece commented Apr 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Documentation

Uh oh!

Uh oh!

merlimat commented Apr 15, 2023

Uh oh!

tisonkun left a comment

Choose a reason for hiding this comment

Uh oh!

onobc commented Apr 15, 2023

Uh oh!

onobc commented Apr 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodece commented Apr 16, 2023

Uh oh!

tisonkun commented Apr 16, 2023

Uh oh!

nodece commented Apr 16, 2023

Uh oh!

cbornet commented Apr 18, 2023

Uh oh!

onobc commented Apr 18, 2023

Uh oh!

merlimat commented Apr 18, 2023

Uh oh!

cbornet commented Apr 18, 2023

Uh oh!

onobc commented May 3, 2023

Uh oh!

tisonkun commented May 3, 2023

Uh oh!

RobertIndie commented May 3, 2023

Uh oh!

tisonkun commented Jul 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nodece commented Apr 14, 2023 •

edited

Loading

onobc commented Apr 16, 2023 •

edited

Loading