Update okhttp to 5.3.2

[jonesbusy]

Update okhttp to 5.x

One of the reason is that some upstream dependencies require 5.X (typically kubernetes-client consumed via kubernetes-client-api plugin) (See: fabric8io/kubernetes-client#7422)

Testing done

Full bom test on jenkinsci/bom#6429

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests that demonstrate the feature works or the issue is fixed

[jglick]

update from 4.x to 5.x is supposed to be binary compatible

So it is not necessary to introduce a separate plugin for okhttp 5?

I think this deserve full bom test?

If you have to ask, the answer is yes.

[jonesbusy]

I don't think it's needed. From what I saw binary compatibility was retained for major (non-alpha) versions.

Update from 3x to 4x was also done on same plugin: #55

Putting this PR in draft until bom tests are done

[jonesbusy]

https://github.com/jenkinsci/bom/pull/6429/checks?check_run_id=65826643820 confirm no issue.

This is aligned with release notes indicating binary compatiblity between 4 and 5 (non-alpha version) (CF : https://square.github.io/okhttp/changelogs/changelog/)

[jonesbusy]

@jenkinsci/company-cloudbees-developers @jenkinsci/okhttp-api-plugin-developers Would you mind taking a look at it? I'm also not sure if you have also internal tests to run inside CB

This is required for for kubernetes-api-client update to 7.6.x (Support K8S 1.35)

I will perform some test on all dependends plugin when this PR is merged and released

Many thanks!

[tumaf33]

@jonesbusy @jtnord

The OkHttp 5.x FastFallbackExchangeFinder throws InterruptedException after extended uptime, breaking all GitHub API calls

Here is more details:

Environment
Jenkins: 2.504.1
Java: JDK 17
okhttp-api plugin: 5.3.2-200 (OkHttp 5.x)
Platform: EKS

Problem
After upgrading from okhttp-api 4.x to 5.3.2, all Jenkins jobs started failing to connect to GitHub after approximately 15 days of uptime. The GitHub App credential could not generate installation tokens, and "Test Connection" from the Credentials plugin returned a 504 error.

A second Jenkins environment on the same VPC, same EKS cluster, running okhttp-api 4.x (OkHttp 4.x) did not exhibit this issue.

Stack Trace

java.lang.InterruptedException
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base/java.util.concurrent.LinkedBlockingDeque.pollFirst(Unknown Source)
    at java.base/java.util.concurrent.LinkedBlockingDeque.poll(Unknown Source)
    at PluginClassLoader for okhttp-api//okhttp3.internal.connection.FastFallbackExchangeFinder.awaitTcpConnect(FastFallbackExchangeFinder.kt:162)
    at PluginClassLoader for okhttp-api//okhttp3.internal.connection.FastFallbackExchangeFinder.find(FastFallbackExchangeFinder.kt:69)
    at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:298)
    at PluginClassLoader for okhttp-api//okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
    at PluginClassLoader for okhttp-api//okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
    at PluginClassLoader for okhttp-api//okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:101)
    at PluginClassLoader for okhttp-api//okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
    at PluginClassLoader for okhttp-api//okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:85)
    at PluginClassLoader for okhttp-api//okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
    at PluginClassLoader for okhttp-api//okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:74)
    at PluginClassLoader for okhttp-api//okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
    at PluginClassLoader for github-api//org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$UnexpectedException.lambda$static$0(ObsoleteUrlFactory.java:1364)
    at PluginClassLoader for okhttp-api//okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
    at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:226)
    at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall.execute(RealCall.kt:178)
    at PluginClassLoader for github-api//org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$OkHttpURLConnection.getResponse(ObsoleteUrlFactory.java:670)
    at PluginClassLoader for github-api//org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$OkHttpURLConnection.getResponseCode(ObsoleteUrlFactory.java:701)
    at PluginClassLoader for github-api//org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$DelegatingHttpsURLConnection.getResponseCode(ObsoleteUrlFactory.java:1063)
    at PluginClassLoader for github-api//org.kohsuke.github.internal.GitHubConnectorHttpConnectorAdapter.send(GitHubConnectorHttpConnectorAdapter.java:87)
    at PluginClassLoader for github-api//org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:461)
    at PluginClassLoader for github-api//org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:427)
    at PluginClassLoader for github-api//org.kohsuke.github.Requester.fetch(Requester.java:85)
    at PluginClassLoader for github-api//org.kohsuke.github.GitHub.getApp(GitHub.java:1185)
    at PluginClassLoader for github-branch-source//org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:291)
Caused: java.lang.IllegalArgumentException: Failed to generate GitHub App installation token for app ID 1234567
    at PluginClassLoader for github-branch-source//org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.generateAppInstallationToken(GitHubAppCredentials.java:354)
    at PluginClassLoader for github-branch-source//org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getToken(GitHubAppCredentials.java:393)
    at PluginClassLoader for github-branch-source//org.jenkinsci.plugins.github_branch_source.GitHubAppCredentials.getPassword(GitHubAppCredentials.java:429)
    at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createPasswordFile(CliGitAPIImpl.java:2550)
    at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2147)
    at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:638)
    at PluginClassLoader for git//jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:408)
    at PluginClassLoader for scm-api//jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:219)
    at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:120)
    at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:72)
    at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:317)
    at hudson.model.ResourceController.execute(ResourceController.java:101)
    at hudson.model.Executor.run(Executor.java:446)

From what I understand, FastFallbackExchangeFinder is new in OkHttp 5.x (does not exist in 4.x). Its awaitTcpConnect method uses LinkedBlockingDeque.poll() with awaitNanos, which is interruptible, it checks the calling thread's interrupt status. In addition, Jenkins routinely interrupts threads during build cancellations and timeouts. interrupt flags appear to accumulate on OkHttp's TaskRunner thread pool, eventually causing every HTTP connection attempt to throw InterruptedException.

This did not occur with OkHttp 4.x.

How I fixed?
Restarting the Jenkins controller (pod restart) probably clears the JVM thread state and resolves the issue.

Questions
Is OkHttp 5.x's FastFallbackExchangeFinder properly handling thread interrupts in the Jenkins context?
Should the plugin clear or ignore interrupt flags before making OkHttp calls?
Would it be feasible to pin OkHttp to 4.x until this is resolved upstream?

I appreciate your response and advise.

[bitwiseman]

@tumaf33 File an issue for this rather than commenting on merged PR.

[tumaf33]

@bitwiseman - the Issues tab is missing, can someone please enable it so I can open one?

[jtnord]

the Issues tab is missing, can someone please enable it so I can open one?

I doubt this is an issue in this project - the code here does not appear to be involved, more likely I would suggest an issue in either github-branch-source (or whatever is creating the OKHttpClient), or upstream in the okhttp-api itself.

Not all plugins use GitHub issues, you can find the issue tracker using https://plugins.jenkins.io/ and then entering the plugin followed by "report new issue".

From what I understand, FastFallbackExchangeFinder is new in OkHttp 5.x (does not exist in 4.x). Its awaitTcpConnect method uses LinkedBlockingDeque.poll() with awaitNanos, which is interruptible, it checks the calling thread's interrupt status.

Which then throws an InteruptedException and clears the threads interrupted status per
https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/locks/Condition.html#awaitNanos(long) ?