Retry flaky Vert.x WebSocket server tests (#5410)

## Problem

The Vert.x WebSocket server tests (`CatsVertxServerTest`,
`ZioVertxServerTest`, `VertxServerTest`) intermittently fail CI with a
3-minute `TimeoutException` — e.g. recently on scala-steward dependency
PRs (#5404), unrelated to the bumped dependency.

## Root cause (upstream Vert.x, not tapir)

A race in `HttpServerRequest.toWebSocket()`: Vert.x writes+flushes the
HTTP 101 in one event-loop task and completes the returned `Future`
(where we register our frame handler) in a later task. A client frame
arriving in that ~1-3 ms window is read and then **silently dropped** —
`WebSocketImplBase.receiveFrame` discards TEXT/BINARY frames when no
handler is registered, and the server WebSocket's inbound queue starts
unpaused. The echo never returns, so the test times out.

This is **not fixable via Vert.x's public per-request API**:
`toWebSocket()` hands back an already-flowing socket, and the
intermediate handshake object isn't exposed (its `pause()` throws
`UnsupportedOperationException`). The only race-free API is the
server-level `HttpServer.webSocketHandler`, which bypasses tapir's
`Router`. The bug is present in every released Vert.x and on current
`master`; it will be reported upstream (the fix there is to start the
server WebSocket paused / buffer inbound frames until a handler is
registered).

## This change

Interim, test-only: retry failed Vert.x server tests using the same
`withFixture` retry already used by `JdkHttpServerTest` and
`ZioHttpServerTest`. `retries = 3` (lower than those suites' `5`,
because a WS flake hangs ~3 min per attempt rather than failing fast).
Each attempt re-runs with a fresh timeout budget. No production code
changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: adamw

server/jdkhttp-server/src/test/scala/sttp/tapir/server/jdkhttp/JdkHttpServerTest.scala

@@ -1,32 +1,17 @@
 package sttp.tapir.server.jdkhttp
 
 import cats.effect.{IO, Resource}
-import org.scalatest.{EitherValues, Exceptional, FutureOutcome}
+import org.scalatest.EitherValues
 import sttp.tapir.server.jdkhttp.internal.idMonad
 import sttp.tapir.server.tests._
 import sttp.tapir.tests.{Test, TestSuite}
 
-import scala.concurrent.Future
-
 class JdkHttpServerTest extends TestSuite with EitherValues {
   // these tests often fail on CI with:
   // "Cause: java.io.IOException: HTTP/1.1 header parser received no bytes"
   // "Cause: java.io.EOFException: EOF reached while reading"
   // for an unknown reason; adding retries to avoid flaky tests
-  val retries = 5
-
-  override def withFixture(test: NoArgAsyncTest): FutureOutcome = withFixture(test, retries)
-
-  def withFixture(test: NoArgAsyncTest, count: Int): FutureOutcome = {
-    val outcome = super.withFixture(test)
-    new FutureOutcome(outcome.toFuture.flatMap {
-      case Exceptional(e) =>
-        println(s"Test ${test.name} failed, retrying.")
-        e.printStackTrace()
-        (if (count == 1) super.withFixture(test) else withFixture(test, count - 1)).toFuture
-      case other => Future.successful(other)
-    })
-  }
+  override def retries = 5
 
   override def tests: Resource[IO, List[Test]] =
     backendResource.flatMap { backend =>

server/vertx-server/cats/src/test/scala/sttp/tapir/server/vertx/cats/CatsVertxServerTest.scala

@@ -13,6 +13,12 @@ import scala.concurrent.duration._
 
 class CatsVertxServerTest extends TestSuite {
 
+  // The Vert.x WebSocket tests hit a rare upstream flake: a client frame sent in the brief window after the 101
+  // handshake but before Vert.x completes toWebSocket() (so before we can register a frame handler) is silently
+  // dropped, so the echo never returns and the test times out. Not fixable via Vert.x's public per-request API
+  // (toWebSocket() returns an already-flowing socket); to be reported upstream. Retry failed tests to avoid flaky CI.
+  override def retries = 3
+
   def vertxResource: Resource[IO, Vertx] =
     Resource.make(IO.delay(Vertx.vertx()))(vertx =>
       // vertx.close sometimes never completes; bound the wait so suite cleanup can't hang the (non-forked) test JVM

server/vertx-server/src/test/scala/sttp/tapir/server/vertx/VertxServerTest.scala

@@ -14,6 +14,13 @@ import scala.concurrent.Promise
 import io.vertx.core.buffer.Buffer
 
 class VertxServerTest extends TestSuite {
+
+  // The Vert.x WebSocket tests hit a rare upstream flake: a client frame sent in the brief window after the 101
+  // handshake but before Vert.x completes toWebSocket() (so before we can register a frame handler) is silently
+  // dropped, so the echo never returns and the test times out. Not fixable via Vert.x's public per-request API
+  // (toWebSocket() returns an already-flowing socket); to be reported upstream. Retry failed tests to avoid flaky CI.
+  override def retries = 3
+
   def vertxResource: Resource[IO, Vertx] =
     Resource.make(IO.delay(Vertx.vertx()))(vertx => IO.delay(vertx.close()).void)
 

server/vertx-server/zio/src/test/scala/sttp/tapir/server/vertx/zio/ZioVertxServerTest.scala

@@ -16,6 +16,13 @@ import sttp.tapir.ztapir.RIOMonadError
 import zio.stream.ZSink
 
 class ZioVertxServerTest extends TestSuite with OptionValues {
+
+  // The Vert.x WebSocket tests hit a rare upstream flake: a client frame sent in the brief window after the 101
+  // handshake but before Vert.x completes toWebSocket() (so before we can register a frame handler) is silently
+  // dropped, so the echo never returns and the test times out. Not fixable via Vert.x's public per-request API
+  // (toWebSocket() returns an already-flowing socket); to be reported upstream. Retry failed tests to avoid flaky CI.
+  override def retries = 3
+
   def vertxResource: Resource[IO, Vertx] =
     Resource.make(IO.delay(Vertx.vertx()))(vertx => IO.delay(vertx.close()).void)
 

server/zio-http-server/src/test/scala/sttp/tapir/server/ziohttp/ZioHttpServerTest.scala

@@ -6,8 +6,6 @@ import cats.implicits.toTraverseOps
 import io.netty.channel.ChannelFactory
 import io.netty.channel.ServerChannel
 import org.scalatest.Assertion
-import org.scalatest.Exceptional
-import org.scalatest.FutureOutcome
 import org.scalatest.matchers.should.Matchers._
 import sttp.capabilities.zio.ZioStreams
 import sttp.client4._
@@ -46,7 +44,6 @@ import zio.stream.ZStream
 
 import java.nio.charset.StandardCharsets
 import java.time
-import scala.concurrent.Future
 import scala.concurrent.duration.DurationInt
 import zio.stream.ZSink
 import zio.http.netty.NettyConfig
@@ -58,20 +55,7 @@ class ZioHttpServerTest extends TestSuite {
 
   // zio-http tests often fail with "Cause: java.io.IOException: parsing HTTP/1.1 status line, receiving [DEFAULT], parser state [STATUS_LINE]"
   // until this is fixed, adding retries to avoid flaky tests
-  val retries = 5
-
-  override def withFixture(test: NoArgAsyncTest): FutureOutcome = withFixture(test, retries)
-
-  def withFixture(test: NoArgAsyncTest, count: Int): FutureOutcome = {
-    val outcome = super.withFixture(test)
-    new FutureOutcome(outcome.toFuture.flatMap {
-      case Exceptional(e) =>
-        println(s"Test ${test.name} failed, retrying.")
-        e.printStackTrace()
-        (if (count == 1) super.withFixture(test) else withFixture(test, count - 1)).toFuture
-      case other => Future.successful(other)
-    })
-  }
+  override def retries = 5
 
   override def tests: Resource[IO, List[Test]] = backendResource.flatMap { backend =>
     implicit val r: Runtime[Any] = Runtime.default

tests/src/main/scalajvm/sttp/tapir/tests/TestSuite.scala

@@ -4,15 +4,33 @@ import cats.effect.std.Dispatcher
 import cats.effect.unsafe.implicits.global
 import cats.effect.{IO, Resource}
 import org.scalactic.source.Position
-import org.scalatest.BeforeAndAfterAll
 import org.scalatest.funsuite.AsyncFunSuite
+import org.scalatest.{BeforeAndAfterAll, Exceptional, FutureOutcome}
 
+import scala.concurrent.Future
 import scala.concurrent.duration._
 
 trait TestSuite extends AsyncFunSuite with BeforeAndAfterAll {
   def tests: Resource[IO, List[Test]]
   def testNameFilter: Option[String] = None // define to run a single test (temporarily for debugging)
 
+  /** The number of times a failing test is retried before it is reported as failed. Override with a positive value in
+    * suites whose backend is subject to a known, unavoidable flake (e.g. an upstream bug) to keep CI green; keep the
+    * override well-commented with the reason. `0` (the default) means each test runs exactly once, as usual.
+    */
+  def retries: Int = 0
+
+  override def withFixture(test: NoArgAsyncTest): FutureOutcome = withRetries(test, retries)
+
+  private def withRetries(test: NoArgAsyncTest, remaining: Int): FutureOutcome =
+    new FutureOutcome(super.withFixture(test).toFuture.flatMap {
+      case Exceptional(e) if remaining > 0 =>
+        println(s"Test '${test.name}' failed, retrying ($remaining ${if (remaining == 1) "retry" else "retries"} left).")
+        e.printStackTrace()
+        withRetries(test, remaining - 1).toFuture
+      case other => Future.successful(other)
+    })
+
   protected val (dispatcher, shutdownDispatcher) = Dispatcher.parallel[IO].allocated.unsafeRunSync()
 
   // we need to register the tests when the class is constructed, as otherwise scalatest skips it