[Bug]: failed to close response

[jasperjanuar]

Module

Core

Testcontainers version

1.17.6

Using the latest Testcontainers version?

Yes

Host OS

Mac OS 13.1

Host Arch

M1 Pro

Docker version

Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.19.4
 Git commit:        3a2c30b63a
 Built:             Thu Dec 15 15:37:38 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.15.0 (93002)
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:41 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Migrating to Spring Boot 3.0.1 and application is up and running, but tests never start due hanging connection DEBUG com.github.dockerjava.zerodep.ApacheDockerHttpClientImpl$ApacheResponse - Failed to close the response java.nio.channels.ClosedChannelException: null in Testcontainer initialization

Relevant log output

15:53:00.192 [docker-java-stream--1362902404] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.wire - http-outgoing-1 << "[\r][\n]"
15:53:00.201 [Test worker] DEBUG org.testcontainers.containers.output.WaitingConsumer - STDERR: 2023-01-19 14:53:00.192 UTC [1] LOG:  database system is ready to accept connections
15:53:00.201 [Test worker] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-0000000A: cancel
15:53:00.201 [docker-java-stream--1362902404] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.wire - http-outgoing-1 << "[read] I/O error: null"
15:53:00.201 [Test worker] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.DefaultManagedHttpClientConnection - http-outgoing-1: close connection IMMEDIATE
15:53:00.201 [docker-java-stream--1362902404] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.wire - http-outgoing-1 << "[read] I/O error: null"
15:53:00.202 [Test worker] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-0000000A: endpoint closed
15:53:00.202 [Test worker] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-0000000A: discarding endpoint
15:53:00.202 [Test worker] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-0000000A: releasing endpoint
15:53:00.202 [Test worker] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-0000000A: connection is not kept alive
15:53:00.202 [docker-java-stream--1362902404] DEBUG com.github.dockerjava.zerodep.ApacheDockerHttpClientImpl$ApacheResponse - Failed to close the response
java.nio.channels.ClosedChannelException: null
	at java.base/sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:215)
	at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:403)
	at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
	at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:107)
	at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:101)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.LoggingInputStream.read(LoggingInputStream.java:81)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:147)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:314)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.io.Closer.close(Closer.java:48)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.IncomingHttpEntity.close(IncomingHttpEntity.java:111)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.io.entity.HttpEntityWrapper.close(HttpEntityWrapper.java:120)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.io.Closer.close(Closer.java:48)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.message.BasicClassicHttpResponse.close(BasicClassicHttpResponse.java:93)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpResponse.close(CloseableHttpResponse.java:200)
	at com.github.dockerjava.zerodep.ApacheDockerHttpClientImpl$ApacheResponse.close(ApacheDockerHttpClientImpl.java:256)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:277)
	at java.base/java.lang.Thread.run(Thread.java:833)
15:53:00.202 [Test worker] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-0000000A: connection released [route: {}->unix://localhost:2375][total available: 0; route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]

Additional Information

I've tried downgrading to 1.17.3 which I was using previously but with no success

[vcvitaly]

@jasperjanuar Could you provide an example of a test? I could try taking a look..

[marrek13]

I can reproduce the same issue even with such a simple code

@SpringBootTest
@Testcontainers
class TestcontainersTestApplicationTests {

    companion object {
        protected val postgreSqlContainer = PostgreSQLContainer("postgres:13.9").apply { start() }
    }

    @Test
    fun test() {
        assertEquals(1, 1)
    }
}

[jsauvain]

Seems pretty critical IMHO

[vcvitaly]

Ok, sorry, I finally found some time to look into this... I cannot reproduce the issue with such code, I mean the error is there, but the test finishes successfully, @marrek13 or @jsauvain could you upload a minimal reproducible example? I'd be happy to debug that.

This test finished successfully:

@SpringBootTest
@Testcontainers
public class SimpleTest {

    private static final PostgreSQLContainer<?> postgreSqlContainer = new PostgreSQLContainer<>("postgres:13.9");

    @BeforeAll
    static void beforeAll() {
        postgreSqlContainer.start();
    }

    @Test
    void test() {
        Assertions.assertEquals(1, 1);
    }

    @DynamicPropertySource
    static void postgresqlProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.datasource.url", postgreSqlContainer::getJdbcUrl);
        registry.add("spring.datasource.password", postgreSqlContainer::getPassword);
        registry.add("spring.datasource.username", postgreSqlContainer::getUsername);
    }
}

[lucassaldanha]

the error is there, but the test finishes successfully

We are experiencing the same thing. We get many of those errors in our logs, however, the containers start successfully and the test passes.

@vcvitaly did you have a chance to look at it and understand why these errors are popping up?

For reference, we use it in our project Teku (https://github.com/ConsenSys/teku).

Testcontainers version: 1.17.6
OS: Mac OS 13.2.1
Arch: M2 Max

[Desire456]

Also have this one. Any news guys?

[akulik512]

Hello
I'm getting the same error.

Version:
Testcontainers version: 1.18.0
OS: Linux 5.19.0-40-generic and 22.04.1-Ubuntu

Test case

@Testcontainers
@SpringBootTest
class QueueListenerIT {

  private static final String QUEUE_NAME = "test-queue";

  @Container
  static LocalStackContainer localStack =
      new LocalStackContainer(DockerImageName.parse("localstack/localstack:0.13.0"))
          .withServices(SQS);

  @BeforeAll
  static void beforeAll() throws IOException, InterruptedException {
    localStack.execInContainer("awslocal", "sqs", "create-queue", "--queue-name", QUEUE_NAME);
  }

  @DynamicPropertySource
  static void overrideConfiguration(DynamicPropertyRegistry registry) {
    registry.add("event-processing.order-event-queue", () -> QUEUE_NAME);
    registry.add("cloud.aws.sqs.endpoint", () -> localStack.getEndpointOverride(SQS));
    registry.add("cloud.aws.credentials.access-key", localStack::getAccessKey);
    registry.add("cloud.aws.credentials.secret-key", localStack::getSecretKey);
  }

  @Autowired
  private QueueMessagingTemplate queueMessagingTemplate;

  @Test
  void messageShouldBeUploadedToBucketOnceConsumedFromQueue() {

    queueMessagingTemplate.send(QUEUE_NAME, new GenericMessage<>("""
        {
           "id": "42",
           "message": "Please delivery ASAP",
           "product": "MacBook Pro",
           "orderedAt": "2021-11-11 12:00:00",
           "expressDelivery": true
        }
      """, Map.of("contentType", "application/json")));
  }
  
}

My pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.0.5</version>
    <relativePath/> <!-- lookup parent from repository -->
  </parent>

  <groupId>com.akulik</groupId>
  <artifactId>app</artifactId>

  <properties>
    <java.version>17</java.version>
    <awssdk-bom.version>2.20.51</awssdk-bom.version>
    <testcontainers.version>1.18.0</testcontainers.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
      <groupId>io.awspring.cloud</groupId>
      <artifactId>spring-cloud-starter-aws-messaging</artifactId>
      <version>2.4.4</version>
    </dependency>
    <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>sqs</artifactId>
    </dependency>
    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <scope>provided</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.awaitility</groupId>
      <artifactId>awaitility</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>junit-jupiter</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>localstack</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>bom</artifactId>
        <version>${awssdk-bom.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
      <dependency>
        <groupId>org.testcontainers</groupId>
        <artifactId>testcontainers-bom</artifactId>
        <version>${testcontainers.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <build>
    <plugins>
      <!-- Build plugin -->
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
    </plugins>
  </build>

</project>

SQS config

@Configuration
public class SQSClientConfig {

  @Value("${cloud.aws.region.static}")
  private String region;

  @Bean
  public QueueMessagingTemplate queueMessagingTemplate() {
    return new QueueMessagingTemplate(amazonSQSAsync());
  }

  @Bean
  @Primary
  public AmazonSQSAsync amazonSQSAsync() {
    return AmazonSQSAsyncClientBuilder.standard()
        .withRegion(region)
        .build();
  }

}

error log.txt

[akulik512]

Hey @bsideup
The number of likes and comments is growing 😞
Maybe you have any thoughts about it?

[gebinic]

I'm using testcontainers 1.18.0 on Ubuntu 20.04.6 LTS and docker 23.0.4 and my tests log the same exception. All tests are successful, too.

[ddaniel7]

any news?

[vcvitaly]

Since the error is logged at Debug level I'm not even sure this is even an issue, however it's best to get an answer from maintainers, probably from @bsideup. The error is logged since Testcontainers 1.16.1 when the following update was done:

Update to docker-java 3.2.12 (https://github.com/testcontainers/testcontainers-java/pull/4459) @bsideup

This repo reproduces the issue:
https://github.com/vcvitaly/testcontainers-java-6420

[MeloszDev]

Any updates on this? I'm experiencing this in my professional work right now.

[srikarbalmuri]

@bsideup any update on this issue?

[angelomelonas]

Also currently experiencing this issue.