[Bug]: Parallel container startup hungs when withCreateContainerCmdModifier is used

[xak2000]

Module

Core

Testcontainers version

1.19.8

Using the latest Testcontainers version?

Yes

Host OS

Windows 11

Host Arch

x86

Docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:48:43 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.30.0 (149282)
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:48:28 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Startables.deepStart(container).join(); hungs indefinitely when container has withCreateContainerCmdModifier defined.

Example:

public class AppTest {

    @SuppressWarnings("resource")
    static GenericContainer<?> gcs = new GenericContainer<>(DockerImageName.parse("fsouza/fake-gcs-server:1"))
            .withExposedPorts(4443)
            .withCreateContainerCmdModifier(cmd -> cmd.withEntrypoint("/bin/fake-gcs-server", "-scheme", "http"));

    static {
        startGlobalContainersParallel();
    }

    static void startGlobalContainersSequentially() {
        gcs.start();
    }

    static void startGlobalContainersParallel() {
        Startables.deepStart(gcs).join();
    }

    @Test
    public void shouldAnswerWithTrue() {
        assertTrue(true);
    }

}

If Startables.deepStart is not used (i.e. instead of calling startGlobalContainersParallel() we call startGlobalContainersSequentially()), then all work fine.

Removing withCreateContainerCmdModifier also helps.

I tried to add withCreateContainerCmdModifier to other containers (e.g. MySQLContainer) and it hungs too, so this bug doesn't depend on the container.

Relevant log output

19:12:53.402 [testcontainers-lifecycle-0] INFO tc.testcontainers/ryuk:0.7.0 -- Container testcontainers/ryuk:0.7.0 started in PT0.7214507S
19:12:53.406 [testcontainers-ryuk] DEBUG org.testcontainers.utility.ResourceReaper -- Sending 'label=org.testcontainers%3Dtrue&label=org.testcontainers.lang%3Djava&label=org.testcontainers.version%3D1.19.8&label=org.testcontainers.sessionId%3D2300acd2-8cc9-41ef-b3e1-37ea382ee1a8' to Ryuk
19:12:53.406 [testcontainers-ryuk] DEBUG org.testcontainers.utility.RyukResourceReaper -- Received 'ACK' from Ryuk
19:12:53.406 [testcontainers-lifecycle-0] INFO org.testcontainers.utility.RyukResourceReaper -- Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.DockerClientFactory -- Checks are enabled
19:12:53.407 [testcontainers-lifecycle-0] INFO org.testcontainers.DockerClientFactory -- Checking the system...
19:12:53.407 [testcontainers-lifecycle-0] INFO org.testcontainers.DockerClientFactory -- ✔︎ Docker server version should be at least 1.6.0
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.utility.PrefixingImageNameSubstitutor -- No prefix is configured
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.utility.ImageNameSubstitutor -- Did not find a substitute image for fsouza/fake-gcs-server:1 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.images.AbstractImagePullPolicy -- Using locally available and not pulling image: fsouza/fake-gcs-server:1
19:12:53.407 [testcontainers-lifecycle-0] DEBUG tc.fsouza/fake-gcs-server:1 -- Starting container: fsouza/fake-gcs-server:1
19:12:53.407 [testcontainers-lifecycle-0] DEBUG tc.fsouza/fake-gcs-server:1 -- Trying to start container: fsouza/fake-gcs-server:1 (attempt 1/1)
19:12:53.407 [testcontainers-lifecycle-0] DEBUG tc.fsouza/fake-gcs-server:1 -- Starting container: fsouza/fake-gcs-server:1
19:12:53.407 [testcontainers-lifecycle-0] INFO tc.fsouza/fake-gcs-server:1 -- Creating container for image: fsouza/fake-gcs-server:1
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.utility.RegistryAuthLocator -- Looking up auth config for image: fsouza/fake-gcs-server:1 at registry: https://index.docker.io/v1/
19:12:53.408 [testcontainers-lifecycle-0] DEBUG org.testcontainers.utility.RegistryAuthLocator -- No matching Auth Configs - falling back to defaultAuthConfig [null]
19:12:53.408 [testcontainers-lifecycle-0] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig -- Effective auth config [null]

Additional Information

Minimal reproduce: tc-parallel-start-bug.zip.

Docs aboutt parallel container startup: https://java.testcontainers.org/features/advanced_options/#parallel-container-startup

[eddumelendez]

Thanks for reporting the issue. BTW, there is a module for the image you are reporting. See https://testcontainers.com/modules/fake-gcs-server/

[kristofdho]

I just ran into this with the latest 1.x version. Is there any workaround to this?

[xak2000]

The only workaround I know of is to not use parallel startup.

[aoitori6]

This seems related to https://bugs.openjdk.org/browse/JDK-8213361.

I tried creating a concrete class implementing the lambda instead, and that worked with parallel startup.

[xak2000]

Thank you @aoitori6. Very valuable finding.

Based on the discussion around JDK-8213361 I think the pattern like this should not be recommended then.

static {
   Startables.deepStart(/*...*/).join();
}

Because, even if we avoid lambda, it is still possible to cause deadlock on class initialization by accessing any static field/method of the current class from a CreateContainerCmdModifier implementation (or any other callable) called from another thread.

Or maybe it is possible to move execution of all callables (e.g. CreateContainerCmdModifiers) to the current thread and execute only actual containers startup logic on the thread pool?