[Bug]: Breaking change in 2.0.2 -> Docker 24.x

[jakub-pietrzak]

Module

Core

Testcontainers version

2.0.2

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host Arch

ARM

Docker version

Client: Docker Engine - Community
 Version:           24.0.9
 API version:       1.43
 Go version:        go1.20.13
 Git commit:        2936816
 Built:             Thu Feb  1 00:48:39 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.9
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.13
  Git commit:       fca702d
  Built:            Thu Feb  1 00:48:39 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

With #11216, the default docker API version has been changed to 1.44.

See also #11212

This is a breaking change for docker 24 which supports up to 1.43.

The logs are from a GitHub Actions running on BuildJet where I can not really influence docker version.

Relevant log output

Starting a Gradle Daemon (subsequent builds will be faster)
Calculating task graph as no cached configuration is available for tasks: assemble
> Task :x:processResources
> Task :x:generateGitProperties
> Task :xxx:generateJooqClasses FAILED

Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
> Task :app:generateJooqClasses FAILED
	UnixSocketClientProviderStrategy: failed with exception BadRequestException (Status 400: {"message":"client version 1.44 is too new. Maximum supported API version is 1.43"}
)
	DockerDesktopClientProviderStrategy: failed with exception NullPointerException (Cannot invoke "java.nio.file.Path.toString()" because the return value of "dev.monosoul.jooq.shadow.org.testcontainers.dockerclient.DockerDesktopClientProviderStrategy.getSocketPath()" is null)As no valid configuration was found, execution cannot continue.

Additional Information

The workaround using api.version suggested in #11212 works, but I would not expect such a breaking change in a patch version.

Is it possible to implement some kind of version negotiation instead?

No response

[jarlah]

Bumping this. But not only 2.x

[taole33]

I checked git blame and found the root cause. While the logic to set a default API version has existed for 5 years, the default value was recently updated to VERSION_1_44 (about 3 weeks ago).

The java file is here

The Problem: Docker Engine v24 supports up to API v1.43. By forcing v1.44, Testcontainers breaks compatibility for all v24 users.

Proposed Fix: We should either:

Revert the default value to a safe baseline (e.g., 1.35 or 1.41).

Or simply remove this default setting block to let docker-java negotiate the version automatically (which seems to be the preferred behavior for compatibility).

If there are no objections, I can submit a PR to remove this block (or downgrade the version).

[xak2000]

The problem is that while Docker Engine v24 supports up to API v1.43, Docker Engine v25 supports starting from API v1.44. So, there is no version that is compatible with both.

So, no default setting will satisfy everyone. Either people with the older version or with the newer version of Docker will need to use a "workaround". Therefore, "revert the default" will not do any good because there is no safe baseline.

let docker-java negotiate

Are you sure docker-java will do this? Will negotiate? If yes, then this could be a good solution. If no, then testcontainers-java should do the negotiation, in my opinion.

[taole33]

Are you sure docker-java will do this?

Yes. First, I confirmed locally that docker-java performs auto-negotiation correctly.

Furthermore, I actually removed the 3 lines in DockerClientProviderStrategy.java to test the fix:

Java

    if (configBuilder.build().getApiVersion() == RemoteApiVersion.UNKNOWN_VERSION) {
        configBuilder.withApiVersion(RemoteApiVersion.VERSION_1_44);
    }

With this change applied, I verified that the tests passed successfully on both Docker v24 and Docker v29.

It seems docker-java correctly handles the versions defined here: RemoteApiVersion.java
(There are a lot of version , but I've only confirmed that the tests pass on Docker v24 and v29.)