[Bug]: Can't connect to Ryuk when using Colima

[skagedal]

Module

Core

Testcontainers version

1.17.6

Using the latest Testcontainers version?

Yes

Host OS

macOS Ventura 13.1 (22C65)

Host Arch

ARM

Docker version

$ docker version
Client: Docker Engine - Community
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.19.5
 Git commit:        715524332f
 Built:             Wed Jan 18 20:42:16 2023
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974ae9e6c219862907efdd76ec2e77ec930b
  Built:            Wed Oct 19 02:58:31 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        5fd4c4d144137e991c4acebb2146ab1483a97925
 docker-init:
  Version:          0.19.0
  GitCommit:

What happened?

See the simple reproducer case in https://github.com/skagedal/testcontainers-hello. This has a test case that simply spins up a MySQL container using testcontainers. I run it with an environment as described here, i.e. with the TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE and DOCKER_HOST environment variables set (but not the TESTCONTAINERS_HOST_OVERRIDE):

$ echo $DOCKER_HOST
unix:///Users/simon/.colima/docker.sock

$ echo $TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE
/var/run/docker.sock

$ echo $TESTCONTAINERS_HOST_OVERRIDE

When running the project, I get many log warnings about being unable to connect to Ryuk:

$ ./gradlew test --rerun-tasks

> Task :app:test

AppTest > testcontainers_mysql() STANDARD_ERROR
    [Test worker] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
    [Test worker] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
    [Test worker] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///Users/simon/.colima/docker.sock
    [Test worker] INFO org.testcontainers.DockerClientFactory - Docker host IP address is localhost
    [Test worker] INFO org.testcontainers.DockerClientFactory - Connected to docker:
      Server Version: 20.10.20
      API Version: 1.41
      Operating System: Alpine Linux v3.16
      Total Memory: 11949 MB
    [Test worker] INFO org.testcontainers.DockerClientFactory - Checking the system...
    [Test worker] INFO org.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0
    [Test worker] INFO 🐳 [mysql:8.0.31] - Creating container for image: mysql:8.0.31
    [Test worker] INFO 🐳 [testcontainers/ryuk:0.3.4] - Creating container for image: testcontainers/ryuk:0.3.4
    [Test worker] INFO 🐳 [testcontainers/ryuk:0.3.4] - Container testcontainers/ryuk:0.3.4 is starting: 0e8b1edeb20fa240b4ec4a8897c994124ed0d9abfd8c48ac50897af1e685318c
    [Test worker] INFO 🐳 [testcontainers/ryuk:0.3.4] - Container testcontainers/ryuk:0.3.4 started in PT0.490917S
    [testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at localhost:49155
    java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
        at java.base/java.lang.Thread.run(Thread.java:833)
    [testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at localhost:49155
    java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
        at java.base/java.lang.Thread.run(Thread.java:833)
    [testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at localhost:49155
    java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
        at java.base/java.lang.Thread.run(Thread.java:833)
    [testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at localhost:49155
    java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
        at java.base/java.lang.Thread.run(Thread.java:833)
    [testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at localhost:49155
    java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
        at java.base/java.lang.Thread.run(Thread.java:833)
    [testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at localhost:49155
    java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
        at java.base/java.lang.Thread.run(Thread.java:833)
    [Test worker] INFO 🐳 [mysql:8.0.31] - Container mysql:8.0.31 is starting: 30f0ab5cad5fcf6ef4413c8949f2a069414557d40b4dc18a1d15d4af76c7153b
    [Test worker] INFO 🐳 [mysql:8.0.31] - Waiting for database connection to become available at jdbc:mysql://localhost:49156/test using query 'SELECT 1'
    [Test worker] INFO 🐳 [mysql:8.0.31] - Container is started (JDBC URL: jdbc:mysql://localhost:49156/test)
    [Test worker] INFO 🐳 [mysql:8.0.31] - Container mysql:8.0.31 started in PT11.480455S

AppTest > testcontainers_mysql() STANDARD_OUT
    Version is: 8.0.31

BUILD SUCCESSFUL in 13s
3 actionable tasks: 3 executed

I'm unsure if this means that it eventually manages to connect to Ryuk or not, but in any case the warnings are unwanted.

If I also add the environment variable TESTCONTAINERS_HOST_OVERRIDE to the IP address that colima listens to, as suggested by this comment, then things run smoothly:

$ echo $TESTCONTAINERS_HOST_OVERRIDE
192.168.106.2
$ ./gradlew test --rerun-tasks

> Task :app:test

AppTest > testcontainers_mysql() STANDARD_ERROR
    [Test worker] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
    [Test worker] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
    [Test worker] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///Users/simon/.colima/docker.sock
    [Test worker] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 192.168.106.2
    [Test worker] INFO org.testcontainers.DockerClientFactory - Connected to docker:
      Server Version: 20.10.20
      API Version: 1.41
      Operating System: Alpine Linux v3.16
      Total Memory: 11949 MB
    [Test worker] INFO org.testcontainers.DockerClientFactory - Checking the system...
    [Test worker] INFO org.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0
    [Test worker] INFO 🐳 [mysql:8.0.31] - Creating container for image: mysql:8.0.31
    [Test worker] INFO 🐳 [testcontainers/ryuk:0.3.4] - Creating container for image: testcontainers/ryuk:0.3.4
    [Test worker] INFO 🐳 [testcontainers/ryuk:0.3.4] - Container testcontainers/ryuk:0.3.4 is starting: a49651f0b12d297e648f5fbfc44337d2352b3cd73b5db451073735264e710b51
    [Test worker] INFO 🐳 [testcontainers/ryuk:0.3.4] - Container testcontainers/ryuk:0.3.4 started in PT0.516247S
    [Test worker] INFO 🐳 [mysql:8.0.31] - Container mysql:8.0.31 is starting: 8e162a50412fc9e13edeff34c960f736f0bc7a87c792a371c1910bb89ef458e0
    [Test worker] INFO 🐳 [mysql:8.0.31] - Waiting for database connection to become available at jdbc:mysql://192.168.106.2:49158/test using query 'SELECT 1'
    [Test worker] INFO 🐳 [mysql:8.0.31] - Container is started (JDBC URL: jdbc:mysql://192.168.106.2:49158/test)
    [Test worker] INFO 🐳 [mysql:8.0.31] - Container mysql:8.0.31 started in PT8.849308S

AppTest > testcontainers_mysql() STANDARD_OUT
    Version is: 8.0.31

BUILD SUCCESSFUL in 10s
3 actionable tasks: 3 executed

(This setup also seems to helps with better connectivity to other testcontainers in our properietary code, however I was not able to create a reproducer for this.)

Relevant log output

No response

Additional Information

Colima version and status:

$ colima version
colima version 0.5.2
git commit: 6b5b6fe0540e708f0c9d6e8919fab292c671fc72

runtime: docker
arch: aarch64
client: v20.10.23
server: v20.10.20

testcontainers-hello on  main via ☕ v17.0.6 on ☁️ (eu-west-1)
$ colima status
INFO[0000] colima is running using QEMU
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] address: 192.168.106.2
INFO[0000] socket: unix:///Users/simon/.colima/default/docker.sock

[HofmeisterAn]

Is this maybe just a verbose output (showing the first attempts to connect)? ./gradlew test --rerun-tasks creates two containers. Ryuk's container logs show a client connection:

atomicjar@Mac-AH-Y0T testcontainers-hello % docker ps
CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS                  PORTS                                                    NAMES
022f16bc803a   mysql:8.0.31                "docker-entrypoint.s…"   1 second ago    Up Less than a second   33060/tcp, 0.0.0.0:49162->3306/tcp, :::49162->3306/tcp   happy_keller
fdeec0a4d722   testcontainers/ryuk:0.3.4   "/app"                   2 seconds ago   Up 1 second             0.0.0.0:49161->8080/tcp, :::49161->8080/tcp              testcontainers-ryuk-32189de6-68e0-4178-9f60-720c9bd0c4df

atomicjar@Mac-AH-Y0T testcontainers-hello % docker logs fdeec0a4d722
2023/02/01 21:37:43 Pinging Docker...
2023/02/01 21:37:43 Docker daemon is available!
2023/02/01 21:37:43 Starting on port 8080...
2023/02/01 21:37:43 Started!
2023/02/01 21:37:44 New client connected: 172.17.0.1:48806
2023/02/01 21:37:44 Received the first connection
2023/02/01 21:37:44 Adding {"label":{"org.testcontainers.lang=java":true,"org.testcontainers.sessionId=32189de6-68e0-4178-9f60-720c9bd0c4df":true,"org.testcontainers.version=1.17.6":true,"org.testcontainers=true":true}}
2023/02/01 21:37:52 EOF
2023/02/01 21:37:52 Client disconnected: 172.17.0.1:48806

While I receive the same warnings mentioned above.

[ananni13]

I encountered the same problem today, using Testcontainers 1.17.6, macOS 13.1 and colima 0.5.2.

Setting DOCKER_HOST to "unix://${HOME}/.colima/default/docker.sock" (instead of "unix://${HOME}/.colima/docker.sock") solved the issue for me.

[skagedal]

@HofmeisterAn:

Is this maybe just a verbose output?

Yeah, I guess you are right! Then it is of course a much less severe issue. But still a bit of an issue? These logs make you feel like there is an issue, and if you have a different issue (like your tests being unable to connect to their testcontainers, for some reason), then you might waste time chasing a red herring.

Also, how would we explain that I don't seem to get those logs when running towards Docker Desktop or when setting up the TESTCONTAINERS_HOST_OVERRIDE to the IP address? Is it just that Ryuk is quicker to start up under these conditions, or that the mode that is triggered to connect in those cases do not cause the verbose logs on reconnects?

[skagedal]

@ananni13: This does not seem to make a difference for me... Running the tests many times now under both conditions, I can see now that the number of "Connection refused" logs will vary, and sometimes they will indeed be zero, which brings further credence to what @HofmeisterAn is saying is true. So maybe you were just lucky after having changed the variable and unlucky before?

[HofmeisterAn]

These logs make you feel like there is an issue, and if you have a different issue (like your tests being unable to connect to their testcontainers, for some reason), then you might waste time chasing a red herring.

I agree, the message is unfortunate and may lead you to the wrong track. It looks like there is an issue even there isn't.

This does not seem to make a difference for me...

For me neither.

Is it just that Ryuk is quicker to start up under these conditions, or that the mode that is triggered to connect in those cases do not cause the verbose logs on reconnects?

I do not think Ryuk is starting quicker, I assume Colima forwards the traffic (immediately) to the right place. I haven not found much more information about --network-address.

There is not much magic going on in tc-java. It just tries to connect to the Ryuk (the container). Maybe it makes sense to reach out to the Colima people and get a better understanding of the --network-address impact.

[leeychee]

For now, the test container could only read TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE from system env, but not ~/.testcontainers.properties.

Considering this configuration is part of test environment, like DOCKER_HOST (which could read from ~/.testcontainers.properties), Could we support read TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE from this personal configuration file?

[patope]

I got ryuk running with setting

TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock

Mapping docker socket from host's home folder do not work with colima, because home folder is mounted from host to colima's VM using sshfs filesystem. IMO solution here would be to define own docker client strategy for colima. Similar to DockerDesktopClientProviderStrategy, but with colima defaults.

Edit:
Another solution is

ln -s ~/.colima/default/docker.sock ~/.docker/run/docker.sock

This enables DockerDesktopClientProviderStrategy to work with colima

[eddumelendez]

Please, consider reporting this to Colima project. There is something we don't know about it and can not fix it on our own. This is similar to Rancher Desktop, some users report it to Rancher project and just last week we got a contribution from them to improve our docs.

[afganhalbana]

just want to share my findings if you are running on Mac M1 Sonoma, make sure the colima can retrieved the ip address.

colima ls

PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK     RUNTIME    ADDRESS
default    Running    aarch64    2       2GiB      60GiB    docker     192.168.64.2

if the address was blank, meaning that probably configured colima wrong
check this

then make sure to define this environment variable, mine is defined on ~/.zshrc

export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
export TESTCONTAINERS_HOST_OVERRIDE=$(colima ls -j | jq -r '.address')
export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock"

[eddumelendez]

Thanks for sharing, @afganhalbana! Do you mind updating our docs?

[afganhalbana]

sure will do @eddumelendez

[alheiros-at-tide]

just want to share my findings if you are running on Mac M1 Sonoma, make sure the colima can retrieved the ip address.

colima ls

PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK     RUNTIME    ADDRESS
default    Running    aarch64    2       2GiB      60GiB    docker     192.168.64.2

if the address was blank, meaning that probably configured colima wrong check this

then make sure to define this environment variable, mine is defined on ~/.zshrc

export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
export TESTCONTAINERS_HOST_OVERRIDE=$(colima ls -j | jq -r '.address')
export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock"

thanks so much, this worked a treat!