[Bug]: MS Hosted Agents dont work after 16th of November

[jkerken]

Testcontainers version

4.0.0

Using the latest Testcontainers version?

No, but 4.9.0 also fails the same

Host OS

Linux

Host arch

x86

.NET version

8.0.409

Docker version

Client: Docker Engine - Community
 Version:           28.0.4
 API version:       1.48
 Go version:        go1.23.7
 Git commit:        b8034c0
 Built:             Tue Mar 25 15:07:16 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.4
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.7
  Git commit:       6430e49
  Built:            Tue Mar 25 15:07:16 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.1.5
  GitCommit:        fcd43222d6b07379a4be9786bda52438f0dd16a1
 runc:
  Version:          1.3.3
  GitCommit:        v1.3.3-0-gd842d771
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client: Docker Engine - Community
 Version:    28.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.38.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose


Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 28.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: fcd43222d6b07379a4be9786bda52438f0dd16a1
 runc version: v1.3.3-0-gd842d771
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.11.0-1018-azure
 Operating System: Ubuntu 24.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.758GiB
 Name: runnervmqppfj
 ID: 25309008-d3b6-4795-9b53-5704e2dd5a06
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: githubactions
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

We had a running unit test setup running fine on MS Hosted agents until the 16th after that it now gives the following errors:

/_/src/Testcontainers/Clients/DockerContainerOperations.cs(213): error test failed: ---- Docker.DotNet.DockerImageNotFoundException : Docker API responded with status code=NotFound, response={"message":"No such image: localstack/localstack:4.1.1"} /_/src/Testcontainers/Clients/DockerContainerOperations.cs(213): error test failed: ---- Docker.DotNet.DockerImageNotFoundException : Docker API responded with status code=NotFound, response={"message":"No such image: mcr.microsoft.com/mssql/server:2022-latest"}

We also have succesfully working:

• Kafka Test Container

Relevant log output

Additional information

No response

[coderabbitai]

Related Issues

• [Bug]: "Initialization has been cancelled" when running in Azure Pipelines #1435: [Bug]: "Initialization has been cancelled" when running in Azure Pipelines
• [Bug]: ImageFromDockerfileBuilder breaking change in 4.8.0 #1557: [Bug]: ImageFromDockerfileBuilder breaking change in 4.8.0
• [Bug]: image created has no associated tags #1524: [Bug]: image created has no associated tags
• [Bug]: LocalStack module fails to start container when image version contains checksum complaining about legacy mode testcontainers-go#3128: [Bug]: LocalStack module fails to start container when image version contains checksum complaining about legacy mode

Suggested Issue Assignees

• thomas-gandy
• alexaka1
• lsfera
• trallnag

Actions

• 🗺️ Generate an implementation plan for this issue

---

_{ℹ️ Note: Issue enrichment is currently in early access.}

Disable automatic issue enrichment

_{To disable automatic issue enrichment, add the following to your .coderabbit.yaml:}

issue_enrichment:
  auto_enrich:
    enabled: false

[HofmeisterAn]

I'm running a bunch of projects on Azure- and GitHub-hosted agents/runners and haven't seen anything like this. It looks like the registry doesn't contain the images.

Are you using a private registry?

Have you tried running a manual docker pull?

Could you please enable and share the TC log messages?

dotnet test -l "console;verbosity=detailed"

[neoxack]

4.8.1 working fine

4.9.0 - DotNet.Testcontainers.Builders.DockerUnavailableException: Docker is either not running or misconfigured. Please ensure that Docker is running and that the endpoint is properly configured.

Details: 
    Failed to connect to Docker endpoint at 'tcp://localhost:2376/'.
    Failed to connect to Docker endpoint at 'tcp://localhost:2376/'.
    Failed to connect to Docker endpoint at 'tcp://localhost:2376/'.
    Failed to connect to Docker endpoint at 'unix:///var/run/docker.sock'.

[HofmeisterAn]

4.9.0 - DotNet.Testcontainers.Builders.DockerUnavailableException: Docker is either not running or misconfigured. Please ensure that Docker is running and that the endpoint is properly configured.

@neoxack Which Docker Engine APIs do these container runtimes use? See: #1576 and the release notes that include information about the required changes due to the breaking changes in Docker Engine v29. Are you running Docker or a different container runtime? Your issue isn't related to this one.

[jkerken]

@HofmeisterAn There is no private registry.

PrePulling the images works as a workaround

Turns out if I move the test to the start of the pipeline they work...
It seems like they fail because the disk of the agents is getting full during the job.

Output Before the disk is full:

xUnit.net v3 In-Process Runner v3.1.0+03a071627b (64-bit .NET 8.0.16)
  Discovering: DevSpells.Databases.MsSql.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  DevSpells.Databases.MsSql.Tests (8 test cases to be run)
  Starting:    DevSpells.Databases.MsSql.Tests (parallel test collections = on [2 threads], stop on fail = off, explicit = off, seed = 689313163)
[testcontainers.org 00:00:00.33] Connected to Docker:
  Host: unix:///var/run/docker.sock
  Server Version: 28.0.4
  Kernel Version: 6.11.0-1018-azure
  API Version: 1.48
  Operating System: Ubuntu 24.04.3 LTS
  Total Memory: 7.76 GB
[testcontainers.org 00:00:01.45] Docker container 88a445feaf5b created
[testcontainers.org 00:00:01.51] Start Docker container 88a445feaf5b
[testcontainers.org 00:00:01.76] Wait for Docker container 88a445feaf5b to complete readiness checks
[testcontainers.org 00:00:01.77] Docker container 88a445feaf5b ready
[testcontainers.org 00:00:01.79] Searching Docker registry credential in Auths
[testcontainers.org 00:00:01.79] Searching Docker registry credential in CredHelpers
[testcontainers.org 00:00:01.79] Searching Docker registry credential in CredsStore
[testcontainers.org 00:00:01.80] Searching Docker registry credential in Auths
[testcontainers.org 00:00:01.80] Docker registry credential mcr.microsoft.com not found
[testcontainers.org 00:00:22.10] Docker image mcr.microsoft.com/mssql/server:2022-latest created
[testcontainers.org 00:00:22.12] Docker container 9b0786e5d2d0 created
[testcontainers.org 00:00:22.13] Start Docker container 9b0786e5d2d0
[testcontainers.org 00:00:22.26] Wait for Docker container 9b0786e5d2d0 to complete readiness checks
[testcontainers.org 00:00:22.27] Execute "/bin/sh -c find /opt/mssql-tools*/bin/sqlcmd -type f -print -quit" at Docker container 9b0786e5d2d0
[testcontainers.org 00:00:22.35] Execute "/opt/mssql-tools18/bin/sqlcmd -C -Q SELECT 1;" at Docker container 9b0786e5d2d0
[testcontainers.org 00:00:27.94] Docker container 9b0786e5d2d0 ready
[testcontainers.org 00:00:30.13] Delete Docker container 9b0786e5d2d0
  Finished:    DevSpells.Databases.MsSql.Tests (ID = '5b33b87cb5165fd8764b8530b1fade3de76fa50ee7973b4042f4c789bdae0cd3')
=== TEST EXECUTION SUMMARY ===
   DevSpells.Databases.MsSql.Tests  Total: 8, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0, Time: 30.769s

I cant really get a log when this occurs, because I dont have a mean to artificially fill the disk. The agents dies when the disk is completly full then