dockerd crashes ~18s after startup in v1.13.1 due to BuildKit healthcheck fatal failure (Docker 29.x)

[Kump3r]

Describe the bug

Summary

Starting with the Docker 29.x image bundled in v1.13.1, dockerd reliably crashes
~18 seconds after startup, causing all put steps to fail. The root cause is
BuildKit's internal gRPC session healthcheck failing fatally due to a
"only one connection allowed" socket transport restriction in the containerized
Concourse worker environment.

Symptoms

• put steps using docker-image-resource fail consistently across retries
• Hijacking the container shows /var/run/docker.sock does not exist (daemon has exited)
• start_docker appears to succeed (prints "waiting for docker to come up...") but
  the daemon exits shortly after

Root cause

/tmp/docker.log from a failed task shows:

INFO  API listen on /var/run/docker.sock              ← daemon up
WARN  healthcheck failed: "only one connection allowed"
ERROR healthcheck failed fatally: session healthcheck failed fatally
INFO  Processing signal 'terminated'                  ← daemon exits ~18s after start

Docker 29.x has BuildKit deeply integrated into the daemon (always-on). BuildKit's
internal gRPC session healthcheck attempts to open a second connection on the Unix
socket, but the containerized environment only permits one connection at a time.
The healthcheck fails fatally and brings down dockerd with it.

This did not occur in older Docker versions bundled with v1.12.4 where BuildKit was
not integrated at this level.

Environment

• docker-image-resource version: v1.13.1
• Docker version in image: 29.5.1
• Base OS: Wolfi (containerized)
• Kernel: 5.15.0-181-generic
• Cgroup: v1

Workaround

Pin to v1.12.4 in your pipeline:

resource_types:
- name: docker-image
  type: registry-image
  source:
    repository: concourse/docker-image-resource
    tag: 1.12.4

Possible fixes

In assets/common.sh, writing a daemon.json disabling the BuildKit integration
before starting dockerd, or passing the flag directly:

Option A — daemon.json

echo '{"features":{"buildkit":false}}' > /etc/docker/daemon.json

Option B — flag

dockerd --feature=buildkit=false ...

Note: setting DOCKER_BUILDKIT=0 is not sufficient in Docker 29.x as BuildKit is now integrated into the daemon itself, not just the CLI. So I think the docker_buildkit: false will no longer work.

Reproduction steps

1. Use v1.13.1 to build and push an iamge

Expected behavior

Push succeeds as in older registry image resources

Additional context

No response

[Kump3r]

docker.log attaching the full docker log.

[Kump3r]

Pipeline used to reproduce:

---
# Test pipeline to reproduce docker-image-resource BuildKit crash (Docker 29.x)
#
# fly -t <target> set-pipeline \
#   -p poc-docker-image-resource-test \
#   -c poc-docker-image-resource-test.yml \
#   -v dockerhub_username=<user> \
#   -v dockerhub_password=<token>

resources:
- name: test-image-v1-13-1
  type: docker-image
  source:
    repository: ((dockerhub_username))/poc-docker-image-resource-test
    tag: poc-test-1.13.1
    username: ((dockerhub_username))
    password: ((dockerhub_password))

jobs:
- name: build-and-push-v1-13-1-should-fail
  plan:
  - task: create-dockerfile
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
      outputs:
      - name: build-context
      run:
        path: sh
        args:
        - -c
        - |
          cat > build-context/Dockerfile <<'EOF'
          FROM alpine
          LABEL test=poc-docker-image-resource-1.13.1
          EOF
  - put: test-image-v1-13-1
    params:
      build: build-context

[Kump3r]

Disabling the buildkit is not a feasible solution, as most modern development requires it as far as I was able to understand, but to fix the BuildKit session socket limit, as the root cause is "only one connection allowed" on the Unix socket inside the Concourse containerd container might be a little more complex. Currently we are blocked using the oci-build-resource as it requires privilege escalation.