Docker does not respect per-repo authn challenges if /v2/ returns 200

[ezekg]

Description

If docker login is used with a private registry that serves both unauthenticated and authenticated requests, depending on a repo's access control policy, docker pull does not respect the registry's fine-grained access control policies. This is because Docker (sometimes) uses the GET /v2/ endpoint to determine if a registry requires authentication.

1. If Docker receives a 200 from the GET /v2/ endpoint, it will never pull credentials to authenticate with that registry, even when a particular request to a repo sends a WWW-Authenticate authentication challenge.
2. If Docker receives a 401 from the GET /v2/ endpoint, it will pull credentials and provide them when it's presented with an authentication challenge. Docker will send credentials with every subsequent request to that registry, even if a particular repo does not send an authentication challenge.

Docker is making an assumption that a registry either requires authentication or it doesn't. But that's a rather large assumption. In the real world, repos will have a variety of access control policies, and Docker should appropriately respond to authentication challenges as it is presented with them.

The problem becomes apparent in a multi-tenant registry that has per-repo access control policies. There's no way of knowing what endpoint the client will request after GET /v2/ nor which tenant the subsequent request will be for, because there's no information in GET /v2/. Thus, relying on GET /v2/ for an accurate and global authentication challenge is impossible in the real-world.

This behavior is very confusing, and it was very hard to debug, because this behavior does not happen for everybody. At first I thought it was a misconfigured credential store, but eventually I was able to reproduce it on a brand new Linux server and I narrowed it down to the storage engines that Docker uses, overlay2 vs containerd.

By default, as far as I understand, Linux uses the graphdriver storage engine (overlay2), but Docker Desktop now defaults to using containerd. As we will see, there is a difference in behavior.

Reproduce

Below is a toy private registry that can be used to reproduce the behavior:

https://gist.github.com/ezekg/bc4f9afaac2ecb633113b494306fcd4a

Start the server with go run registry.go and make the following requests:

docker login localhost:3000 --username foo --password bar
docker pull localhost:3000/foo
docker pull localhost:3000/bar

(You can use ngrok or caddy to serve via TLS if you don't want to set localhost:3000 as an insecure registry.)

Note how the HEAD request to /foo fails, even though the server responded with a 401 status and an authentication challege, yet it continues to make a subsequent unauthenticated GET request:

2025/09/04 09:11:58 request: id=HaRnGLAj method="HEAD" path="/v2/foo/manifests/latest" auth=""
2025/09/04 09:11:58 response: id=HaRnGLAj status=401 bytes=0 challenges=["Basic realm=\"app\""]
2025/09/04 09:11:58 request: id=mMHDuYvm method="GET" path="/v2/foo/manifests/latest" auth=""
2025/09/04 09:11:58 response: id=mMHDuYvm status=200 bytes=0 challenges=["Basic realm=\"app\""]

When we make the request to /bar, it succeeds, since no authentication challenge was given:

2025/09/04 09:12:13 request: id=BpGuAWad method="HEAD" path="/v2/bar/manifests/latest" auth=""
2025/09/04 09:12:13 response: id=BpGuAWad status=200 bytes=0 challenges=[]
2025/09/04 09:12:13 request: id=BvMicvba method="GET" path="/v2/bar/manifests/latest" auth=""
2025/09/04 09:12:13 response: id=BvMicvba status=200 bytes=0 challenges=[]

If we update the /v2 route to respond with a 401 plus a challenge, the reverse happens — /foo succeeds, but then there are problems with /bar:

 case "/v2", "/v2/":
   res.Header().Set("Docker-Distribution-Api-Version", "registry/2.0")
+  if auth := req.Header.Get("Authorization"); !strings.HasPrefix(auth, "Basic ") {
+    res.Header().Set("WWW-Authenticate", `Basic realm="app"`)
+    res.WriteHeader(http.StatusUnauthorized)
+    return
+  }
   res.WriteHeader(http.StatusOK)
   return

docker login localhost:3000 --username foo --password bar
docker pull localhost:3000/foo
docker pull localhost:3000/bar

2025/09/04 09:43:23 serving on localhost:3000...
2025/09/04 09:43:30 request: id=Pegayvhl method="GET" path="/v2/" auth=""
2025/09/04 09:43:30 response: id=Pegayvhl status=401 bytes=0 challenges=["Basic realm=\"app\""]
2025/09/04 09:43:30 request: id=VHAYRlCC method="GET" path="/v2/" auth="Basic Zm9vOmJhcg=="
2025/09/04 09:43:30 response: id=VHAYRlCC status=200 bytes=0 challenges=[]
2025/09/04 09:43:32 request: id=mXmrgmVl method="GET" path="/v2/" auth=""
2025/09/04 09:43:32 response: id=mXmrgmVl status=401 bytes=0 challenges=["Basic realm=\"app\""]
2025/09/04 09:43:32 request: id=MPuqhjBd method="HEAD" path="/v2/foo/manifests/latest" auth="Basic Zm9vOmJhcg=="
2025/09/04 09:43:32 response: id=MPuqhjBd status=200 bytes=0 challenges=[]
2025/09/04 09:43:32 request: id=okEKrWqh method="GET" path="/v2/foo/manifests/latest" auth="Basic Zm9vOmJhcg=="
2025/09/04 09:43:32 response: id=okEKrWqh status=200 bytes=0 challenges=[]
2025/09/04 09:43:35 request: id=nvnEApia method="GET" path="/v2/" auth=""
2025/09/04 09:43:35 response: id=nvnEApia status=401 bytes=0 challenges=["Basic realm=\"app\""]
2025/09/04 09:43:35 request: id=IXxyZAqA method="HEAD" path="/v2/bar/manifests/latest" auth="Basic Zm9vOmJhcg=="
2025/09/04 09:43:35 response: id=IXxyZAqA status=200 bytes=0 challenges=[]
2025/09/04 09:43:35 request: id=CNEWFkCJ method="GET" path="/v2/bar/manifests/latest" auth="Basic Zm9vOmJhcg=="
2025/09/04 09:43:35 response: id=CNEWFkCJ status=200 bytes=0 challenges=[]
2025/09/04 09:44:07 request: id=bncAdRDB method="GET" path="/v2/" auth=""
2025/09/04 09:44:07 response: id=bncAdRDB status=401 bytes=0 challenges=["Basic realm=\"app\""]

Note however that requests to /bar are now sending authentication, even though its access control policy doesn't require it.

And if we logout, now /bar actually requires authentication, because it can't pass the GET /v2/ challenge:

docker logout localhost:3000
docker pull localhost:3000/bar

2025/09/04 09:54:25 request: id=XyzkbOYJ method="GET" path="/v2/" auth=""
2025/09/04 09:54:25 response: id=XyzkbOYJ status=401 bytes=0 challenges=["Basic realm=\"app\""]
2025/09/04 09:54:28 request: id=RmdrKBxB method="GET" path="/v2/" auth=""
2025/09/04 09:54:28 response: id=RmdrKBxB status=401 bytes=0 challenges=["Basic realm=\"app\""]

This is because, as far as I understand, the overlay2 storage engine that Docker uses by default, solely uses GET /v2/ to determine authentication requirements. It does not respect per-repo authentication challenges.

Expected behavior

Docker should respect and appropriately respond to authentication challenges as it is presented with them.

Reverting the changes above and switching to the containerd storage engine resolves the issue:

docker login localhost:3000 --username foo --password bar
docker pull localhost:3000/foo
docker pull localhost:3000/bar

2025/09/04 09:48:05 serving on localhost:3000...
2025/09/04 09:48:09 request: id=qUkKEtls method="GET" path="/v2/" auth=""
2025/09/04 09:48:09 response: id=qUkKEtls status=200 bytes=0 challenges=[]
2025/09/04 09:48:09 request: id=nlinzpGC method="GET" path="/v2/" auth=""
2025/09/04 09:48:09 response: id=nlinzpGC status=200 bytes=0 challenges=[]
2025/09/04 09:48:12 request: id=xOUrhDAN method="HEAD" path="/v2/foo/manifests/latest" auth=""
2025/09/04 09:48:12 response: id=xOUrhDAN status=401 bytes=0 challenges=["Basic realm=\"app\""]
2025/09/04 09:48:12 request: id=hnZJyIsB method="HEAD" path="/v2/foo/manifests/latest" auth="Basic Zm9vOmJhcg=="
2025/09/04 09:48:12 response: id=hnZJyIsB status=200 bytes=0 challenges=[]
2025/09/04 09:48:12 request: id=kPeRusEu method="GET" path="/v2/foo/manifests/latest" auth="Basic Zm9vOmJhcg=="
2025/09/04 09:48:12 response: id=kPeRusEu status=200 bytes=0 challenges=[]
2025/09/04 09:48:14 request: id=ipKxBCNp method="HEAD" path="/v2/bar/manifests/latest" auth=""
2025/09/04 09:48:14 response: id=ipKxBCNp status=200 bytes=0 challenges=[]
2025/09/04 09:48:14 request: id=uJQVhqRt method="GET" path="/v2/bar/manifests/latest" auth=""
2025/09/04 09:48:14 response: id=uJQVhqRt status=200 bytes=0 challenges=[]

With containerd, per-repo authentication challenges are respected.

Docker version

Client: Docker Engine - Community
 Version:           27.5.1
 API version:       1.47
 Go version:        go1.22.11
 Git commit:        9f9e405
 Built:             Wed Jan 22 13:41:48 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.5.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.11
  Git commit:       4c9b3b0
  Built:            Wed Jan 22 13:41:48 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.2.4
  GitCommit:        v1.2.4-0-g6c52b3f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client: Docker Engine - Community
 Version:    27.5.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.20.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.5.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-52-generic
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.875GiB
 Name: docker-ce-ubuntu-2gb-ash-1
 ID: 2161eaf8-64b8-4aa5-93ec-fb31e476b76a
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 23
  Goroutines: 41
  System Time: 2025-09-04T15:16:13.32925468Z
  EventsListeners: 0
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Notes

• This may be a docker/cli issue more than a moby/moby issue, but I wasn't sure.

[ezekg]

Going to tag @thaJeztah here since you made this related comment: docker/cli#6141 (comment)

After digging in to history and comparing behavior on older daemons (which due to the various bugs did NOT prompt), I'm indeed considering that we could just remove the interactive prompt altogether.

I do want to have a look though, because the PrivilegeFunc is a much more elegant solution than the static RegistryAuth field, and perhaps we should consider making PrivilegeFunc the default way to obtain auth if needed (which could mean; try pull without any auth first, then fallback to trying to get auth from store).

Seems related, and as noted in OP, this may be a docker/cli issue vs a moby/moby issue.

[thaJeztah]

Oh! I thought I replied, but looks like I didn't; this is currently a known limitation; the bare /v2/ endpoint was originally designed for feature-discovery, including authentication; https://github.com/opencontainers/distribution-spec/blob/main/spec.md#determining-support

The registry client currently used is designed around that functionality, which was needed for the transition to the "v2" specification;

• Unprotected pull from Docker Registry V2 #17317
• Support for read-only pull and auth push distribution/distribution#1230 (comment)
• Unprotected pull from Docker Registry V2 distribution/distribution#1028

That transition is complete now; the docker daemon no longer supports v1 registries, and all mainstream registries now should be compliant with the OCI distribution spec v2. We are in progress of migrating the image store, and more parts of the "distribution" related code to use containerd (which re-implemented many parts with the intent to ultimately replace the existing implementations). The containerd implementations were written when the v2 specification was already established, and has optimisations for those, such as skipping the feature detection. Most of the work related to the image storage is completed, but parts related to "distribution" pushing/pulling will be next; this will still be a sizable amount of work as there's many parts of the codebase that are based around the original implementation, so we may need to take some intermediate steps;

• replace uses of distributions registry-client because it's deprecated and removed #46940
• fork distribution client v2 #48805

I can't give an ETA for that work yet; there's still quite some parts moving, including a large overhaul of auth-related bits, but revisiting some of the "intermediate" steps is still in my backlog.