Managed ECH HTTPS records omit ALPN, causing clients to prefer h2 over h3

[h3nnes]

Issue Details

Short description

When using Caddy with ECH enabled, h3 breaks and clients fall back to h2.
There are QUIC errors, as shown in the attached 'https-record-h2-fallback.txt' curl log.
While debugging, I removed the HTTPS record and restarted Caddy with the ECH configuration still enabled in Caddyfile. After that, h3 connections started working again, as can be seen in the 'no-https-record-h3-works.txt' curl log.

Setup

• Caddy version:

v2.11.3-0.20260421185931-aed1af59763d h1:mZleOSpQzf10ACi1frV+ZLeX7t1XgI1FzKga3KxCuKk=`

• Caddy compile command:

./xcaddy build master --with github.com/caddy-dns/cloudflare@master --output caddy

• golang version:

go version go1.25.0 linux/amd64

• DNS records:

$ dig ech.h-neef.de
89.144.8.160

$ dig speed.h-neef.de
89.144.8.160

$ dig +short @cloudflare-dns.com +https HTTPS speed.h-neef.de
2 . ech=AEj+DQBEzQAgACBVpRfw+C+q3Mq5XAwSxVEbY1Unray6SACeDyfhd1t4OAAMAAEAAQABAAIAAQADHQ1lY2guaC1uZWVmLmRlAAA=

• Debian 13 amd64 VPS

• Open FW ports 80,443 (TCP/UDP)

• Minimal Caddyfile to reproduce:

{
    debug
    acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}
    dns cloudflare {env.CLOUDFLARE_API_TOKEN}
    ech ech.h-neef.de
}

speed.h-neef.de {
    respond "Hi!"
}

Steps to reproduce

1. Build newest Caddy with a DNS provider
2. Take the attached ECH-enabled Caddyfile and modify domains and global DNS config
3. Make a request to your site, it will use h2 as the protocol
4. After that, either just remove the HTTPS record for the domain and restart Caddy, or remove the HTTPS records, remove the ECH config in Caddyfile and restart Caddy
5. Make a new request to your site, it will use h3 as the protocol

Logs

Attached the two logs from http3 and ECH enabled curl.

https-record-h2-fallback.txt

no-https-record-h3-works.txt

The first log file https-record-h2-fallback.txt contains the first request while Caddy was correctly configured with ECH and there was an existing HTTPS DNS records. h3 was attempted, but failed and clients fell back to h2.
The second log file no-https-record-h3-works.txt contains the second request after the HTTPS DNS record was removed at my DNS provider and Caddy was restarted (with ECH config still in Caddyfile).

Assistance Disclosure

AI not used

If AI was used, describe the extent to which it was used.

No response

[steadytao]

This should be fixed by #7653. The issue here appears to be the managed HTTPS RR contents, not ECH specifically. Right now the published record includes ech but not alpn which can cause clients to choose the TLS/TCP path and negotiate h2 instead of going to h3 over QUIC. So unless there is a second separate QUIC bug here (which fortunately there doesn't seem to be), I would expect this issue to be resolved once that change is merged or if you test a build that includes that PR.

[h3nnes]

@steadytao
Just edited the HTTPS record and added alpn and ip hints:

$ dig +short @cloudflare-dns.com +https HTTPS speed.h-neef.de
2 . alpn="h3,h2" ipv4hint=89.144.8.160 ech=AEj+DQBEHAAgACDefVccUBaIRt+WzPMXdaLMkUSj1K+nctbQxOtgBaSgeQAMAAEAAQABAAIAAQADHQ1lY2guaC1uZWVmLmRlAAA= ipv6hint=2a0c:4ac1:4:193:dead:beef:cafe:affe

Then I restarted Caddy (and checked that the record is still the same), but still, only h2 connections are established even after re-opening the browser.
Here's the curl log attached:

edited-https-record-h2-fallback.txt

Notably, it says QUIC connect to 89.144.8.160 port 443 failed: SSL peer certificate or SSH remote key was not OK

Any idea what else I could test?

[h3nnes]

@steadytao Quick update:
Built Caddy with your feature branch feat-https-rr-alpn:

./xcaddy build feat-https-rr-alpn --with github.com/caddy-dns/cloudflare@master --output caddy

And checked the HTTPS record after starting Caddy (looks like http/1.1 was added automatically):

$ dig +short @cloudflare-dns.com +https HTTPS speed.h-neef.de
2 . alpn="h3,h2,http/1.1" ipv4hint=89.144.8.160 ech=AEj+DQBEQwAgACDiRuROrZtqRt1Xp65F3niMaxosdyY7DhtjgpCNwlJvbQAMAAEAAQABAAIAAQADHQ1lY2guaC1uZWVmLmRlAAA= ipv6hint=2a0c:4ac1:4:193:dead:beef:cafe:affe

Unfortunately, clients still fall back to h2.

curl log attached again:

caddy-httpsrr-branch-h2-fallback.txt

[steadytao]

#7653 did fix one of the issues which was the HTTPS RR alpn omission but it doesn't fix the remaining failure you showed.

I tracked the follow-up locally and the QUIC / HTTP/3 path was rebuilding a minimal TLS config with GetConfigForClient only so GetEncryptedClientHelloKeys was not being forwarded there. That means ECH could work on the normal TLS path while still failing on the QUIC path.

I opened a separate follow-up PR for that QUIC-side ECH wiring bug. Thank you.

[steadytao]

Ah I forgot to mention but if you want to test both fixes before merge, build from feat-https-rr-alpn and cherry-pick fix-quic-ech-key-propagation on top. That gives you the HTTPS RR alpn fix plus the QUIC-side ECH locally.

git fetch origin feat-https-rr-alpn fix-quic-ech-key-propagation

git switch -c test-ech-h3 origin/feat-https-rr-alpn
git cherry-pick origin/fix-quic-ech-key-propagation

xcaddy build --with github.com/caddy-dns/cloudflare@master --output caddy

[h3nnes]

@steadytao
Thank you, #7670 does the trick!
Browser, Wireshark and curl confirm working ECH with h3 and QUIC 🎉

I've rebuild caddy like this:

$ git clone https://github.com/caddyserver/caddy
$ cd caddy

$ git config --global user.email "h3nnes@noreply.local"
$ git config --global user.name "h3nnes"

$ git fetch origin feat-https-rr-alpn fix-quic-ech-key-propagation
$ git switch -c test-ech-h3 origin/feat-https-rr-alpn
$ git checkout .
$ git cherry-pick origin/fix-quic-ech-key-propagation

$ cd ..

# Rebuild Caddy
$ ./xcaddy build --with github.com/caddyserver/caddy/v2=./caddy --with github.com/caddy-dns/cloudflare@master --output caddy

# Set permissions
$ sudo setcap 'cap_net_bind_service,cap_net_admin=+ep' caddy

Small update:
Out of curiosity, I wanted to check what would happen if the alpn indicator is removed now from the HTTPS DNS record.
Turns out clients will fall back to h2 again, which means alpn indicator is in fact necessary for successful h3/QUIC negotiations (as expected, but good to know that it does make a difference)

[mholt]

Thank you both!