Why is traffic always routed via relays?

[khartahk]

question re: caddy-tailscale plugin:

Why does the connection to my node always go via a relay when on my phone either wifi or mobile data, it doesn't matter, but doesn't go via relay when I try to connect to the same server via the tailscale service running on that same node?

And why am I bothered by that you might ask, coz there my speed is being throttled to 1Mb/s and to stream a video that's just not enough.

I worked all day trying to debug this. I've gotten so far that I had to figure out:

• how to set up Hairpin NAT on my Mikrotik router
• then I could spin up my own derp server on the same node as the caddy is running
• had to figure out how to run that same caddy on tailscale/ interface and on 0.0.0.0
• installed layer4 caddy plugin to forward udp packets to stun, since i can't run derp directly because of other stuff running ther.

Finally set everyrhing up and android now shows my phone going through that relay and there is a slight speed increase but still not enough. What am I missing here?

You could try something like:

Caddyfile:
{
	tailscale {
		server_name {
			port 12345
		}
       }
}
...

docker-compose:
...
  caddy:
    ...
    ports:
      - 12345:12345/udp

Port 12345 above is the port that will be used for the Tailscale Wireguard VPN. You don't need to expose other ports unless you want the server accessible both on the LAN directly and Tailscale.

If you are using IPv4, make sure you use an endpoint-independent NAT: https://help.mikrotik.com/docs/spaces/ROS/pages/3211299/NAT#NAT-Endpoint-IndependentNAT.

You should find that your connections are direct on the LAN and at worst very quickly move from relay to direct when remote.

[khartahk]

I've added both lines to my router but the connections stays rellayed.

I do also have NAT-PMP enabled on the router and I can see that tailscale is punching holes on the WAN side to enable direct connections so I know that part is working.

You might need to post more about your network setup and caddy and router configurations then for others to help.

[khartahk]

OK, so my setup is as follows:

mikrotik router -> server (running tailscale and caddy-tailscale) -> client device

when a client device is on the wifi (same network as server) or on mobile data the connection to caddy when I try to ping it via the clients tailscale app is always relayed. If I try to ping the same server directly to his hostname it is always direct no matter if it's on wifi or on mobile.

So my question is why is caddy tailscale behaving differently then the tailscale client?

OK now the config:

cat /etc/caddy/caddy_config
{
	acme_dns cloudflare {
		api_token {env.CLOUDFLARE_DNS_API_TOKEN}
	}

	tailscale {
		ephemeral false
		server_name {
			port 42789
		}
	}

	log tailscale {
		level ERROR
	}

	layer4 {
		tailscale/:22 {
			@ssh ssh
			route @ssh {
				proxy 127.0.0.1:2222
			}
		}
	}

	log {
		level ERROR
	}
}

immich.rudnik.si {
	bind tailscale/

	reverse_proxy localhost:2283
}

On the router I've as I said enabled NAT-PMP and that show that the ports are being oppened up from all the tailsclae devices, that are running on the network.

Also the server is showing that it has oppened up two ports one in the range of 51qwe and the other 53xyz, both UDP - which I'm guessing are from the caddy-tailscale and tailscale. I've disabled both the tailscale app and caddy-tailscale to see what happens and when I stop caddy and start it again the port mapping is added in a moment. For tailscale I haven't observed any change so perhaps it's retaining the mapping for longer?

Looking at where the ports are mapped to the tailscale client is always using the same port for the host device: 41641, (this port is used on 3 different devices as the local port where the firewall sends the incoming packets - the outer port is always different) but caddy-tailscale is choosing different ports as the local ones. And yes I can confirm that caddy is the one adding the port mappings.

And here's the output from netstat on the server:

sudo netstat -tlpnu | grep -E '(caddy|tailscale)'                       
tcp        0      0 0.0.0.0:45785           0.0.0.0:*               LISTEN      432614/caddy        
udp        0      0 0.0.0.0:40000           0.0.0.0:*                           2091/tailscaled     
udp        0      0 0.0.0.0:41641           0.0.0.0:*                           2091/tailscaled     
udp        0      0 0.0.0.0:52429           0.0.0.0:*                           432614/caddy        
udp6       0      0 :::40000                :::*                                2091/tailscaled     
udp6       0      0 :::48437                :::*                                432614/caddy        
udp6       0      0 :::41641                :::*                                2091/tailscaled     
udp6       0      0 :::3478                 :::*                                432614/caddy

Your syntax is not quite right, causing the port that caddy-tailscale selects to not match what you specified (42789). Mine looks more like this:

Try changing:

		server_name {
			port 42789

to:

		immich {
			port 42789

and

immich.rudnik.si {
	bind tailscale/immich

Your docker-compose should then also have:

    ports:
      - 42789:42789/udp

You should then see port 42789 bound. In the above "immich" and "tailscale/immich" are the Tailscale node that will be registered for the server.

If you have access to IPv6 you can avoid NAT issues. Otherwise, you do need endpoint-independent NAT. Make sure that your NAT mapping is actually endpoint independent: whatever port 41641/udp and 42789/udp is translated to, it is translated the same for every external destination IP over a long period of time. I am not able to help with your Mikrotik configuration but one of the AI can write the config for you.

Looking at my router config, you probably want to use the randomize-port option as otherwise I'm not sure how it would handle multiple internal nodes using source port 41641/udp. You may need a longer NAT session timeout; I followed the Tailscale guide for PANW firewalls when configuring my router, which sets the timeout for the Tailscale ports to 4 minutes.

You shouldn't need other unusual configurations. You definitely should be getting direct connections within the same LAN and if it is not working it makes me wonder if you have a host firewall blocking.

NAT-PMP is fine but not essential -- you can get direct connections with no inbound open ports required. You do not need to have hairpin NAT, and while it shouldn't cause the problems you are seeing it is not helping -- your nodes can talk directly on the LAN and will find their public and private IP addresses from the Tailscale coordination server. Adding hair pinning might mean traffic that could be switched src->dst directly instead goes src->router->nat->dst.

This and the subsequent section on troubleshooting may help: https://tailscale.com/kb/1411/device-connectivity#connectivity-matrix

[khartahk]

I've fixed the port so now it's using the one I've set. I've added the endpoint-indenpendent-nat rules to the firewall, yet still when doing a ping on my phone either via WiFi or mobile data it's still relayed.

But when I run a tailscale ping this from my laptop it says direct connection - on WiFi and LAN.

I've removed the hair pinning - as I don't need it if I'm not running DERP server, and when I disabled NAT-PMP the direct connections were still established from basicaly any other device but my phone.

• my laptop in same LAN
• a server behind a router in another city on residential internet
• a server in the oracle cloud

All but my phone work - so I'm at a loss still :(

But thank you for the ideas.

Looking at peer relays now just to see if that helps a bit.

That’s an improvement! You should not need to do strange things and in theory your phone should work as well as your laptop, especially on the LAN.

The diagnostics on the phone are not as complete as on Linux/Mac/Windows. In the web console you can click on your phone and then scroll down to Client Connectivity to see how it is different.

[khartahk]

They both have client conectivity at UDP set to yes, all the other are no. But all endpoints are to some weird IP: 46.122.70.30 ... 🤔

On the laptop the connectivity is to my public IP and local LAN IP.

I will guess that the 46.* IP is your ISP CGNAT (A1 Slovenia). Your phone should have an endpoint on your LAN in order to make direct connection via LAN. I can’t explain why you don’t have that but will suggest it will work better once that is resolved. Have you done anything unusual to your phone?

Also A1 seems to support IPv6 on both mobile and home broadband — you may want to contact them to learn how to enable as that will avoid NAT and may help with connectivity — see below.

In my case in the connectivity report I see for my devices:

• private home LAN IPv4
• NAT IPv4
• IPv6 from ISP prefix delegation

And for caddy-Tailscale:

• docker container network IPv4 — NOT my LAN IPv4!
• NAT IPv4
• IPv6 from ISP prefix delegation

That is interesting that caddy-Tailscale is not announcing an IPv4 for my private LAN — but makes sense as it is running in a docker network and not host networking. If you are using IPv4 only that may not be helping and may be the difference between caddy-Tailscale and Tailscale for you. I never noticed that before and am not sure how to tell caddy to announce a different node endpoint IP.

I won’t be able to try until the weekend but if you are in the same situation then possibly connecting caddy to the LAN via macvlan network and then also to the container bridge networks will work — caddy-Tailscale will then have an IPv4 on your LAN to send the Tailscale coordination servers.

[khartahk]

I have stock Android that comes from the manufacturer. What is interesting is that caddy-tailscale is announcing some networks on my server that are going through a VPN and a network inside docker - but I'm not running this in docker, it's a service running directly on host. How can I forcecaddy-tailscale to only sue certain network adapters? i think that could solve this issue as I can see my LAN, public IP, namespaced subnet that has access to the internet via vpn and the podman/docker subnet.

I’m not aware of a way within caddy-Tailscale to filter the interfaces that tsnet sees. I think you could use macvlan to present the container with an interface on the LAN without the other interfaces.

Just be aware that if you are using host networking now (sounds to be the case) and change caddy to macvlan then caddy will not be able to reach the containers. I think you would give caddy 2 (or more) interfaces, one in macvlan and then one+ on container bridge networks.