Peers can't contact eachother over the network

[Twi]

Describe the bug
I want to create a network such that peers can contact eachother as if they were on the same physical network segment.

To Reproduce
Steps to reproduce the behavior:

apiVersion: v1
kind: Namespace
metadata:
  name: wireguard
---
apiVersion: vpn.wireguard-operator.io/v1alpha1
kind: Wireguard
metadata:
  name: "ponyville"
  namespace: wireguard
spec:
  mtu: "1380"
  serviceType: "NodePort"
  enableIpForwardOnPodInit: true
---
apiVersion: vpn.wireguard-operator.io/v1alpha1
kind: WireguardPeer
metadata:
  name: rainbow-dash
  namespace: wireguard
spec:
  wireguardRef: "ponyville"
---
apiVersion: vpn.wireguard-operator.io/v1alpha1
kind: WireguardPeer
metadata:
  name: rarity
  namespace: wireguard
spec:
  wireguardRef: "ponyville"

Expected behavior
Node rainbow-dash to be able to ping node rarity and connect over TCP/UDP/IP.

Additional context
Add any other context about the problem here.

[jodevsa]

Hi @Twi,

Thank you for openning the issue. I have tried to reproduce and at least I can verify that ICMP connectivity (ping) between the nodes is possible.

How are you verifying this?

[jodevsa]

HTTP (TCP) is also possible between the peers

[jodevsa]

Can you try to also check connectivity from the wireguard pod itself. here are some tips on how you can debug this:

find out the wireguard pod

kubectl get pods -lapp=wireguard -n NAMESPACE_YOU_USED

bash into the wireguard pod

kubectl exec -it <POD_NAME> -c agent -- bash

list connected peers

wg

list iptables rules

iptables -L

those should be similar to what I have as you haven't configured any networkpolicy

connectivity check

ping 10.8.0.2
ping 10.8.0.3

You could also do a tcpdump on the pod and analyse the logs. I'll be happy to help you debug this further and available for a quick call if needed

[SuperCoolAlan]

I am experiencing a similar issue where I have confirmed that each peer can ping the server, but the peers cannot route to each other. I have been able to confirm that each peer can ping the wg server at 10.8.0.1 but tcpdump on the peer shows no response ever making it back for ICMP requests made peer-to-peer.

wg show

iptables -L

iptables -L output seems similar to the example provided above

Peer configuration

I have tested multiple types of peers, both running in the cluster and out of the cluster. Behavior is the same for both which leads me to believe that I am facing a server configuration or firewall issue.

Peer in same cluster as wg server

Here is one peer's configuration. I am running a Gluetun container as a sidecar

[Interface]
PrivateKey = <redacted>
Address = 10.8.0.4
DNS = 10.96.0.10, podname.svc.cluster.local
MTU = 1380
[Peer]
PublicKey = B6WMlMhtiorCmokBqspyelkFnLX9x4t6i8NMPFkiTnk=
AllowedIPs = 10.8.0.4/24
Endpoint = <redacted>

Peer out of cluster

[Interface]
PrivateKey = <redacted>
Address = 10.8.0.3
MTU = 1380
[Peer]
PublicKey = B6WMlMhtiorCmokBqspyelkFnLX9x4t6i8NMPFkiTnk=
AllowedIPs = 10.8.0.3/24
Endpoint = <redacted>

Next steps

I'd like to see if the server is able to route to each peer but the container image that is deployed via the operator is restrictive. Is there a pre-built debugging image available that will host the server and also include debugging tools/permissions? If so, how can I provide the operator with this image? I am using ghcr.io/jodevsa/wireguard-operator/agent:v2.7.0

[nikolaigut]

I am facing the same issue.

From a peer, it is possible to ping the wireguard server 10.8.0.1, but not other peers.

The tracepath command returns a no reply after the gateway:

 tracepath 10.8.0.41
 1?: [LOCALHOST]                      pmtu 1380
 1:  10.8.0.1                                             28.494ms 
 1:  10.8.0.1                                             27.240ms 
 2:  no reply
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
13:  no reply
14:  no reply
15:  no reply
16:  no reply
17:  no reply
18:  no reply
19:  no reply
20:  no reply
21:  no reply
22:  no reply
23:  no reply
24:  no reply
25:  no reply
26:  no reply
27:  no reply
28:  no reply
29:  no reply
30:  no reply
     Too many hops: pmtu 1380
     Resume: pmtu 1380

The ping to peers from the wireguard server is working.

[nikolaigut]

I made extended analysis on my problem, and it turned out, that no firewall rules allowed connections from the wireguard IP range.

The reason was, that I've set the AllowedIPs property of the wireguard configuration to specific IP addresses. After I included the wireguard IP range to the AllowedIPs property of the wireguard peer configuration, it works. Now my peers can communicate with the other peers.

@Twi can you try it with including the wireguard IP range into the AllowedIPs property (default is 0.0.0.0 which includes the wireguard IP range).