False negative results for flow validation in connectivity tests

[wondering639]

Bug report

When running the connectivity test including flow validation, some tests fail. Without flow validation all tests pass.
One of the failing tests is 'no-policies/pod-to-service' and I did further research into that one case.

Let me give an overview of relevant IPs first:

Item	Name / Namespace	IP Address
Client executing curl	cilium-test-1/client2-78bbbfb455-9hsjb	10.244.2.117
Service echo-same-node	cilium-test-1/echo-same-node	10.111.229.242
Pod behind service echo-same-node	cilium-test-1/echo-same-node-64d7f567d8-4m6xp	10.244.2.101

The test 'no-policies/pod-to-service' as the name implies calls curl on the client with the service name "echo-same-node", not with the pod IP behind the service. So it does curl -v http://echo-same-node:8080

The test expects to see the service IP in the flow, but this is not the case, I think due to Socket LB, but being new to Cilium I am not 100% sure about it:

root@worker2:/home/cilium# cilium-dbg status --verbose | grep -i "socket lb"
Socket LB:            Enabled
Socket LB Tracing:    Enabled
Socket LB Coverage:   Full

Here's the failing test output, note that is expects to see the service IP, not the Pod IP that's behind the service:

cilium hubble port-forward &

cilium connectivity test \
	--namespace-labels pod-security.kubernetes.io/enforce=privileged \
	--test 'no-policies/pod-to-service'

# ... some output removed ...
[.] Action [no-policies/pod-to-service:curl-ipv4-0: cilium-test-1/client2-78bbbfb455-9hsjb (10.244.2.117) -> cilium-test-1/echo-same-node (10.111.229.242:8080)]
📄 Following flows...
📄 Validating flows for peer cilium-test-1/client2-78bbbfb455-9hsjb
✅ DNS request found at 0
✅ DNS response found at 1

# HERE IS THE INTERESTING LINE: dst=10.111.229.242 is the IP of the service and it is not seen in the flow
ℹ️  SYN and(ip(src=10.244.2.117,dst=10.111.229.242),tcp(dstPort=8080),tcpflags(syn)) not found

# ... some output removed (obviously all other checks expecting the service IP also fail) ...

When testing and observing manually, one can see that there is a translation of the service IP directly to the pod IP taking place.
Therefore the service IP is not visible in the flow (except for the translation entry), instead the pod IP is:

# with hubble:
root@worker2:/home/cilium# hubble observe --follow --ip-translation=false |egrep "10.111.229.242|10.244.2.101|10.244.2.117"
# RELEVANT LINE: 10.111.229.242 is the service IP, we see XLATE for it
Jun 18 07:22:17.280: 10.244.2.117 (ID:27117) <> 10.111.229.242:8080 (world) SOCK_XLATE_POINT_UNKNOWN TRACED (TCP)
# RELEVANT LINE: 10.244.2.101 is the IP of the only Pod of the service, note that we see its IP from now on, not the service IP
Jun 18 07:22:17.280: 10.244.2.117:39522 (ID:27117) -> 10.244.2.101:8080 (ID:1266) to-endpoint FORWARDED (TCP Flags: SYN)
Jun 18 07:22:17.280: 10.244.2.117:39522 (ID:27117) <- 10.244.2.101:8080 (ID:1266) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Jun 18 07:22:17.280: 10.244.2.117:39522 (ID:27117) -> 10.244.2.101:8080 (ID:1266) to-endpoint FORWARDED (TCP Flags: ACK)
Jun 18 07:22:17.281: 10.244.2.117:39522 (ID:27117) -> 10.244.2.101:8080 (ID:1266) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jun 18 07:22:17.282: 10.244.2.117:39522 (ID:27117) <- 10.244.2.101:8080 (ID:1266) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jun 18 07:22:17.284: 10.244.2.117:39522 (ID:27117) -> 10.244.2.101:8080 (ID:1266) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Jun 18 07:22:17.285: 10.244.2.117:39522 (ID:27117) <- 10.244.2.101:8080 (ID:1266) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Jun 18 07:22:17.285: 10.244.2.117:39522 (ID:27117) -> 10.244.2.101:8080 (ID:1266) to-endpoint FORWARDED (TCP Flags: ACK)

# with cilium-dbg:
root@worker2:/home/cilium# cilium-dbg monitor -v -t trace-sock | egrep "10.111.229.242|10.244.2.101|10.244.2.117"
Listening for events on 2 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
# RELEVANT LINE: note the [pre-xlate-fwd] tag
CPU 01: [pre-xlate-fwd] cgroup_id: 2724 sock_cookie: 739024, dst [10.111.229.242]:8080 tcp

This are the versions I use (nevertheless to my understanding the root cause is in the tests in the Cilium CLI)

• Talos 1.13.4
• Kubernetes 1.36
• Cilium 1.19.5
• Cilium CLI 0.19.4

Other remarks:

• kubeproxy fully replaced by Cilium
• not using KubeSpan from Talos

Command and parameters used to install Cilium:

helm install cilium oci://quay.io/cilium/charts/cilium --version 1.19.5 \
   --namespace kube-system \
   --set ipam.mode=kubernetes \
   --set kubeProxyReplacement=true \
   --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
   --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
   --set cgroup.autoMount.enabled=false \
   --set cgroup.hostRoot=/sys/fs/cgroup \
   --set k8sServiceHost=${ENDPOINT_VALUE} \
   --set k8sServicePort=${ENDPOINT_PORT} \
   --set hubble.enabled=true \
   --set hubble.relay.enabled=true

Other failing tests

These tests involving service are also failing for me. I did no further research, but probably they have the same root cause:

Test [allow-all-except-world]:
  🟥 allow-all-except-world/pod-to-service:curl-ipv4-0: cilium-test-1/client-657b75749d-6q4x7 (10.244.2.52) -> cilium-test-1/echo-other-node (10.100.236.119:8080): Flow validation failed for peer cilium-test-1/client-657b75749d-6q4x7: 2 failures (first: -1, last: -1, matched: 0)
  🟥 allow-all-except-world/pod-to-service:curl-ipv4-1: cilium-test-1/client-657b75749d-6q4x7 (10.244.2.52) -> cilium-test-1/echo-same-node (10.111.229.242:8080): Flow validation failed for peer cilium-test-1/client-657b75749d-6q4x7: 2 failures (first: -1, last: -1, matched: 0)
  🟥 allow-all-except-world/pod-to-service:curl-ipv4-2: cilium-test-1/client2-78bbbfb455-9hsjb (10.244.2.117) -> cilium-test-1/echo-other-node (10.100.236.119:8080): Flow validation failed for peer cilium-test-1/client2-78bbbfb455-9hsjb: 2 failures (first: -1, last: -1, matched: 0)
  🟥 allow-all-except-world/pod-to-service:curl-ipv4-3: cilium-test-1/client2-78bbbfb455-9hsjb (10.244.2.117) -> cilium-test-1/echo-same-node (10.111.229.242:8080): Flow validation failed for peer cilium-test-1/client2-78bbbfb455-9hsjb: 2 failures (first: -1, last: -1, matched: 0)
  🟥 allow-all-except-world/pod-to-service:curl-ipv4-4: cilium-test-1/client3-58f7994d44-n4mkv (10.244.1.26) -> cilium-test-1/echo-same-node (10.111.229.242:8080): Flow validation failed for peer cilium-test-1/client3-58f7994d44-n4mkv: 2 failures (first: -1, last: -1, matched: 0)
  🟥 allow-all-except-world/pod-to-service:curl-ipv4-5: cilium-test-1/client3-58f7994d44-n4mkv (10.244.1.26) -> cilium-test-1/echo-other-node (10.100.236.119:8080): Flow validation failed for peer cilium-test-1/client3-58f7994d44-n4mkv: 2 failures (first: -1, last: -1, matched: 0)
Test [pod-to-itself-via-service]:
  🟥 pod-to-itself-via-service/pod-to-itself-via-service:curl-ipv4-0: cilium-test-1/l7-lb-7894747948-qlkml (10.244.1.81) -> cilium-test-1/l7-lb-non-l7 (10.100.73.222:8080): Flow validation failed for peer cilium-test-1/l7-lb-7894747948-qlkml: 2 failures (first: -1, last: -1, matched: 0)

Also failing for me are these tests, though I think they could have a different root cause as they don't use a service. I did not find the time to check this though, just mentioning for completeness:

Test [to-fqdns]:
  🟥 to-fqdns/pod-to-world:http-to-one.one.one.one.-ipv4-0: cilium-test-1/client-657b75749d-6q4x7 (10.244.2.52) -> one.one.one.one.-http (one.one.one.one.:80): Flow validation failed for peer cilium-test-1/client-657b75749d-6q4x7: 1 failures (first: 1, last: 7, matched: 4)
  🟥 to-fqdns/pod-to-world:http-to-one.one.one.one.-ipv4-1: cilium-test-1/client2-78bbbfb455-9hsjb (10.244.2.117) -> one.one.one.one.-http (one.one.one.one.:80): Flow validation failed for peer cilium-test-1/client2-78bbbfb455-9hsjb: 1 failures (first: 1, last: 7, matched: 4)
  🟥 to-fqdns/pod-to-world:http-to-one.one.one.one.-ipv4-2: cilium-test-1/client3-58f7994d44-n4mkv (10.244.1.26) -> one.one.one.one.-http (one.one.one.one.:80): Flow validation failed for peer cilium-test-1/client3-58f7994d44-n4mkv: 1 failures (first: 1, last: 9, matched: 4)