The establishment of iptables rules lags behind Pod traffic occurrence

[jimmy-zh]

What happened:

In some cases, NPC's establishment of iptables rules lags behind Pod traffic occurrence, which results in network isolation failing to take effect. Maybe this is a knotty problem in consideration of K8s event synchronization mechanism.

How to reproduce it:

1. Deploy the following Network Policy in the K8s cluster:

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  name: test-np
  namespace: diagnosis
spec:
  egress:
  - to:
    - ipBlock:
        cidr: 14.215.0.0/16
  podSelector: {}
  policyTypes:
  - Egress

2. Deploy the following Deployment in the K8s cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: diagnosis
  namespace: diagnosis
spec:
  replicas: 2
  selector:
    matchLabels:
      app: diagnosis
  template:
    metadata:
      labels:
        app: diagnosis
    spec:
      nodeName: 10.0.1.30
      containers:
      - name: diagnosis
        command: ["/bin/sh"]
        args: ["-c","ping 220.181.38.150"]
        image: jimmyzh/diagnosis:v1

3. Observe the logs of certain Pods:

kubectl logs diagnosis-66f959fd76-2wpln -n diagnosis | head

PING 220.181.38.150 (220.181.38.150) 56(84) bytes of data.
64 bytes from 220.181.38.150: icmp_seq=1 ttl=53 time=3.66 ms
64 bytes from 220.181.38.150: icmp_seq=2 ttl=53 time=3.57 ms
64 bytes from 220.181.38.150: icmp_seq=3 ttl=53 time=3.58 ms

4. Observe the iptables rules:

iptables-save -t filter

-A KUBE-POD-FW-3F2W5JGDV6M5LYIN -m comment --comment "rule for stateful firewall for pod" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-POD-FW-3F2W5JGDV6M5LYIN -m comment --comment "run through nw policy test-np" -j KUBE-NWPLCY-ONWOLJXYLLQ2RYI2
-A KUBE-POD-FW-3F2W5JGDV6M5LYIN -m comment --comment "default rule to REJECT traffic destined for POD name:diagnosis-66f959fd76-2wpln namespace: diagnosis" -j REJECT --reject-with icmp-port-unreachable

5. Drop the stateful firewall rule manually:

iptables -t filter -D KUBE-POD-FW-3F2W5JGDV6M5LYIN -m comment --comment "rule for stateful firewall for pod" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

6. Observe the logs of the Pod above:

kubectl logs diagnosis-66f959fd76-2wpln -n diagnosis | tail

64 bytes from 220.181.38.150: icmp_seq=19347 ttl=53 time=3.59 ms
64 bytes from 220.181.38.150: icmp_seq=19348 ttl=53 time=3.58 ms
From 172.22.0.65 icmp_seq=19349 Destination Port Unreachable
From 172.22.0.65 icmp_seq=19350 Destination Port Unreachable
From 172.22.0.65 icmp_seq=19351 Destination Port Unreachable

The result indicates that the Pod traffic occurs before the establishment of iptables rules, and the related stateful firewall rule aggravates the ineffectiveness of isolation.

7. Add a 10 second sleep before the ping action in the Deployment above:

      containers:
      - name: diagnosis
        command: ["/bin/sh"]
        args: ["-c","sleep 10;ping 220.181.38.150"]
        image: jimmyzh/diagnosis:v1

8. Observe the logs of the new Pod:

kubectl logs diagnosis-796c7d859f-cpskr -n diagnosis | head

PING 220.181.38.150 (220.181.38.150) 56(84) bytes of data.
From 172.22.0.65 icmp_seq=1 Destination Port Unreachable
From 172.22.0.65 icmp_seq=2 Destination Port Unreachable
From 172.22.0.65 icmp_seq=3 Destination Port Unreachable

The result turns out to be normal after a sleep action.

[aauren]

@jimmy-zh I've suspected something similarly before. Just out of curiosity, would you mind adding a 60 second sleep before your ping to see what the results are?

If you're correct, I would expect the pings to fail from the beginning, as the iptables rules would have come into place before the first ping and restricted the traffic.

[jimmy-zh]

@jimmy-zh I've suspected something similarly before. Just out of curiosity, would you mind adding a 60 second sleep before your ping to see what the results are?

@aauren No, certainly not. Actually, I've done similar tests before. I added two steps to the issue description to show the new result.

[jimmy-zh]

AFAIK most of Network Policy Controllers are implemented in a similar way:

1. Watch Network Policy related events, such as Pod Adding.
2. Synchronize iptables rules to achieve traffic filtering.

This mechanism does not seem to guarantee that iptables rules are established before the Pod runs.

@murali-reddy @aauren Please tell me what you think about this problem. Thanks!

[murali-reddy]

If --run-firewall=true then we can change the default behaviour during the pod start to till network policies are evaluated for the pod is to drop traffic for both ingress and egress. perhaps this is right thing to do from security standpoint. Right now kube-router for a pod allows both ingress and egress traffic till network policies are evaluated for the pod which will decide to allow/drop both ingress and egress traffics.

But we run into the converse problem where traffic from the pod is dropped which will be eventually permitted by network policies. However this is less of problem for TCP traffic due to retry.

AFAIK there is no correct way to achive both the cases, as setting up the pod involes various asynchronous actions.

pod “ready++” can be of some help in addressing some cases but can not be a general solution. Let me investigate bit further if there is better solution.

[aauren]

I'm going to close this issue for now.

There is still an open conversation around whether or not kube-router could / should drop all pod traffic until it is able to evaluate all of the possible network policies that might apply to the pod.

Right now kube-router still allows traffic to be open until the network policy is applied. Since we then allow established / related flows there is still the possibilities that some flows that might be otherwise disallowed might be allowed going forward if a traffic flow is established fast enough.

At some point in the future we may choose to change this default or to allow the user to configure this policy. However, for the time being, we are in a much better place with the advent of kube-router 1.2.X. Where once it used to take 1 - 5 minutes to setup network policy even in small - medium sized clusters, we are now below 5 seconds even for large clusters. And we are continuing to improve that.

Given that < 5 seconds is also about how long it takes to spin up a pod and for processes to start, I think that we are well within range of this no longer being a problem.

If someone runs into something down the road, feel free to ping and we can re-open this issue.

[morremeyer]

I am running into this issue from the opposite side: I run a k3s cluster that by default has a NetworkPolicy that only allow Egress traffic to coredns so that DNS resolution works.

For some tasks, I run CronJobs (e.g. database backups). The startup of those pods (from pod creation to the container running) oftentimes takes less than a second.

This is not enough time for kube-router to apply the iptables rules derived from the NetworkPolices that allow those (Cron)Job pods access to the database. Therefore, they fail to reach their targets and go into an Error state.

If policy application were faster or there was some possiblity to delay pod startup until the policies are applied, this case could/would be solved.

[aauren]

Unfortunately, Kubernetes doesn't really give us hooks into delaying pod start based upon CNI readiness. While I feel like we have gotten kube-router to implement policy pretty quickly, because of the nature of iptables, we are unlikely to be able to apply policy in less than a second in a consistent way on anything but the smallest of nodes.

I would recommend that you add a step that attempts to resolve DNS before starting your application and have that pause the launch of your database backup until DNS is resolve-able. If you wanted to separate out this functionality, you could also implement this functionality as a initContainer.

Alternatively, you could implement retries to your job as well.

[morremeyer]

I'm thinking about implementing it as an initContainer, but it would make all of those jobs more complex.

Anyway, the situation here is clear, thanks for the feedback.

[disconn3ct]

If someone runs into something down the road, feel free to ping and we can re-open this issue.

I'm running a k3s cluster on RPI and getting a reproducible delay of at least 45s before the rules are applied. Does that count as something?

I'm happy to gather any info that can help. I really need networkpolicies to work reliably and I don't want to go back to Calico. (I saw above that it isn't a trivial fix, but was the default-deny flag ever exposed? I am fine with blocking connections until the correct rules are applied.)

[aauren]

@disconn3ct 45 seconds seems pretty long, I definitely wouldn't expect that. Can you post the version of kube-router that you are running? And the options you're running it with?

Additionally, it would help if you could enable verbose logging with -v 2 and give us the logs for an entire full policy sync.

The relevant logs will be bookend-ed with something like the following:

Starting sync of iptables with version: <version>
...
sync iptables took <time>

[disconn3ct]

I'm using the version embedded with k3s v1.24.9+k3s1
Starting the netpol controller version v1.5.2-0.20221026101626-e01045262706, built on 2022-12-21T00:28:07Z, go1.18.9

I can't find any way to pass arguments to it through k3s. Do you know of a way?

Unfortunately I missed part of your earlier message though - so long as the established/related problem isn't solved, kube-router can't meet my needs. Security via race conditions make me very nervous. Sorry to resurface an old bug for nothing.