Missing kernel modules while creating Calico on k3s flist

[AhmedHanafy725]

 Warning  Unhealthy  4m30s  kubelet            Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/bird/bird.ctl: connect: no such file or directory
  Warning  Unhealthy  4m29s  kubelet            Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused
  Warning  Unhealthy  4m19s  kubelet            Readiness probe failed: 2025-09-01 09:35:41.886 [INFO][531] confd/health.go 180: Number of node(s) with BGP peering established = 1
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  4m9s  kubelet  Readiness probe failed: 2025-09-01 09:35:51.616 [INFO][936] confd/health.go 180: Number of node(s) with BGP peering established = 1
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  3m49s  kubelet  Readiness probe failed: 2025-09-01 09:36:11.494 [INFO][1919] confd/health.go 180: Number of node(s) with BGP peering established = 1
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  3m39s  kubelet  Readiness probe failed: 2025-09-01 09:36:21.511 [INFO][2458] confd/health.go 180: Number of node(s) with BGP peering established = 1
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  3m29s  kubelet  Readiness probe failed: 2025-09-01 09:36:31.472 [INFO][3004] confd/health.go 180: Number of node(s) with BGP peering established = 1
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  3m29s  kubelet  Readiness probe failed: 2025-09-01 09:36:31.573 [INFO][3016] confd/health.go 180: Number of node(s) with BGP peering established = 1
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  3m19s  kubelet  Readiness probe failed: 2025-09-01 09:36:41.557 [INFO][3535] confd/health.go 180: Number of node(s) with BGP peering established = 1
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  56s (x16 over 3m9s)  kubelet  (combined from similar events): Liveness probe failed: calico/node is not ready: Felix is not live: Get "http://localhost:9099/liveness": dial tcp 127.0.0.1:9099: connect: connection refused
root@mre0rts:~# kubectl logs -n kube-system -l k8s-app=calico-node
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
2025-09-01 09:41:48.977 [ERROR][20032] felix/ipsets.go 599: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.11: Kernel error received: Invalid argument\n"
2025-09-01 09:41:48.977 [WARNING][20032] felix/ipsets.go 346: Failed to resync with dataplane error=exit status 1 family="inet"
2025-09-01 09:41:48.981 [ERROR][20032] felix/ipsets.go 599: Bad return code from 'ipset list'. error=exit status 1 family="inet6" stderr="ipset v7.11: Kernel error received: Invalid argument\n"
2025-09-01 09:41:48.981 [WARNING][20032] felix/ipsets.go 346: Failed to resync with dataplane error=exit status 1 family="inet6"
2025-09-01 09:41:49.234 [INFO][20032] felix/ipsets.go 337: Retrying after an ipsets update failure... family="inet"
2025-09-01 09:41:49.235 [ERROR][20032] felix/ipsets.go 599: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.11: Kernel error received: Invalid argument\n"
2025-09-01 09:41:49.235 [WARNING][20032] felix/ipsets.go 346: Failed to resync with dataplane error=exit status 1 family="inet"
2025-09-01 09:41:49.239 [INFO][20032] felix/ipsets.go 337: Retrying after an ipsets update failure... family="inet6"
2025-09-01 09:41:49.248 [ERROR][20032] felix/ipsets.go 599: Bad return code from 'ipset list'. error=exit status 1 family="inet6" stderr="ipset v7.11: Kernel error received: Invalid argument\n"
2025-09-01 09:41:49.248 [WARNING][20032] felix/ipsets.go 346: Failed to resync with dataplane error=exit status 1 family="inet6"
2025-09-01 09:41:49.096 [ERROR][20079] felix/ipsets.go 599: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.11: Kernel error received: Invalid argument\n"
2025-09-01 09:41:49.096 [WARNING][20079] felix/ipsets.go 346: Failed to resync with dataplane error=exit status 1 family="inet"
2025-09-01 09:41:49.097 [ERROR][20079] felix/ipsets.go 599: Bad return code from 'ipset list'. error=exit status 1 family="inet6" stderr="ipset v7.11: Kernel error received: Invalid argument\n"
2025-09-01 09:41:49.097 [WARNING][20079] felix/ipsets.go 346: Failed to resync with dataplane error=exit status 1 family="inet6"
2025-09-01 09:41:49.225 [INFO][20079] felix/ipsets.go 337: Retrying after an ipsets update failure... family="inet"
2025-09-01 09:41:49.225 [INFO][20079] felix/ipsets.go 337: Retrying after an ipsets update failure... family="inet6"
2025-09-01 09:41:49.226 [ERROR][20079] felix/ipsets.go 599: Bad return code from 'ipset list'. error=exit status 1 family="inet" stderr="ipset v7.11: Kernel error received: Invalid argument\n"
2025-09-01 09:41:49.226 [WARNING][20079] felix/ipsets.go 346: Failed to resync with dataplane error=exit status 1 family="inet"
2025-09-01 09:41:49.227 [ERROR][20079] felix/ipsets.go 599: Bad return code from 'ipset list'. error=exit status 1 family="inet6" stderr="ipset v7.11: Kernel error received: Invalid argument\n"
2025-09-01 09:41:49.227 [WARNING][20079] felix/ipsets.go 346: Failed to resync with dataplane error=exit status 1 family="inet6"

looks it requires some modules like ip_set, ip_set_hash_ip, ip_set_hash_net

[ashraffouda]

I believe this will require changing the image of the current running nodes which will be difficult to ask all farmers to do

[delandtj]

1: when farmers have nodes that are meant to be used as kXs nodes, the nodes will be fully rented, no? No issue then as rebooting it with a new version will certainly help.
2: let's get to the basics first:

• a kubernetes node will always be a VM, even if the node is fully rented, so I don't get the 'flist' part: is it a VM? Or do you intend to run the kXs on the node itself ?
• first make sure we can effectively run calico over nodes (VMs) that are connected through Mycelium. That means: a VM has received a private IP address as usual and an ip address from mycelium.
  Since we're talking about VMs here, there should be no need to have Zos rebuilt with the ipset modult (of which I'm baffled it's not included)

3: Ahmed, can you show me what you're trying to do ?

[ashraffouda]

@xmonader what do u think here? np with having new image and rebooting nodes that will run k8s?

[xmonader]

That doesn't solve it for shared nodes where ppl can deploy their clusters too

[xmonader]

That doesn't solve it for shared nodes where ppl can deploy their clusters too