Scapy v2.6.0 doesn't correctly read routes in containers

[michaelvdnet]

Brief description

The new rtnetlink implementation builds an incorrect route table on Azure Container Apps.
This breaks pretty much all inet functionality as Ether layer dst and IP layer src fields are no longer correctly set due to lookup logic involving the route table.

Scapy version

2.6.1

Python version

3.11.11

Operating system

python:11-alpine image -> alpine 3.21.3 -> linux 5.15.173.1-1.cm2

Additional environment information

No response

How to reproduce

>>> scapy.__version__
>>> conf.route  
>>> conf.route.route()
>>> p = Ether()/IP(dst="<snip>")/ICMP()
>>> p.show()
>>> srp1(p)

Actual result

>>> scapy.__version__
'2.6.1.dev49'
>>> conf.route
Network          Netmask          Gateway      Iface  Output IP     Metric
0.0.0.0          0.0.0.0          169.254.1.1  eth0   0.0.0.0       0     
100.100.0.14     255.255.255.255  0.0.0.0      eth0   100.100.0.14  0     
100.100.127.255  255.255.255.255  0.0.0.0      eth0   100.100.0.14  0     
127.0.0.0        255.0.0.0        0.0.0.0      lo     127.0.0.1     0     
127.0.0.1        255.255.255.255  0.0.0.0      lo     127.0.0.1     0     
127.255.255.255  255.255.255.255  0.0.0.0      lo     127.0.0.1     0     
169.254.1.1      255.255.255.255  0.0.0.0      eth0   0.0.0.0       0     
>>> conf.route.route()
('lo', '0.0.0.0', '0.0.0.0')
>>> p = Ether()/IP(dst="<snip>")/ICMP()
>>> p.show()
###[ Ethernet ]###
  dst       = None
  src       = 26:5d:4b:dd:17:a0
  type      = IPv4
###[ IP ]###
     version   = 4
     ihl       = None
     tos       = 0x0
     len       = None
     id        = 1
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = None
     src       = 0.0.0.0
     dst       = <snip>
     \options   \
###[ ICMP ]###
        type      = echo-request
        code      = 0
        chksum    = None
        id        = 0x0
        seq       = 0x0
        unused    = b''

>>> srp1(p)
Begin emission
WARNING: No broadcast address found for iface eth0

..............^CWARNING: MAC address to reach destination not found. Using broadcast.

Finished sending 0 packets

Received 14 packets, got 0 answers, remaining 0 packets

Expected result

>>> scapy.__version__
'2.5.0'
>>> conf.route
Network      Netmask          Gateway      Iface  Output IP     Metric
0.0.0.0      0.0.0.0          169.254.1.1  eth0   100.100.0.33  0     
127.0.0.0    255.0.0.0        0.0.0.0      lo     127.0.0.1     1     
169.254.1.1  255.255.255.255  0.0.0.0      eth0   100.100.0.33  0     
>>> conf.route.route()
('eth0', '100.100.0.33', '169.254.1.1')
>>> p = Ether()/IP(dst="<snip>")/ICMP()
>>> p.show()
WARNING: No broadcast address found for iface eth0

###[ Ethernet ]### 
  dst       = aa:aa:aa:aa:aa:aa
  src       = ae:f4:4a:dd:f9:60
  type      = IPv4
###[ IP ]### 
     version   = 4
     ihl       = None
     tos       = 0x0
     len       = None
     id        = 1
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = None
     src       = 100.100.0.33
     dst       = <snip>
     \options   \
###[ ICMP ]### 
        type      = echo-request
        code      = 0
        chksum    = None
        id        = 0x0
        seq       = 0x0
        unused    = ''

>>> srp1(p)
Begin emission:
WARNING: No broadcast address found for iface eth0

Finished sending 1 packets.
*
Received 1 packets, got 1 answers, remaining 0 packets
<Ether  dst=ae:f4:4a:dd:f9:60 src=aa:aa:aa:aa:aa:aa type=IPv4 |<IP  version=4 ihl=5 tos=0x0 len=28 id=51819 flags= frag=0 ttl=62 proto=icmp chksum=0x4116 src=<snip> dst=100.100.0.33 |<ICMP  type=echo-reply code=0 chksum=0x0 id=0x0 seq=0x0 |>>>

Related resources

No response

[michaelvdnet]

After some diggin, it looks like the kernel never returns the rta_type=RTA_PREFSRC rtmsg_rtattr causing the Output IP to be set to the default 0.0.0.0 address.

Full rt message:

 <rtmsghdr  nlmsg_len=52 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740497776 nlmsg_pid=2153 |<rtmsg  rtm_family=AF_INET rtm_dst_len=0 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_MAIN rtm_protocol=RTPROT_STATIC rtm_scope=RT_SCOPE_UNIVERSE rtm_type=RTN_UNICAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_MAIN |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_GATEWAY rta_data=169.254.1.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>,

Manually patching scapy/arch/linx/rtnetlink.py:898 to the hardcoded IP fixes the issue:

- addr = "0.0.0.0"
+ addr = "100.100.0.14"

>>> conf.route
Network          Netmask          Gateway      Iface  Output IP     Metric
0.0.0.0          0.0.0.0          169.254.1.1  eth0   100.100.1.14  0     
100.100.0.14     255.255.255.255  0.0.0.0      eth0   100.100.0.14  0     
100.100.127.255  255.255.255.255  0.0.0.0      eth0   100.100.0.14  0     
127.0.0.0        255.0.0.0        0.0.0.0      lo     127.0.0.1     0     
127.0.0.1        255.255.255.255  0.0.0.0      lo     127.0.0.1     0     
127.255.255.255  255.255.255.255  0.0.0.0      lo     127.0.0.1     0     
169.254.1.1      255.255.255.255  0.0.0.0      eth0   100.100.1.14  0     
>>> conf.route.route()
('eth0', '100.100.1.14', '169.254.1.1')
>>> p = Ether()/IP(dst="<snip>")/ICMP()
>>> p.show()
###[ Ethernet ]###
  dst       = None
  src       = 26:5d:4b:dd:17:a0
  type      = IPv4
###[ IP ]###
     version   = 4
     ihl       = None
     tos       = 0x0
     len       = None
     id        = 1
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = None
     src       = 100.100.0.14
     dst       = <snip>
     \options   \
###[ ICMP ]###
        type      = echo-request
        code      = 0
        chksum    = None
        id        = 0x0
        seq       = 0x0
        unused    = b''

>>> srp1(p)
Begin emission
WARNING: No broadcast address found for iface eth0

.
Finished sending 1 packets
*
Received 2 packets, got 1 answers, remaining 0 packets
<Ether  dst=26:5d:4b:dd:17:a0 src=aa:aa:aa:aa:aa:aa type=IPv4 |<IP  version=4 ihl=5 tos=0x0 len=28 id=17951 flags= frag=0 ttl=62 proto=icmp chksum=0xc575 src=<snip> dst=100.100.0.14 |<ICMP  type=echo-reply code=0 chksum=0x0 id=0x0 seq=0x0 unused=b'' |>>>

Sadly today is the first day I've heard about rtmlink so can't be of much help.
Not sure if this is a problem in the Container Apps kernel or something that can be fixed on library level.
If it's a kernal problem, it might be nice to have some extra checks or workarounds available

[gpotter2]

Could you share the full output of

from scapy.arch.linux.rtnetlink import _read_routes
_read_routes(socket.AF_INET)

and your ip route show table all result?

Thanks

[michaelvdnet]

Ofcourse!

>>> pprint.pp(_read_routes(socket.AF_INET))
[<rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_HOST rtm_type=RTN_LOCAL rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=100.100.0.14 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=100.100.0.14 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>,
 <rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_LINK rtm_type=RTN_BROADCAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=100.100.127.255 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=100.100.0.14 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>,
 <rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=8 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_HOST rtm_type=RTN_LOCAL rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=127.0.0.0 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=127.0.0.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=1 |>] |>>,
 <rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_HOST rtm_type=RTN_LOCAL rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=127.0.0.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=127.0.0.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=1 |>] |>>,
 <rtmsghdr  nlmsg_len=60 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_LOCAL rtm_protocol=RTPROT_KERNEL rtm_scope=RT_SCOPE_LINK rtm_type=RTN_BROADCAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_LOCAL |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=127.255.255.255 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_PREFSRC rta_data=127.0.0.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=1 |>] |>>,
 <rtmsghdr  nlmsg_len=52 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=0 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_MAIN rtm_protocol=RTPROT_STATIC rtm_scope=RT_SCOPE_UNIVERSE rtm_type=RTN_UNICAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_MAIN |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_GATEWAY rta_data=169.254.1.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>,
 <rtmsghdr  nlmsg_len=52 nlmsg_type=RTM_NEWROUTE nlmsg_flags=NLM_F_MULTI+NLM_F_DUMP_FILTERED nlmsg_seq=1740559641 nlmsg_pid=63617 |<rtmsg  rtm_family=AF_INET rtm_dst_len=32 rtm_src_len=0 rtm_tos=0 rtm_table=RT_TABLE_MAIN rtm_protocol=RTPROT_STATIC rtm_scope=RT_SCOPE_LINK rtm_type=RTN_UNICAST rtm_flags= data=[<rtmsg_rtattr  rta_len=8 rta_type=RTA_TABLE rta_data=RT_TABLE_MAIN |>, <rtmsg_rtattr  rta_len=8 rta_type=RTA_DST rta_data=169.254.1.1 |>, <rtmsg_rtattr  rta_len=8 rta_type=RTS_OIF rta_data=39 |>] |>>]

/app # ip route show table all
default via 169.254.1.1 dev eth0 
169.254.1.1 dev eth0 scope link 
local 100.100.0.14 dev eth0 table local scope host  src 100.100.0.14 
broadcast 100.100.127.255 dev eth0 table local scope link  src 100.100.0.14 
local 127.0.0.0/8 dev lo table local scope host  src 127.0.0.1 
local 127.0.0.1 dev lo table local scope host  src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local scope link  src 127.0.0.1 
fe80::/64 dev eth0  metric 256 
local ::1 dev lo table local  metric 0 
local fe80::245d:4bff:fedd:17a0 dev eth0 table local  metric 0 
multicast ff00::/8 dev eth0 table local  metric 256

Looks like src is missing on the default route here too. I'm not sure if that's something on the container image level or at the kernel level which I can't control.

I'll test with the debian base image and see if that fixes it Did not change anything.

[michaelvdnet]

A few stackoverflow resources tell me the src property is more of a hint and not a required property (connectivity still works). Would it be a solution to fall back to iface IP when none is found in the route table?

[michaelvdnet]

Looks like the issue also arises on my macbook running rancher desktop, same container image:

root@4297ea22def2:/# ip route show table all
default via 10.4.0.1 dev eth0 
10.4.0.0/24 dev eth0 proto kernel scope link src 10.4.0.18 
local 10.4.0.18 dev eth0 table local proto kernel scope host src 10.4.0.18 
broadcast 10.4.0.255 dev eth0 table local proto kernel scope link src 10.4.0.18 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
fe80::/64 dev eth0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::ecab:afff:feec:1341 dev eth0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium

>>> conf.route
Network          Netmask          Gateway   Iface  Output IP  Metric
0.0.0.0          0.0.0.0          10.4.0.1  eth0   0.0.0.0    0     
10.4.0.0         255.255.255.0    0.0.0.0   eth0   10.4.0.18  0     
10.4.0.18        255.255.255.255  0.0.0.0   eth0   10.4.0.18  0     
10.4.0.255       255.255.255.255  0.0.0.0   eth0   10.4.0.18  0     
127.0.0.0        255.0.0.0        0.0.0.0   lo     127.0.0.1  0     
127.0.0.1        255.255.255.255  0.0.0.0   lo     127.0.0.1  0     
127.255.255.255  255.255.255.255  0.0.0.0   lo     127.0.0.1  0     
224.0.0.0        240.0.0.0        0.0.0.0   eth0   10.4.0.18  250   
>>> conf.route.route()
('lo', '0.0.0.0', '0.0.0.0')
>>>
>>> p = Ether()/IP(dst="1.1.1.1")/ICMP()
>>> p.show()
###[ Ethernet ]###
  dst       = None
  src       = ee:ab:af:ec:13:41
  type      = IPv4
###[ IP ]###
     version   = 4
     ihl       = None
     tos       = 0x0
     len       = None
     id        = 1
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = None
     src       = 10.4.0.18
     dst       = 1.1.1.1
     \options   \
###[ ICMP ]###
        type      = echo-request
        code      = 0
        chksum    = None
        id        = 0x0
        seq       = 0x0
        unused    = b''
>>> srp1(p, verbose=False, timeout=5)
>>> 

Weird that it does set the src address correctly on the packet

[gpotter2]

Thanks. I'm not sure that I understand the issue. What you provided in the last comment (including the fact that the output IP is 0.0.0.0 in the routing table) is perfectly expected. You'll notice that that's not what's kept in the final packet.

Going back to your initial question, the fact that you're using APIPA adresses sounds to me like your network configuration is already broken. I'd investigate further on that end.

You should have the same result with ip route get <an ip> and conf.route.route("<the IP>"). If that's not the case then there could be a bug, but until then I'm a bit unsure of what your issue is.

[michaelvdnet]

It does get the correct interface, but not the correct src ip:

# ip route get 1.1.1.1
1.1.1.1 via 169.254.1.1 dev eth0 src 100.100.0.8 uid 0 
    cache 
# python3
>>> Python 3.11.11 (main, Feb 25 2025, 05:21:12) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
from scapy.all import *
>>> conf.route.route("1.1.1.1")
('eth0', '0.0.0.0', '169.254.1.1')

I've traced it down a bit and:

1. the getmacbyip function uses the ARP packet
2. The ARP packet uses the SourceIPField as psrc
3. The SourceIPField uses pkt.route[1] or conf.route.route()[1] which is 0.0.0.0
4. An ARP request like Who has 1.1.1.1. Tell 0.0.0.0 gets sent out
5. We don't get a dst MAC so nobody responds to our packet

The IP packet also uses the SourceIPField so this not only breaks l2 but also l3.

[michaelvdnet]

Stepping into getmacbyip("1.1.1.1") with pdb results in:

(Pdb) where
  <stdin>(1)<module>()
  <stdin>(3)trace()
  /usr/local/lib/python3.11/site-packages/scapy/layers/l2.py(175)getmacbyip()
-> res = srp1(Ether(dst=ETHER_BROADCAST) / ARP(op="who-has", pdst=ip),
> /usr/local/lib/python3.11/site-packages/scapy/sendrecv.py(769)srp1()
-> ans, _ = srp(*args, **kargs)
(Pdb) !args[0].show()
###[ Ethernet ]###
  dst       = ff:ff:ff:ff:ff:ff
  src       = 6e:0e:fc:91:06:6a
  type      = ARP
###[ ARP ]###
     hwtype    = Ethernet (10Mb)
     ptype     = IPv4
     hwlen     = None
     plen      = None
     op        = who-has
     hwsrc     = 6e:0e:fc:91:06:6a
     psrc      = 0.0.0.0
     hwdst     = 00:00:00:00:00:00
     pdst      = 169.254.1.1

Setting psrc manually on ARP fixes it:

>>> srp1(Ether(dst="ff:ff:ff:ff:ff:ff")/ARP(op="who-has", pdst="169.254.1.1"), timeout=4)
Begin emission

Finished sending 1 packets
......................
Received 22 packets, got 0 answers, remaining 1 packets
>>> srp1(Ether(dst="ff:ff:ff:ff:ff:ff")/ARP(op="who-has", psrc="100.100.0.8", pdst="169.254.1.1"), timeout=4)

Finished sending 1 packets
Begin emission
......*
Received 7 packets, got 1 answers, remaining 0 packets
<Ether  dst=6e:0e:fc:91:06:6a src=aa:aa:aa:aa:aa:aa type=ARP |<ARP  hwtype=Ethernet (10Mb) ptype=IPv4 hwlen=6 plen=4 op=is-at hwsrc=aa:aa:aa:aa:aa:aa psrc=169.254.1.1 hwdst=6e:0e:fc:91:06:6a pdst=100.100.0.8 |>>

[gpotter2]

Hi & very sorry for the huge delay.
I believe that #4753 fixes the routes issue you've been experiencing.

If that's not the case, feel free to reopen.