[BUG] UDP/IPv4 transmission does not ARP-query gateways

[nwf]

Describe the bug

FreeRTOS-Plus-TCP (tested with v4.2.0, but v4.3, which I'll cite herein, looks from inspection to behave identically here) fails to trigger an ARP request for its gateway when sending UDPv4 datagrams to off-segment addresses. As a result, the system is effectively unable to send such UDP packets until, AFAICT, either

• the application manually requests an ARP query (for the gateway?), or
• the gateway sends an ARP request for the system, which will suffice to populate the gateway's entry in the ARP cache.

Specifically, assuming I understand the packet traces and debug logs below and am reading the code correctly, the UDP/IPv4 egress path of course queries the ARP table and that has handling for gateways, but when a gateway is needed, the UDPv4 code, despite the comment naming the correct variable, queries for the same address as the gateway handling did and so takes the wrong path rather than the one that generates an ARP query.

Incidentally, there's code in the IP packet ingress handling path to refresh the ARP table or trigger an ARP request, but... it's disabled for UDP packets, which DHCP uses. Perhaps it should be disabled for {multi,broad}cast packets instead, and allow unicast (UDP and otherwise) packets to trigger it? This would have masked the above issue in my case, and in many common cases, because often the DHCP server and the default gateway are one and the same. I'm not sure if that's an argument for or against adopting this behavior!

Target

• Development board: CHERIoT Sonata
• Instruction Set Architecture: CHERIoT (a RV32E derivative)
• IDE and version: n/a
• Toolchain and version: CHERIoT LLVM 20

The curious are welcome to see the CHERIoT network stack interfaces to FreeRTOS-Plus-TCP, but I do not believe that our interface code differs significantly for the purposes of this bug from any other application.

Host

• Host OS: Linux
• Version: Debian Trixie

To Reproduce
Run a FreeRTOS-Plus-TCP application that performs DHCP and attempts to send a UDP packet (perhaps specifically not DNS).

Expected behavior
I expect FreeRTOS-Plus-TCP to either

• initiate an ARP request when sending a UDP datagram to an address it does not know how to reach, buffering the UDP packet for transmission upon resolution, or
• indicate failure to the application's UDP sendto.

Wireshark logs

Here's an example application running on Sonata and trying to do DHCP followed by SNTP. We can see that there's a while a startup where the system won't generate UDP packets destined for off-segment addresses (via the gateway).

• The system initializes and performs DHCP successfully:

  02:14:55.803009 3a:30:25:24:fe:7a > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 300: (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 286)
      0.0.0.0.68 > 255.255.255.255.67: [no cksum] BOOTP/DHCP, Request from 3a:30:25:24:fe:7a, length 258, xid 0xcaabede, Flags [Broadcast] (0x8000)
  02:14:55.805299 b4:fb:e4:20:ca:6b > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0xc0, ttl 64, id 1029, offset 0, flags [none], proto UDP (17), length 328)
      172.29.7.1.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0xcaabede, Flags [Broadcast] (0x8000)
  02:14:55.817799 3a:30:25:24:fe:7a > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 307: (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 293)
      0.0.0.0.68 > 255.255.255.255.67: [no cksum] BOOTP/DHCP, Request from 3a:30:25:24:fe:7a, length 265, xid 0xcaabede, Flags [Broadcast] (0x8000)
  02:14:56.269416 b4:fb:e4:20:ca:6b > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 367: (tos 0xc0, ttl 64, id 1034, offset 0, flags [none], proto UDP (17), length 353)
      172.29.7.1.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 325, xid 0xcaabede, Flags [Broadcast] (0x8000)
  

  The system logs this startup and DHCP state machine traversal thus:

  FreeRTOS_AddEndPoint: MAC: fe-7a IPv4: c0a801f8ip
  ip_thread_start
  ip_thread_entry starting, thread ID is 0x1
  prvIPTask started
  vIPSetDHCP_RATimerEnableState: Off
  prvCloseDHCPSocket[fe-7a]: closed, user count 0
  vDHCPProcessEndPoint: enter 0
  DHCP-socket[fe-7a]: DHCP Socket Create
  prvCreateDHCPSocket[fe-7a]: open, user count 1
  prvInitialiseDHCP: start after 25 ticks
  vDHCP_RATimerReload: 25
  vDHCPProcessEndPoint: exit 1
  vDHCPProcessEndPoint: enter 1
  vDHCPProcess: discover
  vDHCPProcessEndPoint: exit 2
  vDHCPProcessEndPoint: enter 2
  vDHCPProcess: discover
  vDHCPProcess: timeout 1000 ticks
  vDHCPProcess: offer ac1d0761ip for MAC address fe-7a
  vDHCPProcess: reply ac1d0761ip
  vDHCPProcessEndPoint: exit 3
  vDHCPProcessEndPoint: enter 3
  vDHCPProcess: offer ac1d0761ip for MAC address fe-7a
  vDHCPProcess: acked ac1d0761ip
  prvCloseDHCPSocket[fe-7a]: closed, user count 0
  vDHCP_RATimerReload: 4320000
  vDHCPProcessEndPoint: exit 5
  

• Immediately thereafter, the system does a gratuitous ARP announcement as duplicate check:

  02:14:56.282088 3a:30:25:24:fe:7a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 172.29.7.97 tell 172.29.7.97, length 46
  

  No response is received, as no duplicate exists on the network.

• The application now attempts to send a UDP packet (SNTP, specifically, using coreSNTP). No packets, UDP or ARP or otherwise, are emitted and the system logs the following:

  SNTP Info: Sending serialized SNTP request packet to the server: Addr=2214620366, Port=123
  FreeRTOS_FindEndPointOnNetMask[4]: No match for ce6c0084ip
  ARP ce6c0084ip miss using ac1d0701ip
  FreeRTOS_FindEndPointOnNetMask[11]: No match for ce6c0084ip 
  

  Note in particular that the 2nd FindEndPointOnNetMask call is for the same, off-segment address as the first! (Some additional instrumentation shows that the sendto has returned the expected 48, indicating success, which is a bit rude.) coreSNTP eventually times out and reports failure to the application, which goes to sleep (before retrying).

• While the application is asleep, the gateway sends an ARP request to refresh its cache entry for the system, and the system responds:

  02:15:01.393897 b4:fb:e4:20:ca:6b > 3a:30:25:24:fe:7a, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 172.29.7.97 tell 172.29.7.1, length 46
  02:15:01.398879 3a:30:25:24:fe:7a > b4:fb:e4:20:ca:6b, ethertype ARP (0x0806), length 64: Ethernet (len 6), IPv4 (len 4), Reply 172.29.7.97 is-at 3a:30:25:24:fe:7a, length 50
  

  The system logs this and the insertion into its ARP cache:

  pxEasyFit: ARP ac1d0701ip -> ac1d0761ip
  ipARP_REQUEST from ac1d0701ip to ac1d0761ip end-point ac1d0761ip
  

  This happens only because the gateway and the DHCP server are one and the same. Were the gateway a different node, it might never issue an ARP request for the system.

• The application wakes up and retries SNTP. At this point, there is a hit in the ARP cache and a packet is sent:

  02:15:06.321627 3a:30:25:24:fe:7a > b4:fb:e4:20:ca:6b, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 76)
      172.29.7.97.15848 > 149.56.19.163.123: [no cksum] NTPv4, Client, length 48
  

  The system logs:

  SNTP Info: Sending serialized SNTP request packet to the server: Addr=2214620366, Port=123
  FreeRTOS_FindEndPointOnNetMask[4]: No match for ce6c0084ip
  ARP ce6c0084ip hit using ac1d0701ip
  

[jasonpcarroll]

Hi @nwf,

I will reach out to someone more familiar with FreeRTOS-Plus-TCP, but what you've wrote looks correct from my understanding. Would you want to contribute and open a PR once you have or if you have fixed this on your end?

Best,
Jason Carroll

[tony-josi-aws]

@nwf

Thanks for creating the detailed bug report. Great to see FreeRTOS+TCP used in CHERIoT Sonata.

but when a gateway is needed, the UDPv4 code, despite the comment naming the correct variable, queries for the same address as the gateway handling did and so takes the wrong path rather than the one that generates an ARP query.

This seems like a bug to me; does the following patch help?

diff --git a/source/FreeRTOS_UDP_IPv4.c b/source/FreeRTOS_UDP_IPv4.c
index 2fceed8a..d5fde024 100644
--- a/source/FreeRTOS_UDP_IPv4.c
+++ b/source/FreeRTOS_UDP_IPv4.c
@@ -273,8 +273,7 @@ void vProcessGeneratedUDPPacket_IPv4( NetworkBufferDescriptor_t * const pxNetwor
 
             /* 'ulIPAddress' might have become the address of the Gateway.
              * Find the route again. */
-
-            pxNetworkBuffer->pxEndPoint = FreeRTOS_FindEndPointOnNetMask( pxNetworkBuffer->xIPAddress.ulIP_IPv4 );
+            pxNetworkBuffer->pxEndPoint = FreeRTOS_FindEndPointOnNetMask( ulIPAddress );
 
             if( pxNetworkBuffer->pxEndPoint == NULL )
             {

Perhaps it should be disabled for {multi,broad}cast packets instead, and allow unicast (UDP and otherwise) packets to trigger it?

During ingress, some DHCP servers can send unicast OFFER messages in response to the DISCOVER messages, especially if the client has connected to that server before or the BOOTP flag has the unicast bit set.

Example (OFFER is unicast):

Some additional instrumentation shows that the sendto has returned the expected 48, indicating success, which is a bit rude

The reason for a successful return code despite a packet miss is that FreeRTOS_sendto does the ARP lookup and the actual send inside FreeRTOS+TCP's IP-Task rather than the task from which FreeRTOS_sendto is called, so that the calls to FreeRTOS_sendto don't block unless network buffers are not available or the IP-Task's queue is full. Since it's UDP and the reliability isn't guaranteed by default, the expectation is that if we are able to queue the packet to IP-Task, we should be able to send it unless we have a problem with ARP, which happens to be the case here.

[nwf]

This seems like a bug to me; does the following patch help?

I believe it should, yes. Will test to be sure.

Perhaps it should be disabled for {multi,broad}cast packets instead, and allow unicast (UDP and otherwise) packets to trigger it?

During ingress, some DHCP servers can send unicast OFFER messages in response to the DISCOVER messages, especially if the client has connected to that server before or the BOOTP flag has the unicast bit set.

That's true but not, I think, directly in reply to the question I was (meaning to be) asking. Trying again:

1. Should FreeRTOS-Plus-TCP (be allowed to) use ingress unicast UDP/IP datagrams to populate its ARP table?

2. If so, should its DHCP client default to requesting unicast replies (and fall back to broadcast)? At present, AFAIR, it behaves the other way around, with the first requests requesting broadcast replies.

The reason for a successful return code despite a packet miss is that FreeRTOS_sendto does the ARP lookup and the actual send inside FreeRTOS+TCP's IP-Task rather than the task from which FreeRTOS_sendto is called, so that the calls to FreeRTOS_sendto don't block unless network buffers are not available or the IP-Task's queue is full. Since it's UDP and the reliability isn't guaranteed by default, the expectation is that if we are able to queue the packet to IP-Task, we should be able to send it unless we have a problem with ARP, which happens to be the case here.

Understood. Thank you.

[nwf]

This seems like a bug to me; does the following patch help?

I believe it should, yes. Will test to be sure.

Hrm. Things are decidedly better, to be sure, but the actual UDP datagram never leaves the device, still. The device logs:

Network stack wrapper: Sending 0x30-byte UDP packet
Buffer management: Allocated 0x60 byte buffer: 0x10f248 (v:1 0x10f248-0x10f2b8 l:0x70 o:0x0 p: G RWcgm- -- ---), descriptor: 0x10f1e0 (v:1 0x10f1e0-0x10f240 l:0x60 o:0x0 p: G RWcgm- -- ---)
FreeRTOS_FindEndPointOnNetMask[4]: No match for 1785a8f7ip
ARP 1785a8f7ip miss using ac1d0701ip
Buffer management: Freeing descriptor: 0x10f1e0 (v:1 0x10f1e0-0x10f240 l:0x60 o:0x0 p: G RWcgm- -- ---) and buffer 0x10f248 (v:1 0x10f248-0x10f2b8 l:0x70 o:0x0 p: G RWcgm- -- ---)
Buffer management: Allocated 0x48 byte buffer: 0x10f328 (v:1 0x10f328-0x10f380 l:0x58 o:0x0 p: G RWcgm- -- ---), descriptor: 0x10f2c0 (v:1 0x10f2c0-0x10f320 l:0x60 o:0x0 p: G RWcgm- -- ---)
pxEasyFit: ARP ac1d0761ip -> ac1d0701ip
Network stack wrapper: Blocking call returned 48
Network stack wrapper: Send returned 48
NTP Client: STR OK @ 0x21c
ipARP_REPLY from ac1d0701ip to ac1d0761ip end-point ac1d0761ip
Buffer management: Freeing descriptor: 0x10f2c0 (v:1 0x10f2c0-0x10f320 l:0x60 o:0x0 p: G RWcgm- -- ---) and buffer 0x10f328 (v:1 0x10f328-0x10f380 l:0x58 o:0x0 p: G RWcgm- -- ---)

(The STR OK line is after coreSNTP has completed its sendto.) The Buffer management logs indicate that the stack has dropped the initial request (descriptor: 0x10f1e0 / buffer: 0x10f248) before allocating one for the ARP reply (d: 0x10f2c0 / b: 0x10f328). (ETA: the chatter that looks like (v:1 0x...-0x... ... p: G RWcgm- -- ----) is a rendering of CHERI capabilities, which are pointers that carry, among other things, bounds and permissions. It can be glossed over for present purposes, but if curiosity strikes, I'm happy to explain in more detail.)

And indeed, this is borne out on the wire:

13:55:51.483614 b4:fb:e4:20:ca:6b > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 367: (tos 0xc0, ttl 64, id 8428, offset 0, flags [none], proto UDP (17), length 353)                                                                  
13:55:51.496740 3a:30:25:24:fe:7a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 172.29.7.97 tell 172.29.7.97, length 46                                                            
13:55:51.542705 3a:30:25:24:fe:7a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 172.29.7.1 tell 172.29.7.97, length 46
13:55:51.542878 b4:fb:e4:20:ca:6b > 3a:30:25:24:fe:7a, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply 172.29.7.1 is-at b4:fb:e4:20:ca:6b, length 46                                                                
13:55:56.588608 b4:fb:e4:20:ca:6b > 3a:30:25:24:fe:7a, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 172.29.7.97 tell 172.29.7.1, length 46
13:55:56.593588 3a:30:25:24:fe:7a > b4:fb:e4:20:ca:6b, ethertype ARP (0x0806), length 64: Ethernet (len 6), IPv4 (len 4), Reply 172.29.7.97 is-at 3a:30:25:24:fe:7a, length 50                                                               
13:56:01.534143 3a:30:25:24:fe:7a > b4:fb:e4:20:ca:6b, ethertype IPv4 (0x0800), length 90: (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 76) 
    172.29.7.97.63342 > 23.133.168.247.123: [no cksum] NTPv4, Client, length 48

The device finishes DHCP around 13:55:51.49, it does the first sendto around 13:55:51.54, resulting in an ARP exchange, but does not actually send a SNTP UDP datagram until after the application times out and tries invoking coreSNTP again, around 13:56:01.53.

[tony-josi-aws]

Hrm. Things are decidedly better, to be sure

Thanks for verifying the patch and sharing the feedback.

but the actual UDP datagram never leaves the device, still.

In the case of an ARP miss while sending a UDP packet, the actual UDP datagram is not sent from the device because the IP task (which sends out the packet) is not intended to do anything that is blocking, so it cannot wait for the result of an ARP request and then deliver the UDP packet. Due to that reason, we don't allocate a new network buffer for sending the ARP request; instead, we use the initial UDP packet's network buffer and convert it to an ARP request packet and send it over the network, essentially losing that initial UDP packet.

If you wish to avoid the scenario of losing your first UDP packet to the above-mentioned scenario, you may try using the xARPWaitResolution API, which can be called from the user application before the actual UDP transmission. The API looks up a given IP address (ulIPAddress) in the ARP cache; if it's a miss, then it sends out an ARP request and waits for a maximum timeout (uxTicksToWait) for the ARP reply (ie, till we get an ARP entry for that address in the cache), and if it doesn't get a reply, it retries for ipconfigMAX_ARP_RETRANSMISSIONS times. This API also takes care of the scenario where the given IP address (ulIPAddress) is outside of the subnet and we need to pass the network packet to the gateway.

1. Should FreeRTOS-Plus-TCP (be allowed to) use ingress unicast UDP/IP datagrams to populate its ARP table?
2. If so, should its DHCP client default to requesting unicast replies (and fall back to broadcast)? At present, AFAIR, it behaves the other way around, with the first requests requesting broadcast replies.

In FreeRTOS+TCP we populate the ARP cache table with a new entry from an ingress packet only after we have confirmed its address by sending out an ARP request. There is a slight difference in the case of UDP packets, where we also make sure that there is a corresponding socket bound to the same port number with which the incoming/ingress UDP packet is destined. [refer]

The reason for this behavior is to not flood the ARP cache with nonrelevant entries.

At present, AFAIR, it behaves the other way around, with the first requests requesting broadcast replies.

That's incorrect; currently FreeRTOS+TCP sends the initial DHCP discover message with unicast replies requested. [refer]

This broadcast bit is toggled for the subsequent retries if the previous one fails.

[nwf]

At present, AFAIR, it behaves the other way around, with the first requests requesting broadcast replies.

That's incorrect; currently FreeRTOS+TCP sends the initial DHCP discover message with unicast replies requested. [refer]

Fascinating. Thank you for making me look more carefully at the early system logs and not just tcpdump. The first packet my network sees is the second DHCP discover transmitted, but indeed, FreeRTOS+TCP sure tries to send one without the broadcast flag first.

[tony-josi-aws]

@nwf

Which version of FreeRTOS+TCP are you using in your project? There was a recent PR [and follow-up PR] that got merged to the mainline, which resolved an issue with the TCP/IP stack dropping unicast packets (including DHCP packets) before the endpoint or the network is up if the ipconfigETHERNET_DRIVER_FILTERS_PACKETS is disabled in FreeRTOSIPConfig.h.

If you are using an older version, that might explain why DHCP had to do a second retry (with broadcast flag enabled) in order to succeed.

[nwf]

Oh, thanks for the heads up. We're using v4.2.0, so probably should catch up (especially once the patch in #1244 (comment) lands). That said, it looks like the first DHCP Discover packet is making it all the way down to the ethernet driver and not getting sent by the hardware, rather than the reply not making it back up the stack.

[tony-josi-aws]

@nwf

Since the PR is merged to main, I'm closing this issue. Please feel free to reopen or open another issue if you feel otherwise.