Remote MQTT + Linux + 4 robots = Huge delays

[KKalem]

if OS == Linux && MQTT Broker NOT localhost && NumRemoteRobots >= 4:
   RemoteRobots are DELAYED

Tested scenarios:

• Windows 10, 11 Desktops, Laptops
  • Remote broker
    • 1,4,100,200 Robots --> NO DELAYS
• Ubuntu 22.04, 24.05 Laptops
  • Remote broker
    • 1 robot --> NO DELAY
    • 4 robot --> 15s DELAY
    • more than 4 robots --> MINUTES of DELAY
  • Local broker
    • 1,4,100 Robots --> NO DELAYS

Test methods:

• SAM robots publishing mock waraps stuff to mqtt.
  • Check remote robot states in Unity
  • Check MQTT Explorer
• Published time into mqtt
  • Used SAM to produce "load"
  • Check Unity remote robot states
  • Check MQTT Explorer

In every case, MQTT Explorer seems to receive everything just fine, all the time, even when Unity is delayed by minutes.
Meaning:

• publications are fine
• Hardware is fine
• Broker is fine

Because if any of these 3 had a problem, MQTT Explorer would not see the data either.

The problem seems to be specifically in the network-->Unity path.
I can not fix this. Problem happens before Unity but after network card, only on Linux.

[NinjaTuna007]

I'm running Unity sim on Ubuntu 24 with a single sam on my end, and MQTT commands to the agent are dropped/delayed. I have turned off the WaraMQTT object in Unity to reduce overhead (if any) and am using the str_json_mqtt_bridge to pass data back and forth.

Insane Ping-pong delays (also two instead of one for some reason):

Commands

When running a command from MQTTExplorer, the feedback on the BT terminal is almost instantaneous with start-task and $abort signals. No delays.

When sending commands from Unity, commands show up quickly on MQTT and are implemented equally quickly on the agent. The "running" and "executing tasks" windows in sim are updated with ~10 seconds delays (high variance). When Unity's WaraMQTT object is activated, these things don't update at all (unless I disconnect and reconnect to the MQTT broker in sim).

Unity Sim shows this:

even when BT terminal shows this at the same time:

Unity Logs with failed mqtt command stuff (happens when you try to send a command

Interesting observation: Unity takes some time (~10 seconds after connecting with WaraMQTT active, ~5 seconds without) to "load up" Tasks Available for the Agent even though they're being published continuously by the vehicle. Before this time, the Tasks Available list appears as an empty dropdown.

Have tested this with @KKalem's windows desktop, performance is v smooth. Issue is definitely linux specific.

Performance deteriorates with time (i.e. delays increase), which makes this feel like a buffer filling up problem. Periodic flushes exist too, making for a very interesting periodic pattern:

agent is in the blue (~10 seconds) --> agent becomes yellow (~5-6 seconds) --> agent is back in the blue, ad infinitum.

Feels like a buffer filling up and then being flushed.

---

TLDR:

• Unity buttonclicks are quick to show up in MQTT
• MQTT Events take a long time to show up in Unity.

[NinjaTuna007]

HiveMQ with TSL is Better (single agent MQTT responsiveness)

• Single agent communicating with HiveMQ is as good as localhost in terms of responsiveness.
• Increasing number to 5 makes it much slower.
• For reference, localhost + 12 agents is 60 ms ping and perfectly functional.

[NinjaTuna007]

Useful Commands to mess with Linux Socket Buffers:

sudo sysctl -w net.core.rmem_default=524288
sudo sysctl -w net.core.rmem_max=2097152

I used rmem_default and max of 2500B (small) and 2500000B (large) to see if this affects performance and reliability of the Unity-MQTT pipeline.

Status: No measurable improvement; if anything, the delays are slightly longer (though still within the std-dev, so can't say if they're statistically significant) in both cases.