Disconnect issue on Linux devices

[Mowglli]

Disconnect issue, when manually powering off client
I have been playing with Jamulus where i have created a local server on a Raspberry Pi 3B and 4 clients on 4 different raspberry pi's. I have created a script that automaticlly connects to the local server when powering on a client. I have noticed that if i cut the power on one of the connected clients, the rest of the clients connected to the server cant communicate with each other for about 30 sec - 1 min. I have also noticed that this does not happen if i manually press the disconnect button before powering off.

Is it possible for me to fix it, so i can be able to just power off a client, and the communication is unharmed. So i dont have to manually disconnect before powering off.
Best Regards Mowglli

[hoffie]

Thanks for reporting this. Just to make sure I understand correctly:

When you provoke an unclean shutdown of one of the Jamulus clients, your other clients stop hearing each other. Can you confirm?

Can you also please add what exact Jamulus versions you are using and where you got them from?
What distribution/version are you running?

That certainly should not happen. I will try to reproduce later. If I can't, we might need a tcpdump from your server to understand what's happening.

[Mowglli]

Hello hoffie,
Yes that is ecaxtly what i mean!
I am using Jamulus Version 3.6.2, i got it from Github and compiled it myself following the "install guide", provided on the side.

Installed on a Raspberry pi 3B with Raspbian GNU/Linux 10 (buster).

I have tried to get you some data from Wireshack, where 10.10.10.158 is my laptop connected to a closed local network the server is hosting on. 10.10.10.189 is the raspberry pi i provoke a power off on.
Hope it is usefull, otherwise i am willing to provide more test if necessary.

(I have the same problem with only server and clients running on Raspberry Pi's. So i dont think it is a "cross-platform" issue)
Jamulus-dump.zip

[hoffie]

I have tried to get you some data from Wireshack, where 10.10.10.158 is my laptop connected to a closed local network the server is hosting on. 10.10.10.189 is the raspberry pi i provoke a power off on.
Hope it is usefull, otherwise i am willing to provide more test if necessary.

Hrm, I just had a quick look and did not see any traffic from/to 10.10.10.189 at all. I assume that this is a dump which was generated on your laptop?

From a first glance, the dump seems to confirm a problem: Initially, audio data both from and to your server can be observed. After some time (No. 8565, 11.2s) the traffic from server to client becomes less and less frequent until it stops completely (21255, 39.8s).

Could you get a dump from your server when that problem appears?
Also, has this been ever working (e.g. in some previous version) or don't you know?

I will try to reproduce later, first on x86_64 and later on a raspberry.

[Mowglli]

My bad.
I've tested the scenario again, with wireshark running on the server. The test is started and after 10 sec i pull out the power plug on the client 10.10.10.189. Wireshark is running 30 seconds past that.
Hope it can help you. Thanks!
Jamulus_dump_server.zip

[hoffie]

I tried to reproduce this on a regular x86_64 machine with a server and two clients and stopping one of the clients abruptly (SIGSTOP). In this case, the server kept sending audio as expected. First, it continued sending to both clients, later it stopped sending to the shutdown client as expected.
Will try on my Raspberry as well.

In the meantime, could you run some further tests?

• Does the issue occur when killing the client using killall -9 Jamulus?
• Does it occur when blackholing the client on the server-side using iptables -I INPUT -s 10.10.10.189 -j DROP?
• Does it issue only occur when cutting power?
• What does your power supply setup look like? I just remember that Raspberries tend to be react sensitively on power changes. Could the power cut of the client influence the power supplied to your server?
• Does the server Raspberry still respond properly? Try running a ping running, observe top using ssh, etc.

[Mowglli]

I have testet with a server and 3 clients.

Does the issue occur when killing the client using killall -9 Jamulus?
When running the command it diconnects the client and doesnt interfer with the communication between the clients still connected at all.

Does it occur when blackholing the client on the server-side using iptables -I INPUT -s 10.10.10.189 -j DROP?
It dosen't happen when backholding the client.

I think the issue only appears when cutting power.
I have tried with a powerbank aswell as with the original powersupply.
On all testing i have been doing, each raspberry have been given a powerbank for themself or a powersupply for themself. With that i mean that on one test all raspberrys were powered with the same type.

Does the server Raspberry still respond properly? Try running a ping running, observe top using ssh, etc.
I have tried running a ping to the server, and then powering of another client. the ping time spikes to 10750 ms, and gradually reduces down to normal.

[hoffie]

I have tried running a ping to the server, and then powering of another client. the ping time spikes to 10750 ms, and gradually reduces down to normal.

I'd look into that. I'm more and more confident that this is not an issue in Jamulus itself, but rather somewhere in the network.
Could it be that a "dieing" Raspberry sends some half-broken signals on its Ethernet cable so that the switch gets confused?

You could

• try a different switch
• try repeating the test when the client is connected via WiFi
• try checking error counters on the server-side (ethtool)
• try checking error counters on the switch-side (if it is a managed switch)

Not sure if this is still an issue, but older switches had limits on how many packets they could process. If those limits exceeded, they would go into a hub-mode which would essentially broadcast all packets to all interfaces. Maybe this mode is triggered by Jamulus (which certainly does send lots of packtes in short time)? This could allow garbled packets to reach your server.

[Mowglli]

I think you're right. I have been running Jamulus only on a local and closed WiFi-hotspot created by the Raspberry Pi running the server. I'ill look more into, what is actually happening on the network-side of things.

Im not planning on using ethernet-cables and switches for the application.
Thanks again

[hoffie]

Ok, cool. Unless you find specific hints that Jamulus behaves strangely, I'd suggest to close this issue. Can you do that if you agree?

[Mowglli]

Hello, yes i will close this issue. I have recorded a video regarding my problem. Are you willing to provide me with a email, so i can send you the link. Or i can alternativly link it here. It is just to better explain how this wierd error acors and the constellation im using your software in