dump1090-fa versus Piaware

[rumicagit]

Hi all,
I've Piaware running successful for almost one year.
I found and report around 800/900 planes a day.

This week I was snooping around dump1090-fa logs and I found that in the logs I've around 1500/1600 distinct planes discovered.
It seems that nor all the planes are reported to Piaware and I don't know why...
It seems that the difference is only during day. Analyzing planes detected by dump1090-fa and Piaware the night values are similar. The difference start to increase around 10A.M.
I thought that could be lack os capacity of may Pi4 (2GB) but memory and CPU are quite low.

Could be some timers? Or lack of capacity to report on "busy hours"? Anyone with the same problem?

Regards,

RC

PiAware master process (piaware) is running with pid 625.
PiAware ADS-B client (faup1090) is running with pid 1084.
PiAware ADS-B UAT client (faup978) is not running (disabled by configuration settings)
PiAware mlat client (fa-mlat-client) is running with pid 1072.
Local ADS-B receiver (dump1090-fa) is running with pid 983.

dump1090-fa (pid 983) is listening for ES connections on port 30005.
faup1090 is connected to the ADS-B receiver.
piaware is connected to FlightAware.

dump1090 is producing data on localhost:30005.

allow-auto-updates yes # value set at /boot/piaware-config.txt:67
allow-manual-updates yes # value set at /boot/piaware-config.txt:71
allow-mlat yes # value set at /boot/piaware-config.txt:75
allow-modeac yes # value set at /boot/piaware-config.txt:79
image-type piaware # value set at /usr/share/piaware-support/piaware-image-config.txt:5
manage-config yes # value set at /usr/share/piaware-support/piaware-image-config.txt:4
network-config-style buster # value set at /usr/share/piaware-support/piaware-image-config.txt:6
rtlsdr-gain max # value set at /boot/piaware-config.txt:60
wired-network yes # value set at /boot/piaware-config.txt:27
wireless-network yes # value set at /boot/piaware-config.txt:33
wireless-password # value set at /boot/piaware-config.txt:49
wireless-ssid "xxxxx" # value set at /boot/piaware-config.txt:48

[mutability]

How are you measuring the number of aircraft in each case?

[rumicagit]

Hello,
In /run/dump1090-fa you have all the planes found by dump1090-fa (aircraft.json). You can copy the file every minute to a temp directory and at the end of the day, found all the received messages. Easily you can find the number of distinct aircarfts (nor repeated).
Rumica

[rumicagit]

In /var/log/fr24feed/fr24feed.log file you can also see lots of ignored aircrafts... but I don't know the meaning.

2022-09-12 18:54:10 | 22-09-12 18:54:10.778 [D][:] sent 1 of 49 aircraft, 48 ignored, 0 outdated
2022-09-12 18:54:16 | [feed][i]sent 2,47 AC
2022-09-12 18:54:16 | 22-09-12 18:54:16.348 [D][:] sent 2 of 49 aircraft, 47 ignored, 0 outdated
2022-09-12 18:54:21 | [feed][i]sent 1,45 AC
2022-09-12 18:54:21 | 22-09-12 18:54:21.748 [D][:] sent 1 of 46 aircraft, 45 ignored, 0 outdated
2022-09-12 18:54:27 | [feed][i]sent 1,45 AC
2022-09-12 18:54:27 | 22-09-12 18:54:27.378 [D][:] sent 1 of 46 aircraft, 45 ignored, 0 outdated
2022-09-12 18:54:31 | [mlat][i]Pinging the server
2022-09-12 18:54:31 | [mlat][i]Stats 9914779/587
2022-09-12 18:54:32 | [feed][i]sent 1,40 AC
2022-09-12 18:54:32 | 22-09-12 18:54:32.948 [D][:] sent 1 of 41 aircraft, 40 ignored, 0 outdated
2022-09-12 18:54:38 | [feed][i]sent 1,44 AC
2022-09-12 18:54:38 | 22-09-12 18:54:38.568 [D][:] sent 1 of 45 aircraft, 44 ignored, 0 outdated
2022-09-12 18:54:43 | [feed][i]sent 2,42 AC
2022-09-12 18:54:43 | 22-09-12 18:54:43.978 [D][:] sent 2 of 44 aircraft, 42 ignored, 0 outdated
2022-09-12 18:54:49 | [feed][i]sent 1,46 AC
2022-09-12 18:54:49 | 22-09-12 18:54:49.388 [D][:] sent 1 of 47 aircraft, 46 ignored, 0 outdated

[mutability]

I'm not sure how fr24feed logging is relevant to piaware. What are you comparing your aircraft.json numbers to?

[rumicagit]

I'm comparing the aircraft.json with piaware stats page:
https://flightaware.com/adsb/stats/user/rumica#stats-160874
As you can see I've around 800/900 planes reported, but in dump1090-fa I've found distint planes (not repeated) the following values of last days:
1602 2022-09-09
1725 2022-09-10
1692 2022-09-11

Regarding fr24feed: I've fr24feed also installed at same time. The behaviour is the same because it's based also in dump1090-fa.

Regards.

[bovine]

Stats shown on the website reflect numbers based on further stages of processing and validation done by our servers. The numbers will sometimes be slightly lower than the raw counts measured on the device after these layers of processing. (For example, some messages may be missing lat/lon so the server will not count those.)

[rumicagit]

Hi bovine,
Thanks for your feedback.
I understand your explanation, but as you said "slightly lower". My calculations points to around 50% missing.
Regards.

[mutability]

It'll be specific to the pattern of traffic you see locally.

There's not much I can usefully say about why your site sees a particular pattern of data, especially without seeing the exact implementation of how you're counting aircraft. piaware's implementation of what is uploaded lives here: https://github.com/flightaware/dump1090/blob/849a3b73299b4f56620ab16a6b62d88e17f35608/net_io.c#L2606 .. you could compare your implementation to that to look for differences. In particular: you'd need to exclude aircraft with no "useful" information available -- if you hear just a few DF11s and no other data, then that will show up in aircraft.json but won't be uploaded by piaware -- and you'd need to ensure you're using the same error-correction settings that faup1090 uses.