Checksum error and then a string of unknown headers

[gos-h]

I have been experiencing an elusive problem recently that I am looking for advice on. I am connecting to a u-blox ZED-F9P receiver over USB with PyGPSClient. I am sending the default NMEA messages, as well as RXM-RAWX and RXM-SFRBX messages with a 10Hz observation rate and 1Hz NAV rate.

When I first connect, everything seems alright. However, at some point I will start getting errors where one message will fail the checksum, followed by a string of messages with unknown headers (I'm guessing the software is parsing the incoming bytes wrong and gets out of sync somehow). This is creating big gaps in my data. I have been having difficulty troubleshooting this as it is intermittent and usually takes 30+ minutes to start showing up.

Has anyone experienced this before? I am using a baud rate of 115200 and decent quality USB cables. There shouldn't be anything else attempting to access the COM port.

[semuadmin]

Hi @gos-h

I'm unable to reproduce your specific issue but at a measurement interval of 100ms (10Hz), you're almost certainly exceeding the capacity of the ZED-F9P's internal and/or external TX output buffers. If you enable the System Monitor widget you can see how much stress the device's transmission buffers are under. Here's a ZED-F9P (HPG 1.51) running at 10Hz and 115200 baud with a full cohort of NMEA, RXM-RAWX and RXM-SFRBX output messages:

You can see the I/O buffer is completely maxed out after approximately 30 seconds of running, and you're starting to see GNTXT txbuf alloc (buffer overflow) error messages. u-blox receivers often exhibit indeterminate behaviour when the TX buffers are under stress.

You can try the following...

1. Increasing the baud rate still further (F9P can go up to 921600, but bear in mind not all hosts can support that).
2. Reduce the output message cohort - for example, disable (or reduce the output rate of) all the GSV and GSA messages.
3. Disable NMEA output and enable minimal UBX instead (e.g. NAV-POS, NAV-SAT, NAV-DOP).
4. Increasing the measurement interval to, say, 250ms (4 Hz).

... but in general terms there may be little point in running RAW output at a rate higher than 1 Hz as most GNSS constellations simply don't output raw data that quickly. There are numerous posts on this in the u-blox support forum e.g. https://portal.u-blox.com/s/question/0D52p00008HKCS1CAP/how-to-get-10hz-refresh-rate-on-rawx-message-in-ucenter-windows.

What specifically are you looking to achieve here?

[semuadmin]

@gos-h

semuconsulting/pygnssutils#143

As a general observation, if you're looking to record extended GNSS data logs (say, 30 mins or more) and data continuity is key, you may be better off with an unattended (i.e. CLI) rather than attended (i.e. GUI) data logger.

You can always replay the logs in a GUI at a later date if you have a specific need to eyeball the data.

Check out the gnssstreamer CLI utility.

Here, for example, is a CLI gnsstreamer command which I regularly use to a) configure an F9P to output the requisite UBX RAW & NAV message cohort, and b) record the binary data output to a file:

gnssstreamer -C gnssstreamer_zedf9p_raw.conf

The gnssstreamer_zedf9praw.conf text file contains a series of command line arguments:

port=/dev/ttyACM0
baudrate=115200
timeout=3
verbosity=2
format=2
cliinput=5
input=/home/semuadmin/Downloads/pygpsconfig_zedf9p_raw.ubx
clioutput=1
output=/home/semuadmin/Downloads/pygpsdata_zedf9p_rawlog.ubx

The pygpsconfig_zedf9p_raw.ubx binary file contains a series of UBX CFG-VALSET configuration commands previously recorded using PyGPSClient's 'Configuration Command Recorder' facility.

If you wanted to start the above command under Linux as a unattended 'daemon' and just leave it running in the background, you could use something like this (to terminate, use the Linux kill command):

setsid gnssstreamer -C gnssstreamer_zedf9p_raw.conf >gnssstreamer.log 2>&1 < /dev/null &

(I have this set up as an alias in my .zshrc profile)

[semuadmin]

Hi @gos-h Did this address your query at all?

[gos-h]

Hi @semuadmin, thank you very much for this detailed response. I was unaware of the system monitor window so thank you for bringing that to my attention. I am confused as to how baud rate plays into this when I am connected over USB. It seems like the i/o buffer should be cleared easily at the 12 Mbit/s since, with 40 satellites at 10Hz, the RXM-RAWX bandwidth requirement is about 13kbit/s.

It took around 40 minutes of running for the I/O buffer (which had previously been staying around 9% with peaks up to 60% every once in a while) to suddenly max out and I saw the same warnings I had previously encountered.

Interestingly, after this screenshot, the buffer went back down to 9% but I continued seeing these errors sporadically.

[semuadmin]

I am confused as to how baud rate plays into this when I am connected over USB. It seems like the i/o buffer should be cleared easily at the 12 Mbit/s since, with 40 satellites at 10Hz, the RXM-RAWX bandwidth requirement is about 13kbit/s.

... and yet 😉

I'm afraid I can't fully account for the behaviour - you'd need to ask u-blox technical support for that. My guess is that the module implements an internal USB-UART conversion and the bottleneck builds up somewhere there. Unfortunately it's not something I can easily mitigate in code, but in my experience increasing the nominal UART1/2 baud rate to, say, 460800, usually addresses the problem.

[semuadmin]

Hi again @gos-h,

I saw your posting on the u-blox support forum: https://portal.u-blox.com/s/question/0D5bG00001I5d17SAB/zedf9p-io-buffer-filling-up-causing-errors. I bow to @cturvey's expertise on all things u-blox, but I think his remarks more or less tie in with what I've said above.

As well as increasing the baud rate, reducing the output message cohort and reducing the measurement/navigation rate, you could also try turning off any unwanted PyGPSClient "widgets" (e.g. the console, skyview or map), as they each consume resources when visible.

I'd be happy to investigate this further, just to discount the possibility of it being a PyGPSClient long-term stability thing.

To be clear, there are undoubtedly circumstances in which you can 'overload' PyGPSClient's internal processing - Python (and especially tkinter) is never going to be as performant as C/C++, and running a full cohort of NMEA, UBX and RTCM3 messages at, say, 20Hz with every "widget" enabled will definitely grind its gears! But you'd invariably see the effects more or less immediately, rather than after a period of minutes or hours. I don't believe that's what's happening here - in my experience the u-blox I/O buffers max out long before PyGPSClient does.

If you'd like me to investigate further, can you let me have the following information:

1. Details of the specific hardware platform, O/S version and Python/tkinter version you're running on.
2. Details of the specific ZED-F9P board you're using, and its current firmware level (e.g. HPG 1.51).
3. Details of your ZED-F9P configuration - specifically:
   • CFG_RATE_MEAS
   • CFG_RATE_NAV
   • CFG_UART1_BAUDRATE
   • CFG_UART2_BAUDRATE
   • CFG_MSGOUT_NMEA... values for all NMEA messages you have enabled
   • CFG_MSGOUT_UBX_NAV... values for all UBX NAV messages you have enabled
   • CFG_MSGOUT_UBX_RXM... values for all UBX RXM messages you have enabled
   • CFG_MSGOUT_RTCM_3X... values for all RTCM3 messages you have enabled

Tip: you can use the ubxsave CLI utility (which is installed with PyGPSClient) to dump the entire configuration (this will produce a file named something like ubxconfig-20260709114413.ubx, which you can send to me):

ubxsave -P /dev/ttyACM0 --baudrate 115200 --timeout 3