GPS logging capability

[CGurity]

This PR adds the relevant code to support the inclusion of coordinates into the generated PCAP files. There are 3 modes of GPS acquisition/logging:

1. Disabled: the default one, nothing changes from the current use of Rayhunter
2. Fixed coordinates: for use in stationary places, the user has the possibility of setting up the latitude and longitude manually for that specific session in the configuration
3. API endpoint: The endpoint api/gps is opened to receive latitude, longitude, and GPS capture UNIX timestamp (this is especially useful for analysis to check how close in time the coordinates were logged, to account for potential GPS coverage issues, spotty reception of coordinates, etc.). This was included thinking of a companion mobile app that can send the coordinates from a phone or any other setup that can generate requests with the relevant info.

The code was extensively tested on the TP-Link M7350, including GPS mode changes, downloading old session information, etc.

Some visual elements were added, like the configuration fields, GPS modes in the session panels, and a GPS div with the last received coordinates, if applicable.

There was the suggestion to use the Kismet GPS format (https://www.kismetwireless.net/docs/dev/pcapng_gps/#gps-custom-option ), however, while completely possible by currently used libraries, the result is not parsed by Wireshark and requires an extra layer of code to process. So I decided to keep this information in the packet comments for now.

Please be mindful that this will generate PCAP files with the location of the sensor operator, while this is super valuable to answer the when and where parts of a detection, this is highly sensitive information. A comprehensive threat model should include how to process this information. For instance, it would be a very good practice to only publish PCAPs that are anonymized or at least with truncated decimals in the coordinates to reduce precision, and explore informed consent processes for operators (this risk consideration also applies to collecting IMSIs, but it is definitely intensified by collecting GPS data)

The API endpoint might be tested with this very simple python script:

import requests
from datetime import datetime, timezone
import time
import random

# The API endpoint
url = "http://192.168.0.1:8080/api/gps"

# Initial semirandom data
latitude = 40.472757
longitude = -6.653948

while True:
    utc_timestamp = datetime.now(timezone.utc).timestamp()
    data = {
    "latitude": latitude,
    "longitude": longitude,
    "timestamp": str(int(utc_timestamp))
    }
    # A POST request to the API
    response = requests.post(url, json=data)
    # Print the response
    # print(response.json())
    print(data)
    a = random.uniform(-0.0001, 0.0001)
    latitude = latitude + a
    longitude =longitude - a
    time.sleep(2.5)

Claude Code was used to speed up the process; however, every modification was reviewed, approved, and tested personally on device.

Pull Request Checklist

• The Rayhunter team has recently expressed interest in reviewing a PR for this.
  • If not, this PR may be closed due our limited resources and need to prioritize how we spend them.
• Added or updated any documentation as needed to support the changes in this PR.
• Code has been linted and run through cargo fmt.
• If any new functionality has been added, unit tests were also added.
• CONTRIBUTING.md has been read.

You must check one of:

• No generative AI (including LLMs) tools were used to create this PR.
• Generative AI was used to create this PR. I certify that I have read and understand the code, and that all comments and descriptions were authored by myself and are not the product of generative AI.

[CGurity]

References issues #20 #736 and Discussion #447

[untitaker]

the code seems perfectly sound. Note that we're adding a separate ndjson file for all the GPS data, and the PCAP download now joins QMDL and this additional file into one format.

I wonder if we should compress this file in a similar fashion as we plan to do with the QMDL: #970 but I think this would require a larger refactor, and we can do it later.

Alternatively we could write GPS data straight into the QMDL?

But I think this is a good first approximation, and whether these files become too large entirely depends on how often the user submits GPS position updates.

[wgreenberg]

thanks so much for writing this @CGurity!

i have a couple comments ranging from minor nitpicks to moderate refactors, but i agree w/ @untitaker that the general approach here is sound.

[wgreenberg]

hmm, i think it's misleading to give a 0 unix timestamp for the fixed coordinate. instead, what would you think about using the rayhunter startup time (i.e. the current time when this code runs) here?

[wgreenberg]

or perhaps better yet, maybe we should omit a timestamp entirely for the fixed/synthetic data points

[wgreenberg]

reminder about this

[wgreenberg]

what do you think of using the same serialized JSON format used above here? it's slightly less user-friendly, but is definitely still legible to a human, and would make this much more easy to parse if we wanted to write automated tools to process this data later

[untitaker]

@CGurity currently CI fails. appears to be mostly formatting related

[CGurity]

Sorry, the new review request was sent by accident, last time I went through this process, GitHub wasn't even MS property. Apologies, reviewing everything.

[CGurity]

All requests addressed, most important considerations are:

• Now, the GPS sidecar file for each session is the only source of data for creating PCAPs for that session
• The numbering logic behind configurations using dropdowns is changed by the Rust enum approach. Please check if the display options setting is working well, since I can't test it
• Again, everything related to the GPS features was tested on device, including changing modes and auditing PCAPs to check that the coordinates and timestamps were populated correctly.

Again apologies for any process hiccups, I'm still rusty on this, and thank you for the review :)

[cooperq]

@CGurity sorry there is now a merge conflict to be resolved. Also the to_datetime method is unused which is causing a linter failure.

[CGurity]

No problem! I'll address both things the best I can. I'll let you know if something is too difficult for me.

[CGurity]

The issues were addressed, also the following testing commands were run to anticipate further issues:

cargo clippy --package rayhunter-daemon 2>&1 | head -80
cargo check --package rayhunter-daemon
cargo test --package rayhunter-daemon
cargo fmt --package rayhunter-daemon -- --check

Some refactoring to address outputs for these commands. Again, the resulting code was just tested in TP-Link M7350 hardware and everything GPS related works as expected.

[wgreenberg]

thanks! i'm hoping to get back to reviewing this today or tomorrow

[wgreenberg]

thanks @CGurity. there's still some outstanding issues from my last review, and i have a couple more comments, but it's looking closer!

[wgreenberg]

i don't think the request handler for /api/gps ever returns 404 right?

[CGurity]

From my testing, I have seen 404 in two scenarios:

• Trying to get GPS coordinates when in disabled mode (expected)
• Trying to get GPS coordinates in API mode before it actually received coordinated for the first time (also expected?)

[wgreenberg]

if we didn't find GPS data, i think returning 404 would be appropriate here

[CGurity]

I propose generating the pcap and logging the lack of GPS data (given the way everything is generated, I don't see this happening except in a couple of border cases, so I'm a bit wary of adding too much more code to cover this). Let me know what do you think.

[BeigeBox]

Took a pass looking specifically for things that'd affect users in the field, since you and wgreenberg have been through the shape of the code already. Found half a dozen worth flagging, in rough order of impact:

1. Client-supplied timestamp gets thrown away on write to the sidecar and Utc::now() goes in instead, which is the opposite of what the PR description says ("especially useful for analysis to check how close in time the coordinates were logged").
2. Fixed mode silently no-ops if one of the two coords is missing from config. No error, UI just sits on "Awaiting GPS data..." forever.
3. POST returns 200 but drops the record entirely when no recording is active. Companion app sees success, data goes nowhere.
4. gps_fixed_latitude/_longitude persist in config.toml after the user switches mode to Disabled.
5. No server-side range validation on posted lat/lon. The HTML5 min/max on the form is cosmetic.
6. Frontend's gps_mode is read once and doesn't refresh after config changes, so the GPS card visibility lags until hard reload.

Details inline. None of them look hard to address.

[CGurity]

Code updated with corrections, again thank you for catching these issues.

[BeigeBox]

Quick follow up on my earlier timestamp comment. Now that we're storing the client supplied timestamp (good), out of order records are actually more likely than before — a phone app can queue fixes while offline and POST them late, or retry. partition_point at pcap.rs:104 returns whatever on unsorted input, so worth adding a sort here before returning:

records.sort_by_key(|r| r.unix_ts);
records

Cheap, robust.

[CGurity]

Last fix incorporated. Also rebased with main for the recent changes incorporated. GPS features tested again for this round.

[cooperq]

just flagging that while we are waiting for will's time to free up to finish reviewing this there are some merge conflicts and failing tests that could be addressed @CGurity

[CGurity]

Thank you very much for the update and the support! No pressure on my side. While I'm comfortable enough with the additions I'm proposing in this PR, I'm a bit worried I make a mistake rebasing with the recent merged code given I'm less familiar with those features and I'm less confident in testing them effectively. I'll be alert in case there is anything else I can help with, and again thank you :)

[untitaker]

Rebased the PR, and addressed all remaining comments. I think the remaining point is, do we want to silently drop GPS data if it's malformed, when the user tries to download PCAP? I think yes, this feature is a lot of new potentially buggy code, and we don't want to break other functionality while we still figure things out.

[untitaker]

I compacted the UI a bit and removed the extra card for GPS data. This is a bit less to scroll on mobile, at least

[wgreenberg]

this is looking good, thanks @untitaker. i have one proposal which, if you agree, hopefully won't be too much extra work

[wgreenberg]

what do you think about just using Rayhunter's current time for this, rather than accepting a timestamp from the client? we already have an API endpoint to sync Rayhunter's clock, and are using it for pcap data anyhow, so IMO using it removes the chance for a client to send us bogus time data that doesn't gel with the corresponding packet data.

[untitaker]

right. i mean, our use of timestamps is getting messy overall:

1. system timestamp/manifest timestamp can be synced with the browser
2. packet timestamp is just whatever the modem returns (i.e. not corrected at all)
3. then GPS timestamps are user-submitted

i think it's probably best to grab the most recent packet timestamp and use that as GPS timestamp?

that the packet timestamp is not corrected is another issue, which is nontrivial to fix

[untitaker]

we already have an API endpoint to sync Rayhunter's clock, and are using it for pcap data anyhow,

that's actually not how it works, we only use it for manifest/metadata, and don't correct anything in the pcap. this was scoped out because the modem and the overall linux system have two different clocks with different drift, and messing with the pcap was deemed too magical.

[wgreenberg]

right. i mean, our use of timestamps is getting messy overall:

1. system timestamp/manifest timestamp can be synced with the browser

2. packet timestamp is just whatever the modem returns (i.e. not corrected at all)

3. then GPS timestamps are user-submitted

i think it's probably best to grab the most recent packet timestamp and use that as GPS timestamp?

agreed, that seems like a fine compromise

that's actually not how it works, we only use it for manifest/metadata, and don't correct anything in the pcap. this was scoped out because the modem and the overall linux system have two different clocks with different drift, and messing with the pcap was deemed too magical.

ah right, i'd forgotten how cursed the timing situation was for diag data. thanks for the refresher

[untitaker]

I've updated the code to record both the current system timestamp AND the latest packet timestamp for every GPS update, probably useful for debugging.

It would be kind of neat to be able to write the current system time with every QMDL message into .qmdl, not sure if the format allows for a good place for that.

Getting the latest packet timestamp was a bit hairy to do without RwLock, so I moved the code for writing gps data into DiagTask.

[wgreenberg]

afaik a QMDL message can only store a single timestamp, but theoretically we could decide to rely on the system clock + rayhunter offset and override that timestamp

[wgreenberg]

nice, thanks @untitaker, and thanks @CGurity for making this happen!