Dendrometer Hub LoRa Batch Packet Receive Failure (Handshake / Scheduling)

[nicholas-shatalov]

Dendrometer hub stops receiving any data from LoRa dendrometers after an unusual packet receive behavior occurs. It seems that one packet was in the process of being received from one sensor, and then another sensor tried to transmit another packet to the hub. The hub didn't finish processing the first packet and started processing the second. The hub "froze" and completely stopped receiving data after this occurred.

LoRa Feather m0 on Dendro Sensors
Feather m0 with 4g board and solar charging board on Dendrometer hub
Hypnos 3.3

Behavior occurred randomly during normal batch transmission periods

Ideally, the hub does not freeze and continues working.

[nicholasqle]

Additional information: I have the full output logs and function summaries if needed

Code

#include "arduino_secrets.h"

#include <Loom_Manager.h> //4.7

#include <Hardware/Loom_Hypnos/Loom_Hypnos.h>
#include <Radio/Loom_LoRa/Loom_LoRa.h>
#include <Sensors/Loom_Analog/Loom_Analog.h>
#include <Internet/Connectivity/Loom_LTE/Loom_LTE.h>
#include <Internet/Logging/Loom_MongoDB/Loom_MongoDB.h>

const unsigned long REPORT_INTERVAL = 1 * 60 * 60 * 1000;

Manager manager("Hub", 0);
Loom_Hypnos hypnos(manager, HYPNOS_VERSION::V3_3, TIME_ZONE::PST);
Loom_Analog batteryVoltage(manager);
Loom_LoRa lora(manager);
Loom_LTE lte(manager, "hologram", "", "", A5);
Loom_MongoDB mqtt(manager, lte, SECRET_BROKER, SECRET_PORT, DATABASE, BROKER_USER, BROKER_PASS);
Loom_BatchSD batchSD(hypnos, 16); //set batch size uploading

int packetNumber = 0;
void setup()
{
    /* Enables logging logs to the SD card for later viewing under the 'debug' folder */
  ENABLE_SD_LOGGING;   
  
  /* Enables generation of function summaries */
  ENABLE_FUNC_SUMMARIES;
    // Start the serial interface
    manager.beginSerial();

    // Enable the power rails on the hypnos
    hypnos.enable();

    setRTC();

    // Sets the LTE board to use batch SD to only start when we actually need to publish data
    lte.setBatchSD(batchSD);


    // load MQTT credentials from the SD card, if they exist
    mqtt.loadConfigFromJSON(hypnos.readFile("mqtt_creds.json"));

    // Initialize the modules
    manager.initialize();
}

void loop()
{
    // Wait 5 seconds for a message
    do{
      if (lora.receiveBatch(5000, &packetNumber))
      {
          manager.display_data();
          hypnos.logToSD();
          mqtt.publish(batchSD);
      }
    }while(packetNumber > 0);
  static unsigned long timer = millis();
  if (millis() - timer > REPORT_INTERVAL)
      {
          manager.set_device_name("Hub");
          manager.set_instance_num(0);

          manager.measure();
          manager.package();
          manager.display_data();
          mqtt.publish();
          
          timer = millis();
      }
}


void setRTC()
{
    if (!Serial)
        return;

    Serial.println(F("Adjust RTC time? (y/n)"));
    unsigned long timer = millis();
    while (!Serial.available() && (millis() - timer) < 7000)
        ;
    if (!Serial.available())
        return;
    int val = Serial.read();
    delay(50);
    while (Serial.available())
        Serial.read(); // flush the input buffer to avoid invalid input to rtc function

    if (val == 'y')
    {
        hypnos.set_custom_time();
    }
}

Text version of output

[2026.02.02 04:17:43] [DEBUG] [oom_MongoDB.cpp:publish:188] Batch not ready to publish: 9/16
[2026.02.02 04:17:44] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:45] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:45] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:46] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:46] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:47] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:47] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:48] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:48] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:49] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:49] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:50] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:50] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:51] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:51] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:53] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:53] [DEBUG] [oom_Manager.cpp:display_data:198] Data Json: 

{
  "type": "data",
  "id": {
    "name": "LBDendrometer_",
    "instance": 2
  },
  "contents": [
    {
      "module": "Packet",
      "data": {
        "Number": 1082
      }
    },
    {
      "module": "Analog",
      "data": {
        "Vbat": 4.09438467,
        "Vbat_MV": 4079.882568
      }
    },
    {
      "module": "SHT31",
      "data": {
        "Temperature_C": 3.269999981,
        "Humidity_%RH": 99.98999786,
        "VPD": 0.000077251
      }
    },
    {
      "module": "LoRa",
      "data": {
        "RSSI": -68
      }
    },
    {
      "module": "AS5311",
      "data": {
        "Alignment": "Green",
        "mag": 1015,
        "pos_raw": 1934,
        "pos_avg": 1934
      }
    },
    {
      "module": "displacement",
      "data": {
        "um": 2001.953125
      }
    },
    {
      "module": "Button",
      "data": {
        "Pressed?": false
      }
    }
  ],
  "timestamp": {
    "time_utc": "2026-02-02T10:45:44Z",
    "time_local": "2026-02-02T2:45:44"
  }
}
[2026.02.02 04:17:53] [DEBUG] [DManager.cpp:log:163] Successfully logged data to Hub17.csv
[2026.02.02 04:17:53] [DEBUG] [oom_MongoDB.cpp:publish:188] Batch not ready to publish: 10/16
[2026.02.02 04:17:53] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:54] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:54] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:17:55] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 2
[2026.02.02 04:17:55] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:00] [WARNING] [oom_LoRa.cpp:receiveFromLoRa:171] No message received
[2026.02.02 04:18:00] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:05] [WARNING] [oom_LoRa.cpp:receiveFromLoRa:171] No message received
[2026.02.02 04:18:05] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:08] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 6
[2026.02.02 04:18:08] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:09] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 6
[2026.02.02 04:18:09] [WARNING] [oom_LoRa.cpp:handleLostFrag:313] Dropping fragmented packet body with no header received from 6
[2026.02.02 04:18:09] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:11] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 6
[2026.02.02 04:18:11] [WARNING] [oom_LoRa.cpp:handleLostFrag:313] Dropping fragmented packet body with no header received from 6
[2026.02.02 04:18:11] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:12] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 6
[2026.02.02 04:18:12] [WARNING] [oom_LoRa.cpp:handleLostFrag:313] Dropping fragmented packet body with no header received from 6
[2026.02.02 04:18:12] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:13] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 6
[2026.02.02 04:18:13] [WARNING] [oom_LoRa.cpp:handleLostFrag:313] Dropping fragmented packet body with no header received from 6
[2026.02.02 04:18:13] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:14] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 6
[2026.02.02 04:18:14] [WARNING] [oom_LoRa.cpp:handleLostFrag:313] Dropping fragmented packet body with no header received from 6
[2026.02.02 04:18:14] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:16] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 6
[2026.02.02 04:18:16] [WARNING] [oom_LoRa.cpp:handleLostFrag:313] Dropping fragmented packet body with no header received from 6
[2026.02.02 04:18:16] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:17] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 6
[2026.02.02 04:18:17] [WARNING] [oom_LoRa.cpp:handleLostFrag:313] Dropping fragmented packet body with no header received from 6
[2026.02.02 04:18:17] [DEBUG] [oom_LoRa.cpp:receiveFromLoRa:160] Waiting for message...
[2026.02.02 04:18:19] [DEBUG] [oom_LoRa.cpp:receiveFrag:194] Received packet from 6

No further output after this point

[SeamusTappert]

Let's start by creating error handling for if a fragment header is received after a body fragment. We can do this by just creating temporary JSON and then just reassigning the header to it after we receive the header.

Next lets implement a system to prevent transmission while the Hub is still receiving fragments from another node. The conflict lies within some sort of conflict happening because two nodes are trying to transmit to the Hub at the same time, so ensuring that only one node is transmitting should address this issue.

[SeamusTappert]

As a quick fix we can have the transmitting hubs be scheduled in increments of 5 minutes to prevent two devices trying to transmit at the same time.

[SeamusTappert]

I found what is causing this error:

Since fromAddress is initialized to a garbage value, it will accept transmissions from any device and within the underlying recvfromACK function it will overwrite this fromAddress value. So when the receiver is in the process of receiving packets from device 2 and then receives one from device 6 it will overwrite this fromAddress value to 6, which will then break the entirety of the fragment processing logic.

[SeamusTappert]

We will need to change the fromAddress logic to only be overwritten when the fragment is complete. In addition, we should implement a timing system based on device IDs to prevent multiple transmissions at the same time.

[nicholas-shatalov]

Here is a google drive link with some screen recordings of more evidence of this issue.
https://drive.google.com/drive/folders/1djUMpaG7wfAPtOX8TSLSA4gobmgNx2Af?usp=sharing

[DEusername]

UPDATES (from prior to Nicholas' video that I forgot to append to this issue)

• 2026/02/25: Made a compiling version of the handshake system on both the sender and receiver to ensure only single target communication at a time on receiver's end.
• 2026/03/08: Created a version of the handshake system that worked between a sender and a receiver. Needs further testing on with multiple senders trying to communicate to a receiver at the same time.

[DEusername]

UPDATE 2026/03/13:

• Watched through Nicholas' video and it seems that the batch message sending process is failing when multiple devices are trying to connect and transmit data to the hub. The second sending device overrides the first devices transmission, cutting of the first sending device's connection before the first sending device is able to complete it's transmission.

[DEusername]

UPDATE: 2026/03/15:

• Fixed a bug that caused the handshake system to drop the existing handshake connection when it failed to fully transmit a denial response to a 2nd sender that had sent a handshake request.