Skip to content

Won't populate prometheus/grafana after a few minutes #201

@dziban303

Description

@dziban303

I get the sense this project is abandoned but in case it isn't, I've found that when connecting to a node via TCP it will provide stats that appear in grafana but it stops after a few minutes. The container needs to be restarted, and it will work for a few minutes, and then stop again. There are no errors in the container log, in fact it seems to be still connected and getting packets, but nothing appears in grafana.

One thing that's odd is that after a few minutes it starts saying NodeInfo is now yet known for Node 1770346504, ignoring the packet 3707814273 — the referenced node is the one I'm connected to, how would it have forgotten what node that is? But it also forgets who all the other nodes are, as seen in the container log output:

meshtastic-exporter  | 2025-12-02 09:46:20,349 - meshtastic_prometheus_exporter - INFO - Share ideas and vote for new features https://github.com/hacktegic/meshtastic-prometheus-exporter/discussions/categories/ideas
meshtastic-exporter  | 2025-12-02 09:46:20,356 - meshtastic_prometheus_exporter - WARNING - Sentry error reporting is disabled. To enable automatic error reporting to project maintainers in case of runtime errors, set the ENABLE_SENTRY environment variable to 1.
meshtastic-exporter  | 2025-12-02 09:46:20,885 - meshtastic_prometheus_exporter - INFO - NodeDB is available, saving metadata in cache for 102 nodes
meshtastic-exporter  | 2025-12-02 09:46:20,887 - meshtastic_prometheus_exporter - INFO - Connected to device over TCPInterface
meshtastic-exporter  | 2025-12-02 09:46:20,889 - meshtastic_prometheus_exporter - INFO - MeshPacket 744657259 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:46:23,400 - meshtastic_prometheus_exporter - INFO - MeshPacket 2082172268 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:46:40,402 - meshtastic_prometheus_exporter - INFO - MeshPacket 3211477984 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:47:19,470 - meshtastic_prometheus_exporter - INFO - MeshPacket 318253421 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:47:23,472 - meshtastic_prometheus_exporter - INFO - MeshPacket 3061550446 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:47:27,727 - meshtastic_prometheus_exporter - INFO - MeshPacket 2438029035 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:48:19,500 - meshtastic_prometheus_exporter - INFO - MeshPacket 157989231 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:48:23,502 - meshtastic_prometheus_exporter - INFO - MeshPacket 2370556272 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:49:05,798 - meshtastic_prometheus_exporter - INFO - MeshPacket 11457217 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:49:19,552 - meshtastic_prometheus_exporter - INFO - MeshPacket 1095154033 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:49:23,553 - meshtastic_prometheus_exporter - INFO - MeshPacket 1958717810 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:50:19,561 - meshtastic_prometheus_exporter - INFO - MeshPacket 2029888883 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:50:23,576 - meshtastic_prometheus_exporter - INFO - MeshPacket 968604020 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:51:19,589 - meshtastic_prometheus_exporter - INFO - MeshPacket 593134965 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:51:23,598 - meshtastic_prometheus_exporter - INFO - MeshPacket 1661506934 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:51:39,363 - meshtastic_prometheus_exporter - INFO - MeshPacket 4096282476 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:51:56,616 - meshtastic_prometheus_exporter - INFO - MeshPacket 602798914 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:52:19,372 - meshtastic_prometheus_exporter - INFO - MeshPacket 4132035959 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:52:23,627 - meshtastic_prometheus_exporter - INFO - MeshPacket 1764901240 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:53:12,153 - meshtastic_prometheus_exporter - INFO - MeshPacket 3568356553 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:53:19,407 - meshtastic_prometheus_exporter - INFO - MeshPacket 670640505 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:53:23,659 - meshtastic_prometheus_exporter - INFO - MeshPacket 4037880186 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:53:24,911 - meshtastic_prometheus_exporter - INFO - MeshPacket 3289969849 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:53:47,919 - meshtastic_prometheus_exporter - INFO - MeshPacket 3214252218 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:53:55,207 - meshtastic_prometheus_exporter - INFO - Skipping encrypted packet 659201744
meshtastic-exporter  | 2025-12-02 09:54:14,465 - meshtastic_prometheus_exporter - INFO - MeshPacket 3331629046 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:54:19,467 - meshtastic_prometheus_exporter - INFO - MeshPacket 4022609275 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:54:23,470 - meshtastic_prometheus_exporter - INFO - MeshPacket 3425116540 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:55:07,236 - meshtastic_prometheus_exporter - INFO - MeshPacket 714159937 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:55:19,490 - meshtastic_prometheus_exporter - INFO - MeshPacket 3422129533 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:55:23,740 - meshtastic_prometheus_exporter - INFO - MeshPacket 2547790206 is environment metrics telemetry
meshtastic-exporter  | 2025-12-02 09:56:19,524 - meshtastic_prometheus_exporter - INFO - MeshPacket 3163955583 is device metrics telemetry
meshtastic-exporter  | 2025-12-02 09:56:23,528 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 1474814336
meshtastic-exporter  | 2025-12-02 09:56:36,285 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 2766308302, ignoring the packet 361429179
meshtastic-exporter  | 2025-12-02 09:56:55,803 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 145464524, ignoring the packet 2303272355
meshtastic-exporter  | 2025-12-02 09:57:10,057 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 517604496, ignoring the packet 1873235564
meshtastic-exporter  | 2025-12-02 09:57:19,562 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 3707814273
meshtastic-exporter  | 2025-12-02 09:57:23,577 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 3357974914
meshtastic-exporter  | 2025-12-02 09:58:19,591 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 3672052099
meshtastic-exporter  | 2025-12-02 09:58:19,842 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 2997304120, ignoring the packet 1420016749
meshtastic-exporter  | 2025-12-02 09:58:23,596 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 3116728708
meshtastic-exporter  | 2025-12-02 09:59:19,612 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 3239118213
meshtastic-exporter  | 2025-12-02 09:59:19,617 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 3083636102
meshtastic-exporter  | 2025-12-02 09:59:23,615 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 2300830087
meshtastic-exporter  | 2025-12-02 10:00:19,632 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 3326036360
meshtastic-exporter  | 2025-12-02 10:00:21,636 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1517395281, ignoring the packet 3784165165
meshtastic-exporter  | 2025-12-02 10:00:23,638 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 3196071305
meshtastic-exporter  | 2025-12-02 10:01:19,399 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 273431946
meshtastic-exporter  | 2025-12-02 10:01:23,657 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 1770346504, ignoring the packet 1882518923
meshtastic-exporter  | 2025-12-02 10:01:49,170 - meshtastic_prometheus_exporter - INFO - NodeInfo is now yet known for Node 86743048, ignoring the packet 154552044

and when it starts ignoring all the packets, it stops sending metrics to prom and then grafana.

I'm not even sure where to begin troubleshooting this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions