[Bug] Node Network In/Out always 0 — /cluster/resources?type=node doesn't expose netin/netout

[SpyrosPsarras]

Describe the bug

Dashboard "Network In / Network Out" columns for Proxmox nodes are always 0.

Root cause: the Proxmox API endpoint PegaProx queries (/cluster/resources?type=node) does not include netin/netout fields — those only exist on type=vm records. The inline code comment at pegaprox/core/manager.py:971 claims otherwise; it is incorrect.

Location in source (v0.9.10.2)

pegaprox/core/manager.py lines 971–980:

# MK: /nodes/{node}/status doesn't have netin/netout, only /cluster/resources does
net_by_node = {}
try:
    res_url = f"https://{host}:{self.api_port}/api2/json/cluster/resources?type=node"
    res_r = self._create_session().get(res_url, timeout=10)
    if res_r.status_code == 200:
        for nr in res_r.json().get('data', []):
            net_by_node[nr.get('node', '')] = {
                'netin': nr.get('netin', 0),
                'netout': nr.get('netout', 0)
            }

Since the node records have no netin/netout, nr.get('netin', 0) always returns 0.

Evidence (Proxmox-side, via pvesh)

pvesh get /cluster/resources --type node returns only these keys:

cgroup-mode, cpu, disk, hastate, id, level, maxcpu, maxdisk,
maxmem, mem, node, status, type, uptime

No netin/netout.

pvesh get /cluster/resources --type vm does include netin, netout (works correctly for VMs).

pvesh get /nodes/{node}/rrddata --timeframe hour returns samples that do carry netin/netout — this is where node throughput actually lives. Example latest sample on our cluster:

netin:  844333.85 (bytes/sec, AVERAGE)
netout: 863012.88 (bytes/sec, AVERAGE)

Steps to Reproduce

1. Connect PegaProx to any Proxmox VE 9.x cluster
2. Open the cluster dashboard
3. Observe the Network In / Network Out columns for the node rows — always 0, regardless of actual node traffic
4. Confirm from the Proxmox side:

   pvesh get /cluster/resources --type node --output-format json | jq '.[0] | keys'

   netin/netout are not in the returned keys.

Expected behavior

Node Network In / Network Out columns should show the current (or recent-average) throughput for each node's primary network, matching what the Proxmox VE web UI shows on the node summary graph.

Proposed fix

The repo already has a helper for exactly this — PVEManager.get_node_rrddata() at manager.py:11671 ("NS: Added Jan 2026 - Same format as VM rrddata for consistency"). It's just not used by the dashboard net_by_node block.

Sketch:

net_by_node = {}
for node in nodes:
    nname = node['node']
    try:
        rrd_url = (
            f"https://{host}:{self.api_port}/api2/json/nodes/{nname}"
            f"/rrddata?timeframe=hour&cf=AVERAGE"
        )
        rrd_r = self._create_session().get(rrd_url, timeout=10)
        if rrd_r.status_code == 200:
            samples = rrd_r.json().get('data', [])
            # walk back to most recent sample with non-null netin/netout
            for s in reversed(samples):
                if s.get('netin') is not None and s.get('netout') is not None:
                    net_by_node[nname] = {
                        'netin': s['netin'],
                        'netout': s['netout'],
                    }
                    break
    except Exception:
        pass

Notes for the implementer:

• Units change: rrddata returns a rate (bytes/sec, AVERAGE over the timeframe step), not cumulative bytes. If the UI does its own delta between refreshes assuming cumulative bytes, the value should be treated as a rate directly. End-user effect: panel finally shows real throughput instead of 0.
• Extra HTTP calls: adds N calls per refresh (1 per node). Mitigation: fold the RRD fetch into the existing fetch_node_details(node) function so it benefits from the GEVENT_POOL parallel path already conditional in manager.py.

Environment

• PegaProx Version: 0.9.10.2 (also reproduced in 0.9.4 — code path unchanged across releases)
• Installation Method: Manual (git clone / update.sh archive-based update)
• OS: Debian 13 trixie (LXC, unprivileged)
• Browser: Firefox 150
• Behind Reverse Proxy? No (direct https://host:5000)
• UI? Modern Layout

Proxmox VE on the managed cluster: 9.1.11 (Debian 13 hosts).

Logs

No errors emitted — the HTTP call to /cluster/resources?type=node succeeds with HTTP 200, and the .get('netin', 0) quietly returns 0 because the key is absent. So nothing surfaces in logs; the symptom is purely the always-zero panel.

Screenshots

N/A — every node row simply shows 0 B/s for both columns. The same panel renders correct values for CPU, RAM, disk, load average; only the two network columns are flat.

Checklist

• I have searched existing issues to make sure this is not a duplicate (searched for netin, netout, network, cluster/resources — closed Vm network stats. #205 is about VM stats, not nodes)
• I am using the latest version of PegaProx (0.9.10.2)

[mkellermann97]

Hey @SpyrosPsarras — this is the level of bug report we wish every ticket looked like: file:line citation, pvesh evidence on both sides, and a fix sketch using the right helper. Verified end-to-end against PVE 9.x: confirmed netin/netout are absent on /cluster/resources?type=node rows and present on /nodes/{name}/rrddata samples. The old comment at manager.py:971 was just wrong.

Patch landed on Testing (commit 2772f0c). Folded the RRD pull into fetch_node_details() so it rides the existing gevent parallel pool — wall-clock doesn't double, per-node serial latency goes up a bit but that's absorbed. Walks samples in reverse for the first one where both netin AND netout are non-null (PVE's last 1–2 averaging windows are usually still null while it averages). Falls back to (0, 0) only if every sample is null, which preserves the old "show 0 if we don't know" behaviour.

Nico cuts releases and he's on holiday through May 18, so this rolls into the next release whenever he gets to it — no fixed ETA from me.

Well-investigated reports like this are exactly why the project's still alive. Donation-funded at https://opencollective.com/pegaprox if you ever feel like buying us a round in return.

— MK

[mkellermann97]

Hey @SpyrosPsarras — v0.9.10.3 shipped: https://github.com/PegaProx/project-pegaprox/releases/tag/v0.9.10.3

Your fix is in. The old /cluster/resources?type=node path is gone; fetch_node_details() now pulls /nodes/{name}/rrddata?timeframe=hour&cf=AVERAGE alongside the status call, walks samples in reverse for the most recent non-null netin+netout pair, and returns it as a 4-tuple — folded into the existing gevent parallel pool so wall-clock doesn't double. Node Network In/Out columns finally show real throughput.

Closing as completed. Thanks again for the file:line-cited bug report — that one practically wrote itself.

Donations at https://opencollective.com/pegaprox if well-investigated-reports-paying-back earns its keep.

— Marcus

[SpyrosPsarras]

Re-opening this thread because v0.9.10.3 still shows 0.0 MB/s on the dashboard Node Network In/Out cards in my cluster — the backend half of 2772f0c landed correctly (/api/clusters/<id>/metrics now returns netin/netout as bytes/sec from rrddata), but the frontend consumer in web/src/tables.js NodeCard was not updated and still does counter-delta-over-time:

const netinRate = timeDiff > 0
  ? Math.max(0, (metrics.netin - lastNetRef.current.netin) / timeDiff)
  : 0;

Delta of a near-stable rate ≈ 0 → .toFixed(1) renders "0.0", exactly the original symptom.

Follow-up patch (drop the delta math + lastNetRef, divide directly) opened as #435 — small two-file diff, includes a rebuilt web/index.html. VM-row counter math is left alone (VMs on /cluster/resources?type=vm are still cumulative, that math is correct).