v2.3.0 Release Changes

TODO.md

@@ -321,6 +321,37 @@ The following were implemented in the WiFi Optimizer feature:
 
 ## Monitoring
 
+### PON supplemental counters: wrap-aware deltas (SFP Stats)
+The augmented PON provider's frame/allocation counters (GEM tx/rx, LAN frames, allocations) are 32-bit
+on the ONT and wrap (~4.29B unsigned, or ~2.15B if signed). Every path handles the wrap **safely**
+today - the chart delta guard (`cur >= prev ? cur - prev : null`), the alert spike check
+(`if (delta < 0) delta = 0`), and ISP Health's positive-increment total all treat a wrap like a
+counter reset. The only cost is a single dropped data point on the frame charts at each wrap, which on
+a busy link can be ~hourly for the high-rate counters.
+
+Follow-up (low priority): reconstruct the true delta across a wrap instead of gapping it. Use
+`sfp_uptime_s` to distinguish a wrap (uptime kept climbing) from a real reset/reboot (uptime dropped),
+then add the modular span. Blocker: we'd have to know the counter width (2^31 vs 2^32) per field, and a
+wrong guess produces a garbage spike - so the safe null-gap stays the default until the widths are
+confirmed. Not worth it for a rare one-point gap; revisit if the frame charts read as too sparse.
+
+### Upstream path discovery: DynamoDB regional endpoints for transit trace exposure (Setup tab)
+Transit Health involvement weighting and destination->transit jitter absolution both key off which
+monitored internet targets a transit ASN provably carries (trace ancestry). The current CDN/DNS
+targets peer at the local IX, so they cross no transit and give the attribution nothing to work with.
+AWS DynamoDB regional endpoints ARE ICMP-pingable and DO ride paid transit, so they're good extra
+trace targets (and can double as monitored targets):
+
+    dynamodb.<region>.amazonaws.com   (nearest region(s) first, e.g. us-west-2, us-west-1, us-east-2, us-east-1)
+
+- **Not anycast** - resolve + latency-rank to the nearest region(s), else you trace a transcontinental
+  path that misrepresents the transit.
+- **Attribution still needs work (observed in testing):** the endpoint pings and the path is NOT
+  peered, but the intermediate transit hops don't answer traceroute (all stars) - so the current
+  ancestry can't tell WHICH transit ASN it crossed. DynamoDB gives a clean AWS target that genuinely
+  transits; surfacing which transit still needs the responding-hop-ASN capture (record every
+  responding hop's ASN, not just monitored-target IPs), and pure-star segments recover nothing.
+
 ### Investigation Functions (Network Performance tab)
 The Investigate card currently jumps the latency charts to the most recent **packet-loss** and **loaded-loss** events and steps event-to-event (coalesced, peak-loss minute). Ideas to extend it:
 

docs/features/netopt-custom-pon-contract.md

@@ -0,0 +1,156 @@
+# Network Optimizer Custom PON Stats - JSON Contract v1
+
+The **Network Optimizer Custom (HTTP JSON)** ONT provider polls a plain HTTP
+endpoint that returns PON-layer statistics as JSON. The contract is
+vendor-neutral (field semantics follow ITU-T G.984.x / G.9807.1 terminology), so
+anyone can implement it for their ONT hardware - a GPON/XGS-PON SFP stick, a
+full ONT, or a small relay script on the gateway that gathers stats from the
+device and serves them.
+
+Configure it under Settings - ONT Device Monitoring with provider
+"Network Optimizer Custom (HTTP JSON)". Two modes:
+
+- **Standalone**: polled like any other ONT provider; PON state and FEC/BIP
+  counters flow to the ONT Stats tab and ONT alerts.
+- **Attached to a monitored SFP module** (the "Attach to SFP Module" setting):
+  polled on the gateway SFP collection cycle instead, with the full metric set
+  merged into that module's `sfp` measurement and charted on the SFP Stats tab.
+  Use this when the ONT *is* the SFP module in your gateway, so its DDM optics
+  readings and its PON internals form one time series.
+
+## Endpoint semantics
+
+- `GET http://<host>:<port>/` (any path is ignored by the reference
+  implementation; the provider requests `/`). Default port: **10012**.
+- Response: `Content-Type: application/json`, UTF-8, a single JSON object.
+- No authentication. Serve it on a management/LAN interface only.
+- The provider allows up to **15 seconds** per request and never issues
+  concurrent requests, so an implementation may gather stats on demand (e.g. an
+  SSH round-trip into an SFP stick) and may be a one-shot listener loop.
+- Stats should be gathered fresh per request (or cached only briefly).
+
+### Failure shape
+
+When the implementation cannot reach the underlying device, return HTTP 200
+with an error object instead of stats:
+
+```json
+{
+  "error": "sfp_unreachable",
+  "message": "Could not reach SFP at 169.254.1.1:30007"
+}
+```
+
+A non-empty `error` marks the poll failed; `message` is a human-readable
+detail shown in the UI. Any transport failure (refused, timeout, non-JSON) is
+treated the same way.
+
+## Payload
+
+Every section and every field is **optional** - serve what the hardware
+exposes; absent values are simply not recorded. All counters are **cumulative
+since ONT boot** (Network Optimizer derives per-interval deltas and uses
+`sfp_uptime_s` to recognize counter resets). Numbers may be JSON numbers or
+numeric strings; 64-bit counters are expected.
+
+```json
+{
+  "lan": {
+    "mode": 15,             // host-side PHY mode (raw device enum, informational)
+    "link_status": 5,       // host (module-to-gateway) link state, raw enum
+    "phy_duplex": 1         // raw enum, informational
+  },
+  "lan_counters": {         // host-link MAC counters (module <-> gateway)
+    "tx_frames": 671630919,
+    "rx_frames": 850075595,
+    "tx_drop_events": 0,
+    "rx_fcs_err": 0,        // FCS/checksum errors - host-link signal integrity
+    "buffer_overflow": 0
+  },
+  "ploam": {
+    "curr_state": 5,        // ITU-T activation state, numeric: 1-7 = O1-O7 (5 = O5 Operation)
+    "previous_state": 4,    // state before the last transition
+    "elapsed_msec": 12345   // ms in the current state (uint32 on many devices; wraps ~49.7 days)
+  },
+  "gtc_status": {
+    "ds_state": 3,          // downstream GTC sync state (raw enum; 3 = sync)
+    "onu_id": 49,           // ONU ID assigned by the OLT; changes on re-ranging
+    "ds_fec_enable": 0,     // OLT profile: downstream FEC on/off (0/1)
+    "us_fec_enable": 0,     // OLT profile: upstream FEC on/off (0/1)
+    "onu_response_time": 34992  // raw ranging response time / equalization delay
+  },
+  "gtc_counters": {
+    "bip": 0,                     // BIP parity errors (0 on a healthy link)
+    "hec_error_corr": 0,          // GTC header errors, corrected
+    "hec_error_uncorr": 1,        // GTC header errors, uncorrectable
+    "bwmap_error_corr": 0,        // upstream bandwidth-map errors, corrected
+    "bwmap_error_uncorr": 0,      // upstream bandwidth-map errors, uncorrectable
+    "fec_error_corr": 0,          // corrected FEC bit errors (informational)
+    "fec_words_corr": 0,          // corrected FEC codewords (early warning)
+    "fec_words_uncorr": 0,        // UNCORRECTABLE FEC codewords - the data-loss signal
+    "fec_words_total": 0,
+    "fec_seconds": 0,
+    "tx_gem_frames_total": 848555332,       // (X)GEM frames sent upstream
+    "tx_gem_bytes_total": 1774937051,       // (often 32-bit; unreliable at line rate)
+    "tx_gem_idle_frames_total": 1076427353, // idle frames = granted-but-unused upstream capacity
+    "rx_gem_frames_total": 682156109,
+    "rx_gem_bytes_total": 0,
+    "rx_gem_frames_dropped": 1,
+    "omci_drop": 0,
+    "drop": 1,
+    "rx_oversized_frames": 0,
+    "allocations_total": 1629046208,        // upstream grants received from the OLT
+    "allocations_lost": 20                  // missed grants - scheduling/resync indicator
+  },
+  "gpe_pon": {              // PON-side bridge port counters (packet engine)
+    "ibp_good": 670798566,  "ibp_discard": 0,   // ingress good/discarded
+    "ebp_good": 848843235,  "ebp_discard": 0,   // egress good/discarded
+    "learning_discard": 0                       // MAC-learning-limit discards
+  },
+  "gpe_lan": {              // host-side bridge port, same fields as gpe_pon
+    "ibp_good": 848843244,  "ibp_discard": 0,
+    "ebp_good": 670798569,  "ebp_discard": 0,
+    "learning_discard": 0
+  },
+  "sfp_uptime_s": 358825    // seconds since the ONT module booted (counter-reset anchor)
+}
+```
+
+## What Network Optimizer records
+
+Not every field is stored. The curated set, and how the standard concepts map
+onto Network Optimizer's existing schema:
+
+| Contract field | Stored as | Notes |
+|---|---|---|
+| `ploam.curr_state` | `pon_link_status` | same encoding as the ont measurement: `initial`, `standby`, `serial_number`, `ranging`, `operation`, `popup`, `emergency_stop` |
+| `ploam.previous_state` | `pon_link_status_prev` | same encoding |
+| `ploam.elapsed_msec` | `ploam_elapsed_ms` | |
+| `gtc_status.ds_state` | `gtc_ds_state` | |
+| `gtc_status.onu_id` | `onu_id` | |
+| `gtc_status.ds_fec_enable` / `us_fec_enable` | `ds_fec_enabled` / `us_fec_enabled` | |
+| `gtc_status.onu_response_time` | `onu_response_time` | |
+| `gtc_counters.bip` | `bip_errors` | standard field |
+| `gtc_counters.fec_words_uncorr` | `fec_errors` | standard field: uncorrectable codewords |
+| `gtc_counters.fec_words_corr` | `fec_corrected_words` | |
+| `gtc_counters.hec_error_corr` / `uncorr` | `hec_corrected` / `hec_uncorrected` | |
+| `gtc_counters.bwmap_error_corr` / `uncorr` | `bwmap_corrected` / `bwmap_uncorrected` | |
+| `gtc_counters.tx_gem_frames_total` etc. | `gem_tx_frames`, `gem_tx_idle_frames`, `gem_rx_frames`, `gem_rx_dropped` | |
+| `gtc_counters.allocations_total` / `lost` | `alloc_total` / `alloc_lost` | |
+| `gpe_pon` / `gpe_lan` discards | `gpe_{pon,lan}_{ingress,egress,learning}_discard` | good-frame counters are not stored |
+| `lan.link_status` | `lan_link_status` | |
+| `lan_counters.*` | `lan_tx_frames`, `lan_rx_frames`, `lan_tx_drop_events`, `lan_rx_fcs_err`, `lan_buffer_overflow` | |
+| `sfp_uptime_s` | `sfp_uptime_s` | |
+
+Not recorded: `lan.mode`, `lan.phy_duplex` (static config), GEM byte counters
+(32-bit wrap), `drop` / `omci_drop` / `rx_oversized_frames`,
+`fec_error_corr` / `fec_words_total` / `fec_seconds`, and the `gpe_*` good-frame
+counters (redundant with `lan_counters` and GEM frame counters).
+
+## Reference implementation
+
+The first serving-side implementation is a pair of shell scripts on a UniFi
+gateway that SSH into a Lantiq-based GPON SFP stick, run the vendor `onu` CLI
+(`ploamsg`, `gtcsg`, `gtctcg`, `gpebptcg`, `lanpsg`, `lantcg`), and serve the
+JSON via a one-shot netcat accept loop on port 10012. Any equivalent works -
+the provider only sees the HTTP contract above.

scripts/agent/install-agent-gateway.sh

@@ -59,19 +59,34 @@ while [ $# -gt 0 ]; do
     esac
 done
 
-err() { echo "Error: $*" >&2; exit 1; }
+# --- Output helpers -----------------------------------------------------------
+# Colorized, structured output; colors collapse to empty when stdout isn't a
+# terminal, so piped/logged output stays clean.
+if [ -t 1 ]; then
+    _b=$'\e[1m'; _dim=$'\e[2m'; _grn=$'\e[32m'; _ylw=$'\e[33m'; _red=$'\e[31m'; _cyn=$'\e[36m'; _rst=$'\e[0m'
+else
+    _b=; _dim=; _grn=; _ylw=; _red=; _cyn=; _rst=
+fi
+_rule="$(printf '\xe2\x94\x80%.0s' {1..52})"   # ── divider between sections
+step() { printf '\n%s%s%s\n%s==>%s %s%s%s\n' "$_dim" "$_rule" "$_rst" "${_cyn}${_b}" "$_rst" "$_b" "$*" "$_rst"; }
+ok()   { printf '  %s\xe2\x9c\x93%s %s\n' "$_grn" "$_rst" "$*"; }
+note() { printf '  %s%s%s\n' "$_dim" "$*" "$_rst"; }
+warn() { printf '  %s\xe2\x9a\xa0%s  %s\n' "$_ylw" "$_rst" "$*"; }
+err()  { printf '%sError:%s %s\n' "${_red}${_b}" "$_rst" "$*" >&2; exit 1; }
 
 [ "$(id -u)" -eq 0 ] || err "Run as root (the gateway's default SSH user is root)."
 command -v systemctl >/dev/null 2>&1 || err "systemd is required (systemctl not found)."
 
 # --- Teardown --------------------------------------------------------------
 if [ "$UNINSTALL" = true ]; then
-    echo "Removing ${SERVICE_NAME} and ${INSTALL_DIR}..."
+    step "Removing the Network Optimizer agent"
+    note "${SERVICE_NAME} + ${INSTALL_DIR}"
     systemctl disable --now "${SERVICE_NAME}.service" 2>/dev/null || true
     rm -f "/etc/systemd/system/${SERVICE_NAME}.service"
     systemctl daemon-reload 2>/dev/null || true
     rm -rf "$INSTALL_DIR"
-    echo "Done - the gateway is back to stock."
+    ok "Removed - the gateway is back to stock."
+    printf '\n'
     exit 0
 fi
 
@@ -96,36 +111,39 @@ esac
 # is already accounted for in MemAvailable).
 MIN_AVAILABLE_MB=256
 if ! systemctl is-active --quiet "${SERVICE_NAME}.service"; then
+    step "Memory pre-flight"
     AVAILABLE_MB="$(awk '/MemAvailable/ {print int($2/1024)}' /proc/meminfo)"
     if [ -z "$AVAILABLE_MB" ]; then
-        echo "Warning: could not read MemAvailable from /proc/meminfo - skipping the memory check." >&2
+        warn "could not read MemAvailable from /proc/meminfo - skipping the memory check."
     elif [ "$AVAILABLE_MB" -lt "$MIN_AVAILABLE_MB" ]; then
         err "only ${AVAILABLE_MB} MB of memory is available; the agent needs ${MIN_AVAILABLE_MB} MB of headroom so it stays well clear of routing/IPS. Free up memory (e.g. remove unused UniFi applications) or run the agent on a separate box (see install-native.sh)."
     else
-        echo "Memory check: ${AVAILABLE_MB} MB available (need ${MIN_AVAILABLE_MB} MB) - OK"
+        ok "${AVAILABLE_MB} MB available (need ${MIN_AVAILABLE_MB} MB)"
     fi
 fi
 
-echo "Installing Network Optimizer agent to ${INSTALL_DIR} (${RID}, monitoring-only)"
+printf '\n%sNetwork Optimizer on-site agent (gateway, monitoring-only)%s\n' "$_b" "$_rst"
+note "Installing to ${INSTALL_DIR}  (${RID})"
 mkdir -p "$INSTALL_DIR"
 
 # Download to a temp name and rename into place: writing over the binary while
 # the agent is running fails with ETXTBSY, but rename swaps the directory entry
 # and the running process keeps its old inode until the restart below.
-echo "Downloading agent binary..."
-curl -fSL "${RELEASE_BASE}/NetworkOptimizer.Agent-${RID}" -o "${INSTALL_DIR}/NetworkOptimizer.Agent.new"
+step "Downloading agent binary"
+curl -fsSL "${RELEASE_BASE}/NetworkOptimizer.Agent-${RID}" -o "${INSTALL_DIR}/NetworkOptimizer.Agent.new"
 chmod +x "${INSTALL_DIR}/NetworkOptimizer.Agent.new"
 mv -f "${INSTALL_DIR}/NetworkOptimizer.Agent.new" "${INSTALL_DIR}/NetworkOptimizer.Agent"
+ok "agent (${RID})"
 
 CONFIG="${INSTALL_DIR}/agent.json"
 
+step "Configuring the agent"
 # Preserve an already-enrolled config so re-running to update the binary never
 # wipes the persisted key.
 if grep -q '"agentKey"' "$CONFIG" 2>/dev/null; then
-    echo "Existing enrolled agent config found - keeping it."
+    note "Existing enrollment found - keeping agent.json"
 else
     [ -n "$TOKEN" ] || err "--token is required for a first-time install."
-    echo "Writing ${CONFIG}"
     {
         echo "{"
         echo "  \"serverUrl\": \"${SERVER%/}\","
@@ -135,9 +153,10 @@ else
         echo "}"
     } > "$CONFIG"
     chmod 600 "$CONFIG"
+    ok "Wrote ${CONFIG}"
 fi
 
-echo "Installing ${SERVICE_NAME}.service"
+step "Installing the agent service"
 cat > "/etc/systemd/system/${SERVICE_NAME}.service" <<UNIT
 [Unit]
 Description=Network Optimizer Agent (${SERVICE_NAME})
@@ -163,15 +182,15 @@ WantedBy=multi-user.target
 UNIT
 
 systemctl daemon-reload
-systemctl enable "${SERVICE_NAME}.service"
+systemctl enable --quiet "${SERVICE_NAME}.service"
 # restart (not `enable --now`) so an upgrade re-run moves an already-running
 # agent onto the new binary; it starts a stopped/fresh service just the same
 systemctl restart "${SERVICE_NAME}.service"
-
-echo
-echo "Agent started (monitoring-only). It enrolls, then holds a tunnel to ${SERVER%/}."
-echo "Watch it come Online in the web UI, or follow logs:"
-echo "  journalctl -u ${SERVICE_NAME} -f"
-echo
-echo "Remove it again with:"
-echo "  bash <(curl -fsSL https://raw.githubusercontent.com/Ozark-Connect/NetworkOptimizer/main/scripts/agent/install-agent-gateway.sh) --uninstall"
+ok "${SERVICE_NAME}.service installed and started"
+
+step "Done"
+ok "Agent installed and running (monitoring-only)"
+note "It enrolls, then holds a tunnel to ${SERVER%/} - watch it come Online in the web UI."
+note "Logs:   journalctl -u ${SERVICE_NAME} -f"
+note "Remove: bash <(curl -fsSL https://raw.githubusercontent.com/Ozark-Connect/NetworkOptimizer/main/scripts/agent/install-agent-gateway.sh) --uninstall"
+printf '\n'

scripts/agent/install-docker.sh

@@ -37,7 +37,20 @@ while [ $# -gt 0 ]; do
     esac
 done
 
-err() { echo "Error: $*" >&2; exit 1; }
+# --- Output helpers -----------------------------------------------------------
+# Colorized, structured output; colors collapse to empty when stdout isn't a
+# terminal, so piped/logged output stays clean.
+if [ -t 1 ]; then
+    _b=$'\e[1m'; _dim=$'\e[2m'; _grn=$'\e[32m'; _ylw=$'\e[33m'; _red=$'\e[31m'; _cyn=$'\e[36m'; _rst=$'\e[0m'
+else
+    _b=; _dim=; _grn=; _ylw=; _red=; _cyn=; _rst=
+fi
+_rule="$(printf '\xe2\x94\x80%.0s' {1..52})"   # ── divider between sections
+step() { printf '\n%s%s%s\n%s==>%s %s%s%s\n' "$_dim" "$_rule" "$_rst" "${_cyn}${_b}" "$_rst" "$_b" "$*" "$_rst"; }
+ok()   { printf '  %s\xe2\x9c\x93%s %s\n' "$_grn" "$_rst" "$*"; }
+note() { printf '  %s%s%s\n' "$_dim" "$*" "$_rst"; }
+warn() { printf '  %s\xe2\x9a\xa0%s  %s\n' "$_ylw" "$_rst" "$*"; }
+err()  { printf '%sError:%s %s\n' "${_red}${_b}" "$_rst" "$*" >&2; exit 1; }
 
 [ -n "$SERVER" ] || err "--server is required (the central server's HTTPS address)"
 case "$SERVER" in
@@ -61,21 +74,23 @@ if [ "$(id -u)" -ne 0 ]; then
     SUDO="sudo"
 fi
 
-echo "Installing Network Optimizer agent to ${INSTALL_DIR}"
+printf '\n%sNetwork Optimizer on-site agent (Docker)%s\n' "$_b" "$_rst"
+note "Installing to ${INSTALL_DIR}"
 $SUDO mkdir -p "${INSTALL_DIR}/data"
 
-# Compose template
+step "Fetching the compose template"
 $SUDO curl -fsSL "$COMPOSE_URL" -o "${INSTALL_DIR}/docker-compose.yml"
+ok "docker-compose.yml"
 
 CONFIG="${INSTALL_DIR}/data/agent.json"
 
+step "Configuring the agent"
 # Preserve an already-enrolled config so re-running the installer (e.g. to
 # update the image) never wipes the persisted agent key.
 if $SUDO grep -q '"agentKey"' "$CONFIG" 2>/dev/null; then
-    echo "Existing enrolled agent config found - keeping it."
+    note "Existing enrollment found - keeping agent.json"
 else
     [ -n "$TOKEN" ] || err "--token is required for a first-time install"
-    echo "Writing ${CONFIG}"
     TMP_CONFIG="$(mktemp)"
     {
         echo "{"
@@ -90,13 +105,16 @@ else
     } > "$TMP_CONFIG"
     $SUDO cp "$TMP_CONFIG" "$CONFIG"
     rm -f "$TMP_CONFIG"
+    ok "Wrote ${CONFIG}"
 fi
 
-echo "Starting agent..."
+step "Starting the agent"
 $SUDO $COMPOSE -f "${INSTALL_DIR}/docker-compose.yml" pull
 $SUDO $COMPOSE -f "${INSTALL_DIR}/docker-compose.yml" up -d
+ok "container up"
 
-echo
-echo "Agent started. It enrolls, then holds a tunnel to ${SERVER%/}."
-echo "Watch it come Online in the web UI, or follow logs:"
-echo "  ${SUDO} docker logs -f network-optimizer-agent"
+step "Done"
+ok "Agent started"
+note "It enrolls, then holds a tunnel to ${SERVER%/} - watch it come Online in the web UI."
+note "Logs: ${SUDO} docker logs -f network-optimizer-agent"
+printf '\n'