WAN stability detection and backoff to prevent SFE kernel errors (#517)

* Add WAN stability detection and backoff to prevent SFE kernel errors

When WAN interfaces flap (reboots, GPON swaps, ISP outages), wansteer's
health checker toggled state every ~6-10s, flushing SFE 3-6 times/min
while the gateway's own failover engine manipulated the same conntrack
entries. This race produced sfe_ipv4_remove_connection kernel errors.

Changes:
- Startup grace period: wait for WAN interfaces to be link-up and stable
  before applying rules (prevents stale assumptions post-reboot)
- Instability detection: track state transition frequency per-WAN,
  mark as "unstable" when transitions exceed threshold in window
- Backoff mode: when any WAN is unstable, suppress SFE/conntrack flushes
  from health transitions and reconciliation drift detection. On recovery,
  single forced flush + full rule reapply for clean state.
- SFE flush coalescing: minimum 10s interval between flushes, redundant
  calls within cooldown are skipped

New config options (all with sensible defaults):
  startup_grace_seconds (30), instability_threshold (3),
  instability_window_seconds (300), backoff_recovery_seconds (60),
  sfe_flush_cooldown_seconds (10)

* Fix reconciler false drift detection when WAN is down

expectedRuleCount() compared against the full config, but reapplyRules()
disables traffic classes targeting unhealthy WANs. This mismatch caused
the reconciler to detect "drift" every 30s while a WAN was down,
triggering unnecessary SFE flushes each time.

Fix: pass unhealthy WANs to expectedRuleCount so it skips the same
traffic classes that reapplyRules disables.

* Refactor: deduplicate onStateChange callback, remove closure capture

- Extract onHealthChange to a named function used by both initial setup
  and SIGHUP reload (was duplicated inline)
- Move stability check into health tick case (remove separate ticker)
- Pass inBackoff explicitly through health checker instead of closing
  over a variable from the main loop scope

* Show warning when deployed wansteer binary version is outdated

Compares the app's assembly version against the wansteer status JSON
version field after each status poll. Shows an alert-warning banner
in the status card when versions don't match.

Suppressed when both are dev, or either has a non-release version
string (alpha, pre-release, +metadata). Warns when one is dev and
the other is a tagged release, or when both are different tagged
releases.

* Stamp Go binary versions in macOS install script

The Mac install script was building cfspeedtest, uwnspeedtest, and
wansteer without passing -X main.version, so all binaries reported
"dev". Now uses git describe --tags --always to stamp the version,
matching what the MSI build and Dockerfile already do.

* Show wansteer version in daemon status metrics

* Fix status JSON parse failure from missing newline after cat

The SSH command concatenated the ---VERSION--- delimiter onto the
last line of the status JSON (cat doesn't guarantee a trailing
newline). This caused JsonSerializer to fail parsing every poll,
leaving _parsedStatus null and hiding uptime, rule count, WAN
health, and version metrics.

Fix: add 'echo' after cat to ensure a newline separator.

* Remove per-WAN health metrics from status card

* Fix uptime display timezone - use DateTimeOffset to preserve offset

Author: tvancott42

scripts/install-macos-native.sh

@@ -261,6 +261,11 @@ echo "[3b/9] Building Go binaries..."
 if command -v go &> /dev/null; then
     mkdir -p "$INSTALL_DIR/tools"
 
+    # Get version from git tags for Go binary version stamps
+    GO_VERSION=$(cd "$REPO_ROOT" && git describe --tags --always 2>/dev/null || echo "dev")
+    GO_VERSION="${GO_VERSION#v}" # strip leading v
+    echo "Go binary version: $GO_VERSION"
+
     # Detect Go architecture for local binary
     GO_ARCH="amd64"
     if [ "$ARCH" = "arm64" ]; then
@@ -271,7 +276,7 @@ if command -v go &> /dev/null; then
     if [ -d "$CFSPEEDTEST_SRC" ]; then
         cd "$CFSPEEDTEST_SRC"
         CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -trimpath \
-            -ldflags "-s -w" \
+            -ldflags "-s -w -X main.version=$GO_VERSION" \
             -o "$INSTALL_DIR/tools/cfspeedtest-linux-arm64" .
         echo "Built cfspeedtest for linux/arm64"
     else
@@ -283,12 +288,12 @@ if command -v go &> /dev/null; then
         cd "$UWNSPEEDTEST_SRC"
         # Build local binary for server-side WAN speed tests
         CGO_ENABLED=0 GOOS=darwin GOARCH=$GO_ARCH go build -a -trimpath \
-            -ldflags "-s -w" \
+            -ldflags "-s -w -X main.version=$GO_VERSION" \
             -o "$INSTALL_DIR/tools/uwnspeedtest-darwin-$GO_ARCH" .
         echo "Built uwnspeedtest for darwin/$GO_ARCH (local)"
         # Build gateway binary for deployment via SSH to UniFi gateways
         CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -trimpath \
-            -ldflags "-s -w" \
+            -ldflags "-s -w -X main.version=$GO_VERSION" \
             -o "$INSTALL_DIR/tools/uwnspeedtest-linux-arm64" .
         echo "Built uwnspeedtest for linux/arm64 (gateway)"
     else
@@ -300,7 +305,7 @@ if command -v go &> /dev/null; then
         cd "$WANSTEER_SRC"
         # Build gateway binary for WAN steering (deployed via SSH to UniFi gateways)
         CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -trimpath \
-            -ldflags "-s -w" \
+            -ldflags "-s -w -X main.version=$GO_VERSION" \
             -o "$INSTALL_DIR/tools/wansteer-linux-arm64" .
         echo "Built wansteer for linux/arm64 (gateway)"
     else

src/NetworkOptimizer.Web/Components/Pages/WanSteering.razor

@@ -12,6 +12,8 @@
 @using Microsoft.EntityFrameworkCore
 @using System.Text.Json
 @using NetworkOptimizer.Core.Helpers
+@using System.Reflection
+@using System.Text.RegularExpressions
 @inject IJSRuntime JS
 
 <PageTitle>WAN Steering - Network Optimizer</PageTitle>
@@ -82,6 +84,13 @@
                         @(_status.BinaryDeployed ? "Deployed" : "Not Deployed")
                     </div>
                 </div>
+                @if (_parsedStatus != null && !string.IsNullOrEmpty(_parsedStatus.Version))
+                {
+                    <div class="metric">
+                        <div class="metric-label">Version</div>
+                        <div class="metric-value">@_parsedStatus.Version</div>
+                    </div>
+                }
                 <div class="metric">
                     <div class="metric-label">Daemon</div>
                     <div class="metric-value">
@@ -99,18 +108,15 @@
                         <div class="metric-label">Active Rules</div>
                         <div class="metric-value">@_parsedStatus.RuleCount</div>
                     </div>
-                    @foreach (var kvp in _parsedStatus.WanHealth)
-                    {
-                        <div class="metric">
-                            <div class="metric-label">@kvp.Key</div>
-                            <div class="metric-value">
-                                <span class="status-indicator @(kvp.Value.Healthy ? "status-active" : "status-inactive")"></span>
-                                @(kvp.Value.Healthy ? "Healthy" : "Down")
-                            </div>
-                        </div>
-                    }
                 }
             </div>
+            @if (_showVersionWarning && _parsedStatus != null)
+            {
+                <div class="alert alert-warning" style="margin-top: 1rem;">
+                    <strong>WAN Steer binary outdated.</strong>
+                    Gateway is running @(_parsedStatus.Version) but the app is @_appVersion. Redeploy to update.
+                </div>
+            }
         }
     </div>
 </div>
@@ -496,6 +502,8 @@
     private List<WanSteerWanInfo> _wanInterfaces = new();
     private WanSteerStatus? _status;
     private ParsedDaemonStatus? _parsedStatus;
+    private bool _showVersionWarning;
+    private string _appVersion = "";
 
     // Rule editing
     private int? _editingRuleId; // null = not editing, 0 = adding new, >0 = editing existing
@@ -1066,10 +1074,10 @@
         return parts.Count > 0 ? string.Join(" | ", parts) : "Match all traffic";
     }
 
-    private static string FormatUptime(DateTime startedAt)
+    private static string FormatUptime(DateTimeOffset startedAt)
     {
         if (startedAt == default) return "-";
-        var span = DateTime.UtcNow - startedAt;
+        var span = DateTimeOffset.UtcNow - startedAt;
         if (span.TotalDays >= 1)
             return $"{(int)span.TotalDays}d {span.Hours}h";
         if (span.TotalHours >= 1)
@@ -1082,18 +1090,53 @@
     private void ParseStatusJson()
     {
         _parsedStatus = null;
+        _showVersionWarning = false;
         if (string.IsNullOrWhiteSpace(_status?.StatusJson)) return;
 
         try
         {
             _parsedStatus = JsonSerializer.Deserialize<ParsedDaemonStatus>(_status.StatusJson, _jsonOptions);
+            CheckVersionMismatch();
         }
         catch (Exception ex)
         {
             Logger.LogDebug(ex, "Failed to parse WAN Steering status JSON");
         }
     }
 
+    private void CheckVersionMismatch()
+    {
+        _showVersionWarning = false;
+        if (_parsedStatus == null) return;
+
+        _appVersion = Assembly.GetExecutingAssembly()
+            .GetCustomAttribute<AssemblyInformationalVersionAttribute>()
+            ?.InformationalVersion ?? "";
+
+        var binaryVersion = _parsedStatus.Version ?? "";
+
+        // Extract base version (X.Y.Z) from both, stripping v prefix, pre-release, and metadata.
+        // e.g., "v1.14.7" -> "1.14.7", "1.14.7-alpha.0.2+hash" -> "1.14.7", "dev" -> "dev"
+        var appBase = ExtractBaseVersion(_appVersion);
+        var binBase = ExtractBaseVersion(binaryVersion);
+
+        // Source builds produce 0.0.0 - suppress
+        if (appBase == "0.0.0") return;
+
+        // Binary is "dev" or a different base version: warn
+        if (binBase != appBase)
+        {
+            _showVersionWarning = true;
+        }
+    }
+
+    // Extracts X.Y.Z from version strings like "v1.14.7", "1.14.7-alpha.0.2+hash", "dev"
+    private static string ExtractBaseVersion(string version)
+    {
+        var match = Regex.Match(version, @"v?(\d+\.\d+\.\d+)");
+        return match.Success ? match.Groups[1].Value : version;
+    }
+
     private static readonly JsonSerializerOptions _jsonOptions = new()
     {
         PropertyNameCaseInsensitive = true,
@@ -1217,8 +1260,8 @@
     {
         public string Version { get; set; } = "";
         public bool Running { get; set; }
-        public DateTime StartedAt { get; set; }
-        public DateTime LastReconcile { get; set; }
+        public DateTimeOffset StartedAt { get; set; }
+        public DateTimeOffset LastReconcile { get; set; }
         public int RuleCount { get; set; }
         public int ReconcileCount { get; set; }
         public Dictionary<string, ParsedWanHealth> WanHealth { get; set; } = new();

src/NetworkOptimizer.Web/Services/WanSteerDeploymentService.cs

@@ -48,7 +48,7 @@ public async Task<WanSteerStatus> GetStatusAsync()
         {
             var combinedCommand =
                 "echo '---PROCESS---'; pgrep -x wansteer > /dev/null 2>&1 && echo running || echo stopped; " +
-                "echo '---STATUS---'; cat /tmp/wan-steer-status.json 2>/dev/null || echo '{}'; " +
+                "echo '---STATUS---'; cat /tmp/wan-steer-status.json 2>/dev/null || echo '{}'; echo; " +
                 "echo '---VERSION---'; /data/wan-steer/wansteer -version 2>/dev/null || echo 'not installed'; " +
                 "echo '---BINARY---'; test -x /data/wan-steer/wansteer && echo 'exists' || echo 'missing'";
 

src/wansteer/config.go

@@ -17,6 +17,27 @@ type Config struct {
 	HealthPassThreshold int                     `json:"health_pass_threshold"`
 	TrafficClasses      []TrafficClass          `json:"traffic_classes"`
 	StatusFile          string                  `json:"status_file"`
+
+	// Stability detection: prevent SFE kernel errors during WAN flapping.
+
+	// StartupGraceSeconds is how long to wait after start for WAN interfaces
+	// to stabilize before applying rules or starting health checks.
+	StartupGraceSeconds int `json:"startup_grace_seconds"`
+
+	// InstabilityThreshold is the number of state transitions within
+	// InstabilityWindowSeconds that marks a WAN as "unstable".
+	InstabilityThreshold int `json:"instability_threshold"`
+
+	// InstabilityWindowSeconds is the sliding window for counting state transitions.
+	InstabilityWindowSeconds int `json:"instability_window_seconds"`
+
+	// BackoffRecoverySeconds is how long all WANs must be stable before
+	// exiting backoff mode (single flush + full reapply on exit).
+	BackoffRecoverySeconds int `json:"backoff_recovery_seconds"`
+
+	// SFEFlushCooldownSeconds is the minimum interval between SFE flushes.
+	// Calls within the cooldown window are skipped.
+	SFEFlushCooldownSeconds int `json:"sfe_flush_cooldown_seconds"`
 }
 
 // WANInterface describes a WAN link the daemon can steer traffic to.
@@ -90,6 +111,21 @@ func loadConfig(path string) (*Config, error) {
 	if cfg.StatusFile == "" {
 		cfg.StatusFile = "/tmp/wan-steer-status.json"
 	}
+	if cfg.StartupGraceSeconds <= 0 {
+		cfg.StartupGraceSeconds = 30
+	}
+	if cfg.InstabilityThreshold <= 0 {
+		cfg.InstabilityThreshold = 3
+	}
+	if cfg.InstabilityWindowSeconds <= 0 {
+		cfg.InstabilityWindowSeconds = 300
+	}
+	if cfg.BackoffRecoverySeconds <= 0 {
+		cfg.BackoffRecoverySeconds = 60
+	}
+	if cfg.SFEFlushCooldownSeconds <= 0 {
+		cfg.SFEFlushCooldownSeconds = 10
+	}
 
 	return &cfg, nil
 }