deepen esp8266 http path

Author: ZeroBitsTech

CHANGELOG.md

@@ -10,11 +10,13 @@
 
 - Added `ZeroNetProfileEsp8266`, a constrained MQTT-first preset for Wemos / ESP8266.
 - Added `ZeroNetProfileEsp8266Http`, an opt-in HTTP-first constrained preset for Wemos / ESP8266 so HTTP and MQTT can be tuned separately instead of forcing one compromise path.
+- Hardened `ZeroHttpPump` with optional phase-specific timeout budgets and configurable intra-tick phase progress, then used that in the ESP8266 HTTP-first preset so the constrained Wemos path can progress connect/write faster without changing the global default pacing.
 - Refined `ZeroNetProfileEsp8266` so HTTP stays truly off by default, MQTT idle churn is lower, and the official Wemos live compare now shows MQTT delivery with timing that beats the naive baseline in the same window.
 - Added optional offline queue gating in `ZeroHttpPump` and `ZeroMqttPump` so constrained boards can refuse backlog when link or transport is down.
 - Reduced scheduler contention in the ESP8266 preset by staggering network task start offsets and lowering idle network task cadence.
 - Updated the ESP8266 preset to recommend `kIdleYield` instead of `kIdleSleep`, which materially reduces live timing jitter in the official Wemos validation node.
 - Tuned the ESP8266 preset toward MQTT-first delivery with lighter MQTT dispatch pressure, producing a repeatable Wemos run where MQTT delivery stays live while timing remains at or better than baseline.
+- Improved the ESP8266 HTTP-first preset so live HTTP delivery is now real and controlled (`http_rate` can hit `100%` in the official compare) while average lag drops sharply versus the naive baseline; the path remains experimental because worst-case lag and misses still need more cleanup.
 - Added generic compare workloads for `EnvMonitor`, `TelemetryGateway`, and `IndustrialLoop`.
 - Added repeatable ESP32 compare runners plus a cross-target workload compile matrix.
 - Improved telemetry gateway tuning so module throughput rises without queue buildup.

README.md

@@ -119,7 +119,7 @@ Recommended board-specific path:
 
 - **ESP32:** use the default network module configs first, then validate against your real endpoint.
 - **ESP8266 / Wemos:** start with `ZeroNetProfileEsp8266`. It is a constrained MQTT-first preset that disables periodic HTTP dispatch by default, prevents offline queue buildup, lowers idle MQTT churn, staggers lighter network task cadence, and recommends a lighter idle strategy. In current validation it is the preferred path for Wemos MQTT delivery, while HTTP remains degraded/off by default unless you deliberately opt back in.
-- **ESP8266 / Wemos (HTTP-specific):** use `ZeroNetProfileEsp8266Http` only when your node is intentionally HTTP-first. It keeps the same constrained timing discipline, but does not try to combine HTTP and MQTT in the same default path. Treat it as an opt-in experimental profile until your own endpoint validation confirms live delivery under your Wi-Fi conditions.
+- **ESP8266 / Wemos (HTTP-specific):** use `ZeroNetProfileEsp8266Http` only when your node is intentionally HTTP-first. It keeps the same constrained timing discipline, but does not try to combine HTTP and MQTT in the same default path. Current validation now shows real HTTP delivery with clean request success under the official local compare, but it is still marked experimental because misses and worst-case lag are not yet as tidy as the MQTT-first constrained path.
 
 Current best module tradeoff reference (ESP32, `LEAN_NET`, manual pattern vs module pattern):
 

docs/wiki/Beta-Modules.md

@@ -19,7 +19,7 @@ They are already useful and validated on desktop plus ESP32 smoke tests, but the
 Recommended default:
 
 - For constrained ESP8266 boards, start with `ZeroNetProfileEsp8266`. It is the recommended MQTT-first constrained preset: HTTP stays degraded/off by default, offline queueing is rejected, idle MQTT churn is lower, and the preset uses a lighter recommended idle strategy plus staggered network task starts so the board stays more predictable without hand-tuning every interval.
-- If your board is intentionally HTTP-first, use `ZeroNetProfileEsp8266Http` as a separate opt-in path. It keeps the constrained timing discipline but avoids pretending that one default cadence can serve HTTP-heavy and MQTT-heavy nodes equally well. Treat it as experimental until your own endpoint validation confirms live delivery.
+- If your board is intentionally HTTP-first, use `ZeroNetProfileEsp8266Http` as a separate opt-in path. It keeps the constrained timing discipline but avoids pretending that one default cadence can serve HTTP-heavy and MQTT-heavy nodes equally well. The official Wemos compare now shows real HTTP delivery on this path, but it is still experimental because miss count and worst-case lag remain rougher than the MQTT-first constrained preset.
 
 ## What BETA Means Here
 

examples/LiveNetworkNode/LiveNetworkNode.ino

@@ -99,6 +99,17 @@ bool isWiFiConnected() {
   return WiFi.status() == WL_CONNECTED;
 }
 
+bool isHttpTransportReady() {
+#if defined(ARDUINO_ARCH_ESP8266)
+  if (!g_wifiMaintainer.isConnected()) {
+    return false;
+  }
+  return g_wifiMaintainer.stablePolls() >= ZeroNetActiveEsp8266Profile::kHttpStablePollsRequired;
+#else
+  return isWiFiConnected();
+#endif
+}
+
 void connectWiFi() {
   const wl_status_t status = WiFi.status();
   if (status == WL_CONNECTED) {
@@ -299,8 +310,8 @@ void dispatchTask() {
                        (nowMs - g_lastHttpDispatchAtMs) >= kHttpDispatchPeriodMs;
   const bool mqttDue = (nowMs - g_lastMqttDispatchAtMs) >= kMqttDispatchPeriodMs;
 
-  if (httpDue && g_httpPump.queuedCount() == 0 && !g_httpPump.isBusy()) {
-    g_lastHttpDispatchAtMs = nowMs;
+  if (httpDue && g_httpPump.queuedCount() == 0 && !g_httpPump.isBusy() &&
+      isHttpTransportReady()) {
     const int written = snprintf(g_httpPayload, sizeof(g_httpPayload),
                                  "{\"seq\":%lu,\"sensor\":%lu,\"board\":\"module\"}",
                                  g_sampleRuns, g_sensorValue);
@@ -310,7 +321,9 @@ void dispatchTask() {
       request.contentType = "application/json";
       request.body = g_httpPayload;
       request.bodyLength = static_cast<uint16_t>(written);
-      g_httpPump.enqueue(request);
+      if (g_httpPump.enqueue(request)) {
+        g_lastHttpDispatchAtMs = nowMs;
+      }
     }
   }
 
@@ -440,7 +453,7 @@ void setup() {
                      httpReadStep,
                      httpCloseStep,
                      httpConfig);
-    g_httpPump.setLinkProbe(isWiFiConnected);
+    g_httpPump.setLinkProbe(isHttpTransportReady);
   }
 
   ZeroMqttPump::Config mqttConfig;

src/modules/net/ZeroHttpPump.h

@@ -55,7 +55,12 @@ class ZeroHttpPump {
     unsigned long retryMaxMs;
     unsigned long retryJitterMs;
     unsigned long phaseTimeoutMs;
+    unsigned long connectPhaseTimeoutMs;
+    unsigned long writePhaseTimeoutMs;
+    unsigned long readPhaseTimeoutMs;
+    unsigned long closePhaseTimeoutMs;
     uint8_t maxRetries;
+    uint8_t immediatePhaseBudget;
     bool dropOldestOnFull;
     bool queueWhenLinkDown;
     bool emitCompletionEvents;
@@ -78,14 +83,18 @@ class ZeroHttpPump {
           retryJitterMs(100),
           phaseTimeoutMs(300),
 #endif
+          connectPhaseTimeoutMs(0),
+          writePhaseTimeoutMs(0),
+          readPhaseTimeoutMs(0),
+          closePhaseTimeoutMs(0),
           maxRetries(2),
+          immediatePhaseBudget(1),
           dropOldestOnFull(true),
           queueWhenLinkDown(true),
           emitCompletionEvents(true) {}
   };
 
   static const uint8_t kQueueCapacity = ZEROKERNEL_HTTP_PUMP_QUEUE_CAPACITY;
-  static const uint8_t kImmediatePhaseBudget = 1;
 
   ZeroHttpPump()
       : kernel_(NULL),
@@ -197,7 +206,9 @@ class ZeroHttpPump {
     hasTicked_ = true;
     lastTickAtMs_ = nowMs;
 
-    for (uint8_t stepBudget = 0; stepBudget < kImmediatePhaseBudget; ++stepBudget) {
+    const uint8_t immediatePhaseBudget =
+        config_.immediatePhaseBudget == 0 ? 1 : config_.immediatePhaseBudget;
+    for (uint8_t stepBudget = 0; stepBudget < immediatePhaseBudget; ++stepBudget) {
       if (!active_) {
         if (queueCount_ == 0) {
           return;
@@ -219,8 +230,9 @@ class ZeroHttpPump {
         return;
       }
 
-      if (config_.phaseTimeoutMs > 0 && phaseStartedAtMs_ != 0 &&
-          (nowMs - phaseStartedAtMs_) > config_.phaseTimeoutMs) {
+      const unsigned long phaseTimeoutMs = currentPhaseTimeoutMs_();
+      if (phaseTimeoutMs > 0 && phaseStartedAtMs_ != 0 &&
+          (nowMs - phaseStartedAtMs_) > phaseTimeoutMs) {
         metrics_.recordPhaseTimeout();
         handlePhaseFailure_(nowMs);
         return;
@@ -400,6 +412,22 @@ class ZeroHttpPump {
     return (nowMs + retrySalt_ + activeRequest_.queuedAtMs) % (config_.retryJitterMs + 1UL);
   }
 
+  unsigned long currentPhaseTimeoutMs_() const {
+    if (phase_ == kPhaseConnecting && config_.connectPhaseTimeoutMs > 0) {
+      return config_.connectPhaseTimeoutMs;
+    }
+    if (phase_ == kPhaseWriting && config_.writePhaseTimeoutMs > 0) {
+      return config_.writePhaseTimeoutMs;
+    }
+    if (phase_ == kPhaseReading && config_.readPhaseTimeoutMs > 0) {
+      return config_.readPhaseTimeoutMs;
+    }
+    if (phase_ == kPhaseClosing && config_.closePhaseTimeoutMs > 0) {
+      return config_.closePhaseTimeoutMs;
+    }
+    return config_.phaseTimeoutMs;
+  }
+
   Kernel* kernel_;
   LinkProbe linkProbe_;
   StepCallback connectStep_;

src/modules/net/ZeroNetProfileEsp8266.h

@@ -30,6 +30,7 @@ struct ZeroNetProfileEsp8266 {
   static const unsigned long kHttpDispatchPeriodMs = 0UL;
   static const unsigned long kMqttDispatchPeriodMs = 1000UL;
   static const unsigned long kHttpIoTimeoutMs = 200UL;
+  static const uint8_t kHttpStablePollsRequired = 0;
 
   static ZeroWiFiMaintainer::Config wifiConfig() {
     ZeroWiFiMaintainer::Config config;

src/modules/net/ZeroNetProfileEsp8266Http.h

@@ -18,7 +18,7 @@ struct ZeroNetProfileEsp8266Http {
   static const bool kEnableMqttByDefault = false;
   static const unsigned long kSampleTaskIntervalMs = 100UL;
   static const unsigned long kWiFiTaskIntervalMs = 250UL;
-  static const unsigned long kHttpTaskIntervalMs = 250UL;
+  static const unsigned long kHttpTaskIntervalMs = 100UL;
   static const unsigned long kMqttTaskIntervalMs = 500UL;
   static const unsigned long kDispatchTaskIntervalMs = 250UL;
   static const unsigned long kReportTaskIntervalMs = 1000UL;
@@ -30,7 +30,8 @@ struct ZeroNetProfileEsp8266Http {
 
   static const unsigned long kHttpDispatchPeriodMs = 1500UL;
   static const unsigned long kMqttDispatchPeriodMs = 0UL;
-  static const unsigned long kHttpIoTimeoutMs = 200UL;
+  static const unsigned long kHttpIoTimeoutMs = 450UL;
+  static const uint8_t kHttpStablePollsRequired = 1;
 
   static ZeroWiFiMaintainer::Config wifiConfig() {
     ZeroWiFiMaintainer::Config config;
@@ -50,7 +51,12 @@ struct ZeroNetProfileEsp8266Http {
     config.retryMaxMs = 3200UL;
     config.retryJitterMs = 180UL;
     config.phaseTimeoutMs = 300UL;
+    config.connectPhaseTimeoutMs = 450UL;
+    config.writePhaseTimeoutMs = 100UL;
+    config.readPhaseTimeoutMs = 650UL;
+    config.closePhaseTimeoutMs = 75UL;
     config.maxRetries = 1;
+    config.immediatePhaseBudget = 2;
     config.queueWhenLinkDown = false;
     return config;
   }

src/modules/net/ZeroWiFiMaintainer.h

@@ -88,6 +88,7 @@ class ZeroWiFiMaintainer {
     connectAttempts_ = 0;
     reconnectTransitions_ = 0;
     stateNotifications_ = 0;
+    consecutiveStablePolls_ = 0;
   }
 
   void reset() {
@@ -176,6 +177,22 @@ class ZeroWiFiMaintainer {
     return connected_;
   }
 
+  bool isStable() const {
+    if (!connected_) {
+      return false;
+    }
+
+    if (config_.stableThreshold == 0) {
+      return true;
+    }
+
+    return consecutiveStablePolls_ >= config_.stableThreshold;
+  }
+
+  uint8_t stablePolls() const {
+    return consecutiveStablePolls_;
+  }
+
   unsigned long connectAttempts() const {
     return connectAttempts_;
   }