Watermark propagation stalls indefinitely in v1.7.5 Rust runtime due to frequent micro-regressions

[yuanzaiyang]

Describe the bug
Watermark propagation stalls indefinitely in Numaflow v1.7.5 (Rust runtime). Even with idleSource and maxDelay properly configured, the Source vertex frequently logs Watermark regression detected, skipping
publish, leading to a complete stop in watermark updates for downstream vertices.

This issue is particularly severe for Reduce (Aggregator) vertices, which become stuck in the STILL WAITING for window to close state indefinitely. The watermark on the Dashboard shows "Not Available" for
downstream edges of Aggregators, even when data is actively being processed. This behavior suggests a regression in how the new Rust data plane handles micro-jitters in data timestamps and gRPC signal
delivery.

To Reproduce
Steps to reproduce the behavior:

1. Deploy a pipeline with a Kafka Source and a Reduce/Aggregator vertex using Numaflow v1.7.5.
2. Configure a fixed window (e.g., 60s) and set maxDelay (e.g., 120s) and idleSource.
3. Feed a Kafka topic with data across multiple partitions where slight timestamp jitters (sub-second regressions) occur naturally.
4. Observe the Source vertex logs: it frequently skips watermark publishing due to "regression detected".
5. Observe the Aggregator vertex: it logs STILL WAITING for window to close but never receives the None signal to close the window, despite the wall time being well past the window_end.
6. The UI shows Watermark: Not Available on the edge following the Aggregator.

Expected behavior

1. Watermarks should advance monotonically, utilizing maxDelay to absorb micro-jitters without skipping publishing entirely.
2. idleSource should force watermark advancement even if some partitions are slow or have slight regressions.
3. Reduce windows should close and propagate watermarks to downstream sinks once the watermark (adjusted by lateness) exceeds the window end time.

Screenshots

Environment (please complete the following information):

• Kubernetes: v1.34
• Numaflow: v1.7.5
• Data Plane Runtime: Rust
• SDK: Rust SDK v0.5.0

Additional context

1. Micro-Regression Sensitivity: The system seems extremely sensitive to even small regressions (e.g., < 500ms), which causes the Source to stop publishing watermarks.
2. Resource Impact: Increasing CPU/Memory resources and adjusting NATS maxackpending did not resolve the stall, suggesting a logic error in the Rust-based watermark progressor or gRPC stream handling.
3. Consistency: The issue is reproducible across multiple independent pipelines, including those with only Map vertices, where downstream watermarks also intermittently show "Not Available".
4. Window Stall: In Aggregator UDF logs, we see thousands of STILL WAITING entries, confirming that the window closure signal is never sent by the Sidecar when the watermark is stuck at the source.

[syayi]

Thanks @yuanzaiyang for the detailed report, valid feedback, and we're looking into it.

While we dig in, we'd love to learn more about how you're using Numaflow. What does your use case look like? (e.g., what kind of data are you processing, scale/throughput, etc.)

We're always looking to better understand how the community is using Numaflow in the real world

[vaibhavtiwari33]

@yuanzaiyang Could you please also provide me the specific idleSource config you're using to allow me to reproduce this issue accurately.

[yuanzaiyang]

谢谢@yuanzaiyang详细报告和有效反馈意见，我们正在调查中。

在深入研究的过程中，我们很想了解您是如何使用 Numaflow 的。您的使用场景是怎样的？（例如，您正在处理什么类型的数据，规模/吞吐量等等。）

我们一直在努力更好地了解社区如何在实际应用中使用Numaflow。

Usage Scenario
We are using Numaflow as a high-performance Observability Pipeline for our large-scale E-commerce and Logistics platform. The primary goal is to perform real-time log analysis, statistics aggregation, and
exception pattern detection from thousands of microservices (e.g., Order Management, TMS, Pricing, and Social MQ services).

Data Type & Architecture

• Data Format: We process high-volume OTLP (OpenTelemetry) logs and structured Java application logs.
• Pipeline Flow: Our typical architecture is Kafka Source -> Transformer (Rust) -> Aggregators (Reduce/Windowing) -> Kafka Sink.
• Transformation: We use the Numaflow Rust SDK to perform complex transformations, log redaction, and metadata extraction.
• Aggregation: We use Fixed Windows (typically 1 min) to calculate per-service throughput, error rates, and to perform "Drain" log template clustering.

Scale & Throughput

• Throughput: Currently, we handle around 1,000 - 2,000 messages per second per vertex on average, with significant spikes during peak hours.
• Window Volume: A single 1-minute window for some high-traffic services can aggregate over 100,000 logs.
• Infrastructure: Running on AWS EKS (EC2 nodes).

[yuanzaiyang]

idleSource

Specific idleSource Configuration

Here is the exact watermark configuration we used in our YAML:

1 watermark:
2 disabled: false
3 # Initially we used 0s, but we also tested with 30s/60s/120s.
4 # Even with higher maxDelay, the stall persists.
5 maxDelay: 0s
6 idleSource:
7 threshold: 30s
8 stepInterval: 10s
9 incrementBy: 10s

Important Observation: Windowed vs. Non-Windowed Pipelines

To help narrow down the root cause, I want to emphasize that we encountered this issue in two different types of pipelines:

1. Non-Windowed Pipeline (e.g., Istio Gateway Logs)

• Architecture: Kafka Source -> Map UDF -> Kafka Sink.
• Behavior: Data flows correctly through the Map node to the Sink (throughput is normal). However, the Watermark is "Not Available" on the edge between the Map and the Sink.
• Impact: Even though the sink is functional, the lack of watermark propagation makes it impossible to monitor Max Lag or use it for downstream time-sensitive logic.

2. Windowed Pipeline (e.g., Application Logs Aggregator)

• Architecture: Kafka Source -> Reduce UDF (1 min Fixed Window) -> Kafka Sink.
• Behavior: Due to the same watermark stall at the source, the Window never closes.
• Impact: Because the window remains open indefinitely (stuck in STILL WAITING for window to close), no data is ever sent to the Kafka Sink. The data simply accumulates in the inter-step buffer (ISB) until
  it hits the maxmsgs limit.

In my non-windowed pipeline, the data isn't blocked because it's a stateless Map, but in the windowed pipeline, the missing watermark is fatal.

[vigith]

@yuanzaiyang do you mind running against our nightly build (quay.io/numaproj/numaflow:nightly-20260513) which has few fixes on watermark progression esp. with Kafka?