Skip to content

Commit ac7d761

Browse files
committed
[Python] Bound Watch state with a timestamp cursor
Follow-up to the experimental Watch transform that documents the event-time contract, makes late data observable, and adds an opt-in O(1) bound on the per-input state. Watermark/event-time: - Document the event-time/watermark contract on the module and PollResult. - Warn (throttled) when an output is emitted behind the current watermark, so out-of-order late data is observable instead of silently dropped, and point to PollResult.with_watermark for out-of-order sources. Scalability: - Add opt-in Watch.with_timestamp_cursor(): dedup by a high-water-mark timestamp instead of by value identity. Watch keeps only the greatest event time emitted per input and emits the polled outputs strictly past it, so the per-input state is a single timestamp that never grows and the poll result is not hashed, giving O(1) state and per-checkpoint encoding. For sources whose outputs carry strictly increasing event-time timestamps; an output at or below the cursor is treated as already seen, so out-of-order or equal-timestamp outputs are dropped (use the default exact dedup otherwise). - Reuse the parent dedup map on idle rounds to drop the O(N) per-round copy. Tests cover the watermark contract, the out-of-order late warning, cursor emit-after-high-water-mark, O(1) state, relist-once, complete, cursor coder round-trip, and end-to-end cursor dedup. 26 tests and 5 subtests pass.
1 parent a577bb6 commit ac7d761

2 files changed

Lines changed: 472 additions & 38 deletions

File tree

0 commit comments

Comments
 (0)