Commit 71d6282
committed
[Python] Bound Watch state with a timestamp cursor
Follow-up to the experimental Watch transform that documents the event-time
contract, makes late data observable, and adds an opt-in O(1) bound on the
per-input state.
Watermark/event-time:
- Document the event-time/watermark contract on the module and PollResult.
- Warn (throttled) when an output is emitted behind the current watermark, so
out-of-order late data is observable instead of silently dropped, and point
to PollResult.with_watermark for out-of-order sources.
Scalability:
- Add opt-in Watch.with_timestamp_cursor(): dedup by a high-water-mark timestamp
instead of by value identity. Watch keeps only the greatest event time emitted
per input and emits the polled outputs strictly past it, so the per-input
state is a single timestamp that never grows and the poll result is not
hashed, giving O(1) state and per-checkpoint encoding. For sources whose
outputs carry strictly increasing event-time timestamps; an output at or below
the cursor is treated as already seen, so out-of-order or equal-timestamp
outputs are dropped (use the default exact dedup otherwise).
- Reuse the parent dedup map on idle rounds to drop the O(N) per-round copy.
Tests cover the watermark contract, the out-of-order late warning, cursor
emit-after-high-water-mark, O(1) state, relist-once, complete, cursor coder
round-trip, and end-to-end cursor dedup. 26 tests and 5 subtests pass.1 parent 9acab87 commit 71d6282
2 files changed
Lines changed: 472 additions & 38 deletions
0 commit comments