Commit 0cd6937
committed
[Python] Add Watch transform with growth_of polling SDF
Add an experimental Watch transform that watches a growing set of outputs
per input element. Watch.growth_of(poll_fn) runs a periodic poll loop as a
splittable DoFn and emits an unbounded PCollection of (input, output) pairs.
Each process() performs one poll round and self-checkpoints via
defer_remainder. New outputs are deduplicated with a stable 128-bit blake2b
hash of the encoded output, and a manual watermark estimator advances per
poll. Per-input termination supports never() and after_total_of(); polling
also stops when a poll returns PollResult.complete(). The DoFn is its own
RestrictionProvider, and restriction state serializes through a tagged
GrowthState coder.
Tests cover termination conditions, coder round-trips, the restriction
tracker claim/checkpoint/dedup logic, and DirectRunner end-to-end runs.1 parent 8801afd commit 0cd6937
2 files changed
Lines changed: 1078 additions & 0 deletions
0 commit comments