This repository was archived by the owner on Nov 11, 2022. It is now read-only.
Version 1.3.0
- Improved
IterableLikeCoderto efficiently encode small values. This change is backward compatible; however, if you have a running pipeline that was constructed with SDK version 1.3.0 or later, it may not be possible to "update" that pipeline with a replacement that was constructed using SDK version 1.2.1 or older. Updating a running pipeline with a pipeline constructed using a new SDK version, however, should be successful. - When
TextIO.WriteorAvroIO.Writeoutputs to a fixed number of files, added a reshard (shuffle) step immediately prior to the write step. The cost of this reshard is often exceeded by additional parallelism available to the preceding stage. - Added support for RFC 3339 timestamps in
PubsubIO. This allows reading from Cloud Pub/Sub topics published by Cloud Logging without losing timestamp information. - Improved memory management to help prevent pipelines in the streaming execution mode from stalling when running with high memory utilization. This particularly benefits pipelines with large
GroupByKeyresults. - Added ability to customize timestamps of emitted windows. Previously, the watermark was held to the earliest timestamp of any buffered input. With this change, you can choose a later time to allow the watermark to progress further. For example, using the end of the window will prevent long-lived sessions from holding up the output. See
Window.Bound.withOutputTime(). - Added a simplified syntax for early and late firings with an
AfterWatermarktrigger, as follows:AfterWatermark.pastEndOfWindow().withEarlyFirings(...).withLateFirings(...).