Skip to content
This repository was archived by the owner on Nov 11, 2022. It is now read-only.

Version 1.3.0

Choose a tag to compare

@davorbonaci davorbonaci released this 06 Dec 22:02
· 1093 commits to master since this release
  • Improved IterableLikeCoder to efficiently encode small values. This change is backward compatible; however, if you have a running pipeline that was constructed with SDK version 1.3.0 or later, it may not be possible to "update" that pipeline with a replacement that was constructed using SDK version 1.2.1 or older. Updating a running pipeline with a pipeline constructed using a new SDK version, however, should be successful.
  • When TextIO.Write or AvroIO.Write outputs to a fixed number of files, added a reshard (shuffle) step immediately prior to the write step. The cost of this reshard is often exceeded by additional parallelism available to the preceding stage.
  • Added support for RFC 3339 timestamps in PubsubIO. This allows reading from Cloud Pub/Sub topics published by Cloud Logging without losing timestamp information.
  • Improved memory management to help prevent pipelines in the streaming execution mode from stalling when running with high memory utilization. This particularly benefits pipelines with large GroupByKey results.
  • Added ability to customize timestamps of emitted windows. Previously, the watermark was held to the earliest timestamp of any buffered input. With this change, you can choose a later time to allow the watermark to progress further. For example, using the end of the window will prevent long-lived sessions from holding up the output. See Window.Bound.withOutputTime().
  • Added a simplified syntax for early and late firings with an AfterWatermark trigger, as follows: AfterWatermark.pastEndOfWindow().withEarlyFirings(...).withLateFirings(...).