I'm experiencing a situation when an incoming message isn't acknowledged (therefore in sometime broker resend it) and the watermark is not updated while new messages are coming continuously.
After some time I've discovered that this situation is related to the fact that finalizaCheckpoint is not being called.
I took a look at the Pubsub IO implementation and found that they expect such situation and do not compute watermark and ack messages in finalizeCheckpoint. Here is the comment about that: https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L289
Should MQTT IO do the same?
Imported from Jira BEAM-3696. Original Jira may contain additional context.
Reported by: kolchinmax.
I'm experiencing a situation when an incoming message isn't acknowledged (therefore in sometime broker resend it) and the watermark is not updated while new messages are coming continuously.
After some time I've discovered that this situation is related to the fact that finalizaCheckpoint is not being called.
I took a look at the Pubsub IO implementation and found that they expect such situation and do not compute watermark and ack messages in finalizeCheckpoint. Here is the comment about that: https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L289
Should MQTT IO do the same?
Imported from Jira BEAM-3696. Original Jira may contain additional context.
Reported by: kolchinmax.