Unable to subscribe to several PubSub subscriptions with a valueprovider

[damccorm]

Deploying as a template a streaming job in Google Dataflow, I can't create more than one PubSub subscriptions, when subscription name is determined at runtime (with a ValueProvider):

      final List<PCollection<String>> pCollections = new ArrayList<>();
        for (final String
topic : topics) {
            final ValueProvider<String> vpSub = ValueProvider.NestedValueProvider.of(options.getSubscriptionPrefix(),

                   prefix -> prefix + topic);
            PCollection<String> messages =
        
           pipeline.apply("Sub-read-" + topic, PubsubIO.readStrings().fromSubscription(vpSub))
   
                        .apply("Windowing", Window.into(duration));
            pCollections.add(messages);

       }

During the launching of the job, I got the error in the logs:

Workflow failed. Causes: The pubsub configuration contains errors: Subscription 'projects/_project_/subscriptions/_prefix_'
is consumed by multiple stages, this will result in undefined behavior.

It seems that the coded SerializableFunction that I provide to the instances of NestedValueProvider are not called.

It could be due to https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java  StreamingPubsubIOReadTranslator#translate, especially these lines :

      if (overriddenTransform.getSubscriptionProvider() != null) {
        if (overriddenTransform.getSubscriptionProvider().isAccessible())
{
          stepContext.addInput(
              PropertyNames.PUBSUB_SUBSCRIPTION,
             
overriddenTransform.getSubscription().getV1Beta1Path());
        } else {
          stepContext.addInput(

             PropertyNames.PUBSUB_SUBSCRIPTION_OVERRIDE,
              ((NestedValueProvider) overriddenTransform.getSubscriptionProvider()).propertyName());

       }
      }

Indeed this code :

• Forces to use a NestedValueProvider if value is not accessible before runtime, due to the vehement cast,
• Results in having the same incorrect subscription value at runtime (so only the prefix in my case) for all the reads of the loop, at least at validation time

 

Seems to be the same issue reported there : https://issuetracker.google.com/u/0/issues/157584222

Imported from Jira BEAM-11635. Original Jira may contain additional context.
Reported by: Jean-Gabriel.

[JavaIlya]

Hi @damccorm we've faced this bug in our commercial project actually, we have a workaround for it by providing subs through a runtime value provider directly, but I thought mb I could check this thing, and after investigating for a while you are actually right, this part of code is causing this thing, but I don't understand how does it work, thought mb you as apache beam commiter have some pointers for me.

1. This property names in steps, they are provided but I don't see where is it used, it looks like it's some other code?
2. There's some code below "PropertyNames.PUBSUB_SERIALIZED_ATTRIBUTES_FN" where we serialize a function which I suppose will be called later, shouldn't we use simillair approach for this thing?

Or mb if you raised it and never fixed it's for some reason not possible by design, idk, would be happy to have your answer, if it's possible for me to help I would be also happy to do so, but for now I'm stuck