What happened?
I've been trying out the WriteToPubSub option in a python Dataflow job, with a formatting step like this:
yield PubsubMessage(
data=json.dumps(dataclasses.asdict(my_data)).encode('utf-8'),
ordering_key=ordering_key,
attributes={
"timestamp": str(my_data.timestamp),
}
)
right before the write step, where the call to WriteToPubSub looks as follows:
WriteToPubSub(
topic='projects/my_project/topics/my_topic_name',
with_attributes=True)
What I observe is that my job succeeds, but no messages are published. There are also no errors displayed in either the job logs or worker logs.
I've confirmed I can publish the messages in the same Dataflow job if I write my own Pub/Sub writer transform that uses the python client library for Pub/Sub, so I believe that rules out issues related to service account permissions, networking, the size of the messages, or a misconfigured topic. That's fine for now but I'd prefer to use the official WriteToPubSub transform.
If my code above looks correct, is there a way to expose errors that occur in WriteToPubSub so I can debug this more effectively?
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
What happened?
I've been trying out the WriteToPubSub option in a python Dataflow job, with a formatting step like this:
right before the write step, where the call to WriteToPubSub looks as follows:
What I observe is that my job succeeds, but no messages are published. There are also no errors displayed in either the job logs or worker logs.
I've confirmed I can publish the messages in the same Dataflow job if I write my own Pub/Sub writer transform that uses the python client library for Pub/Sub, so I believe that rules out issues related to service account permissions, networking, the size of the messages, or a misconfigured topic. That's fine for now but I'd prefer to use the official WriteToPubSub transform.
If my code above looks correct, is there a way to expose errors that occur in WriteToPubSub so I can debug this more effectively?
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components