[Bug]: Dataflow keeps retrying to obtain an already deleted BigQuery dataset

[fozzie15]

What happened?

max retry is not applying to getDatasetLocation() and attempts to retry indefinitely.

Max retries is seen in the stack trace here at 9, but beam does not let go after the 9 retries, and the logs are filled with continuous attempts to getDatasetLocation (see below.) This is a singular instance of the stack trace that repeats indefinitely, hanging the job:

Error message from worker: java.lang.RuntimeException: unable to obtain dataset for dataset nonexistent_dataset in project <project_id>
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.getDatasetLocation(BigQueryHelpers.java:598)
org.apache.beam.sdk.io.gcp.bigquery.WriteTables.startLoad(WriteTables.java:506)
org.apache.beam.sdk.io.gcp.bigquery.WriteTables.access$700(WriteTables.java:97)
org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:277)
Caused by: java.io.IOException: Unable to get dataset: nonexistent_dataset, aborting after 9 retries.
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.executeWithRetries(BigQueryServicesImpl.java:1476)

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Suvrat1629]

@fozzie15 @liferoad Is this issue available to work on could I take a stab at this?

[liferoad]

@fozzie15 @liferoad Is this issue available to work on could I take a stab at this?

Sounds good. Feel free to take the issue (https://beam.apache.org/contribute/). Since the issue is a little old, I suggest you first verify whether this still occurs with the latest Beam.

[github-actions]

This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.