I’m using BigQuery as a cache to load data into BigQuery directly (as recommended in the documentation). This avoids the unnecessary overhead of creating a local cache.
However, the current implementation hardcodes the dataset location to "US" and doesn’t provide a way to change it. This becomes a problem when your existing datasets are in other locations, since BigQuery doesn’t support querying across regions.
Here's the relevant code:
def bigquery_cache_to_destination_configuration(
cache: BigQueryCache,
) -> DestinationBigquery:
"""Get the destination configuration from the BigQuery cache."""
credentials_json: str | None = (
SecretString(Path(cache.credentials_path).read_text(encoding="utf-8"))
if cache.credentials_path
else None
)
return DestinationBigquery(
project_id=cache.project_name,
dataset_id=cache.dataset_name,
dataset_location="US", # <-- Hardcoded
credentials_json=credentials_json,
loading_method=BatchedStandardInserts(),
)Let's make dataset_location configurable, either by accepting it from the BigQueryConfig object or exposing it as a parameter in the cache setup. This would allow better compatibility with non-US datasets and avoid location mismatch errors.
I’m using BigQuery as a cache to load data into BigQuery directly (as recommended in the documentation). This avoids the unnecessary overhead of creating a local cache.
However, the current implementation hardcodes the dataset location to "US" and doesn’t provide a way to change it. This becomes a problem when your existing datasets are in other locations, since BigQuery doesn’t support querying across regions.
Here's the relevant code:
Let's make
dataset_locationconfigurable, either by accepting it from theBigQueryConfigobject or exposing it as a parameter in the cache setup. This would allow better compatibility with non-US datasets and avoid location mismatch errors.