Skip to content

Commit fcb71ac

Browse files
authored
fix: support setting location in BigQuery Airflow op (#1558)
1 parent f8e45bb commit fcb71ac

2 files changed

Lines changed: 24 additions & 2 deletions

File tree

docs/integrations/engines/bigquery.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ pip install "sqlmesh[bigquery]"
4646

4747
### Connection info
4848

49-
The operator requires an [Airflow connection](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html) to determine the target BigQuery account. Please see [GoogleBaseHook](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/common/hooks/base_google/index.html#airflow.providers.google.common.hooks.base_google.GoogleBaseHook) and [GCP connection](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html)for more details. The goal is to create a working GCP connection on Airflow. The only difference is that you should use the `sqlmesh_google_cloud_bigquery_default` (by default) connection ID instead of the `google_cloud_default` one in the Airflow guide.
49+
The operator requires an [Airflow connection](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html) to determine the target BigQuery account. Please see [GoogleBaseHook](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/common/hooks/base_google/index.html#airflow.providers.google.common.hooks.base_google.GoogleBaseHook) and [GCP connection](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html)for more details. Use the `sqlmesh_google_cloud_bigquery_default` (by default) connection ID instead of the `google_cloud_default` one in the Airflow guide.
5050

5151
By default, the connection ID is set to `sqlmesh_google_cloud_bigquery_default`, but it can be overridden using the `engine_operator_args` parameter to the `SQLMeshAirflow` instance as in the example below:
5252
```python linenums="1"
@@ -58,6 +58,20 @@ sqlmesh_airflow = SQLMeshAirflow(
5858
)
5959
```
6060

61+
#### Optional Arguments
62+
63+
* `location`: Sets the default location for datasets and tables. If not set, BigQuery defaults to US for new datasets. See `location` in [Connection options](#connection-options) for more details.
64+
65+
```python linenums="1"
66+
sqlmesh_airflow = SQLMeshAirflow(
67+
"bigquery",
68+
engine_operator_args={
69+
"bigquery_conn_id": "<Connection ID>",
70+
"location": "<location>"
71+
},
72+
)
73+
```
74+
6175
## Connection Methods
6276
- [oauth](https://google-auth.readthedocs.io/en/master/reference/google.auth.html#google.auth.default) (default)
6377
- Related Credential Configuration:

sqlmesh/schedulers/airflow/hooks/bigquery.py

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ def __init__(
3131
gcp_conn_id: str = default_conn_name,
3232
delegate_to: str | None = None,
3333
impersonation_chain: str | t.Sequence[str] | None = None,
34+
location: t.Optional[str] = None,
3435
**kwargs: t.Any,
3536
) -> None:
3637
GoogleBaseHook.__init__(
@@ -39,9 +40,11 @@ def __init__(
3940
delegate_to=delegate_to,
4041
impersonation_chain=impersonation_chain,
4142
)
43+
self.location = location
4244

4345
def get_conn(self) -> Connection:
4446
"""Returns a BigQuery DBAPI connection object."""
47+
from google.api_core.client_info import ClientInfo
4548
from google.cloud.bigquery import Client
4649
from google.cloud.bigquery.dbapi import Connection
4750

@@ -50,5 +53,10 @@ def get_conn(self) -> Connection:
5053
creds, project_id = self._get_credentials_and_project_id() # type: ignore
5154
except AttributeError:
5255
creds, project_id = self.get_credentials_and_project_id() # type: ignore
53-
client = Client(project=project_id, credentials=creds)
56+
client = Client(
57+
project=project_id,
58+
credentials=creds,
59+
location=self.location,
60+
client_info=ClientInfo(user_agent="sqlmesh"),
61+
)
5462
return Connection(client=client)

0 commit comments

Comments
 (0)