You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Move license check from init() to constructor.
Reorder config fields logically (core fields first, advanced last).
Use sentence-per-line in config descriptions for easier diffs.
Add newTestOutput helper for tests.
Rewrite test comments as given/when/then.
Inject test license via license.InjectTestService.
Regenerate docs.
Copy file name to clipboardExpand all lines: docs/modules/components/pages/outputs/gcp_bigquery_write_api.adoc
+93-94Lines changed: 93 additions & 94 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -43,13 +43,13 @@ output:
43
43
dataset: "" # No default (required)
44
44
table: "" # No default (required)
45
45
message_format: json
46
-
credentials_json: ""
47
46
max_in_flight: 64
48
47
batching:
49
48
count: 0
50
49
byte_size: 0
51
50
period: ""
52
51
check: ""
52
+
credentials_json: ""
53
53
```
54
54
55
55
--
@@ -66,6 +66,13 @@ output:
66
66
dataset: "" # No default (required)
67
67
table: "" # No default (required)
68
68
message_format: json
69
+
max_in_flight: 64
70
+
batching:
71
+
count: 0
72
+
byte_size: 0
73
+
period: ""
74
+
check: ""
75
+
processors: [] # No default (optional)
69
76
credentials_json: ""
70
77
target_principal: ""
71
78
delegates: []
@@ -74,31 +81,23 @@ output:
74
81
endpoint:
75
82
http: ""
76
83
grpc: ""
77
-
max_in_flight: 64
78
-
batching:
79
-
count: 0
80
-
byte_size: 0
81
-
period: ""
82
-
check: ""
83
-
processors: [] # No default (optional)
84
84
```
85
85
86
86
--
87
87
======
88
88
89
-
Writes messages to a BigQuery table using the Storage Write API, which provides
90
-
higher throughput and lower latency than the legacy streaming API or load jobs.
89
+
Writes messages to a BigQuery table using the Storage Write API.
90
+
This provides higher throughput and lower latency than the legacy streaming API or load jobs.
91
91
92
-
Messages can be formatted as JSON (default) or raw protobuf bytes. When using
93
-
JSON format the component automatically fetches the table schema and converts
94
-
each message to the corresponding proto representation.
92
+
Messages can be formatted as JSON (default) or raw protobuf bytes.
93
+
When using JSON format the component automatically fetches the table schema and converts each message to the corresponding proto representation.
95
94
96
95
WARNING: The proto3 JSON mapping encodes int64 and uint64 values as strings.
97
-
JSON messages with integer fields must use string values (e.g. `"age": "30"`
98
-
not `"age": 30`), otherwise the write will fail with an unmarshalling error.
96
+
JSON messages with integer fields must use string values (e.g. `"age": "30"` not `"age": 30`).
97
+
Otherwise the write will fail with an unmarshalling error.
99
98
100
-
When batching is enabled the table name is resolved from the first message in
101
-
each batch; all messages in the same batch are written to that table.
99
+
When batching is enabled the table name is resolved from the first message in each batch.
100
+
All messages in the same batch are written to that table.
102
101
103
102
104
103
== Fields
@@ -131,7 +130,7 @@ This field supports xref:configuration:interpolation.adoc#bloblang-queries[inter
131
130
132
131
=== `message_format`
133
132
134
-
The format of input messages. Use 'json' to have the component convert JSON to proto automatically, or 'protobuf' to supply raw proto-encoded bytes.
133
+
The format of input messages. Use 'json' to have the component convert JSON to proto automatically. Use 'protobuf' to supply raw proto-encoded bytes.
135
134
136
135
137
136
*Type*: `string`
@@ -143,82 +142,6 @@ Options:
143
142
, `protobuf`
144
143
.
145
144
146
-
=== `credentials_json`
147
-
148
-
An optional JSON string containing GCP credentials. If empty, credentials are loaded from the environment.
149
-
[CAUTION]
150
-
====
151
-
This field contains sensitive information that usually shouldn't be added to a config directly, read our xref:configuration:secrets.adoc[secrets page for more info].
152
-
====
153
-
154
-
155
-
156
-
*Type*: `string`
157
-
158
-
*Default*: `""`
159
-
160
-
=== `target_principal`
161
-
162
-
Service account email to impersonate. When set, the output obtains tokens acting as this service account. Requires the caller to have roles/iam.serviceAccountTokenCreator on the target.
163
-
164
-
165
-
*Type*: `string`
166
-
167
-
*Default*: `""`
168
-
169
-
=== `delegates`
170
-
171
-
Optional delegation chain for chained service account impersonation. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next in the chain.
172
-
173
-
174
-
*Type*: `array`
175
-
176
-
*Default*: `[]`
177
-
178
-
=== `stream_idle_timeout`
179
-
180
-
How long a cached stream can remain unused before being closed. Relevant when the table field uses interpolation to route to many tables.
181
-
182
-
183
-
*Type*: `string`
184
-
185
-
*Default*: `"5m"`
186
-
187
-
=== `stream_sweep_interval`
188
-
189
-
How often to check for idle streams to close.
190
-
191
-
192
-
*Type*: `string`
193
-
194
-
*Default*: `"1m"`
195
-
196
-
=== `endpoint`
197
-
198
-
Optional endpoint overrides for the BigQuery and Storage Write API clients.
199
-
200
-
201
-
*Type*: `object`
202
-
203
-
204
-
=== `endpoint.http`
205
-
206
-
Override the BigQuery HTTP endpoint. Useful for local emulators.
207
-
208
-
209
-
*Type*: `string`
210
-
211
-
*Default*: `""`
212
-
213
-
=== `endpoint.grpc`
214
-
215
-
Override the BigQuery Storage gRPC endpoint. Useful for local emulators.
216
-
217
-
218
-
*Type*: `string`
219
-
220
-
*Default*: `""`
221
-
222
145
=== `max_in_flight`
223
146
224
147
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
@@ -330,4 +253,80 @@ processors:
330
253
format: json_array
331
254
```
332
255
256
+
=== `credentials_json`
257
+
258
+
An optional JSON string containing GCP credentials. If empty, credentials are loaded from the environment.
259
+
[CAUTION]
260
+
====
261
+
This field contains sensitive information that usually shouldn't be added to a config directly, read our xref:configuration:secrets.adoc[secrets page for more info].
262
+
====
263
+
264
+
265
+
266
+
*Type*: `string`
267
+
268
+
*Default*: `""`
269
+
270
+
=== `target_principal`
271
+
272
+
Service account email to impersonate. When set, the output obtains tokens acting as this service account. Requires the caller to have roles/iam.serviceAccountTokenCreator on the target.
273
+
274
+
275
+
*Type*: `string`
276
+
277
+
*Default*: `""`
278
+
279
+
=== `delegates`
280
+
281
+
Optional delegation chain for chained service account impersonation. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next in the chain.
282
+
283
+
284
+
*Type*: `array`
285
+
286
+
*Default*: `[]`
287
+
288
+
=== `stream_idle_timeout`
289
+
290
+
How long a cached stream can remain unused before being closed. Relevant when the table field uses interpolation to route to many tables.
291
+
292
+
293
+
*Type*: `string`
294
+
295
+
*Default*: `"5m"`
296
+
297
+
=== `stream_sweep_interval`
298
+
299
+
How often to check for idle streams to close.
300
+
301
+
302
+
*Type*: `string`
303
+
304
+
*Default*: `"1m"`
305
+
306
+
=== `endpoint`
307
+
308
+
Optional endpoint overrides for the BigQuery and Storage Write API clients.
309
+
310
+
311
+
*Type*: `object`
312
+
313
+
314
+
=== `endpoint.http`
315
+
316
+
Override the BigQuery HTTP endpoint. Useful for local emulators.
317
+
318
+
319
+
*Type*: `string`
320
+
321
+
*Default*: `""`
322
+
323
+
=== `endpoint.grpc`
324
+
325
+
Override the BigQuery Storage gRPC endpoint. Useful for local emulators.
Description("The format of input messages. Use 'json' to have the component convert JSON to proto automatically, or 'protobuf' to supply raw proto-encoded bytes.").
103
+
Description("The format of input messages."+
104
+
" Use 'json' to have the component convert JSON to proto automatically."+
105
+
" Use 'protobuf' to supply raw proto-encoded bytes.").
105
106
Default("json"),
107
+
service.NewOutputMaxInFlightField().Default(64),
108
+
service.NewBatchPolicyField(bqwaFieldBatching),
106
109
service.NewStringField(bqwaFieldCredentialsJSON).
107
-
Description("An optional JSON string containing GCP credentials. If empty, credentials are loaded from the environment.").
" If empty, credentials are loaded from the environment.").
108
112
Secret().
109
113
Default(""),
110
114
service.NewStringField(bqwaFieldTargetPrincipal).
111
-
Description("Service account email to impersonate. When set, the output obtains tokens acting as this service account. Requires the caller to have roles/iam.serviceAccountTokenCreator on the target.").
115
+
Description("Service account email to impersonate."+
116
+
" When set, the output obtains tokens acting as this service account."+
117
+
" Requires the caller to have roles/iam.serviceAccountTokenCreator on the target.").
112
118
Advanced().
113
119
Default(""),
114
120
service.NewStringListField(bqwaFieldDelegates).
115
-
Description("Optional delegation chain for chained service account impersonation. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next in the chain.").
121
+
Description("Optional delegation chain for chained service account impersonation."+
122
+
" Each service account must be granted roles/iam.serviceAccountTokenCreator on the next in the chain.").
Description("How long a cached stream can remain unused before being closed. Relevant when the table field uses interpolation to route to many tables.").
126
+
Description("How long a cached stream can remain unused before being closed."+
127
+
" Relevant when the table field uses interpolation to route to many tables.").
0 commit comments