Skip to content

Commit d148ddd

Browse files
jubradclaude
andcommitted
docs: update Kafka connection examples for BOOTSTRAP BROKER syntax
Updates user-facing documentation to show the new BOOTSTRAP BROKER and MATCHING broker rules syntax for Kafka PrivateLink connections. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent c34f80c commit d148ddd

2 files changed

Lines changed: 73 additions & 11 deletions

File tree

doc/user/content/sql/create-connection.md

Lines changed: 29 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -320,8 +320,34 @@ SSH bastion host.
320320
{{< include-md file="shared-content/aws-privatelink-cloud-only-note.md" >}}
321321

322322
Depending on the hosted service you are connecting to, you might need to specify
323-
a PrivateLink connection [per advertised broker](#kafka-privatelink-syntax)
324-
(e.g. Amazon MSK), or a single [default PrivateLink connection](#kafka-privatelink-default) (e.g. Redpanda Cloud).
323+
a PrivateLink connection and [per-availability-zone routing rules for brokers](#kafka-privatelinks) (e.g. Confluent Cloud),
324+
a PrivateLink connection [per advertised broker](#kafka-privatelink-syntax) (e.g. Amazon MSK),
325+
or a single [default PrivateLink connection](#kafka-privatelink-default) (e.g. Redpanda Cloud).
326+
327+
##### Dynamic broker discovery {#kafka-privatelinks}
328+
329+
Confluent Cloud does not require listing every broker individually.
330+
Instead, specify wildcard patterns for routing dynamically discovered brokers
331+
to the correct availability-zone-specific PrivateLink endpoint.
332+
For bootstrap brokers, use exact-match patterns without wildcards.
333+
334+
{{% include-syntax file="examples/create_connection" example="syntax-kafka-aws-privatelinks" %}}
335+
336+
```mzsql
337+
CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
338+
SERVICE NAME 'com.amazonaws.vpce.us-east-1.vpce-svc-0e123abc123198abc',
339+
AVAILABILITY ZONES ('use1-az1', 'use1-az4')
340+
);
341+
342+
CREATE CONNECTION kafka_connection TO KAFKA (
343+
BOOTSTRAP BROKER 'lkc-xxx.domyyy.us-east-1.aws.confluent.cloud:9092'
344+
USING AWS PRIVATELINK privatelink_svc (PORT 9092),
345+
BROKERS (
346+
MATCHING '*.use1-az1.*' USING AWS PRIVATELINK privatelink_svc (AVAILABILITY ZONE = 'use1-az1'),
347+
MATCHING '*.use1-az4.*' USING AWS PRIVATELINK privatelink_svc (AVAILABILITY ZONE = 'use1-az4')
348+
)
349+
);
350+
```
325351

326352
##### Broker connection syntax {#kafka-privatelink-syntax}
327353

@@ -347,7 +373,7 @@ broker that you want to connect to via the tunnel.
347373
Field | Value | Required | Description
348374
----------------------------------------|------------------|:--------:|-------------------------------
349375
`AWS PRIVATELINK` | object name | ✓ | The name of an [AWS PrivateLink connection](#aws-privatelink) through which network traffic for this broker should be routed.
350-
`AVAILABILITY ZONE` | `text` | | The ID of the availability zone of the AWS PrivateLink service in which the broker is accessible. If unspecified, traffic will be routed to each availability zone declared in the [AWS PrivateLink connection](#aws-privatelink) in sequence until the correct availability zone for the broker is discovered. If specified, Materialize will always route connections via the specified availability zone.
376+
`AVAILABILITY ZONE` | `text` | | The ID of the availability zone of the AWS PrivateLink service in which the broker is accessible.
351377
`PORT` | `integer` | | The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
352378

353379
##### Example {#kafka-privatelink-example}
@@ -388,13 +414,6 @@ PrivateLink connection and the port of the bootstrap server instead.
388414

389415
{{% include-syntax file="examples/create_connection" example="syntax-kafka-default-aws-privatelink" %}}
390416

391-
##### Default connection options {#kafka-privatelink-default-options}
392-
393-
Field | Value | Required | Description
394-
----------------------------------------|------------------|:--------:|-------------------------------
395-
`AWS PRIVATELINK` | object name | ✓ | The name of an [AWS PrivateLink connection](#aws-privatelink) through which network traffic for this broker should be routed.
396-
`PORT` | `integer` | | The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
397-
398417
##### Example {#kafka-privatelink-default-example}
399418

400419
```mzsql

doc/user/data/examples/create_connection.yml

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -242,11 +242,54 @@
242242
syntax_elements:
243243
- name: "`AWS PRIVATELINK <privatelink_connection_name>`"
244244
description: |
245+
*Value:* object name. Required.
246+
245247
The name of an AWS PrivateLink connection through which network traffic
246248
should be routed.
247249
- name: "`PORT`"
248250
description: |
249-
The port of the AWS PrivateLink service to connect to.
251+
*Value:* `integer`
252+
253+
The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
254+
255+
- name: "syntax-kafka-aws-privatelinks"
256+
code: |
257+
CREATE CONNECTION <connection_name> TO KAFKA (
258+
BOOTSTRAP BROKER '<broker_address>'
259+
USING AWS PRIVATELINK <privatelink_connection> (PORT <port>),
260+
BROKERS (
261+
MATCHING '<pattern1>' USING AWS PRIVATELINK <privatelink_connection1> (
262+
PORT <port1>,
263+
AVAILABILITY ZONE = '<az_id1>'
264+
),
265+
MATCHING '<pattern2>' USING AWS PRIVATELINK <privatelink_connection2>
266+
),
267+
...
268+
);
269+
syntax_elements:
270+
- name: "`BOOTSTRAP BROKER`"
271+
description: |
272+
Specifies a broker address used for the initial connection to the Kafka cluster.
273+
The broker is routed through the specified AWS PrivateLink connection.
274+
- name: "`MATCHING '<pattern>' USING AWS PRIVATELINK <connection_name>`"
275+
description: |
276+
Routes brokers whose advertised `host:port` matches `<pattern>` through
277+
the named AWS PrivateLink connection.
278+
A pattern may begin with `*` to match any prefix. A pattern may end with `*` to match any suffix.
279+
All other characters in the pattern are matched literally.
280+
Rules are evaluated in order. The first matching rule wins.
281+
If no rule matches, Materialize will attempt to connect to the broker directly, without tunneling.
282+
- name: "`AVAILABILITY ZONE`"
283+
description: |
284+
*Value:* `text`
285+
286+
The ID of the availability zone of the AWS PrivateLink service in which
287+
the broker is accessible.
288+
- name: "`PORT`"
289+
description: |
290+
*Value:* `integer`
291+
292+
The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
250293
251294
- name: "syntax-csr"
252295
code: |

0 commit comments

Comments
 (0)