Skip to content

Commit effe4f2

Browse files
jubradclaude
andcommitted
docs: update Kafka connection examples for BOOTSTRAP BROKER syntax
Updates user-facing documentation to show the new BOOTSTRAP BROKER and MATCHING broker rules syntax for Kafka PrivateLink connections. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 924a1ff commit effe4f2

2 files changed

Lines changed: 83 additions & 11 deletions

File tree

doc/user/content/sql/create-connection.md

Lines changed: 37 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -320,8 +320,42 @@ SSH bastion host.
320320
{{< include-md file="shared-content/aws-privatelink-cloud-only-note.md" >}}
321321

322322
Depending on the hosted service you are connecting to, you might need to specify
323-
a PrivateLink connection [per advertised broker](#kafka-privatelink-syntax)
324-
(e.g. Amazon MSK), or a single [default PrivateLink connection](#kafka-privatelink-default) (e.g. Redpanda Cloud).
323+
a PrivateLink connection and [per-availability-zone routing rules for brokers](#kafka-privatelinks) (e.g. Confluent Cloud),
324+
a PrivateLink connection [per advertised broker](#kafka-privatelink-syntax) (e.g. Amazon MSK),
325+
or a single [default PrivateLink connection](#kafka-privatelink-default) (e.g. Redpanda Cloud).
326+
327+
##### Dynamic broker discovery {#kafka-privatelinks}
328+
329+
Confluent Cloud does not require listing every broker individually.
330+
Instead, specify a `BOOTSTRAP BROKER` for the initial connection and
331+
wildcard `MATCHING` rules for routing dynamically discovered brokers
332+
through PrivateLink.
333+
334+
{{% include-syntax file="examples/create_connection" example="syntax-kafka-aws-privatelinks" %}}
335+
336+
When your Confluent cluster is deployed across multiple AZs and your
337+
PrivateLink connection is configured with matching AZs, you should only
338+
specify the `AWS PRIVATELINK` on the `BOOTSTRAP BROKER`, not the
339+
`AVAILABILITY ZONE`. This will allow Materialize to attempt to find the
340+
bootstrap broker across availability zones. The `MATCHING` rules should
341+
specify `AVAILABILITY ZONE` to route discovered brokers through their
342+
specific AZ endpoint.
343+
344+
```mzsql
345+
CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
346+
SERVICE NAME 'com.amazonaws.vpce.us-east-1.vpce-svc-0e123abc123198abc',
347+
AVAILABILITY ZONES ('use1-az1', 'use1-az4')
348+
);
349+
350+
CREATE CONNECTION kafka_connection TO KAFKA (
351+
BOOTSTRAP BROKER 'lkc-xxx.domyyy.us-east-1.aws.confluent.cloud:9092'
352+
USING AWS PRIVATELINK privatelink_svc,
353+
BROKERS (
354+
MATCHING '*.use1-az1.*' USING AWS PRIVATELINK privatelink_svc (AVAILABILITY ZONE = 'use1-az1'),
355+
MATCHING '*.use1-az4.*' USING AWS PRIVATELINK privatelink_svc (AVAILABILITY ZONE = 'use1-az4')
356+
)
357+
);
358+
```
325359

326360
##### Broker connection syntax {#kafka-privatelink-syntax}
327361

@@ -347,7 +381,7 @@ broker that you want to connect to via the tunnel.
347381
Field | Value | Required | Description
348382
----------------------------------------|------------------|:--------:|-------------------------------
349383
`AWS PRIVATELINK` | object name | ✓ | The name of an [AWS PrivateLink connection](#aws-privatelink) through which network traffic for this broker should be routed.
350-
`AVAILABILITY ZONE` | `text` | | The ID of the availability zone of the AWS PrivateLink service in which the broker is accessible. If unspecified, traffic will be routed to each availability zone declared in the [AWS PrivateLink connection](#aws-privatelink) in sequence until the correct availability zone for the broker is discovered. If specified, Materialize will always route connections via the specified availability zone.
384+
`AVAILABILITY ZONE` | `text` | | The ID of the availability zone of the AWS PrivateLink service in which the broker is accessible.
351385
`PORT` | `integer` | | The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
352386

353387
##### Example {#kafka-privatelink-example}
@@ -388,13 +422,6 @@ PrivateLink connection and the port of the bootstrap server instead.
388422

389423
{{% include-syntax file="examples/create_connection" example="syntax-kafka-default-aws-privatelink" %}}
390424

391-
##### Default connection options {#kafka-privatelink-default-options}
392-
393-
Field | Value | Required | Description
394-
----------------------------------------|------------------|:--------:|-------------------------------
395-
`AWS PRIVATELINK` | object name | ✓ | The name of an [AWS PrivateLink connection](#aws-privatelink) through which network traffic for this broker should be routed.
396-
`PORT` | `integer` | | The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
397-
398425
##### Example {#kafka-privatelink-default-example}
399426

400427
```mzsql

doc/user/data/examples/create_connection.yml

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -242,11 +242,56 @@
242242
syntax_elements:
243243
- name: "`AWS PRIVATELINK <privatelink_connection_name>`"
244244
description: |
245+
*Value:* object name. Required.
246+
245247
The name of an AWS PrivateLink connection through which network traffic
246248
should be routed.
247249
- name: "`PORT`"
248250
description: |
249-
The port of the AWS PrivateLink service to connect to.
251+
*Value:* `integer`
252+
253+
The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
254+
255+
- name: "syntax-kafka-aws-privatelinks"
256+
code: |
257+
CREATE CONNECTION <connection_name> TO KAFKA (
258+
BOOTSTRAP BROKER '<broker_address>'
259+
USING AWS PRIVATELINK <privatelink_connection> (PORT <port>),
260+
BROKERS (
261+
MATCHING '<pattern1>' USING AWS PRIVATELINK <privatelink_connection1> (
262+
PORT <port1>,
263+
AVAILABILITY ZONE = '<az_id1>'
264+
),
265+
MATCHING '<pattern2>' USING AWS PRIVATELINK <privatelink_connection2>
266+
),
267+
...
268+
);
269+
syntax_elements:
270+
- name: "`BOOTSTRAP BROKER`"
271+
description: |
272+
Specifies a broker address used for the initial connection to the Kafka cluster.
273+
The broker is routed through the specified AWS PrivateLink connection.
274+
- name: "`MATCHING '<pattern>' USING AWS PRIVATELINK <connection_name>`"
275+
description: |
276+
Routes brokers whose advertised `host:port` matches `<pattern>` through
277+
the named AWS PrivateLink connection.
278+
A pattern may begin with `*` to match any prefix. A pattern may end with `*` to match any suffix.
279+
All other characters in the pattern are matched literally.
280+
Rules are evaluated in order. The first matching rule wins.
281+
If no rule matches, Materialize will attempt to connect to the broker directly, without tunneling.
282+
- name: "`AVAILABILITY ZONE`"
283+
description: |
284+
*Value:* `text`. Optional.
285+
286+
The ID of the availability zone of the AWS PrivateLink service in which
287+
the broker is accessible. If omitted, traffic routes through the default
288+
PrivateLink endpoint, which distributes across all configured availability
289+
zones. Specify this only when you need to pin brokers to specific AZs.
290+
- name: "`PORT`"
291+
description: |
292+
*Value:* `integer`. Optional.
293+
294+
The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
250295
251296
- name: "syntax-csr"
252297
code: |

0 commit comments

Comments
 (0)