You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update Kafka connection examples for BOOTSTRAP BROKER syntax
Updates user-facing documentation to show the new BOOTSTRAP BROKER
and MATCHING broker rules syntax for Kafka PrivateLink connections.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: doc/user/content/ingest-data/kafka/confluent-cloud.md
+95-19Lines changed: 95 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -76,7 +76,7 @@ of the following steps:
76
76
77
77
Otherwise, you can find more information about how to do that [here](https://docs.confluent.io/cloud/current/get-started/index.html#step-2-create-a-ak-topic).
78
78
79
-
4.#### Create a source in Materialize
79
+
4.#### Create a connection in Materialize
80
80
81
81
a. Open the [Confluent Cloud dashboard](https://confluent.cloud/) and select your cluster.
82
82
@@ -87,39 +87,115 @@ of the following steps:
87
87
d. Connect to Materialize using the [SQL Shell](/console/),
88
88
or your preferred SQL client.
89
89
90
-
e. Run the following command. Replace `<confluent_cloud>` with whatever you
91
-
want to name your source. The broker URL is what you copied in step c of
92
-
this subsection. The `<topic-name>` is the name of the topic you created in
93
-
Step 4. The `<your-api-key>` and `<your-api-secret>` are from the _Create
94
-
an API Key_ step.
90
+
e. Create the connection. The exact steps depend on your networking
91
+
configuration, so start by selecting the relevant option.
92
+
93
+
{{< tabs >}}
94
+
{{< tab "Public cluster">}}
95
95
96
96
```mzsql
97
-
CREATE SECRET confluent_username AS '<your-api-key>';
98
-
CREATE SECRET confluent_password AS '<your-api-secret>';
97
+
CREATE SECRET confluent_username AS '<your-api-key>';
98
+
CREATE SECRET confluent_password AS '<your-api-secret>';
99
99
100
-
CREATE CONNECTION <confluent_cloud> TO KAFKA (
100
+
CREATE CONNECTION confluent_cloud TO KAFKA (
101
101
BROKER '<confluent-broker-url>',
102
102
SASL MECHANISMS = 'PLAIN',
103
103
SASL USERNAME = SECRET confluent_username,
104
104
SASL PASSWORD = SECRET confluent_password
105
-
);
105
+
);
106
+
```
107
+
108
+
{{< /tab >}}
109
+
110
+
{{< tab "Use AWS PrivateLink (Cloud-only)" >}}
111
+
112
+
[AWS PrivateLink](https://aws.amazon.com/privatelink/) lets you connect
113
+
Materialize to your Confluent Cloud cluster without exposing traffic to the
114
+
public internet.
115
+
116
+
1. In the [Confluent Cloud console](https://confluent.cloud/), navigate to
117
+
your cluster's **Networking** settings and set up a PrivateLink endpoint.
118
+
Record the **VPC Endpoint Service Name** and the **DNS domain**.
119
+
120
+
1. In the Materialize [SQL shell](/console/), create a
121
+
[PrivateLink connection](/ingest-data/network-security/privatelink/) using
122
+
the service name from the previous step. Be sure to specify **all
123
+
availability zones** of your Confluent Cloud cluster.
124
+
125
+
```mzsql
126
+
CREATE CONNECTION confluent_privatelink TO AWS PRIVATELINK (
127
+
SERVICE NAME 'com.amazonaws.vpce.us-east-1.vpce-svc-0e123abc123198abc',
128
+
AVAILABILITY ZONES ('use1-az1', 'use1-az4', 'use1-az6')
129
+
);
130
+
```
131
+
132
+
1. Retrieve the AWS principal for the PrivateLink connection:
133
+
134
+
```mzsql
135
+
SELECT principal
136
+
FROM mz_aws_privatelink_connections plc
137
+
JOIN mz_connections c ON plc.id = c.id
138
+
WHERE c.name = 'confluent_privatelink';
139
+
```
140
+
141
+
1. In the Confluent Cloud console, add the AWS principal to the PrivateLink
142
+
access list.
106
143
107
-
CREATE SOURCE <source-name>
144
+
1. In Materialize, validate the PrivateLink connection:
145
+
146
+
```mzsql
147
+
VALIDATE CONNECTION confluent_privatelink;
148
+
```
149
+
150
+
If no validation error is returned, move to the next step.
151
+
152
+
1. Create the Kafka connection. The `BOOTSTRAP BROKER` does not need an
153
+
`AVAILABILITY ZONE` — Materialize will find the bootstrap broker across
154
+
availability zones. The `MATCHING` rules should specify `AVAILABILITY ZONE`
155
+
to route discovered brokers through their specific AZ endpoint.
156
+
157
+
```mzsql
158
+
CREATE SECRET confluent_username AS '<your-api-key>';
159
+
CREATE SECRET confluent_password AS '<your-api-secret>';
160
+
161
+
CREATE CONNECTION confluent_cloud TO KAFKA (
162
+
BOOTSTRAP BROKER '<confluent-broker-url>'
163
+
USING AWS PRIVATELINK confluent_privatelink,
164
+
BROKERS (
165
+
MATCHING '*.use1-az1.*' USING AWS PRIVATELINK confluent_privatelink (AVAILABILITY ZONE = 'use1-az1'),
166
+
MATCHING '*.use1-az4.*' USING AWS PRIVATELINK confluent_privatelink (AVAILABILITY ZONE = 'use1-az4'),
167
+
MATCHING '*.use1-az6.*' USING AWS PRIVATELINK confluent_privatelink (AVAILABILITY ZONE = 'use1-az6')
168
+
),
169
+
SASL MECHANISMS = 'PLAIN',
170
+
SASL USERNAME = SECRET confluent_username,
171
+
SASL PASSWORD = SECRET confluent_password
172
+
);
173
+
```
174
+
175
+
The `MATCHING` patterns correspond to the AZ-specific DNS subdomains
176
+
from your Confluent Cloud networking settings. Adjust the patterns and
177
+
availability zones to match your cluster's configuration.
178
+
179
+
{{< /tab >}}
180
+
{{< /tabs >}}
181
+
182
+
5. #### Start ingesting data
183
+
184
+
Once you have created the connection, create a source and start ingesting
185
+
data from your topic. By default, the source will be created in the active
186
+
cluster; to use a different cluster, use the `IN CLUSTER` clause.
187
+
188
+
```mzsql
189
+
CREATE SOURCE confluent_source
108
190
FROM KAFKA CONNECTION confluent_cloud (TOPIC '<topic-name>')
109
191
FORMAT JSON;
110
192
```
111
-
By default, the source will be created in the active cluster; to use a different
112
-
cluster, use the `IN CLUSTER` clause.
113
193
114
-
f. If the command executes without an error and outputs _CREATE SOURCE_, it
194
+
If the command executes without an error and outputs _CREATE SOURCE_, it
115
195
means that you have successfully connected Materialize to your Confluent
116
196
Cloud Kafka cluster.
117
197
118
-
**Note:** The example above walked through creating a source, which is a way
119
-
of connecting Materialize to an external data source. We created a
120
-
connection to Confluent Cloud Kafka using SASL authentication and credentials
121
-
securely stored as secrets in Materialize's secret management system. For
122
-
input formats, we used `JSON`, but you can also ingest Kafka messages
198
+
**Note:** The example above used `JSON`, but you can also ingest Kafka messages
123
199
formatted in other supported formats; e.g., [Avro and CSV](/sql/create-source/kafka/#syntax).
124
200
You can find more details about the various different supported formats and
125
201
possible configurations in the [reference documentation](/sql/create-source/kafka/).
`AWS PRIVATELINK` | object name | ✓ | The name of an [AWS PrivateLink connection](#aws-privatelink) through which network traffic for this broker should be routed.
350
-
`AVAILABILITY ZONE` | `text` | | The ID of the availability zone of the AWS PrivateLink service in which the broker is accessible. If unspecified, traffic will be routed to each availability zone declared in the [AWS PrivateLink connection](#aws-privatelink) in sequence until the correct availability zone for the broker is discovered. If specified, Materialize will always route connections via the specified availability zone.
384
+
`AVAILABILITY ZONE` | `text` | | The ID of the availability zone of the AWS PrivateLink service in which the broker is accessible.
351
385
`PORT` | `integer` | | The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
352
386
353
387
##### Example {#kafka-privatelink-example}
@@ -388,13 +422,6 @@ PrivateLink connection and the port of the bootstrap server instead.
`AWS PRIVATELINK` | object name | ✓ | The name of an [AWS PrivateLink connection](#aws-privatelink) through which network traffic for this broker should be routed.
396
-
`PORT` | `integer` | | The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
397
-
398
425
##### Example {#kafka-privatelink-default-example}
0 commit comments