You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update Kafka connection examples for MATCHING broker rules
Updates user-facing documentation to show the new MATCHING broker rules
syntax for Kafka PrivateLink connections. Adds a PrivateLink tab to the
Confluent Cloud guide.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: doc/user/content/ingest-data/kafka/confluent-cloud.md
+94-19Lines changed: 94 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -76,7 +76,7 @@ of the following steps:
76
76
77
77
Otherwise, you can find more information about how to do that [here](https://docs.confluent.io/cloud/current/get-started/index.html#step-2-create-a-ak-topic).
78
78
79
-
4.#### Create a source in Materialize
79
+
4.#### Create a connection in Materialize
80
80
81
81
a. Open the [Confluent Cloud dashboard](https://confluent.cloud/) and select your cluster.
82
82
@@ -87,39 +87,114 @@ of the following steps:
87
87
d. Connect to Materialize using the [SQL Shell](/console/),
88
88
or your preferred SQL client.
89
89
90
-
e. Run the following command. Replace `<confluent_cloud>` with whatever you
91
-
want to name your source. The broker URL is what you copied in step c of
92
-
this subsection. The `<topic-name>` is the name of the topic you created in
93
-
Step 4. The `<your-api-key>` and `<your-api-secret>` are from the _Create
94
-
an API Key_ step.
90
+
e. Create the connection. The exact steps depend on your networking
91
+
configuration, so start by selecting the relevant option.
92
+
93
+
{{< tabs >}}
94
+
{{< tab "Public cluster">}}
95
95
96
96
```mzsql
97
-
CREATE SECRET confluent_username AS '<your-api-key>';
98
-
CREATE SECRET confluent_password AS '<your-api-secret>';
97
+
CREATE SECRET confluent_username AS '<your-api-key>';
98
+
CREATE SECRET confluent_password AS '<your-api-secret>';
99
99
100
-
CREATE CONNECTION <confluent_cloud> TO KAFKA (
100
+
CREATE CONNECTION confluent_cloud TO KAFKA (
101
101
BROKER '<confluent-broker-url>',
102
102
SASL MECHANISMS = 'PLAIN',
103
103
SASL USERNAME = SECRET confluent_username,
104
104
SASL PASSWORD = SECRET confluent_password
105
-
);
105
+
);
106
+
```
107
+
108
+
{{< /tab >}}
109
+
110
+
{{< tab "Use AWS PrivateLink (Cloud-only)" >}}
111
+
112
+
[AWS PrivateLink](https://aws.amazon.com/privatelink/) lets you connect
113
+
Materialize to your Confluent Cloud cluster without exposing traffic to the
114
+
public internet.
115
+
116
+
1. In the [Confluent Cloud console](https://confluent.cloud/), navigate to
117
+
your cluster's **Networking** settings and set up a PrivateLink endpoint.
118
+
Record the **VPC Endpoint Service Name** and the **DNS domain**.
119
+
120
+
1. In the Materialize [SQL shell](/console/), create a
121
+
[PrivateLink connection](/ingest-data/network-security/privatelink/) using
122
+
the service name from the previous step. Be sure to specify **all
123
+
availability zones** of your Confluent Cloud cluster.
124
+
125
+
```mzsql
126
+
CREATE CONNECTION confluent_privatelink TO AWS PRIVATELINK (
127
+
SERVICE NAME 'com.amazonaws.vpce.us-east-1.vpce-svc-0e123abc123198abc',
128
+
AVAILABILITY ZONES ('use1-az1', 'use1-az4', 'use1-az6')
129
+
);
130
+
```
131
+
132
+
1. Retrieve the AWS principal for the PrivateLink connection:
133
+
134
+
```mzsql
135
+
SELECT principal
136
+
FROM mz_aws_privatelink_connections plc
137
+
JOIN mz_connections c ON plc.id = c.id
138
+
WHERE c.name = 'confluent_privatelink';
139
+
```
140
+
141
+
1. In the Confluent Cloud console, add the AWS principal to the PrivateLink
142
+
access list.
106
143
107
-
CREATE SOURCE <source-name>
144
+
1. In Materialize, validate the PrivateLink connection:
145
+
146
+
```mzsql
147
+
VALIDATE CONNECTION confluent_privatelink;
148
+
```
149
+
150
+
If no validation error is returned, move to the next step.
151
+
152
+
1. Create the Kafka connection. The static broker (used for bootstrapping)
153
+
does not need an `AVAILABILITY ZONE` — Materialize will find it across
154
+
availability zones. The `MATCHING` rules should specify `AVAILABILITY ZONE`
155
+
to route discovered brokers through their specific AZ endpoint.
156
+
157
+
```mzsql
158
+
CREATE SECRET confluent_username AS '<your-api-key>';
159
+
CREATE SECRET confluent_password AS '<your-api-secret>';
160
+
161
+
CREATE CONNECTION confluent_cloud TO KAFKA (
162
+
BROKERS (
163
+
'<confluent-broker-url>' USING AWS PRIVATELINK confluent_privatelink,
164
+
MATCHING '*.use1-az1.*' USING AWS PRIVATELINK confluent_privatelink (AVAILABILITY ZONE = 'use1-az1'),
165
+
MATCHING '*.use1-az4.*' USING AWS PRIVATELINK confluent_privatelink (AVAILABILITY ZONE = 'use1-az4'),
166
+
MATCHING '*.use1-az6.*' USING AWS PRIVATELINK confluent_privatelink (AVAILABILITY ZONE = 'use1-az6')
167
+
),
168
+
SASL MECHANISMS = 'PLAIN',
169
+
SASL USERNAME = SECRET confluent_username,
170
+
SASL PASSWORD = SECRET confluent_password
171
+
);
172
+
```
173
+
174
+
The `MATCHING` patterns correspond to the AZ-specific DNS subdomains
175
+
from your Confluent Cloud networking settings. Adjust the patterns and
176
+
availability zones to match your cluster's configuration.
177
+
178
+
{{< /tab >}}
179
+
{{< /tabs >}}
180
+
181
+
5. #### Start ingesting data
182
+
183
+
Once you have created the connection, create a source and start ingesting
184
+
data from your topic. By default, the source will be created in the active
185
+
cluster; to use a different cluster, use the `IN CLUSTER` clause.
186
+
187
+
```mzsql
188
+
CREATE SOURCE confluent_source
108
189
FROM KAFKA CONNECTION confluent_cloud (TOPIC '<topic-name>')
109
190
FORMAT JSON;
110
191
```
111
-
By default, the source will be created in the active cluster; to use a different
112
-
cluster, use the `IN CLUSTER` clause.
113
192
114
-
f. If the command executes without an error and outputs _CREATE SOURCE_, it
193
+
If the command executes without an error and outputs _CREATE SOURCE_, it
115
194
means that you have successfully connected Materialize to your Confluent
116
195
Cloud Kafka cluster.
117
196
118
-
**Note:** The example above walked through creating a source, which is a way
119
-
of connecting Materialize to an external data source. We created a
120
-
connection to Confluent Cloud Kafka using SASL authentication and credentials
121
-
securely stored as secrets in Materialize's secret management system. For
122
-
input formats, we used `JSON`, but you can also ingest Kafka messages
197
+
**Note:** The example above used `JSON`, but you can also ingest Kafka messages
123
198
formatted in other supported formats; e.g., [Avro and CSV](/sql/create-source/kafka/#syntax).
124
199
You can find more details about the various different supported formats and
125
200
possible configurations in the [reference documentation](/sql/create-source/kafka/).
`AWS PRIVATELINK` | object name | ✓ | The name of an [AWS PrivateLink connection](#aws-privatelink) through which network traffic for this broker should be routed.
350
-
`AVAILABILITY ZONE` | `text` | | The ID of the availability zone of the AWS PrivateLink service in which the broker is accessible. If unspecified, traffic will be routed to each availability zone declared in the [AWS PrivateLink connection](#aws-privatelink) in sequence until the correct availability zone for the broker is discovered. If specified, Materialize will always route connections via the specified availability zone.
384
+
`AVAILABILITY ZONE` | `text` | | The ID of the availability zone of the AWS PrivateLink service in which the broker is accessible.
351
385
`PORT` | `integer` | | The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
352
386
353
387
##### Example {#kafka-privatelink-example}
@@ -388,13 +422,6 @@ PrivateLink connection and the port of the bootstrap server instead.
`AWS PRIVATELINK` | object name | ✓ | The name of an [AWS PrivateLink connection](#aws-privatelink) through which network traffic for this broker should be routed.
396
-
`PORT` | `integer` | | The port of the AWS PrivateLink service to connect to. Defaults to the broker's port.
397
-
398
425
##### Example {#kafka-privatelink-default-example}
0 commit comments