Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 52 additions & 1 deletion docs/integrations/data-ingestion/clickpipes/aws-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
- category: 'clickpipes'
---

import PrivatePreviewBadge from '@theme/badges/PrivatePreviewBadge';
import cp_service from '@site/static/images/integrations/data-ingestion/clickpipes/cp_service.png';
import cp_step0 from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step0.png';
import cp_rpe_select from '@site/static/images/integrations/data-ingestion/clickpipes/cp_rpe_select.png';
Expand Down Expand Up @@ -189,6 +190,8 @@
and configuring the VPC endpoint service to use the NLB.

VPC endpoint service can be [configured with a private DNS](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html), that will be accessible in a ClickPipes VPC.
If your service can't use PrivateLink managed private DNS (for example, the provider requires
consumers to manage DNS resolution, as Confluent Cloud does), see [Custom private DNS](#custom-private-dns).

It's a preferred choice for:

Expand All @@ -208,6 +211,52 @@
can be configured for ClickPipes. Add [your ClickPipe region](#aws-privatelink-regions) to the allowed regions in your VPC endpoint service.
:::

## Custom private DNS {#custom-private-dns}

<PrivatePreviewBadge/>

:::info
Custom private DNS is in **Private Preview**. Reach out to the [ClickHouse support team](https://clickhouse.com/support/program) to enable it for your service.
:::

AWS PrivateLink provides [managed private DNS](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html)
for VPC endpoint services with a verified domain. Some services require each consumer to manage DNS resolution in their own VPC, e.g., Confluent Cloud requires consumers to resolve broker hostnames to the PrivateLink endpoint.

For these cases, ClickPipes supports attaching custom private DNS names to a reverse private
endpoint. ClickPipes resolves these names to the endpoint's private addresses, so your data
source can be reached by its own hostname over private connectivity.

Custom private DNS complements managed private DNS — it doesn't replace it. If your
PrivateLink service already provides private DNS names, you don't need custom names.

Custom private DNS names are supported for the [VPC endpoint service](#vpc-endpoint-service)
and [VPC resource](#vpc-resource) endpoint types. MSK multi-VPC provides managed private DNS

Check notice on line 233 in docs/integrations/data-ingestion/clickpipes/aws-privatelink.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Uppercase

Suggestion: Instead of uppercase for 'MSK', use lowercase or backticks (`) if possible. Otherwise, ask a Technical Writer to add this word or acronym to the rule's exception list.
out-of-the-box and doesn't support custom names.

The following rules apply to custom private DNS names:

- Exact names (`kafka.internal.example.com`) and wildcard names (`*.example.com`) are
supported. A wildcard matches a single DNS label — for example,
`*.abcde12345.us-east-1.aws.confluent.cloud` matches
`b0-lkc123.abcde12345.us-east-1.aws.confluent.cloud`.
- Names must be unique across all reverse private endpoints of a ClickHouse service,
including overlaps between wildcard and exact names.
- Names under reserved suffixes (e.g., `local`, `localhost`, `internal`, `corp`,
`private`) are rejected.

To configure custom private DNS names:

- In the ClickHouse Cloud console, fill in the `Custom private DNS name` field when
[creating a reverse private endpoint](#creating-clickpipe). The field is shown once the
feature is enabled for your service.
- With [OpenAPI](https://clickhouse.com/docs/cloud/manage/api/swagger), set
`customPrivateDnsMappings` when creating a reverse private endpoint, or update an existing
endpoint with a `PATCH` request. Updates replace the full list of mappings; an empty list
removes all custom names.
- With Terraform, use the
[`clickhouse_clickpipes_reverse_private_endpoint_custom_private_dns`](https://registry.terraform.io/providers/ClickHouse/clickhouse/latest/docs/resources/clickpipes_reverse_private_endpoint_custom_private_dns)
resource to manage mappings on an existing reverse private endpoint.

## Creating a ClickPipe with reverse private endpoint {#creating-clickpipe}

<VerticalStepper headerLevel="list">
Expand Down Expand Up @@ -246,6 +295,7 @@
- For VPC resource, provide the configuration share ARN and configuration ID.
- For MSK multi-VPC, provide the cluster ARN and authentication method used with a created endpoint.
- For VPC endpoint service, provide the service name.
- Optionally, provide a [custom private DNS](#custom-private-dns) name.

7. Click on `Create` and wait for the reverse private endpoint to be ready.

Expand All @@ -258,7 +308,8 @@
8. Once the endpoint is ready, you can use a DNS name to connect to the data source.

On a list of endpoints, you can see the DNS name for the available endpoint.
It can be either an internally ClickPipes provisioned DNS name or a private DNS name supplied by a PrivateLink service.
It can be an internally ClickPipes provisioned DNS name, a private DNS name supplied by a PrivateLink service,
or a [custom private DNS name](#custom-private-dns).
DNS name isn't a complete network address.
Add the port according to the data source.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ To integrate with a schema registry during ClickPipes configuration, you must us

ClickPipes connects to the schema registry over HTTPS at the URL you provide. The schema registry does not need to be publicly accessible.

If your Kafka brokers are reached through a [reverse private endpoint](/integrations/clickpipes/aws-privatelink) (AWS PrivateLink or GCP Private Service Connect), the schema registry can use the same private connectivity. ClickPipes resolves the registry hostname through the reverse private endpoint's private DNS, so a registry hosted privately alongside your brokers is reachable as long as its hostname resolves to the reverse private endpoint's private IP addresses (via the endpoint's private DNS support or a custom private DNS mapping).
If your Kafka brokers are reached through a [reverse private endpoint](/integrations/clickpipes/aws-privatelink) (AWS PrivateLink or GCP Private Service Connect), the schema registry can use the same private connectivity. ClickPipes resolves the registry hostname through the reverse private endpoint's private DNS, so a registry hosted privately alongside your brokers is reachable as long as its hostname resolves to the reverse private endpoint's private IP addresses (via the endpoint's private DNS support or a [custom private DNS mapping](/integrations/clickpipes/aws-privatelink#custom-private-dns)).

Keep the following in mind:

Expand Down
Loading