Skip to content

Commit 0205335

Browse files
aimurphyaimurphy
authored
DOC-5600 2102x (#187)
* bring new files * nav, attribs, duplicate page * remove duplicate elasticsearch page * remove experimental sinks and sources * remove unsupported connectors * missing attributes * attributes hacks, missing attributes * fix link * bring main topic content * elastic redirect * change example syntax * fix file type * once more
1 parent 8a3e902 commit 0205335

23 files changed

Lines changed: 796 additions & 463 deletions

‎antora.yml‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,4 +17,16 @@ asciidoc:
1717
starlight-rabbitmq: 'Starlight for RabbitMQ'
1818
pulsar-reg: 'Apache Pulsar(TM)'
1919
pulsar: 'Apache Pulsar'
20-
pulsar-short: 'Pulsar'
20+
pulsar-short: 'Pulsar'
21+
22+
# Required for include::common partials that are shared with Astra Streaming
23+
web-ui: 'Admin Console'
24+
astra: 'Astra'
25+
astra-db: 'Astra DB'
26+
scb: 'Secure Connect Bundle (SCB)'
27+
scb-short: 'SCB'
28+
scb-brief: 'Secure Connect Bundle'
29+
cass-reg: 'Apache Cassandra(R)'
30+
cass: 'Apache Cassandra'
31+
cass-short: 'Cassandra'
32+
debezium-version: '1.7'

‎modules/ROOT/nav.adoc‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,21 @@
1919
* xref:operations:auth.adoc[]
2020
* xref:operations:functions.adoc[]
2121
* xref:operations:functions-transform.adoc[]
22-
* xref:operations:io-connectors.adoc[]
23-
* xref:operations:io-elastic-sink.adoc[]
22+
* I/O connectors
23+
** xref:connectors:index.adoc[]
24+
** Sinks
25+
*** xref:connectors:sinks/astra-db.adoc[]
26+
*** xref:connectors:sinks/elastic-search.adoc[]
27+
*** xref:connectors:sinks/jdbc-clickhouse.adoc[]
28+
*** xref:connectors:sinks/jdbc-mariadb.adoc[]
29+
*** xref:connectors:sinks/jdbc-postgres.adoc[]
30+
*** xref:connectors:sinks/kafka.adoc[]
31+
*** xref:connectors:sinks/kinesis.adoc[]
32+
** Sources
33+
*** xref:connectors:sources/debezium-mongodb.adoc[]
34+
*** xref:connectors:sources/debezium-mysql.adoc[]
35+
*** xref:connectors:sources/debezium-postgres.adoc[]
36+
*** xref:connectors:sources/kafka.adoc[]
37+
*** xref:connectors:sources/kinesis.adoc[]
2438
* xref:operations:scale-cluster.adoc[]
2539
* xref:operations:troubleshooting.adoc[]

‎modules/ROOT/pages/faqs.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ https://github.com/datastax/pulsar-heartbeat[{pulsar-short} Heartbeat] monitors
8080

8181
== What are the features provided by {company} {pulsar} Connector (`pulsar-sink`) that are not supported in `kafka-sink`?
8282

83-
The https://pulsar.apache.org/docs/en/io-overview/[{pulsar-short} IO framework] provides many features that are not possible in Kafka, and has different compression formats and auth/security features. The features are handled by {pulsar-short}. For more, see xref:operations:io-connectors.adoc[Luna Streaming IO Connectors].
83+
The https://pulsar.apache.org/docs/en/io-overview/[{pulsar-short} IO framework] provides many features that are not possible in Kafka, and has different compression formats and auth/security features. The features are handled by {pulsar-short}. For more, see xref:connectors:index.adoc[Luna Streaming IO Connectors].
8484

8585
The {company} {pulsar} Connector allows single-record acknowledgement and negative acknowledgements.
8686

‎modules/ROOT/pages/index.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ In addition to the distribution of https://pulsar.apache.org/en/versions/[{pulsa
1717

1818
* A xref:install-upgrade:quickstart-helm-installs.adoc[Helm chart] that deploys and manages {pulsar-short} on your current CNCF-conformant Kubernetes infrastructure
1919

20-
* Cassandra, Elastic, Kinesis, Kafka, and JDBC xref:operations:io-connectors.adoc[connectors]
20+
* Cassandra, Elastic, Kinesis, Kafka, and JDBC xref:connectors:index.adoc[connectors]
2121

2222
* xref:components:admin-console-vm.adoc[{pulsar-short} Admin Console] for simplified administration of your {pulsar-short} environment
2323

‎modules/ROOT/partials/sinks/edit.adoc‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
To update a connector, pass the new configuration definition to the connector.
2+
For example, if you created the connector with a configuration file, you can pass an updated configuration file.
3+
4+
You can include the entire configuration or only the properties that you want to change.
5+
Additionally, some properties can be modified with specific arguments, such as `--parallelism`.
6+
7+
To get the current configuration, see xref:connectors:index.adoc#get-sink-connector-configuration-data[Get sink connector configuration data].
8+
9+
[source,shell,subs="+attributes"]
10+
----
11+
./bin/pulsar-admin sinks update \
12+
--sink-type {connectorType} \
13+
--name "$SINK_NAME" \
14+
--inputs "persistent://$TENANT/$NAMESPACE/$TOPIC" \
15+
--tenant "$TENANT" \
16+
--parallelism 2
17+
----

‎modules/ROOT/partials/sinks/get-started.adoc‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
. Optional: If you are using the `pulsar-admin` CLI, set the following commonly-used environment variables:
2+
+
3+
[source,shell,subs="+quotes"]
4+
----
5+
export TENANT="**TENANT_NAME**"
6+
export TOPIC="**INPUT_TOPIC_NAME**"
7+
export NAMESPACE="**NAMESPACE_NAME**"
8+
export SINK_NAME="**SINK_CONNECTOR_UNIQUE_NAME**"
9+
----
10+
+
11+
`**SINK_NAME**` is the name for your new sink connector.
12+
{company} recommends a memorable, human-readable name that summarizes the connector's purpose.
13+
For example: `{connectorType}-sink-prod-us-east-1`.
14+
15+
. Create the connector using JSON-formatted connector configuration data.
16+
You can pass the configuration directly or with a configuration file.
17+
+
18+
[source,shell,subs="+attributes"]
19+
----
20+
./bin/pulsar-admin sinks create \
21+
--sink-type {connectorType} \
22+
--name "$SINK_NAME" \
23+
--inputs "persistent://$TENANT/$NAMESPACE/$TOPIC" \
24+
--tenant "$TENANT" \
25+
--sink-config-file configs.json
26+
----
27+
+
28+
.Example configuration data structure
29+
[source,json]
30+
----
31+
include::common:streaming:example$connectors/sinks/{connectorType}/sample-data.json[]
32+
----

‎modules/ROOT/partials/sources/edit.adoc‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
To update a connector, pass the new configuration definition to the connector.
2+
For example, if you created the connector with a configuration file, you can pass an updated configuration file.
3+
4+
You can include the entire configuration or only the properties that you want to change.
5+
Additionally, some properties can be modified with specific arguments, such as `--parallelism`.
6+
7+
To get the current configuration, see xref:connectors:index.adoc#get-source-connector-configuration-data[Get source connector configuration data].
8+
9+
[source,shell,subs="+attributes"]
10+
----
11+
./bin/pulsar-admin sources update \
12+
--source-type {connectorType} \
13+
--name "$SOURCE_NAME" \
14+
--destination-topic-name "persistent://$TENANT/$NAMESPACE/$TOPIC" \
15+
--tenant "$TENANT" \
16+
--parallelism 2
17+
----

‎modules/ROOT/partials/sources/get-started.adoc‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
. Optional: If you are using the `pulsar-admin` CLI, set the following commonly-used environment variables:
2+
+
3+
[source,shell,subs="+quotes"]
4+
----
5+
export TENANT="**TENANT_NAME**"
6+
export TOPIC="**OUTPUT_TOPIC_NAME**"
7+
export NAMESPACE="**NAMESPACE_NAME**"
8+
export SOURCE_NAME="**SOURCE_CONNECTOR_UNIQUE_NAME**"
9+
----
10+
+
11+
`**SOURCE_NAME**` is the name for your new source connector.
12+
{company} recommends a memorable, human-readable name that summarizes the connector's purpose.
13+
For example: `{connectorType}-source-prod-us-east-1`.
14+
15+
. Create the connector using JSON-formatted connector configuration data.
16+
You can pass the configuration directly or with a configuration file.
17+
+
18+
[source,shell,subs="+attributes"]
19+
----
20+
./bin/pulsar-admin sources create \
21+
--source-type {connectorType} \
22+
--name "$SOURCE_NAME" \
23+
--destination-topic-name "persistent://$TENANT/$NAMESPACE/$TOPIC" \
24+
--tenant "$TENANT" \
25+
--source-config-file configs.json
26+
----
27+
+
28+
.Example configuration data structure
29+
[source,json]
30+
----
31+
include::common:streaming:example$connectors/sources/{connectorType}/sample-data.json[]
32+
----

‎modules/connectors/pages/index.adoc‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
= {pulsar-short} I/O connectors in {product}
2+
:navtitle: {pulsar-short} I/O connectors reference
3+
:page-aliases: operations:io-connectors.adoc
4+
5+
{product} offers fully-managed versions of https://pulsar.apache.org/docs/en/io-overview/[{pulsar-reg} sink and source connectors].
6+
7+
[TIP]
8+
====
9+
There are three versions of the {company} Luna Streaming distribution.
10+
The `lunastreaming-all` version includes all connectors.
11+
====
12+
13+
[#sink-connectors]
14+
== Sink connectors
15+
16+
Sink connectors export messages from {pulsar-short} topics to external services.
17+
18+
Supported sink connectors::
19+
The following sink connectors are included in {product}:
20+
+
21+
* xref:connectors:sinks/astra-db.adoc[{astra-db} and {cass} sink]
22+
* xref:connectors:sinks/kafka.adoc[Apache Kafka sink]
23+
* xref:connectors:sinks/kinesis.adoc[AWS Kinesis sink]
24+
* xref:connectors:sinks/elastic-search.adoc[Elasticsearch sink]
25+
* xref:connectors:sinks/jdbc-clickhouse.adoc[JDBC ClickHouse sink]
26+
* xref:connectors:sinks/jdbc-mariadb.adoc[JDBC MariaDB sink]
27+
* xref:connectors:sinks/jdbc-postgres.adoc[JDBC PostgreSQL sink]
28+
29+
[#source-connectors]
30+
== Source connectors
31+
32+
Source connectors ingest messages from external services into {pulsar-short} topics.
33+
34+
Supported source connectors::
35+
The following source connectors are included in {product}:
36+
+
37+
* xref:connectors:sources/kafka.adoc[Apache Kafka source]
38+
* xref:connectors:sources/kinesis.adoc[AWS Kinesis source]
39+
* xref:connectors:sources/debezium-mongodb.adoc[Debezium MongoDB source]
40+
* xref:connectors:sources/debezium-mysql.adoc[Debezium MySQL source]
41+
* xref:connectors:sources/debezium-postgres.adoc[Debezium PostgreSQL source]
42+
43+
[#create-and-manage-connectors]
44+
== Create and manage connectors
45+
46+
You can use the {web-ui} and the `xref:components:admin-console-tutorial.adoc[pulsar-admin]` CLI to create, monitor, and manage sink and source connectors.
47+
48+
Although you use the same base commands to create and update all {product} {pulsar-short} connectors, each connector has different configuration options.
49+
For example commands and configuration details, see the documentation for your preferred <<sink-connectors,sink connectors>> and <<source-connectors,source connectors>>.
50+
51+
=== `pulsar-admin` CLI sink operations
52+
53+
Get available sink connectors::
54+
Get a list of sink connectors that are available in your {product} {pulsar-short} tenant:
55+
+
56+
[source,shell]
57+
----
58+
./bin/pulsar-admin sinks available-sinks
59+
----
60+
61+
[#get-sink-connector-configuration-data]
62+
Get sink connector configuration data::
63+
Get the configuration for an existing sink connector:
64+
+
65+
[source,shell]
66+
----
67+
# Get information about a connector
68+
./bin/pulsar-admin sinks get \
69+
--namespace "$NAMESPACE" \
70+
--name "$SINK_NAME" \
71+
--tenant "$TENANT"
72+
----
73+
74+
Start a sink connector::
75+
+
76+
[source,shell]
77+
----
78+
# Start all instances of a connector
79+
./bin/pulsar-admin sinks start \
80+
--namespace "$NAMESPACE" \
81+
--name "$SINK_NAME" \
82+
--tenant "$TENANT"
83+
84+
# optionally add --instance-id to only start an individual instance
85+
----
86+
87+
Stop a sink connector::
88+
+
89+
[source,shell]
90+
----
91+
# Stop all instances of a connector
92+
./bin/pulsar-admin sinks stop \
93+
--namespace "$NAMESPACE" \
94+
--name "$SINK_NAME" \
95+
--tenant "$TENANT"
96+
97+
# optionally add --instance-id to only stop an individual instance
98+
----
99+
100+
Restart a sink connector::
101+
+
102+
[source,shell]
103+
----
104+
# Restart all instances of a connector
105+
./bin/pulsar-admin sinks restart \
106+
--namespace "$NAMESPACE" \
107+
--name "$SINK_NAME" \
108+
--tenant "$TENANT"
109+
110+
# optionally add --instance-id to only restart an individual instance
111+
----
112+
113+
Get sink connector status::
114+
+
115+
[source,shell]
116+
----
117+
# Check connector status
118+
./bin/pulsar-admin sinks status \
119+
--instance-id "$SINK_INSTANCE_ID" \
120+
--namespace "$NAMESPACE" \
121+
--name "$SINK_NAME" \
122+
--tenant "$TENANT"
123+
----
124+
125+
Delete a sink connector::
126+
+
127+
[source,shell]
128+
----
129+
# Delete all instances of a connector
130+
./bin/pulsar-admin sinks delete \
131+
--namespace "$NAMESPACE" \
132+
--name "$SINK_NAME" \
133+
--tenant "$TENANT"
134+
----
135+
136+
=== `pulsar-admin` CLI source operations
137+
138+
Get available source connectors::
139+
Get a list of source connectors that are available in your {product} {pulsar-short} tenant:
140+
+
141+
[source,shell]
142+
----
143+
./bin/pulsar-admin sources available-sources
144+
----
145+
146+
[#get-source-connector-configuration-data]
147+
Get source connector configuration data::
148+
Get the configuration for an existing source connector:
149+
+
150+
[source,shell]
151+
----
152+
# Get information about connector
153+
./bin/pulsar-admin sources get \
154+
--namespace "$NAMESPACE" \
155+
--name "$SOURCE_NAME" \
156+
--tenant "$TENANT"
157+
----
158+
159+
Start a source connector::
160+
+
161+
[source,shell]
162+
----
163+
# Start all instances of a connector
164+
./bin/pulsar-admin sources start \
165+
--namespace "$NAMESPACE" \
166+
--name "$SOURCE_NAME" \
167+
--tenant "$TENANT"
168+
169+
# optionally add --instance-id to only start an individual instance
170+
----
171+
172+
Stop a source connector::
173+
+
174+
[source,shell]
175+
----
176+
# Stop all instances of a connector
177+
./bin/pulsar-admin sources stop \
178+
--namespace "$NAMESPACE" \
179+
--name "$SOURCE_NAME" \
180+
--tenant "$TENANT"
181+
182+
# optionally add --instance-id to only stop an individual instance
183+
----
184+
185+
Restart a source connector::
186+
+
187+
[source,shell]
188+
----
189+
# Restart all instances of a connector
190+
./bin/pulsar-admin sources restart \
191+
--namespace "$NAMESPACE" \
192+
--name "$SOURCE_NAME" \
193+
--tenant "$TENANT"
194+
195+
# optionally add --instance-id to only restart an individual instance
196+
----
197+
198+
Get source connector status::
199+
+
200+
[source,shell]
201+
----
202+
# Check connector status
203+
./bin/pulsar-admin sources status \
204+
--instance-id "$SOURCE_INSTANCE_ID" \
205+
--namespace "$NAMESPACE" \
206+
--name "$SOURCE_NAME" \
207+
--tenant "$TENANT"
208+
----
209+
210+
Delete a source connector::
211+
+
212+
[source,shell]
213+
----
214+
# Delete all instances of a connector
215+
./bin/pulsar-admin sources delete \
216+
--namespace "$NAMESPACE" \
217+
--name "$SOURCE_NAME" \
218+
--tenant "$TENANT"
219+
----
220+
221+
== Connector metrics
222+
223+
{product} exposes Prometheus-formatted metrics for every connector.
224+
For more information, see xref:components:pulsar-monitor.adoc[].

0 commit comments

Comments
 (0)