You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/modules/ROOT/pages/backfill-cli.adoc
+35-35Lines changed: 35 additions & 35 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -50,7 +50,7 @@ Run as a `pulsar-admin` extension::
50
50
The `pulsar-admin` extension is packaged with the IBM Elite Support for {pulsar} (formerly {company} Luna Streaming) distribution in the /cliextensions folder, so you don't need to build from source unless you want to make changes to the code.
51
51
+
52
52
.. Move the generated NAR archive to the /cliextensions folder of your {pulsar-short} installation (e.g. /pulsar/cliextensions).
53
-
..Modify the client.conf file of your {pulsar-short} installation to include: `customCommandFactories=cassandra-cdc`.
53
+
..Modify the client.conf file of your {pulsar-short} installation to include: `customCommandFactories=cassandra-cdc`.
54
54
.. Run the following command (this assumes the xref:6.9@dse:installing:tarball-dse.adoc[default tarball installation of {dse-short}]):
55
55
+
56
56
[source,shell]
@@ -264,86 +264,86 @@ In both the `pulsar-admin` extension and the standalone Java application, {cass-
264
264
|Parameter
265
265
|Description
266
266
267
-
|--data-dir=PATH
267
+
|`--data-dir=PATH`
268
268
|The directory where data is exported to and imported from. The
269
269
default is a 'data' subdirectory in the current working directory.
270
270
The data directory is created if it doesn't exist. Tables are exported in subdirectories of the data directory specified here;
271
271
there is one subdirectory per keyspace inside the data
272
272
directory, then one subdirectory per table inside each keyspace
273
273
directory.
274
274
275
-
|--help, -h
275
+
|`--help`, `-h`
276
276
|Displays this help message
277
277
278
-
|--dsbulk-log-dir=PATH, -l
278
+
|`--dsbulk-log-dir=PATH`, `-l`
279
279
|The directory where {dsbulk-short} should store its logs. The default is a
280
280
'logs' subdirectory in the current working directory. This
281
281
subdirectory is created if it doesn't exist. Each {dsbulk-short}
282
282
operation creates a subdirectory inside the log directory
283
283
specified here. This command isn't available in the `pulsar-admin` extension.
284
284
285
-
|--export-bundle=PATH
285
+
|`--export-bundle=PATH`
286
286
|The path to a {scb} to connect to an {astra-db} database. Options --export-host and --export-bundle are mutually exclusive.
287
287
288
-
|--export-consistency=CONSISTENCY
288
+
|`--export-consistency=CONSISTENCY`
289
289
|The consistency level to use when exporting data. The default is
290
290
LOCAL_QUORUM.
291
291
292
-
|--export-max-concurrent-files=NUM\|AUTO
292
+
|`--export-max-concurrent-files=NUM|AUTO`
293
293
|The maximum number of concurrent files to write to. Must be a positive
294
294
number or the special value AUTO. The default is AUTO.
295
295
296
-
|--export-max-concurrent-queries=NUM\|AUTO
296
+
|`--export-max-concurrent-queries=NUM|AUTO`
297
297
|The maximum number of concurrent queries to execute. Must be a
298
298
positive number or the special value AUTO. The default is AUTO.
299
299
300
-
|--export-splits=NUM\|NC
300
+
|`--export-splits=NUM|NC`
301
301
|The maximum number of token range queries to generate. Use the NC
302
302
syntax to specify a multiple of the number of available cores, e.g.
303
303
8C = 8 times the number of available cores. The default is 8C. This
304
304
is an advanced setting; you should rarely need to modify the default
305
305
value.
306
306
307
-
|--export-dsbulk-option=OPT=VALUE
307
+
|`--export-dsbulk-option=OPT=VALUE`
308
308
|An extra {dsbulk-short} option to use when exporting. Any valid {dsbulk-short} option
309
309
can be specified here, and it is passed as-is to the {dsbulk-short}
310
310
process. {dsbulk-short} options, including driver options, must be passed as
311
311
'--long.option.name=<value>'. Short options aren't supported.
312
312
313
-
|--export-host=HOST[:PORT]
313
+
|`--export-host=HOST[:PORT]`
314
314
|The host name or IP and, optionally, the port of a node from the
315
315
{cass-short} cluster. If the port isn't specified, it defaults to
316
316
9042. This option can be specified multiple times. Options
317
317
--export-host and --export-bundle are mutually exclusive.
318
318
319
-
|--export-password
319
+
|`--export-password`
320
320
|The password to use to authenticate against the origin cluster.
321
321
Options --export-username and --export-password must be provided
322
322
together, or not at all. Omit the parameter value to be prompted for
323
323
the password interactively.
324
324
325
-
|--export-protocol-version=VERSION
325
+
|`--export-protocol-version=VERSION`
326
326
|The protocol version to use to connect to the {cass-short} cluster, e.g.
327
327
'V4'. If not specified, the driver negotiates the highest
328
328
version supported by both the client and the server.
329
329
330
-
|--export-username=STRING
330
+
|`--export-username=STRING`
331
331
|The username to use to authenticate against the origin cluster.
332
332
Options --export-username and --export-password must be provided
333
333
together, or not at all.
334
334
335
-
|--keyspace=<keyspace>, -k
335
+
|`--keyspace=<keyspace>`, `-k`
336
336
|The name of the keyspace where the table to be exported exists
337
337
338
-
|--max-rows-per-second=PATH
338
+
|`--max-rows-per-second=PATH`
339
339
|The maximum number of rows per second to read from the {cass-short}
340
340
table. Setting this option to any negative value or zero
341
341
disables it. The default is -1.
342
342
343
-
|--table=<table>, -t
343
+
|`--table=<table>`, `-t`
344
344
|The name of the table to export data from for cdc back filling
345
345
346
-
|--version, -v
346
+
|`--version`, `-v`
347
347
|Displays version info.
348
348
|===
349
349
@@ -357,52 +357,52 @@ These parameters should be passed as command line arguments in the standalone Ja
357
357
|===
358
358
|Parameter |Description
359
359
360
-
|--events-topic-prefix=<topicPrefix>
360
+
|`--events-topic-prefix=<topicPrefix>`
361
361
|The event topic name prefix. The `<keyspace_name>.<table_name>` is appended to that prefix to build the topic name.
Copy file name to clipboardExpand all lines: docs/modules/ROOT/pages/cdc-cassandra-events.adoc
+17-15Lines changed: 17 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ The {product} agent pushes the mutation primary key for the CDC-enabled table in
7
7
8
8
In order to support https://pulsar.apache.org/docs/en/concepts-topic-compaction/[{pulsar-short} Topic Compaction], the message key is encoded separately from the message payload, in the message metadata.
9
9
10
-
Finally, the following CQL data types are encoded as AVRO logical types:
10
+
The following CQL data types are encoded as https://avro.apache.org/docs/current/spec.html#Logical+Types[AVRO logical types]:
11
11
12
12
* `Date`
13
13
* `Decimal`
@@ -16,15 +16,13 @@ Finally, the following CQL data types are encoded as AVRO logical types:
16
16
* `Varint`
17
17
* `Uuid`, `timeuuid`
18
18
19
-
See https://avro.apache.org/docs/current/spec.html#Logical+Types[AVRO Logical Types] for more info on AVRO.
20
-
21
19
== Change Event's Key
22
20
23
21
For a given table, the change event's key is an AVRO record that contains a field for each column in the primary key of the table at the time the event was created. Both the events and the data topics (also called the dirty and the clean topics) have the same message key, an AVRO record including the primary key columns.
24
22
25
23
== `INSERT` Event
26
24
27
-
Let's create a {cass-short} table to illustrate what happens:
25
+
Create a {cass-short} table to test this behavior:
28
26
29
27
[source,bash]
30
28
----
@@ -49,7 +47,7 @@ CREATE TABLE ks1.tbl1 (
49
47
AND speculative_retry = '99PERCENTILE';
50
48
----
51
49
52
-
Then insert a row:
50
+
Insert a row:
53
51
54
52
[source,bash]
55
53
----
@@ -89,6 +87,10 @@ You can check the connector status with the following command. The connector mus
89
87
[source,bash]
90
88
----
91
89
bin/pulsar-admin source status --name cassandra-source-ks1-table1
90
+
----
91
+
92
+
[source,json]
93
+
----
92
94
{
93
95
"numInstances" : 1,
94
96
"numRunning" : 1,
@@ -115,22 +117,24 @@ bin/pulsar-admin source status --name cassandra-source-ks1-table1
115
117
116
118
If you're having issues consuming CDC events, check the source connector logs on your {pulsar-short} function workers and the data topic schema.
117
119
118
-
=== Check the source connector logs
119
-
120
+
Check the source connector logs::
120
121
Check the source connector logs on your {pulsar-short} function workers. The name of the logs depends on the connectors' name.
Copy file name to clipboardExpand all lines: docs/modules/ROOT/pages/faqs.adoc
+14-14Lines changed: 14 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,7 +23,7 @@ From there, the data can be published to external platforms like Elasticsearch,
23
23
24
24
== How do I install {product-short}?
25
25
26
-
Follow the xref:install.adoc[installation instructions].
26
+
Follow the xref:ROOT:install.adoc[installation instructions].
27
27
28
28
== What are the requirements for {product-short}?
29
29
@@ -65,7 +65,7 @@ If the {pulsar-short} cluster is down, the change agent continues trying to send
65
65
When the disk space of the `cdc_raw` directory reaches your `cdc_total_space_in_mb` {cass-short} setting (less than 4 GB by default), writes to CDC-enabled tables fail with a `CDCWriteException`.
66
66
The following warning message is included in {cass-short} logs:
67
67
68
-
[source,bash]
68
+
[source,console]
69
69
----
70
70
WARN [CoreThread-5] 2021-10-29 09:12:52,790 NoSpamLogger.java:98 - Rejecting Mutation containing CDC-enabled table. Free up space in /mnt/data/cdc_raw.
71
71
----
@@ -93,7 +93,7 @@ SELECT * FROM system_distributed.cdc_local WHERE keyspace_name = 'keyspace_name'
93
93
94
94
There are three possible statuses:
95
95
96
-
Enabled::
96
+
`enabled`::
97
97
If the CDC status is `enabled`, then CDC is enabled on the table.
98
98
+
99
99
From this status, you can disable CDC on the table by running the following CQL query:
@@ -103,7 +103,7 @@ From this status, you can disable CDC on the table by running the following CQL
103
103
ALTER TABLE keyspace_name.table_name WITH cdc = {'enabled': false};
104
104
----
105
105
106
-
Disabled::
106
+
`disabled`::
107
107
If the CDC status is `disabled` then CDC is disabled on the table.
108
108
+
109
109
From this status, you can enable CDC on the table by running the following CQL query:
@@ -113,7 +113,7 @@ From this status, you can enable CDC on the table by running the following CQL q
113
113
ALTER TABLE keyspace_name.table_name WITH cdc = {'enabled': true};
114
114
----
115
115
116
-
Null::
116
+
`null`::
117
117
If the CDC status is `null` then CDC isn't enabled on the table.
118
118
+
119
119
From this status, you can enable CDC on the table by running the following CQL query:
@@ -134,7 +134,7 @@ SELECT * FROM system_distributed.cdc_local WHERE keyspace_name = 'cdc' AND table
134
134
135
135
There are three possible statuses:
136
136
137
-
Running::
137
+
`running`::
138
138
If the `status` column is `running`, then the agent is running.
139
139
+
140
140
From this status, you can stop the agent by running the following CQL query:
@@ -144,7 +144,7 @@ From this status, you can stop the agent by running the following CQL query:
144
144
ALTER TABLE cdc.raw_cdc WITH cdc = {'enabled': false};
145
145
----
146
146
147
-
Stopped::
147
+
`stopped`::
148
148
If the `status` column is `stopped` then the agent isn't running.
149
149
+
150
150
From this status, you can start the agent by running the following CQL query:
@@ -154,7 +154,7 @@ From this status, you can start the agent by running the following CQL query:
154
154
ALTER TABLE cdc.raw_cdc WITH cdc = {'enabled': true};
155
155
----
156
156
157
-
Null::
157
+
`null`::
158
158
If the `status` column is `null`, then the agent isn't running.
159
159
+
160
160
From this status, you can start the agent by running the following CQL query:
@@ -179,7 +179,7 @@ The design of CDC in {cass-short} assumed that when table changes are synchroniz
179
179
There is a max log size setting that disables writes to the table when the set threshold is reached.
180
180
If a connection to the {pulsar-short} cluster is needed for the log to be drained, and it isn't responsive, then the log begins to fill, which can impact a table's write availability.
181
181
182
-
For more, see the xref:cdc-for-cassandra:ROOT:install.adoc#scaling-up-your-configuration[Scaling up your CDC configuration].
182
+
For more, see the xref:ROOT:install.adoc#scaling-up-your-configuration[Scaling up your CDC configuration].
183
183
184
184
== Does the {csc_pulsar_first} use a dead-letter topic?
185
185
@@ -227,11 +227,11 @@ The most manageable way to handle this is to use the {pulsar-short} {cass-short}
227
227
228
228
The {cass-short} sink requires the following provisions:
229
229
230
-
- Use the CDC data topic as its source of messages
231
-
- Provide a secure bundle (creds) to another {cass-short} cluster
232
-
- Map message values to a specific table in the other cluster
233
-
- Use the {pulsar-short} delivery guarantee to ensure success
234
-
- Use the {pulsar-short} connector health metrics to monitor failures
230
+
* Use the CDC data topic as its source of messages
231
+
* Provide a secure bundle (creds) to another {cass-short} cluster
232
+
* Map message values to a specific table in the other cluster
233
+
* Use the {pulsar-short} delivery guarantee to ensure success
234
+
* Use the {pulsar-short} connector health metrics to monitor failures
0 commit comments