Skip to content

Commit 5b6a022

Browse files
razvanmaltesander
razvan
and
maltesander
authored
feat: create client.properties for use with kafka scripts (#898)
* create client.properties and update tls tests * fix: loose ends and cleanups (#900) * group kerberos code together * clean up bootstrap listener ports * update metrics.py to actually test for the existence of metrics * read jmx config from server.yaml instead of broker.yaml * update docs * update changelog * add comment about the kerberos situation * update docs to remove kcat references * remove comments from client.properties * fix topic creation * remove kcat from upgrade test * update changelog * Update CHANGELOG.md Co-authored-by: Malte Sander <contact@maltesander.com> * Apply suggestions from code review Co-authored-by: Malte Sander <contact@maltesander.com> * implement review feedback --------- Co-authored-by: Malte Sander <contact@maltesander.com>
1 parent d1c5516 commit 5b6a022

20 files changed

Lines changed: 303 additions & 386 deletions

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@ All notable changes to this project will be documented in this file.
1010
- Add experimental support for Kafka KRaft mode ([#889]).
1111
- Add experimental support for Kafka `4.1.0` ([#889]).
1212
- Add `prometheus.io/path|port|scheme` annotations to metrics service ([#897]).
13+
- Add `client.properties` to the Kafka configuration config map ([#898]).
14+
15+
Use this file together with the Kafka client shell scripts and preconfigured TLS settings.
16+
Unfortunately, when Kerberos is enabled this file is incomplete and must be edited first before it can be used.
1317

1418
### Changed
1519

@@ -21,11 +25,15 @@ All notable changes to this project will be documented in this file.
2125
- use the new `server.yaml` for jmx configuration
2226
- update metrics tests
2327
- update monitoring doc
28+
- Replace `kcat` with Kafka client scripts wherever possible ([#898]).
29+
30+
At the moment, `kcat` is still used for liveliness probes and Kerberos tests.
2431

2532
[#889]: https://github.com/stackabletech/kafka-operator/pull/889
2633
[#890]: https://github.com/stackabletech/kafka-operator/pull/890
2734
[#892]: https://github.com/stackabletech/kafka-operator/pull/892
2835
[#897]: https://github.com/stackabletech/kafka-operator/pull/897
36+
[#898]: https://github.com/stackabletech/kafka-operator/pull/898
2937
[#900]: https://github.com/stackabletech/kafka-operator/pull/900
3038

3139
## [25.7.0] - 2025-07-23

‎docs/modules/kafka/examples/getting_started/getting_started.sh‎

Lines changed: 30 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -90,28 +90,33 @@ trap "kill $PORT_FORWARD_PID" EXIT
9090

9191
sleep 15
9292

93-
echo "Creating test data"
94-
# tag::kcat-create-data[]
95-
echo "some test data" > data
96-
# end::kcat-create-data[]
97-
98-
echo "Writing test data"
99-
# tag::kcat-write-data[]
100-
kcat -b localhost:9092 -t test-data-topic -P data
101-
# end::kcat-write-data[]
102-
103-
echo "Reading test data"
104-
# tag::kcat-read-data[]
105-
kcat -b localhost:9092 -t test-data-topic -C -e > read-data.out
106-
# end::kcat-read-data[]
107-
108-
echo "Check contents"
109-
# tag::kcat-check-data[]
110-
grep "some test data" read-data.out
111-
# end::kcat-check-data[]
112-
113-
echo "Cleanup"
114-
# tag::kcat-cleanup-data[]
115-
rm data
116-
rm read-data.out
117-
# end::kcat-cleanup-data[]
93+
echo "Creating test topic test-data-topic"
94+
# tag::create-topic[]
95+
kubectl exec -n default simple-kafka-broker-default-0 -c kafka -t -- /stackable/kafka/bin/kafka-topics.sh \
96+
--create \
97+
--topic test-data-topic \
98+
--partitions 1 \
99+
--bootstrap-server localhost:9092
100+
# end::create-topic[]
101+
102+
echo "Publish test data"
103+
# tag::write-data[]
104+
kubectl exec -n default simple-kafka-broker-default-0 -c kafka -t -- /stackable/kafka/bin/kafka-producer-perf-test.sh \
105+
--producer-props bootstrap.servers=localhost:9092 \
106+
--topic test-data-topic \
107+
--payload-monotonic \
108+
--throughput 1 \
109+
--num-records 5
110+
# end::write-data[]
111+
112+
echo "Consume test data"
113+
# tag::read-data[]
114+
kubectl exec -n default simple-kafka-broker-default-0 -c kafka -t -- /stackable/kafka/bin/kafka-console-consumer.sh \
115+
--bootstrap-server localhost:9092 \
116+
--topic test-data-topic \
117+
--offset earliest \
118+
--partition 0 \
119+
--timeout-ms 1000
120+
# end::read-data[]
121+
122+
echo "Success!"

‎docs/modules/kafka/examples/getting_started/getting_started.sh.j2‎

Lines changed: 30 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -90,28 +90,33 @@ trap "kill $PORT_FORWARD_PID" EXIT
9090

9191
sleep 15
9292

93-
echo "Creating test data"
94-
# tag::kcat-create-data[]
95-
echo "some test data" > data
96-
# end::kcat-create-data[]
97-
98-
echo "Writing test data"
99-
# tag::kcat-write-data[]
100-
kcat -b localhost:9092 -t test-data-topic -P data
101-
# end::kcat-write-data[]
102-
103-
echo "Reading test data"
104-
# tag::kcat-read-data[]
105-
kcat -b localhost:9092 -t test-data-topic -C -e > read-data.out
106-
# end::kcat-read-data[]
107-
108-
echo "Check contents"
109-
# tag::kcat-check-data[]
110-
grep "some test data" read-data.out
111-
# end::kcat-check-data[]
112-
113-
echo "Cleanup"
114-
# tag::kcat-cleanup-data[]
115-
rm data
116-
rm read-data.out
117-
# end::kcat-cleanup-data[]
93+
echo "Creating test topic test-data-topic"
94+
# tag::create-topic[]
95+
kubectl exec -n default simple-kafka-broker-default-0 -c kafka -t -- /stackable/kafka/bin/kafka-topics.sh \
96+
--create \
97+
--topic test-data-topic \
98+
--partitions 1 \
99+
--bootstrap-server localhost:9092
100+
# end::create-topic[]
101+
102+
echo "Publish test data"
103+
# tag::write-data[]
104+
kubectl exec -n default simple-kafka-broker-default-0 -c kafka -t -- /stackable/kafka/bin/kafka-producer-perf-test.sh \
105+
--producer-props bootstrap.servers=localhost:9092 \
106+
--topic test-data-topic \
107+
--payload-monotonic \
108+
--throughput 1 \
109+
--num-records 5
110+
# end::write-data[]
111+
112+
echo "Consume test data"
113+
# tag::read-data[]
114+
kubectl exec -n default simple-kafka-broker-default-0 -c kafka -t -- /stackable/kafka/bin/kafka-console-consumer.sh \
115+
--bootstrap-server localhost:9092 \
116+
--topic test-data-topic \
117+
--offset earliest \
118+
--partition 0 \
119+
--timeout-ms 1000
120+
# end::read-data[]
121+
122+
echo "Success!"

‎docs/modules/kafka/pages/getting_started/first_steps.adoc‎

Lines changed: 52 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
= First steps
2-
:description: Deploy and verify a Kafka cluster on Kubernetes with Stackable Operators, including ZooKeeper setup and data testing using kcat.
3-
:kcat-install: https://github.com/edenhill/kcat#install
2+
:description: Deploy and verify a Kafka cluster on Kubernetes with Stackable Operators, including ZooKeeper setup and data testing.
43

54
After going through the xref:getting_started/installation.adoc[] section and having installed all the operators, you now deploy a Kafka cluster and the required dependencies.
65
Afterward you can <<_verify_that_it_works, verify that it works>> by producing test data into a topic and consuming it.
@@ -65,58 +64,90 @@ This creates the actual Kafka instance.
6564

6665
== Verify that it works
6766

68-
Next you produce data into a topic and read it via {kcat-install}[kcat].
69-
Depending on your platform you may need to replace `kafkacat` in the commands below with `kcat`.
67+
Next, use the Kafka client scripts to create a topic and publish and consume data.
7068

71-
First, make sure that all the Pods in the StatefulSets are ready:
69+
The Kafka operator has created a service called `simple-kafka-broker-default-bootstrap`.
70+
This service represents the endpoint clients should initially connect to in order to publish and consume data.
71+
First, make sure that the service exists and it is healthy:
7272

7373
[source,bash]
7474
----
75-
kubectl get statefulset
75+
kubectl describe svc simple-kafka-broker-default-bootstrap
7676
----
7777

78-
The output should show all pods ready:
78+
The output should look somewhat like this:
7979

8080
----
81-
NAME READY AGE
82-
simple-kafka-broker-default 3/3 5m
83-
simple-zk-server-default 3/3 7m
81+
Name: simple-kafka-broker-default-bootstrap
82+
Namespace: default
83+
Labels: app.kubernetes.io/component=broker
84+
app.kubernetes.io/instance=simple-kafka-broker-default-bootstrap
85+
app.kubernetes.io/managed-by=listeners.stackable.tech_listener
86+
app.kubernetes.io/name=listener
87+
app.kubernetes.io/role-group=default
88+
app.kubernetes.io/version=3.9.1-stackable0.0.0-dev
89+
stackable.tech/vendor=Stackable
90+
Annotations: <none>
91+
Selector: listener.stackable.tech/mnt.9555cbb6f38d4b0ca1771e6d83d28e27=simple-kafka-broker-default-bootstrap
92+
Type: NodePort
93+
IP Family Policy: SingleStack
94+
IP Families: IPv4
95+
IP: 10.105.88.52
96+
IPs: 10.105.88.52
97+
Port: kafka 9092/TCP
98+
TargetPort: 9092/TCP
99+
NodePort: kafka 32608/TCP
100+
Endpoints: 10.244.4.22:9092,10.244.4.24:9092,10.244.4.23:9092
101+
Session Affinity: None
102+
External Traffic Policy: Local
103+
Internal Traffic Policy: Cluster
104+
Events: <none>
84105
----
85106

86-
Then, create a port-forward for the Kafka Broker:
107+
The output shows that there are three endpoints serviced here.
108+
They correspond to the three broker pods belonging to the Kafka cluster.
109+
110+
Then, create a port-forward on this service:
87111

88112
----
89113
include::example$getting_started/getting_started.sh[tag=port-forwarding]
90114
----
91115

92-
Create a file containing some data:
116+
Now, create a new topic called `test-data-topic`:
93117

94118
----
95-
include::example$getting_started/getting_started.sh[tag=kcat-create-data]
119+
include::example$getting_started/getting_started.sh[tag=create-topic]
96120
----
97121

98-
Write that data:
122+
Use the Kafka performance producer script to send a couple of messages to the topic previously created:
99123

100124
----
101-
include::example$getting_started/getting_started.sh[tag=kcat-write-data]
125+
include::example$getting_started/getting_started.sh[tag=write-data]
102126
----
103127

104-
Read that data:
128+
The output should contain the following line:
105129

106130
----
107-
include::example$getting_started/getting_started.sh[tag=kcat-read-data]
131+
5 records sent, 1.138434 records/sec (0.00 MB/sec), 83.40 ms avg latency, 395.00 ms max latency, 3 ms 50th, 395 ms 95th, 395 ms 99th, 395 ms 99.9th.
108132
----
109133

110-
Check the content:
134+
This confirms that there were five messages sent to the topic and it also displays performance timers.
135+
We are not interested in any performance indicators but appreciate the fact that there were five unique messages that we consume later.
136+
137+
Now let's consume the messages from above:
111138

112139
----
113-
include::example$getting_started/getting_started.sh[tag=kcat-check-data]
140+
include::example$getting_started/getting_started.sh[tag=read-data]
114141
----
115142

116-
And clean up:
143+
The consumer should print the messages in between logging statements
117144

118145
----
119-
include::example$getting_started/getting_started.sh[tag=kcat-cleanup-data]
146+
0
147+
1
148+
2
149+
3
150+
4
120151
----
121152

122153
You successfully created a Kafka cluster and produced and consumed data.

‎docs/modules/kafka/pages/getting_started/index.adoc‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ You need:
1111
* a Kubernetes cluster
1212
* kubectl
1313
* optional: Helm
14-
* https://github.com/edenhill/kcat#install[kcat] for testing
1514

1615
Resource sizing depends on cluster type(s), usage and scope, but as a starting point a minimum of the following resources is recommended for this operator:
1716

‎docs/modules/kafka/pages/index.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ It is commonly used for real-time data processing, data ingestion, event streami
2121

2222
== Getting started
2323

24-
Follow the xref:kafka:getting_started/index.adoc[] which guides you through installing The Stackable Kafka and ZooKeeper operators, setting up ZooKeeper and Kafka and testing your Kafka using `kcat`.
24+
Follow the xref:kafka:getting_started/index.adoc[] which guides you through installing The Stackable Kafka and ZooKeeper operators, setting up ZooKeeper and Kafka and testing your Kafka installation.
2525

2626
== Resources
2727

0 commit comments

Comments
 (0)