Skip to content

Commit 627059b

Browse files
xeniapexeniape
authored
Merge branch 'main' into chore/update-monitoring-stack-for-nifi-metrics
2 parents 6fedbbe + 1beca43 commit 627059b

12 files changed

Lines changed: 150 additions & 56 deletions

File tree

‎demos/data-lakehouse-iceberg-trino-spark/create-spark-ingestion-job.yaml‎

Lines changed: 87 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ spec:
1111
spec:
1212
serviceAccountName: demo-serviceaccount
1313
initContainers:
14-
- name: wait-for-kafka
14+
- name: wait-for-resources
1515
image: oci.stackable.tech/sdp/tools:1.0.0-stackable0.0.0-dev
1616
command:
1717
- bash
@@ -23,6 +23,73 @@ spec:
2323
kubectl wait --for=condition=ready --timeout=30m pod -l app=minio,release=minio,stackable.tech/vendor=Stackable
2424
echo 'Waiting for all kafka brokers to be ready'
2525
kubectl wait --for=condition=ready --timeout=30m pod -l app.kubernetes.io/name=kafka,app.kubernetes.io/instance=kafka
26+
echo 'Waiting for all nifi instances to be ready'
27+
kubectl wait --for=condition=ready --timeout=30m pod -l app.kubernetes.io/name=nifi,app.kubernetes.io/instance=nifi
28+
- name: wait-for-kafka-topics
29+
image: oci.stackable.tech/sdp/kafka:3.9.1-stackable0.0.0-dev
30+
command:
31+
- bash
32+
- -euo
33+
- pipefail
34+
- -c
35+
- |
36+
#!/usr/bin/bash
37+
#
38+
# Wait for Kafka topics created by the Nifi workflows to be ready.
39+
# Also wait for all topic partitions to have a leader.
40+
# This is required for the Spark streaming job to be able to read from the topics.
41+
# Without this check, the Spark job might hang indefinitely without processing any events.
42+
43+
BROKER="${BROKER:-kafka-broker-default-0-listener-broker:9093}"
44+
45+
log() {
46+
level="$1"
47+
shift
48+
echo "[$level] $*"
49+
}
50+
51+
check_leaders() {
52+
local topic=$1
53+
local failed=0
54+
55+
log INFO "Starting leader check on Kafka broker: $BROKER for topic: $topic"
56+
metadata=$(kcat -b "$BROKER" -X security.protocol=SSL -X ssl.key.location=/stackable/tls-kcat/tls.key -X ssl.certificate.location=/stackable/tls-kcat/tls.crt -X ssl.ca.location=/stackable/tls-kcat/ca.crt -L -t "$topic" 2>/dev/null)
57+
58+
if [[ -z "$metadata" ]]; then
59+
log ERROR "Failed to retrieve metadata for topic: $topic"
60+
return 1
61+
fi
62+
63+
log DEBUG "Metadata for $topic:"
64+
echo "$metadata"
65+
66+
if echo "$metadata" | grep -q 'leader: -1'; then
67+
log ERROR "Found 'leader: -1' in topic '$topic'; topic not ready yet!"
68+
return 1
69+
fi
70+
71+
if echo "$metadata" | grep -q 'Broker: Leader not available'; then
72+
log ERROR "Topic '$topic' not available yet"
73+
return 1
74+
fi
75+
76+
log INFO "Check topic '$topic' was successful"
77+
return 0
78+
}
79+
80+
for topic in "shared_bikes_bike_status" "shared_bikes_station_status" "shared_bikes_station_information" "water_levels_measurements" "water_levels_stations"
81+
do
82+
result=$(check_leaders "$topic")
83+
echo "$result"
84+
if [ "$result" == "1" ]
85+
then
86+
exit 1
87+
fi
88+
done
89+
exit 0
90+
volumeMounts:
91+
- name: tls-kcat
92+
mountPath: /stackable/tls-kcat
2693
containers:
2794
- name: create-spark-ingestion-job
2895
image: oci.stackable.tech/sdp/tools:1.0.0-stackable0.0.0-dev
@@ -40,7 +107,26 @@ spec:
40107
- name: manifest
41108
configMap:
42109
name: create-spark-ingestion-job-manifest
110+
- name: tls-kcat
111+
ephemeral:
112+
volumeClaimTemplate:
113+
metadata:
114+
annotations:
115+
secrets.stackable.tech/backend.autotls.cert.lifetime: "1d"
116+
secrets.stackable.tech/class: "tls"
117+
secrets.stackable.tech/format: "tls-pem"
118+
secrets.stackable.tech/scope: "pod"
119+
spec:
120+
accessModes:
121+
- ReadWriteOnce
122+
resources:
123+
requests:
124+
storage: "1"
125+
storageClassName: secrets.stackable.tech
126+
volumeMode: Filesystem
43127
restartPolicy: OnFailure
128+
securityContext:
129+
fsGroup: 1000
44130
backoffLimit: 50
45131
---
46132
apiVersion: v1

‎demos/demos-v2.yaml‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,9 @@ demos:
2828
- plainYaml: https://raw.githubusercontent.com/stackabletech/demos/main/demos/hbase-hdfs-load-cycling-data/create-hfile-and-import-to-hbase.yaml
2929
supportedNamespaces: []
3030
resourceRequests:
31-
cpu: "3"
32-
memory: 5638Mi
33-
pvc: 16Gi
31+
cpu: 1950m
32+
memory: 9216Mi
33+
pvc: 21Gi
3434
end-to-end-security:
3535
description: Demonstrates end-to-end security across multiple products
3636
stackableStack: end-to-end-security
@@ -218,7 +218,6 @@ demos:
218218
- nifi
219219
- jupyterhub
220220
- grafana-dashboards
221-
- zookeeper
222221
manifests:
223222
- plainYaml: https://raw.githubusercontent.com/stackabletech/demos/main/demos/signal-processing/serviceaccount.yaml
224223
- plainYaml: https://raw.githubusercontent.com/stackabletech/demos/main/demos/signal-processing/create-timescale-tables.yaml

‎demos/nifi-kafka-druid-water-level-data/create-druid-ingestion-job.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,6 @@ data:
190190
],
191191
"targetRowsPerSegment": 5000000
192192
},
193-
"maxNumSegmentsToMerge": 500,
193+
"maxNumSegmentsToMerge": 500
194194
}
195195
}

‎demos/trino-taxi-data/superset-assets.zip‎

-183 Bytes
Binary file not shown.

‎docs/modules/demos/images/jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data/jupyter_hub_run_notebook.png‎

-343 KB
Binary file not shown.

‎docs/modules/demos/pages/hbase-hdfs-load-cycling-data.adoc‎

Lines changed: 33 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,9 @@ WARNING: This demo should not be run alongside other demos.
2121

2222
To run this demo, your system needs at least:
2323

24-
* 3 {k8s-cpu}[cpu units] (core/hyperthread)
25-
* 6GiB memory
26-
* 16GiB disk storage
24+
* 2 {k8s-cpu}[cpu units] (core/hyperthread)
25+
* 10 GiB memory
26+
* 22 GiB disk storage
2727
2828
== Overview
2929

@@ -52,26 +52,31 @@ To list the installed Stackable services run the following command: `stackablect
5252
----
5353
$ stackablectl stacklet list
5454
55-
┌───────────┬───────────┬───────────┬──────────────────────────────────────────────────────────────┬─────────────────────────────────┐
56-
│ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │
57-
╞═══════════╪═══════════╪═══════════╪══════════════════════════════════════════════════════════════╪═════════════════════════════════╡
58-
│ hbase ┆ hbase ┆ default ┆ regionserver 172.18.0.2:31521 ┆ Available, Reconciling, Running │
59-
│ ┆ ┆ ┆ ui-http 172.18.0.2:32064 ┆ │
60-
│ ┆ ┆ ┆ metrics 172.18.0.2:31372 ┆ │
61-
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
62-
│ hdfs ┆ hdfs ┆ default ┆ datanode-default-0-listener-data 172.18.0.2:31990 ┆ Available, Reconciling, Running │
63-
│ ┆ ┆ ┆ datanode-default-0-listener-http http://172.18.0.2:30659 ┆ │
64-
│ ┆ ┆ ┆ datanode-default-0-listener-ipc 172.18.0.2:30678 ┆ │
65-
│ ┆ ┆ ┆ datanode-default-0-listener-metrics 172.18.0.2:31531 ┆ │
66-
│ ┆ ┆ ┆ namenode-default-0-http http://172.18.0.2:32543 ┆ │
67-
│ ┆ ┆ ┆ namenode-default-0-metrics 172.18.0.2:30098 ┆ │
68-
│ ┆ ┆ ┆ namenode-default-0-rpc 172.18.0.2:30915 ┆ │
69-
│ ┆ ┆ ┆ namenode-default-1-http http://172.18.0.2:31333 ┆ │
70-
│ ┆ ┆ ┆ namenode-default-1-metrics 172.18.0.2:30862 ┆ │
71-
│ ┆ ┆ ┆ namenode-default-1-rpc 172.18.0.2:31440 ┆ │
72-
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
73-
│ zookeeper ┆ zookeeper ┆ default ┆ ┆ Available, Reconciling, Running │
74-
└───────────┴───────────┴───────────┴──────────────────────────────────────────────────────────────┴─────────────────────────────────┘
55+
┌───────────┬───────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────┐
56+
│ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │
57+
╞═══════════╪═══════════╪═══════════╪═══════════════════════════════════════════════════════════════════════════════════════════════╪═════════════════════════════════╡
58+
│ hbase ┆ hbase ┆ default ┆ master-default-0-listener-master 172.19.0.5:32001 ┆ Available, Reconciling, Running │
59+
│ ┆ ┆ ┆ master-default-0-listener-ui-http http://172.19.0.5:30709 ┆ │
60+
│ ┆ ┆ ┆ regionserver-default-0-listener-regionserver 172.19.0.4:32684 ┆ │
61+
│ ┆ ┆ ┆ regionserver-default-0-listener-ui-http http://172.19.0.4:31899 ┆ │
62+
│ ┆ ┆ ┆ regionserver-default-1-listener-regionserver 172.19.0.5:30604 ┆ │
63+
│ ┆ ┆ ┆ regionserver-default-1-listener-ui-http http://172.19.0.5:30274 ┆ │
64+
│ ┆ ┆ ┆ restserver-default-0-rest-http 172.19.0.5:32590 ┆ │
65+
│ ┆ ┆ ┆ restserver-default-0-ui-http http://172.19.0.5:31997 ┆ │
66+
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
67+
│ hdfs ┆ hdfs ┆ default ┆ datanode-default-0-listener-data 172.19.0.4:30883 ┆ Available, Reconciling, Running │
68+
│ ┆ ┆ ┆ datanode-default-0-listener-http http://172.19.0.4:32095 ┆ │
69+
│ ┆ ┆ ┆ datanode-default-0-listener-ipc 172.19.0.4:30086 ┆ │
70+
│ ┆ ┆ ┆ datanode-default-0-listener-metrics 172.19.0.4:31954 ┆ │
71+
│ ┆ ┆ ┆ namenode-default-0-http http://172.19.0.4:32035 ┆ │
72+
│ ┆ ┆ ┆ namenode-default-0-metrics 172.19.0.4:30468 ┆ │
73+
│ ┆ ┆ ┆ namenode-default-0-rpc 172.19.0.4:31966 ┆ │
74+
│ ┆ ┆ ┆ namenode-default-1-http http://172.19.0.5:31952 ┆ │
75+
│ ┆ ┆ ┆ namenode-default-1-metrics 172.19.0.5:32760 ┆ │
76+
│ ┆ ┆ ┆ namenode-default-1-rpc 172.19.0.5:32022 ┆ │
77+
├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
78+
│ zookeeper ┆ zookeeper ┆ default ┆ server-zk zookeeper-server.default.svc.cluster.local:2282 ┆ Available, Reconciling, Running │
79+
└───────────┴───────────┴───────────┴───────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────┘
7580
7681
----
7782

@@ -159,7 +164,8 @@ Took 13.4666 seconds
159164

160165
== Inspecting the Table
161166

162-
You can now use the table and the data. You can use all available HBase shell commands.
167+
You can now use the table and the data.
168+
You can use all available HBase shell commands.
163169

164170
[source,sql]
165171
----
@@ -192,10 +198,11 @@ COLUMN FAMILIES DESCRIPTION
192198
[TIP]
193199
====
194200
Run `stackablectl stacklet list` to get the address of the _ui-http_ endpoint.
195-
If the UI is unavailable, do a port-forward `kubectl port-forward hbase-master-default-0 16010`.
201+
Depending on the type of listener class that has been defined, the address will either be directly reachable via e.g. the NodePort IP and an externally reachable port, or via a port-forward (with e.g. `kubectl port-forward hbase-master-default-0 16010`).
196202
====
197203

198-
The HBase web UI will give you information on the status and metrics of your HBase cluster. See below for the start page.
204+
The HBase web UI will give you information on the status and metrics of your HBase cluster.
205+
See below for the start page.
199206

200207
image::hbase-hdfs-load-cycling-data/hbase-ui-start-page.png[]
201208

‎docs/modules/demos/pages/jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data.adoc‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
:jupyter: https://jupyter.org
1313
:spark-connect: https://spark.apache.org/docs/latest/spark-connect-overview.html
1414
:spark-connect-client: https://github.com/stackabletech/docker-images/blob/main/spark-connect-client/Dockerfile
15+
:repo-url: https://github.com/stackabletech/demos
1516

1617
This demo showcases the integration between {jupyterlab}[JupyterLab], {spark-connect}[Spark Connect] and {hadoop}[Apache Hadoop] deployed on the Stackable Data Platform (SDP) Kubernetes cluster.
1718
The SDP makes this integration easy by publishing a discovery ConfigMap for the HDFS cluster and a Spark Connect service.
@@ -115,12 +116,10 @@ image::jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data/jupyterlab_login.png[
115116
Log in with token `adminadmin`.
116117
You should arrive at your workspace:
117118

118-
image::jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data/jupyterlab_workspace.png[]
119+
image::jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data/jupyterlab_workspace.png[link={repo-url}/blob/{page-origin-refname}/stacks/jupyterhub-pyspark-hdfs/notebook.ipynb,linktarget=_blank,alt="JupyterLab Notebook"]
119120

120121
Now you can double-click on the `notebook` folder on the left, open and run the contained file.
121-
Click on the double arrow (⏩️) to execute the Python scripts (click on the image below to go to the notebook file).
122-
123-
image::jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data/jupyter_hub_run_notebook.png[link=https://github.com/stackabletech/demos/blob/main/stacks/jupyterhub-pyspark-hdfs/notebook.ipynb,window=_blank]
122+
Click on the double arrow (⏩️) to execute the Python scripts (click on the image above to go to the notebook file).
124123

125124
The Python notebook uses libraries such as `pandas` and `scikit-learn` to analyze the data.
126125
In addition, since the model training is delegated to a Spark Connect server, some of these dependencies, most notably `scikit-learn`, must also be made available on the Spark Connect pods.

0 commit comments

Comments
 (0)