Skip to content

Commit 4adb68f

Browse files
Refine documentation for ClickHouse on GCP: update titles, enhance clarity, and improve image descriptions across multiple sections.
1 parent 4c5bdf7 commit 4adb68f

7 files changed

Lines changed: 57 additions & 41 deletions

File tree

content/learning-paths/servers-and-cloud-computing/clickhouse-gcp/_index.md

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,5 @@
11
---
2-
title: Build a Real-Time Analytics Pipeline with ClickHouse on Google Cloud Axion (Arm-based C4A VMs)
3-
4-
draft: true
5-
cascade:
6-
draft: true
2+
title: Build a real-time analytics pipeline with ClickHouse on Google Cloud Axion
73

84
minutes_to_complete: 50
95

content/learning-paths/servers-and-cloud-computing/clickhouse-gcp/benchmarking.md

Lines changed: 26 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ weight: 9
55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
8+
## ClickHouse benchmarking on Axion processors
89

910

1011
## ClickHouse Benchmarking on Axion Processors
@@ -23,13 +24,16 @@ SET use_query_cache = 0;
2324
```
2425
This ensures every query is executed fully and not served from cache.
2526

26-
### Validate Dataset Size
27-
Ensures enough data is present to produce meaningful latency results.
27+
### Validate dataset size
28+
29+
Ensure enough data is present to produce meaningful latency results by checking the current row count.
2830

2931
```console
3032
SELECT count(*) FROM realtime.logs;
3133
```
32-
You should see an output similar to:
34+
35+
The output is similar to:
36+
3337
```output
3438
┌─count()─┐
3539
1. │ 5000013 │ -- 5.00 million
@@ -48,20 +52,22 @@ SELECT
4852
FROM numbers(1000000);
4953
```
5054

51-
You should see an output similar to:
55+
The output is similar to:
56+
5257
```output
5358
Query id: 8fcbefab-fa40-4124-8f23-516fca2b8fdd
5459
Ok.
5560
1000000 rows in set. Elapsed: 0.058 sec. Processed 1.00 million rows, 8.00 MB (17.15 million rows/s., 137.20 MB/s.)
5661
Peak memory usage: 106.54 MiB.
5762
```
5863

59-
### Define Benchmark Queries
64+
### Define benchmark queries
65+
6066
These queries represent common real-time analytics patterns:
6167

62-
- **Filtered count** service-level analytics
63-
- **Time-windowed count** recent activity
64-
- **Aggregation by service** grouped analytics
68+
- **Filtered count**: service-level analytics
69+
- **Time-windowed count**: recent activity
70+
- **Aggregation by service**: grouped analytics
6571

6672
Each query scans and processes millions of rows to stress the execution engine.
6773

@@ -73,7 +79,7 @@ FROM realtime.logs
7379
WHERE service = 'service-5';
7480
```
7581

76-
You should see an output similar to:
82+
The output is similar to:
7783
```output
7884
Query id: cfbab386-7168-42ce-a752-2d5146f68b48
7985
@@ -92,7 +98,8 @@ FROM realtime.logs
9298
WHERE event_time >= now() - INTERVAL 10 MINUTE;
9399
```
94100

95-
You should see an output similar to:
101+
The output is similar to:
102+
96103
```output
97104
Query id: 7654746b-3068-4663-a5c6-6944d9c2d2b9
98105
┌─count()─┐
@@ -144,8 +151,9 @@ Query id: c48c0d30-0ef6-4fb9-bbb9-815a509a5f91
144151
Peak memory usage: 7.18 MiB.
145152
```
146153

147-
### Run Repeatable Latency Measurements
148-
To calculate reliable latency metrics, the same query is executed multiple times(10) using `clickhouse-client --time`.
154+
### Run repeatable latency measurements
155+
156+
To calculate reliable latency metrics, execute the same query multiple times (10 iterations) using `clickhouse-client --time`.
149157

150158
```sql
151159
clickhouse-client --time --query "
@@ -249,12 +257,15 @@ The 10th value in the sorted list is your p95 latency.
249257
p95 latency = 0.011 seconds ≈ 11 ms
250258
```
251259
252-
The ClickHouse query was executed 10 times on a GCP Axion (Arm) VM. Observed p95 query latency was ~11 ms, demonstrating consistently low-latency analytical performance on Arm-based infrastructure.
260+
After executing the ClickHouse query 10 times on the GCP Axion (Arm) VM, the observed p95 query latency was ~11 ms, demonstrating consistently low-latency analytical performance on Arm-based infrastructure.
261+
262+
## What you've accomplished
253263
254-
### Benchmark summary
255-
Results from the earlier run on the `c4a-standard-4` (4 vCPU, 16 GB memory) Arm64 VM in GCP (SUSE):
264+
You've successfully completed a comprehensive ClickHouse benchmarking exercise on Google Axion (Arm64) processors. Key results from the `c4a-standard-4` (4 vCPU, 16 GB memory) Arm64 VM running SUSE:
256265
257266
- ClickHouse on **Google Axion (Arm64)** delivered consistently low query latency, even while scanning ~6 million rows per query.
258267
- Across **10 repeat executions, the p95 latency was ~11 ms**, indicating stable and predictable performance.
259268
- Disabling the query cache ensured true execution latency, not cache-assisted results.
260269
- Analytical queries sustained **500M+ rows/sec throughput** with minimal memory usage.
270+
271+
Throughout this Learning Path, you provisioned an Arm-based VM on Google Cloud, deployed ClickHouse, configured a real-time streaming pipeline with Pub/Sub and Dataflow, and validated end-to-end analytical performance. You can now deploy, optimize, and benchmark ClickHouse workloads on Google Cloud Arm infrastructure.

content/learning-paths/servers-and-cloud-computing/clickhouse-gcp/dataflow-streaming-etl.md

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,11 @@ weight: 8
66
layout: learningpathall
77
---
88

9-
## Dataflow Streaming ETL (Pub/Sub → ClickHouse)
10-
This section implements a real-time streaming ETL pipeline that ingests events from Pub/Sub, processes them using Dataflow (Apache Beam), and writes them into ClickHouse running on a GCP Axion (Arm64) VM.
9+
## Dataflow streaming ETL (Pub/Sub → ClickHouse)
1110

12-
## Pipeline Overview
13-
Flow
11+
In this section you'll implement a real-time streaming ETL pipeline that ingests events from Pub/Sub, processes them using Dataflow (Apache Beam), and writes them into ClickHouse running on a GCP Axion (Arm64) VM.
12+
13+
## Pipeline overview
1414

1515
```bash
1616
Pub/Sub → Dataflow (Apache Beam) → ClickHouse (Axion VM)
@@ -23,7 +23,8 @@ Pub/Sub → Dataflow (Apache Beam) → ClickHouse (Axion VM)
2323
- ClickHouse: real-time analytical storage on Arm64
2424

2525
### Install Python 3.11 on the Axion VM
26-
Install Python 3.11 and the required system packages
26+
27+
Install Python 3.11 and the required system packages:
2728

2829
```console
2930
sudo zypper refresh
@@ -37,15 +38,17 @@ python3.11 --version
3738
pip3.11 --version
3839
```
3940

40-
### Create a Python Virtual Environment (Recommended)
41+
### Create a Python virtual environment (recommended)
42+
4143
Using a virtual environment avoids dependency conflicts with the system Python.
4244

4345
```console
4446
python3.11 -m venv beam-venv
4547
source beam-venv/bin/activate
4648
```
4749

48-
### Install Apache Beam with GCP Support
50+
### Install Apache Beam with GCP support
51+
4952
Install Apache Beam and the required dependencies for Dataflow:
5053

5154
```console
@@ -68,7 +71,7 @@ Connect to ClickHouse on the Axion VM:
6871
clickhouse client
6972
```
7073

71-
**Creates the target database and table for streaming inserts:**
74+
Create the target database and table for streaming inserts:
7275

7376
```sql
7477
CREATE DATABASE IF NOT EXISTS realtime;
@@ -104,7 +107,8 @@ Query id: aa25de9d-c07f-4538-803f-5473744631bc
104107
exit;
105108
```
106109

107-
### Validate Pub/Sub (Before Dataflow)
110+
### Validate Pub/Sub before Dataflow
111+
108112
Before running Dataflow, confirm that messages can be published and pulled.
109113

110114
**Publish a test message:**
@@ -276,4 +280,8 @@ This confirms:
276280
- ClickHouse ingests data on Axion (Arm64) via HTTP
277281
- The end-to-end real-time pipeline is operational
278282
279-
This pipeline serves as the foundation for ClickHouse latency benchmarking and real-time analytics on Google Axion.
283+
## What you've accomplished and what's next
284+
285+
You've successfully built a complete streaming ETL pipeline connecting Google Cloud Pub/Sub to ClickHouse running on an Arm-based Axion VM. The pipeline processes real-time events through Dataflow and stores them in ClickHouse for analytics.
286+
287+
This pipeline serves as the foundation for ClickHouse latency benchmarking and real-time analytics on Google Axion, which you'll perform in the next section.

content/learning-paths/servers-and-cloud-computing/clickhouse-gcp/gcp_firewall_setup.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,19 +20,19 @@ To expose the TCP port 8123, create a firewall rule.
2020

2121
Navigate to the [Google Cloud Console](https://console.cloud.google.com/), go to **VPC Network > Firewall**, and select **Create firewall rule**.
2222

23-
![Create a firewall rule alt-text#center](images/firewall-rule1.png "Create a firewall rule")
23+
![Google Cloud Console VPC Network Firewall page with Create Firewall Rule button highlighted alt-txt#center](images/firewall-rule1.png "Create a firewall rule")
2424

2525
Set the **Name** of the new rule to "allow-tcp-8123". Select your network that you intend to bind to your VM (default is "autoscaling-net", but your organization might have others).
2626

2727

28-
![Create a firewall rule alt-text#center](images/network-rule2.png "Creating the TCP/8123 firewall rule")
28+
![Firewall rule configuration screen showing name field and network selection dropdown alt-txt#center](images/network-rule2.png "Creating the TCP/8123 firewall rule")
2929

3030
Next, Set **Direction of traffic** to "Ingress". Set **Allow on match** to "Allow" and **Targets** to "Specified target tags". Set **Source IPv4 ranges** to "0.0.0.0/0".
3131

32-
![Create a firewall rule alt-text#center](images/network-rule3.png "Creating the TCP/8123 firewall rule")
32+
![Firewall rule settings displaying Ingress direction, Allow action, and source IP range configuration alt-txt#center](images/network-rule3.png "Creating the TCP/8123 firewall rule")
3333

3434
Finally, select **Specified protocols and ports** under the **Protocols and ports** section. Select the **TCP** checkbox, enter "8123" in the **Ports** text field, and select **Create**.
3535

36-
![Specifying the TCP port to expose alt-text#center](images/network-port.png "Specifying the TCP port to expose")
36+
![Protocols and ports section with TCP checkbox selected and port 8123 entered in text field alt-txt#center](images/network-port.png "Specifying the TCP port to expose")
3737

3838
The network firewall rule is now created, and you can continue with the VM creation.

content/learning-paths/servers-and-cloud-computing/clickhouse-gcp/installation.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,11 @@ layout: learningpathall
77
---
88

99
## Install ClickHouse and gcloud CLI on GCP VM
10-
This guide covers the installation of **Google Cloud CLI (gcloud)** and **ClickHouse** on a GCP SUSE Linux Arm64 (Axion C4A) VM.
10+
11+
In this section you'll install Google Cloud CLI (gcloud) and ClickHouse on your GCP SUSE Linux Arm64 (Axion C4A) VM.
1112
These tools are required to:
12-
- Interact with GCP services such as **Pub/Sub** and **Dataflow**
13-
- Store and query real-time analytics data efficiently using **ClickHouse on Arm64**
13+
- Interact with GCP services such as Pub/Sub and Dataflow
14+
- Store and query real-time analytics data efficiently using ClickHouse on Arm64
1415

1516
### Install Google Cloud CLI (gcloud)
1617
The Google Cloud CLI is required to authenticate with GCP, publish Pub/Sub messages, and submit Dataflow jobs from the VM.

content/learning-paths/servers-and-cloud-computing/clickhouse-gcp/instance.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ To create a virtual machine based on the C4A instance type:
2424
- Set **Series** to `C4A`.
2525
- Select `c4a-standard-4` for machine type.
2626

27-
![Create a Google Axion C4A Arm virtual machine in the Google Cloud Console with c4a-standard-4 selected alt-text#center](images/gcp-vm.png "Creating a Google Axion C4A Arm virtual machine in Google Cloud Console")
27+
![Google Cloud Console VM creation screen showing machine configuration with C4A series and c4a-standard-4 selected alt-txt#center](images/gcp-vm.png "Creating a Google Axion C4A Arm virtual machine in Google Cloud Console")
2828

2929

3030
- Under **OS and Storage**, select **Change**, then choose an Arm64-based OS image. For this Learning Path, use **SUSE Linux Enterprise Server**.
@@ -34,10 +34,10 @@ To create a virtual machine based on the C4A instance type:
3434
- Click **Create** to launch the instance.
3535
- Once created, you should see a "SSH" option to the right in your list of VM instances. Click on this to launch a SSH shell into your VM instance:
3636

37-
![Invoke a SSH session via your browser alt-text#center](images/gcp-ssh.png "Invoke a SSH session into your running VM instance")
37+
![Google Cloud Console VM instances list with SSH button highlighted for connecting to running instance alt-txt#center](images/gcp-ssh.png "Invoke a SSH session into your running VM instance")
3838

3939
- A window from your browser should come up and you should now see a shell into your VM instance:
4040

41-
![Terminal Shell in your VM instance alt-text#center](images/gcp-shell.png "Terminal shell in your VM instance")
41+
![Browser-based terminal window showing command prompt connected to VM instance alt-txt#center](images/gcp-shell.png "Terminal shell in your VM instance")
4242

4343
Next, let's install ClickHouse!

content/learning-paths/servers-and-cloud-computing/clickhouse-gcp/pub_sub_creation.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ Grant the following roles to the **Compute Engine default service account**:
8888
4. Assign the roles listed above
8989
5. Save
9090

91-
![ GCP onsole alt-text#center](images/roles.png "Figure 4: Required IAM Roles")
91+
![Google Cloud Console IAM page displaying assigned roles for Compute Engine service account including Dataflow and Pub/Sub permissions alt-txt#center](images/roles.png "Figure 4: Required IAM Roles")
9292

9393
VM OAuth scopes are limited by default. IAM roles are authoritative.
9494

0 commit comments

Comments
 (0)