You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/flink-on-gcp/background.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,22 +1,22 @@
1
1
---
2
-
title: Getting started with Apache Flink on Google Axion C4A (Arm Neoverse-V2)
2
+
title: Get started with Apache Flink on Google Axion C4A
3
3
4
4
weight: 2
5
5
6
6
layout: "learningpathall"
7
7
---
8
8
9
-
## Google Axion C4A Arm instances in Google Cloud
9
+
## Explore Google Axion C4A Arm instances
10
10
11
11
Google Axion C4A is a family of Arm-based virtual machines built on Google’s custom Axion CPU, which is based on Arm Neoverse-V2 cores. Designed for high-performance and energy-efficient computing, these virtual machines offer strong performance for modern cloud workloads such as CI/CD pipelines, microservices, media processing, and general-purpose applications.
12
12
13
-
The C4A series provides a cost-effective alternative to x86 virtual machines while leveraging the scalability and performance benefits of the Arm architecture in Google Cloud.
13
+
The C4A series provides a cost-effective alternative to x86 virtual machines while leveraging the scalability and performance benefits of Arm architecture in Google Cloud.
14
14
15
-
To learn more about Google Axion, refer to the [Introducing Google Axion Processors, our new Arm-based CPUs](https://cloud.google.com/blog/products/compute/introducing-googles-new-arm-based-cpu) blog.
15
+
To learn more about Google Axion, see the Google blog [Introducing Google Axion Processors, our new Arm-based CPUs](https://cloud.google.com/blog/products/compute/introducing-googles-new-arm-based-cpu).
16
16
17
-
## Apache Flink
17
+
## Explore Apache Flink
18
18
19
-
[Apache Flink](https://flink.apache.org/) is an open-source, distributed stream and batch data processing framework** developed under the [Apache Software Foundation](https://www.apache.org/).
19
+
[Apache Flink](https://flink.apache.org/) is an open-source, distributed stream and batch data processing framework developed under the [Apache Software Foundation](https://www.apache.org/).
20
20
21
21
Flink is designed for high-performance, low-latency, and stateful computations on both unbounded (streaming) and bounded (batch) data. It provides a robust runtime and APIs in Java, Scala, and Python for building scalable, fault-tolerant data processing pipelines.
title: Apache Flink Baseline Testing on Google Axion C4A Arm Virtual Machine
2
+
title: Test Flink baseline functionality
3
3
weight: 5
4
4
5
5
### FIXED, DO NOT MODIFY
@@ -12,7 +12,7 @@ In this section you will perform baseline testing for Apache Flink after install
12
12
### Install Maven (Required to Build and Run Flink Jobs)
13
13
Before running Flink jobs, ensure that Maven is installed on your VM. Many Flink examples and real-world jobs require Apache Maven to compile Java applications.
14
14
15
-
Download Maven and extract it:
15
+
### Install Maven
16
16
17
17
```console
18
18
cd /opt
@@ -21,21 +21,22 @@ sudo tar -xvzf apache-maven-3.8.6-bin.tar.gz
21
21
sudo mv apache-maven-3.8.6 /opt/maven
22
22
```
23
23
24
-
###Configure Environment Variables
24
+
## Configure environment variables
25
25
Configure the environment so Maven commands can be run system-wide:
OS name: "linux", version: "5.14.21-150500.55.124-default", arch: "aarch64", family: "unix"
46
47
```
47
48
48
-
At this point, both Java and Maven are installed and ready to use.
49
+
### Start the Flink cluster
49
50
50
-
### Start the Flink Cluster
51
51
Before launching Flink, open port 8081 in the Google Cloud Firewall Rules so that the Web UI is reachable externally.
52
52
53
-
Start the standalone Flink cluster using the provided startup script:
54
-
55
53
```console
56
54
cd $FLINK_HOME
57
55
./bin/start-cluster.sh
58
56
```
59
57
60
-
You should see output similar to:
58
+
The output is similar to:
59
+
61
60
```output
62
61
Starting cluster.
63
62
[INFO] 1 instance(s) of standalonesession are already running on lpprojectsusearm64.
64
63
Starting standalonesession daemon on host lpprojectsusearm64.
65
64
Starting taskexecutor daemon on host lpprojectsusearm64.
66
65
```
67
66
68
-
Verify that the Flink Processes (JobManager and TaskManager) are running:
67
+
Verify that the Flink processes (JobManager and TaskManager) are running:
69
68
70
69
```console
71
70
jps
72
71
```
73
72
74
-
You should see output similar to:
73
+
The output is similar to:
74
+
75
75
```output
76
76
21723 StandaloneSessionClusterEntrypoint
77
77
2621 Jps
78
78
2559 TaskManagerRunner
79
79
```
80
-
StandaloneSessionClusterEntrypoint is the JobManager process
81
-
TaskManagerRunner is the worker responsible for executing tasks and maintaining state.
80
+
81
+
`StandaloneSessionClusterEntrypoint` is the JobManager process, and `TaskManagerRunner` is the worker responsible for executing tasks and maintaining state.
![Screenshot of the Apache Flink Dashboard web interface showing the Overview page with cluster status, available task slots, running jobs count, and system metrics displayed in a clean web UI alt-text#center](images/flink-dashboard.png"Flink Dashboard")
92
90
93
91
A successfully loaded dashboard confirms the cluster network and UI functionality. This serves as the baseline for network and UI validation.
94
92
95
-
### Run a Simple Example Job
96
-
A basic sanity test is to run the built-in WordCount example:
93
+
### Run a simple example job
94
+
A basic check is to run the built-in WordCount example:
97
95
98
96
```console
99
97
cd $FLINK_HOME
100
98
./bin/flink run examples/streaming/WordCount.jar
101
99
```
102
-
You can monitor the job in the Web UI or check console logs. A successful WordCount run confirms that your Flink cluster lifecycle works end-to-end.
103
100
104
-
![Flink Dashboard alt-text#center](images/wordcount.png"Figure 2: Word Count Job")
101
+
You can monitor the job in the Web UI or check console logs.
102
+
103
+
![Screenshot of the Flink Dashboard showing a completed WordCount job with execution details, task metrics, and job timeline visible in the web interface alt-text#center](images/wordcount.png"WordCount job in Flink Dashboard")
105
104
106
-
Flink baseline testing has been completed. You can now proceed to Flink benchmarking.
105
+
Flink baseline testing is complete. You can now proceed to Flink benchmarking.
This section walks you through running Apache Flink microbenchmarks on a Google Cloud Axion C4A (Arm64) SUSE VM. You will clone the official Flink-benchmarks repository, build the benchmark suite, explore available tests, and run the Remote Channel Throughput Benchmark, one of the key indicators of Flink’s communication and data-transfer performance.
11
+
This section walks you through running Apache Flink microbenchmarks on a Google Cloud Axion C4A (Arm64) SUSE VM. You will clone the official Flink-benchmarks repository, build the benchmark suite, and run the Remote Channel Throughput Benchmark.
12
12
13
-
### Clone the Repository
13
+
### Clone the repository
14
14
Clone the official Flink microbenchmarks repository:
15
15
16
16
```console
@@ -20,39 +20,41 @@ cd flink-benchmarks
20
20
```
21
21
This repository contains microbenchmarks built using JMH (Java Microbenchmark Harness), widely used for JVM-level performance testing.
22
22
23
-
### Build the Benchmarks with Maven
23
+
### Build the benchmarks with Maven
24
24
Compile the benchmarks and create the executable JAR:
25
25
26
26
```console
27
27
mvn clean package -DskipTests
28
28
```
29
29
What this does:
30
-
* clean removes previous build artifacts
31
-
* package compiles the code and produces benchmark JARs
30
+
*`clean` removes previous build artifacts
31
+
*`package` compiles the code and produces benchmark JARs
32
32
* -DskipTests speeds up the build since unit tests aren’t needed for benchmarking
33
33
34
-
After building, the compiled **benchmarks.jar** files appear under:
34
+
After building, the compiled benchmarks.jar files appear under:
benchmarks.jar — Contains all Flink microbenchmarks packaged with JMH.
54
54
55
-
### List Available Benchmarks
55
+
`benchmarks.jar` contains all Flink microbenchmarks packaged with JMH.
56
+
57
+
## List available benchmarks
56
58
View all benchmarks included in the JAR:
57
59
58
60
```console
@@ -61,16 +63,18 @@ java -jar benchmarks.jar -l
61
63
-`-l` → Lists all benchmarks packaged in the JAR.
62
64
- This helps you identify which benchmarks you want to execute on your VM.
63
65
64
-
###Run Selected Benchmarks
66
+
## Run the Remote Channel Throughput benchmark
65
67
While the Flink benchmarking project includes multiple suites for state backends, windowing, checkpointing, and scheduler performance, in this Learning path you will run the Remote Channel Throughput benchmark to evaluate network and I/O performance.
66
68
67
-
**Remote Channel Throughput**: This benchmark measures the data transfer rate between remote channels in Flink, helping to evaluate network and I/O performance.
69
+
Remote Channel Throughput: This benchmark measures the data transfer rate between remote channels in Flink, helping to evaluate network and I/O performance.
### Apache Flink performance benchmarking observations on Arm64
108
+
##What you've accomplished and what's next
105
109
106
-
- Both the ALIGNED mode and DEBLOAT modes demonstrate a strong throughput on the Arm64 VM.
107
-
-The benchmark confirms that the Arm64 architecture efficiently handles Flink's remote channel throughput workloads.
110
+
You've successfully deployed Apache Flink on a Google Axion C4A Arm-based virtual machine, validated its functionality, and measured its performance using JMH-based microbenchmarks.
111
+
The benchmark results confirm that Google Axion C4A Arm-based instances deliver strong throughput in both ALIGNED and DEBLOAT modes, demonstrating that Arm64 architecture efficiently handles Flink's remote channel throughput workloads.
108
112
109
-
Overall, Arm64 VMs have shown that they are highly suitable for real-time Flink workloads, especially streaming analytics, ETL pipelines, and JVM-based microbenchmarks.
113
+
Arm64 VMs are highly suitable for real-time Flink workloads, including streaming analytics, ETL pipelines, and JVM-based microbenchmarks. The consistent performance across different modes demonstrates that Arm-based infrastructure can effectively handle Flink's demanding real-time processing requirements.
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/flink-on-gcp/installation.md
+13-11Lines changed: 13 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,12 +6,12 @@ weight: 4
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Install Apache Flink on GCP VM
10
-
In this section you will install Apache Flink and its required dependencies on a Google Cloud Platform (GCP) SUSE Arm64 Virtual Machine (VM). By the end, you will have a fully configured Flink environment ready for local execution, standalone clusters, or benchmarking Flink workloads on Arm.
9
+
## Install Apache Flink
10
+
11
+
This section shows you how to install Apache Flink and its required dependencies on a Google Cloud Platform (GCP) SUSE Arm64 virtual machine.
12
+
13
+
### Update the system and install Java
11
14
12
-
### Update the System and Install Java
13
-
Apache Flink requires a Java runtime (JRE) and development kit (JDK).
This command retrieves the official Flink binary distribution for installation on your VM.
30
30
31
31
{{% notice Note %}}
32
-
Flink 2.0.0 introduced Disaggregated State Management architecture, which enables more efficient resource utilization in cloud-native environments, ensuring high-performance real-time processing while minimizing resource overhead.
33
-
You can view [this release note](https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/)
32
+
Flink 2.0.0 introduced Disaggregated State Management architecture, which enables more efficient resource utilization in cloud-native environments, ensuring high-performance real-time processing while minimizing resource overhead. See [the Flink 2.0.0 Release Note](https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/).
34
33
35
34
For best performance on Arm, the [Arm Ecosystem Dashboard](https://developer.arm.com/ecosystem-dashboard/) recommends using Flink ≥ 2.0.0.
title: Create a Google Axion C4A Arm virtual machine on GCP
2
+
title: Create a Google Axion C4A Arm virtual machine
3
3
weight: 3
4
4
5
5
### FIXED, DO NOT MODIFY
@@ -8,13 +8,13 @@ layout: learningpathall
8
8
9
9
## Overview
10
10
11
-
In this section, you will create a Google Axion C4A Arm virtual machine on Google Cloud Platform (GCP) using the `c4a-standard-4` (4 vCPUs, 16 GB memory) machine type in the Google Cloud Console.
11
+
This section shows you how to create a Google Axion C4A Arm virtual machine on Google Cloud Platform (GCP) using the `c4a-standard-4` (4 vCPUs, 16 GB memory) machine type in the Google Cloud Console.
12
12
13
13
{{% notice Note %}}
14
-
For support on GCP setup, see the Learning Path [Getting started with Google Cloud Platform](https://learn.arm.com/learning-paths/servers-and-cloud-computing/csp/google/).
14
+
For support on GCP setup, see the Learning Path [Getting started with Google Cloud Platform](/learning-paths/servers-and-cloud-computing/csp/google/).
15
15
{{% /notice %}}
16
16
17
-
## Provision a Google Axion C4A Arm VM in Google Cloud Console
17
+
## Provision a Google Axion C4A Arm VM
18
18
19
19
To create a virtual machine based on the C4A instance type:
20
20
- Navigate to the [Google Cloud Console](https://console.cloud.google.com/).
@@ -24,8 +24,8 @@ To create a virtual machine based on the C4A instance type:
24
24
- Set **Series** to `C4A`.
25
25
- Select `c4a-standard-4` for machine type.
26
26
27
-
![Create a Google Axion C4A Arm virtual machine in the Google Cloud Console with c4a-standard-4 selected alt-text#center](images/gcp-vm.png"Creating a Google Axion C4A Arm virtual machine in Google Cloud Console")
27
+
![Screenshot of the Google Cloud Console showing the Compute Engine VM instance creation page with Machine configuration section expanded. The Series dropdown shows C4A selected, and the Machine type field displays c4a-standard-4 with specifications of 4 vCPUs and 16 GB memory visible alt-text#center](images/gcp-vm.png"Creating a Google Axion C4A Arm virtual machine in Google Cloud Console")
28
28
29
-
- Under **OS and Storage**, select **Change**, then choose an Arm64-based OS image. For this Learning Path, use **SUSE Linux Enterprise Server**. Pick the preferred version for your Operating System. Ensure you select the **Arm image** variant. Click**Select**.
29
+
- Under **OS and Storage**, select **Change**, then choose an Arm64-based OS image. For this Learning Path, use **SUSE Linux Enterprise Server**. Pick the preferred version for your Operating System. Ensure you select the **Arm image** variant. Select**Select**.
30
30
- Under **Networking**, enable **Allow HTTP traffic**.
0 commit comments