Skip to content

Commit d12e437

Browse files
Final tweaks, including labelling diagram.
1 parent d2b2bd3 commit d12e437

7 files changed

Lines changed: 42 additions & 37 deletions

File tree

content/learning-paths/servers-and-cloud-computing/spark-on-azure/_index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Run Spark applications on the Microsoft Azure Cobalt 100 processors
2+
title: Run Spark applications on Microsoft Azure Cobalt 100 processors
33

44
minutes_to_complete: 60
55

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
---
2-
title: Overview of Microsoft Azure Cobalt 100, Azure Linux 3.0, and Apache Spark
2+
title: Getting started with Microsoft Azure Cobalt 100, Azure Linux 3.0, and Apache Spark
33
weight: 2
44

55
### FIXED, DO NOT MODIFY
66
layout: "learningpathall"
77
---
88

9-
## Learn about Azure Cobalt 100, Azure Linux 3.0, and Apache Spark
9+
## Key technologies for running Apache Spark on Azure Cobalt 100
1010

1111
This section introduces the key technologies you will use when running Spark applications on Microsoft Azure Cobalt 100 processors. You will learn about the Azure Cobalt 100 Arm-based processor, Azure Linux 3.0, and Apache Spark.
1212

@@ -16,16 +16,18 @@ Azure Cobalt 100 is Microsoft’s first-generation Arm-based processor, designed
1616

1717
Typical workloads include web and application servers, data analytics, open-source databases, and caching systems.
1818

19-
To learn more, see the blog [Announcing the preview of new Azure virtual machines based on the Azure Cobalt 100 processor](https://techcommunity.microsoft.com/blog/azurecompute/announcing-the-preview-of-new-azure-vms-based-on-the-azure-cobalt-100-processor/4146353).
19+
To learn more, see the Microsoft blog [Announcing the preview of new Azure virtual machines based on the Azure Cobalt 100 processor](https://techcommunity.microsoft.com/blog/azurecompute/announcing-the-preview-of-new-azure-vms-based-on-the-azure-cobalt-100-processor/4146353).
2020

2121
## Azure Linux 3.0
2222

23-
Azure Linux 3.0 is Microsoft’s lightweight Linux distribution optimized for cloud-native workloads on Azure. It is designed for performance, security, and reliability, and is supported by Microsoft. Azure Linux 3.0 is tailored for containers, microservices, and Kubernetes.
23+
Azure Linux 3.0 is Microsoft’s lightweight Linux distribution optimized for cloud-native workloads on Azure. It is designed for performance, security, and reliability. Azure Linux 3.0 is tailored for containers, microservices, and Kubernetes.
2424

2525
With native support for Arm64 (AArch64) architecture, Azure Linux 3.0 enables efficient execution of workloads on Arm-based infrastructure, making it a scalable and cost-effective choice for cloud deployments.
2626

2727
## Apache Spark
2828

2929
Apache Spark is an open-source, distributed computing system for fast, general-purpose big data processing. It provides high-level APIs in Java, Scala, Python, and R, and supports in-memory computation for improved performance.
3030

31-
Spark is widely used for large-scale data analytics, machine learning, and real-time data processing. Learn more at the [Apache Spark official website](https://spark.apache.org/) and in the [official documentation](https://spark.apache.org/docs/latest/).
31+
Spark is widely used for large-scale data analytics, machine learning, and real-time data processing.
32+
33+
Learn more at the [Apache Spark official website](https://spark.apache.org/) and in the [official documentation](https://spark.apache.org/docs/latest/).

content/learning-paths/servers-and-cloud-computing/spark-on-azure/baseline.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
---
2-
title: Functional validation of Apache Spark on Azure Cobalt 100 Arm64
2+
title: Validate Apache Spark on Azure Cobalt 100 Arm64 VMs
33
weight: 6
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9-
## Functional validation of Apache Spark on Azure Cobalt 100 Arm64
9+
## Run a functional test of Apache Spark on Azure Cobalt 100 Arm64
1010

1111
After installing Apache Spark on your Arm64 virtual machine, you can perform simple baseline testing to validate that Spark runs correctly and produces the expected output.
1212

@@ -47,6 +47,6 @@ You should see output similar to:
4747

4848
## Output summary
4949

50-
- Spark successfully generated code (10.5 ms) and executed a simple DataFrame operation
51-
- The test data **[1, "ARM64"]** and **[2, "Azure"]** was displayed before cleanly shutting down (exitCode 0)
52-
- This confirms a working Spark deployment on Arm64
50+
- Spark successfully generated code (10.5 ms) and executed a simple DataFrame operation.
51+
- The test data **[1, "ARM64"]** and **[2, "Azure"]** was displayed before cleanly shutting down (exitCode 0).
52+
- This confirms a working Spark deployment on Arm64.

content/learning-paths/servers-and-cloud-computing/spark-on-azure/benchmarking.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,7 @@ You should see output similar to the following:
181181
[info] broadcast nested loop join wholestage on 18857 18928 84 1.1 899.2 1.4X
182182
[success] Total time: 1644 s (27:24), completed Jul 25, 2025, 6:27:46 AM
183183
```
184-
### Benchmark Results Table Explained:
184+
## Benchmark Results Table Explained:
185185

186186
- **Best Time (ms):** Fastest execution time observed (in milliseconds).
187187
- **Avg Time (ms):** Average time across all iterations.
@@ -191,11 +191,11 @@ You should see output similar to the following:
191191
- **Relative Speed comparison:** baseline (1.0X) is the slower version.
192192

193193
{{% notice Note %}}
194-
Benchmarks were performed in both an Azure Linux 3.0 Docker container and an Azure Linux 3.0 virtual machine. Results were found to be comparable.
194+
Benchmark results on Azure Linux 3.0 were consistent across both Docker containers and virtual machines.
195195
{{% /notice %}}
196196

197197

198-
### Benchmark summary on Arm64:
198+
## Benchmark summary on Arm64:
199199
For easier comparison, shown here is a summary of benchmark results collected on an Arm64 `D4ps_v6` Azure virtual machine created from a custom Azure Linux 3.0 image using the AArch64 ISO.
200200
| Benchmark | Wholestage | Best Time (ms) | Avg Time (ms) | Stdev (ms) | Rate (M/s) | Per Row (ns) | Relative |
201201
|----------------------------------------|------------|----------------|----------------|------------|-------------|----------------|----------|
@@ -222,7 +222,7 @@ For easier comparison, shown here is a summary of benchmark results collected on
222222
| Broadcast nested loop join | Off | 26847 | 26870 | 32 | 0.8 | 1280.2 | 1.0X |
223223
| | On | 18857 | 18928 | 84 | 1.1 | 899.2 | 1.4X |
224224

225-
### Benchmark summary on x86_64:
225+
## Benchmark summary on x86_64:
226226
Shown here is a summary of the benchmark results collected on an `x86_64` `D4s_v4` Azure virtual machine using the Azure Linux 3.0 image published by Ntegral Inc.
227227
| Benchmark | Wholestage | Best Time (ms) | Avg Time (ms) | Stdev (ms) | Rate (M/s) | Per Row (ns) | Relative |
228228
|------------------------------------------|------------|----------------|----------------|------------|-------------|----------------|----------|
@@ -250,7 +250,7 @@ Shown here is a summary of the benchmark results collected on an `x86_64` `D4s_v
250250
| | On | 31254 | 31346 | 78 | 0.7 | 1490.3 | 1.2X |
251251

252252

253-
### Benchmark comparison insights
253+
## Benchmark comparison insights
254254

255255
When comparing the results on Arm64 vs x86_64 virtual machines:
256256

content/learning-paths/servers-and-cloud-computing/spark-on-azure/container-setup.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ You can deploy your Spark workload either in an Azure Linux 3.0 Docker container
1414

1515
The Azure Linux Container Host is an operating system image optimized for running container workloads on Azure Kubernetes Service (AKS). Microsoft maintains the Azure Linux Container Host, which is based on CBL-Mariner, an open-source Linux distribution created by Microsoft.
1616

17-
To learn more, see [What is Azure Linux Container Host for AKS](https://learn.microsoft.com/en-us/azure/azure-linux/intro-azure-linux).
17+
To learn more, see [What is Azure Linux Container Host for AKS?](https://learn.microsoft.com/en-us/azure/azure-linux/intro-azure-linux)
1818

1919
Azure Linux 3.0 supports AArch64. However, a standalone virtual machine image for Azure Linux 3.0 or CBL Mariner 3.0 is not yet available for Arm. To use the default Microsoft software stack, you can run a Docker container with Azure Linux 3.0 as the base image and run your Spark application inside the container.
2020

@@ -36,8 +36,10 @@ Currently, the Azure Marketplace offers official virtual machine images of Azure
3636

3737
For detailed steps, see [Create an Azure Linux 3.0 virtual machine with Cobalt 100 processors](/learning-paths/servers-and-cloud-computing/azure-vm).
3838

39-
---
39+
## Next steps
4040

41+
{{% notice Note %}}
4142
Whether you use an Azure Linux 3.0 Docker container or a virtual machine created from a custom image, the Spark deployment and benchmarking steps in the following sections remain the same.
43+
{{% /notice %}}
4244

4345
Once the setup is complete, continue to the next section to install and run Spark.
Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,31 @@
11
---
2-
title: Create an Arm-based virtual machine using Azure Cobalt 100
2+
title: Create an Azure Cobalt 100 Arm64 virtual machine
33
weight: 3
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9-
## Create an Arm-based virtual machine using Azure Cobalt 100
9+
## Create an Azure Cobalt 100 Arm64 VM using the Azure portal
1010

11-
You can create an Azure Cobalt 100 Arm-based virtual machine in several ways: using the Microsoft Azure console, the Azure CLI, or your preferred Infrastructure as Code (IaC) tool. In this Learning Path, you will use the Azure console to create a virtual machine with the Azure Cobalt 100 Arm-based processor.
11+
You can create an Azure Cobalt 100 Arm64 virtual machine in several ways, including the Azure portal, the Azure CLI, or an Infrastructure as Code (IaC) tool.
1212

13-
## Create an Arm-based Azure virtual machine
13+
In this Learning Path, you’ll use the Azure portal to create a VM with the Cobalt 100 processor, following a process similar to creating any other virtual machine in Azure.
1414

15-
Creating a virtual machine based on Azure Cobalt 100 is similar to creating any other virtual machine in Azure. To create an Azure virtual machine, open the Azure portal and go to **Virtual Machines**.
15+
## Step-by-step: create the virtual machine
1616

17-
1. Select **Create**
18-
2. Enter details such as **Name** and **Region**
19-
3. Choose the image for your virtual machine (for example, Ubuntu 24.04) and select **Arm64** as the architecture
20-
4. In the **Size** field, click **See all sizes**, then choose the D-Series v6 family of virtual machines
21-
5. Select **D4ps_v6** from the list and create the virtual machine
17+
1. In the Azure portal, go to **Virtual Machines** and select **Create**.
18+
2. Enter details such as **Name** and **Region**.
19+
3. Choose the image for your virtual machine (for example, Ubuntu 24.04) and select **Arm64** as the architecture.
20+
4. In the **Size** field, select **See all sizes**, then choose the D-Series v6 family of virtual machines.
21+
5. Select **D4ps_v6** from the list and create the virtual machine.
2222

23-
![Azure portal showing VM creation with D4ps_v6 selected](./instance-new.png)
23+
![Azure portal VM creation — Azure Cobalt 100 Arm64 virtual machine (D4ps_v6) alt-text#center](./instance-new.png "Figure 1: Create an Azure Cobalt 100 Arm64 VM in the Azure portal")
2424

25-
Once the virtual machine is running, you can SSH into it using the generated PEM key and the public IP address of the instance.
25+
Once the Arm64 virtual machine is running, you can SSH into it using the generated PEM key and the public IP address of the instance.
2626

2727
{{% notice Note %}}
2828

29-
To learn more about Arm-based virtual machines in Azure, see [Get started with Arm-based cloud instances](/learning-paths/servers-and-cloud-computing/csp/azure).
29+
To learn more about Arm-based virtual machines in Azure, see the Learning Path [Get started with Arm-based cloud instances](/learning-paths/servers-and-cloud-computing/csp/azure).
3030

3131
{{% /notice %}}

content/learning-paths/servers-and-cloud-computing/spark-on-azure/deploy.md

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,17 @@
11
---
2-
title: Install Apache Spark on Azure Cobalt 100 processors
2+
title: Install Apache Spark on Azure Cobalt 100 Arm64 processors
33
weight: 5
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9-
## Install Apache Spark
9+
## Install Apache Spark on Azure Cobalt 100 Arm64
1010
Within your running docker container image or your custom Azure Linux VM, follow the instructions to install Spark.
1111

1212
Start by installing Java, Python, and other essential tools:
13-
### Install Required Packages
13+
14+
## Install Java, Python, and tools for Apache Spark
1415

1516
```console
1617
sudo tdnf update -y
@@ -38,7 +39,7 @@ The output will look like:
3839
Python 3.12.9
3940
```
4041

41-
### Install Apache Spark on Arm
42+
## Download and install Apache Spark on Azure Cobalt 100 (Arm64)
4243

4344
You can now download and configure Apache Spark on your Arm-based machine:
4445

@@ -47,7 +48,7 @@ wget https://downloads.apache.org/spark/spark-3.5.6/spark-3.5.6-bin-hadoop3.tgz
4748
tar -xzf spark-3.5.6-bin-hadoop3.tgz
4849
sudo mv spark-3.5.6-bin-hadoop3 /opt/spark
4950
```
50-
### Set Environment Variables
51+
## Configure environment variables for Apache Spark
5152
Add this line to ~/.bashrc or ~/.zshrc to make the change persistent across terminal sessions.
5253

5354
```cosole
@@ -61,7 +62,7 @@ Apply changes immediately in your running shell:
6162
source ~/.bashrc
6263
```
6364

64-
### Verify Spark Installation
65+
## Verify Apache Spark installation on Azure Cobalt 100
6566

6667
```console
6768
spark-submit --version

0 commit comments

Comments
 (0)