diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/_index.md b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/_index.md new file mode 100644 index 0000000000..7317b853a9 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/_index.md @@ -0,0 +1,70 @@ +--- +title: Run Apache Spark SQL workloads on Azure Cobalt 100 Arm64 using Gluten and Velox for accelerated analytics + +draft: true +cascade: + draft: true + +minutes_to_complete: 120 + +who_is_this_for: This is an intermediate topic for data engineers, platform engineers, and developers who want to build and optimize high-performance Spark SQL workloads using native execution engines on Arm-based cloud environments. + +learning_objectives: + - Install and configure Hadoop, Spark, and Hive on Azure Cobalt 100 Arm64 virtual machines + - Build and integrate Gluten with the Velox backend for native query execution + - Configure Spark SQL for columnar and vectorized execution + - Generate and load TPC-DS datasets for benchmarking + - Run Spark SQL workloads and compare performance between vanilla Spark and Gluten + Velox + +prerequisites: + - A [Microsoft Azure account](https://azure.microsoft.com/) with access to Cobalt 100 based instances (Dpsv6) + - Basic knowledge of Linux command-line operations + - Familiarity with SSH and remote server access + - Basic understanding of distributed systems and Apache Spark + +author: Pareena Verma + +### Tags +skilllevels: Introductory +subjects: Performance and Architecture +cloud_service_providers: + - Microsoft Azure + +armips: + - Neoverse + +tools_software_languages: + - Apache Spark + - Hadoop + - Hive + - Gluten + - Velox + + +operatingsystems: + - Linux + +further_reading: + - resource: + title: Apache Spark Documentation + link: https://spark.apache.org/docs/latest/ + type: documentation + - resource: + title: Gluten Project (Apache Incubator) + link: https://github.com/apache/incubator-gluten + type: documentation + - resource: + title: Velox Execution Engine + link: https://github.com/facebookincubator/velox + type: documentation + - resource: + title: TPC-DS Benchmark Overview + link: https://www.tpc.org/tpcds/ + type: documentation + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 +layout: "learningpathall" +learning_path_main_page: "yes" +--- diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/_next-steps.md new file mode 100644 index 0000000000..c3db0de5a2 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/background.md b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/background.md new file mode 100644 index 0000000000..790f393337 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/background.md @@ -0,0 +1,63 @@ +--- +title: "Overview of Azure Cobalt 100 and Apache Spark with Gluten and Velox" + +weight: 2 + +layout: "learningpathall" +--- + +## Azure Cobalt 100 Arm-based processor + +Azure’s Cobalt 100 is Microsoft’s first-generation, in-house Arm-based processor. Built on Arm Neoverse N2, Cobalt 100 is a 64-bit CPU that delivers strong performance and energy efficiency for cloud-native, scale-out Linux workloads such as web and application servers, data analytics, open-source databases, and caching systems. Running at 3.4 GHz, Cobalt 100 allocates a dedicated physical core for each vCPU, which helps ensure consistent and predictable performance. + +To learn more, see the Microsoft blog [Announcing the preview of new Azure VMs based on the Azure Cobalt 100 processor](https://techcommunity.microsoft.com/blog/azurecompute/announcing-the-preview-of-new-azure-vms-based-on-the-azure-cobalt-100-processor/4146353). + +## Apache Spark with Gluten and Velox + +Apache Spark is an open-source distributed data processing engine designed for large-scale data analytics. It provides high-level APIs for SQL, streaming, machine learning, and graph processing, and is widely used for building data pipelines and analytical workloads. + +By default, Spark executes queries using the JVM (Java Virtual Machine), which can introduce overhead in CPU-intensive workloads. To address this, modern acceleration frameworks like **Gluten** and **Velox** enable native execution for improved performance. + +**Gluten** is an open-source Spark plugin that offloads Spark SQL execution from the JVM to native engines. It acts as a bridge between Spark and high-performance backends, enabling efficient query execution while maintaining compatibility with existing Spark workloads. + +**Velox** is a high-performance, vectorized execution engine written in C++. It is optimized for modern hardware, including Arm64 architectures such as Azure Cobalt 100. Velox processes data in a columnar format and uses vectorized execution to significantly reduce CPU overhead and improve query performance. + +Together, **Gluten + Velox** provide: + +- Native (off-JVM) execution of Spark SQL queries +- Vectorized processing for faster computation +- Reduced memory and CPU overhead +- Improved performance on Arm-based infrastructure + +To learn more, see: +- [Apache Spark Documentation](https://spark.apache.org/docs/latest/) +- [Gluten Project](https://github.com/apache/incubator-gluten) +- [Velox Engine](https://github.com/facebookincubator/velox) + + +### Key Capabilities + +- **Native Query Execution:** + Spark SQL queries are executed using Velox instead of JVM-based execution. + +- **Columnar Processing:** + Data is processed in columnar batches, improving cache efficiency and throughput. + +- **Vectorized Execution:** + Multiple data values are processed in a single CPU instruction, accelerating computation. + +- **Hardware Optimization:** + Velox is optimized for modern CPUs, including Arm64 (Azure Cobalt 100), delivering better performance per core. + +### In This Learning Path + +In this Learning Path, you will: + +- Deploy Apache Spark on an Azure Cobalt 100 Arm64 virtual machine +- Build and integrate Gluten with the Velox backend +- Configure Spark to use native execution +- Run Spark SQL workloads using Gluten + Velox +- Generate and load TPC-DS benchmark datasets +- Execute analytical queries and measure performance +- Compare accelerated workloads against vanilla Spark + diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/benchmarking.md b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/benchmarking.md new file mode 100644 index 0000000000..2d8515431d --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/benchmarking.md @@ -0,0 +1,427 @@ +--- +title: Run TPC-DS Benchmark on Spark with Gluten + Velox (Arm64) +weight: 5 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Run TPC-DS Benchmark on Spark (Gluten + Velox) + +This guide walks through running a **stable TPC-DS benchmark (10GB)** on Spark with Gluten + Velox. + +You will: + +- Generate dataset +- Upload to HDFS +- Convert to Parquet +- Run SQL queries +- Measure performance + +## Why This Approach? + +Traditional benchmarking methods (like spark-sql-perf) often fail due to: + +- Schema mismatches +- Missing columns +- Complex query dependencies + +To solve this, we use: + +- **Parquet format** → schema-safe & optimized +- **Manual SQL queries** → controlled execution +- **Local execution mode** → stable resource usage + +## Generate TPC-DS data + +```console +cd /opt +git clone https://github.com/gregrahn/tpcds-kit.git +cd tpcds-kit/tools +make +``` + +This builds the `dsdgen` tool used for data generation.\ + +## Generate 10GB dataset (stable) + +```console +mkdir -p /opt/tpcds-data + +./dsdgen -scale 10 -dir /opt/tpcds-data +``` + +- dsdgen generates industry-standard benchmark data +- Scale 10 = ~10GB dataset +- Used to simulate real analytics workloads + +**Verify:** + +```console +ls -lh /opt/tpcds-data | head +du -sh /opt/tpcds-data +``` +The output is similar to: + +```output +~12GB data +``` +Confirms successful dataset generation + +## Upload Data to HDFS + +```console +hdfs dfsadmin -safemode leave + +hdfs dfs -mkdir -p /ds/tpcds10_raw +hdfs dfs -put /opt/tpcds-data/* /ds/tpcds10_raw/ +``` + +**Why?** + +- HDFS acts as distributed storage layer +- Required for Spark-based analytics +- Safe mode must be disabled before writing + +## Verify HDFS data + +```console +hdfs dfs -du -h /ds/tpcds10_raw | head -30 +``` + +The output is similar to: + +```output +7.4 K 7.4 K /ds/tpcds10_raw/call_center +7.4 K 7.4 K /ds/tpcds10_raw/call_center.dat +1.6 M 1.6 M /ds/tpcds10_raw/catalog_page +1.6 M 1.6 M /ds/tpcds10_raw/catalog_page.dat +211.3 M 211.3 M /ds/tpcds10_raw/catalog_returns +211.3 M 211.3 M /ds/tpcds10_raw/catalog_returns.dat +2.8 G 2.8 G /ds/tpcds10_raw/catalog_sales +2.8 G 2.8 G /ds/tpcds10_raw/catalog_sales.dat +167.1 M 167.1 M /ds/tpcds10_raw/customer +63.8 M 63.8 M /ds/tpcds10_raw/customer.dat +0 0 /ds/tpcds10_raw/customer_address +26.4 M 26.4 M /ds/tpcds10_raw/customer_address.dat +0 0 /ds/tpcds10_raw/customer_demographics +76.9 M 76.9 M /ds/tpcds10_raw/customer_demographics.dat +9.8 M 9.8 M /ds/tpcds10_raw/date_dim +9.8 M 9.8 M /ds/tpcds10_raw/date_dim.dat +60 60 /ds/tpcds10_raw/dbgen_version.dat +81 81 /ds/tpcds10_raw/dbgen_version_1_4.dat +148.1 K 148.1 K /ds/tpcds10_raw/household_demographics +148.1 K 148.1 K /ds/tpcds10_raw/household_demographics.dat +328 328 /ds/tpcds10_raw/income_band +328 328 /ds/tpcds10_raw/income_band.dat +2.6 G 2.6 G /ds/tpcds10_raw/inventory +2.6 G 2.6 G /ds/tpcds10_raw/inventory.dat +27.5 M 27.5 M /ds/tpcds10_raw/item +``` + +Confirms all tables uploaded correctly + +## Prepare Parquet Directory + +```console +mkdir -p /opt/tpcds10_parquet +rm -rf /opt/tpcds10_parquet/* +``` + +- Parquet is a columnar format +- Faster than CSV +- Reduces IO and improves query speed + +## Start Spark (LOCAL MODE – STABLE) + +```console +$SPARK_HOME/bin/spark-shell \ +--master local[4] \ +--driver-memory 6g \ +--conf spark.sql.shuffle.partitions=32 \ +--conf spark.sql.adaptive.enabled=true \ +--conf spark.plugins="" +``` + +**Why Local Mode?** + +- Avoids YARN resource issues +- Ensures consistent benchmarking +- Ideal for single-node Arm VM + +## Convert CSV → Parquet + +```scala +val rawBase = "file:///opt/tpcds-data" +val pqBase = "file:///opt/tpcds10_parquet" + +val tables = Seq( +"call_center","catalog_page","catalog_returns","catalog_sales", +"customer","customer_address","customer_demographics", +"date_dim","household_demographics","income_band", +"inventory","item","promotion","reason","ship_mode", +"store","store_returns","store_sales", +"time_dim","warehouse","web_page","web_returns","web_sales","web_site" +) + +tables.foreach { t => + val df = spark.read + .option("delimiter", "|") + .option("inferSchema", "true") + .option("header", "false") + .csv(s"$rawBase/${t}*") + + df.write.mode("overwrite").parquet(s"$pqBase/$t") +} +``` + +- CSV = raw, slow +- Parquet = optimized, compressed +- Enables vectorized execution (important for Arm) + +## Validate Parquet Data + +```console +spark.read.parquet("file:///opt/tpcds10_parquet/store_sales").count() +spark.read.parquet("file:///opt/tpcds10_parquet/catalog_sales").count() +spark.read.parquet("file:///opt/tpcds10_parquet/web_sales").count() +``` + +The output is similar to: + +```output +res1: Long = 28800991 store_sales → 28M rows +res2: Long = 14401261 catalog_sales → 14M rows +res3: Long = 7197566 web_sales → 7M rows +``` +Confirms data correctness. + +## Register Tables + +```console +tables.foreach { t => + val df = spark.read.parquet(s"file:///opt/tpcds10_parquet/$t") + df.createOrReplaceTempView(t) +} +``` + +- Creates SQL-accessible tables +- Enables querying via Spark SQL + +## Verify tables + +```scala +spark.sql("show tables").show(50, false) +``` +All tables should be visible. + +The output is similar to: + +```output ++---------+----------------------+-----------+ +|namespace|tableName |isTemporary| ++---------+----------------------+-----------+ +| |call_center |true | +| |catalog_page |true | +| |catalog_returns |true | +| |catalog_sales |true | +| |customer |true | +| |customer_address |true | +| |customer_demographics |true | +| |date_dim |true | +| |household_demographics|true | +| |income_band |true | +| |inventory |true | +| |item |true | +| |promotion |true | +| |reason |true | +| |ship_mode |true | +| |store |true | +| |store_returns |true | +| |store_sales |true | +| |time_dim |true | +| |warehouse |true | +| |web_page |true | +| |web_returns |true | +| |web_sales |true | +| |web_site |true | ++---------+----------------------+-----------+ +``` + + +## Benchmark Function + +```scala +def timedQuery(name: String, sqlText: String): Unit = { + val t0 = System.nanoTime() + val df = spark.sql(sqlText) + df.count() + val t1 = System.nanoTime() + + println(s"$name took " + (t1 - t0) / 1e9 + " seconds") +} +``` + +- Measures query execution time +- Helps compare performance +- Standard benchmarking approach + +## Run Benchmark Queries + +### 1. Store Sales Aggregation + +```scala +timedQuery("q_store_sales_by_item", +""" +SELECT _c2 AS item_sk, SUM(_c22) AS total_sales +FROM store_sales +GROUP BY _c2 +ORDER BY total_sales DESC +""") + +The output is similar to: + +```output +q_store_sales_by_item took 1.548731698 seconds +``` + +### 2. Catalog Sales + +```scala +timedQuery("q_catalog_sales_by_item", +""" +SELECT _c15 AS item_sk, SUM(_c23) AS total_sales +FROM catalog_sales +GROUP BY _c15 +ORDER BY total_sales DESC +""") +``` + +The output is similar to: + +```output +q_catalog_sales_by_item took 0.795856122 seconds +``` + +### 3. Web Sales + +```scala +timedQuery("q_web_sales_by_item", +""" +SELECT _c3 AS item_sk, SUM(_c21) AS total_sales +FROM web_sales +GROUP BY _c3 +ORDER BY total_sales DESC +""") +``` + +The output is similar to: + +```output +q_web_sales_by_item took 0.423602822 seconds +``` + +### 4. Returns + +```scala +timedQuery("q_store_returns_by_item", +""" +SELECT _c2 AS item_sk, SUM(_c16) AS total_returns +FROM store_returns +GROUP BY _c2 +ORDER BY total_returns DESC +""") +``` + +The output is similar to: + +```output +q_store_returns_by_item took 0.264841719 seconds +``` + +### 5. Join Query + +```scala +timedQuery("q_join_store_sales_item", +""" +SELECT i._c0 AS item_sk, COUNT(*) AS cnt, SUM(s._c22) AS total_sales +FROM store_sales s +JOIN item i +ON s._c2 = i._c0 +GROUP BY i._c0 +ORDER BY total_sales DESC +""") +``` + +The output is similar to: + +```output +q_join_store_sales_item took 2.203225285 seconds +``` + +**What This Means** + +- ArmM VM handles analytics efficiently +- Parquet significantly improves performance +- Spark execution is stable and fast + +## Sample Result + +```scala +spark.sql(""" +SELECT _c2 AS item_sk, SUM(_c22) AS total_sales +FROM store_sales +GROUP BY _c2 +ORDER BY total_sales DESC +LIMIT 10 +""").show(false) +``` + +The output is similar to: + +```output ++-------+-------------------+ +|item_sk|total_sales | ++-------+-------------------+ +|40386 |-19058.600000000006| +|27714 |-24582.960000000003| +|45492 |-25120.229999999996| +|100578 |-25404.539999999997| +|62538 |-27089.640000000003| +|87474 |-27629.73 | +|47112 |-28303.099999999995| +|73650 |-30656.89 | +|43380 |-31415.079999999998| +|12552 |-31864.710000000006| ++-------+-------------------+ +``` + +## Save Benchmark Output + +```scala +import java.io.PrintWriter + +val out = new PrintWriter("/opt/tpcds10_benchmark_results.txt") + +out.println("TPC-DS 10GB Local Benchmark completed") + +out.close() +``` + +## FINAL RESULT + +- Data Generated: 10GB +- Format: Parquet +- Execution Mode: Local +- Queries Executed: 5 +- Execution Time: sub-second to ~2 sec + + +## What You Have Accomplished + +- Generated industry-standard benchmark dataset +- Built a stable Spark SQL execution pipeline +- Converted raw data into optimized format (Parquet) +- Executed analytical queries successfully +- Measured real performance on Arm64 + diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/final-vm.png b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/final-vm.png new file mode 100644 index 0000000000..5f09e1d343 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/final-vm.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/instance.png b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/instance.png new file mode 100644 index 0000000000..7f6f8cb94f Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/instance.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/instance1.png b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/instance1.png new file mode 100644 index 0000000000..b9d22c352d Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/instance1.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/instance4.png b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/instance4.png new file mode 100644 index 0000000000..2a0ff1e3b0 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/instance4.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/ubuntu-pro.png b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/ubuntu-pro.png new file mode 100644 index 0000000000..7c23b77783 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/images/ubuntu-pro.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/instance.md b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/instance.md new file mode 100644 index 0000000000..cd88503d77 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/instance.md @@ -0,0 +1,88 @@ +--- +title: Create an Azure Cobalt 100 Arm64 virtual machine +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Prerequisites and setup + +There are several common ways to create an Arm-based Cobalt 100 virtual machine, and you can choose the method that best fits your workflow or requirements: + +- The Azure Portal +- The Azure CLI +- An infrastructure as code (IaC) tool + +In this section, you'll launch the Azure Portal to create a virtual machine with the Arm-based Azure Cobalt 100 processor. + +This Learning Path focuses on general-purpose virtual machines in the Dpsv6 series. For more information, see the [Microsoft Azure guide for the Dpsv6 size series](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dpsv6-series). + +While the steps to create this instance are included here for convenience, you can also refer to the [Deploy a Cobalt 100 virtual machine on Azure Learning Path](/learning-paths/servers-and-cloud-computing/cobalt/). + +## Create an Arm-based Azure virtual machine + +Creating a virtual machine based on Azure Cobalt 100 is no different from creating any other virtual machine in Azure. To create an Azure virtual machine: + +- Launch the Azure portal and navigate to **Virtual Machines**. +- Select **Create**, and select **Virtual Machine** from the drop-down list. +- Inside the **Basic** tab, fill in the instance details such as **Virtual machine name** and **Region**. +- Select the image for your virtual machine (for example, Ubuntu 22.04 LTS Server) and select **Arm64** as the VM architecture. +- In the **Size** field, select **See all sizes** and select the D-Series v6 family of virtual machines. +- Select **D8ps_v6** from the list as shown in the diagram below: + +![Azure Portal showing D-Series v6 VM size selection with D8ps_v6 highlighted alt-txt#center](images/instance.png "Select D8ps_v6 from the D-Series v6 family") + +- For **Authentication type**, select **SSH public key**. + +{{% notice Note %}} +Azure generates an SSH key pair for you and lets you save it for future use. This method is fast, secure, and easy for connecting to your virtual machine. +{{% /notice %}} + +- Fill in the **Administrator username** for your VM. +- Select **Generate new key pair**, and select **RSA SSH Format** as the SSH Key Type. + +{{% notice Note %}} +RSA offers better security with keys longer than 3072 bits. +{{% /notice %}} + +- Give your SSH key a key pair name. +- In the **Inbound port rules**, select **HTTP (80)** and **SSH (22)** as the inbound ports, as shown below: + +![Azure Portal showing inbound port rules with HTTP (80) and SSH (22) selected alt-txt#center](images/instance1.png "Configure inbound port rules for HTTP and SSH access") + +- Now select the **Review + Create** tab and review the configuration for your virtual machine. It should look like the following: + +![Azure Portal Review + Create tab showing VM configuration summary ready for deployment alt-txt#center](images/ubuntu-pro.png "Review VM configuration before creation") + +- When you're happy with your selection, select the **Create** button and then **Download Private key and Create Resource** button. + +![Azure Portal showing Create button and SSH key download dialog alt-txt#center](images/instance4.png "Download SSH key and create the virtual machine") + +Your virtual machine should be ready and running in a few minutes. You can SSH into the virtual machine using the private key, along with the public IP details. + +![Azure Portal showing successful VM deployment with confirmation details alt-txt#center](images/final-vm.png "Successful VM deployment confirmation") + +{{% notice Note %}}To learn more about Arm-based virtual machines in Azure, see "Getting Started with Microsoft Azure" in [Get started with Arm-based cloud instances](/learning-paths/servers-and-cloud-computing/csp/azure/).{{% /notice %}} + +## What you've learned and what's next + +You have successfully created an Azure Cobalt 100 Arm64 virtual machine running **Ubuntu 22.04 LTS Server** with SSH authentication configured. The VM is now fully prepared for running distributed data processing workloads. + +On this VM, you have: + +- Set up a stable ARM64 environment +- Configured SSH access and hostname for cluster communication +- Prepared the system for big data stack installation (Hadoop, Spark, Hive) +- Ensured compatibility for Java 17 and ARM-based execution + +## What’s Next + +On this VM, you will now build a **high-performance Spark SQL analytics platform** using modern acceleration technologies. + +**You will:** + +- Install and configure **Hadoop (HDFS + YARN)** +- Install and configure **Apache Spark** +- Set up **Hive Metastore (MySQL-based)** +- Build and integrate **Gluten + Velox (native engine)** diff --git a/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/setup-and-gluten.md b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/setup-and-gluten.md new file mode 100644 index 0000000000..94922753da --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/spark-velox-cobalt/setup-and-gluten.md @@ -0,0 +1,393 @@ +--- +title: Deploy Spark SQL with Gluten + Velox on Arm64 (Stable Setup) +weight: 4 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Deploy Apache Spark with Gluten + Velox on Arm64 + +This guide helps you **set up Spark with native acceleration (Gluten + Velox)** on Arm64 (Azure Cobalt 100). + +We will build everything step-by-step from scratch. + +- Apache Hadoop +- Apache Spark +- Apache Hive Metastore +- Gluten + Velox native engine + +## Objective + +In this guide, you will: + +- Install Hadoop, Spark, and Hive +- Configure a single-node cluster +- Fix Java 17 compatibility issues +- Build Gluten with Velox backend +- Enable native execution (off-JVM) +- Prepare system for benchmarking + + +## Why Gluten + Velox? + +- Spark (default) runs on JVM ❌ +- Gluten + Velox runs queries in native C++ engine - + +**Benefits:** + +- Faster execution +- Lower CPU usage +- Better ARM performance + +## Environment + +| Component | Value | +|----------|------| +| Architecture | Arm64 | +| OS | Ubuntu 22.04 / 24.04 | +| CPU | 4–8 vCPU | +| RAM | 8–32 GB | +| Disk | ≥ 80 GB | + +## System preparation + +We install all required tools for: + +- Java (Spark/Hadoop) +- Build tools (Gluten) +- Database (Hive metastore) + +```console +sudo -i + +apt update -y +apt install -y \ +openjdk-17-jdk wget tar git curl unzip build-essential \ +python3-pip mysql-server maven cmake ninja-build pkg-config libssl-dev +``` + +These tools are required for: + +- Java runtime (Spark/Hadoop) +- Building Gluten (C++ dependencies) +- Hive metastore (MySQL) + +## Configure hostname +Hadoop requires proper hostname for internal communication. + +```console +hostnamectl set-hostname spark-master +exec console +``` + +**Why this matters:** + +- Required for Hadoop services +- Prevents cluster communication issues + +## Configure hosts + +Prevents connection errors (very important) + +```console +echo "127.0.0.1 spark-master" >> /etc/hosts +``` + +## Setup passwordless SSH + +**Why?** + +- Hadoop services use SSH internally. + +```console +ssh-keygen -t rsa -P "" +cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys +``` + +**Why:** + +- Required for Hadoop daemons +- Enables internal communication + +## Install Hadoop + +**Why?** + +Hadoop provides: + +- HDFS → Storage +- YARN → Resource manager + +```console +cd /opt + +wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1/hadoop-3.3.1-aarch64.tar.gz +tar -xvf hadoop-3.3.1-aarch64.tar.gz +ln -s hadoop-3.3.1 hadoop +``` + +## Install Spark + +Spark is the main engine for SQL and analytics. + +```console +wget https://archive.apache.org/dist/spark/spark-3.4.2/spark-3.4.2-bin-hadoop3.tgz +tar -xvf spark-3.4.2-bin-hadoop3.tgz +ln -s spark-3.4.2-bin-hadoop3 spark +``` + +## Install Hive + +Hive provides: + +- Metadata (table structure) +- SQL layer for Spark + +```cnsole +wget https://archive.apache.org/dist/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz +tar -xvf apache-hive-3.1.3-bin.tar.gz +ln -s apache-hive-3.1.3-bin hive +``` + +## Environment variables + +```console +cat >> ~/.consolerc < $HADOOP_HOME/etc/hadoop/core-site.xml < + +fs.defaultFS +hdfs://spark-master:9000 + + +EOF +``` + +**hdfs-site.xml** + +```console +cat > $HADOOP_HOME/etc/hadoop/core-site.xml < + +fs.defaultFS +hdfs://spark-master:9000 + + +EOF +``` + +**yarn-site.xml (ARM optimized)** + +```console +cat > $HADOOP_HOME/etc/hadoop/yarn-site.xml < +yarn.resourcemanager.hostnamespark-master +yarn.nodemanager.resource.memory-mb8192 +yarn.nodemanager.resource.cpu-vcores4 + +EOF +``` + +## Java 17 compatibility fix + +- Without this → Hadoop & Spark crash + +```console +cat >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh < $SPARK_HOME/conf/spark-defaults.conf <