diff --git a/content/install-guides/bolt.md b/content/install-guides/bolt.md deleted file mode 100644 index 1cfcc5d734..0000000000 --- a/content/install-guides/bolt.md +++ /dev/null @@ -1,243 +0,0 @@ ---- -title: BOLT - -additional_search_terms: -- linux -- optimization -- PGO - -minutes_to_complete: 20 - -author: Jonathan Davies - -official_docs: https://github.com/llvm/llvm-project/tree/main/bolt -description: Install BOLT on Linux AArch64 to optimize binary code layout using performance profiles and improve the runtime performance of large applications on Arm. - -test_images: -- ubuntu:latest -test_maintenance: true - -layout: installtoolsall -tool_install: true -multi_install: false -multitool_install_part: false -weight: 1 ---- - -BOLT is an open-source post-link binary optimization tool developed to speed up large applications. It does this by optimizing the application's code layout based on performance profile samples collected during execution. - -In this guide, you'll learn how to download and install BOLT. The instructions are for Debian-based Linux distributions, but can be adapted for other Linux distributions. - -{{% notice Note %}} -BOLT is provided as a built-in, ready-to-use component of the [Arm Toolchain for Linux](https://developer.arm.com/documentation/110477) suite. For more -information, see [How to use BOLT with our toolchain](https://developer.arm.com/documentation/110477/211/How-to-use-BOLT-with-our-toolchain). -{{% /notice %}} - -## Before you begin - -Follow these steps: - -1. Install Git - -[Install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) for your operating system. - -Many Linux distributions include Git so you may not need to install it. - -2. Install CMake - -```bash { target="ubuntu:latest" } -sudo apt install cmake -y -``` - -Check it is installed: - -```bash { target="ubuntu:latest" } -cmake --version -``` - -The output is similar to: - -```output -cmake version 3.28.3 - -CMake suite maintained and supported by Kitware (kitware.com/cmake). -``` - -For more information, see [CMake install guide.](/install-guides/cmake/) - -3. Install Ninja - -```bash { target="ubuntu:latest" } -sudo apt install ninja-build -y -``` - -Check it is installed: - -```bash { target="ubuntu:latest" } -ninja --version -``` - -The output is similar to: - -```output -1.11.1 -``` - -4. Install Clang - -```bash { target="ubuntu:latest" } -sudo apt install clang -y -``` - -Check it is installed: - -```bash { target="ubuntu:latest" } -clang --version -``` - -The output is similar to: - -```output -Ubuntu clang version 18.1.3 (1ubuntu1) -Target: aarch64-unknown-linux-gnu -Thread model: posix -InstalledDir: /usr/bin -``` - -5. Install xz-utils - -```bash -sudo apt-get install xz-utils -y -``` - -## Install BOLT - -You can install BOLT by building the source code or by downloading a binary release from GitHub. - -### Build and install BOLT from source code - -To build and install BOLT from source code, follow these steps: - -1. Clone the repository - -```bash -git clone https://github.com/llvm/llvm-project.git -``` - -2. Build BOLT and run it - -```bash -cd llvm-project -mkdir build -cd build -cmake -G Ninja ../llvm -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="bolt;clang;lld" -``` - -```console -ninja bolt -``` -{{% notice Note %}} -The build time depends on your machine configuration and may take several minutes to complete. -{{% /notice %}} - -3. Add the path to BOLT in your `.bashrc` file - -```bash -echo 'export PATH="$PATH:$HOME/llvm-project/build/bin"' >> ~/.bashrc -source ~/.bashrc -``` - -You are now ready to [verify BOLT is installed](#verify). - -### Install BOLT using a binary release - -To install BOLT using a binary release, follow these steps: - -{{% notice Note %}} -The following commands use BOLT version 22.1.3. The same commands work with other versions. Replace the file used in these steps with the file for your version of choice. To find the latest version, see [LLVM Project releases](https://github.com/llvm/llvm-project/releases). -{{% /notice %}} - -1. Download a binary release - -For Arm Linux, use the file with `ARM64` in the name: - -```bash -wget https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.3/LLVM-22.1.3-Linux-ARM64.tar.xz -``` - -2. Extract the downloaded file - -```bash -tar -xvf LLVM-22.1.3-Linux-ARM64.tar.xz -``` - -3. Add the path to BOLT in your `.bashrc` file - -```bash -echo 'export PATH="$PATH:$HOME/LLVM-22.1.3-Linux-ARM64/bin"' >> ~/.bashrc -source ~/.bashrc -``` - -### Verify BOLT is installed {#verify} - -To verify BOLT is installed, follow these steps: - -1. Confirm BOLT applications `perf2bolt` and `llvm-bolt` are installed - -Check the `perf2bolt` command: - -```console -perf2bolt -``` - -The output is similar to: - -```output -perf2bolt: Not enough positional command line arguments specified! -Must specify at least 1 positional argument: See: perf2bolt --help -``` - -Check the `llvm-bolt` command: - -```console -llvm-bolt -``` - -The output is similar to: - -```output -llvm-bolt: Not enough positional command line arguments specified! -Must specify at least 1 positional argument: See: llvm-bolt --help -``` - -2. Print the BOLT version - -```console -llvm-bolt --version -``` - -The output is similar to: - -```output -LLVM (http://llvm.org/): - LLVM version 22.1.3 - Optimized build. -BOLT revision e9846648fd6183ee6d8cbdb4502213fcf902a211 - - Registered Targets: - aarch64 - AArch64 (little endian) - aarch64_32 - AArch64 (little endian ILP32) - aarch64_be - AArch64 (big endian) - arm64 - ARM64 (little endian) - arm64_32 - ARM64 (little endian ILP32) - riscv32 - 32-bit RISC-V - riscv32be - 32-bit big endian RISC-V - riscv64 - 64-bit RISC-V - riscv64be - 64-bit big endian RISC-V - x86 - 32-bit X86: Pentium-Pro and above - x86-64 - 64-bit X86: EM64T and AMD64 -``` - -You'll see additional Registered Targets if you downloaded a binary release. - -You are now ready to use BOLT on your Linux machine. diff --git a/content/install-guides/llvm.md b/content/install-guides/llvm.md new file mode 100644 index 0000000000..20c54d51f4 --- /dev/null +++ b/content/install-guides/llvm.md @@ -0,0 +1,333 @@ +--- +title: LLVM toolchain for Linux on Arm + +additional_search_terms: +- bolt +- llvm-bolt +- perf2bolt +- llvm +- clang +- compiler +- lld +- pgo +- lto +- profdata +- profgen + +minutes_to_complete: 10 + +author: Paschalis Mpeis, Jonathan Davies + +official_docs: https://releases.llvm.org/download.html +description: Install the LLVM toolchain on Arm Linux to use Clang, LLD, BOLT, and profiling utilities. + +test_images: +- ubuntu:latest +test_maintenance: false + +layout: installtoolsall +tool_install: true +multi_install: false +multitool_install_part: false +weight: 1 +--- + +The LLVM toolchain includes Clang, LLD, BOLT, and other utilities for compiling, linking, profiling, and optimizing applications on Arm Linux. + +In this guide, you’ll learn how to install LLVM from a prebuilt release or from source, install BOLT from Linux distribution packages, and verify the installed tools. + +{{% notice Note %}} +BOLT is provided as a built-in, ready-to-use component of the [Arm Toolchain for Linux](https://developer.arm.com/documentation/110477) suite. For more +information, see [How to use BOLT with our toolchain](https://developer.arm.com/documentation/110477/211/How-to-use-BOLT-with-our-toolchain). +{{% /notice %}} + +## Before you begin {#before-you-begin} + +Confirm you are using an Arm machine by running: + +Run: + +```bash +uname -m +``` + +The output is similar to: + +```output +aarch64 +``` + +If you see a different result, you are not using an Arm computer running 64-bit Linux. + +There are three ways to install the LLVM tools covered in this guide: + +- [Install LLVM from a prebuilt release](#install-llvm-releases) to install Clang, LLD, BOLT, and the LLVM profiling tools. +- [Build LLVM from source](#build-llvm-source) if you need to build the LLVM toolchain yourself. +- [Install BOLT using a package manager](#install-bolt-packages) if you only need the BOLT tools and your Linux distribution provides a suitable version. + +## Install from LLVM Releases {#install-llvm-releases} + +You can install LLVM by downloading a prebuilt binary release from GitHub. + +Install the tools needed to download and extract the archive. +For Debian-based Linux distributions, including Ubuntu, install `wget` and `xz-utils`: + +```bash { target="ubuntu:latest" } +sudo apt update +sudo apt install wget xz-utils -y +``` + +The following commands use LLVM version 22.1.8. To install a different version, replace the filename in the commands below. For the latest releases, see [LLVM Project releases](https://github.com/llvm/llvm-project/releases). + +The commands extract LLVM to `$HOME/toolchain`. To install LLVM in a different location, adjust the extraction path and update the PATH environment variable accordingly. + +1. Download a binary release + +For Arm Linux, use the archive with `Linux-ARM64` in its filename: + + +```bash +mkdir -p $HOME/toolchain +cd $HOME/toolchain +wget https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/LLVM-22.1.8-Linux-ARM64.tar.xz +``` + +2. Extract the downloaded file + +```bash +tar -xvf LLVM-22.1.8-Linux-ARM64.tar.xz +``` + +3. Add the LLVM `bin` directory to your `PATH` + +This enables you to run LLVM tools such as `clang` and `lld` from any directory. +The command updates `PATH` for your current terminal session. +To make the change persistent, add the same command to your shell profile, such as `~/.bashrc`. + +```bash +export PATH="$HOME/toolchain/LLVM-22.1.8-Linux-ARM64/bin:$PATH" +``` + +## Install from Sources {#build-llvm-source} + +### Install source build dependencies + +Before building LLVM from source, install the required build tools. + +1. Install Git + +[Install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) for your operating system. + +Many Linux distributions include Git so you may not need to install it. + +2. Install CMake + +```bash { target="ubuntu:latest" } +sudo apt install cmake -y +``` + +Check it is installed: + +```bash { target="ubuntu:latest" } +cmake --version +``` + +The output is similar to: + +```output +cmake version 3.28.3 + +CMake suite maintained and supported by Kitware (kitware.com/cmake). +``` + +For more information, see [CMake install guide.](/install-guides/cmake/) + +3. Install Ninja + +```bash { target="ubuntu:latest" } +sudo apt install ninja-build -y +``` + +Check it is installed: + +```bash { target="ubuntu:latest" } +ninja --version +``` + +The output is similar to: + +```output +1.11.1 +``` + +4. Install Clang + +```bash { target="ubuntu:latest" } +sudo apt install clang -y +``` + +Check it is installed: + +```bash { target="ubuntu:latest" } +clang --version +``` + +The output is similar to: + +```output +Ubuntu clang version 18.1.3 (1ubuntu1) +Target: aarch64-unknown-linux-gnu +Thread model: posix +InstalledDir: /usr/bin +``` + +### Build BOLT from source {#build-bolt-source} + +To build and install BOLT from source code, follow these steps: + +1. Clone the repository + +```bash +git clone https://github.com/llvm/llvm-project.git +``` + +2. Build BOLT and run it + +```bash +cd llvm-project +mkdir build +cd build +cmake -G Ninja ../llvm -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="bolt;clang;lld" +``` + +```console +ninja bolt +``` +{{% notice Note %}} +The build time depends on your machine configuration and may take several minutes to complete. +{{% /notice %}} + +3. Add the path to BOLT in your `.bashrc` file + +```bash +echo 'export PATH="$PATH:$HOME/llvm-project/build/bin"' >> ~/.bashrc +source ~/.bashrc +``` + +After the build completes, continue to [verify the BOLT tools](#verify-bolt-tools). + +## Install BOLT using a package manager {#install-bolt-packages} + +Use a package manager if you prefer a system-managed installation. Package versions depend on your Linux distribution. + +{{< tabpane code=true >}} + +{{< tab header="Ubuntu/Debian" language="bash">}} +sudo apt update +sudo apt install llvm-bolt +{{< /tab >}} + +{{< tab header="Fedora" language="bash">}} +sudo dnf install llvm-bolt +{{< /tab >}} + +{{< tab header="openSUSE" language="bash">}} +sudo zypper install llvm-bolt +{{< /tab >}} + +{{< /tabpane >}} + +BOLT is available on Ubuntu 25.04 and later, Debian 13 and later, Fedora 42 and later, and on openSUSE Tumbleweed + +## Verify Clang compiler and LLD linker tools {#verify-compiler-tools} + +Run the following commands: + +```console +clang --version +clang++ --version +ld.lld --version +``` + +The output is similar to: + +```output +clang version 22.1.8 (https://github.com/llvm/llvm-project ca7933e47d3a3451d81e72ac174dcb5aa28b59d1) +Target: aarch64-unknown-linux-gnu +Thread model: posix +InstalledDir: /home/ubuntu/toolchain/LLVM-22.1.8-Linux-ARM64/bin + +clang version 22.1.8 (https://github.com/llvm/llvm-project ca7933e47d3a3451d81e72ac174dcb5aa28b59d1) +Target: aarch64-unknown-linux-gnu +Thread model: posix +InstalledDir: /home/ubuntu/toolchain/LLVM-22.1.8-Linux-ARM64/bin + +LLD 22.1.8 (https://github.com/llvm/llvm-project ca7933e47d3a3451d81e72ac174dcb5aa28b59d1) (compatible with GNU linkers) +``` + +Each command should print LLVM version information. + +### Compile and run a C++ example {#compile-cpp-example} + +Use a text editor to copy and paste the following code into a file named `hello.cpp`: + +```cpp { file_name="hello.cpp" } +#include + +int main() +{ + std::cout << "Hello, LLVM on Arm\n"; + return 0; +} +``` + +Compile the example with `clang++` and link it with LLD: + +```bash +clang++ -fuse-ld=lld hello.cpp -o hello +``` + +Run the example: + +```bash +./hello +``` + +The output is: + +```output +Hello, LLVM on Arm +``` + +You are now ready to use LLVM on your Arm Linux system. + + +## Verify the PGO profiling tools {#verify-pgo-tools} + +Run the following commands: + +```console +llvm-profdata --version +llvm-profgen --version +``` + +Each command should print LLVM version information. + +## Verify BOLT tools {#verify-bolt-tools} + +Run the following commands: + +```console +perf2bolt --version +llvm-bolt --version +``` + +The output is similar to: + +```output +LLVM (http://llvm.org/): + LLVM version 22.1.8 + Optimized build. +``` + +Each command should print LLVM version information. diff --git a/content/learning-paths/servers-and-cloud-computing/bolt-demo/_index.md b/content/learning-paths/servers-and-cloud-computing/bolt-demo/_index.md index 2e3c75d0f3..bd1bd60f9c 100644 --- a/content/learning-paths/servers-and-cloud-computing/bolt-demo/_index.md +++ b/content/learning-paths/servers-and-cloud-computing/bolt-demo/_index.md @@ -14,7 +14,7 @@ learning_objectives: - Evaluate the impact of BOLT optimizations using performance metrics and profiling data prerequisites: - - An AArch64 system running Linux with [perf](/install-guides/perf/) installed + - An AArch64 system running Linux with [LLVM BOLT](/install-guides/llvm/#verify-bolt-tools) and [perf](/install-guides/perf/) installed - Linux kernel version 6.17 or later to enable Branch Record Buffer Extension ([BRBE profiling](/learning-paths/servers-and-cloud-computing/bolt-demo/brbe/)) - Linux kernel version 6.14 or later for Arm Statistical Profiling Extension ([SPE profiling](/learning-paths/servers-and-cloud-computing/bolt-demo/spe/)) - GCC version 13.3 or later to compile the example program ([GCC](/install-guides/gcc/) ) @@ -121,4 +121,3 @@ weight: 1 # _index.md always has weight of 1 to order corr layout: "learningpathall" # All files under learning paths have this same wrapper learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. --- - diff --git a/content/learning-paths/servers-and-cloud-computing/bolt-demo/install.md b/content/learning-paths/servers-and-cloud-computing/bolt-demo/install.md deleted file mode 100644 index 2552772735..0000000000 --- a/content/learning-paths/servers-and-cloud-computing/bolt-demo/install.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -title: Install BOLT on Linux -weight: 3 - -### FIXED, DO NOT MODIFY -layout: learningpathall ---- - -## Install BOLT on Linux - -This guide requires LLVM BOLT 22.1.0 or later for SPE profiling and required options. -Package manager versions might be older, so verify the installed version before continuing. - - -## Install BOLT from LLVM releases - -Install BOLT from a prebuilt [LLVM release](https://github.com/llvm/llvm-project/releases). -This method provides a consistent version across systems. It also lets you use newer releases when available. -The following example uses LLVM 22.1.5, the latest LLVM release available at the time of writing (May 2026). - -{{% notice Please Note %}} - -If you are using a 1st generation Arm AGI CPU, we recommend installing the latest LLVM release to ensure support for the processor. However due to backwards compatibility, LLVM BOLT 22.1.0 or later can still be used to complete this learning path. - -Arm AGI CPU support is expected to be introduced no earlier than LLVM 23. - -{{% /notice %}} - - -Download and extract LLVM: - -```bash -wget https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.5/LLVM-22.1.5-Linux-ARM64.tar.xz -tar xf LLVM-22.1.5-Linux-ARM64.tar.xz -``` - -Add LLVM tools to your PATH: - -```bash -export PATH="$(pwd)/LLVM-22.1.5-Linux-ARM64/bin:$PATH" -``` - - -## Install BOLT using a package manager - -Use a package manager if you prefer a system-managed installation. Package versions depend on your Linux distribution. - - - -{{< tabpane code=true >}} - -{{< tab header="Ubuntu/Debian" language="bash">}} -sudo apt update -sudo apt install llvm-bolt -{{< /tab >}} - -{{< tab header="Fedora" language="bash">}} -sudo dnf install llvm-bolt -{{< /tab >}} - -{{< tab header="openSUSE" language="bash">}} -sudo zypper install llvm-bolt -{{< /tab >}} - -{{< /tabpane >}} - - -BOLT is available on Ubuntu 25.04 and later, Debian 13 and later, Fedora 42 and later, and on openSUSE Tumbleweed. - -## Verify the installation - -```bash -llvm-bolt --version -``` - -The command prints the BOLT version. If the command fails, check your PATH. diff --git a/content/learning-paths/servers-and-cloud-computing/bolt-demo/setup.md b/content/learning-paths/servers-and-cloud-computing/bolt-demo/setup.md index fcc69af504..2c1f4fdbc8 100644 --- a/content/learning-paths/servers-and-cloud-computing/bolt-demo/setup.md +++ b/content/learning-paths/servers-and-cloud-computing/bolt-demo/setup.md @@ -8,6 +8,19 @@ layout: learningpathall ## Set up your environment + +This Learning Path assumes that you have installed LLVM BOLT before continuing. If BOLT is not installed, [install LLVM from a prebuilt release](/install-guides/llvm/#install-llvm-releases), or [install BOLT from your Linux distribution packages](/install-guides/llvm/#install-bolt-packages) if they provide a suitable version. + +This Learning Path requires LLVM BOLT 22.1.0 or later for SPE profiling and the required BOLT options. Package manager versions might be older, so [verify the BOLT tools](/install-guides/llvm/#verify-bolt-tools) before continuing. + +{{% notice Please Note %}} + +If you are using a 1st generation Arm AGI CPU, we recommend installing the latest LLVM release to ensure support for the processor. However due to backwards compatibility, LLVM BOLT 22.1.0 or later can still be used to complete this learning path. + +For Arm AGI CPU support, we recommend using LLVM 23 or later. + +{{% /notice %}} + On your AArch64 Linux machine, navigate to your home directory (or another empty working directory) and download the `bsort.cpp` source file: ```bash diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/_index.md b/content/learning-paths/servers-and-cloud-computing/pgo/_index.md new file mode 100644 index 0000000000..93d2bc3063 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/_index.md @@ -0,0 +1,70 @@ +--- +title: "Optimize AArch64 code with LLVM LTO and PGO" +description: Learn how to use LLVM Link-Time Optimization and Profile-Guided Optimization on AArch64 Linux. +minutes_to_complete: 45 +aliases: + - /learning-paths/servers-and-cloud-computing/pgo-test/ + +who_is_this_for: This is an introductory topic for developers who compile C or C++ applications on AArch64 Linux and want to use Link-Time Optimization (LTO) with Profile-Guided Optimization (PGO). + + +learning_objectives: + - Understand the basics of Link-Time Optimization (LTO) and Profile-Guided Optimization (PGO) + - Build Thin-LTO and Full-LTO binaries with Clang on AArch64 + - Generate instrumentation-based profile data with Clang on AArch64 + - Generate sample-based profile data with Clang on AArch64 + - Use generated profile data to optimize a small example application + + +prerequisites: + - LLVM installed with Clang, LLD, `llvm-profdata`, and `llvm-profgen` available in your `PATH`. For setup instructions, see [LLVM toolchain for Linux on Arm](/install-guides/llvm/). + - Linux kernel version 6.17 or later for the Branch Record Buffer Extension (BRBE) profile-guided optimization workflow + + +author: Paschalis Mpeis + +### Tags +skilllevels: Introductory +subjects: Performance and Architecture +armips: + - Neoverse + - Cortex-A +tools_software_languages: + - Clang + - LLVM + - LTO + - PGO + - perf + +operatingsystems: + - Linux + +further_reading: + - resource: + title: Clang profile-guided optimization + link: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization + type: documentation + - resource: + title: llvm-profdata command guide + link: https://llvm.org/docs/CommandGuide/llvm-profdata.html + type: documentation + - resource: + title: llvm-profgen command guide + link: https://llvm.org/docs/CommandGuide/llvm-profgen.html + type: documentation + - resource: + title: LLVM LTO + link: https://llvm.org/docs/LinkTimeOptimization.html + type: documentation + - resource: + title: LLVM Thin-LTO + link: https://clang.llvm.org/docs/Thin-LTO.html + type: documentation + + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/pgo/_next-steps.md new file mode 100644 index 0000000000..c3db0de5a2 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/csir-pgo.md b/content/learning-paths/servers-and-cloud-computing/pgo/csir-pgo.md new file mode 100644 index 0000000000..83196f7eda --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/csir-pgo.md @@ -0,0 +1,86 @@ +--- +title: Optimize with CSIR-PGO +weight: 8 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is CSIR-PGO? + +CSIR-PGO extends IR-PGO by adding a second, context-sensitive profiling pass. +The first pass is the standard IR-PGO instrumentation pass. The second pass instruments the program after inlining, which enables LLVM to distinguish execution counts from different calling contexts. +The additional context can improve optimization when function behavior depends on the call site, but it does not guarantee better performance for every program. + +Use CSIR-PGO when you want to provide LLVM with more detailed profile information and can afford an extra build and training run. + +## Build the context-sensitive instrumented binary + +First, generate a standard IR-PGO profile as described in the [previous section](/learning-paths/servers-and-cloud-computing/pgo/ir-pgo/). +We reuse the resulting `prof/ir.profdata` profile to build a second instrumented binary with `-fcs-profile-generate`, which adds context-sensitive instrumentation after inlining. + +```bash +clang++ -O3 -flto -fuse-ld=lld \ + -fprofile-use=prof/ir.profdata \ + -fcs-profile-generate=prof/csir \ + bsort.cpp -o out/bsort.csirpgo.instr +``` + +Run the context-sensitive instrumented binary: + +```bash +./out/bsort.csirpgo.instr +``` + +After the program exits, the raw profiles are written to: + +```bash +ls prof/csir/*.profraw +``` + +## Merge, Convert, and Inspect the profile + +As with IR-PGO, the context-sensitive training run can produce one or more raw `.profraw` files. Merge those files together with the existing `prof/ir.profdata` profile using `llvm-profdata` before using the merged profile during the final optimized build. + + +```bash +llvm-profdata merge prof/ir.profdata prof/csir -output=prof/csir.profdata +``` + +You can inspect the merged profile to see block counts. In the example below, the function `sort_array` has 6 context-sensitive counters with several recorded hits. You can inspect all functions with `--all-functions`, but the output can be extensive for large applications. + +```bash { command_line="user@host | 2-13" } +llvm-profdata show --showcs --counts --function=sort_array prof/csir.profdata +Counters: + ld-temp.o;_Z10sort_arrayPi: + Hash: 0x18c2aba34f0cfff9 + Counters: 6 + Block counts: [24763682, 25224415, 9999, 9882, 1, 9881] +Instrumentation level: IR entry_first = 0 instrument_loop_entries = 0 +Functions shown: 1 +Total functions: 12 +Maximum function count: 24763682 +Maximum internal block count: 25224415 +Total number of blocks: 32 +Total count: 75242276 +``` + +## Build with CSIR-PGO and LTO + +Build the optimized binary using the merged CS-IR profile: + + +```bash +clang++ -O3 -flto -fuse-ld=lld -fprofile-use=prof/csir.profdata \ + bsort.cpp -o out/bsort.csirpgo.opt +``` + +Run the optimized binary: + +```bash { command_line="user@host | 2-3" } +./out/bsort.opt.csir +``` + +## What you've learned and what's next + +You've added a second context-sensitive profiling pass on top of IR-PGO and built a CSIR-PGO optimized binary with LTO. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/fe-pgo.md b/content/learning-paths/servers-and-cloud-computing/pgo/fe-pgo.md new file mode 100644 index 0000000000..65d20f14e5 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/fe-pgo.md @@ -0,0 +1,86 @@ +--- +title: Optimize with FE-PGO +weight: 6 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is FE-PGO? + +FE-PGO uses Clang frontend instrumentation to collect execution counts while the program runs. Clang adds the counters before lowering the program to LLVM IR, so the profile data maps closely to the source code. After the training run, Clang uses the collected profile during an optimized build. + +Use FE-PGO when source-level profile information is important. For performance-focused PGO, IR-PGO and CSIR-PGO are usually better defaults and are covered next. + +## Build the instrumented binary + +Build an instrumented binary. The `-fprofile-instr-generate` option tells Clang to add frontend counters and write the raw profile. + + +```bash +clang++ -O3 -flto -fuse-ld=lld \ + -fprofile-instr-generate=prof/fe.profraw \ + bsort.cpp -o out/bsort.fepgo.instr +``` + +Run the instrumented binary: + +```bash +./out/bsort.fepgo.instr +``` + +After the program exits, the raw profile should be written to: + +```bash +ls prof/fe.profraw +``` + + +## Convert and Inspect the profile data + +Before Clang can use the profile during an optimized build, convert it to the `.profdata` format. + +```bash +llvm-profdata merge prof/fe.profraw -output=prof/fe.profdata +``` +You can inspect the merged profile to see block counts. In the example below we see that the function `sort_array` had 2 counters with hits matching our problem size. You can observe every function using `--all-functions` but can generate a large output on big binaries. + +```bash { command_line="user@host | 2-14" } +llvm-profdata show --counts --function=sort_array prof/fe.profdata +Counters: + _Z10sort_arrayPi: + Hash: 0x00000000000046d1 + Counters: 2 + Function count: 1 + Block counts: [10000] +Instrumentation level: Front-end +Functions shown: 1 +Total functions: 16 +Maximum function count: 5044883 +Maximum internal block count: 49988097 +Total number of blocks: 39 +Total count: 100476579 +``` + +## Build with FE-PGO and LTO + +Build the optimized binary using the converted profile: + +```bash +clang++ -O3 -flto -fuse-ld=lld \ + -fprofile-instr-use=prof/fe.profdata \ + bsort.cpp -o out/bsort.fepgo.opt +``` + +Run the optimized binary: + +```bash { command_line="user@host | 2-3" } +./out/bsort.fepgo.opt +``` + +## What you've learned and what's next + + +You've collected an FE-PGO profile, converted it with `llvm-profdata`, and used it with Thin-LTO to build an optimized binary. + +Next, you'll build the same example with IR-PGO and Thin-LTO. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/ir-pgo.md b/content/learning-paths/servers-and-cloud-computing/pgo/ir-pgo.md new file mode 100644 index 0000000000..f0033c4950 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/ir-pgo.md @@ -0,0 +1,86 @@ +--- +title: Optimize with IR-PGO +weight: 7 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is IR-PGO? + +IR-PGO uses instrumentation at the LLVM IR level to collect execution counts by adding counters before LLVM optimizes the IR. + +Compared to FE-PGO, it usually has lower instrumentation overhead, generates smaller profiles, and is generally the better choice for optimization. + +## Build the instrumented binary + +Build an instrumented binary. The `-fprofile-generate` option tells LLVM to add IR-level counters and write raw profiles to the specified directory. + +```bash +clang++ -O3 -flto -fuse-ld=lld \ + -fprofile-generate=prof/ir \ + bsort.cpp -o out/bsort.irpgo.instr +``` + +Run the instrumented binary: + +```bash +./out/bsort.irpgo.instr +``` + +After the program exits, the raw profiles are written to: + +```bash +ls prof/ir/*.profraw +``` + +## Merge, Convert, and Inspect the profile data + +When profile filename patterns are used, Clang can write multiple raw `.profraw` files. This is useful when profiling multiple processes, collecting profiles on different hosts, running the program multiple times, or avoiding profile file overwrites in parallel runs. For more information, see [Clang profile filename patterns](https://clang.llvm.org/docs/UsersManual.html#profiling-with-instrumentation). + +Before Clang can use the profile during an optimized build, merge the raw profiles with `llvm-profdata`. This step also converts them to the `.profdata` format used during optimization. + + +```bash +llvm-profdata merge prof/ir -output=prof/ir.profdata +``` + +You can inspect the merged profile to see block counts. In the example below, the function `sort_array` has 6 counters with several recorded hits. You can inspect all functions with `--all-functions`, but the output can be extensive for large applications. + +```bash { command_line="user@host | 2-8" } +llvm-profdata show --counts --function=sort_array prof/ir.profdata +Counters: + _Z10sort_arrayPi: + Hash: 0x08380d8f3e6d4c88 + Counters: 6 + Block counts: [9882, 49988097, 25224415, 10000, 1, 9882] +Instrumentation level: IR entry_first = 0 instrument_loop_entries = 0 +Functions shown: 1 +Total functions: 13 +Maximum function count: 9882 +Maximum internal block count: 49988097 +Total number of blocks: 38 +Total count: 150915523 +``` + +## Build with IR-PGO and LTO + +Build the optimized binary using the merged profile: + +```bash +clang++ -O3 -flto -fuse-ld=lld -fprofile-use=prof/ir.profdata \ + bsort.cpp -o out/bsort.irpgo.opt +``` + +Run the optimized binary: + +```bash +./out/bsort.irpgo.opt +``` + + +## What you've learned and what's next + +You've collected an IR-PGO profile, merged it with `llvm-profdata`, and used it with Thin-LTO to build an optimized binary. + +Next, you'll extend the IR-PGO workflow with a context-sensitive profiling pass. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/lto.md b/content/learning-paths/servers-and-cloud-computing/pgo/lto.md new file mode 100644 index 0000000000..492e15b60a --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/lto.md @@ -0,0 +1,72 @@ +--- +title: Build with LTO +weight: 4 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is LTO? + +Link-Time Optimization (LTO) enables optimization across source file boundaries during the link stage. Without LTO, the compiler optimizes each source file independently before the linker combines the resulting object files. As a result, the optimizer cannot make optimization decisions based on the whole program. + +For more information, see the [LLVM Link Time Optimization](https://llvm.org/docs/LinkTimeOptimization.html) design documentation. + +## Full vs Thin-LTO + +LLVM supports two main LTO modes: Full-LTO and Thin-LTO. + +Both modes make Clang emit LLVM bitcode during compilation. The difference is how LLVM performs optimization during the link stage. + +Full-LTO merges all input bitcode into a single LLVM module. LLVM then optimizes the program as a single unit before generating native code. +This gives the optimizer a complete view of the program, but it can increase link time and memory usage. + +Thin-LTO keeps the build more scalable. Each compiled module includes a compact summary. During the link stage, LLVM combines these summaries into a global index, determines which functions to import across module boundaries, and then optimizes each module in parallel. +Thin-LTO also supports incremental builds by caching compilation results and rebuilding only the modules whose generated code changes. + +## Build with LTO + +LTO is disabled by default. Use `-flto=full` to enable Full-LTO or `-flto=thin` to enable Thin-LTO. +If you specify `-flto` without a value, Clang uses Full-LTO. The following commands use `-fuse-ld=lld` to select the LLVM linker. + +{{< tabpane code=true >}} + {{< tab header="Full-LTO" language="bash">}} +clang -O3 -flto=full -c bsort.cpp -o out/bsort.lto.full.o +clang -O3 -flto=full -fuse-ld=lld out/bsort.lto.full.o -o out/bsort.lto.full + {{< /tab >}} + {{< tab header="Thin-LTO" language="bash">}} +clang -O3 -flto=thin -c bsort.cpp -o out/bsort.lto.thin.o +clang -O3 -flto=thin -fuse-ld=lld out/bsort.lto.thin.o -o out/bsort.lto.thin + {{< /tab >}} +{{< /tabpane >}} + + +## Verify the LTO object files + +Use the bitcode analyzer (`llvm-bcanalyzer`) on an object file to verify that it contains LLVM bitcode and to identify the LTO mode. +For Thin-LTO, look for `GLOBALVAL_SUMMARY_BLOCK`. For Full-LTO, look for `FULL_LTO_GLOBALVAL_SUMMARY_BLOCK`. + +{{< tabpane code=true >}} + {{< tab header="Full-LTO" language="bash" output_lines="2-4">}} +llvm-bcanalyzer -dump bsort.lto.full.o | grep 'SUMMARY_BLOCK' + + + Block ID #24 (FULL_LTO_GLOBALVAL_SUMMARY_BLOCK): + {{< /tab >}} + {{< tab header="Thin-LTO" language="bash" output_lines="2-4">}} +llvm-bcanalyzer -dump bsort.lto.thin.o | grep 'SUMMARY_BLOCK' + + + Block ID #20 (GLOBALVAL_SUMMARY_BLOCK): + {{< /tab >}} + {{< tab header="No LTO" language="bash" output_lines="2">}} +llvm-bcanalyzer -dump bsort.nolto.o +llvm-bcanalyzer: Invalid record at top-level + {{< /tab >}} +{{< /tabpane >}} + +## What you've learned and what's next + +You've built the example with Full-LTO and Thin-LTO and verified that each build uses the expected LTO mode. + +Next, you'll collect sampled profile data and use it to build the application with S-PGO and Thin-LTO. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/overview.md b/content/learning-paths/servers-and-cloud-computing/pgo/overview.md new file mode 100644 index 0000000000..dde2eaaa7f --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/overview.md @@ -0,0 +1,36 @@ +--- +title: Understand PGO and LTO +weight: 2 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Overview + +This Learning Path demonstrates how to use LLVM Link-Time Optimization (LTO) and Profile-Guided Optimization (PGO) together on AArch64 Linux. + +LTO gives the compiler visibility of the entire program during the link stage. Instead of optimizing each source file independently, LLVM performs additional whole-program optimizations across source file boundaries. PGO uses runtime behavior to guide compiler decisions. When the compiler knows which functions and branches are frequently executed, it can make better decisions about function inlining, branch prediction, code layout, and other optimizations. + +LTO and PGO are complementary. LTO enables whole-program optimizations, while PGO provides runtime profile data that guides optimization decisions. + +This guide focuses on using LLVM LTO and PGO. +The example application and build configuration are intentionally simple to demonstrate the LLVM workflows and compiler options. Real-world applications are typically more complex and might require different build configurations and optimization strategies. + +The following sections cover LLVM LTO and several LLVM PGO workflows, each with different trade-offs: + +- [LLVM LTO](/learning-paths/servers-and-cloud-computing/pgo/lto/) gives the compiler visibility across source file boundaries at link time. This provides the baseline for the later profile-guided workflows. + +- [LLVM S-PGO](/learning-paths/servers-and-cloud-computing/pgo/s-pgo/) uses sampled execution data, usually collected with `perf`, instead of compiler-inserted instrumentation. It has much lower profiling overhead than instrumentation, but it requires hardware and operating system support. + +- [LLVM FE-PGO](/learning-paths/servers-and-cloud-computing/pgo/fe-pgo/) instruments the program in the Clang frontend to record execution counts. It can be useful when profile data needs to map closely to the source code, but IR-PGO is usually the better starting point for optimization. + +- [LLVM IR-PGO](/learning-paths/servers-and-cloud-computing/pgo/ir-pgo/) instruments the program at the LLVM IR level to record execution counts. This is the default instrumentation-based workflow in this guide and is typically the preferred option for performance optimization with Clang. + +- [LLVM CSIR-PGO](/learning-paths/servers-and-cloud-computing/pgo/csir-pgo/) adds a second, context-sensitive profiling pass after an initial IR-PGO build. This can give LLVM more precise profile data, but it requires an extra build and profiling run. + +## What you've learned and what's next + +You’ve now seen an overview of LLVM LTO, PGO, and the optimization workflows covered in this guide. + +In the next section, you'll prepare the test directory and verify the LLVM tools. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/s-pgo.md b/content/learning-paths/servers-and-cloud-computing/pgo/s-pgo.md new file mode 100644 index 0000000000..34d175cf9d --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/s-pgo.md @@ -0,0 +1,142 @@ +--- +title: Optimize with S-PGO (AFDO) +weight: 5 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is S-PGO? + +{{% notice Note %}} +S-PGO is also called Sample-PGO, AFDO, or AutoFDO in LLVM documentation and related tools. +{{% /notice %}} + +S-PGO is sample-based profile-guided optimization. +Instead of adding software counters to the program, you run an optimized binary with a profiler such as Linux `perf`, which records hardware events while the program executes. +`llvm-profgen` converts the raw sample data into an LLVM sample profile. Clang then uses that profile during an optimized build with `-fprofile-sample-use`. + +S-PGO requires hardware and operating system support for sampling. +It also uses debug information to map the collected samples back to functions and source locations. + +## When to use S-PGO + +Use S-PGO when you want lower profiling overhead than instrumentation-based PGO. A typical use case is collecting profile data in production environments, where instrumentation-based profiling can introduce too much overhead. + + +## Build a binary for sampling + + +Build the binary with optimization enabled and add the information needed to match the collected samples to the program. +This guide uses line-table debug information, profiling-specific debug information, unique internal names, and pseudo-probes. +For more information about these options, see the [Clang profile-guided optimization guide](https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization). + +To generate the binary for sampling, run: + +``` +clang++ -O3 -flto=thin -fuse-ld=lld \ + -gline-tables-only \ + -fdebug-info-for-profiling \ + -funique-internal-linkage-names \ + -fpseudo-probe-for-profiling \ + bsort.cpp -o out/bsort.spgo +``` + +If your build system changes source paths between the profiling build and the optimized build, omit `-funique-internal-linkage-names` from both builds. + +Check that the binary contains line-table information and pseudo-probe sections: + +```bash { command_line="user@host | 2-5" } +llvm-readelf --sections out/bsort.spgo | grep -E 'debug_line|pseudo_probe' + [28] .pseudo_probe_desc PROGBITS 0000000000000000 01cdce 000464 00 0 0 1 + [35] .debug_line PROGBITS 0000000000000000 01da48 00049d 00 0 0 1 + [36] .debug_line_str PROGBITS 0000000000000000 01dee5 000043 01 MS 0 0 1 + [37] .pseudo_probe PROGBITS 0000000000000000 01df28 000199 00 L 13 0 1 +``` + + +## Collect a sampling profile + +Collect a `perf` profile with branch stack data: + +```bash { command_line="user@host | 2-5" } +perf record -j any,u -o prof/brbe.data -- ./out/bsort.spgo +Bubble sorting 10000 elements +140 ms (first=100669 last=2147469841) +[ perf record: Woken up 2 times to write data ] +[ perf record: Captured and wrote 0.438 MB prof/brbe.data (566 samples) ] +``` + +## Convert the sampling profile + +Convert the raw `perf` data into an LLVM sample profile: + + +``` +llvm-profgen \ + --binary=out/bsort.spgo \ + --perfdata=prof/brbe.data \ + --output=prof/brbe.data.prof +``` + +Inspect the generated sample profile: + +```bash { command_line="user@host | 2-12" } +llvm-profdata show --sample --all-functions prof/brbe.data.prof | grep -E 'Function:|inlined callee:' +Function: main: CFG checksum 1688854155231231 + 4: inlined callee: _ZL11start_timerv.__uniq.184325335692493633500970462303439801414: CFG checksum 281479271677951 + 5: inlined callee: _Z10sort_arrayPi: CFG checksum 563057241526008 + 8: inlined callee: _Z11bubble_sortPii: CFG checksum 281822477581176 + 6: inlined callee: _ZL10stop_timerv.__uniq.184325335692493633500970462303439801414: CFG checksum 562954248388607 + 7: inlined callee: _ZL16print_first_lastPKii.__uniq.184325335692493633500970462303439801414: CFG checksum 281546317938031 +Function: _ZL5swap4PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +Function: _ZL5swap3PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +Function: _ZL5swap1PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +Function: _ZL5swap2PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +Function: _ZL5swap5PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +``` + +The output should show sample data for the workload. For large applications, the output can be extensive. + + +## Build with S-PGO and LTO + +Build the optimized binary using the sample profile. + +The `-fsample-profile-use-profi` option infers missing block and edge counts to improve profile quality. + + +```bash +clang++ -O3 -flto=thin -fuse-ld=lld \ + -fsample-profile-use-profi \ + -fdebug-info-for-profiling -funique-internal-linkage-names \ + -fpseudo-probe-for-profiling \ + -fprofile-sample-use=prof/brbe.data.prof \ + -Rpass=sample-profile-inline -fdiagnostics-show-hotness \ + bsort.cpp -o out/bsort.spgo.opt +``` + +Use `-Rpass=sample-profile-inline` and `-fdiagnostics-show-hotness` to emit optimization remarks and verify that Clang used the sample profile during optimization. + +Example output: + +``` +bsort.cpp:107:5: remark: '_ZL11start_timerv.__uniq.184325335692493633500970462303439801414' inlined into 'main' to match profiling context with (cost=-14990, threshold=45) + at callsite main:5:5; [-Rpass=sample-profile-inline] + 107 | start_timer(); + | ^ + +bsort.cpp:108:5: '_Z10sort_arrayPi' inlined into 'main' to match profiling context with (cost=-14945, threshold=45) at callsite main:6:5; (hotness: 1) +``` + +Finally, run the optimized binary: +``` +./out/bsort.spgo.opt +``` + + +## What you've learned and what's next + +You've collected a sampled profile, converted it with `llvm-profgen`, and used it with Thin-LTO to build an optimized binary. + +Next, you'll try FE-PGO with Thin-LTO. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/setup.md b/content/learning-paths/servers-and-cloud-computing/pgo/setup.md new file mode 100644 index 0000000000..7db7986263 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/setup.md @@ -0,0 +1,50 @@ +--- +title: Prepare your environment +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Set up your environment + +On your AArch64 Linux machine, navigate to your home directory or another empty working directory and download the `bsort.cpp` source file: + +```bash +wget https://learn.arm.com/learning-paths/servers-and-cloud-computing/bolt-demo/bsort.cpp +``` + +Create the following directories to organize generated files from this example: + +```bash +mkdir -p out prof +``` + +- **out**: Stores output binaries +- **prof**: Stores profile data + +If LLVM is not already installed, follow the [LLVM toolchain for Linux on Arm](/install-guides/llvm/) install guide before continuing. + + +## Verify tool availability + +Check that the LLVM tools are available: + +```bash { line_numbers=true } +clang++ --version +ld.lld --version +llvm-profdata --version +llvm-profgen --version +``` + +For S-PGO, also check that perf is available. The BRBE-based S-PGO workflow in this guide requires a perf binary from Linux kernel 6.17. + +```bash +perf --version +``` + +## What you've learned and what's next + +You've created the working directory, verified that the LLVM tools are available, and downloaded the example source. + +Next, you'll build the example with Thin-LTO and Full-LTO.