From 2896cf1ba0f524cf2f42dc9cc07bf8b4b13edde1 Mon Sep 17 00:00:00 2001 From: Paschalis Mpeis Date: Fri, 10 Jul 2026 15:44:49 +0100 Subject: [PATCH 1/2] Add PGO Learning Path --- content/install-guides/llvm.md | 157 ++++++++++++++++++ .../servers-and-cloud-computing/pgo/_index.md | 70 ++++++++ .../pgo/_next-steps.md | 8 + .../pgo/csir-pgo.md | 86 ++++++++++ .../servers-and-cloud-computing/pgo/fe-pgo.md | 86 ++++++++++ .../servers-and-cloud-computing/pgo/ir-pgo.md | 86 ++++++++++ .../servers-and-cloud-computing/pgo/lto.md | 72 ++++++++ .../pgo/overview.md | 36 ++++ .../servers-and-cloud-computing/pgo/s-pgo.md | 142 ++++++++++++++++ .../servers-and-cloud-computing/pgo/setup.md | 50 ++++++ 10 files changed, 793 insertions(+) create mode 100644 content/install-guides/llvm.md create mode 100644 content/learning-paths/servers-and-cloud-computing/pgo/_index.md create mode 100644 content/learning-paths/servers-and-cloud-computing/pgo/_next-steps.md create mode 100644 content/learning-paths/servers-and-cloud-computing/pgo/csir-pgo.md create mode 100644 content/learning-paths/servers-and-cloud-computing/pgo/fe-pgo.md create mode 100644 content/learning-paths/servers-and-cloud-computing/pgo/ir-pgo.md create mode 100644 content/learning-paths/servers-and-cloud-computing/pgo/lto.md create mode 100644 content/learning-paths/servers-and-cloud-computing/pgo/overview.md create mode 100644 content/learning-paths/servers-and-cloud-computing/pgo/s-pgo.md create mode 100644 content/learning-paths/servers-and-cloud-computing/pgo/setup.md diff --git a/content/install-guides/llvm.md b/content/install-guides/llvm.md new file mode 100644 index 0000000000..4f3f51530f --- /dev/null +++ b/content/install-guides/llvm.md @@ -0,0 +1,157 @@ +--- +title: LLVM toolchain for Linux on Arm + +additional_search_terms: +- llvm +- clang +- compiler +- lld +- pgo +- lto +- profdata +- profgen + +minutes_to_complete: 10 + +author: Paschalis Mpeis + +official_docs: https://releases.llvm.org/download.html +description: Install the LLVM toolchain on Arm Linux to use Clang, LLD, and profiling utilities. + +test_images: +- ubuntu:latest +test_maintenance: false + +layout: installtoolsall +tool_install: true +multi_install: false +multitool_install_part: false +weight: 1 +--- + +The LLVM toolchain includes Clang, LLD, and other LLVM utilities for compiling, linking, and optimizing applications. + +In this guide, you’ll learn how to download, extract, and install the LLVM toolchain from a prebuilt binary release on Arm Linux. + +## Before you begin + +Confirm you are using an Arm machine by running: + +Run: + +```bash +uname -m +``` + +The output is similar to: + +```output +aarch64 +``` + +If you see a different result, you are not using an Arm computer running 64-bit Linux. + +Install the tools needed to download and extract the archive. +For Debian-based Linux distributions, including Ubuntu, install `wget` and `xz-utils`: + + +```bash { target="ubuntu:latest" } +sudo apt update +sudo apt install wget xz-utils -y +``` + +## Install LLVM + +You can install LLVM by downloading a prebuilt binary release from GitHub. + +The following commands use LLVM version 22.1.8. To install a different version, replace the filename in the commands below. For the latest releases, see [LLVM Project releases](https://github.com/llvm/llvm-project/releases). + +The commands extract LLVM to `$HOME/toolchain`. To install LLVM in a different location, adjust the extraction path and update the PATH environment variable accordingly. + +1. Download a binary release + +For Arm Linux, use the archive with `Linux-ARM64` in its filename: + + +```bash +mkdir -p $HOME/toolchain +cd $HOME/toolchain +wget https://github.com/llvm/llvm-project/releases/download/llvmorg-22.1.8/LLVM-22.1.8-Linux-ARM64.tar.xz +``` + +2. Extract the downloaded file + +```bash +tar -xvf LLVM-22.1.8-Linux-ARM64.tar.xz +``` + +3. Add the LLVM `bin` directory to your `PATH` + +This enables you to run LLVM tools such as `clang` and `lld` from any directory. +The command updates `PATH` for your current terminal session. +To make the change persistent, add the same command to your shell profile, such as `~/.bashrc`. + +```bash +export PATH="$HOME/toolchain/LLVM-22.1.8-Linux-ARM64/bin:$PATH" +``` +## Verify LLVM is installed {#verify} + +Run the following commands: + +```console +clang --version +clang++ --version +ld.lld --version +``` + +The output is similar to: + +```output +clang version 22.1.8 (https://github.com/llvm/llvm-project ca7933e47d3a3451d81e72ac174dcb5aa28b59d1) +Target: aarch64-unknown-linux-gnu +Thread model: posix +InstalledDir: /home/ubuntu/toolchain/LLVM-22.1.8-Linux-ARM64/bin + +clang version 22.1.8 (https://github.com/llvm/llvm-project ca7933e47d3a3451d81e72ac174dcb5aa28b59d1) +Target: aarch64-unknown-linux-gnu +Thread model: posix +InstalledDir: /home/ubuntu/toolchain/LLVM-22.1.8-Linux-ARM64/bin + +LLD 22.1.8 (https://github.com/llvm/llvm-project ca7933e47d3a3451d81e72ac174dcb5aa28b59d1) (compatible with GNU linkers) +``` + +Each command should print LLVM version information. + +## Compile and run a C++ example + +Use a text editor to copy and paste the following code into a file named `hello.cpp`: + +```cpp { file_name="hello.cpp" } +#include + +int main() +{ + std::cout << "Hello, LLVM on Arm\n"; + return 0; +} +``` + +Compile the example with `clang++` and link it with LLD: + +```bash +clang++ -fuse-ld=lld hello.cpp -o hello +``` + +Run the example: + +```bash +./hello +``` + +The output is: + +```output +Hello, LLVM on Arm +``` + +You are now ready to use LLVM on your Arm Linux system. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/_index.md b/content/learning-paths/servers-and-cloud-computing/pgo/_index.md new file mode 100644 index 0000000000..93d2bc3063 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/_index.md @@ -0,0 +1,70 @@ +--- +title: "Optimize AArch64 code with LLVM LTO and PGO" +description: Learn how to use LLVM Link-Time Optimization and Profile-Guided Optimization on AArch64 Linux. +minutes_to_complete: 45 +aliases: + - /learning-paths/servers-and-cloud-computing/pgo-test/ + +who_is_this_for: This is an introductory topic for developers who compile C or C++ applications on AArch64 Linux and want to use Link-Time Optimization (LTO) with Profile-Guided Optimization (PGO). + + +learning_objectives: + - Understand the basics of Link-Time Optimization (LTO) and Profile-Guided Optimization (PGO) + - Build Thin-LTO and Full-LTO binaries with Clang on AArch64 + - Generate instrumentation-based profile data with Clang on AArch64 + - Generate sample-based profile data with Clang on AArch64 + - Use generated profile data to optimize a small example application + + +prerequisites: + - LLVM installed with Clang, LLD, `llvm-profdata`, and `llvm-profgen` available in your `PATH`. For setup instructions, see [LLVM toolchain for Linux on Arm](/install-guides/llvm/). + - Linux kernel version 6.17 or later for the Branch Record Buffer Extension (BRBE) profile-guided optimization workflow + + +author: Paschalis Mpeis + +### Tags +skilllevels: Introductory +subjects: Performance and Architecture +armips: + - Neoverse + - Cortex-A +tools_software_languages: + - Clang + - LLVM + - LTO + - PGO + - perf + +operatingsystems: + - Linux + +further_reading: + - resource: + title: Clang profile-guided optimization + link: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization + type: documentation + - resource: + title: llvm-profdata command guide + link: https://llvm.org/docs/CommandGuide/llvm-profdata.html + type: documentation + - resource: + title: llvm-profgen command guide + link: https://llvm.org/docs/CommandGuide/llvm-profgen.html + type: documentation + - resource: + title: LLVM LTO + link: https://llvm.org/docs/LinkTimeOptimization.html + type: documentation + - resource: + title: LLVM Thin-LTO + link: https://clang.llvm.org/docs/Thin-LTO.html + type: documentation + + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/pgo/_next-steps.md new file mode 100644 index 0000000000..c3db0de5a2 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/csir-pgo.md b/content/learning-paths/servers-and-cloud-computing/pgo/csir-pgo.md new file mode 100644 index 0000000000..83196f7eda --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/csir-pgo.md @@ -0,0 +1,86 @@ +--- +title: Optimize with CSIR-PGO +weight: 8 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is CSIR-PGO? + +CSIR-PGO extends IR-PGO by adding a second, context-sensitive profiling pass. +The first pass is the standard IR-PGO instrumentation pass. The second pass instruments the program after inlining, which enables LLVM to distinguish execution counts from different calling contexts. +The additional context can improve optimization when function behavior depends on the call site, but it does not guarantee better performance for every program. + +Use CSIR-PGO when you want to provide LLVM with more detailed profile information and can afford an extra build and training run. + +## Build the context-sensitive instrumented binary + +First, generate a standard IR-PGO profile as described in the [previous section](/learning-paths/servers-and-cloud-computing/pgo/ir-pgo/). +We reuse the resulting `prof/ir.profdata` profile to build a second instrumented binary with `-fcs-profile-generate`, which adds context-sensitive instrumentation after inlining. + +```bash +clang++ -O3 -flto -fuse-ld=lld \ + -fprofile-use=prof/ir.profdata \ + -fcs-profile-generate=prof/csir \ + bsort.cpp -o out/bsort.csirpgo.instr +``` + +Run the context-sensitive instrumented binary: + +```bash +./out/bsort.csirpgo.instr +``` + +After the program exits, the raw profiles are written to: + +```bash +ls prof/csir/*.profraw +``` + +## Merge, Convert, and Inspect the profile + +As with IR-PGO, the context-sensitive training run can produce one or more raw `.profraw` files. Merge those files together with the existing `prof/ir.profdata` profile using `llvm-profdata` before using the merged profile during the final optimized build. + + +```bash +llvm-profdata merge prof/ir.profdata prof/csir -output=prof/csir.profdata +``` + +You can inspect the merged profile to see block counts. In the example below, the function `sort_array` has 6 context-sensitive counters with several recorded hits. You can inspect all functions with `--all-functions`, but the output can be extensive for large applications. + +```bash { command_line="user@host | 2-13" } +llvm-profdata show --showcs --counts --function=sort_array prof/csir.profdata +Counters: + ld-temp.o;_Z10sort_arrayPi: + Hash: 0x18c2aba34f0cfff9 + Counters: 6 + Block counts: [24763682, 25224415, 9999, 9882, 1, 9881] +Instrumentation level: IR entry_first = 0 instrument_loop_entries = 0 +Functions shown: 1 +Total functions: 12 +Maximum function count: 24763682 +Maximum internal block count: 25224415 +Total number of blocks: 32 +Total count: 75242276 +``` + +## Build with CSIR-PGO and LTO + +Build the optimized binary using the merged CS-IR profile: + + +```bash +clang++ -O3 -flto -fuse-ld=lld -fprofile-use=prof/csir.profdata \ + bsort.cpp -o out/bsort.csirpgo.opt +``` + +Run the optimized binary: + +```bash { command_line="user@host | 2-3" } +./out/bsort.opt.csir +``` + +## What you've learned and what's next + +You've added a second context-sensitive profiling pass on top of IR-PGO and built a CSIR-PGO optimized binary with LTO. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/fe-pgo.md b/content/learning-paths/servers-and-cloud-computing/pgo/fe-pgo.md new file mode 100644 index 0000000000..65d20f14e5 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/fe-pgo.md @@ -0,0 +1,86 @@ +--- +title: Optimize with FE-PGO +weight: 6 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is FE-PGO? + +FE-PGO uses Clang frontend instrumentation to collect execution counts while the program runs. Clang adds the counters before lowering the program to LLVM IR, so the profile data maps closely to the source code. After the training run, Clang uses the collected profile during an optimized build. + +Use FE-PGO when source-level profile information is important. For performance-focused PGO, IR-PGO and CSIR-PGO are usually better defaults and are covered next. + +## Build the instrumented binary + +Build an instrumented binary. The `-fprofile-instr-generate` option tells Clang to add frontend counters and write the raw profile. + + +```bash +clang++ -O3 -flto -fuse-ld=lld \ + -fprofile-instr-generate=prof/fe.profraw \ + bsort.cpp -o out/bsort.fepgo.instr +``` + +Run the instrumented binary: + +```bash +./out/bsort.fepgo.instr +``` + +After the program exits, the raw profile should be written to: + +```bash +ls prof/fe.profraw +``` + + +## Convert and Inspect the profile data + +Before Clang can use the profile during an optimized build, convert it to the `.profdata` format. + +```bash +llvm-profdata merge prof/fe.profraw -output=prof/fe.profdata +``` +You can inspect the merged profile to see block counts. In the example below we see that the function `sort_array` had 2 counters with hits matching our problem size. You can observe every function using `--all-functions` but can generate a large output on big binaries. + +```bash { command_line="user@host | 2-14" } +llvm-profdata show --counts --function=sort_array prof/fe.profdata +Counters: + _Z10sort_arrayPi: + Hash: 0x00000000000046d1 + Counters: 2 + Function count: 1 + Block counts: [10000] +Instrumentation level: Front-end +Functions shown: 1 +Total functions: 16 +Maximum function count: 5044883 +Maximum internal block count: 49988097 +Total number of blocks: 39 +Total count: 100476579 +``` + +## Build with FE-PGO and LTO + +Build the optimized binary using the converted profile: + +```bash +clang++ -O3 -flto -fuse-ld=lld \ + -fprofile-instr-use=prof/fe.profdata \ + bsort.cpp -o out/bsort.fepgo.opt +``` + +Run the optimized binary: + +```bash { command_line="user@host | 2-3" } +./out/bsort.fepgo.opt +``` + +## What you've learned and what's next + + +You've collected an FE-PGO profile, converted it with `llvm-profdata`, and used it with Thin-LTO to build an optimized binary. + +Next, you'll build the same example with IR-PGO and Thin-LTO. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/ir-pgo.md b/content/learning-paths/servers-and-cloud-computing/pgo/ir-pgo.md new file mode 100644 index 0000000000..f0033c4950 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/ir-pgo.md @@ -0,0 +1,86 @@ +--- +title: Optimize with IR-PGO +weight: 7 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is IR-PGO? + +IR-PGO uses instrumentation at the LLVM IR level to collect execution counts by adding counters before LLVM optimizes the IR. + +Compared to FE-PGO, it usually has lower instrumentation overhead, generates smaller profiles, and is generally the better choice for optimization. + +## Build the instrumented binary + +Build an instrumented binary. The `-fprofile-generate` option tells LLVM to add IR-level counters and write raw profiles to the specified directory. + +```bash +clang++ -O3 -flto -fuse-ld=lld \ + -fprofile-generate=prof/ir \ + bsort.cpp -o out/bsort.irpgo.instr +``` + +Run the instrumented binary: + +```bash +./out/bsort.irpgo.instr +``` + +After the program exits, the raw profiles are written to: + +```bash +ls prof/ir/*.profraw +``` + +## Merge, Convert, and Inspect the profile data + +When profile filename patterns are used, Clang can write multiple raw `.profraw` files. This is useful when profiling multiple processes, collecting profiles on different hosts, running the program multiple times, or avoiding profile file overwrites in parallel runs. For more information, see [Clang profile filename patterns](https://clang.llvm.org/docs/UsersManual.html#profiling-with-instrumentation). + +Before Clang can use the profile during an optimized build, merge the raw profiles with `llvm-profdata`. This step also converts them to the `.profdata` format used during optimization. + + +```bash +llvm-profdata merge prof/ir -output=prof/ir.profdata +``` + +You can inspect the merged profile to see block counts. In the example below, the function `sort_array` has 6 counters with several recorded hits. You can inspect all functions with `--all-functions`, but the output can be extensive for large applications. + +```bash { command_line="user@host | 2-8" } +llvm-profdata show --counts --function=sort_array prof/ir.profdata +Counters: + _Z10sort_arrayPi: + Hash: 0x08380d8f3e6d4c88 + Counters: 6 + Block counts: [9882, 49988097, 25224415, 10000, 1, 9882] +Instrumentation level: IR entry_first = 0 instrument_loop_entries = 0 +Functions shown: 1 +Total functions: 13 +Maximum function count: 9882 +Maximum internal block count: 49988097 +Total number of blocks: 38 +Total count: 150915523 +``` + +## Build with IR-PGO and LTO + +Build the optimized binary using the merged profile: + +```bash +clang++ -O3 -flto -fuse-ld=lld -fprofile-use=prof/ir.profdata \ + bsort.cpp -o out/bsort.irpgo.opt +``` + +Run the optimized binary: + +```bash +./out/bsort.irpgo.opt +``` + + +## What you've learned and what's next + +You've collected an IR-PGO profile, merged it with `llvm-profdata`, and used it with Thin-LTO to build an optimized binary. + +Next, you'll extend the IR-PGO workflow with a context-sensitive profiling pass. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/lto.md b/content/learning-paths/servers-and-cloud-computing/pgo/lto.md new file mode 100644 index 0000000000..492e15b60a --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/lto.md @@ -0,0 +1,72 @@ +--- +title: Build with LTO +weight: 4 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is LTO? + +Link-Time Optimization (LTO) enables optimization across source file boundaries during the link stage. Without LTO, the compiler optimizes each source file independently before the linker combines the resulting object files. As a result, the optimizer cannot make optimization decisions based on the whole program. + +For more information, see the [LLVM Link Time Optimization](https://llvm.org/docs/LinkTimeOptimization.html) design documentation. + +## Full vs Thin-LTO + +LLVM supports two main LTO modes: Full-LTO and Thin-LTO. + +Both modes make Clang emit LLVM bitcode during compilation. The difference is how LLVM performs optimization during the link stage. + +Full-LTO merges all input bitcode into a single LLVM module. LLVM then optimizes the program as a single unit before generating native code. +This gives the optimizer a complete view of the program, but it can increase link time and memory usage. + +Thin-LTO keeps the build more scalable. Each compiled module includes a compact summary. During the link stage, LLVM combines these summaries into a global index, determines which functions to import across module boundaries, and then optimizes each module in parallel. +Thin-LTO also supports incremental builds by caching compilation results and rebuilding only the modules whose generated code changes. + +## Build with LTO + +LTO is disabled by default. Use `-flto=full` to enable Full-LTO or `-flto=thin` to enable Thin-LTO. +If you specify `-flto` without a value, Clang uses Full-LTO. The following commands use `-fuse-ld=lld` to select the LLVM linker. + +{{< tabpane code=true >}} + {{< tab header="Full-LTO" language="bash">}} +clang -O3 -flto=full -c bsort.cpp -o out/bsort.lto.full.o +clang -O3 -flto=full -fuse-ld=lld out/bsort.lto.full.o -o out/bsort.lto.full + {{< /tab >}} + {{< tab header="Thin-LTO" language="bash">}} +clang -O3 -flto=thin -c bsort.cpp -o out/bsort.lto.thin.o +clang -O3 -flto=thin -fuse-ld=lld out/bsort.lto.thin.o -o out/bsort.lto.thin + {{< /tab >}} +{{< /tabpane >}} + + +## Verify the LTO object files + +Use the bitcode analyzer (`llvm-bcanalyzer`) on an object file to verify that it contains LLVM bitcode and to identify the LTO mode. +For Thin-LTO, look for `GLOBALVAL_SUMMARY_BLOCK`. For Full-LTO, look for `FULL_LTO_GLOBALVAL_SUMMARY_BLOCK`. + +{{< tabpane code=true >}} + {{< tab header="Full-LTO" language="bash" output_lines="2-4">}} +llvm-bcanalyzer -dump bsort.lto.full.o | grep 'SUMMARY_BLOCK' + + + Block ID #24 (FULL_LTO_GLOBALVAL_SUMMARY_BLOCK): + {{< /tab >}} + {{< tab header="Thin-LTO" language="bash" output_lines="2-4">}} +llvm-bcanalyzer -dump bsort.lto.thin.o | grep 'SUMMARY_BLOCK' + + + Block ID #20 (GLOBALVAL_SUMMARY_BLOCK): + {{< /tab >}} + {{< tab header="No LTO" language="bash" output_lines="2">}} +llvm-bcanalyzer -dump bsort.nolto.o +llvm-bcanalyzer: Invalid record at top-level + {{< /tab >}} +{{< /tabpane >}} + +## What you've learned and what's next + +You've built the example with Full-LTO and Thin-LTO and verified that each build uses the expected LTO mode. + +Next, you'll collect sampled profile data and use it to build the application with S-PGO and Thin-LTO. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/overview.md b/content/learning-paths/servers-and-cloud-computing/pgo/overview.md new file mode 100644 index 0000000000..dde2eaaa7f --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/overview.md @@ -0,0 +1,36 @@ +--- +title: Understand PGO and LTO +weight: 2 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Overview + +This Learning Path demonstrates how to use LLVM Link-Time Optimization (LTO) and Profile-Guided Optimization (PGO) together on AArch64 Linux. + +LTO gives the compiler visibility of the entire program during the link stage. Instead of optimizing each source file independently, LLVM performs additional whole-program optimizations across source file boundaries. PGO uses runtime behavior to guide compiler decisions. When the compiler knows which functions and branches are frequently executed, it can make better decisions about function inlining, branch prediction, code layout, and other optimizations. + +LTO and PGO are complementary. LTO enables whole-program optimizations, while PGO provides runtime profile data that guides optimization decisions. + +This guide focuses on using LLVM LTO and PGO. +The example application and build configuration are intentionally simple to demonstrate the LLVM workflows and compiler options. Real-world applications are typically more complex and might require different build configurations and optimization strategies. + +The following sections cover LLVM LTO and several LLVM PGO workflows, each with different trade-offs: + +- [LLVM LTO](/learning-paths/servers-and-cloud-computing/pgo/lto/) gives the compiler visibility across source file boundaries at link time. This provides the baseline for the later profile-guided workflows. + +- [LLVM S-PGO](/learning-paths/servers-and-cloud-computing/pgo/s-pgo/) uses sampled execution data, usually collected with `perf`, instead of compiler-inserted instrumentation. It has much lower profiling overhead than instrumentation, but it requires hardware and operating system support. + +- [LLVM FE-PGO](/learning-paths/servers-and-cloud-computing/pgo/fe-pgo/) instruments the program in the Clang frontend to record execution counts. It can be useful when profile data needs to map closely to the source code, but IR-PGO is usually the better starting point for optimization. + +- [LLVM IR-PGO](/learning-paths/servers-and-cloud-computing/pgo/ir-pgo/) instruments the program at the LLVM IR level to record execution counts. This is the default instrumentation-based workflow in this guide and is typically the preferred option for performance optimization with Clang. + +- [LLVM CSIR-PGO](/learning-paths/servers-and-cloud-computing/pgo/csir-pgo/) adds a second, context-sensitive profiling pass after an initial IR-PGO build. This can give LLVM more precise profile data, but it requires an extra build and profiling run. + +## What you've learned and what's next + +You’ve now seen an overview of LLVM LTO, PGO, and the optimization workflows covered in this guide. + +In the next section, you'll prepare the test directory and verify the LLVM tools. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/s-pgo.md b/content/learning-paths/servers-and-cloud-computing/pgo/s-pgo.md new file mode 100644 index 0000000000..34d175cf9d --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/s-pgo.md @@ -0,0 +1,142 @@ +--- +title: Optimize with S-PGO (AFDO) +weight: 5 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is S-PGO? + +{{% notice Note %}} +S-PGO is also called Sample-PGO, AFDO, or AutoFDO in LLVM documentation and related tools. +{{% /notice %}} + +S-PGO is sample-based profile-guided optimization. +Instead of adding software counters to the program, you run an optimized binary with a profiler such as Linux `perf`, which records hardware events while the program executes. +`llvm-profgen` converts the raw sample data into an LLVM sample profile. Clang then uses that profile during an optimized build with `-fprofile-sample-use`. + +S-PGO requires hardware and operating system support for sampling. +It also uses debug information to map the collected samples back to functions and source locations. + +## When to use S-PGO + +Use S-PGO when you want lower profiling overhead than instrumentation-based PGO. A typical use case is collecting profile data in production environments, where instrumentation-based profiling can introduce too much overhead. + + +## Build a binary for sampling + + +Build the binary with optimization enabled and add the information needed to match the collected samples to the program. +This guide uses line-table debug information, profiling-specific debug information, unique internal names, and pseudo-probes. +For more information about these options, see the [Clang profile-guided optimization guide](https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization). + +To generate the binary for sampling, run: + +``` +clang++ -O3 -flto=thin -fuse-ld=lld \ + -gline-tables-only \ + -fdebug-info-for-profiling \ + -funique-internal-linkage-names \ + -fpseudo-probe-for-profiling \ + bsort.cpp -o out/bsort.spgo +``` + +If your build system changes source paths between the profiling build and the optimized build, omit `-funique-internal-linkage-names` from both builds. + +Check that the binary contains line-table information and pseudo-probe sections: + +```bash { command_line="user@host | 2-5" } +llvm-readelf --sections out/bsort.spgo | grep -E 'debug_line|pseudo_probe' + [28] .pseudo_probe_desc PROGBITS 0000000000000000 01cdce 000464 00 0 0 1 + [35] .debug_line PROGBITS 0000000000000000 01da48 00049d 00 0 0 1 + [36] .debug_line_str PROGBITS 0000000000000000 01dee5 000043 01 MS 0 0 1 + [37] .pseudo_probe PROGBITS 0000000000000000 01df28 000199 00 L 13 0 1 +``` + + +## Collect a sampling profile + +Collect a `perf` profile with branch stack data: + +```bash { command_line="user@host | 2-5" } +perf record -j any,u -o prof/brbe.data -- ./out/bsort.spgo +Bubble sorting 10000 elements +140 ms (first=100669 last=2147469841) +[ perf record: Woken up 2 times to write data ] +[ perf record: Captured and wrote 0.438 MB prof/brbe.data (566 samples) ] +``` + +## Convert the sampling profile + +Convert the raw `perf` data into an LLVM sample profile: + + +``` +llvm-profgen \ + --binary=out/bsort.spgo \ + --perfdata=prof/brbe.data \ + --output=prof/brbe.data.prof +``` + +Inspect the generated sample profile: + +```bash { command_line="user@host | 2-12" } +llvm-profdata show --sample --all-functions prof/brbe.data.prof | grep -E 'Function:|inlined callee:' +Function: main: CFG checksum 1688854155231231 + 4: inlined callee: _ZL11start_timerv.__uniq.184325335692493633500970462303439801414: CFG checksum 281479271677951 + 5: inlined callee: _Z10sort_arrayPi: CFG checksum 563057241526008 + 8: inlined callee: _Z11bubble_sortPii: CFG checksum 281822477581176 + 6: inlined callee: _ZL10stop_timerv.__uniq.184325335692493633500970462303439801414: CFG checksum 562954248388607 + 7: inlined callee: _ZL16print_first_lastPKii.__uniq.184325335692493633500970462303439801414: CFG checksum 281546317938031 +Function: _ZL5swap4PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +Function: _ZL5swap3PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +Function: _ZL5swap1PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +Function: _ZL5swap2PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +Function: _ZL5swap5PiS_.__uniq.184325335692493633500970462303439801414: CFG checksum 844617033839767 +``` + +The output should show sample data for the workload. For large applications, the output can be extensive. + + +## Build with S-PGO and LTO + +Build the optimized binary using the sample profile. + +The `-fsample-profile-use-profi` option infers missing block and edge counts to improve profile quality. + + +```bash +clang++ -O3 -flto=thin -fuse-ld=lld \ + -fsample-profile-use-profi \ + -fdebug-info-for-profiling -funique-internal-linkage-names \ + -fpseudo-probe-for-profiling \ + -fprofile-sample-use=prof/brbe.data.prof \ + -Rpass=sample-profile-inline -fdiagnostics-show-hotness \ + bsort.cpp -o out/bsort.spgo.opt +``` + +Use `-Rpass=sample-profile-inline` and `-fdiagnostics-show-hotness` to emit optimization remarks and verify that Clang used the sample profile during optimization. + +Example output: + +``` +bsort.cpp:107:5: remark: '_ZL11start_timerv.__uniq.184325335692493633500970462303439801414' inlined into 'main' to match profiling context with (cost=-14990, threshold=45) + at callsite main:5:5; [-Rpass=sample-profile-inline] + 107 | start_timer(); + | ^ + +bsort.cpp:108:5: '_Z10sort_arrayPi' inlined into 'main' to match profiling context with (cost=-14945, threshold=45) at callsite main:6:5; (hotness: 1) +``` + +Finally, run the optimized binary: +``` +./out/bsort.spgo.opt +``` + + +## What you've learned and what's next + +You've collected a sampled profile, converted it with `llvm-profgen`, and used it with Thin-LTO to build an optimized binary. + +Next, you'll try FE-PGO with Thin-LTO. diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/setup.md b/content/learning-paths/servers-and-cloud-computing/pgo/setup.md new file mode 100644 index 0000000000..7db7986263 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/pgo/setup.md @@ -0,0 +1,50 @@ +--- +title: Prepare your environment +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Set up your environment + +On your AArch64 Linux machine, navigate to your home directory or another empty working directory and download the `bsort.cpp` source file: + +```bash +wget https://learn.arm.com/learning-paths/servers-and-cloud-computing/bolt-demo/bsort.cpp +``` + +Create the following directories to organize generated files from this example: + +```bash +mkdir -p out prof +``` + +- **out**: Stores output binaries +- **prof**: Stores profile data + +If LLVM is not already installed, follow the [LLVM toolchain for Linux on Arm](/install-guides/llvm/) install guide before continuing. + + +## Verify tool availability + +Check that the LLVM tools are available: + +```bash { line_numbers=true } +clang++ --version +ld.lld --version +llvm-profdata --version +llvm-profgen --version +``` + +For S-PGO, also check that perf is available. The BRBE-based S-PGO workflow in this guide requires a perf binary from Linux kernel 6.17. + +```bash +perf --version +``` + +## What you've learned and what's next + +You've created the working directory, verified that the LLVM tools are available, and downloaded the example source. + +Next, you'll build the example with Thin-LTO and Full-LTO. From 964b4280a092785a7029d6297344137adc8dddda Mon Sep 17 00:00:00 2001 From: pareenaverma Date: Mon, 27 Jul 2026 14:59:19 -0400 Subject: [PATCH 2/2] Update draft status and aliases in _index.md Added draft status and updated aliases in the learning path. --- .../servers-and-cloud-computing/pgo/_index.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/pgo/_index.md b/content/learning-paths/servers-and-cloud-computing/pgo/_index.md index 93d2bc3063..638bc7b9c0 100644 --- a/content/learning-paths/servers-and-cloud-computing/pgo/_index.md +++ b/content/learning-paths/servers-and-cloud-computing/pgo/_index.md @@ -1,9 +1,12 @@ --- title: "Optimize AArch64 code with LLVM LTO and PGO" + +draft: true +cascade: + draft: true + description: Learn how to use LLVM Link-Time Optimization and Profile-Guided Optimization on AArch64 Linux. minutes_to_complete: 45 -aliases: - - /learning-paths/servers-and-cloud-computing/pgo-test/ who_is_this_for: This is an introductory topic for developers who compile C or C++ applications on AArch64 Linux and want to use Link-Time Optimization (LTO) with Profile-Guided Optimization (PGO).