Skip to content

Commit 516a33d

Browse files
Updates
1 parent 3eb8449 commit 516a33d

5 files changed

Lines changed: 61 additions & 36 deletions

File tree

content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/_index.md

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,6 @@
11
---
22
title: Analyze cache behavior with Perf C2C on Arm
33

4-
5-
6-
74
minutes_to_complete: 15
85

96
who_is_this_for: This topic is for performance-oriented developers working on Arm-based cloud or server systems who want to optimize memory access patterns and investigate cache inefficiencies using Perf C2C and Arm SPE.

content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/how-to-1.md

Lines changed: 21 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,38 @@
11
---
2-
title: Introduction to Arm SPE and false sharing
2+
title: Arm Statistical Profiling Extension and false sharing
33
weight: 2
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
8-
## Introduction to the Arm Statistical Profiling Extension (SPE)
98

10-
Traditional Linux performance profiling tools often rely on retired instruction counts, which lack visibility into memory addresses, cache latencies, and micro-operation behavior. Moreover, the “skid” phenomenon where events are falsely attributed to later instructions can mislead developers.
9+
## What is the Arm Statistical Profiling Extension (SPE), and what does it do?
1110

12-
SPE integrates sampling directly into the CPU pipeline, triggering on individual micro-operations rather than retired instructions, thereby eliminating skid and blind spots. Each SPE sample record includes relevant metadata, such as data addresses, per-µop pipeline latency, triggered PMU event masks, and the memory hierarchy source, enabling fine-grained and precise cache analysis.
11+
{{% notice Learning goal%}}
12+
In this section, you will learn how the Arm Statistical Profiling Extension (SPE) gives you deeper visibility into how your applications interact with the CPU. It covers how to detect and fix false sharing, which is a hidden performance problem in multithreaded code, by using cache line alignment in C++ and tools like Perf C2C.
13+
{{% /notice %}}
14+
15+
Arm’s Statistical Profiling Extension (SPE) gives you a powerful way to understand what’s really happening inside your applications at the microarchitecture level. Introduced in Armv8.2, SPE captures a statistical view of how instructions move through the CPU, which allows you to dig into issues like memory access latency, cache misses, and pipeline behavior.
16+
17+
Most Linux profiling tools focus on retired instruction counts, which means they miss key details like memory addresses, cache latency, and micro-operation behavior. This can lead to misleading results — especially due to a phenomenon called “skid,” where events are falsely attributed to later instructions.
18+
19+
SPE integrates sampling directly into the CPU pipeline, triggering on individual micro-operations instead of retired instructions. This approach eliminates skid and blind spots. Each SPE sample record includes relevant metadata, such as data addresses, per-µop pipeline latency, triggered PMU event masks, and the memory hierarchy source, enabling fine-grained, precise cache analysis.
1320

1421
SPE helps developers optimize user-space applications by showing where cache latency or memory access delays are happening. Importantly, cache statistics are enabled with the Linux Perf cache-to-cache (C2C) utility.
1522

16-
To learn more, check out the [Arm SPE white paper](https://developer.arm.com/documentation/109429/latest/) for more information.
23+
For more information, see the [*Arm Statistical Profiling Extension: Performance Analysis Methodology White Paper*](https://developer.arm.com/documentation/109429/latest/).
1724

1825
In this Learning Path, you will use SPE and Perf C2C to diagnose a cache issue for an application running on a Neoverse server.
1926

2027
## What is false sharing and why should I care about it?
2128

2229
In large-scale, multithreaded applications, false sharing can degrade performance by introducing hundreds of unnecessary cache line invalidations per second - often with no visible red flags in the source code.
2330

24-
Even when two threads touch entirely separate variables, modern processors move data in fixed-size cache lines (nominally 64-bytes). If those distinct variables happen to occupy bytes within the same line, every time one thread writes its variable the core’s cache must gain exclusive ownership of the whole line, forcing the other core’s copy to be invalidated.
31+
Even when two threads touch entirely separate variables, modern processors move data in fixed-size cache lines, which is typically 64 bytes. If those distinct variables happen to occupy bytes within the same line, every time one thread writes its variable the core’s cache must gain exclusive ownership of the whole line, forcing the other core’s copy to be invalidated.
2532

26-
The second thread, still working on its own variable, then triggers a coherence miss to fetch the line back, and the ping-pong pattern repeats. See the illustration below, taken from the Arm SPE white paper, for a visual explanation.
33+
The second thread, still working on its own variable, then triggers a coherence miss to fetch the line back, and the ping-pong pattern repeats.
34+
35+
The diagram below, taken from the Arm SPE white paper, provides a visual representation of two threads on separate cores alternately gaining exclusive access to the same cache line.
2736

2837
![false_sharing_diagram alt-text#center](./false_sharing_diagram.png "Two threads on separate cores alternately gain exclusive access to the same cache line.")
2938

@@ -77,9 +86,9 @@ int main() {
7786
}
7887
```
7988

80-
The example output below shows the variables e, f, g and h occur at least 64 bytes apart in the byte-addressable architecture. Whereas variables a, b, c, and d occur 8 bytes apart, occupying the same cache line.
89+
The output below shows that the variables e, f, g and h occur at least 64 bytes apart in the byte-addressable architecture. Whereas variables a, b, c, and d occur 8 bytes apart, occupying the same cache line.
8190

82-
Although this is a contrived example, in a production workload there might be several layers of indirection that unintentionally result in false sharing. For these complex cases, to understand the root cause you will use Perf C2C.
91+
Although this is a contrived example, in a production workload there might be several layers of indirection that unintentionally result in false sharing. For these complex cases, to understand the root cause, you can use Perf C2C.
8392

8493
```output
8594
Without Alignment can occupy same cache line
@@ -101,7 +110,8 @@ Address of AlignedType h - 0xffffeb6c6080
101110

102111
## Summary
103112

104-
In this section, you learned what Arm SPE is and why it’s a game-changer for performance profiling. You also saw how a subtle issue like false sharing can slow down multithreaded code — and how to avoid it using alignment in C++.
113+
In this section, you explored what Arm SPE is and why it offers a deeper, more accurate view of application performance. You also examined how a subtle issue like false sharing can impact multithreaded code, and how to mitigate it using data alignment techniques in C++.
114+
115+
Next, you'll set up your environment and use Perf C2C to capture and analyze real cache behavior on an Arm Neoverse system.
105116

106-
Next up: you’ll set up your environment and get ready to run Perf C2C on a real application.
107117

content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/how-to-2.md

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,23 @@
11
---
2-
title: Configure your environment for Arm SPE profiling
2+
title: Environment setup for Arm SPE profiling
33
weight: 3
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
8-
98
## Select a system with SPE support
109

11-
SPE requires both hardware and operating system support. Many cloud instances running Linux do not enable SPE-based profiling.
10+
{{% notice Learning goal%}}
11+
Before you can start profiling cache behavior with Arm SPE and Perf C2C, your system needs to meet a few requirements. In this section, you’ll learn how to check whether your hardware and kernel support Arm SPE, install the necessary tools, and validate that Linux Perf can access the right performance monitoring events. By the end, your environment will be ready to record and analyze memory access patterns using Perf C2C on an Arm Neoverse system.
12+
{{% /notice %}}
13+
14+
SPE requires support from both your hardware and the operating system. Many cloud instances running Linux do not enable SPE-based profiling.
1215

1316
You need to identify a system that supports SPE using the information below.
1417

1518
If you are looking for an AWS system, you can use a `c6g.metal` instance running Amazon Linux 2023 (AL2023).
1619

17-
Check the underlying Neoverse processor and operating system kernel version with the following commands.
20+
Check the underlying Neoverse processor and operating system kernel version with the following commands:
1821

1922
```bash
2023
lscpu | grep -i "model name"
@@ -43,15 +46,15 @@ Run the following command to confirm if the SPE kernel module is loaded:
4346
sudo modprobe arm_spe_pmu
4447
```
4548

46-
If the module is not loaded (blank output), SPE might still be available.
49+
If the module is not loaded (and there is blank output), SPE might still be available.
4750

4851
Run this command to check if SPE is included in the kernel:
4952

5053
```bash
5154
ls /sys/bus/event_source/devices/ | grep arm_spe
5255
```
5356

54-
If SPE is available, the output is:
57+
If SPE is available, the output you will see is:
5558

5659
```output
5760
arm_spe_0
@@ -63,7 +66,7 @@ If the output is blank then SPE is not available.
6366

6467
You can install and run a Python script named Sysreport to summarize your system's performance profiling capabilities.
6568

66-
See [Get ready for performance analysis with Sysreport](https://learn.arm.com/learning-paths/servers-and-cloud-computing/sysreport/) to learn how to install and run it.
69+
See the Learning Path [Get ready for performance analysis with Sysreport](https://learn.arm.com/learning-paths/servers-and-cloud-computing/sysreport/) to learn how to install and run it.
6770

6871
Look at the Sysreport output and confirm SPE is available by checking the `perf sampling` field.
6972

@@ -99,21 +102,21 @@ List of pre-defined events (to be used in -e or -M):
99102
arm_spe_0// [Kernel PMU event]
100103
```
101104

102-
Assign capabilities to Perf by running:
105+
Assign capabilities to `perf` by running:
103106

104107
```bash
105108
sudo setcap cap_perfmon,cap_sys_ptrace,cap_sys_admin+ep $(which perf)
106109
```
107110

108-
If `arm_spe` is not available because of your system configuration or if you don't have PMU permission, the `perf c2c` command fails.
111+
If `arm_spe` isn’t available due to your system configuration or limited PMU access, the `perf c2c` command will fail.
109112

110113
To confirm Perf can access SPE, run:
111114

112115
```bash
113116
perf c2c record
114117
```
115118

116-
The output showing the failure is:
119+
If SPE access is blocked, you’ll see output like this:
117120

118121
```output
119122
failed: memory events not supported
@@ -122,9 +125,11 @@ failed: memory events not supported
122125
{{% notice Note %}}
123126
If you are unable to use SPE it might be a restriction based on your cloud instance size or operating system.
124127

125-
Generally, access to a full server (also known as metal instances) with a relatively new kernel is needed for Arm SPE support.
128+
Generally, access to a full server (also known as metal instances) with a relatively new kernel is required for Arm SPE support.
126129

127130
For more information about enabling SPE, see the [perf-arm-spe manual page](https://man7.org/linux/man-pages/man1/perf-arm-spe.1.html)
128131
{{% /notice %}}
129132

130-
Continue to learn how to use Perf C2C on an example application.
133+
## Summary
134+
135+
You've confirmed that your system supports Arm SPE, installed the necessary tools, and verified that perf can access SPE events. You're now ready to start collecting detailed performance data using perf c2c. In the next section, you’ll run a real application and use Perf C2C to capture cache sharing behavior and uncover memory performance issues.

content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/how-to-3.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ layout: learningpathall
88

99
## Example code
1010

11+
{{% notice Learning Goal%}}
12+
The example code in this section demonstrates how false sharing affects performance by comparing two multithreaded programs — one with cache-aligned data structures and one without. You’ll compile and run both versions, observe the runtime difference, and learn how memory layout affects cache behavior. This sets the stage for analyzing performance with Perf C2C in the next section.
13+
{{% /notice %}}
14+
1115
Use a text editor to copy and paste the C example code below into a file named `false_sharing_example.c`
1216

1317
The code is adapted from [Joe Mario](https://github.com/joemario/perf-c2c-usage-files) and is discussed thoroughly in the Arm Statistical Profiling Extension Whitepaper.
@@ -285,7 +289,7 @@ int main ( int argc, char *argv[] )
285289
286290
### Code explanation
287291
288-
The key data structure that occupies the cache is `struct Buf`. With a 64-byte cache line size, each line can hold 8, 8-byte `long` integers.
292+
The key data structure that occupies the cache is `struct _buf`. With a 64-byte cache line size, each line can hold 8, 8-byte `long` integers.
289293
290294
If you do not pass in the `NO_FALSE_SHARING` macro during compilation the `Buf` data structure will contain the elements below. Each structure neatly occupies the entire 64-byte cache line.
291295
@@ -306,7 +310,7 @@ typedef struct _buf {
306310

307311
Alternatively if you pass in the `NO_FALSE_SHARING` macro during compilation, the `Buf` structure has a different shape.
308312

309-
The 40 bytes of padding pushes the reader variables onto a different cache line. However, notice that this is with the tradeoff the new `Buf` structures occupies multiple cache lines (12 long integers). Therefore it leaves unused cache space of 25% per `Buf` structure.
313+
The 40 bytes of padding pushes the reader variables onto a different cache line. However, notice that this is with the tradeoff the new `Buf` structures occupies multiple cache lines (12 long integers). Therefore it leaves unused cache space of 25% per `Buf` structure. This trade-off uses more memory but eliminates false sharing, improving performance by reducing cache line contention.
310314

311315
```output
312316
typedef struct _buf {
@@ -345,5 +349,6 @@ user 0m8.869s
345349
sys 0m0.000s
346350
```
347351

348-
Continue to the next section to learn how to use Perf C2C to analyze the example code.
352+
## Summary
353+
In this section, you ran a hands-on C example to see how false sharing can significantly degrade performance in multithreaded applications. By comparing two versions of the same program, one with aligned memory access and one without, you saw how something as subtle as cache line layout can result in a 2x difference in runtime. This practical example sets the foundation for using Perf C2C to capture and analyze real cache line sharing behavior in the next section.
349354

content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/how-to-4.md

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,13 @@ weight: 5
66
layout: learningpathall
77
---
88

9-
## Measuring Performance
9+
## Compare Performance with perf stat
1010

11-
A straight forward method to observe the performance characteristics of both binaries would be to use the `perf stat` command.
11+
{{% notice Learning goal %}}
12+
In this section, you’ll learn how to use Linux Perf tools and Arm SPE to identify performance bottlenecks in multithreaded applications. You’ll compare aligned and unaligned workloads, detect cache-related slowdowns like false sharing, and trace memory contention down to the source code using Perf C2C.
13+
{{% /notice %}}
14+
15+
A straightforward method to observe the performance characteristics of both binaries would be to use the `perf stat` command.
1216

1317
For example, run the false sharing version using:
1418

@@ -66,13 +70,13 @@ The output is similar to:
6670

6771
Comparing the results you can see the run time is significantly different (13.01 s vs. 6.49 s).
6872

69-
Additionally, the instructions per cycle (IPC) is notably different, (0.74 vs. 1.70) and looks to be commensurate to run time.
73+
The instructions per cycle (IPC) are also notably different, (0.74 vs. 1.70) and look to be commensurate to run time.
7074

71-
## Understanding the root cause
75+
## Pinpoint pipeline bottlenecks with top-down analysis
7276

7377
There are many root causes of variations in IPC.
7478

75-
To identify an area to focus on we will start off using the [top-down methodology](https://developer.arm.com/documentation/109542/0100/Arm-Topdown-methodology). Install the python script using the [install guide](https://learn.arm.com/install-guides/topdown-tool/).
79+
To identify where the bottleneck occurs, we’ll start by using the [Arm Topdown methodology](https://developer.arm.com/documentation/109542/0100/Arm-Topdown-methodology). Install the python script using the [Telemetry Solution Install Guide](https://learn.arm.com/install-guides/topdown-tool/).
7680

7781
Run the following command to observe the ratio of frontend to backend stall cycles. These indicate which section of the CPU pipeline is waiting on resources and causing slower performance.
7882

@@ -92,7 +96,7 @@ Backend Stalled Cycles. 75.24% cycles
9296

9397
The output shows there are disproportionately more backend stall cycles. This indicates the CPU is waiting for data. You could follow the top-down methodology further looking at the stage 2 microarchitecture analysis, but for sake of brevity you can jump to recording events with SPE.
9498

95-
## Skid when using Perf Record
99+
## Skid: When perf record misleads
96100

97101
The naive approach would be to record the events using the `perf record` subcommand. Running the following commands can be used to demonstrate skid, inaccuracy or "slippage" in the instruction location recorded by the Performance Monitoring Unit (PMU) when a performance event is sampled.
98102

@@ -155,4 +159,8 @@ Looking at the corresponding source code, we observe the following.
155159

156160
The output from SPE-based profiling with Perf C2C shows that attempting to access and increment the `lock0` and `reader1` variable, is causing the bottleneck.
157161

158-
The insight generated from Perf C2C indicates to reorganize the layout of the data structure.
162+
The insight generated from Perf C2C indicates to reorganize the layout of the data structure.
163+
164+
## Summary
165+
166+
In this section, you used multiple tools to analyze and diagnose a real performance issue caused by false sharing. You compared performance between aligned and unaligned code using perf stat, investigated backend stalls with topdown-tool, and saw how standard perf record can mislead due to instruction skid. Finally, you used Perf C2C with Arm SPE to pinpoint the exact variables and code lines causing contention, giving you actionable insight into how to reorganize your data layout for better performance.

0 commit comments

Comments
 (0)