You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/_index.md
-3Lines changed: 0 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,9 +1,6 @@
1
1
---
2
2
title: Analyze cache behavior with Perf C2C on Arm
3
3
4
-
5
-
6
-
7
4
minutes_to_complete: 15
8
5
9
6
who_is_this_for: This topic is for performance-oriented developers working on Arm-based cloud or server systems who want to optimize memory access patterns and investigate cache inefficiencies using Perf C2C and Arm SPE.
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/how-to-1.md
+21-11Lines changed: 21 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,29 +1,38 @@
1
1
---
2
-
title: Introduction to Arm SPE and false sharing
2
+
title: Arm Statistical Profiling Extension and false sharing
3
3
weight: 2
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
-
## Introduction to the Arm Statistical Profiling Extension (SPE)
9
8
10
-
Traditional Linux performance profiling tools often rely on retired instruction counts, which lack visibility into memory addresses, cache latencies, and micro-operation behavior. Moreover, the “skid” phenomenon where events are falsely attributed to later instructions can mislead developers.
9
+
## What is the Arm Statistical Profiling Extension (SPE), and what does it do?
11
10
12
-
SPE integrates sampling directly into the CPU pipeline, triggering on individual micro-operations rather than retired instructions, thereby eliminating skid and blind spots. Each SPE sample record includes relevant metadata, such as data addresses, per-µop pipeline latency, triggered PMU event masks, and the memory hierarchy source, enabling fine-grained and precise cache analysis.
11
+
{{% notice Learning goal%}}
12
+
In this section, you will learn how the Arm Statistical Profiling Extension (SPE) gives you deeper visibility into how your applications interact with the CPU. It covers how to detect and fix false sharing, which is a hidden performance problem in multithreaded code, by using cache line alignment in C++ and tools like Perf C2C.
13
+
{{% /notice %}}
14
+
15
+
Arm’s Statistical Profiling Extension (SPE) gives you a powerful way to understand what’s really happening inside your applications at the microarchitecture level. Introduced in Armv8.2, SPE captures a statistical view of how instructions move through the CPU, which allows you to dig into issues like memory access latency, cache misses, and pipeline behavior.
16
+
17
+
Most Linux profiling tools focus on retired instruction counts, which means they miss key details like memory addresses, cache latency, and micro-operation behavior. This can lead to misleading results — especially due to a phenomenon called “skid,” where events are falsely attributed to later instructions.
18
+
19
+
SPE integrates sampling directly into the CPU pipeline, triggering on individual micro-operations instead of retired instructions. This approach eliminates skid and blind spots. Each SPE sample record includes relevant metadata, such as data addresses, per-µop pipeline latency, triggered PMU event masks, and the memory hierarchy source, enabling fine-grained, precise cache analysis.
13
20
14
21
SPE helps developers optimize user-space applications by showing where cache latency or memory access delays are happening. Importantly, cache statistics are enabled with the Linux Perf cache-to-cache (C2C) utility.
15
22
16
-
To learn more, check out the [Arm SPE white paper](https://developer.arm.com/documentation/109429/latest/) for more information.
23
+
For more information, see the [*Arm Statistical Profiling Extension: Performance Analysis Methodology White Paper*](https://developer.arm.com/documentation/109429/latest/).
17
24
18
25
In this Learning Path, you will use SPE and Perf C2C to diagnose a cache issue for an application running on a Neoverse server.
19
26
20
27
## What is false sharing and why should I care about it?
21
28
22
29
In large-scale, multithreaded applications, false sharing can degrade performance by introducing hundreds of unnecessary cache line invalidations per second - often with no visible red flags in the source code.
23
30
24
-
Even when two threads touch entirely separate variables, modern processors move data in fixed-size cache lines (nominally 64-bytes). If those distinct variables happen to occupy bytes within the same line, every time one thread writes its variable the core’s cache must gain exclusive ownership of the whole line, forcing the other core’s copy to be invalidated.
31
+
Even when two threads touch entirely separate variables, modern processors move data in fixed-size cache lines, which is typically 64 bytes. If those distinct variables happen to occupy bytes within the same line, every time one thread writes its variable the core’s cache must gain exclusive ownership of the whole line, forcing the other core’s copy to be invalidated.
25
32
26
-
The second thread, still working on its own variable, then triggers a coherence miss to fetch the line back, and the ping-pong pattern repeats. See the illustration below, taken from the Arm SPE white paper, for a visual explanation.
33
+
The second thread, still working on its own variable, then triggers a coherence miss to fetch the line back, and the ping-pong pattern repeats.
34
+
35
+
The diagram below, taken from the Arm SPE white paper, provides a visual representation of two threads on separate cores alternately gaining exclusive access to the same cache line.
27
36
28
37

29
38
@@ -77,9 +86,9 @@ int main() {
77
86
}
78
87
```
79
88
80
-
The example output below shows the variables e, f, g and h occur at least 64 bytes apart in the byte-addressable architecture. Whereas variables a, b, c, and d occur 8 bytes apart, occupying the same cache line.
89
+
The output below shows that the variables e, f, g and h occur at least 64 bytes apart in the byte-addressable architecture. Whereas variables a, b, c, and d occur 8 bytes apart, occupying the same cache line.
81
90
82
-
Although this is a contrived example, in a production workload there might be several layers of indirection that unintentionally result in false sharing. For these complex cases, to understand the root cause you will use Perf C2C.
91
+
Although this is a contrived example, in a production workload there might be several layers of indirection that unintentionally result in false sharing. For these complex cases, to understand the root cause, you can use Perf C2C.
83
92
84
93
```output
85
94
Without Alignment can occupy same cache line
@@ -101,7 +110,8 @@ Address of AlignedType h - 0xffffeb6c6080
101
110
102
111
## Summary
103
112
104
-
In this section, you learned what Arm SPE is and why it’s a game-changer for performance profiling. You also saw how a subtle issue like false sharing can slow down multithreaded code — and how to avoid it using alignment in C++.
113
+
In this section, you explored what Arm SPE is and why it offers a deeper, more accurate view of application performance. You also examined how a subtle issue like false sharing can impact multithreaded code, and how to mitigate it using data alignment techniques in C++.
114
+
115
+
Next, you'll set up your environment and use Perf C2C to capture and analyze real cache behavior on an Arm Neoverse system.
105
116
106
-
Next up: you’ll set up your environment and get ready to run Perf C2C on a real application.
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/how-to-2.md
+17-12Lines changed: 17 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,20 +1,23 @@
1
1
---
2
-
title: Configure your environment for Arm SPE profiling
2
+
title: Environment setup for Arm SPE profiling
3
3
weight: 3
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
-
9
8
## Select a system with SPE support
10
9
11
-
SPE requires both hardware and operating system support. Many cloud instances running Linux do not enable SPE-based profiling.
10
+
{{% notice Learning goal%}}
11
+
Before you can start profiling cache behavior with Arm SPE and Perf C2C, your system needs to meet a few requirements. In this section, you’ll learn how to check whether your hardware and kernel support Arm SPE, install the necessary tools, and validate that Linux Perf can access the right performance monitoring events. By the end, your environment will be ready to record and analyze memory access patterns using Perf C2C on an Arm Neoverse system.
12
+
{{% /notice %}}
13
+
14
+
SPE requires support from both your hardware and the operating system. Many cloud instances running Linux do not enable SPE-based profiling.
12
15
13
16
You need to identify a system that supports SPE using the information below.
14
17
15
18
If you are looking for an AWS system, you can use a `c6g.metal` instance running Amazon Linux 2023 (AL2023).
16
19
17
-
Check the underlying Neoverse processor and operating system kernel version with the following commands.
20
+
Check the underlying Neoverse processor and operating system kernel version with the following commands:
18
21
19
22
```bash
20
23
lscpu | grep -i "model name"
@@ -43,15 +46,15 @@ Run the following command to confirm if the SPE kernel module is loaded:
43
46
sudo modprobe arm_spe_pmu
44
47
```
45
48
46
-
If the module is not loaded (blank output), SPE might still be available.
49
+
If the module is not loaded (and there is blank output), SPE might still be available.
47
50
48
51
Run this command to check if SPE is included in the kernel:
49
52
50
53
```bash
51
54
ls /sys/bus/event_source/devices/ | grep arm_spe
52
55
```
53
56
54
-
If SPE is available, the output is:
57
+
If SPE is available, the output you will see is:
55
58
56
59
```output
57
60
arm_spe_0
@@ -63,7 +66,7 @@ If the output is blank then SPE is not available.
63
66
64
67
You can install and run a Python script named Sysreport to summarize your system's performance profiling capabilities.
65
68
66
-
See [Get ready for performance analysis with Sysreport](https://learn.arm.com/learning-paths/servers-and-cloud-computing/sysreport/) to learn how to install and run it.
69
+
See the Learning Path [Get ready for performance analysis with Sysreport](https://learn.arm.com/learning-paths/servers-and-cloud-computing/sysreport/) to learn how to install and run it.
67
70
68
71
Look at the Sysreport output and confirm SPE is available by checking the `perf sampling` field.
69
72
@@ -99,21 +102,21 @@ List of pre-defined events (to be used in -e or -M):
If `arm_spe`is not available because of your system configuration or if you don't have PMU permission, the `perf c2c` command fails.
111
+
If `arm_spe`isn’t available due to your system configuration or limited PMU access, the `perf c2c` command will fail.
109
112
110
113
To confirm Perf can access SPE, run:
111
114
112
115
```bash
113
116
perf c2c record
114
117
```
115
118
116
-
The output showing the failure is:
119
+
If SPE access is blocked, you’ll see output like this:
117
120
118
121
```output
119
122
failed: memory events not supported
@@ -122,9 +125,11 @@ failed: memory events not supported
122
125
{{% notice Note %}}
123
126
If you are unable to use SPE it might be a restriction based on your cloud instance size or operating system.
124
127
125
-
Generally, access to a full server (also known as metal instances) with a relatively new kernel is needed for Arm SPE support.
128
+
Generally, access to a full server (also known as metal instances) with a relatively new kernel is required for Arm SPE support.
126
129
127
130
For more information about enabling SPE, see the [perf-arm-spe manual page](https://man7.org/linux/man-pages/man1/perf-arm-spe.1.html)
128
131
{{% /notice %}}
129
132
130
-
Continue to learn how to use Perf C2C on an example application.
133
+
## Summary
134
+
135
+
You've confirmed that your system supports Arm SPE, installed the necessary tools, and verified that perf can access SPE events. You're now ready to start collecting detailed performance data using perf c2c. In the next section, you’ll run a real application and use Perf C2C to capture cache sharing behavior and uncover memory performance issues.
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/how-to-3.md
+8-3Lines changed: 8 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,6 +8,10 @@ layout: learningpathall
8
8
9
9
## Example code
10
10
11
+
{{% notice Learning Goal%}}
12
+
The example code in this section demonstrates how false sharing affects performance by comparing two multithreaded programs — one with cache-aligned data structures and one without. You’ll compile and run both versions, observe the runtime difference, and learn how memory layout affects cache behavior. This sets the stage for analyzing performance with Perf C2C in the next section.
13
+
{{% /notice %}}
14
+
11
15
Use a text editor to copy and paste the C example code below into a file named `false_sharing_example.c`
12
16
13
17
The code is adapted from [Joe Mario](https://github.com/joemario/perf-c2c-usage-files) and is discussed thoroughly in the Arm Statistical Profiling Extension Whitepaper.
@@ -285,7 +289,7 @@ int main ( int argc, char *argv[] )
285
289
286
290
### Code explanation
287
291
288
-
The key data structure that occupies the cache is `struct Buf`. With a 64-byte cache line size, each line can hold 8, 8-byte `long` integers.
292
+
The key data structure that occupies the cache is `struct _buf`. With a 64-byte cache line size, each line can hold 8, 8-byte `long` integers.
289
293
290
294
If you do not pass in the `NO_FALSE_SHARING` macro during compilation the `Buf` data structure will contain the elements below. Each structure neatly occupies the entire 64-byte cache line.
291
295
@@ -306,7 +310,7 @@ typedef struct _buf {
306
310
307
311
Alternatively if you pass in the `NO_FALSE_SHARING` macro during compilation, the `Buf` structure has a different shape.
308
312
309
-
The 40 bytes of padding pushes the reader variables onto a different cache line. However, notice that this is with the tradeoff the new `Buf` structures occupies multiple cache lines (12 long integers). Therefore it leaves unused cache space of 25% per `Buf` structure.
313
+
The 40 bytes of padding pushes the reader variables onto a different cache line. However, notice that this is with the tradeoff the new `Buf` structures occupies multiple cache lines (12 long integers). Therefore it leaves unused cache space of 25% per `Buf` structure. This trade-off uses more memory but eliminates false sharing, improving performance by reducing cache line contention.
310
314
311
315
```output
312
316
typedef struct _buf {
@@ -345,5 +349,6 @@ user 0m8.869s
345
349
sys 0m0.000s
346
350
```
347
351
348
-
Continue to the next section to learn how to use Perf C2C to analyze the example code.
352
+
## Summary
353
+
In this section, you ran a hands-on C example to see how false sharing can significantly degrade performance in multithreaded applications. By comparing two versions of the same program, one with aligned memory access and one without, you saw how something as subtle as cache line layout can result in a 2x difference in runtime. This practical example sets the foundation for using Perf C2C to capture and analyze real cache line sharing behavior in the next section.
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/false-sharing-arm-spe/how-to-4.md
+15-7Lines changed: 15 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,13 @@ weight: 5
6
6
layout: learningpathall
7
7
---
8
8
9
-
## Measuring Performance
9
+
## Compare Performance with perf stat
10
10
11
-
A straight forward method to observe the performance characteristics of both binaries would be to use the `perf stat` command.
11
+
{{% notice Learning goal %}}
12
+
In this section, you’ll learn how to use Linux Perf tools and Arm SPE to identify performance bottlenecks in multithreaded applications. You’ll compare aligned and unaligned workloads, detect cache-related slowdowns like false sharing, and trace memory contention down to the source code using Perf C2C.
13
+
{{% /notice %}}
14
+
15
+
A straightforward method to observe the performance characteristics of both binaries would be to use the `perf stat` command.
12
16
13
17
For example, run the false sharing version using:
14
18
@@ -66,13 +70,13 @@ The output is similar to:
66
70
67
71
Comparing the results you can see the run time is significantly different (13.01 s vs. 6.49 s).
68
72
69
-
Additionally, the instructions per cycle (IPC) is notably different, (0.74 vs. 1.70) and looks to be commensurate to run time.
73
+
The instructions per cycle (IPC) are also notably different, (0.74 vs. 1.70) and look to be commensurate to run time.
70
74
71
-
## Understanding the root cause
75
+
## Pinpoint pipeline bottlenecks with top-down analysis
72
76
73
77
There are many root causes of variations in IPC.
74
78
75
-
To identify an area to focus on we will start off using the [top-down methodology](https://developer.arm.com/documentation/109542/0100/Arm-Topdown-methodology). Install the python script using the [install guide](https://learn.arm.com/install-guides/topdown-tool/).
79
+
To identify where the bottleneck occurs, we’ll start by using the [Arm Topdown methodology](https://developer.arm.com/documentation/109542/0100/Arm-Topdown-methodology). Install the python script using the [Telemetry Solution Install Guide](https://learn.arm.com/install-guides/topdown-tool/).
76
80
77
81
Run the following command to observe the ratio of frontend to backend stall cycles. These indicate which section of the CPU pipeline is waiting on resources and causing slower performance.
The output shows there are disproportionately more backend stall cycles. This indicates the CPU is waiting for data. You could follow the top-down methodology further looking at the stage 2 microarchitecture analysis, but for sake of brevity you can jump to recording events with SPE.
94
98
95
-
## Skid when using Perf Record
99
+
## Skid: When perf record misleads
96
100
97
101
The naive approach would be to record the events using the `perf record` subcommand. Running the following commands can be used to demonstrate skid, inaccuracy or "slippage" in the instruction location recorded by the Performance Monitoring Unit (PMU) when a performance event is sampled.
98
102
@@ -155,4 +159,8 @@ Looking at the corresponding source code, we observe the following.
155
159
156
160
The output from SPE-based profiling with Perf C2C shows that attempting to access and increment the `lock0` and `reader1` variable, is causing the bottleneck.
157
161
158
-
The insight generated from Perf C2C indicates to reorganize the layout of the data structure.
162
+
The insight generated from Perf C2C indicates to reorganize the layout of the data structure.
163
+
164
+
## Summary
165
+
166
+
In this section, you used multiple tools to analyze and diagnose a real performance issue caused by false sharing. You compared performance between aligned and unaligned code using perf stat, investigated backend stalls with topdown-tool, and saw how standard perf record can mislead due to instruction skid. Finally, you used Perf C2C with Arm SPE to pinpoint the exact variables and code lines causing contention, giving you actionable insight into how to reorganize your data layout for better performance.
0 commit comments