final tech review

Author: jasonrandrews

content/learning-paths/servers-and-cloud-computing/memory-subsystem/_index.md

@@ -13,7 +13,7 @@ who_is_this_for: This is an advanced topic for software developers and performan
 
 learning_objectives:
     - Identify the core topology, cluster layout, and cache hierarchy of an Arm Linux system using standard tools
-    - Measure cache and memory latency using a pointer-chase benchmark from ASCT
+    - Measure cache and memory latency using a pointer-chase benchmark
     - Measure single-core and multi-core streaming bandwidth at each level of the memory hierarchy
     - Evaluate latency behavior under bandwidth pressure 
     - Compare results across Arm systems and draw conclusions

content/learning-paths/servers-and-cloud-computing/memory-subsystem/cache-hierarchy.md

@@ -6,18 +6,16 @@ weight: 3
 layout: learningpathall
 ---
 
-## Why cache hierarchy matters
+## Cache levels and performance clifs
 
-Each memory access is satisfied by the closest level of the hierarchy that contains the requested data. If the data is not found in the private caches, the request falls through to lower levels and eventually to DRAM. Each level trades capacity for speed: L1 is tiny but fast, L2 is larger but slower, and L3 (or a system-level cache) is the largest on-chip cache but has the highest latency. Understanding this hierarchy for your specific system tells you where performance "cliffs" will occur as your working set grows.
+Each memory access is satisfied by the closest level of the hierarchy that contains the requested data. If the data is not found in the private caches, the request falls through to the next cache level and eventually to DRAM. Each level trades capacity for speed: L1 is tiny but fast, L2 is larger but slower, and L3 (or a system-level cache) is the largest on-chip cache but has the highest latency. Understanding this hierarchy for your specific system tells you where performance "cliffs" will occur as your working set grows.
 
 ## Cache levels on Arm systems
 
-Both systems use 4-way L1 caches with 64-byte lines and 8-way L2 caches, also with 64-byte lines. The key architectural differences are in L2 size and whether a shared last-level cache is present.
+Both Graviton systems use 4-way L1 caches with 64-byte lines and 8-way L2 caches, also with 64-byte lines. The key architectural differences are in L2 size and the last level cache size. If you investigate a variety of Arm Linux systems you will see many different cache sizes and configurations.
 
 Graviton2 (Neoverse N1) has a 1 MB private L2 per core and a 32 MB shared L3 across all 64 cores. Graviton4 (Neoverse V2) doubles the L2 to 2 MB and has a 36 MB shared L3. The larger L2 on Neoverse V2 reduces traffic to the shared L3 and lowers contention in multi-threaded workloads, which is especially beneficial for workloads with per-thread working sets between 1 MB and 2 MB.
 
-Some SoCs add a System Level Cache (SLC) beyond L3, sitting between the CPU cluster and the memory controllers and serving both CPU and GPU traffic.
-
 ## Read cache properties from the kernel
 
 You already saw the `sysfs` approach in the previous section. You can also create a structured summary across all cache levels.
@@ -98,7 +96,7 @@ For Arm Neoverse cores, the Technical Reference Manuals (TRMs) list:
 - Prefetcher behavior
 - Expected cycle-count latencies
 
-Compare your `sysfs` findings with the TRM. If the sizes differ, the SoC vendor may have chosen a non-default configuration. For example, Neoverse V2 allows L2 sizes of 1 MB or 2 MB, and the TRM documents both options.
+Compare your `sysfs` findings with the TRM. If the sizes differ, the SoC vendor may have chosen a different configuration. Many Arm CPUs have multiple options for cache size. For example, Neoverse V2 allows L2 sizes of 1 MB or 2 MB, and the TRM documents both options.
 
 {{% notice Tip %}}
 Arm publishes TRMs on the [Arm Developer documentation portal](https://developer.arm.com/documentation/). Search for the specific core name (for example, "Neoverse N1 TRM" or "Neoverse V2 TRM") to find the detailed cache specifications.
@@ -111,4 +109,4 @@ In this section you:
 - Learned the key concepts of cache lines, associativity, prefetching, and ns-vs-cycles that explain benchmark results
 - Cross-checked kernel-reported values against Arm documentation
 
-The next section uses the ASCT `latency-sweep` benchmark to measure the actual access latency at each level of the hierarchy.
+The next section uses the ASCT `latency-sweep` benchmark to measure the actual access latency at each level of the memory hierarchy.

content/learning-paths/servers-and-cloud-computing/memory-subsystem/comparative-analysis.md

@@ -14,15 +14,15 @@ You now have a complete set of memory subsystem measurements for each of your te
 
 Throughout the previous sections, you ran individual ASCT benchmarks. To characterize a new system from scratch, run all memory benchmarks at once.
 
-Run all tests and save the results.
+Run all tests on each system and save the results.
 
 ```bash
 sudo asct run memory loaded-latency --output-dir results_$(hostname)
 ```
 
 This runs `latency-sweep`, `bandwidth-sweep`, `idle-latency`, `peak-bandwidth`, `c2c-latency`, and `loaded-latency` in a single pass, saving all results and plots to the output directory. The directory name includes the hostname so you can easily identify which system produced the data.
 
-## Compare systems with asct diff
+## Compare systems with the diff command
 
 ASCT includes a `diff` command that compares output directories from different runs. It loads the results from each directory, groups fields by benchmark, and reports percentage differences for measurements and raw values for system configuration.
 
@@ -66,7 +66,7 @@ To compare everything except system-info:
 asct diff results_graviton2-c6g/ results_graviton4-c8g/ --benchmarks ^system-info
 ```
 
-Output is:
+The output for everything except the system info is:
 
 ```output
 ,field,recipe,run,comparator,delta,delta_percent,baseline
@@ -147,50 +147,34 @@ Collect the key measurements from each system into a comparison table:
 
 ## Analyze the differences
 
-### Latency
+The conclusions below are from running ASCT on AWS EC2 instances that are easily available for you to try so you can learn the process and learn how to think about the results. There are many more details that go into CPU microarchitecture and system design that are not covered here, but this is a good way to get an initial understanding of the memory system of an Arm Linux server. 
 
-The latency measurements from `latency-sweep` reveal how each generation's cache hierarchy performs:
+### Latency
 
-- **L1 latency** is similar across both generations (1.6 ns vs 1.4 ns). L1 caches are designed for very low latency (typically a few cycles), so the small difference reflects clock speed rather than cache microarchitecture.
-- **L2 latency** improves by 27% on Graviton4 (4.0 ns vs 5.4 ns). Neoverse V2 has a 2 MB private L2 compared to Neoverse N1's 1 MB. Despite the larger size, Neoverse V2's improved cache pipeline delivers lower latency.
-- **LLC latency** improves by 24% on Graviton4 (21.8 ns vs 28.8 ns), despite the L3 being only slightly larger (36 MB vs 32 MB). This reflects microarchitectural improvements in the Neoverse V2 cache interconnect.
-- **DRAM latency** is higher on Graviton4 (114.6 ns vs 95.5 ns, +20%). DDR5 trades slightly higher access latency for significantly more bandwidth per channel compared to DDR4.
+The latency measurements from `latency-sweep` reveal how each generation's cache hierarchy performs. L1 latency is similar across both generations (1.6 ns vs 1.4 ns) because L1 caches are designed for very low latency, so the small difference reflects clock speed rather than cache microarchitecture. L2 latency improves by 27% on Graviton4 (4.0 ns vs 5.4 ns): Neoverse V2 has a 2 MB private L2 compared to Neoverse N1's 1 MB, and despite the larger size, the improved cache pipeline delivers lower latency. LLC latency improves by 24% on Graviton4 (21.8 ns vs 28.8 ns) even though the L3 is only slightly larger (36 MB vs 32 MB), which reflects microarchitectural improvements in the Neoverse V2 cache interconnect. DRAM latency is higher on Graviton4 (114.6 ns vs 95.5 ns, +20%) because DDR5 trades slightly higher access latency for significantly more bandwidth per channel compared to DDR4.
 
 ### Loaded latency
 
-The `loaded-latency` results reveal the latency-bandwidth tradeoff:
-
-- At low load (~3000 NOPs), latency matches the idle DRAM latency from `latency-sweep` (96.7 ns for Graviton2, 115.1 ns for Graviton4), confirming the two benchmarks are consistent.
-- The knee on Graviton2 occurs between 70 and 50 NOPs, where latency nearly doubles from 134 ns to 266 ns. Beyond the knee, latency climbs to 344 ns at saturation.
-- The knee on Graviton4 occurs between 30 and 20 NOPs, at a much higher bandwidth (~340 GB/s vs ~126 GB/s on Graviton2). Graviton4's latency plateaus around 190-198 ns at saturation rather than continuing to climb, suggesting the DDR5 memory controllers handle queue saturation more gracefully.
-- Despite Graviton4's higher idle DRAM latency, its latency at saturation is 43% lower than Graviton2's (197 ns vs 344 ns), making it the better choice for workloads that operate near memory bandwidth limits.
+The `loaded-latency` results reveal the latency-bandwidth tradeoff. At low load (~3000 NOPs), latency matches the idle DRAM latency from `latency-sweep` (96.7 ns for Graviton2, 115.1 ns for Graviton4), confirming the two benchmarks are consistent. The knee on Graviton2 occurs between 70 and 50 NOPs, where latency nearly doubles from 134 ns to 266 ns before climbing to 344 ns at saturation. On Graviton4 the knee occurs between 30 and 20 NOPs at a much higher bandwidth (~340 GB/s vs ~126 GB/s on Graviton2), and latency plateaus around 190–198 ns at saturation rather than continuing to climb, suggesting the DDR5 memory controllers handle queue saturation more gracefully. Despite Graviton4's higher idle DRAM latency, its latency at saturation is 43% lower than Graviton2's (197 ns vs 344 ns), making it the better choice for workloads that operate near memory bandwidth limits.
 
 ### Bandwidth
 
-Single-core bandwidth from `bandwidth-sweep` shows the throughput capacity at each cache level:
-
-- **L1 bandwidth** doubles on Graviton4 (321 vs 159 GB/s, +101%). This likely reflects a combination of higher clock speed and microarchitectural throughput improvements in Neoverse V2.
-- **L2 bandwidth** improves by 29% on Graviton4 (95 vs 73 GB/s), due to the wider L2 fill path and microarchitectural improvements in Neoverse V2.
-- **LLC bandwidth** more than doubles on Graviton4 (80 vs 36 GB/s, +123%), reflecting improvements in the interconnect and shared cache design.
-- **DRAM bandwidth (single core)** improves by 77% on Graviton4 (37 vs 21 GB/s). Neoverse V2 supports more outstanding memory requests than Neoverse N1, and DDR5 provides more bandwidth per channel than DDR4.
-
-Peak bandwidth from `peak-bandwidth` shows the system-level memory controller capacity:
+Single-core bandwidth from `bandwidth-sweep` shows the throughput capacity at each cache level. L1 bandwidth doubles on Graviton4 (321 vs 159 GB/s, +101%), likely reflecting a combination of higher clock speed and microarchitectural throughput improvements in Neoverse V2. L2 bandwidth improves by 29% (95 vs 73 GB/s) due to the wider L2 fill path, and LLC bandwidth more than doubles (80 vs 36 GB/s, +123%), reflecting improvements in the interconnect and shared cache design. Single-core DRAM bandwidth improves by 77% (37 vs 21 GB/s) because Neoverse V2 supports more outstanding memory requests than Neoverse N1 and DDR5 provides more bandwidth per channel than DDR4.
 
-- Graviton4 delivers 2.7x the peak all-reads bandwidth of Graviton2 (465.6 vs 169.7 GB/s). This is the most dramatic difference between the two generations.
-- Compare the "All Reads" figure with the theoretical peak bandwidth from `asct system-info` to see how efficiently each system uses its available bandwidth. Well-configured systems typically achieve 85-95% of theoretical peak.
+Peak bandwidth from `peak-bandwidth` shows the system-level memory controller capacity. Graviton4 delivers 2.7x the peak all-reads bandwidth of Graviton2 (465.6 vs 169.7 GB/s), the most dramatic difference between the two generations. Compare the "All Reads" figure with the theoretical peak bandwidth from `asct system-info` to see how efficiently each system uses its available bandwidth — well-configured systems typically achieve 85–95% of theoretical peak.
 
 ## Cross-validate your findings
 
 The credibility of this type of analysis comes from cross-validation. Check that:
 
-1. **Latency steps match cache sizes**: the `latency-sweep` boundaries should align with the cache sizes reported by `sysfs` and the TRM.
-2. **Bandwidth plateaus match cache sizes**: the `bandwidth-sweep` should show transitions at the same data sizes that `latency-sweep` identified.
-3. **Peak bandwidth matches documented limits**: compare the "All Reads" figure from `peak-bandwidth` against the theoretical peak from `asct system-info`. A well-configured system typically achieves 85-95% of theoretical peak.
-4. **Idle loaded latency matches unloaded latency**: the lowest-load row from `loaded-latency` should be close to the DRAM latency from `latency-sweep`.
-5. **Results are repeatable**: run each benchmark at least twice. If results vary by more than 5%, investigate sources of noise (CPU frequency scaling, background processes).
+1. Latency steps match cache sizes: the `latency-sweep` boundaries should align with the cache sizes reported by `sysfs` and the TRM.
+2. Bandwidth plateaus match cache sizes: the `bandwidth-sweep` should show transitions at the same data sizes that `latency-sweep` identified.
+3. Peak bandwidth matches documented limits: compare the "All Reads" figure from `peak-bandwidth` against the theoretical peak from `asct system-info`. A well-configured system typically achieves 85-95% of theoretical peak.
+4. Idle loaded latency matches unloaded latency: the lowest-load row from `loaded-latency` should be close to the DRAM latency from `latency-sweep`.
+5. Results are repeatable: run each benchmark at least twice. If results vary by more than 5%, investigate sources of noise (CPU frequency scaling, background processes).
 
 {{% notice Tip %}}
-Disable CPU frequency scaling during benchmarks for more consistent results:
+Check if your system allows you to control CPU frequency scaling during benchmarks for more consistent results:
 ```bash
 echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
 ```
@@ -200,10 +184,10 @@ echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governo
 
 The examples here used AWS Graviton instances, but the methodology works on any Arm Linux machine with a NUMA-enabled kernel. To adapt it:
 
-1. **Install ASCT** on the target system following the [install guide](/install-guides/asct/).
-2. **Run all memory benchmarks**: `sudo asct run memory loaded-latency --format=csv --output-dir results_$(hostname)`
-3. **Compare with previous systems**: `asct diff results_new/ --baseline results_reference/`
-4. **Cross-validate**: check that latency boundaries, bandwidth plateaus, and peak numbers are consistent with each other and with the hardware documentation.
+1. Install ASCT on the target system following the [install guide](/install-guides/asct/).
+2. Run all memory benchmarks: `sudo asct run memory loaded-latency --format=csv --output-dir results_$(hostname)`
+3. Compare with previous systems: `asct diff results_new/ --baseline results_reference/`
+4. Cross-validate: check that latency boundaries, bandwidth plateaus, and peak numbers are consistent with each other and with the hardware documentation.
 
 
 ## What you've accomplished

content/learning-paths/servers-and-cloud-computing/memory-subsystem/multicore-bandwidth.md

@@ -17,14 +17,14 @@ ASCT includes two benchmarks that characterize multi-core memory behavior:
 - `peak-bandwidth` uses all cores on all NUMA nodes to measure the maximum achievable memory bandwidth under multiple traffic patterns.
 - `loaded-latency` measures how memory latency degrades as other cores generate increasing bandwidth pressure.
 
-Both benchmarks depend on `latency-sweep` to determine the optimal data size for targeting DRAM. If you haven't already run `latency-sweep`, ASCT runs it automatically as a dependency.
+Both benchmarks depend on `latency-sweep` to determine the optimal data size for targeting DRAM. ASCT runs it automatically as a dependency.
 
 ## Peak bandwidth
 
 The `peak-bandwidth` benchmark makes full use of all cores to measure the maximum system bandwidth. It tests multiple traffic patterns to show how the mix of reads and writes affects peak throughput.
 
 ```bash
-sudo asct run peak-bandwidth  --output-dir peakbw_results
+sudo asct run peak-bandwidth  --output-dir peakbw_results_$(hostname)
 ```
 
 The Graviton2 output:
@@ -71,10 +71,10 @@ The `loaded-latency` benchmark measures how memory latency changes as other core
 The `loaded-latency` benchmark takes several minutes to complete because it runs multiple measurement phases at different traffic levels.
 {{% /notice %}}
 
-To save the output to a specific directory:
+Run the command on both system and save the output to a specific directory:
 
 ```bash
-sudo asct run loaded-latency --output-dir loaded_latency_results
+sudo asct run loaded-latency --output-dir loaded_latency_results_$(hostname)
 ```
 
 The Graviton2 output:

content/learning-paths/servers-and-cloud-computing/memory-subsystem/pointer-chase-latency.md

@@ -26,15 +26,15 @@ asct version
 
 If ASCT is not installed, follow the [ASCT install guide](/install-guides/asct/) to install it.
 
-The Arm System Characterization Tool (ASCT) includes a `latency-sweep` benchmark that implements a pointer chase internally. It sweeps data sizes from 128 bytes to 1 GiB, uses randomized linked lists to defeat prefetching, and allocates 1 GiB huge pages to minimize the effect of page table walks on the measurements.
+The Arm System Characterization Tool (ASCT) includes a `latency-sweep` benchmark that implements a pointer chase. It sweeps data sizes from 128 bytes to 1 GiB, uses randomized linked lists to defeat prefetching, and allocates 1 GiB huge pages to minimize the effect of page table walks on the measurements.
 
 The benchmark automatically identifies cache level boundaries and reports the optimal data size, lower bound, upper bound, and average latency for each level. It also generates a latency-vs-size line graph in the output directory.
 
 ASCT requires `sudo` because it configures huge pages and pins threads to specific cores.
 
 ASCT prints the results to the terminal and saves detailed output (including CSV data and a `latency-sweep.png` plot) in a timestamped directory under the current working directory.
 
-To save the output to a specific directory:
+Run the benchmark on both systems and save the output to a specific directory:
 
 ```bash
 sudo asct run latency-sweep --output-dir latency_results_$(hostname)