From 797eb29273228b13f91d60c6e2aa256fae43d79d Mon Sep 17 00:00:00 2001 From: Charliechen114514 <725610365@qq.com> Date: Thu, 25 Jun 2026 22:17:56 +0800 Subject: [PATCH] release: version v0.7.0 release --- .gitignore | 1 + CONTRIBUTORS.md | 6 + README.md | 10 +- changelogs/v0.7.0.md | 80 +++ documents/en/team/index.md | 9 + .../exercises/01-bounded-queue.md | 2 +- .../10-asan-family-and-memory-safety.md | 411 +++++++++++++++ .../11-memory-safety-asan-valgrind.md | 491 ++++++++++++++++++ ...2-sanitizer-toolchain-and-memory-safety.md | 341 ++++++++++++ documents/team/index.md | 9 + .../exercises/01-bounded-queue.md | 2 +- site/.vitepress/plugins/code-fold-plugin.ts | 4 +- .../theme/components/FontSizeSwitcher.vue | 4 +- 13 files changed, 1359 insertions(+), 11 deletions(-) create mode 100644 changelogs/v0.7.0.md create mode 100644 documents/en/vol6-performance/10-asan-family-and-memory-safety.md create mode 100644 documents/en/vol6-performance/11-memory-safety-asan-valgrind.md create mode 100644 documents/en/vol6-performance/12-sanitizer-toolchain-and-memory-safety.md diff --git a/.gitignore b/.gitignore index 0551cdc74..44e0eccc7 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,7 @@ build/ **/build/ *.build/ +**/_build_ci/ # VitePress node_modules/ diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 711976a79..94225c8f0 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -48,3 +48,9 @@ **贡献类型**：🎨 📝 **贡献渠道**：微信 **贡献说明**：指导文档站 UI 设计，提供阅读体验改进意见 + +### [Nyuudoukumo](https://github.com/Nyuudoukumo) + +**贡献类型**：🐛 💡 +**贡献渠道**：GitHub +**贡献说明**：反馈卷一「Cache 机制与内存层次」示例与解释不一致的问题（#67），促成量纲修正 diff --git a/README.md b/README.md index 9f3b7da59..288a3c9ea 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ --- -![English Coverage](https://img.shields.io/badge/en_coverage-99%25-green.svg) 491/494 docs translated +![English Coverage](https://img.shields.io/badge/en_coverage-100%25-green.svg) 494/494 docs translated ## 这是什么项目 @@ -103,12 +103,12 @@ cmake -S code/examples/chapter05/06_array_vs_stdarray -B build && cmake --build |----|------|:----:|--------| | 卷一 | 基础入门（含 C 速通） | 87 | ✅ 成型 | | 卷二 | 现代特性（RAII / 智能指针 / 移动 / lambda） | 44 | ✅ 成型 | -| 卷三 | 标准库深入 | 8 | 🔨 在建 | -| 卷四 | 高级主题（concepts / 协程 / 模板） | 8 | 🔨 在建 | +| 卷三 | 标准库深入 | 40 | ✅ 成型 | +| 卷四 | 高级主题（concepts / 协程 / 模板 / 设计模式） | 29 | 🔨 在建 | | 卷五 | 并发编程 | 44 | ✅ 成型 | -| 卷六 | 性能优化 | 3 | 🔨 在建 | +| 卷六 | 性能优化 | 6 | 🔨 在建 | | 卷七 | 工程实践（CMake / 工具链 / 调试） | 8 | 🔨 在建 | -| 卷八 | 领域应用（嵌入式 / 网络 / GUI / 存储） | 63 | ✅ 成型 | +| 卷八 | 领域应用（嵌入式 / GUI / 存储等） | 63 | ✅ 成型 | | 卷九 | 开源项目研读（Chromium 等） | 16 | 📚 持续更新 | | 卷十 | 课程与演讲笔记（CppCon 等） | 17 | 📚 持续更新 | diff --git a/changelogs/v0.7.0.md b/changelogs/v0.7.0.md new file mode 100644 index 000000000..6dc6bb819 --- /dev/null +++ b/changelogs/v0.7.0.md @@ -0,0 +1,80 @@ +# v0.7.0 (2026-06-25) + +内容主线里程碑:卷三「标准库」全面成型、卷四补齐 GoF 设计模式系列、卷六补齐内存安全与 sanitizer 三部曲、卷五并发 Lab 重设计为 TDD 工程;平台层落地代码折叠与字号开关,部署链路切换为 GitHub Pages artifact 方式;英文翻译覆盖率达 100%。 + +## 卷三:标准库全面成型 + +按 v0.6.0 路线图「012 卷三全面重建」收尾。本轮把容器以外的标准库全部写完,并把扁平结构重组为 6 个子目录,每篇均带本机 GCC 16.1.1 真实终端输出,零 stub: + +- **containers/(13 篇)** — v0.6.0 里程碑内容,本轮仅作目录迁移,确认完整无损坏。 +- **iterators-algorithms/(7 篇 40–46)** — 迭代器适配器、STL 算法总览(上/下)、数值算法、ranges 与 C++23 新件(`fold`/`contains`/新适配器)、并行算法(`-ltbb`、`par` / `par_unseq` 编译命令)。 +- **strings/(6 篇)** — `char8_t` 与 UTF-8、`string_view`、`charconv`、`format`、`print`、`regex`(按 C++20/23 标注支持度)。 +- **error-utils/(8 篇 61–68)** — `optional`、`variant`、`any`、`expected`(monadic 链 + 与异常的性能实测,频繁失败路径快近两个数量级)、`functional`、`error_code`、`stacktrace`、`source_location`。 +- **io/(3 篇 55–57)** — `iostream`(为何慢、怎么用才不慢)、`fstream`(RAII + 二进制可移植性)、`filesystem`(真实遍历 `/usr/include` 21173 条目 benchmark,冷热页缓存差 ~20×,`directory_entry` 缓存 vs 自由函数重 stat 实测)。 +- **time-numeric/(3 篇 58–60)** — `chrono`(实测 `high_resolution_clock` 在 libstdc++ 是 `system_clock` 别名,纠正老资料说是 `steady_clock` 的说法;C++20 日历/时区/`parse`)、`cmath`(精度陷阱)、`random`(为什么别用 `rand`)。 + +v0.6.0 changelog 标记的卷三待办基本清零;唯一留作可选深化项的是独立的「libc++ 源码阅读」专题——但多篇 deep-dive(`vector`/`string`/`map`/`unordered_map`/`charconv`/`regex` 等)已内嵌实现级源码分析,故该专题文章的缺失不等于内容缺失。 + +## 卷四:GoF 设计模式系列 + +从笔者旧博客迁移 **21 篇 GoF 设计模式**([01-singleton … 21-mediator](documents/vol4-advanced/vol4-generics-patterns/)),一次性补齐 `vol4-generics-patterns` 子区。按创造型(4)/结构型(7)/行为型(10)分组,统一走「朴素写法 → 逐步改进 → 终端实测验证 → 决策表 → 小结」骨架,是现代 C++ 写法而非朴素 OOP: + +- **Visitor** 贴 `-O2` 汇编看 `std::visit` 判别式分派 + 500 万次虚函数 vs variant 实测,带诚实的「variant 并不总是更快」caveat。 +- **Singleton** 用 500 线程实测证 C++11 magic-statics 线程安全,对比 DCLP 的 acquire/release。 +- **Observer** 用 `weak_ptr` 防悬挂 + ASan 复现 use-after-free。 + +21 篇均有配套 CMake 工程([`code/volumn_codes/vol4/design-patterns/`](code/volumn_codes/vol4/design-patterns/)),中英双语对齐。index 另标「规划中」的模板级模式(Policy-Based Design / CRTP / type erasure / Mixin 等)作为未来路线。 + +## 卷六:内存安全与 sanitizer + +新增内存安全三部曲(#80,迁移自作者 `*sans` 笔记): + +- **ASan 家族**(compile-time instrumentation / 1:8 影子内存 / quarantine)+ Heartbleed(CVE-2014-0160)越界读框架;五 sanitizer(ASan/LSan/MSan/TSan/UBSan)对比,ASan↔TSan 互斥用真实编译错误演示。 +- **Valgrind vs ASan**——DBT/JIT(20–50× 慢、免重编译、memcheck A-bit/V-bit 双影子表)vs CTI(~2×、须重编译、影子内存)两条路线对比;五工具分解(memcheck/callgrind/cachegrind/helgrind+drd/massif),六种错误(uninit/UAF/OOB/leak/malloc-delete 错配/double-free)带 GCC 16.1.1 + valgrind 3.25.1 真实输出。 +- **sanitizer 工具链全景**——用户态 `-fsanitize=*` ↔ 内核 KASAN/KMSAN/UBSAN/KCSAN/KFENCE/DAMON(含合并版本),三层防御(开发期全插桩 / 预发布采样 / 生产轻量)。 + + +## 卷五:并发 Lab 重设计 + +并发 Lab(#70)重设计为 TDD 工程:新增 [`code/volumn_codes/vol5-labs/`](code/volumn_codes/vol5-labs/)(standalone FetchContent + Catch2 v3.7.1),`templates/`(空骨架,学习者拷贝去做)/ `examples/`(参考实现)双轨,每个 lab 独立可构建。 + +- **Lab 0(线程生命周期)** 完整:MS1–4 渐进测试 + TSan 友好的并发文件扫描器参考实现(递归收集 → 分片 → `JoiningThread` worker → 归约)。 +- **Lab 1(阻塞队列 / 并发缓存 / 同步原语)** 模板与 MS1–6 测试就绪;参考实现标注「⚠️ 正在开发中」,目前只有空声明,完成后作为对照。 + +两份 lab 手册(`exercises/00-thread-lifecycle.md`、`01-bounded-queue.md`)大改重写 + +## 卷八:领域归属整理 + +`networking` 子区内容下线(挪至 `.claude/drafts/networking/` 待重写,gitignored),卷八回归嵌入式 / GUI / 存储 / 算法 / C++ 深入五个已成型子领域(中英 index 与全仓引用同步清理,零死链)。 + +## 平台与站点 + +- **长代码折叠**(#72,对应 issue #71):构建期 markdown-it 插件,把超阈值行数的代码块整段包进原生 `
`——零 FOUC(静态即折叠态)、无 JS 原生可展开、打印友好;`code-group` 内子块豁免。代码放在 `
` 外、纯 CSS `:has(details[open])` 控展开。 +- **字号开关**(#69):正文五档缩放(超小 / 小 / 正常 / 大 / 超大),桌面 + 移动端导航均有,A-/A+ 步进,localStorage 持久化,首屏由 `` 内联脚本提前应用防闪烁。 +- **部署链路优化**(#78):`gh-pages` 分支部署 → `actions/deploy-pages` artifact 部署。根治了 search 索引 chunk 每次构建换 hash 被 commit 进 gh-pages、致仓库膨胀至 437MB(86% 是 `.git`)的问题;main 分支保持干净,站点已上线。 +- **构建/校验基础设施**: + - `tags.py` 抽为 `VALID_TAGS` 单一数据源(`validate_frontmatter.py` + `check_quality.py` 共用),根治 vol4 设计模式迁移时两套白名单分叉导致 CI 挂的坑;本轮补全 GoF 设计模式 + 内存安全 + 调试/测试等新标签。 + - 新增 `build_examples.py`:host / STM32 分离 + CTest 支持,CI 接入,发现 113 个 host 工程。 + - `translate.py` 两处加固:`normalize_translated_title` 剥离 LLM 重新加回的 `<>`(v-html 吃裸尖括号)+ `yaml.dump width` 防长标题折行被侧边栏截断。 + +## 内容修复 + +- **卷一 Cache 步长实验量纲错误**(总耗时 → 每次访问耗时),#68 —— 源自 [@Nyuudoukumo](https://github.com/Nyuudoukumo) 的反馈 issue #67。 +- **`std::inplace_vector` 代码示例改进**,#73,@YukunJ。 +- **`std::is_trivially_copyable_v` 代码示例错误表述修正**,#75,@YukunJ。 +- **`std::pmr::vector` 仅用栈缓冲、不落堆的实测佐证**,#77,@YukunJ。 + +## 翻译 + +- 英文全量对齐;卷六内存安全 3 篇补译,英文覆盖率 **99% → 100%(494/494)**,所有卷全部双语对齐。 +- README 状态表全量刷新:卷三 8→40 篇(翻为「✅ 成型」)、卷四 8→29 篇(标注含设计模式)、卷六 3→6 篇、卷八描述去掉已下线的「网络」;coverage 徽章由 `coverage.py --update` 自动更新为 100%。 + +## 贡献者 + +Charliechen114514、YukunJ(代码与内容); + +感谢反馈者 **[@Nyuudoukumo](https://github.com/Nyuudoukumo)**(issue #67 卷一 Cache 内容勘误,促成 #68 量纲修正)。已同步进 CONTRIBUTORS.md 与中英 team 页。 + +本版合并 PR #68–#80（#71/#74 为 issue，未计入） + +贡献者 Charliechen114514、YukunJ，反馈者 Nyuudoukumo。 diff --git a/documents/en/team/index.md b/documents/en/team/index.md index fde65c809..e7dc2fbad 100644 --- a/documents/en/team/index.md +++ b/documents/en/team/index.md @@ -98,6 +98,15 @@ The valuable feedback from these folks is equally worth recording. It is your fe

Reported compilation failure in the expected example

+ + Nyuudoukumo +
+ Nyuudoukumo Issue +

Bug Report · Content Errata

+

🐛 💡

+

Reported an inconsistency between the example and explanation in Vol.1 "Cache and Memory Hierarchy" (#67), leading to a units fix

+
+ ## How to Become a Contributor diff --git a/documents/en/vol5-concurrency/exercises/01-bounded-queue.md b/documents/en/vol5-concurrency/exercises/01-bounded-queue.md index ed8df5a13..e0a523969 100644 --- a/documents/en/vol5-concurrency/exercises/01-bounded-queue.md +++ b/documents/en/vol5-concurrency/exercises/01-bounded-queue.md @@ -55,7 +55,7 @@ Upon completion, you should have muscle memory for the mutex + condition_variabl ## Project Scaffold (Get this running first) -Each Lab has two versions under `vol5-labs/`: **`templates/lab1_bounded_queue/`** is the empty implementation skeleton (copy this one to work on), and **`examples/lab1_bounded_queue/`** is the reference implementation (use this to compare if you get stuck, but don't copy it upfront). Both are standalone projects. You will be working on the `templates` version: +Each Lab has two versions under `vol5-labs/`: **`templates/lab1_bounded_queue/`** is the empty implementation skeleton (copy this one to work on), and **`examples/lab1_bounded_queue/`** is the reference implementation (⚠️ under development — currently only empty declarations, no complete reference yet; once complete, use it to compare if you get stuck, but don't copy it upfront). Both are standalone projects. You will be working on the `templates` version: ```text templates/lab1_bounded_queue/ diff --git a/documents/en/vol6-performance/10-asan-family-and-memory-safety.md b/documents/en/vol6-performance/10-asan-family-and-memory-safety.md new file mode 100644 index 000000000..be84374a1 --- /dev/null +++ b/documents/en/vol6-performance/10-asan-family-and-memory-safety.md @@ -0,0 +1,411 @@ +--- +title: 'ASan Tool Family and Memory Safety: Shadow Memory, Heartbleed, and Sanitizer Selection' +description: 'Starting with Heartbleed, we break down the shadow memory trio of AddressSanitizer, test OOB, UAF, global out-of-bounds, and UBSan, and clarify the responsibilities and mutual exclusivity of the five brothers: ASan, LSan, MSan, TSan, and UBSan.' +chapter: 6 +order: 10 +platform: host +difficulty: advanced +cpp_standard: +- 11 +- 14 +- 17 +- 20 +reading_time_minutes: 22 +prerequisites: +- 动态内存管理(new/delete 与智能指针) +- 并发程序调试技巧(ThreadSanitizer) +related: +- 动态内存管理(new/delete 与智能指针) +- 并发程序调试技巧(ThreadSanitizer) +- C 语言动态内存管理(malloc/free 与 valgrind) +tags: +- host +- cpp-modern +- advanced +- 内存安全 +- 调试 +- 内存管理 +translation: + source: documents/vol6-performance/10-asan-family-and-memory-safety.md + source_hash: 8062233844a7990d1e7112dd028dffa5f17ee3f2c46a0f517645d984738e3f8b + translated_at: '2026-06-25T13:28:09.799801+00:00' + engine: anthropic + token_count: 4210 +--- +# The ASan Tool Family and Memory Safety: Shadow Memory, Heartbleed, and Sanitizer Selection + +> PS: This content was migrated from notes I took during college. It has undergone limited verification through research. If you find any technical inaccuracies or outright errors, please report an issue or submit a pull request! + +After writing C/C++ for a while, you have likely been tortured by these classes of problems repeatedly: reading one element past the end of an array, using a pointer after it has been freed, or calling `delete` twice on the same pointer. These errors share a nasty characteristic—they are **undefined behavior** (the infamous UB). It's not that they "always crash," but rather "they run fine in a Debug build, then randomly explode in Release or on a different machine." Even worse, the crash location is often far removed from the actual buggy code, and the stack trace might point to some innocent library function. + +Why does this happen? Because these bugs corrupt the memory manager's own metadata, and the explosion only happens during the next `malloc`/`free` operation that touches that corrupted location. Previous volumes focused on performance; this time, we switch dimensions to another topic: how to use tools to catch bugs before they cause production incidents. The protagonists are AddressSanitizer (ASan) and the entire family of sanitizer tools behind it. + +Don't rush to treat ASan as just a "add a flag and you're done" utility. The design behind it—shadow memory and compile-time instrumentation—is actually one of the most significant engineering advancements in C/C++ memory safety over the last decade. Furthermore, it was originally invented to plug a hole that terrified the entire internet. We start with that hole. + +## The Origin: Heartbleed and Buffer Over-reads + +In April 2014, CVE-2014-0160 was disclosed under the name Heartbleed. This was a vulnerability hidden in a seemingly harmless feature of the OpenSSL implementation—the TLS heartbeat extension. The protocol is simple: the client sends arbitrary data and tells the server, "This data is N bytes long, please read it back verbatim," to verify the connection is still alive. + +The vulnerability lay in the fact that the server **trusted the length N reported by the client without verifying that N did not exceed the actual length of the data it held**. Consequently, an attacker simply needed to report a large N (for example, 64 KB), and the server would "read back" 64 KB from its own process memory to the attacker. What was read could be the TLS private keys of other sessions, user passwords, or session tokens—everything adjacent to that buffer in process memory would be leaked. + +The nature of this bug was not an out-of-bounds **write**, but an out-of-bounds **read** (buffer over-read). Write overflows corrupt data and are easily exposed; over-reads are much quieter, as the process itself does not crash, and data simply silently leaks out. ASan was frequently cited back then because it was one of the few tools capable of **reliably detecting over-reads**—as long as that out-of-bounds memory touched the redzones planted by ASan, a single read would trigger an immediate error. + +We will use a few dozen lines of Modern C++ to recreate a Heartbleed-shaped bug, and then have ASan catch it in the act. This is the flagship demo for this article, and we will refer to it repeatedly. + +```cpp +// oob_read.cpp —— 复刻 Heartbleed 形状的越界读 +// Platform: host Standard: C++20 +// 编译: g++ -std=c++20 -O1 -fsanitize=address -g oob_read.cpp -o oob_read +#include +#include +#include + +// 心跳回显:客户端说"还给我 n 字节"。服务器照办,但不校验 n 上界。 +std::string read_back(const std::array& buf, int n) +{ + return std::string(buf.data(), n); // n 可能远大于 8 +} + +int main() +{ + std::array buf{'H', 'i', '!', 0, 0, 0, 0, 0}; + // 只授权了 8 字节,却要求"读回" 64 字节 —— 经典 over-read + auto leaked = read_back(buf, 64); + std::printf("读到 %zu 字节: %.8s...\n", leaked.size(), leaked.c_str()); +} +``` + +If we compile and run this without ASan, the code will likely "appear to work correctly": the `std::string` constructor dutifully copies 64 bytes starting from `buf.data()`, reading the irrelevant bytes on the stack. The program won't crash. This is exactly what makes over-reads so dangerous. + +However, if we add `-fsanitize=address` and run it again, the situation changes drastically. Here is the output from the local GCC 16.1.1: + +```text +================================================================= +==37023==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x72175e1f0028 at pc 0x761760d29ac2 ... +READ of size 64 at 0x72175e1f0028 thread T0 + #0 0x... in memcpy (/usr/lib/libasan.so.8+0x129ac1) + ... + #6 0x... in read_back[abi:cxx11](std::array const&, int) oob_read.cpp:11 + #7 0x... in main oob_read.cpp:18 + ... + + This frame has 2 object(s): + [32, 40) 'buf' (line 16) + [64, 96) 'leaked' (line 18) <== Memory access at offset 40 partially underflows this variable +SUMMARY: AddressSanitizer: stack-buffer-overflow oob_read.cpp:11 in read_back +``` + +Pay attention to two details. First, the error type is `stack-buffer-overflow`, occurring on line 11 of `read_back`—specifically the `return std::string(buf.data(), n);` line. This precision in source location is exactly why compiling with `-g` is essential. Second, ASan even tells us that there are two objects on the stack: `buf` occupies `[32, 40)` and `leaked` occupies `[64, 96)`. The out-of-bounds read location (offset 40) falls right between them. This level of forensic detail is what fundamentally distinguishes ASan from "adding an assertion and hunting slowly." + +## So, what exactly does ASan do to achieve this? + +### Part One: Compile-Time Instrumentation (CTI) + +ASan is not a post-mortem profiler; rather, it **rewrites your code at compile time**. When you add `-fsanitize=address`, the compiler (whether GCC or Clang) inserts extra check instructions before and after every memory access—every `*p`, every array subscript, every `memcpy`. This technique is called **compile-time instrumentation (CTI)**, also known as static instrumentation. + +Let's first verify that it actually "touches your code." Compile the `oob_read.cpp` from above without ASan, then again with ASan, and compare the **size of their code segments (.text)**—that is, the actual machine instructions packed into the binary: + +```text +普通构建 .text: 2792 字节 +ASan 构建 .text: 5736 字节 (+105%) +``` + +(On the local machine with GCC 16.1.1, `g++ -std=c++20 -O1 -g`, checking the `.text` section with `size`.) The doubled size represents the check instructions the compiler inserts before and after every memory access. Note a pitfall here: **don't compare the total binary file size**—ASan's runtime library, `libasan.so.8`, is **dynamically linked** (visible via `ldd`) and not baked into the executable, so the total file size only increases by about 5%. The `.text` code section, which truly reflects the amount of instrumentation, is where the doubling occurs. The cost is increased size and slower execution—but compared to the bugs it can catch, this overhead is negligible during development. CTI is determined at **compile time**, so you must include `-fsanitize=address` during **compilation**, and also during **linking**. If you add it only when compiling the main program but omit it when linking a third-party `.a` library, memory accesses inside that library won't be instrumented, and ASan will be blind to that code. The complete workflow is: + +```bash +g++ -std=c++20 -O1 -fsanitize=address -g -c a.cpp -o a.o # 编译带 +g++ -std=c++20 -O1 -fsanitize=address -g main.cpp a.o -o app # 链接也带 +``` + +The `-fsanitize=address` flag must be present in **both** the compilation and linking stages. If it is missing from either one, it will not work. + +### The Second Piece: Shadow Memory + +Instrumentation alone is not enough. The check code inserted by instrumentation needs a "ledger" to answer the question: "Is this address accessible right now?" This ledger is the **shadow memory**. + +The core idea is an elegant design: **use one byte of shadow memory to record the accessibility state of eight bytes of actual memory**. In other words, AddressSanitizer maps the entire process address space to a contiguous shadow region in 8-byte chunks, with a ratio of 1:8. This way, checking if an address is valid simply involves calculating its corresponding shadow byte and reading it; there is no need to maintain complex hash tables. + +AddressSanitizer prints the values of the shadow bytes at the end of its report. Let's look at the legend in the actual output: + +```text +Shadow byte legend (one shadow byte represents 8 application bytes): + Addressable: 00 + Partially addressable: 01 02 03 04 05 06 07 + Heap left redzone: fa + Freed heap region: fd + Stack left redzone: f1 + Stack mid redzone: f2 +``` + +This covers the complete semantics of the 1:8 mapping. `00` indicates that all eight bytes are accessible; `01` through `07` indicate that only the first few bytes are valid (for example, `03` means the first three bytes are accessible and the last five are not, used for partially accessible regions at the end of alignments); `fa` is the redzone around heap allocations—ASan secretly surrounds every block of memory you `new` with a "no-entry zone", so if you read `fa`, you have a heap buffer overflow; `fd` is memory that has already been `free`'d, so touching it is a use-after-free; `f1`/`f2` are redzones for stack objects. + +Looking back at the shadow dump in the error report above: + +```text +=>0x72175e1f0000: f1 f1 f1 f1 00[f2]f2 f2 00 00 00 00 f3 f3 f3 f3 +``` + +`00` represents the body of `buf` (8 bytes, plus 1 shadow byte). The immediately following `[f2]` is the mid redzone between stack objects. The address of our out-of-bounds read landed exactly on this `f2`—which ASan spotted immediately. This is why the shadow memory mechanism can achieve byte-level precision. + +### The Third Piece: Runtime Library + Quarantine + +Instrumentation and shadow regions alone aren't enough; someone needs to **keep the ledger**. The ASan runtime library (`libasan`) completely replaces functions like `new`/`delete` and `malloc`/`free` with its own versions. Every time memory is allocated, the runtime paints redzones in the shadow region for it; every time it is freed, the corresponding shadow memory is marked as `fd`. + +A key design here is called **quarantine**. When memory is `free`'d, ASan doesn't return it to the system for reallocation immediately. Instead, it tosses it into a quarantine queue to "cool off." Why? Because for bugs like use-after-free, if you `free` memory and immediately reallocate it to someone else, the shadow state of that block resets to `00`, and subsequent erroneous reads won't be caught. By quarantining it for a while, we ensure that the "freed" state persists long enough to collide with any subsequent invalid accesses. + +However, quarantine isn't unlimited—the queue has a cap. Once full, the oldest freed memory is recycled via FIFO. So, ASan's detection of use-after-free isn't 100%—if the quarantine window has passed and the memory has been reallocated, that specific invalid read might slip through. But with sufficient test coverage, the vast majority of UAFs will be caught. + +### The Cost: 2-4x Overhead, and Why It's Worth It + +With this three-piece combo, ASan typically incurs **2-4x runtime slowdown and 3-5x memory overhead** (shadow memory takes 1/8, plus redzones and quarantine). That sounds like a lot, but it depends on who you compare it to. + +Traditional memory checking tools like Valgrind (Memcheck) use **Dynamic Binary Instrumentation** (DBI)—they don't recompile your program. Instead, at runtime, they translate every machine instruction into their own intermediate representation, analyzing and executing them one by one. High precision and no recompilation needed, but the cost is 20-50x slowdown. A test that takes 1 second normally might take half a minute with Valgrind, making it often impossible to include in daily CI. + +ASan shifts the analysis cost **upfront to compile time** (CTI). At runtime, it only performs table lookups, keeping the overhead down to 2-4x. This magnitude means you can **permanently** enable ASan in development and CI to run full test suites, rather than remembering to manually run Valgrind occasionally. This is ASan's fundamental advantage over Valgrind—it's not that it's more accurate, but that it's **affordable**. + +::: warning No Valgrind Installed Locally +All ASan/UBSan outputs in this article were run locally on real hardware (GCC 16.1.1 / Clang 22, WSL2). Valgrind is not installed in the local environment (`which valgrind` → not found), so actual Valgrind output is not included here. Students who need Valgrind can run `apt install valgrind` on Debian/Ubuntu. For usage, see the Valgrind section in vol1's [C Dynamic Memory Management](../vol1-fundamentals/c_tutorials/14-dynamic-memory.md). Remember the essential difference between the two commands: **ASan is compile-time `-fsanitize=address` (CTI), while Valgrind is runtime `valgrind ./prog` (DBI)**. +::: + +## The Tool Family: Five Sanitizers, Each for a Category + +ASan is actually just one member of a family. This set of tools was originally implemented by Google engineers as patches for GCC and Clang, later becoming standard in mainstream compilers. The family has five members, each watching for a specific class of errors: + +| Tool | Flag | What it Catches | Typical Overhead | +|------|------|-----------------|------------------| +| **ASan** (AddressSanitizer) | `-fsanitize=address` | OOB read/write, use-after-free, double-free, stack/global overflow | 2-4x slowdown | +| **LSan** (LeakSanitizer) | `-fsanitize=leak` | Memory leaks (heap memory not freed at exit) | Near zero overhead | +| **MSan** (MemorySanitizer) | `-fsanitize=memory` | Reading uninitialized memory (use of uninitialized value) | ~3x slowdown | +| **TSan** (ThreadSanitizer) | `-fsanitize=thread` | Data races, deadlocks | 5-15x slowdown | +| **UBSan** (UndefinedBehaviorSanitizer) | `-fsanitize=undefined` | Undefined behavior (signed overflow, null pointer dereference, out-of-bounds shift, etc.) | Configurable, most sub-checks have low overhead | + +Among the five, ASan is the workhorse and almost essential for daily development. LSan is enabled by default with ASan (in GCC/Clang on supported environments). MSan is only fully available on Clang and requires **the entire program** to be compiled as MSan (even libc needs a MSan version, or false positives will be rampant). TSan specifically monitors concurrency; we covered it in detail in vol5's [Debugging Concurrency](../vol5-concurrency/ch08-debug-testing-perf/01-debugging-concurrency.md). UBSan acts as a "finisher," with low overhead that allows it to be combined with others. + +### ASan and TSan are Mutually Exclusive: An Iron Rule + +These five tools cannot be combined arbitrarily. The most important constraint is: **ASan and TSan cannot be enabled simultaneously**. ASan needs its own shadow memory layout, and TSan needs its own; the two mechanisms will conflict. The compiler will reject you outright at compile time: + +```text +$ g++ -std=c++20 -fsanitize=address,thread -g conflict.cpp -o conflict +cc1plus: error: '-fsanitize=thread' is incompatible with '-fsanitize=address' +``` + +The error message is straightforward. The engineering consequence of this constraint is that within a project's CI, memory error detection and data race detection require **two separate builds**—one with ASan and one with TSan—each running the test suite independently. Volume 5's chapter on TSan covered this "dual build" practice in detail, so we will just focus on the conclusion here. + +As for MSan, it is incompatible with both ASan and TSan (it requires all code to be "clean" and run through its own uninitialized memory tracking), and it only supports Clang. Consequently, it is the least used in real-world projects. LSan and UBSan are the two "versatile" options—LSan has almost zero overhead and can be a permanent fixture, while most of UBSan's sub-options can also be enabled alongside ASan. + +## In Practice: ASan Catches Three Typical Errors + +Just talking about principles isn't satisfying. Let's write code for the three types of memory errors that最容易 trip us up in C++, and let ASan catch them one by one. The following three sections are based on actual local runs. + +### Heap Use-After-Free + +Smart pointers can prevent most use-after-free (UAF) issues, but as long as a project still contains raw pointers or C-style APIs, this hole cannot be completely plugged. Here is a minimal example—we release a `unique_ptr`, but then hold onto the raw pointer it previously returned to read data: + +```cpp +// ... code block would follow ... +``` + +```cpp +// uaf.cpp —— use-after-free +// Platform: host Standard: C++20 +// 编译: g++ -std=c++20 -O1 -fsanitize=address -g uaf.cpp -o uaf +#include +#include + +int main() +{ + auto p = std::make_unique(42); + int* raw = p.get(); // 拿到裸指针 + p.reset(); // 这里释放 —— raw 立刻变成悬空指针 + std::printf("悬空指针读到的值: %d\n", *raw); // use-after-free +} +``` + +Run with ASan: + +```text +================================================================= +==37082==ERROR: AddressSanitizer: heap-use-after-free on address 0x7a948abe0010 ... +READ of size 4 at 0x7a948abe0010 thread T0 + #0 0x... in main uaf.cpp:12 + +0x7a948abe0010 is located 0 bytes inside of 4-byte region [0x7a948abe0010,0x7a948abe0014) +freed by thread T0 here: + #0 0x... in operator delete(void*, unsigned long) (/usr/lib/libasan.so.8+0x12e4c1) + ... + #4 0x... in main uaf.cpp:11 + +previously allocated by thread T0 here: + #0 0x... in operator new(unsigned long) (/usr/lib/libasan.so.8+0x12d341) + ... + #2 0x... in main uaf.cpp:9 + +SUMMARY: AddressSanitizer: heap-use-after-free uaf.cpp:12 in main +``` + +This report is the most valuable part of ASan. It doesn't just tell you "the read on line 12 is a use-after-free"; it simultaneously presents **two phases of this memory's history**: allocated by `make_unique` at `uaf.cpp:9` and freed by `reset` at `uaf.cpp:11`. Looking at these two lines, the causal chain of the bug becomes complete—this is exactly the value of the quarantine and redzone mechanisms: freed memory is marked as `fd` instead of being immediately reclaimed, so subsequent erroneous reads can collide with it. + +That `[fd]` in the shadow dump is the smoking gun: + +```text +=>0x7a948abe0000: fa fa[fd]fa fa fa fa fa ... +``` + +`fd` = freed heap region. This is the result of ASan's "bookkeeping" capabilities. + +### Global Buffer Overflow + +Global and static variables are also protected by redzones. ASan will reliably detect out-of-bounds access to global arrays as well: + +```cpp +// global_oob.cpp —— 全局数组越界 +// 编译: g++ -std=c++20 -O1 -fsanitize=address -g global_oob.cpp -o global_oob +#include +int g[4] = {1, 2, 3, 4}; +int main() { std::printf("g[5] = %d\n", g[5]); } +``` + +```text +==38356==ERROR: AddressSanitizer: global-buffer-overflow on address 0x63ca65acd074 ... +SUMMARY: AddressSanitizer: global-buffer-overflow global_oob.cpp:5 in main +``` + +The error type is explicitly marked as `global-buffer-overflow`. ASan uses different redzone encodings to distinguish between stack, heap, and global regions (`f1`/`f2` for stack, `fa` for heap, `f9` for global), so we can see at a glance which type of memory the out-of-bounds access occurred on. + +::: warning Regarding the claim that "Global OOB requires Clang 11" +Some older resources might claim that "ASan requires Clang 11 or higher to detect global variable out-of-bounds access." The historical context for this is that early ASan support for global variable redzones was incomplete; Clang 11 introduced improvements like the ODR indicator (`-fsanitize-address-use-odr-indicator`) to solidify global detection. However, **today**—with GCC 8.3+ and current Clang versions—global out-of-bounds detection is enabled by default and works out of the box. The example above was caught immediately with default settings on the local GCC 16.1.1. Therefore, this "version threshold" is obsolete for modern toolchains, so don't be misled by outdated documentation. +::: + +### Leaks: LSan Wraps Up at Exit + +Finally, let's look at memory leaks. LSan operates differently than the previous sanitizers—it doesn't report errors during execution. Instead, when `main` returns and the program is about to exit, it scans all "live" heap allocations and flags those that are no longer referenced or have not been freed. Let's write a minimal example that intentionally leaks: + +```cpp +// leak.cpp —— 故意泄漏 +// Platform: host Standard: C++20 +// 编译: g++ -std=c++20 -O1 -fsanitize=address -g leak.cpp -o leak +#include +#include +int main() +{ + int* p = (int*)std::malloc(sizeof(int) * 4); // 拿了堆内存 + p[0] = 42; + std::printf("ptr = %p\n", (void*)p); // 让指针逃逸,防止整段被优化器删掉 + // 没有 free,程序退出时 p 指向的内存泄漏 +} +``` + +Running with ASan (GCC 16.1.1 / WSL2, LSan is enabled by default with ASan): + +```text +ptr = 0x730c4cbe0010 +================================================================= +==364484==ERROR: LeakSanitizer: detected memory leaks + +Direct leak of 16 byte(s) in 1 object(s) allocated from: + #0 0x... in malloc (/usr/lib/libasan.so.8+0x12c161) + #1 0x... in main leak.cpp:8 + ... + +SUMMARY: AddressSanitizer: 16 byte(s) leaked in 1 allocation(s). +``` + +Note that the report is printed **after** `main` returns. This is how LSan's "cleanup on exit" mechanism works. Volume 1's [Dynamic Memory Management](../vol1-fundamentals/ch12/02-new-delete.md) also provided an equivalent example for comparison. + +::: warning The "Silent Exit" Trap of LSan +In mainstream Linux environments (GCC 16.1.1 / Clang 22), LSan is enabled by default alongside ASan, and the example above can be caught reliably on a local machine. However, be wary of a real-world pitfall: **leaks are only scanned when the process exits normally**. If your program is killed by `SIGKILL`, calls `_exit` to bypass `atexit` hooks, or runs in certain containers/sandboxes where LSan's exit hooks don't execute, the leak report will **silently vanish**. The program appears to have "no errors," but in reality, LSan never had a chance to scan. + +Troubleshooting steps: Confirm the process exits via a normal `return`. If necessary, explicitly force it on with `ASAN_OPTIONS=detect_leaks=1`. Long-running services (which never exit) cannot use LSan's "cleanup on exit" model; you must switch to Valgrind Massif or heap sampling instead. Do not assume that "no report from LSan means no leaks." +::: + +## UBSan: Turning "Undefined Behavior" from Silent Failures into Errors + +Having covered the mainstays of the ASan family, let's look at UBSan, the finisher. C/C++ has a blood-pressure-raising feature: **Undefined Behavior (UB)**. The compiler's attitude toward UB is: "Since the standard doesn't specify what happens, I'll assume it doesn't happen and optimize freely." The consequence is that errors like signed integer overflow, out-of-bounds shifts, and null pointer dereferences often **appear to run perfectly fine**—until one day you turn on `-O2` or switch compiler versions. The optimizer, relying on the assumption that "this won't overflow," performs aggressive transformations, and the program suddenly produces outrageous results. + +UBSan's strategy is to insert a runtime check next to every operation that could produce UB. If UB actually occurs, it immediately prints a `runtime error: ...` report (by default, it does not abort the program, but this can be configured). The overhead is low, and many sub-features can coexist with ASan. + +Let's look at a minimal example that packs three classic types of UB into one: + +```cpp +// ubsan.cpp —— UBSan 捕获未定义行为 +// Platform: host Standard: C++20 +// 编译: g++ -std=c++20 -O1 -fsanitize=undefined -g ubsan.cpp -o ubsan +#include +#include + +int main() +{ + int arr[4]{1, 2, 3, 4}; + int idx = 10; + std::printf("越界下标 arr[10] = %d\n", arr[idx]); // 下标越界 + + int max = std::numeric_limits::max(); + std::printf("有符号溢出: %d\n", max + 1); // 有符号整数溢出 + + int shift = 32; + std::printf("左移 32 位: %d\n", 1 << shift); // 位移量 >= 位宽 +} +``` + +Run with UBSan: + +```text +ubsan.cpp:11:55: runtime error: index 10 out of bounds for type 'int [4]' +ubsan.cpp:11:16: runtime error: load of address 0x7ffe8a0525c8 with insufficient space for an object of type 'int' +ubsan.cpp:14:16: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int' +ubsan.cpp:17:42: runtime error: shift exponent 32 is too large for 32-bit type 'int' +``` + +All three UBs were caught, pinpointed exactly to `file:line:column`. The list of UBs covered by UBSan is extensive. Common ones include: + +- **Arithmetic**: Signed integer overflow/underflow, division by zero. +- **Shift**: Shift amount negative or greater than/equal to bit width, left shift changing the sign bit. +- **Memory/Pointer**: Null pointer dereference, misaligned memory access, object size mismatch (accessing via wrong pointer type). +- **Array**: Out-of-bounds subscripts (`-fsanitize=bounds`). This overlaps with ASan's OOB detection but focuses differently—ASan looks at redzones, while UBSan looks at array sizes known at compile time. + +The overhead of UBSan depends on which sub-options you enable. `-fsanitize=undefined` is a collection of default sub-options, most of which are lightweight. The truly expensive one is `-fsanitize=integer` (which treats unsigned overflow as an error, high overhead, many false positives, use with caution in production). Recommendation for daily use: enable `-fsanitize=undefined` alongside ASan. It has a low cost and high yield. + +## Selection: Which Tool to Use for a Memory Bug + +Now that all five siblings have been introduced. The question arises—when you are face-to-face with a weird bug, in what order should you select your tools? We locate them by "symptom": + +- **Crashes on run / Segfault / Intermittent crashes**: Start with ASan and run reproduction tests. OOB, UAF, and double-free are the three most common causes of segfaults; ASan catches them all. +- **Intermittent incorrect results / Weird values across functions**: Suspect UAF or data race. First, use ASan to rule out UAF. If ASan is silent, build a separate TSan version to check for data races (remember, the two are mutually exclusive and cannot be enabled simultaneously). +- **Calculated values are ridiculous / Behavior changes under `-O2`**: Almost certainly a UB lock; go straight to UBSan. +- **Reading "looks normal" garbage values / Behavior depends on uninitialized data**: MSan (Note: Clang only, requires full program compilation). +- **Process eats more and more memory / Suspected leak**: LSan (reports on exit), or for long-running services, use Valgrind massif / heap sampling. + +A sound engineering practice is to **have two builds resident in CI**—one set with `ASan+UBSan` and one with `TSan`, running on every commit. The overhead is acceptable (ASan+UBSan is in the 2-4x range), and it buys you the ability to catch the most expensive bugs—like "intermittent crashes after deployment"—before they leave the building. + +::: warning ASan Is Not a Silver Bullet +ASan is powerful, but it has unavoidable limitations that you must keep in mind. + +First, **it only catches paths that are "actually executed"**. CTI is runtime detection; if code isn't run, the check won't trigger. If your test coverage is insufficient and a certain OOB path is never triggered, ASan won't catch it—this is exactly why ASan should be paired with good test cases, or even fuzzing. Fuzzing is responsible for running out rare paths, and ASan is responsible for reporting the moment an error occurs on those paths. + +Second, **it only catches memory errors**. Logic errors (calculation mistakes), concurrency errors (data races), and integer overflow UB are not ASan's concern—the latter belongs to UBSan, the former to TSan. Don't expect one flag to solve all problems. + +Third, **do not enable in production**. 2-4x slowdown and extra memory are a disaster under production load. ASan/UBSan/TSan are tools for the **development/testing/CI stages**; these flags must be removed in release builds. + +Fourth, **it has false positive boundaries**. Certain custom stack unwinding mechanisms (`swapcontext`, `vfork`) can cause ASan's shadow region logic to fail and report false positives. The line `HINT: this may be a false positive if your program uses some custom stack unwind mechanism` in the report is a reminder of this. +::: + +## Summary + +Starting from the Heartbleed over-read vulnerability that terrified the world, we've dissected the ASan tool family in this article. Let's wrap up with a few key conclusions: + +- **The ASan Trio**: Compile-time instrumentation (CTI) rewrites every memory access; shadow memory uses a 1:8 shadow byte to record the accessibility state of every 8 bytes of application memory; a runtime library replaces `new`/`delete` and uses quarantine to isolate freed memory. Overhead is 2-4x slowdown, far lower than Valgrind's 20-50x, making it viable for permanent CI residence. +- **Shadow Memory Encoding**: `00` accessible, `01`-`07` partially accessible, `fa` heap redzone, `fd` freed, `f1`/`f2` stack redzone, `f9` global redzone. The shadow dump at the end of an ASan report is a visual representation of this encoding. +- **Tool Family**: ASan (OOB/UAF), LSan (leaks), MSan (uninitialized reads, Clang only), TSan (data races), UBSan (UB). **ASan and TSan are mutually exclusive**, so maintain separate builds in CI. +- **UBSan is the Finisher**: Low overhead, can coexist with ASan, turning "silent UBs" like signed overflow, shift out-of-bounds, and null pointer dereferences into explicit `runtime error`s. +- **Selection Mnemonic**: For segfaults, use ASan first. For intermittent errors, rule out UAF with ASan then check races with TSan. For ridiculous arithmetic, use UBSan. For memory growth, check LSan. Always disable in production environments. + +True memory safety isn't achieved by catching bugs with tools, but by guarding against them at the syntactic level using RAII, smart pointers, `std::span`, and range `for` loops—mechanisms that make it physically impossible to write OOB or dangling accesses. Those are the topics of vol1 and vol3. The value of the ASan toolset lies in the transition period—before you've replaced every raw pointer and wrapped third-party C libraries. It acts as the "last line of defense," forcing latent memory bugs to reveal themselves during development rather than detonating at 3 AM in production. + +## Reference Resources + +- [AddressSanitizer · google/sanitizers Wiki](https://github.com/google/sanitizers/wiki/AddressSanitizer) — Official ASan documentation, the authoritative source for shadow memory mechanisms, 1:8 mapping, and 2x overhead. +- [Clang: AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html) — Clang-specific ASan docs, including global detection evolution like `-fsanitize-address-use-odr-indicator`. +- [Clang: ThreadSanitizer](https://clang.llvm.org/docs/ThreadSanitizer.html) — TSan documentation, source for ASan↔TSan mutual exclusion (see vol5 Concurrency Debugging). +- [Clang: UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) — List of UBSan sub-options and overhead. +- [Valgrind User Manual](https://valgrind.org/docs/manual/manual.html) — DBI methodology and Memcheck/Helgrind, reference for the 20-50x overhead comparison. diff --git a/documents/en/vol6-performance/11-memory-safety-asan-valgrind.md b/documents/en/vol6-performance/11-memory-safety-asan-valgrind.md new file mode 100644 index 000000000..a3d8f5514 --- /dev/null +++ b/documents/en/vol6-performance/11-memory-safety-asan-valgrind.md @@ -0,0 +1,491 @@ +--- +title: 'Valgrind vs. ASan Comparison: JIT Interpretation vs. Compile-time Instrumentation' +description: Break down the responsibilities of the Valgrind suite (Memcheck, Callgrind, Cachegrind, Helgrind/DRD, and Massif), compile and run six classic memory errors with ASan, and thoroughly explain the essential differences between "dynamic binary translation" and "compile-time shadow memory instrumentation. +chapter: 6 +order: 11 +platform: host +difficulty: advanced +cpp_standard: +- 11 +- 14 +- 17 +- 20 +tags: +- host +- cpp-modern +- advanced +- 内存安全 +- 调试 +- 内存管理 +reading_time_minutes: 28 +prerequisites: +- 动态内存管理（new/delete 与智能指针） +- C 语言动态内存（malloc/free 与 valgrind 速览） +related: +- ASan 工具家族与内存安全：shadow memory、Heartbleed 与 sanitizer 选型 +- 并发程序调试技巧（TSan / Helgrind 深入） +- 动态内存管理 +translation: + source: documents/vol6-performance/11-memory-safety-asan-valgrind.md + source_hash: 6cfba1678566f8d5e146ab1d149ee1297c8f9e5a6c0ef64c81c72a73090a073e + translated_at: '2026-06-25T13:29:35.943066+00:00' + engine: anthropic + token_count: 5270 +--- +# Valgrind vs. ASan: JIT Interpretation vs. Compile-Time Instrumentation + +> PS: This content has been migrated from the author's university notes. Key conclusions have been verified on the local machine using GCC 16.1.1 + Valgrind 3.25.1; if there are any omissions, issues or PRs are welcome. + +Let's start with a scenario we've likely all encountered: a piece of C++ code runs perfectly fine locally, but crashes sporadically in production, or its memory RSS (Resident Set Size) climbs until it gets killed by the OOM Killer. You review the code, and the `new`/`delete` pairs look correct, and the out-of-bounds access is only off by a byte or two—you can't find the problem just by reading the code. For bugs like this, debugging by eye is hopeless; we must rely on tools to "see" every memory access. + +In this article, we categorize memory error detection tools into two camps based on their "implementation route" and break them down. One camp is **Valgrind**—the veteran solution that wraps a "virtual CPU" around your program to interpret execution via JIT (Just-In-Time compilation); the other is **AddressSanitizer (ASan)**—a compile-time solution that inserts check code into your program and uses "shadow memory" for bookkeeping. The original notes only covered Valgrind and didn't mention ASan, yet the latter is the more commonly used path in modern engineering. This article fills that gap and puts the two approaches side by side for comparison. + +## 1. Two Types of Memory Errors, and "Why Code Review Can't Find Them" + +Before using the tools, let's clarify the "enemies" we are hunting. Memory errors generally fall into two categories, and the difficulty of catching them differs significantly: + +**Type 1: Deterministic out-of-bounds / use-after-free / double-free.** The characteristic of these errors is "accessing an address that should not be accessed." They are dangerous but relatively easier to catch—as long as the tool can mark "which memory blocks are legal and which aren't," it can report the error the moment the boundary is crossed. `char buf[8]; buf[8] = 'x';` (off-by-one) and `free(p); return *p;` (dangling pointer) fall into this category. + +**Type 2: Uninitialized reads / memory leaks.** These are more insidious. Uninitialized reads mean "the address is legal, but the value is garbage"—the program doesn't crash, it just silently calculates the wrong result. Memory leaks mean "the address is always legal, just never returned"—the program doesn't crash, but RSS slowly increases. You cannot catch these with a "legal address table" alone; you need a different mechanism: Valgrind maintains a "is this value initialized" tag for every byte, while ASan's leak detector (LSan) scans the heap at program exit to see if there are blocks "allocated but not pointed to." + +The fundamental reason why code review fails is that both types of errors **depend on the runtime memory state**, not on the literal code syntax. Just looking at `*p`, you have no idea if the memory `p` points to is alive or dead, initialized or garbage. This is exactly why we need tools to "record" every allocation, every free, and every read/write—turning the runtime memory state into an audit-able ledger. + +Regarding this "recording" task, Valgrind and ASan take two completely different implementation routes. Let's put the conclusion first, then break it down later. + +| Dimension | Valgrind (memcheck) | AddressSanitizer | +|------|---------------------|------------------| +| How it records | Dynamic Binary Translation: Translates every machine instruction into a checked version at runtime | Compile-time instrumentation: Inserts check code before/after every memory access at compile time | +| Needs recompilation? | **No**, runs on existing binaries | **Yes**, must recompile with `-fsanitize=address` | +| Runtime overhead | 20~50x slower, 2x+ memory (official docs) | ~2x slower, ~3x memory | +| Platforms | Linux/macOS (FreeBSD/Solaris), x86/ARM, etc. | GCC/Clang/MSVC, all platforms including Windows | +| Catches uninitialized reads? | Yes, native (V-bit) | **No**, requires `-fsanitize=memory` (MSan, Clang only) | +| Catches stack overflow | Yes (with full `--tool=memcheck` suite) | Catches stack/global redzones by default; `detect_stack_use_after_return` catches stack-use-after-return | + +Keep this table in mind. Next, starting from the "source of the pain," let's look at how the Valgrind path works. + +## 2. Valgrind: Wrapping a "Virtual CPU" to JIT Interpret Your Program + +### 2.1 What is it actually doing? + +Valgrind is essentially a **dynamic binary translation (DBT) framework**. It isn't a normal checking library; it puts your entire program inside a "virtual CPU" to run. When you type `valgrind ./myprog`, what actually happens is: Valgrind intercepts every machine instruction of yours, **translates it on-the-fly** into a sequence of new instructions that "does the original work + records memory state," and only then executes it. So your program doesn't run directly on the CPU; it is "interpreted" inside Valgrind's core. + +This is the source of its famous side effect—**20 to 50 times slower**, memory usage more than doubled. The Valgrind official manual states: + +> Programs running under Valgrind run significantly more slowly, and use much more memory -- e.g. more than twice as much as normal under the Memcheck tool. + +In other words: a program that runs in 1 second might take half a minute under memcheck. So Valgrind isn't for hanging onto during daily development; it's for when "this program really has a memory bug, and I'm dedicating a chunk of time to hunt it down." + +This JIT interpretation architecture has a huge benefit, and it's the fundamental reason Valgrind hasn't been retired: **no recompilation needed**. You have a binary from ten years ago, where you can't even find all the source code, and you suspect it leaks—just type `valgrind ./old_binary` and it runs. ASan can't do this; ASan must recompile from source. This is the hardest distinction between the two routes. + +### 2.2 The Five-Piece Suite: One Framework, Five Tools + +The essence of Valgrind is "framework + tools." The core handles translation and scheduling, while the specific "what to record, what to report" is handed off to pluggable tools. Using `--tool=` selects which pair of "checking glasses" to wear. Let's look at the core tools listed in the manual: + +**Memcheck**—The memory error detector, Valgrind's default tool, and the one most people actually use when they say "using Valgrind to check memory." It catches the full set (from manual section 4.1): accessing memory that shouldn't be accessed (heap block overflow, stack top overflow, access after free), using uninitialized values, incorrect freeing (double-free, `malloc` with `delete` mismatches), `memcpy` source/destination overlap, passing "suspicious" negative sizes to allocation functions, passing 0 to `realloc`, alignment values that aren't powers of two, and memory leaks. In short: memcheck catches almost all common memory errors in C/C++ programs. + +**Callgrind**—Call graph + cache/branch prediction profiler. It doesn't require special compile-time options (though `-g` is recommended) and writes analysis data to a file at the end of the run, which is then converted to a human-readable format using `callgrind_annotate`. Use it to locate "how many times a function was called and what the call graph looks like." + +**Cachegrind**—Cache profiler. It simulates the CPU's I1/D1/L2 caches and precisely points out cache misses and hits in your program, telling you how many misses and instructions each line of code, function, or module generated. Use it when squeezing out cache performance. + +**Helgrind and DRD**—Both are **thread error detectors**, catching data races, inconsistent lock ordering, and POSIX threads API misuse. The source notes described Helgrind as "still experimental," but this statement is **long outdated**—in the 2026 official manual, both Helgrind and DRD are officially listed as stable tools with independent chapters (Chapters 7 and 8), not experimental features. Incidentally, the source notes only mentioned Helgrind and **missed DRD**—they share the same goal (catching thread bugs) but use different algorithms; DRD is usually faster and has better support for certain scenarios (like many small objects, Boost.Thread, OpenMP). I covered practical TSan/Helgrind usage in Volume 5's [Concurrency Debugging Techniques](../vol5-concurrency/ch08-debug-testing-perf/01-debugging-concurrency.md), so I won't repeat it here. Just remember: "for thread bugs, look to helgrind/drd or the more modern TSan." + +**Massif**—Heap profiler. Measures exactly how much heap memory your program consumes, giving you growth curves for heap blocks, management structures, and the stack. Use it to "slim down" a program or find the biggest RSS consumers. + +> **An easily overlooked division of labor**: memcheck catches "correctness" (can this memory be accessed, is it initialized), while callgrind/cachegrind/massif catch "speed/quantity" (performance and usage). Newcomers often confuse them, thinking Valgrind is just for memory leaks—actually, that's just one tool's job. The performance profiling tools (callgrind/cachegrind/massif) are in a completely different track from ASan; ASan doesn't touch performance profiling. + +### 2.3 Memcheck's Dual-Table Principle: A-bit and V-bit + +How does memcheck manage to catch so many types of memory errors? The key lies in the two "shadow tables" it maintains, covering the entire process address space. Manual section 4.5 explains this clearly: + +**Valid-Address Table (A-bit).** Every byte in the process address space corresponds to 1 bit, recording "whether this address is currently readable/writable." When memory is malloc'd, the A-bit marks those bytes as "valid"; when freed, the mark flips back to "invalid." When an instruction tries to read/write a byte, it checks the A-bit first—if invalid, it's an illegal access, and memcheck reports the error immediately. This layer catches: out-of-bounds, use-after-free, accessing unallocated areas. + +**Valid-Value Table (V-bit).** Every byte in the process address space corresponds to 8 bits; each CPU register also corresponds to a bit vector. They record "whether this value has been initialized." Memory from malloc starts with all V-bits as "uninitialized"; once an instruction writes a definite value to it, the corresponding byte's V-bit flips to "initialized." The key design is: **V-bits "propagate" with the value**—if you read an uninitialized value from memory into a register, the V-bit moves to the register too; if you use it in calculation, the result's V-bit is also "uninitialized." However, memcheck doesn't report as soon as an uninitialized value is read; it only reports when "that value is used to influence program output or generate an address." This delay is intentional—to avoiding flooding the screen with false positives. + +Looking at the two tables together: A-bit manages "is the address legal," V-bit manages "is the value clean." The former catches OOB/UAF, the latter catches uninitialized reads. Double-free and alloc-dealloc mismatches are caught by memcheck's own maintained ledger of "what allocator was used for this block." + +The cost of this "accounting for every byte" mechanism explains the memory doubling mentioned earlier—the A-bit and V-bit themselves take up space. + +## 3. ASan: Compile-Time Instrumentation + Shadow Memory + +### 3.1 The Approach is Completely Reversed + +ASan's implementation route is the exact opposite of Valgrind's. It **does not** wrap a virtual CPU around the program; instead, it **inserts check code into your program at compile time**. When you add `-fsanitize=address`, the compiler inserts a small piece of code before and after every memory read/write: this code checks a "shadow memory" table to determine if the access is legal, and if not, reports an error and aborts. + +So ASan's checking is "the program checking itself," rather than "an external virtual CPU checking for it." This explains the huge difference in overhead between the two routes: ASan only adds a few instructions on the instrumented memory accesses, avoiding the cost of "translating the entire instruction stream," so it is **only about 2 times slower** (Valgrind is 20~50x); the price is that it must be recompiled, and checking only covers instrumented code—dynamically loaded third-party .so files compiled without ASan are out of its scope (Valgrind can handle them, as it intercepts everything at the instruction level). + +### 3.2 Shadow Memory: 8 Bytes → 1 Byte Encoding + +The core mechanism of ASan is shadow memory (for a full breakdown of shadow memory, see this volume's [ASan Tool Family](./10-asan-family-and-memory-safety.md), which also covers how it plugged vulnerabilities like Heartbleed). It maps the entire process address space into a shadow table in groups of 8 bytes—every 8 application bytes correspond to 1 shadow byte. The value of that shadow byte has a specific meaning; I'll paste the diagram generated locally here (the output below is real): + +```text +Shadow byte legend (one shadow byte represents 8 application bytes): + Addressable: 00 + Partially addressable: 01 02 03 04 05 06 07 + Heap left redzone: fa + Freed heap region: fd + Stack left redzone: f1 + Stack mid redzone: f2 + Stack right redzone: f3 + Stack after return: f5 + Stack use after scope: f8 + Global redzone: f9 + Global init order: f6 + Poisoned by user: f7 + Container overflow: fc + Array cookie: ac + Intra object redzone: bb + ASan internal: fe + Left alloca redzone: ca + Right alloca redzone: cb +``` + +Let's translate the elegance of this encoding scheme: + +- Shadow byte is `00`: All 8 bytes are accessible; +- It is `01`~`07`: Only the first N bytes are accessible; the rest are out-of-bounds red zones—this is exactly how ASan catches off-by-one errors. It surrounds every heap block, stack frame, and global variable with a ring of "red zones". The shadow bytes for these red zones are marked `fa`, `f9`, etc. Once you step into a red zone, the instrumentation code checks the shadow byte, sees that it is not "accessible", and reports an error immediately; +- It is `fd`: This memory block has been freed—accessing it again is a use-after-free, and it gets caught on the spot. + +In other words, ASan does not account for address validity "byte-by-byte" like Memcheck does. Instead, it "lays out red zones around valid regions and uses them to define boundaries". This mechanism is extremely effective for out-of-bounds and UAF issues, but **it lacks V-bits**—so ASan cannot detect uninitialized reads. This gap must be filled by MSan (MemorySanitizer, `-fsanitize=memory`), and MSan is only implemented in Clang. **GCC up to version 16.1.1 does not support `-fsanitize=memory`** (tested locally, results in `unrecognized argument`). This is a genuine shortcoming of the ASan approach compared to Memcheck. + +> **Warning**: ASan and other sanitizers have a "one at a time" relationship. `-fsanitize=address` and `-fsanitize=thread` (TSan) **cannot be enabled simultaneously**—they make different assumptions about shadow memory layout, and mixing them will lead to direct errors or abnormal behavior. Therefore, enable ASan when hunting memory errors, and enable TSan separately when hunting concurrent data races. Don't expect to "catch everything in one go". For how to check threading errors, see [Volume 5: Concurrency Debugging](../vol5-concurrency/ch08-debug-testing-perf/01-debugging-concurrency.md). + +## IV. Let's Run It: Six Classic Errors and Real ASan Output + +Explaining the theory isn't satisfying enough. We took the six classic errors from the source notes—which were "all screenshots, no source code"—rewrote them entirely in real code, and compiled and ran them locally using `g++ -std=c++20 -O0 -g -fsanitize=address,undefined` on GCC 16.1.1. Every output snippet below was **actually generated by running the code**, not manually fabricated. + +First, let's wrap all six error types into a single program: + +```cpp +// cases.cpp — 六类经典内存错误，逐个用 ASan 复现 +// 编译: g++ -std=c++20 -O0 -g -fsanitize=address,undefined cases.cpp -o cases +// 运行: ./cases <1..6> 不传参则只跑内存泄漏 +#include +#include + +// 1. 使用未初始化内存（ASan 抓不到，要 MSan） +int case_uninit() { + int* p = (int*)malloc(sizeof(int)); // 内容是垃圾 + int v = *p; // 读到垃圾值，但地址合法 + free(p); + return v; +} + +// 2. use-after-free +int case_uaf() { + int* p = (int*)malloc(sizeof(int)); + *p = 42; + free(p); + return *p; // 读已释放内存 +} + +// 3. 堆缓冲区越界（尾部读写） +int case_oob() { + int* a = (int*)malloc(4 * sizeof(int)); // 只有 a[0..3] + a[4] = 99; // 第 5 个元素越界 + int r = a[4]; + free(a); + return r; +} + +// 4. 内存泄漏（忘记 free） +void case_leak() { + int* p = (int*)malloc(sizeof(int)); + *p = 7; // 故意不 free +} + +// 5. malloc 配 delete（分配/释放不匹配） +void case_mismatch() { + int* p = (int*)malloc(sizeof(int)); + *p = 5; + delete p; // malloc 该配 free +} + +// 6. 双重释放 +void case_double_free() { + int* p = (int*)malloc(sizeof(int)); + free(p); + free(p); // 第二次 free +} + +int main(int argc, char** argv) { + if (argc < 2) { case_leak(); puts("done: leak only"); return 0; } + switch (atoi(argv[1])) { + case 1: printf("uninit=%d\n", case_uninit()); break; + case 2: printf("uaf=%d\n", case_uaf()); break; + case 3: printf("oob=%d\n", case_oob()); break; + case 4: case_leak(); puts("done leak"); break; + case 5: case_mismatch(); puts("done mismatch"); break; + case 6: case_double_free(); puts("done double-free"); break; + default: puts("usage: ./cases [1..6]"); break; + } + return 0; +} +``` + +Remember this compilation command, as we will use it for every case below: `g++ -std=c++20 -O0 -g -fsanitize=address,undefined cases.cpp -o cases`. We use `-g` so that the ASan report includes line numbers, and `-O0` to prevent the compiler from optimizing away our out-of-bounds accesses (at high optimization levels, a "write-then-read" pattern like `a[4]` might be folded; ASan can still catch it, but `-O0` is cleanest for debugging). + +### 4.1 Using Uninitialized Memory — ASan's Blind Spot + +Let's run case 1 first to see how ASan reacts: + +```text +$ ./cases 1 +uninit=-1094795586 +``` + +**ASan stays silent**, and the program simply returns a garbage value (`-1094795586`). This highlights the limitation mentioned earlier: this memory address is valid (allocated via `malloc`), and ASan's shadow memory marks it as "accessible". Since it lacks the V-bit to determine "whether this value has been initialized", ASan misses this error. Memcheck can catch this (using the V-bit), but ASan cannot—to detect this, you would need MSan (`-fsanitize=memory`, Clang only). This represents a **substantive difference in capability** between the two approaches; neither is strictly "stronger"—they simply cover different domains. + +### 4.2 use-after-free —— The red zone catches it immediately + +Running case 2: + +```text +$ ./cases 2 +================================================================= +==44083==ERROR: AddressSanitizer: heap-use-after-free on address 0x799329de0010 ... +READ of size 4 at 0x799329de0010 thread T0 + #0 ... in case_uaf() /tmp/asand/cases.cpp:20 + #1 ... in main /tmp/asand/cases.cpp:56 + ... + +0x799329de0010 is located 0 bytes inside of 4-byte region [0x799329de0010,0x799329de0014) +freed by thread T0 here: + #0 ... in free ... + #1 ... in case_uaf() /tmp/asand/cases.cpp:19 + ... + +previously allocated by thread T0 here: + #0 ... in malloc ... + #1 ... in case_uaf() /tmp/asand/cases.cpp:17 + ... + +SUMMARY: AddressSanitizer: heap-use-after-free /tmp/asand/cases.cpp:20 in case_uaf() +``` + +(I omitted irrelevant lines like the build-id above, but kept all the key information.) As you can see, ASan provides three pieces of information: **where this illegal read occurred** (`return *p` at line 20 in `case_uaf()`), **where this memory was freed** (line 19), and **where it was originally malloc'd** (line 17). Combined, the entire causal chain of "allocate → free → use" becomes clear at a glance. This is the result of the redzone mechanism combined with "setting shadow bytes to `fd` after free"—to ASan, that memory block is no longer "accessible" after `free`, so touching it triggers a report. + +### 4.3 Heap Buffer Overflow — Tail Redzone + +Running case 3 (`a[4]` is out of bounds, as `a` was only allocated for four ints): + +```text +$ ./cases 3 +================================================================= +==44191==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7288a7be0020 ... +WRITE of size 4 at 0x7288a7be0020 thread T0 + #0 ... in case_oob() /tmp/asand/cases.cpp:26 + ... + +0x7288a7be0020 is located 0 bytes after 16-byte region [0x7288a7be0010,0x7288a7be0020) +allocated by thread T0 here: + #0 ... in malloc ... + #1 ... in case_oob() /tmp/asand/cases.cpp:25 + ... +``` + +`located 0 bytes after 16-byte region` — This memory region is 16 bytes (four `int`s), and the access point lands exactly on the **first byte immediately following its end**, which is the start of the trailing redzone. This is the principle behind how ASan catches off-by-one errors: a block returned by `malloc` is immediately followed by a ring of redzones. The shadow bytes for these redzones are `fa` (heap left redzone, though actually representing poisoned areas surrounding the heap block). Since `a[4]` falls into this redzone, the instrumentation checks the shadow byte, sees it is not `00`, and reports the error on the spot. + +> **A point mentioned in the source notes that is easily misunderstood**: The source notes state that "Valgrind does not check statically allocated arrays." While this was true for older versions of Memcheck (historically, out-of-bounds access on stack/global arrays was a weak point for Memcheck), **this is not the case for ASan**. ASan places redzones around stack arrays and global variables (shadow bytes `f1`~`f3` are stack redzones, `f9` is a global redzone), and it is very effective at catching out-of-bounds access on stack arrays. Therefore, the conclusion that "static array out-of-bounds access cannot be detected" applies only to Valgrind, not ASan. Do not conflate the limitations of these two tools. + +### 4.4 Memory Leaks — LSan Scans the Heap at Program Exit + +Running case 4 (`./cases 4`, intentionally omitting `free`): + +```text +$ ./cases 4 + +================================================================= +==44296==ERROR: LeakSanitizer: detected memory leaks + +Direct leak of 4 byte(s) in 1 object(s) allocated from: + #0 ... in malloc ... + #1 ... in case_leak() /tmp/asand/cases.cpp:34 + #2 ... in main /tmp/asand/cases.cpp:58 + ... + +SUMMARY: AddressSanitizer: 4 byte(s) leaked in 1 allocation(s). +``` + +Note that this error is reported by **`LeakSanitizer`**, not ASan itself. LSan is the leak detector bundled by default with ASan. It scans the entire heap **when the program exits normally** to identify blocks that were allocated but are no longer pointed to by any pointer. It reports "definitely lost" from the "still reachable / definitely lost" classification. This approach is consistent with Memcheck's leak detection strategy (both scan the heap at exit), except that LSan is part of the ASan toolchain. + +> **What about daemons?** By default, LSan only scans when the program calls `exit`. Long-running daemons or service processes do not exit on their own. In this case, you can send a signal to trigger an intermediate dump: use `ASAN_OPTIONS=abort_on_error=0:detect_leaks=1` with `kill`, or call the LSan `__lsan_do_leak_check()` API in your code to actively trigger a scan. The corresponding approach for Valgrind is to `kill` the memcheck process from another terminal to force output (a trick mentioned in the source notes). + +### 4.5 malloc with delete — Allocation/Deallocation Mismatch + +Running case 5: + +```text +$ ./cases 5 +================================================================= +==44300==ERROR: AddressSanitizer: alloc-dealloc-mismatch (malloc vs operator delete) ... + #0 ... in operator delete(void*, unsigned long) ... + #1 ... in case_mismatch() /tmp/asand/cases.cpp:42 + ... + +0x71a3249e0010 is located 0 bytes inside of 4-byte region [0x71a3249e0010,0x71a3249e0014) +allocated by thread T0 here: + #0 ... in malloc ... + #1 ... in case_mismatch() /tmp/asand/cases.cpp:40 + ... +``` + +`alloc-dealloc-mismatch (malloc vs operator delete)` — ASan records "who allocated it" for every allocation. When freeing, it checks the match; `malloc` paired with `delete` is a mismatch, triggering an immediate report. Memcheck detects the same category (manual section 4.2.5, "freed with an inappropriate deallocation function"), so their capabilities are aligned. + +> **Platform Note:** This `alloc-dealloc-mismatch` check is **disabled by default** on Windows (MSVC's ASan), because `delete` and `free` are often effectively equivalent on Windows. It is enabled by default on Linux/macOS. If you find these errors are not caught on Windows, check `ASAN_OPTIONS=alloc_dealloc_mismatch=1`. + +### 4.6 Double Free + +Run case 6: + +```text +$ ./cases 6 +================================================================= +==44193==ERROR: AddressSanitizer: attempting double-free on 0x6d0d527e0010 in thread T0: + #0 ... in free ... + #1 ... in case_double_free() /tmp/asand/cases.cpp:49 + ... + +0x6d0d527e0010 is located 0 bytes inside of 4-byte region [0x6d0d527e0010,0x6d0d527e0014) +freed by thread T0 here: + #0 ... in free ... + #1 ... in case_double_free() /tmp/asand/cases.cpp:48 + ... +``` + +`attempting double-free` — After the first `free`, the shadow bytes are set to `fd`. When we attempt to `free` the same address a second time, ASan detects that it is already in the `fd` state (freed) and immediately flags it as a double-free. It also helpfully informs us that "the previous free was on line 48". + +### 4.7 Extra Bonus: Use-After-Return on the Stack + +ASan can also catch something that has historically been difficult for memcheck to detect — **accessing a stack frame after it has returned** (the function has returned, but the caller still holds a pointer to a local variable inside that function). This feature must be explicitly enabled: + +```cpp +// suar2.cpp +#include +static int* g = nullptr; +void stash() { int local = 0xc0ffee; g = &local; } // 把局部变量地址存出去 +int main() { stash(); return *g; } // local 已随 stash 返回而消失 +``` + +```text +$ g++ -std=c++20 -O0 -g -fsanitize=address suar2.cpp -o suar2 +$ ASAN_OPTIONS=detect_stack_use_after_return=1 ./suar2 +================================================================= +==44702==ERROR: AddressSanitizer: stack-use-after-return on address 0x6da50b8f0020 ... +READ of size 4 at 0x6da50b8f0020 thread T0 + #0 ... in main /tmp/asand/suar2.cpp:4 + ... + +Address 0x6da50b8f0020 is located in stack of thread T0 at offset 32 in frame + #0 ... in stash() /tmp/asand/suar2.cpp:3 + + This frame has 1 object(s): + [32, 36) 'local' (line 3) <== Memory access at offset 32 is inside this variable +HINT: this may be a false positive if your program uses some custom stack unwind mechanism ... +SUMMARY: AddressSanitizer: stack-use-after-return /tmp/asand/suar2.cpp:4 in main +``` + +Note that address `0x6da50b8f0020` is located very **early** in the process address space (not in the normal stack region). This is because with `detect_stack_use_after_return` enabled, ASan moves "local variables that might be targeted by escaped pointers" to a dedicated "fake stack". When the function returns, that region of the fake stack is marked as poisoned, and any subsequent access triggers a `stack-use-after-return` error (shadow byte `f5`). This option is disabled by default due to some overhead and a small number of false positives (see that HINT). However, bugs involving "stack memory still in use after a function returns" are extremely difficult to track down, so it is worth knowing this trick exists. + +## 5. How to Use Valgrind: Feeding Those Errors to Memcheck + +Now that we have covered the principles, let's take that same `cases.cpp` from Section 4 (this time compiled without `-fsanitize=`, just a normal build) and run it under Valgrind. We will see how Memcheck reports the same batch of errors—comparing the two reporting styles side-by-side makes the differences clear. The local machine uses Valgrind 3.25.1. + +First, compile a clean version with `-g` (Valgrind doesn't need ASan's instrumentation, but it requires `-g` to report line numbers): + +```bash +g++ -std=c++20 -g -O0 cases.cpp -o cases_plain + +# 最常用：memcheck 全量查泄漏 +valgrind --tool=memcheck --leak-check=full ./cases_plain 4 + +# 更狠：连 still reachable 也列出来 + 跟进子进程 +valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all --trace-children=yes ./cases_plain +``` + +Here are a few key parameters: `--leak-check=full` performs a full leak check (providing line numbers), `--show-leak-kinds=all` lists even "still reachable" blocks (blocks that still have pointers pointing to them and can theoretically be freed) (the older `--show-reachable=yes` is an alias for this, which still works but is deprecated), and `--trace-children=yes` follows child processes spawned by `fork`/`exec`. To switch tools, modify `--tool=`: options include `callgrind`, `cachegrind`, `helgrind`, `drd`, and `massif`. + +### 5.1 Memcheck reports the same UAF like this + +Running case 2 (the use-after-free from Section 4): + +```text +$ valgrind --tool=memcheck --leak-check=full ./cases_plain 2 +==453796== Memcheck, a memory error detector +... +==453796== Invalid read of size 4 +==453796== at 0x40011E9: case_uaf() (cases.cpp:20) +==453796== by 0x4001377: main (cases.cpp:56) +==453796== Address 0x4ee9080 is 0 bytes inside a block of size 4 free'd +==453796== at 0x48529EF: free (vg_replace_malloc.c:989) +==453796== by 0x40011E4: case_uaf() (cases.cpp:19) +==453796== Block was alloc'd at +==453796== at 0x484F8A8: malloc (vg_replace_malloc.c:446) +==453796== by 0x40011CA: case_uaf() (cases.cpp:17) +uaf=42 +... +==453796== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) +``` + +Notice the line numbers—`cases.cpp:20` for the read, `:19` for free, and `:17` for malloc. These are **exactly the same** as those reported by ASan in Section 4 (ASan also reported :20/:19/:17). It is the same bug; both tools pinpoint the exact same lines, just with different wording: + +- ASan says `heap-use-after-free` + `located 0 bytes inside of 4-byte region`; +- memcheck says `Invalid read of size 4` + `Address ... is 0 bytes inside a block of size 4 free'd`. + +memcheck also adds `Block was alloc'd at ... :17`. It relies on the A-bit ledger to record the "lifetime" of this memory (where it was allocated, where it was freed, and where it is now being read), providing the full causal chain in one go. This shares the same logic as ASan's three-part "allocated by / freed by" structure, just with different phrasing. + +### 5.2 Leaks: LEAK SUMMARY vs. LSan + +Running case 4 (intentionally not calling free): + +```text +$ valgrind --tool=memcheck --leak-check=full ./cases_plain 4 +==453446== HEAP SUMMARY: +==453446== in use at exit: 4 bytes in 1 blocks +==453446== total heap usage: 3 allocs, 2 frees, 77,828 bytes allocated +==453446== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1 +==453446== at 0x484F8A8: malloc (vg_replace_malloc.c:446) +==453446== by 0x400123D: case_leak() (cases.cpp:34) +==453446== by 0x40013B5: main (cases.cpp:58) +==453446== LEAK SUMMARY: +==453446== definitely lost: 4 bytes in 1 blocks +==453446== indirectly lost: 0 bytes in 0 blocks +==453446== possibly lost: 0 bytes in 0 blocks +==453446== still reachable: 0 bytes in 0 blocks +==453446== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) +``` + +`definitely lost: 4 bytes` corresponds to the `Direct leak of 4 byte(s)` from LSan (LeakSanitizer) in the previous section. Both tools "scan the heap upon program exit," but Memcheck categorizes leaks into four levels (`definitely lost`, `indirectly lost`, `possibly lost`, `still reachable`), which is more granular, whereas LSan defaults to reporting only `Direct` and `Indirect` leaks. The line number is `:34`, consistent with ASan. + +> **Stop manually downloading and compiling source packages.** The installation process in the original notes was `tar -jxvf valgrind-3.12.0.tar.bz2 && ./configure && make && sudo make install`. Version `3.12.0` is from 2016—**ten years ago**. It has poor support for modern kernels and new CPU instructions (like newer AVX), and tends to throw errors when running newly compiled programs. Now, just use the distribution packages: `apt install valgrind` for Debian/Ubuntu, `dnf install valgrind` for Fedora/RHEL, or `pacman -S valgrind` for Arch. You'll get version 3.2x (3.25.1 on my machine). + +## 6. How to Choose Between the Two Paths + +After all this discussion, when should we actually use which tool? Here is a practical decision matrix: + +**Default to ASan.** For daily development and memory error detection in CI pipelines, ASan is the first choice. It is fast (2x slowdown vs. 20–50x, which CI can tolerate), cross-platform (works on Windows, macOS, and Linux, and is supported by MSVC), and provides clean reports. In modern C++ projects, `-fsanitize=address,undefined` is practically the standard configuration for debug builds. We cover ASan catching leaks in [Dynamic Memory Management](../vol1-fundamentals/ch12/02-new-delete.md) in Volume 1, and TSan for concurrency debugging in Volume 5—these are all tools in this ecosystem. + +**You must use Valgrind in these scenarios:** + +1. **You only have the binary, no source code**, or recompiling is too expensive (e.g., a massive legacy project). ASan requires recompilation, while Valgrind can run on existing binaries. +2. **You need to catch uninitialized reads but only have GCC.** ASan lacks V-bits, and MSan is Clang-only—for GCC projects needing to catch uninitialized reads, Memcheck is the ready solution. +3. **You need performance profiling** (callgrind/cachegrind/massif). ASan has no equivalent for these; if you want to see cache misses, heap growth curves, or call graphs, Valgrind is the only game in town. +4. **You need full coverage, including third-party libraries not compiled with ASan.** Valgrind intercepts at the instruction level, so it can catch memory errors even in `.so` files without source code; ASan only covers instrumented code. + +Conversely, **Valgrind can't do these, or does them poorly, so you need ASan**: catching stack/global buffer overflows (ASan's stack/global redzones are its strength), running fast (CI friendly), Windows platform (Valgrind has basically no Windows support), and catching `stack-use-after-return` (ASan has a dedicated fake stack mechanism). + +To sum it up in one sentence: **ASan is the standard for "development phase," while Valgrind is the specialist for "hard-to-diagnose issues, performance, and legacy binaries."** They aren't replacements; they are complementary—many teams run ASan in CI as the daily gatekeeper, and only bring in Valgrind for a second opinion when ASan misses a weird bug. + +## 7. Back to C++: Tools are a Safety Net, RAII is the Cure + +After talking about tools for so long, we must bring the conversation back to this point: **No matter how strong these tools are, they are "post-mortem bug catchers," not "bug eliminators."** What truly makes memory errors disappear from the root is C++'s RAII and smart pointers. + +Looking back at those six categories of errors, you will find they are **invariably built on "raw malloc/free and raw pointers"**: + +- **Leaks?** Use `std::unique_ptr` / `std::vector`. Objects are automatically released when they go out of scope, so there's no chance to forget `free`. +- **Use-after-free?** The ownership semantics of smart pointers make "whether this memory is still valid" a compile-time constraint. +- **Double-free?** `unique_ptr` cannot be copied and nulls the source after a move, making a double-free physically impossible. +- **Out-of-bounds?** `std::vector` with `.at()` throws exceptions, and `std::span` carries bounds—don't use raw `[]` with manual lengths. + +C-style `malloc`/`free` and raw pointers leave "when to release memory" and "who can access it" entirely to the programmer's memory—human brains inevitably fail at this, which is why we need "bookkeeping tools" like Valgrind and ASan as a safety net. The Modern C++ approach moves this bookkeeping **into the type system**: the resource lifecycle is bound to the object, and the compiler guarantees the release for you. This is the fundamental leap from "tools catching bugs" to "the language eliminating bugs," which is exactly what our [Dynamic Memory Management](../vol1-fundamentals/ch12/02-new-delete.md) chapter in Volume 1 is about. + +However, this **does not** mean Modern C++ projects don't need ASan/Valgrind. As long as your code calls C libraries, uses `new`/`delete`, or touches third-party interfaces lacking RAII wrappers, there is still room for memory errors to slip in. So the correct approach is: **First, use RAII to eliminate 99% of memory errors while coding. Then, use ASan to catch the remaining 1% during testing. Finally, use Valgrind as the bottom line for the weirdest, hardest problems.** Three lines of defense, each indispensable. diff --git a/documents/en/vol6-performance/12-sanitizer-toolchain-and-memory-safety.md b/documents/en/vol6-performance/12-sanitizer-toolchain-and-memory-safety.md new file mode 100644 index 000000000..189e8d227 --- /dev/null +++ b/documents/en/vol6-performance/12-sanitizer-toolchain-and-memory-safety.md @@ -0,0 +1,341 @@ +--- +title: 'Sanitizer Toolchain Overview: From -fsanitize to Kernel KASAN/KFENCE' +description: Comparing user-space `-fsanitize=address/memory/undefined/thread` with kernel-space KASAN/KMSAN/UBSAN/KCSAN/KFENCE side-by-side, we thoroughly explain the two approaches of "compile-time instrumentation vs. sampling" and the layered defense strategy of "debugging vs. production". +chapter: 6 +order: 12 +platform: host +difficulty: advanced +cpp_standard: +- 11 +- 14 +- 17 +- 20 +reading_time_minutes: 22 +prerequisites: +- ASan 工具家族与内存安全:shadow memory、Heartbleed 与 sanitizer 选型 +- Valgrind 与 ASan 对照:JIT 解释 vs 编译期插桩 +related: +- ASan 工具家族与内存安全:shadow memory、Heartbleed 与 sanitizer 选型 +- Valgrind 与 ASan 对照:JIT 解释 vs 编译期插桩 +- 并发程序调试技巧(ThreadSanitizer) +- 动态内存管理(new/delete 与智能指针) +tags: +- host +- cpp-modern +- advanced +- 内存安全 +- 调试 +- 工具链 +translation: + source: documents/vol6-performance/12-sanitizer-toolchain-and-memory-safety.md + source_hash: 6bb4d06235bfda3a14d218ee235d1291d222170d30d403fcd494c82997c838c2 + translated_at: '2026-06-25T13:30:42.897270+00:00' + engine: anthropic + token_count: 3783 +--- +# Sanitizer Toolchain Panorama: From `-fsanitize` to Kernel KASAN/KFENCE + +> PS: This content has been migrated from notes taken during my university years and has been verified and cross-checked; user-mode sanitizers were run locally, while kernel-mode tools could not be run locally and are based on official kernel.org documentation. If there are any omissions, issues or PRs are welcome. + +In the previous two articles, we dissected user-mode ASan / UBSan / MSan / TSan and Valgrind in great detail—how shadow memory keeps accounts, the differences between JIT interpretation and compile-time instrumentation, and why the five sanitizers are mutually exclusive. However, if you only focus on "just adding a flag with `g++ -fsanitize=address`", you will miss the bigger picture: **sanitizers are not exclusive to user mode; there is a full set of corresponding tools in the kernel**, and their design trade-offs are completely different. + +In this article, we will look at the entire sanitizer toolchain on a level playing field. On one side we have user-mode `-fsanitize=*`, and on the other, kernel-mode `CONFIG_KASAN / CONFIG_KMSAN / CONFIG_KFENCE`—they catch the same class of bugs (out-of-bounds, use-after-free, uninitialized, data races), but under completely different constraints: user mode can slow down a program by 2-5x to catch bugs, but the kernel cannot. If the kernel slows down by 5x, the whole machine is effectively dead. Therefore, the kernel has evolved the path of "sampling"—KFENCE uses extremely low overhead to enable "always-on in production," coexisting in a layered fashion with heavy tools like KASAN that are "only enabled during debugging." + +## Closing the Loop on User Mode + +Before moving to the kernel, let's nail down the four user-mode sanitizer flags with real reports, so we can easily compare them with the kernel tools later. The detailed shadow memory principles and the Heartbleed story were thoroughly explained in the previous article; here, we only provide minimal reproducible code and real terminal outputs to facilitate matching "which flag corresponds to each bug." + +The division of labor for the four flags in one sentence: `-fsanitize=address` (ASan, out-of-bounds/UAF/leaks), `-fsanitize=undefined` (UBSan, undefined behavior), `-fsanitize=memory` (MSan, uninitialized reads), `-fsanitize=thread` (TSan, data races). + +### ASan: Catching Three Types of Errors at Once + +Heap out-of-bounds, use-after-free, and memory leaks—ASan catches them all. We write minimal examples for each of the three errors separately (if we put them in one program, ASan will abort at the first error, so we won't see the latter two; hence, we split them up): + +```cpp +// uaf.cpp —— 释放后使用(use-after-free) +#include +int main() { + int* p = new int(7); + delete p; + printf("*p = %d\n", *p); // p 已 delete,悬空 + return 0; +} +``` + +Compile with `g++ -std=c++20 -O0 -g -fsanitize=address -fno-omit-frame-pointer uaf.cpp -o uaf`, and run it: + +```text +================================================================= +==118313==ERROR: AddressSanitizer: heap-use-after-free on address 0x72c9e1de0010 at pc 0x5d222d6ed26f bp 0x7ffc31d299a0 sp 0x7ffc31d29990 +READ of size 4 at 0x72c9e1de0010 thread T0 + #0 0x5d222d6ed26e in main /tmp/sanit/uaf.cpp:6 + ... +SUMMARY: AddressSanitizer: heap-use-after-free /tmp/sanit/uaf.cpp:6 in main +``` + +`-g` enables source code locations like `uaf.cpp:5` in the report. This is the deciding factor for whether ASan is usable—without debug symbols, the report is just a string of addresses, rendering it useless. It also detects out-of-bounds access on the stack. Let's switch to a cross-function stack buffer: + +```cpp +// stack_oob.cpp —— 栈缓冲越界 +#include +void fill(char* p) { // 跨函数,检测能跨栈帧 + for (int i = 0; i <= 8; ++i) p[i] = 'A'; // 合法下标 0..7,8 越界 +} +int main() { + char buf[8]; + fill(buf); + printf("done\n"); + return 0; +} +``` + +Running the binary built with the same flags: + +```text +================================================================= +==119120==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x6ec9ef2f0028 at pc 0x5f38ab644200 bp 0x7fff6db78e20 sp 0x7fff6db78e10 +WRITE of size 1 at 0x6ec9ef2f0028 thread T0 + #0 0x5f38ab6441ff in fill(char*) /tmp/sanit/stack_oob.cpp:4 + #1 0x5f38ab64429d in main /tmp/sanit/stack_oob.cpp:8 + ... +Address 0x6ec9ef2f0028 is located in stack of thread T0 at offset 40 in frame + #0 0x5f38ab644220 in main /tmp/sanit/stack_oob.cpp:6 +``` + +Note that it doesn't just tell you that the bounds were exceeded; it also tells you that "this memory is the `buf` located at offset 40 within the `main` stack frame." The stack redzone even marks the ownership of the array on the stack. This demonstrates the power of shadow memory, which we deconstructed in detail in the previous article, so we won't expand on it here. + +Memory leaks are handled by LeakSanitizer (LSan), which comes bundled with ASan. It performs a scan when the program exits: + +```cpp +// leak.cpp —— 忘记 delete +#include +int main() { + int* leak = new int(99); + *leak = 100; + printf("leak = %d (故意不 delete)\n", *leak); + return 0; +} +``` + +```text +================================================================= +==118322==ERROR: LeakSanitizer: detected memory leaks + +Direct leak of 4 byte(s) in 1 object(s) allocated from: + #0 0x7c2b9dd2d341 in operator new(unsigned long) (/usr/lib/libasan.so.8+0x12d341) + #1 0x609649f361ba in main /tmp/sanit/leak.cpp:4 +``` + +The cost of ASan is substantial: the program runs two to five times slower, and memory usage increases by three to five times. Therefore, **we must remove `-fsanitize=address` for production builds**, enabling it only during debugging and testing. This constraint might sound trivial, but in the kernel world, the same "too much overhead" issue led to the creation of completely different tools—this is the origin of KFENCE, which we will discuss later. + +### UBSan: Specialized for Undefined Behavior + +ASan manages "whether this memory can be touched," while UBSan manages "whether this operation itself is valid." Signed integer overflow, out-of-bounds array access, null pointer dereference, and misaligned shifts are undefined behaviors (UB) in the C++ standard. They might not necessarily crash, but the results are unpredictable: + +```cpp +// ub.cpp —— 三种 UB +#include +#include +int main() { + int32_t big = 2147483647; // INT32_MAX + int32_t sum = big + 1; // (1) 有符号加法溢出 → UB + int arr[4] = {0,1,2,3}; + int idx = 10; + int v = arr[idx]; // (2) 下标越界 → UBSan 的 bounds 检查 + printf("sum=%d v=%d\n", sum, v); + return 0; +} +``` + +Use `g++ -std=c++20 -O0 -g -fsanitize=undefined ub.cpp -o ub` (defaults to recover, printing all instances of undefined behavior and then continuing): + +```text +ub.cpp:6:13: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int' +ub.cpp:9:20: runtime error: index 10 out of bounds for type 'int [4]' +ub.cpp:9:9: runtime error: load of address 0x7fffed140f28 with insufficient space for an object of type 'int' +sum=-2147483648 v=0 +``` + +Here is a real pitfall to watch out for first: **UBSan and ASan can be enabled together** (`-fsanitize=address,undefined`). Many people do this because one handles memory and the other handles arithmetic, making them complementary. However, UBSan defaults to "print and continue" (recover). If you prefer to abort on the first instance of UB (which is closer to production behavior), add `-fno-sanitize-recover=all`. Conversely, ASan aborts immediately upon detection, and this behavior cannot be changed. + +### MSan: Uninitialized Reads, Clang Only + +MSan detects "use of uninitialized values," a class of errors ASan cannot catch—the memory is valid, the access is valid, but the value is garbage. The catch is: **MSan is implemented only in Clang; GCC does not support this flag at all**: + +```cpp +// msan.cpp —— 用了没初始化的变量 +#include +int main() { + int x; // 故意不初始化 + if (x) // 拿垃圾值做分支判断 → MSan 抓 + printf("x is truthy\n"); + else + printf("x is zero\n"); + return 0; +} +``` + +GCC reports an error directly: + +```text +$ g++ -std=c++20 -fsanitize=memory msan.cpp -o msan +g++: error: unrecognized argument to '-fsanitize=' option: 'memory' +``` + +Switching to Clang allows it to compile and run (`clang++ -std=c++20 -O0 -g -fsanitize=memory -fno-omit-frame-pointer msan.cpp -o msan`): + +```text +==118932==WARNING: MemorySanitizer: use-of-uninitialized-value + #0 0x58f3129f5677 (/tmp/sanit/msan+0xd7677) + ... +SUMMARY: MemorySanitizer: use-of-uninitialized-value +``` + +> **Warning**: MSan has a hard constraint—**the entire program (including all linked libraries) must be compiled with MSan instrumentation**. If you directly link a non-instrumented `libc++` or a third-party library using `clang++ -fsanitize=memory`, you will get a bunch of false positives, because MSan treats values returned by libraries as uninitialized. Therefore, MSan is rarely used in practical projects; it usually requires "rebuilding the entire toolchain with MSan" to run cleanly. This was mentioned in the previous article, but I'm emphasizing it here because KMSAN on the kernel side has similar "full-link instrumentation" requirements. + +As for TSan (ThreadSanitizer, for data races), it is mutually exclusive with ASan, has a 5–15x performance overhead, and specializes in catching concurrency bugs. The "Concurrent Programming Debugging Techniques" section in the concurrency volume has already covered this thoroughly, so we will just mark its position in the big picture here and avoid repetition. + +## So, what about the kernel? + +Once you have memorized the four user-space flags, here comes the real point of this article. **The kernel is also C code; it can also have out-of-bounds access, UAF, and data races. Can we just slap `-fsanitize=address` onto the kernel?** + +The answer is: **Yes, and the kernel actually does this, but the cost is so high that you can only enable it during debugging.** This is KASAN—Kernel AddressSanitizer. Its underlying mechanism is the same as user-space ASan (shadow memory + compile-time instrumentation), but the kernel has its own constraints: + +1. **Shadow memory occupies a large chunk of the kernel's virtual address space**. User-space ASan's shadow memory takes up "1/8 of the process address space", but the kernel carves out a significant segment of the kernel VAS directly (from `KASAN_SHADOW_START` to `KASAN_SHADOW_END`). On a 64-bit kernel, the address space is large enough (128 TB) to handle this; on 32-bit, it is much tighter, so early KASAN could only run on 64-bit, until Linus Walleij created a streamlined version for ARM-32 in 5.11. + +2. **Every memory access in the system is instrumented**. The kernel is not a process; it is the underlying layer shared by all processes. Once KASAN is enabled, system-wide performance collapses immediately—which is why `CONFIG_KASAN` is only used for debugging the kernel and is never enabled in production kernels. + +3. **It requires specific memory allocators**. The kernel uses SLAB or SLUB allocators. KASAN needs to plant red zones in allocators and mark freed pages as "poisoned" (`KASAN_SANITIZE_*`) to catch UAF/OOB immediately. This shares the same logic as user-space ASan intercepting `malloc/free`, just swapped for `kmalloc/kfree`. + +The source notes say "KASAN applies to x86_64 and AArch64, 4.x and above", so we need to verify that version number. In reality, KASAN was merged into the mainline in **Linux 4.0** (initially supporting x86_64), with AArch64 following shortly after. **5.11** added the optimized version for ARM-32. The mechanism is correct, but don't just remember it vaguely as "4.x". + +### What does KASAN look like? (Official report style) + +What does a KASAN report look like? Based on the example structure in the kernel.org dev-tools/kasan documentation, replacing the official example's `kmalloc_oob_right` with a virtual `buggy_driver_write` (where fields and hierarchy correspond exactly to the official report), it looks roughly like this: + +```text +================================================================== +BUG: KASAN: slab-out-of-bounds in buggy_driver_write+0x3e/0x60 [buggy] +Write of size 1 at addr ffff888006c42185 by task cat/1234 + +CPU: 0 PID: 1234 Comm: cat Tainted: G B +Call Trace: + dump_stack_lvl+0x49/0x63 + print_report+0x171/0x486 + kasan_report+0xb1/0x130 + buggy_driver_write+0x3e/0x60 [buggy] + ... + +Allocated by task 1234: + kasan_save_stack+0x1e/0x40 + __kasan_kmalloc+0x81/0xa0 + kmalloc_trace+0x21/0x30 + buggy_driver_init+0x2a/0x60 [buggy] + ... + +The buggy address belongs to the object at ffff888006c42180 + which belongs to the cache kmalloc-8 of size 8 +The buggy address is located 5 bytes inside of + 8-byte region [ffff888006c42180, ffff88800642188) +``` + +The report structure is nearly identical to userspace ASan: **it first states where it blew up (slab-out-of-bounds, out-of-bounds write, which driver function), then provides the allocation stack (who allocated this memory, which `kmalloc` call)**. The kernel report adds kernel-allocator-specific details like "which slab cache (`kmalloc-8`)" and "offset within the object." Once you understand userspace ASan reports, you can basically read kernel KASAN reports too. + +## Panoramic Comparison: Userspace ↔ Kernel + +Let's align both sides now. This table is the core of this article—the original note was an external PNG, so we've recreated it here in Markdown: + +| Bug Caught | Userspace Flag | Kernel Tool | Kernel Merged Version | Production Ready? | +|-----------|----------------|------------|-----------------------|-------------------| +| OOB / UAF / Double Free | `-fsanitize=address` (ASan) | **KASAN** | 4.0 (x86_64) / 5.11 (ARM-32 optimization) | No, debug only | +| Uninitialized Read | `-fsanitize=memory` (MSan, Clang only) | **KMSAN** | Patches available from 5.16, **6.1** mainline fully usable, Clang 14.0.6+ only, x86_64 only | No, huge overhead | +| Undefined Behavior (overflow/shift/OOB) | `-fsanitize=undefined` (UBSan) | **UBSAN** | Merged in 4.5 | Partially (see below) | +| Data Race | `-fsanitize=thread` (TSan) | **KCSAN** | Merged in 5.8, sampling-based | No, debug only | +| Memory Leak | ASan includes LSan | **kmemleak** / eBPF `memleak` | kmemleak has existed for a long time | Careful, false positives | +| Sampling Memory Errors | (No userspace equivalent) | **KFENCE** | **5.12** | **Yes, on by default** | +| Access Pattern Analysis (not bug detection) | (None) | **DAMON** | **5.15** | Yes, designed for production | + +There are a few key relationships in this table you must remember: + +- **ASan ↔ KASAN**: The same shadow memory concept moved to the kernel, but the cost is a total system performance collapse, so it's debug-only. +- **MSan ↔ KMSAN**: Both are Clang-only, both require full instrumentation, and both have huge overhead. The KMSAN docs explicitly state "not intended for production use, because it drastically increases kernel memory footprint and slows the whole system down." +- **UBSan ↔ UBSAN**: Kernel UBSAN was merged in 4.5, and **some of its checks (like `CONFIG_UBSAN_BOUNDS`) are enabled by default in modern distro kernels** because the overhead is low—this is one of the few kernel sanitizers that can "stay on." +- **TSan ↔ KCSAN**: Note that TSan uses compile-time full instrumentation, while KCSAN is different—it is **sampling-based** (watchpoints), so the overhead is manageable. However, it detects data races by "chance sampling," not TSan's "theoretically guaranteed detection." Merged in mainline in 5.8 (the google/kernel-sanitizers repo explicitly says "in mainline since 5.8"). + +The original note marks KMSAN as "6.1 and above"—**this version number is correct**, don't get it wrong. KMSAN was maintained as a patch series by Google's Alexander Potapenko for years; by the end of 2021, it was still just a patch branch not in mainline (official kernel.org examples ran on a patched `5.16.0-rc3+` using the google/kmsan branch, not mainline). The Google official repo (google/kmsan) README states: "Linux 6.1+ contains a fully-working KMSAN implementation which can be used out of the box," meaning **fully usable in mainline from 6.1 onwards**. So KMSAN is the latest of this batch of kernel sanitizers to hit mainline. Don't confuse "5.16 patch branch works" with "6.1 mainline"—this is the most common misinterpretation of these version numbers. + +## KFENCE: The Key Move to Production + +KASAN's problem is obvious—it's debug-only. But what do you do when your company's production kernel hits a memory bug? You can't swap a production machine for a KASAN-enabled debug kernel to reproduce it; your service would be down long before that. What's really missing is a **memory error detector with low enough overhead to stay on all the time**. + +Enter KFENCE (Kernel Electric-Fence), **merged into mainline in Linux 5.12**. Its approach is totally different from KASAN: instead of "checking every access," it uses **sampling**: + +- KFENCE maintains a fixed-size object pool (default `CONFIG_KFENCE_NUM_OBJECTS=255`, each object uses 2 pages—one for the object, one as a guard page; object pages and guard pages are interleaved in the pool, so every object page is surrounded by guard pages; the whole pool is about 2 MiB by default). +- The kernel slab allocator (`kmalloc`) is **hooked by a sampling timer into the KFENCE pool**: KFENCE has a millisecond sampling interval (boot param `kfence.sample_interval`, configurable via `CONFIG_KFENCE_SAMPLE_INTERVAL`). The next `kmalloc` after each interval is "hooked" and handed to KFENCE. +- Once in the KFENCE pool, the allocation is placed between two guard pages—any out-of-bounds read/write hits a guard page, immediately triggering a page fault, and the kernel reports the precise error and allocation stack. +- On free, KFENCE marks the page as "inaccessible"; touching it again is a use-after-free, which also triggers an immediate report. + +The cost of sampling is: **the vast majority of allocations never touch KFENCE**, so it misses most bugs—you have to run for a long time and let enough allocations flow through the pool to catch one. But the trade-off is **extremely low overhead** (officials say near-zero, production workloads barely notice it), making it the **first memory sanitizer that can stay on in a production kernel**. In fact, as long as the architecture supports it and SLAB/SLUB is on, KFENCE is on by default in many distros. + +The original note said "KFENCE must run for a long time, but the overhead is low enough to run in production"—the mechanism description is correct. We've added the version number (5.12) and the keyword "sampling," and emphasized the engineering significance of "on by default." It replaces the older `kmemcheck` (which was deleted in 4.15 because the overhead was too high and conflicted with KFENCE's philosophy). + +## DAMON: Another "Sampling" Path, But Not for Bugs + +Speaking of "sampling," we should mention DAMON (Data Access MONitor), because philosophically it's in the same category as KFENCE—**don't track everything, sample representative data**. But DAMON isn't a sanitizer; it doesn't catch bugs, it **monitors memory access patterns**: + +- **Merged into mainline in Linux 5.15**, its goal is to help developers (and the kernel itself) see "how the process is actually accessing memory," to optimize layout and guide reclamation. +- DAMON splits the target process's address space into equal-sized regions, **samples** several representative pages in each region, records access frequency, and forms a histogram. If a region is hot, it subdivides—this "smart zoom" allows it to run cheaply even on huge address spaces. +- The kernel component is the "producer" (producing access patterns), and userspace (or the kernel) is the "consumer." The consumer can even call `madvise()` based on patterns to change memory attributes—like advising the kernel to swap out confirmed cold data. + +DAMON has three interfaces: the userspace `damo` tool (from awslabs/damo), sysfs under `/sys/kernel/mm/damon/admin/`, and a kernel API for kernel developers. The old debugfs interface is deprecated. Looking at KFENCE and DAMON together, you'll see that in the 5.12~5.15 wave, the kernel systematically used "sampling" to plug the gap left by "full instrumentation is too expensive"—KFENCE catches bugs, DAMON watches patterns, both can go to production. + +## Three Layers of Defense: Placing Tools by Scenario + +Putting userspace and kernel sanitizers together, the memory safety toolchain is actually a **layered defense in depth**, each with different overhead/coverage trade-offs: + +::: tip Development Phase: Full Instrumentation, Catch Everything +For self-testing, CI, and fuzzing, **overhead is not an issue, coverage is paramount**. Userspace turns on `-fsanitize=address,undefined` (plus a separate run of `-fsanitize=thread`), and kernel debug builds turn on `CONFIG_KASAN` + `CONFIG_KCSAN` + `CONFIG_UBSAN`. This layer assumes bugs will be caught by full instrumentation, at the cost of slowing the program/system down several times—only acceptable in non-production environments. +::: + +::: tip Testing/Staging: Sampling Instrumentation, Long Runs +For pre-production, canary releases, and long load tests, **we can't accept total system collapse, but need to run long enough to expose rare bugs**. This layer uses KFENCE—sampling, low overhead, always on, letting thousands of allocations flow through the guard page pool to catch those "once in ten thousand runs" OOB and UAF bugs. Userspace lacks a direct equivalent here (Valgrind is too slow, ASan too heavy), so KFENCE's engineering value on the kernel side is particularly prominent. +::: + +::: tip Production: Default Lightweight Checks + Post-Mortem +For real production kernels, **only enable negligible-overhead checks**: KFENCE (on by default), lightweight UBSAN subsets like `CONFIG_UBSAN_BOUNDS`, plus DAMON for access pattern analysis. When accidents happen, rely on post-mortem tools—kernel oops logs, kdump/crash analysis, and eBPF's `memleak-bpfcc` to track unreleased allocations. This layer stops expecting "catching bugs on the spot" and focuses on "keeping enough evidence to investigate later." +::: + +This layering explains why the kernel maintains both KASAN and KFENCE, which seem redundant—**the same bug (e.g., UAF) is caught by KASAN during development and by KFENCE in production**. The tools aren't redundant; the scenarios don't overlap. Userspace currently only has the first layer (development instrumentation) working well; the second and third layers lack tools as mature as the kernel's. This is why "totally solving memory safety in C++ userspace" is harder than in the kernel—the kernel has KFENCE as a production safety net; when userspace has a production UAF, often the only option is to wait for it to crash and inspect the core dump. + +## By the Way: Static Analysis and Post-Mortem Tools + +Besides these runtime sanitizers, both kernel and userspace have a set of tools that **don't run the code, but read code or logs**, mentioned in the original notes. We'll wrap them up briefly here without diving deep: + +- **Static Analysis**: Kernel side has `sparse`, `smatch`, `Coccinelle`, `checkpatch.pl`; userspace has `clang-tidy`, `cppcheck`. They don't run code and have zero overhead, but can only catch "obviously problematic patterns," missing runtime-only UAF/OOB. They complement, not replace, sanitizers—static analysis enforces rules, sanitizers catch runtime issues. +- **Post-Mortem**: Kernel oops/panic logs, `kdump`/`crash` tools for dump analysis, `[K]GDB` debugging. These are forensic methods after the bug has exploded, a different stage from sanitizers that "catch bugs early." + +Userspace C++ post-mortem analysis was seen once in the "Dynamic Memory Management" chapter using `-fsanitize=address` to report leaks on exit, and in "Concurrent Debugging Tips" using TSan to locate concurrent bugs post-mortem. The whole toolchain is a **one-stop shop: Development Sanitizer → Production Lightweight Checks → Post-Mortem Analysis**. If you miss any link, the corresponding class of bugs will bite you repeatedly at that stage. + +## Summary + +This article pulled the sanitizer toolchain from userspace to kernel space. Here are the key takeaways: + +- Userspace has four flags with clear division: ASan (OOB/UAF/Leak), UBSan (Undefined Behavior), MSan (Uninitialized Read, Clang only), TSan (Data Race). Most are mutually exclusive (ASan/MSan/TSan can't run together), but UBSan can stack with ASan. Local GCC 16.1.1 / Clang 22 all ran successfully. +- Kernel has fully corresponding tools: KASAN (↔ASan, 4.0), KMSAN (↔MSan, 6.1+ mainline fully usable), UBSAN (↔UBSan, 4.5), KCSAN (↔TSan, 5.8). Mechanisms are similar, but constrained by the kernel, most are debug-only. +- **KFENCE (5.12) is the watershed**: Using "sampling + guard pages" to crush memory error detection overhead to production levels, on by default, filling the production gap left by KASAN. +- **DAMON (5.15)** follows the same sampling path but doesn't catch bugs; it monitors access patterns to guide memory optimization. +- The whole toolchain is a three-layer defense: Dev full instrumentation (KASAN/ASan) → Staging sampling (KFENCE) → Production lightweight checks + post-mortem (UBSAN subset/kdump). +- Two version numbers from the source notes have been verified: KFENCE version = 5.12 (source didn't label it, added); KMSAN source said "6.1 and above" is actually correct, verified against Google's official repo README confirming 6.1+ mainline usability—patches ran on 5.16 branch, but mainline merge was 6.1. + +Next, we continue on the performance vs. correctness line to see how compiler optimization makes code faster without changing semantics—and when it "secretly" changes semantics, making your carefully crafted concurrent code run differently than you expected. + +## Reference Resources + +- [kernel.org: Kernel Address Sanitizer (KASAN)](https://www.kernel.org/doc/html/latest/dev-tools/kasan.html) — KASAN mechanisms, config options, and example reports +- [kernel.org: Kernel Memory Sanitizer (KMSAN)](https://www.kernel.org/doc/html/latest/dev-tools/kmsan.html) — KMSAN requires Clang 14.0.6+, x86_64 only, explicitly "not for production" +- [kernel.org: Kernel Electric-Fence (KFENCE)](https://www.kernel.org/doc/html/latest/dev-tools/kfence.html) — KFENCE sampling mechanism, `CONFIG_KFENCE_NUM_OBJECTS`, production readiness +- [kernel.org: UndefinedBehaviorSanitizer (UBSAN)](https://www.kernel.org/doc/html/latest/dev-tools/ubsan.html) — Kernel UBSAN sub-checks and overhead +- [kernel.org: Kernel Concurrency Sanitizer (KCSAN)](https://www.kernel.org/doc/html/latest/dev-tools/kcsan.html) — KCSAN watchpoint-based sampling race detection +- [kernel.org: DAMON](https://www.kernel.org/doc/html/latest/admin-guide/mm/damon/usage.html) — DAMON sysfs/schemes interfaces and access pattern monitoring +- [Clang: UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) — Userspace UBSan sub-check list +- [Clang: MemorySanitizer](https://clang.llvm.org/docs/MemorySanitizer.html) — MSan full instrumentation requirements and usage diff --git a/documents/team/index.md b/documents/team/index.md index b20ea8d42..7101a462e 100644 --- a/documents/team/index.md +++ b/documents/team/index.md @@ -93,6 +93,15 @@ description: "感谢所有为本项目做出贡献的人"

报告 expected 示例编译失败问题

+ + Nyuudoukumo +
+ Nyuudoukumo Issue +

问题反馈 · 内容勘误

+

🐛 💡

+

反馈卷一「Cache 机制与内存层次」示例与解释不一致的问题（#67），促成量纲修正

+
+ ## 如何成为贡献者 diff --git a/documents/vol5-concurrency/exercises/01-bounded-queue.md b/documents/vol5-concurrency/exercises/01-bounded-queue.md index 16b712093..9ae0e39fd 100644 --- a/documents/vol5-concurrency/exercises/01-bounded-queue.md +++ b/documents/vol5-concurrency/exercises/01-bounded-queue.md @@ -46,7 +46,7 @@ Lab 0 跑通了多线程骨架——创建线程、RAII 包装、参数安全传 ## 工程脚手架（先把这个跑起来） -每个 Lab 在 vol5-labs/ 下有两份：**`templates/lab1_bounded_queue/`** 是空实现骨架（你拷贝去做），**`examples/lab1_bounded_queue/`** 是参考实现（卡住可对照，别先抄）。两份都是 standalone 工程。你要做的是 templates 那份： +每个 Lab 在 vol5-labs/ 下有两份：**`templates/lab1_bounded_queue/`** 是空实现骨架（你拷贝去做），**`examples/lab1_bounded_queue/`** 是参考实现（⚠️ 正在开发中，目前只有空声明，暂未提供完整参考；完成后卡住可对照，别先抄）。两份都是 standalone 工程。你要做的是 templates 那份： ```text templates/lab1_bounded_queue/ diff --git a/site/.vitepress/plugins/code-fold-plugin.ts b/site/.vitepress/plugins/code-fold-plugin.ts index a2ffc1f70..6d9f48901 100644 --- a/site/.vitepress/plugins/code-fold-plugin.ts +++ b/site/.vitepress/plugins/code-fold-plugin.ts @@ -25,8 +25,8 @@ import type MarkdownIt from 'markdown-it' * - code-group 内的 fence 不折叠(tab 本身已是折叠语义):core ruler 用独立 depth 计数 * 打 token.meta.inCodeGroup 标记 —— 不依赖 render 期才追加的 " active" info。 * - * 阈值 30 依据全站实测(7732 个代码块):>30 行占 ~12%,即 Issue #71 所指「一大坨」; - * 20-30 行可正常阅读的中等块不误伤。改这一处常量即可全局调档。 + * 阈值 20 依据全站实测:20 行以内的中等块保持原样、不误伤阅读,超过的(多为演示性 + * 大段输出或完整工程,即 Issue #71 所指「一大坨」)才折叠。改这一处常量即可全局调档。 */ const FOLD_THRESHOLD = 20 diff --git a/site/.vitepress/theme/components/FontSizeSwitcher.vue b/site/.vitepress/theme/components/FontSizeSwitcher.vue index 809075293..4c2aded05 100644 --- a/site/.vitepress/theme/components/FontSizeSwitcher.vue +++ b/site/.vitepress/theme/components/FontSizeSwitcher.vue @@ -1,9 +1,9 @@