Skip to content

04-rust-in-kernel.md

Latest commit

 

History

History
186 lines (152 loc) · 9.63 KB

04-rust-in-kernel.md

File metadata and controls

186 lines (152 loc) · 9.63 KB

Phase 3 — Improving the kernel with Rust

Goal (from the task): clone the kernel via git and try to improve it — replace some parts with Rust, expanding Rust's use, to improve stability, security and speed while avoiding the perennial problems of the C language family.

This document explains what is realistic, what we actually did, and how to go further. It is deliberately honest: rewriting core subsystems of a 40-million-line C kernel in Rust is a multi-year, community-scale effort (the upstream Rust for Linux project). What one can do concretely — and what we did — is:

  1. Enable first-class Rust support (CONFIG_RUST) in a real, freshly-cloned Linux 7.1 tree.
  2. Build the in-tree Rust example modules to prove the whole toolchain works end-to-end and produces real kernel objects.
  3. Write a new, original kernel component in safe Rust (rust_safe_counter) and integrate it into the kernel's Kconfig/Kbuild so it builds as part of the kernel — demonstrating expanding Rust usage.
  4. Boot the resulting kernel and observe the Rust code running in ring 0.

1. Why Rust in the kernel at all?

The "perennial problems of the C family" the task refers to are, concretely, the bug classes that cause the majority of kernel CVEs. Microsoft and Google have both reported that ~70 % of their serious security bugs are memory-safety issues. In the kernel these are:

C problem What it causes Rust's answer
Use-after-free / double-free RCE, privilege escalation Ownership + borrow checker; Drop/RAII frees exactly once
Buffer overflow / OOB access RCE, info-leak Bounds-checked slices; typed user-copy API
Data races Corruption, deadlock-adjacent bugs Send/Sync + lock holds the data — forgetting the lock is a compile error
NULL-pointer deref DoS / oops No implicit null; absence is Option<T>, must be handled
Uninitialised memory Info-leak Values must be initialised before use
Integer overflow → UB Subtle exploits checked_/saturating_/wrapping_ arithmetic, explicit
Forgotten error path cleanup Leaks, UAF ? + Drop clean up automatically

Crucially, Rust delivers this at zero runtime cost — the checks are at compile time, the generated machine code is on par with C. That is why it can improve security and stability without sacrificing speed, which C-vs-managed-language tradeoffs normally force.

2. What is already Rust in Linux 7.1 (the cloned tree)

Rust support has matured far beyond "hello world". In the 7.1 tree we cloned, rust/kernel/ already provides safe abstractions for large parts of the driver API surface. A sample of the modules present:

acpi  alloc  auxiliary  block  clk  configfs  cpu  cpufreq  cpumask  cred
debugfs  device  devres  dma  driver  error  faux  firmware  fs  gpu  i2c
io  ioctl  iov  irq  jump_label  kunit  list  maple_tree  miscdevice  ...

and the samples/rust/ directory ships working example drivers: misc device, configfs, debugfs, DMA, and real bus drivers for PCI, USB, I²C, platform and auxiliary devices. Upstream already has production Rust code such as the Nova (NVIDIA GSP) GPU driver scaffolding, the Apple AGX GPU driver (downstream/Asahi), null_blk, PHY drivers, and the DRM/GPU abstractions.

So "replacing parts with Rust" upstream happens at the leaf level first: new drivers and self-contained modules are written in Rust, where a bug can only hurt that driver, not the whole kernel. Core subsystems (scheduler, mm, vfs) stay C for now because the safe Rust abstractions over them are still being built.

3. What we did — rust_safe_counter

We wrote an original module, samples/rust/rust_safe_counter.rs, and wired it into samples/rust/Kconfig (CONFIG_SAMPLE_RUST_SAFE_COUNTER) and samples/rust/Makefile. It is a misc character device, /dev/rust-safe-counter, exposing a 64-bit counter through ioctls (INC / GET / RESET / HELLO).

It is small on purpose, but every line demonstrates a safety property that the equivalent C driver would have to get right by hand, every time:

  • No data races, enforced by the compiler. The shared count lives inside a Mutex<Inner>. In Rust you cannot touch the data without taking the lock — self.inner.lock() returns a guard that is the access path. There is no way to "forget the spin_lock" the way you can in C, because there is no separate unlocked pointer to the data.

  • No use-after-free, no manual kfree. The device state is a Pin<KBox<Self>>; it is freed exactly once, automatically, when the last reference drops (PinnedDrop). No goto err_free ladders, no double-free.

  • No integer-overflow UB. inc() uses count.saturating_add(1) — explicit, defined behaviour instead of C's silent wraparound/UB.

  • No out-of-bounds user copies. get() returns the value to userspace through UserSlice::new(arg, size).writer().write::<u64>(&value) — a typed copy that cannot exceed the size the ioctl declared.

  • No unhandled error paths. Registration uses try_pin_init! + ?; if the misc device fails to register, the half-built object is torn down correctly and the error propagates — there is no partially-initialised device left behind.

Because it is built-in (=y), its init() runs during boot and prints to the kernel log, which is how we verify "Rust is really executing in ring 0" without needing userspace module tooling. (See 06-results.md for the captured dmesg.)

4. The realistic path to "replacing C with Rust"

If you wanted to take this further on this machine, the pragmatic order is:

  1. New leaf drivers in Rust — the lowest-risk, highest-value step. Anything you would write as a new misc/platform/PCI driver, write in Rust using the abstractions in rust/kernel/. A bug there cannot corrupt the core kernel.
  2. Rewrite an isolated, self-contained driver that already exists in C and is a frequent source of bugs (e.g. a parser of untrusted input — a filesystem for a removable format, a network protocol helper). These are exactly where memory-safety bugs cluster.
  3. Build/extend safe abstractions (rust/kernel/*.rs) over a C subsystem you want to reach, then write the consumer in Rust. This is the upstream model: the unsafe C FFI is confined to a thin, audited abstraction layer; everything above it is safe.
  4. Leave the hot, mature core (scheduler/mm/vfs) in C until the community's safe abstractions for it land. Rewriting them now would add risk, not remove it — contradicting the stability goal.

Why not "rewrite the whole kernel in Rust"?

  • The safe abstractions for many subsystems don't exist yet — you'd be writing oceans of unsafe, which throws away the very guarantees that justify the work.
  • You'd fork away from upstream and inherit the entire maintenance burden.
  • The mature C core is not where most exploitable bugs are; new/leaf code is. Targeting Rust there is where the security ROI actually is.

5. Toolchain notes (what makes the build work)

  • rustc 1.91.1 — the kernel requires rustc >= 1.85.0 (it enforces only a minimum); 1.91.1 is recent but safely within the tested range. Pinned to the clone with rustup override set 1.91.1.
  • bindgen 0.71.1 (bindgen-0.71 from Ubuntu) — kernel minimum is 0.71.1. Generates the Rust↔C bindings from kernel headers.
  • rust-src for 1.91.1 — the kernel recompiles core/alloc itself.
  • LLVM=1 (clang/lld 18) — Rust-for-Linux strongly prefers a single LLVM-based toolchain so bindgen's libclang and the C compiler agree on ABI/layout. (Our bindgen links libclang-20 while clang is 18 — this is a non-fatal warning; rust_is_available.sh still reports "Rust is available!".)
  • All Rust-aware make invocations therefore carry: make LLVM=1 BINDGEN=bindgen-0.71 ...

6. Results — it actually runs in ring 0

Metric Value
Build result exit 0 (9 min 25 s, bzImage)
bzImage 15 MB (15 139 840 B)
Version 7.1.0-rust-dirty (dirty = our Kconfig/Makefile/source additions)
Toolchain clang 18 + LLD + rustc 1.91.1 + bindgen 0.71.1
Our module object samples/rust/rust_safe_counter.o — compiled clean, no errors
CONFIG_RUST / CONFIG_SAMPLE_RUST_SAFE_COUNTER y / y

Boot test (q35, KVM, 2 vCPUs) — the four built-in Rust modules announce themselves at boot, then our userspace test drives the driver:

rust_minimal: Rust minimal sample (init)
rust_minimal: Am I built-in? true
rust_misc_device: Initialising Rust Misc Device Sample
rust_safe_counter: init — safe Rust device coming up        <-- our module
rust_safe_counter: built-in = true
rust_print: [rust_print_main.rs:35:5] c = "hello, world"
x86/mm: Checked W+X mappings: passed, no W+X pages found.
...
misc rust-safe-counter: opened
misc rust-safe-counter: hello from safe Rust
misc rust-safe-counter: reset
misc rust-safe-counter: count -> 1
misc rust-safe-counter: count -> 2
misc rust-safe-counter: count -> 3
[rust-test] counter value = 3 (expected 3) -> PASS          <-- our driver works
reboot: Power down

The userspace program (artifacts/rust/rust_test_init.c) opened /dev/rust-safe-counter, issued HELLO/RESET/INC×3/GET via ioctl, and read back exactly 3 — every one of those calls executed our safe Rust code in the kernel, with the shared counter protected by a Mutex the compiler forced us to hold. That is the whole thesis in one boot: Rust code, with C-free safety guarantees, running as a real kernel driver.

Artifacts: artifacts/rust/ (bzImage, config, boot.log, rust_safe_counter.rs, rust_test_init.c).