Ensure a requested amount of stack is avaiable before hostcalls (#567)

add a configurable guarantee of stack space available for hostcalls

this is enforced by generating trampoline functions to ensure this guarantee can be upheld before performing calls to host code

additionally, provide documentation around hostcalls and Lucet's implementation thereof

Author: iximeow

benchmarks/lucet-benchmarks/src/seq.rs

@@ -80,10 +80,7 @@ fn instantiate_with_dense_heap<R: RegionCreate + 'static>(c: &mut Criterion) {
         region.new_instance(module).unwrap()
     }
 
-    let limits = Limits {
-        heap_memory_size: 1024 * 1024 * 1024,
-        ..Limits::default()
-    };
+    let limits = Limits::default().with_heap_memory_size(1024 * 1024 * 1024);
 
     let region = R::create(1, &limits).unwrap();
 
@@ -103,10 +100,7 @@ fn instantiate_with_sparse_heap<R: RegionCreate + 'static>(c: &mut Criterion) {
         region.new_instance(module).unwrap()
     }
 
-    let limits = Limits {
-        heap_memory_size: 1024 * 1024 * 1024,
-        ..Limits::default()
-    };
+    let limits = Limits::default().with_heap_memory_size(1024 * 1024 * 1024);
 
     let region = R::create(1, &limits).unwrap();
 
@@ -153,10 +147,7 @@ fn hello_drop_instance<R: RegionCreate + 'static>(c: &mut Criterion) {
 fn drop_instance_with_dense_heap<R: RegionCreate + 'static>(c: &mut Criterion) {
     fn body(_inst: InstanceHandle) {}
 
-    let limits = Limits {
-        heap_memory_size: 1024 * 1024 * 1024,
-        ..Limits::default()
-    };
+    let limits = Limits::default().with_heap_memory_size(1024 * 1024 * 1024);
 
     let region = R::create(1, &limits).unwrap();
 
@@ -178,10 +169,7 @@ fn drop_instance_with_dense_heap<R: RegionCreate + 'static>(c: &mut Criterion) {
 fn drop_instance_with_sparse_heap<R: RegionCreate + 'static>(c: &mut Criterion) {
     fn body(_inst: InstanceHandle) {}
 
-    let limits = Limits {
-        heap_memory_size: 1024 * 1024 * 1024,
-        ..Limits::default()
-    };
+    let limits = Limits::default().with_heap_memory_size(1024 * 1024 * 1024);
 
     let region = R::create(1, &limits).unwrap();
 

docs/src/SUMMARY.md

@@ -10,6 +10,7 @@
   - [Module integrity and authentication](./Integrity-and-authentication.md)
 - [Lucet components](./Lucet-components.md)
   - [`lucetc`](./lucetc.md)
+    - [`Hostcalls`](./lucetc/hostcalls.md)
   - [`lucet-runtime`](./lucet-runtime.md)
     - [`KillSwitch`](./lucet-runtime/killswitch.md)
   - [`lucet-wasi`](./lucet-wasi.md)

docs/src/lucetc/hostcalls.md

@@ -0,0 +1,130 @@
+# Hostcalls
+
+Hostcalls are how Lucet guests interact with the world outside the WebAssembly
+VM. For example, all functions in [WASI](https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-intro.md) are implemented in terms of
+hostcalls that can be exposed to a WebAssembly guest. This chapter discusses
+implementation details of hostcalls as Lucet implements them.
+
+Lucet implements hostcalls as imports of symbols specified by the
+`bindings.json` provided to `lucetc`. This maps namespaced functions provided
+to the WebAssembly module to symbol names the module should import when loaded.
+Functionally, `lucet-runtime` currently relies on the dynamic linker to be able
+to locate and fix up refereneces to these imported functions, and will fail to
+load a module if the dynamic linker can't resolve all imports.
+
+Hostcalls have an important intersection with safety properties Lucet seeks to
+uphold: if a fault occurs in a WebAssembly guest, it should be isolated to that
+guest, and the host should, generally, be able to continue execution. However,
+a fault in a hostcall is a fault _outside_ the WebAssembly guest, back in
+whatever code the host running `lucet-runtime` has provided. A fault here is
+well outside any guarantees WebAssembly can offer, and the only sound option
+Lucet has is to raise that issue in the host, if it has the option, and hope
+the host knows what to do with it.
+
+## Stack overflows
+
+Generally, "kick the problem to the host and hope they know what to do" works.
+For a general memory fault in host code, that gets handled no differently.
+Language-specific features, like Rust panics, work too; if a `ud2` is present
+in the hostcall's body, the `SIGILL` still causes the same kind of panic - not
+a recoverable issue, but Lucet doesn't cause new problems here.
+
+Unfortunately, memory faults like `SIGBUS` and `SIGSEGV` are typically fatal to
+host applications. Memory faults in unpredictable locations even moreso. A
+naive hostcall implementation scheme by calling import functions raises a real
+risk here: if a WebAssembly guest consumes most, but not all, of the Lucet
+guest's stack, _then_ makes a hostcall, the hostcall may consume the rest of
+the guest's stack space and experience a stack overflow.
+
+To mitigate the risk of an unknown WebAssembly guest being able to cause host
+faults essentially on-demand, Lucet guards hostcalls by a trampoline function
+that performs safety checks before actually making the call into host code.
+Currently, there is one check: is there enough guest stack space remaining to
+uphold some guaranteed amount available for hostcalls?
+
+By guaranteeing some minimum available space, the problem of hostcall stack use
+becomes the same as not overflowing stacks generally; if a host expects to
+handle stack overflows in some manner, it probably still can, and if it allows
+the system to do what it will on stack overflows, a hostcall overflowing the
+guaranteed space will still observe a normal stack overflow. Hostcalls must
+conform to the same requirements code would have without Lucet inolved, except
+that the Lucet hostcall stack reservation may be more or less than the system's
+configured thread size.
+
+The good news is that while the hostcall stack reservation is a fixed size, it
+is customizable: the field
+[hostcall_reservation](https://docs.rs/lucet-runtime/0.7.0/lucet_runtime/struct.Limits.html#structfield.heap_memory_size)
+in `Limits` specifies the space Lucet will require to be available, with a
+default of 32KiB. Lucet requires that `hostcall_reservation` is between zero
+and the guest's entire stack. Finally, a `hostcall_reservation` equal to the
+entire guest stack size is allowed, and a de facto denial of hostcalls to the
+guest - a Lucet guest will always have some stack space reserved for the
+runtime-required backstop, so the availability check would always fail.
+
+In terms of implementation, hostcall checks are done in entirely synthetic
+functions generated by lucetc, prefixed with `trampoline_`. The trampoline
+functions themselves are very simple, and have a shape like the following
+Cranelift IR:
+```
+; A trampoline has the same signature as its hostcall - args plus a vmctx.
+function %trampoline_$HOSTCALL($HOSTCALL_ARGS.., i64 vmctx) -> $HOSTCALL_RESULT {
+    gv_vmctx = vmctx
+    heap0 = static gv0
+
+block0($HOSTCALL_ARGS.., vmctx: i64):
+  ; The stack limit is recorded as part of the instance, just before the start of the guest heap.
+  stack_limit_addr = heap_addr.i64 heap0, gv_vmctx, -$STACK_LIMIT_OFFSET
+  stack_limit = load.i64 stack_limit_addr
+  ; Compare the current stack pointer to stack_limit.
+  ; The stack pointer is the LHS of this comparison, `stack_limit` the RHS.
+  stack_cmp = ifcmp_sp stack_limit
+  ; If the limit is greater than or equal to the stack pointer, there is
+  ; insufficient space for the hostcall. Branch to the fail block and trap.
+  ;
+  ; "greater than or equal" may be surprising - it might be more natural to
+  ; consider this comparison with arguments reversed; if the stack pointer is
+  ; less than the limit, there is insufficient space.
+  ;
+  ; Even phrased like this, "less than" may be surprising, but is correct since
+  ; the stack grows downward. Given an example layout:
+  ; 0x0000: start of stack - start of the stack's allocation
+  ; 0x2000: hostcall limit - reserve 0x2000 bytes
+  ; 0x8000: base of stack - initial guest stack pointer
+  ;
+  ; and knowledge that the stack grows downward, the space from 0x2000 to
+  ; 0x0000 is the reserved space, and a stack pointer below 0x2000 is in the
+  ; reserved area, and thus voids the guarantee of reserved space by Lucet.
+  brif ugte stack_cmp, stack_check_fail
+  jump do_hostcall($HOSTCALL_ARGS.., vmctx)
+
+do_hostcall($HOSTCALL_ARGS.., vmctx: i64):
+  $HOSTCALL_RESULT.. = call $HOSTCALL($HOSTCALL_ARGS.., vmctx)
+  return $HOSTCALL_RESULT
+
+stack_check_fail():
+  ; If the stack check fails, it's raised as a stack overflow in guest code.
+  ; This is reasonably close to the actual occurrance, and allows the guest to be
+  ; unwound.
+  trap stk_ovf
+```
+
+### Lucet implementation considerations
+
+Why do this trampoline and stack usage test, instead of observing failures in
+guest stack guards? The issue here is twofold: first, we can't robustly
+distinguish a stack overflow from an unlucky errant memory access for other
+reasons. If hosts are built using stack probes, stack overflows will probably
+be observed in places we expect, with patterns we can expect. But this is by no
+means a guarantee, and stack accesses might not be through the stack pointer
+directly (perhaps the address is loaded into an alternate register, and a fault
+occurs without referencing the stack pointer at all). Second, if a hostcall
+fault were to be recovered by Lucet, the runtime may have to unwind a guest that
+already has exhausted its stack space. This would require temporarily making
+the stack guard writable to support instigating a guest unwind, and probably
+motivate a second "real" guard page for safety against unforseen circumstances
+where unwinding might consume significant amounts of stack space.
+
+Given the complexity and fallibility of trying to recover from a stack overflow
+in guest code, we've concluded that pushing the error out of host code, with
+hostcalls constrained to the same kind of boundaries as would otherwise be
+expected, is a reasonable compromise.

lucet-module/src/runtime.rs

@@ -5,4 +5,5 @@
 pub struct InstanceRuntimeData {
     pub globals_ptr: *mut i64,
     pub instruction_count: u64,
+    pub stack_limit: u64,
 }

lucet-runtime/include/lucet_types.h

@@ -120,6 +120,10 @@ struct lucet_alloc_limits {
      * Size of the guest stack. (default 128K)
      */
     uint64_t stack_size;
+    /**
+     * Amount of the guest stack that must be available for hostcalls. (default 32K)
+     */
+    uint64_t hostcall_reservation;
     /**
      * Size of the globals region in bytes; each global uses 8 bytes. (default 4K)
      */

lucet-runtime/lucet-runtime-internals/src/alloc/mod.rs

@@ -432,6 +432,8 @@ pub struct Limits {
     pub heap_address_space_size: usize,
     /// Size of the guest stack. (default 128K)
     pub stack_size: usize,
+    /// Amount of the guest stack that must be available for hostcalls. (default 32K)
+    pub hostcall_reservation: usize,
     /// Size of the globals region in bytes; each global uses 8 bytes. (default 4K)
     pub globals_size: usize,
     /// Size of the signal stack in bytes. (default SIGSTKSZ for release builds, at least 12K for
@@ -484,13 +486,42 @@ impl Limits {
             heap_memory_size: 16 * 64 * 1024,
             heap_address_space_size: 0x0002_0000_0000,
             stack_size: 128 * 1024,
+            hostcall_reservation: 32 * 1024,
             globals_size: 4096,
             signal_stack_size: DEFAULT_SIGNAL_STACK_SIZE,
         }
     }
-}
 
-impl Limits {
+    pub const fn with_heap_memory_size(mut self, heap_memory_size: usize) -> Self {
+        self.heap_memory_size = heap_memory_size;
+        self
+    }
+
+    pub const fn with_heap_address_space_size(mut self, heap_address_space_size: usize) -> Self {
+        self.heap_address_space_size = heap_address_space_size;
+        self
+    }
+
+    pub const fn with_stack_size(mut self, stack_size: usize) -> Self {
+        self.stack_size = stack_size;
+        self
+    }
+
+    pub const fn with_hostcall_reservation(mut self, hostcall_reservation: usize) -> Self {
+        self.hostcall_reservation = hostcall_reservation;
+        self
+    }
+
+    pub const fn with_globals_size(mut self, globals_size: usize) -> Self {
+        self.globals_size = globals_size;
+        self
+    }
+
+    pub const fn with_signal_stack_size(mut self, signal_stack_size: usize) -> Self {
+        self.signal_stack_size = signal_stack_size;
+        self
+    }
+
     pub fn total_memory_size(&self) -> usize {
         // Memory is laid out as follows:
         // * the instance (up to instance_heap_offset)
@@ -544,6 +575,13 @@ impl Limits {
         if self.stack_size <= 0 {
             return Err(Error::InvalidArgument("stack size must be greater than 0"));
         }
+        // We allow `hostcall_reservation == self.stack_size`, a circumstance that guarantees
+        // any hostcalls will fail with a StackOverflow.
+        if self.hostcall_reservation > self.stack_size {
+            return Err(Error::InvalidArgument(
+                "hostcall reserved space must not be greater than stack size",
+            ));
+        }
         if self.signal_stack_size < MINSIGSTKSZ {
             tracing::info!(
                 "signal stack size of {} requires manual configuration of signal stacks",

lucet-runtime/lucet-runtime-internals/src/alloc/tests.rs

@@ -666,6 +666,7 @@ macro_rules! alloc_tests {
             heap_memory_size: 4096,
             heap_address_space_size: 2 * 4096,
             stack_size: 4096,
+            hostcall_reservation: 4096,
             globals_size: 4096,
             ..Limits::default()
         };

lucet-runtime/lucet-runtime-internals/src/c_api.rs

@@ -142,6 +142,8 @@ pub struct lucet_alloc_limits {
     pub heap_address_space_size: u64,
     /// Size of the guest stack. (default 128K)
     pub stack_size: u64,
+    /// Amount of the guest stack that must be available for hostcalls. (default 32K)
+    pub hostcall_reservation: u64,
     /// Size of the globals region in bytes; each global uses 8 bytes. (default 4K)
     pub globals_size: u64,
     /// Size of the signal stack in bytes. (default SIGSTKSZ for release builds, at least 12K for
@@ -168,6 +170,7 @@ impl From<&Limits> for lucet_alloc_limits {
             heap_memory_size: limits.heap_memory_size as u64,
             heap_address_space_size: limits.heap_address_space_size as u64,
             stack_size: limits.stack_size as u64,
+            hostcall_reservation: limits.hostcall_reservation as u64,
             globals_size: limits.globals_size as u64,
             signal_stack_size: limits.signal_stack_size as u64,
         }
@@ -182,13 +185,13 @@ impl From<lucet_alloc_limits> for Limits {
 
 impl From<&lucet_alloc_limits> for Limits {
     fn from(limits: &lucet_alloc_limits) -> Limits {
-        Limits {
-            heap_memory_size: limits.heap_memory_size as usize,
-            heap_address_space_size: limits.heap_address_space_size as usize,
-            stack_size: limits.stack_size as usize,
-            globals_size: limits.globals_size as usize,
-            signal_stack_size: limits.signal_stack_size as usize,
-        }
+        Limits::default()
+            .with_heap_memory_size(limits.heap_memory_size as usize)
+            .with_heap_address_space_size(limits.heap_address_space_size as usize)
+            .with_stack_size(limits.stack_size as usize)
+            .with_hostcall_reservation(limits.hostcall_reservation as usize)
+            .with_globals_size(limits.globals_size as usize)
+            .with_signal_stack_size(limits.signal_stack_size as usize)
     }
 }
 

lucet-runtime/lucet-runtime-internals/src/instance.rs

@@ -851,6 +851,23 @@ impl Instance {
     pub fn set_instruction_count(&mut self, instruction_count: u64) {
         self.get_instance_implicits_mut().instruction_count = instruction_count;
     }
+
+    #[inline]
+    pub fn set_hostcall_stack_reservation(&mut self) {
+        let slot = self
+            .alloc
+            .slot
+            .as_ref()
+            .expect("reachable instance has a slot");
+
+        let reservation = slot.limits.hostcall_reservation;
+
+        // The `.stack` field is a pointer to the lowest address of the stack - the start of its
+        // allocation. Because the stack grows downward, this is the end of the stack space. So the
+        // limit we'll need to check for hostcalls is some reserved space upwards from here, to
+        // meet some guest stack pointer early.
+        self.get_instance_implicits_mut().stack_limit = slot.stack as u64 + reservation as u64;
+    }
 }
 
 // Private API
@@ -887,6 +904,8 @@ impl Instance {
         };
         inst.set_globals_ptr(globals_ptr);
         inst.set_instruction_count(0);
+        // Ensure the hostcall limit tracked in this instance's guest-shared data is up-to-date.
+        inst.set_hostcall_stack_reservation();
 
         assert_eq!(mem::size_of::<Instance>(), HOST_PAGE_SIZE_EXPECTED);
         let unpadded_size = offset_of!(Instance, _padding);

lucet-runtime/lucet-runtime-tests/guests/guest_fault/bindings.json

@@ -0,0 +1,5 @@
+{
+    "env": {
+        "onetwothree": "onetwothree"
+    }
+}