Only allow async ABI options for async-typed functions

Author: lukewagner

design/mvp/CanonicalABI.md

@@ -777,15 +777,22 @@ class Task(Supertask):
     self.threads = []
 ```
 
-The `Task.needs_exclusive` predicate returns whether the Canonical ABI options
-indicate that the core wasm being executed does not expect to be reentered
-(e.g., because the code is using a single global linear memory shadow stack).
-Concretely, this is assumed to be the case when core wasm is lifted
-synchronously or with `async callback`. This predicate is used by the other
-`Task` methods to determine whether to acquire/release the component instance's
-`exclusive` lock.
+The `Task.needs_exclusive` method returns whether an `async`-typed function's
+ABI options indicate that the Core WebAssembly code requires serialized
+execution (with the common reason being that there is a single, global linear
+memory shadow stack). This serialized execution is implemented by
+acquiring/releasing the component-instance-wide `exclusive` lock before/after
+executing Core WebAssembly code executing on the task's *implicit thread*
+(explicit threads created by `thread.new-indirect` ignore the `exclusive` lock).
+Specifically, sync- and stackless-async-lifted (`async callback`) functions
+require the `exclusive` lock and stackful-async-lifted (`async`) functions
+ignore the `exclusive` lock (just like explicit threads). Note that
+non-`async`-typed functions' implicit threads also ignore the `exclusive` lock
+since they must complete synchronously without blocking and thus don't have to
+worry about non-LIFO stack interleaving.
 ```python
   def needs_exclusive(self):
+    assert(self.ft.async_)
     return not self.opts.async_ or self.opts.callback
 ```
 
@@ -3452,7 +3459,9 @@ present, is validated as such:
 * if `realloc` is present then `memory` must be present
 * `post-return` - only allowed on [`canon lift`](#canon-lift), which has rules
   for validation
-* 🔀 `async` - cannot be present with `post-return`
+* 🔀 `async` - is only allowed when used with an `async` function type in
+  [`canon lift`](#canon-lift) or [`canon lower`](#canon-lower) and cannot be
+  present with `post-return`
 * 🔀,not(🚟) `async` - `callback` must also be present. Note that with the 🚟
   feature (the "stackful" ABI), this restriction is lifted.
 * 🔀 `callback` - the function has type `(func (param i32 i32 i32) (result i32))`
@@ -3580,7 +3589,7 @@ function (specified as a `funcidx` immediate in `canon lift`) until the
     [packed] = call_and_trap_on_throw(callee, flat_args)
     code,si = unpack_callback_result(packed)
     while code != CallbackCode.EXIT:
-      assert(inst.exclusive is task)
+      assert(task.needs_exclusive() and inst.exclusive is task)
       inst.exclusive = None
       match code:
         case CallbackCode.YIELD:

design/mvp/Concurrency.md

@@ -74,16 +74,6 @@ the same way that they already bind to various OS's concurrent I/O APIs (such
 as `select`, `epoll`, `io_uring`, `kqueue` and Overlapped I/O) making the
 Component Model "just another OS" from the language toolchain's perspective.
 
-The new async ABI can be used alongside or instead of the existing Preview 2
-"sync ABI" to call or implement *any* WIT function type. When *calling* an
-imported function via the async ABI, if the callee [blocks](#blocking), control
-flow is returned immediately to the caller, and the callee continues executing
-concurrently. When *implementing* an exported function via the async ABI,
-multiple concurrent export calls are allowed to be made by the caller.
-Critically, both sync-ABI-calls-async-ABI and async-ABI-calls-sync-ABI pairings
-have well-defined, composable behavior for both inter-component and
-intra-component calls.
-
 In addition to adding a new async *ABI* for use by the language's compiler and
 runtime, the Component Model also adds a new `async` [effect type] that can be
 added to function types (in both WIT and raw component function type
@@ -103,6 +93,16 @@ invariant is necessary to allow non-`async` component exports to be called in
 synchronous contexts (like event listeners, callbacks, getters, setters and
 constructors).
 
+The new async ABI can be used alongside or instead of the existing Preview 2
+"sync ABI" to call or implement any `async`-typed functions. When *calling* an
+imported function via the async ABI, if the `async` callee [blocks](#blocking),
+control flow is returned immediately to the caller, and the callee continues
+executing concurrently. When *implementing* an `async` function via the async
+ABI, multiple concurrent export calls are allowed to be made by the caller.
+Critically, both sync-ABI-calls-async-ABI and async-ABI-calls-sync-ABI pairings
+have well-defined, composable behavior for both inter-component and
+intra-component calls.
+
 Because `async` function exports may be implemented with the *sync* ABI and
 then call `async` function imports using the *sync* ABI, traditional sync code
 can compile directly to components exporting `async` functions without having
@@ -873,19 +873,12 @@ JS [top-level `await`] or I/O in C++ constructors executing during `start`.
 
 ## Async ABI
 
-At an ABI level, native async in the Component Model defines for every WIT
-function an async-oriented core function signature that can be used instead of
-or in addition to the existing (Preview-2-defined) synchronous core function
-signature. This async-oriented core function signature is intended to be called
-or implemented by generated bindings which then map the low-level core async
-protocol to the languages' higher-level native concurrency features.
-
-Note that *every* WIT-level function type can be lifted and lowered using the
-async (or sync) ABI. While calling a non-`async`-typed function import using
-the async ABI will never returned that the call "blocked" (as guaranteed by the
-Component Model trapping if the callee would have blocked), the async ABI is
-still allowed to be used (for the benefit of code generators that only want
-to think about one ABI).
+At an ABI level, native async in the Component Model defines for every
+`async`-typed function a non-blocking core function signature that can be
+used instead of or in addition to the existing (Preview-2-defined) synchronous
+core function signature. This non-blocking core function signature is intended
+to be called or implemented by generated bindings which then map the low-level
+core async protocol to the languages' higher-level native concurrency features.
 
 ### Async Import ABI
 

design/mvp/Explainer.md

@@ -1328,13 +1328,10 @@ be deallocated and destructors called. This immediate is always optional but,
 if present, is validated to have parameters matching the callee's return type
 and empty results.
 
-🔀 The `async` option specifies that the component wants to make (for imports)
-or support (for exports) multiple concurrent (asynchronous) calls. This option
-can be applied to any component-level function type and changes the derived
-Canonical ABI significantly. See the [concurrency explainer] for more details.
-When a function signature contains a `future` or `stream`, validation of `canon
-lower` requires the `async` option to be set (since a synchronous call to a
-function using these types is highly likely to deadlock).
+🔀 The `async` option may only be used with `async` function types and specifies
+that the component wants to make (for imports) or support (for exports) multiple
+concurrent (asynchronous) calls. This option changes the derived Canonical ABI
+significantly; see the [concurrency explainer] for more details.
 
 🔀 The `(callback ...)` option may only be present in `canon lift` when the
 `async` option has also been set and specifies a core function that is

design/mvp/canonical-abi/definitions.py

@@ -449,6 +449,7 @@ def __init__(self, ft, opts, inst, supertask, on_start, on_resolve):
     self.threads = []
 
   def needs_exclusive(self):
+    assert(self.ft.async_)
     return not self.opts.async_ or self.opts.callback
 
   def may_block(self):
@@ -2104,7 +2105,7 @@ def thread_func():
     [packed] = call_and_trap_on_throw(callee, flat_args)
     code,si = unpack_callback_result(packed)
     while code != CallbackCode.EXIT:
-      assert(inst.exclusive is task)
+      assert(task.needs_exclusive() and inst.exclusive is task)
       inst.exclusive = None
       match code:
         case CallbackCode.YIELD:

test/async/cross-abi-calls.wast

@@ -175,62 +175,67 @@
       (export "task.return16" (func $task.return16))
       (export "task.return17" (func $task.return17))
     ))))
-    (func (export "sync-4-param") (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
+    (func (export "sync-4-param") async (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
       (canon lift (core func $core "sync-4-param"))
     )
-    (func (export "sync-5-param") (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64) (param "e" u32)
+    (func (export "sync-5-param") async (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64) (param "e" u32)
       (canon lift (core func $core "sync-5-param"))
     )
-    (func (export "sync-17-param") (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
+    (func (export "sync-17-param") async
+                                   (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
                                    (param "e" u32) (param "f" u64) (param "g" f32) (param "h" f64)
                                    (param "i" u32) (param "j" u64) (param "k" f32) (param "l" f64)
                                    (param "m" u32) (param "n" u64) (param "o" f32) (param "p" f64)
                                    (param "q" u32)
       (canon lift (core func $core "sync-17-param") (memory $memory "mem") (realloc (func $memory "realloc")))
     )
-    (func (export "sync-1-result") (result f64)
+    (func (export "sync-1-result") async (result f64)
       (canon lift (core func $core "sync-1-result"))
     )
-    (func (export "sync-16-result") (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64))
+    (func (export "sync-16-result") async (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64))
       (canon lift (core func $core "sync-16-result") (memory $memory "mem"))
     )
-    (func (export "sync-17-result") (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32))
+    (func (export "sync-17-result") async (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32))
       (canon lift (core func $core "sync-17-result") (memory $memory "mem"))
     )
-    (func (export "async-4-param") (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
+    (func (export "async-4-param") async
+                                   (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
       (canon lift (core func $core "async-4-param") async (callback (func $core "unreachable-cb")))
     )
-    (func (export "async-5-param") (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64) (param "e" u32)
+    (func (export "async-5-param") async
+                                   (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64) (param "e" u32)
       (canon lift (core func $core "async-5-param") async (callback (func $core "unreachable-cb")))
     )
-    (func (export "async-17-param") (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
+    (func (export "async-17-param") async
+                                    (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
                                     (param "e" u32) (param "f" u64) (param "g" f32) (param "h" f64)
                                     (param "i" u32) (param "j" u64) (param "k" f32) (param "l" f64)
                                     (param "m" u32) (param "n" u64) (param "o" f32) (param "p" f64)
                                     (param "q" u32)
       (canon lift (core func $core "async-17-param") async (callback (func $core "unreachable-cb")) (memory $memory "mem") (realloc (func $memory "realloc")))
     )
-    (func (export "async-1-result") (result f64)
+    (func (export "async-1-result") async (result f64)
       (canon lift (core func $core "async-1-result") async (callback (func $core "unreachable-cb")))
     )
-    (func (export "async-16-result") (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64))
+    (func (export "async-16-result") async (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64))
       (canon lift (core func $core "async-16-result") async (callback (func $core "unreachable-cb")))
     )
-    (func (export "async-17-result") (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32))
+    (func (export "async-17-result") async (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32))
       (canon lift (core func $core "async-17-result") async (callback (func $core "unreachable-cb")) (memory $memory "mem") (realloc (func $memory "realloc")))
     )
   )
   (component $Bottom
-    (import "func-4-param" (func $func-4-param (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)))
-    (import "func-5-param" (func $func-5-param (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64) (param "e" u32)))
-    (import "func-17-param" (func $func-17-param (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
+    (import "func-4-param" (func $func-4-param async (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)))
+    (import "func-5-param" (func $func-5-param async (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64) (param "e" u32)))
+    (import "func-17-param" (func $func-17-param async
+                                                 (param "a" u32) (param "b" u64) (param "c" f32) (param "d" f64)
                                                  (param "e" u32) (param "f" u64) (param "g" f32) (param "h" f64)
                                                  (param "i" u32) (param "j" u64) (param "k" f32) (param "l" f64)
                                                  (param "m" u32) (param "n" u64) (param "o" f32) (param "p" f64)
                                                  (param "q" u32)))
-    (import "func-1-result" (func $func-1-result (result f64)))
-    (import "func-16-result" (func $func-16-result (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64))))
-    (import "func-17-result" (func $func-17-result (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32))))
+    (import "func-1-result" (func $func-1-result async (result f64)))
+    (import "func-16-result" (func $func-16-result async (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64))))
+    (import "func-17-result" (func $func-17-result async (result (tuple u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32 u64 f32 f64 u32))))
     (core module $Memory (memory (export "mem") 1))
     (core instance $memory (instantiate $Memory))
     (core module $Core
@@ -408,18 +413,18 @@
       (export "sync-17-result" (func $sync-17-result))
       (export "async-17-result" (func $async-17-result))
     ))))
-    (func (export "call-sync-4-param") (result u32) (canon lift (core func $core "call-sync-4-param")))
-    (func (export "call-async-4-param") (result u32) (canon lift (core func $core "call-async-4-param")))
-    (func (export "call-sync-5-param") (result u32) (canon lift (core func $core "call-sync-5-param")))
-    (func (export "call-async-5-param") (result u32) (canon lift (core func $core "call-async-5-param")))
-    (func (export "call-sync-17-param") (result u32) (canon lift (core func $core "call-sync-17-param")))
-    (func (export "call-async-17-param") (result u32) (canon lift (core func $core "call-async-17-param")))
-    (func (export "call-sync-1-result") (result u32) (canon lift (core func $core "call-sync-1-result")))
-    (func (export "call-async-1-result") (result u32) (canon lift (core func $core "call-async-1-result")))
-    (func (export "call-sync-16-result") (result u32) (canon lift (core func $core "call-sync-16-result")))
-    (func (export "call-async-16-result") (result u32) (canon lift (core func $core "call-async-16-result")))
-    (func (export "call-sync-17-result") (result u32) (canon lift (core func $core "call-sync-17-result")))
-    (func (export "call-async-17-result") (result u32) (canon lift (core func $core "call-async-17-result")))
+    (func (export "call-sync-4-param") async (result u32) (canon lift (core func $core "call-sync-4-param")))
+    (func (export "call-async-4-param") async (result u32) (canon lift (core func $core "call-async-4-param")))
+    (func (export "call-sync-5-param") async (result u32) (canon lift (core func $core "call-sync-5-param")))
+    (func (export "call-async-5-param") async (result u32) (canon lift (core func $core "call-async-5-param")))
+    (func (export "call-sync-17-param") async (result u32) (canon lift (core func $core "call-sync-17-param")))
+    (func (export "call-async-17-param") async (result u32) (canon lift (core func $core "call-async-17-param")))
+    (func (export "call-sync-1-result") async (result u32) (canon lift (core func $core "call-sync-1-result")))
+    (func (export "call-async-1-result") async (result u32) (canon lift (core func $core "call-async-1-result")))
+    (func (export "call-sync-16-result") async (result u32) (canon lift (core func $core "call-sync-16-result")))
+    (func (export "call-async-16-result") async (result u32) (canon lift (core func $core "call-async-16-result")))
+    (func (export "call-sync-17-result") async (result u32) (canon lift (core func $core "call-sync-17-result")))
+    (func (export "call-async-17-result") async (result u32) (canon lift (core func $core "call-async-17-result")))
   )
   (instance $top (instantiate $Top))
   (instance $bottom-to-sync (instantiate $Bottom

test/async/trap-on-reenter.wast

@@ -12,13 +12,13 @@
     )
   )
   (core instance $core_inner (instantiate $CoreInner))
-  (func $a (canon lift
+  (func $a async (canon lift
     (core func $core_inner "a")
     async (callback (func $core_inner "a-cb"))
   ))
 
   (component $Child
-    (import "a" (func $a))
+    (import "a" (func $a async))
 
     (core module $Memory (memory (export "mem") 1))
     (core instance $memory (instantiate $Memory))
@@ -37,7 +37,7 @@
     (core instance $core_child (instantiate $CoreChild (with "" (instance
       (export "a" (func $a'))
     ))))
-    (func (export "b") (canon lift
+    (func (export "b") async (canon lift
       (core func $core_child "b")
       async (callback (func $core_child "b-cb"))
     ))
@@ -57,7 +57,7 @@
   (core instance $core_outer (instantiate $CoreOuter (with "" (instance
     (export "b" (func $b))
   ))))
-  (func $c (export "c") (canon lift
+  (func $c (export "c") async (canon lift
     (core func $core_outer "c")
     async (callback (func $core_outer "c-cb"))
   ))