Skip to content

Commit 08caef6

Browse files
committed
Update playground-how-it-works for on-demand WASM module loading
The playground now loads only the coreutils multicall binary eagerly and fetches the standalone modules (grep, find/locate/updatedb, diff/cmp, sed) on demand. Update the diagrams, components table, dispatch notes and add an on-demand loading section to match.
1 parent d203905 commit 08caef6

1 file changed

Lines changed: 36 additions & 8 deletions

File tree

content/playground-how-it-works.md

Lines changed: 36 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,20 +6,22 @@ template = "page.html"
66
<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>
77
<script>mermaid.initialize({ startOnLoad: true, theme: 'dark' });</script>
88

9-
The [uutils playground](/playground) lets you run real Rust coreutils directly in your browser, with no server, no installation, and no network round-trips after the initial page load. This page explains the architecture behind it.
9+
The [uutils playground](/playground) lets you run real Rust coreutils directly in your browser, with no server and no installation. This page explains the architecture behind it.
1010

1111
## High-Level Overview
1212

1313
<pre class="mermaid">
1414
flowchart LR
1515
A["Browser"] -->|"1. Load page"| B["Zola static site"]
16-
B -->|"2. Fetch WASM binary"| C["uutils.wasm<br/>(~10 MB)"]
16+
B -->|"2. Fetch core binary"| C["uutils.wasm<br/>(coreutils multicall)"]
1717
A -->|"3. User types command"| D["JavaScript shell"]
18-
D -->|"4. Execute via WASI"| E["WebAssembly runtime"]
18+
D -->|"4a. On-demand fetch<br/>(grep, find, diff, sed…)"| F["Standalone<br/>WASM modules"]
19+
D -->|"4b. Execute via WASI"| E["WebAssembly runtime"]
20+
F --> E
1921
E -->|"5. Output"| A
2022
</pre>
2123

22-
Everything runs **client-side**. Once the WASM binary is downloaded, the playground works entirely offline.
24+
Everything runs **client-side**. The page loads only the **coreutils multicall binary** up front; the optional standalone modules (`grep`, `find`/`locate`/`updatedb`, `diff`/`cmp`, `sed`) are fetched **on demand** the first time you use them — either by clicking their "Load" button or simply by running the command. Once a module is downloaded, it works entirely offline.
2325

2426
## Architecture
2527

@@ -50,7 +52,8 @@ flowchart TB
5052
| **[xterm.js](https://xtermjs.org/)** | Terminal emulator rendered in the browser. Handles cursor, colors, input, scrollback. |
5153
| **JavaScript shell** | Parses command lines, manages pipes, handles builtins (`help`, `clear`, `cd`, `locale`), and dispatches to WASM. |
5254
| **[browser_wasi_shim](https://github.com/bjorn3/browser_wasi_shim)** | Implements the WASI (WebAssembly System Interface) in JavaScript so that uutils can perform I/O operations. |
53-
| **uutils.wasm** | The actual Rust coreutils, compiled with the `feat_wasm` feature to a single WASM binary containing 60+ commands. |
55+
| **uutils.wasm** | The Rust coreutils, compiled with the `feat_wasm` feature to a single multicall WASM binary containing 60+ commands. This is the only binary loaded eagerly at page load. |
56+
| **Standalone modules** | Separate uutils projects shipped as their own WASM binaries, **loaded on demand**: `grep.wasm` ([uutils/grep](https://github.com/uutils/grep)), `find.wasm`/`locate.wasm`/`updatedb.wasm` ([uutils/findutils](https://github.com/uutils/findutils)), `diffutils.wasm` providing `diff`/`cmp` ([uutils/diffutils](https://github.com/uutils/diffutils)), and `sed.wasm` ([uutils/sed](https://github.com/uutils/sed)). |
5457
| **Virtual filesystem** | An in-memory filesystem backed by WASI shim `PreopenDirectory`, pre-populated with sample files. Persists across commands within a session. |
5558

5659
## Lifecycle of a Command
@@ -95,11 +98,14 @@ sequenceDiagram
9598
Key details:
9699

97100
- **Pipeline execution**: each pipe stage is a fresh WASM instantiation. The stdout of one stage becomes the stdin of the next.
98-
- **Command dispatch**: every command goes through `["coreutils", command, ...args]` - the WASM binary is a multicall binary, similar to BusyBox.
101+
- **Command dispatch**: coreutils commands go through `["coreutils", command, ...args]` - the core WASM binary is a multicall binary, similar to BusyBox. Standalone modules (`grep`, `find`, `diff`, `sed`…) are invoked **directly by their own name** as `argv[0]`, since each is its own binary rather than a multicall entry.
102+
- **On-demand loading**: if a command lives in a standalone module that hasn't been fetched yet, the shell loads that module first (printing a `loading <module>… done` notice), then runs the command.
99103
- **Path resolution**: relative paths are resolved against a virtual `cwd` maintained by the JS shell.
100104

101105
## WASM Loading & Initialization
102106

107+
The playground splits its WASM into one eagerly-loaded core binary and several optional modules fetched lazily, keeping the initial page download small.
108+
103109
<pre class="mermaid">
104110
flowchart TB
105111
A["Page load"] --> B{"SharedArrayBuffer<br/>available?"}
@@ -118,10 +124,30 @@ flowchart TB
118124
K -->|No| M["Show prompt"]
119125
</pre>
120126

121-
- The WASM binary is compiled with `WebAssembly.compileStreaming()` for best performance, with a fallback to `arrayBuffer()` if the server doesn't set `application/wasm` content-type.
122-
- Commands are disabled until the WASM binary finishes loading. The terminal shows a loading message and a prompt appears once it's ready.
127+
- Only the **coreutils multicall binary** loads eagerly. The standalone modules (`grep`, `find`/`locate`/`updatedb`, `diffutils`, `sed`) are **not** part of this startup fetch.
128+
- WASM binaries are compiled with `WebAssembly.compileStreaming()` for best performance, with a fallback to `arrayBuffer()` if the server doesn't set the `application/wasm` content-type.
129+
- Commands are disabled until the core binary finishes loading. The terminal shows a loading message and a prompt appears once it's ready.
123130
- The `SharedArrayBuffer` polyfill stub prevents `ReferenceError` in browsers without cross-origin isolation headers.
124131

132+
### On-Demand Loading of Standalone Modules
133+
134+
<pre class="mermaid">
135+
flowchart TB
136+
A["User runs grep/find/diff/sed<br/>(or clicks its Load button)"] --> B{"Module already<br/>compiled?"}
137+
B -->|Yes| F["Run command"]
138+
B -->|No| C{"Fetch in flight?"}
139+
C -->|Yes| D["Share the existing<br/>in-flight fetch"]
140+
C -->|No| E["fetch + compileStreaming<br/>/wasm/&lt;module&gt;.wasm"]
141+
D --> G["Dispatch<br/>uutils:program-loaded"]
142+
E --> G
143+
G --> F
144+
</pre>
145+
146+
- Each module is fetched **once**, the first time it's needed. Concurrent callers share a single in-flight fetch, and the compiled module is cached for the rest of the session.
147+
- A single module can back several commands: `diffutils.wasm` provides both `diff` and `cmp`, and the **"Load" button for `find`** brings in `find`, `locate` and `updatedb` together.
148+
- On success the page dispatches a `uutils:program-loaded` event, which the "Load" buttons listen for so their label flips to `✓ <name> loaded` — whether the module was loaded by the button or by running the command.
149+
- If a module's binary isn't present (e.g. local dev without a CI build), the command reports that it's unavailable instead of breaking the terminal.
150+
125151
## Command Parsing & Pipes
126152

127153
The shell implements a simple but functional parser:
@@ -192,6 +218,8 @@ flowchart LR
192218

193219
Utilities are excluded when they depend on OS-level syscalls not available in WASI - for example, `df` needs filesystem stats, `du` needs directory traversal with metadata, and `chown`/`chcon` need permission and SELinux APIs.
194220

221+
> **Note:** `grep`, `find`/`locate`/`updatedb`, `diff`/`cmp` and `sed` are **not** part of the coreutils `feat_wasm` set — they live in separate uutils projects ([grep](https://github.com/uutils/grep), [findutils](https://github.com/uutils/findutils), [diffutils](https://github.com/uutils/diffutils), [sed](https://github.com/uutils/sed)) and are compiled to their own WASM modules, loaded on demand as described above. (`xargs` is intentionally absent: it must spawn child processes, which the browser WASI sandbox can't do.)
222+
195223
### Multicall Binary: How Command Dispatch Works
196224

197225
<pre class="mermaid">

0 commit comments

Comments
 (0)